From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

arXiv:2608.27442 · cs.SE, cs.AI, cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.SE, cs.AI, cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Code: https://github.com/DeepSoftwareAnalytics/MCR-bench

Terminology

• Benchmark — 37× in this paper · explained in 2 episodes
• Benchmarking — 17× in this paper · explained in 3 episodes
• Pull Request — 15× in this paper
• Consistency — 11× in this paper · explained in 3 episodes
• Accuracy — 9× in this paper · explained in 1 episode
• F1 Score — 9× in this paper · explained in 3 episodes
• Verification — 9× in this paper · explained in 2 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• False Negatives — 5× in this paper
• International Conference on Software Engineering — 5× in this paper
• False Positives — 4× in this paper
• Core — 3× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Sycophancy — 2× in this paper · explained in 4 episodes
• Actionability — 1× in this paper · explained in 3 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Analyzer — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Conference on Source Code Analysis and Manipulation — 1× in this paper
• Contextual Understanding — 1× in this paper · explained in 2 episodes
• Elias — 1× in this paper · explained in 2 episodes
• Error Taxonomy — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• International Conference on Mining Software Repositories — 1× in this paper
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Multi-agent Collaborative Framework — 1× in this paper · explained in 2 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• Quadratic Weighted Kappa — 1× in this paper
• Semantic Understanding — 1× in this paper · explained in 2 episodes
• Temporal Reasoning — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle
• CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects
• Retrieval-Augmented Code Review Comment Generation
• Deep Assessment of Code Review Generation Approaches: Beyond Lexical Similarity
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuning
• StarCoder 2 and The Stack v2: The Next Generation
• ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation
• When More Retrieval Hurts: Retrieval-Augmented Code Review Generation
• Kimi K2: Open Agentic Intelligence
• Towards an Understanding of Context Utilization in Code Intelligence
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models
• Qwen3 Technical Report
• Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
• SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation
• Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
• AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
• HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
• A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends
• MemoryBank: Enhancing Large Language Models with Long-Term Memory

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties