Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

arXiv:2609.33812 · cs.SE, cs.LG · Submitted 2026-09-27 · Read on arXiv

cs.SE, cs.LG

Submitted: 2026-09-27

Updated: 2026-09-27

Code: https://github.com/earino/identical-runs-different-results

Project page: https://szilard.github.io/xgboost-autoresearch

Terminology

• AUC — 36× in this paper · explained in 2 episodes
• Benchmark — 17× in this paper · explained in 2 episodes
• Trace — 10× in this paper · explained in 2 episodes
• International Conference on Learning Representations — 4× in this paper
• Benchmarking — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• AI Agents — 2× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Feature Engineering — 2× in this paper · explained in 3 episodes
• Reward Hacking — 2× in this paper · explained in 8 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes

Sources

• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• On Randomness in Agentic Evals
• Accounting for Variance in Machine Learning Benchmarks
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows
• Evaluating Large Language Models Trained on Code
• Generalization in Adaptive Data Analysis and Holdout Reuse
• An Empirical Study of Harness Design for Coding Agents
• Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Towards a Science of AI Agent Reliability
• When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
• Establishing Best Practices for Building Rigorous Agentic Benchmarks

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties