RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

arXiv:2609.28850 · cs.AI, cs.LG, cs.SE · Submitted 2026-09-23 · Read on arXiv

cs.AI, cs.LG, cs.SE

Submitted: 2026-09-23

Updated: 2026-09-23

Code: https://github.com/mithils3/reclaim

Terminology

• Benchmark — 117× in this paper · explained in 2 episodes
• Provenance — 36× in this paper · explained in 3 episodes
• Accuracy — 32× in this paper · explained in 1 episode
• Verification — 29× in this paper · explained in 2 episodes
• Benchmarking — 14× in this paper · explained in 3 episodes
• Trace — 14× in this paper · explained in 2 episodes
• AI Agents — 12× in this paper · explained in 2 episodes
• AUC — 10× in this paper · explained in 2 episodes
• Perplexity — 9× in this paper · explained in 3 episodes
• International Conference on Machine Learning — 8× in this paper
• Core — 6× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 6× in this paper
• Differentially Private — 5× in this paper · explained in 3 episodes
• Generative AI — 5× in this paper · explained in 2 episodes
• Average Precision — 4× in this paper
• Failure Modes — 4× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• In Conference on Language Modeling — 3× in this paper
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• Meeting of the Association for Computational Linguistics — 3× in this paper
• Aggregation — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Decode — 2× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
• Failure Taxonomy — 2× in this paper · explained in 2 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• International Conference on Software Engineering — 2× in this paper
• LLM judge — 2× in this paper · explained in 2 episodes
• Mean Absolute Scaled Error — 2× in this paper
• Minimal Reproduction Example — 2× in this paper
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Reward Hacking — 2× in this paper · explained in 8 episodes
• Transactions on Machine Learning Research — 2× in this paper
• Transformer — 2× in this paper · explained in 2 episodes
• alpha — 2× in this paper · explained in 1 episode
• Accountability — 1× in this paper · explained in 2 episodes
• Actor — 1× in this paper · explained in 3 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Agent Harness — 1× in this paper · explained in 1 episode
• Agentic AI — 1× in this paper · explained in 1 episode
• Agentic AI systems — 1× in this paper · explained in 1 episode
• Area Under ROC Curve — 1× in this paper
• Combinatorial Optimization — 1× in this paper · explained in 2 episodes
• Empirical Methods in Natural Language Processing — 1× in this paper
• Grounding — 1× in this paper · explained in 3 episodes
• Hamming Distance — 1× in this paper · explained in 2 episodes
• In Job Scheduling Strategies for Parallel Processing — 1× in this paper
• KV cache — 1× in this paper · explained in 3 episodes
• Multi-Agent System — 1× in this paper · explained in 5 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Rectified Flow — 1× in this paper · explained in 3 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Total Average Return — 1× in this paper
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Improving Generalization of Neural Combinatorial Optimization for Vehicle Routing Problems via Test-Time Projection Learning
• MirrorCode: AI can rebuild entire programs from behavior alone
• From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
• AI Coding Agents Can Reproduce Social Science Findings
• SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
• Recovering Wasted Compute in Autoresearch Agents
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
• Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
• RESCORE: LLM-Driven Simulation Recovery in Control Systems Research Papers
• Exploring the use of AI authors and reviewers at Agents4Science
• On Randomness in Agentic Evals
• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
• Accounting for Variance in Machine Learning Benchmarks
• You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
• Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
• Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection