ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs

ARAC: Benchmarking Auto-Research’s Alignment and Completeness on End-to-End Researchs proposes ARAC-Bench, a Researcher-Mimicking Evaluation framework that shifts the objective from matching final answers to reproducing high-quality human research processes. The framework operates through two synergistic components: the Academic Cognition Skills (ACS) system, which is the first to transform implicit reviewer expertise into stage-calibrated, quantifiable rubrics, and a three-stage capability diagnostic protocol that decomposes the research process under strict modular constraints into three traceable, mutually independent dimensions: Proposal, Experiment, and Synthesis.

The ACS is derived from core scientific questions extracted from 7,000 AI papers accepted at NeurIPS, ICLR, and ICML over the past two years, along with cognitive strategies employed by authors and reviewers during rebuttals and discussions. The ACS knowledge base encompasses five major themes: Large Language Models, Multimodal Learning, Diffusion Models, Reinforcement Learning, and Deep Learning, and 121 sub-themes. ACS evaluation follows a progressive operational pathway: it begins with a comprehensive analysis of the research problem’s context, from which a set of highly relevant core questions is extracted, then systematically matched against a predefined library of skill standards, with the top five most closely aligned skill standards selected as the primary basis for final assessment.

The three-stage diagnostic protocol enforces strict temporal isolation: the literature search and knowledge base access of all evaluated frameworks are hard-truncated to mid-2025, physically eliminating any possibility that a model could directly retrieve the real papers serving as Gold References. The Proposal stage scores 40 points and includes Related Work Scoring (5 points, using the cited reference list of the original paper as ground truth), Proposal Details Scoring (25 points, entirely driven by ACS with three-level anchor scoring for each skill point), and Benchmark Selection Scoring (10 points, subdivided into selection rationality and accessibility). The Experiment stage scores 35 points and includes Code Implementation Scoring (30 points, using a standard module library containing 2,869 functional units scored on functional correctness, module completeness, and code robustness, with missing modules scored as 0) and Hyperparameter Setting Scoring (5 points, assessing whether settings conform to domain priors). The Synthesis stage scores 25 points and includes Basic Analysis Scoring (10 points, evaluating completeness of Introduction, Related Work, and Preliminaries) and Methodological Deep Analysis Scoring (15 points, driven by attribution reasoning and critical synthesis skills via a three-layer progressive verification protocol).

Systematic evaluation of 11 state-of-the-art frameworks yields a best alignment score of only 67.9 of 100, revealing a significant gap in simulating rigorous human methodology. The best-performing framework achieved an overall alignment of merely 67.9%, indicating a significant gap of over 32 percentage from methodological completeness. Substantial capability differentiation was observed among frameworks: frameworks like ARIS, AI-v2 and AutoSci demonstrated relative advantages in the Proposal and Synthesis stages, but exhibited cliff-like declines in the Experiment stage; 7 frameworks like AutoSci, Dr.Claw and Claw-AI clustered in the 50–60 range.

Validation against Ph.D. Candidates rankings shows a strong correlation of 0.8141, confirming that ARAC-Bench reliably reflects the dimensions researchers truly value. Using the human comprehensive ranking as the benchmark, Pearson correlation coefficients for ARAC-Bench scores achieved 0.8788, 0.6606, and 0.9030 for the Proposal, Experiment, and Synthesis dimensions respectively. The high correlations for Proposal and Synthesis indicate that the evaluation criteria for these two stages are already highly aligned with domain experts’ cognitive judgments; the correlation for the Experiment dimension, while relatively lower, remains at an acceptable moderately strong level.

Ablation studies demonstrated that under the condition of removing Inspiration and retaining only domain background, the average Idea score dropped from 14.02 to 11.14, an average decrease of 20.54%. Notably, higher-scoring frameworks suffered more significant impact from Inspiration removal: AutoResearchClaw’s decline reached 30.38%, far exceeding EvoScientist’s 11.56%. This nonlinear effect indicates that stronger frameworks do not merely read clues from Inspiration more effectively, but rather leverage them more effectively as cognitive pivots for reasoning expansion. Integration of open-source academic skill packages into the ClaudeCode framework produced relatively notable improvements in the Proposal and Synthesis stages, while no significant improvement was observed in the Experiment stage.

ARAC-Bench provides not only a fine-grained diagnostic tool but also a scalable reward signal for training the next generation of autonomous research systems. The paper concludes that the best system achieves only 67.9 overall alignment, with the Experiment stage constituting the primary bottleneck. Notably, strong coding capability alone, as exhibited by general-purpose tools, does not translate into scientific reasoning depth, confirming that methodological completeness is the true frontier for autonomous research.

Improvements to AI Systems:

1. Stage-Calibrated Rubric Integration: Implement the ACS (Academic Cognition Skills) system as a dynamic, internal scoring module within autonomous research agents. The AI can self-assess its Proposal, Experiment, and Synthesis outputs against the 121 sub-theme skill standards, enabling real-time correction of methodological gaps before final submission.

2. Temporal Isolation Enforcement: Hard-truncate the AI’s training data and retrieval access to mid-2025, mirroring ARAC-Bench’s protocol. This forces the system to generate novel hypotheses and experimental designs from first principles rather than pattern-matching existing papers, improving originality and reducing data leakage risks.

3. Experiment-Stage Specialized Module: Since the Experiment stage is the primary bottleneck (scoring 35 points with code implementation at 30), integrate a dedicated code-verification layer that uses the 2,869 functional unit library. The AI can pre-check its own code for functional correctness, module completeness, and robustness, flagging missing modules as zero-score risks before execution.

4. Inspiration-Driven Cognitive Pivoting: Enhance reasoning expansion by explicitly modeling Inspiration as a separate input channel (not just domain background). The AI should be trained to use external clues as cognitive pivots—generating multiple divergent research directions from a single inspiration point, then pruning via ACS-based scoring. This addresses the observed 20-30% performance drop when inspiration is removed.

5. Three-Layer Progressive Verification for Synthesis: Implement a nested verification protocol for deep analysis: (a) attribution reasoning (trace every claim to a specific source or experiment), (b) critical synthesis (identify contradictions and gaps across related works), and (c) methodological justification (explain why chosen methods are superior to alternatives). This directly targets the Synthesis stage’s 15-point deep analysis component.

6. Cross-Stage Dependency Tracking: Build a feedback loop where Proposal-stage decisions (e.g., benchmark selection) are automatically validated against Experiment-stage feasibility (e.g., code availability in the standard module library). This prevents the cliff-like declines seen in frameworks like ARIS and AI-v2, which excel in early stages but fail during implementation.

7. Human-Aligned Reward Shaping: Use the validated correlation coefficients (0.8788 for Proposal, 0.6603 for Synthesis) to weight reward signals in reinforcement learning. Prioritize Proposal and Synthesis improvements first (as they align with human judgment), then iteratively refine Experiment-stage rewards using the moderately strong 0.6606 correlation as a baseline.

What the Improved AI System Can Do:

• Self-Diagnose Research Quality: Automatically generate a stage-by-stage scorecard (Proposal/Experiment/Synthesis) before final output, identifying specific skill deficiencies (e.g., weak benchmark accessibility rationale or missing code robustness check) and suggesting corrective actions.

• Generate Methodologically Complete Research Proposals: Produce proposals that not only match human-level related work citation (using ground-truth reference lists) but also justify benchmark choices with explicit rationality and accessibility arguments, scoring 40/40 on the Proposal stage.

• Execute Experiments with Verified Code: Write code that passes functional correctness checks against the 2,869-unit library, with automatic fallback to alternative implementations when a module is missing, reducing Experiment-stage score loss.

• Produce Synthesis with Traceable Reasoning: Generate analyses where every claim is attributed to a specific experiment or citation, and where critical synthesis explicitly addresses contradictions between prior works—matching the three-layer verification protocol.

• Adapt to Inspiration Sensitivity: When given a vague inspiration (e.g., improve diffusion model efficiency), the system will generate 5-10 diverse research angles, score each against ACS rubrics, and select the most methodologically sound path—avoiding the 30% performance collapse seen in current top frameworks.

• Achieve >80% Alignment with Human Researchers: By integrating the validated scoring dimensions and temporal isolation, the system can close the 32-point gap, targeting an overall alignment score above 80/100, with particular gains in the Experiment stage (from 60% to >75% correlation with expert judgment).