LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform

Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. The gist: even the strongest current systems win only 23.0% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60%.

LitReview Arena Platform and Protocol

The paper introduces LitReview Arena, a battle-style evaluation platform designed with a structured protocol tailored specifically to literature review quality. This protocol involves domain experts with AI paper-writing experience comparing anonymized drafts matched to topics within their expertise and providing dimension-wise outcomes over five specific criteria. The authors argue that existing evaluations fail because they lack the necessary subjective, dynamic human preference signal required for assessing non-verifiable qualities like structure and research directions.

Benchmark Construction (LitReviewBench)

The LitReviewBench benchmark is constructed in three stages: (1) defining a survey draft generation task from high-quality survey literature and mapping instances into a consistent topic taxonomy; (2) collecting blind expert preferences via pairwise comparisons on the LitReview Arena; and (3) freezing the resulting arena logs into a versioned offline benchmark that supports direct testing and standardized leaderboards. The source pool for topics is curated from OpenAlex papers, focusing on recent surveys with high citation counts, ensuring underlying research questions are valuable.

Evaluation Dimensions

The protocol defines five tailored dimensions to capture literature review quality: D1-literature coverage, D2-claim support, D3-paper structure, D4-research suggestions quality, and D5-overall utility. These dimensions explicitly capture both citation-facing criteria and the non-verifiable dimensions that dominate research utility. The atomic unit for evaluation is a battle record containing the topic query, paired drafts, and expert outcomes for each dimension using a four-way outcome set: A (Draft A better), B (Draft B better), Tie (both equally good), and BothBad (neither acceptable).

Results on SOTA Models and Agents

The study collected approximately 3k expert judgments. When competing directly against human drafts, state-of-the-art models win only 23.0% of decisive matches on D5 Overall Utility. The largest gaps are observed in Paper Structure (D3) and Research Suggestions (D4), which trail human experts by roughly 200 points, indicating that coherent landscape organization and non-obvious gap identification remain difficult even for strong systems. Agentic models substantially outperform pure language models across dimensions, but at a substantial test-time computing cost—on average consuming roughly 15× the token budget for the same topic.

Expert-Aligned Evaluator (LitJudge)

To make expert-aligned evaluation practical at low ongoing cost, the authors release LitJudge, an expertcalibrated evaluator. Using LitReviewBench as a calibration signal, LitJudge improves expert-ranking alignment from Spearman’s ρ ≈ 0.467 to ρ ≈ 0.792 on D5 Overall Utility, comparable to inter-expert consistency. The calibrator conditions on case context that matches the test instance in structure and topic, including Structure-Similar Examples, Content-Similar Examples, and Gap Anchors derived from expert drafts. This enables scalable offline assessment with low marginal cost.

Implications for Model Development

The findings suggest that utility depends less on citation volume than on conceptual coherence and non-trivial future directions, as D3 and D4 are the strongest predictors of expert preference (Spearman correlations with D5 are 0.99 for D3 and 0.96 for D4). The paper concludes that future literature review agents should move from broad paper collection toward defensible synthesis, grounded claims, and field-level reasoning. The LitReview Arena, LitReviewBench, and LitJudge align development signals with actual researcher needs by turning expert peer review preferences into a scalable and repeatable benchmark interface.

Limitations

The study notes several limitations: the main benchmark is anchored in AI topics; broader validation across fields with different evidential norms remains future work; expert-calibrated preferences can entrench biases if structure matching favors familiar rhetorical forms or if gap anchors over-represent fashionable directions, potentially amplifying mainstream academic styles. Additionally, the current benchmark does not cover long-horizon settings such as maintaining living reviews. Complementary evaluations, including targeted factual audits and longitudinal update tasks, are promising directions.

Impact Statement

This work aims to improve the evaluation of AI systems that generate scientific literature reviews by making assessment more transparent, reproducible, and aligned with expert judgment. The benchmark and LitJudge may help researchers identify weaknesses in coverage, claim support, structure, and research suggestions before such systems are used in research workflows. LitJudge should therefore be used as a development and triage signal rather than a replacement for human expert review.

Here are specific improvements to AI systems based on the LitReview Arena framework, along with what those improved systems can achieve:

1. Improve Literature Review Agents by shifting focus from mere retrieval/fluency to Argumentative Synthesis and Gap Identification. Instead of just listing papers (D1), the system should prioritize organizing literature into a coherent field landscape (D3) that explains relationships between sub-areas. This means the improved system can produce reviews where the structure itself is an argument about how different research approaches relate, rather than a simple summary.

2. Enhance Research Direction Generation by focusing on Non-Obvious Gap Identification (D4). The system should be trained to move beyond generic future work templates and instead surface gaps that are specific, grounded in cited literature, and suggest experiments that resolve uncertainty based on existing limitations (e.g., While X paper addresses A, the current literature lacks a framework for integrating B into C).

3. Implement Expert-Aligned Evaluation Feedback Loops via an Expert-in-the-Loop Calibrated Evaluator (LitJudge). The improved system should be trained using feedback from this evaluator, which leverages structure-matched examples (D3), content-similar precedents (D1/D2), and expert gap anchors (D4). This allows the AI to learn the nuanced quality signal that human experts prioritize over surface-level metrics like citation count.

4. Develop Systems with Explicit Claim Grounding Mechanisms (D2). The system must be engineered to rigorously verify that its claims are directly supported by relevant citations, moving beyond simple claim generation. This improves the reliability of the review by ensuring every assertion is tethered to evidence, reducing hallucinations and weak assertions in synthesis-heavy sections.

5. Optimize Computational Efficiency for Synthesis Tasks. Since agentic models currently incur a high token cost (approx. 15x), future improvements should focus on developing specialized agents or retrieval/synthesis pipelines that achieve expert-level D3/D4 performance at a fraction of the current computational budget, making high-quality literature reviews economically viable for researchers.

6. Create Versioned, Reproducible Benchmarks from Live Feedback. The system development pipeline should incorporate a mechanism to freeze live preference data (from platforms like LitReview Arena) into versioned benchmarks (LitReviewBench). This allows developers to systematically track improvements across different iterations and ensure that performance gains are measurable against a stable, expert-validated standard.

Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. We introduce LitReview Arena, a battle-style evaluation platform with a structured protocol tailored to literature review quality: domain experts with AI paper-writing experience compare anonymized drafts, are matched to topics within their expertise, and provide dimension-wise outcomes over five literature-review-specific criteria. From this protocol, we collect approximately 3k expert judgments, each containing five dimension-wise outcomes, and show that even the strongest current systems win only 23.0% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60%. We further find that existing LLM-as-a-judge methods are substantially misaligned with human experts (Spearman's rho=0.467), especially on synthesis-heavy criteria such as paper structure and research suggestions. Using the collected preference data, we provide an expert-calibrated evaluator, LitJudge, which improves alignment to Spearman's rho=0.78, comparable to inter-expert consistency; code and data are publicly available at https://github.com/VanellopeAsher/LitReview-Arena.