Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Affiliation: Max Planck Institute for Intelligent Systems, Tübingen; Tübingen AI Center; ETH Zürich

Published: arXiv:2410.13341v3 [cs.LG] 6 Jan 2026

"High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work."

The paper addresses the challenge of evaluating newly released AI models as they advance in capabilities. Expert data annotation is not only slow and costly. Traditional benchmarking also struggles to keep up with rapidly changing model capabilities across an expanding range of tasks. The authors examine the model-as-judge paradigm, where a strong existing model to provide judgments about other models is used to replace human annotators.

However, the authors note that When used as judges, models exhibit a range of biases that can skew model comparisons and result in misleading model rankings. Recently proposed debiasing methods promise a compelling way forward. Using a small number of ground truth labels, these methods can potentially debias a large number of model predictions, thus restoring their utility for benchmarking purposes.

The paper focuses on the evaluation frontier: Newly released models for which we have little intuition as of yet. What makes this case so challenging is that the new model is likely better than the judge in some ways. The central question is whether debiasing methods together with the model-as-judge paradigm can, in principle, provide an adequate solution to scalable evaluation at the frontier.

The main prediction is sobering: "Whenever the judge model performs worse at its task than the evaluated model, the optimal debiasing method is no better than using twice the ground truth data. This shows that, although there is merit to debiasing, at the evaluation frontier its economic gains are not greater than a factor-two savings in annotation cost."

The paper analyzes aggregated binary evaluations: Given a prompt x, a model m receives a binary score s(x, m(x)) depending on its output m(x). The model is evaluated based on its expected score E s(X, m(X)) where the expectation is taken over the prompt distribution X ∼ D, as well as additional randomness in the scores s(·, m(·)) introduced by the evaluation protocol.

This setup covers:

• Accuracy in classification and Q&A benchmarks: the accuracy score s(x, m) equals one whenever m(x) equals the label y(x)

• Arena-style benchmarks: The score s(x, m) indicates whether m(x) was judged to be a better response than m′(x)

• Safety benchmarks: The score s(x, m) indicates whether the model's response m(x) is safe

A judge model provides a proxy score s̃ that is cheaper to obtain. The joint distribution of the two induced random variables s(m) and s̃(m) is specified by three parameters:

• b(m) = P s(m) = 1 — the model's expected real score

• p(m) = P s̃(m) = s(m) s(m) = 1 — the true positive rate

• q(m) = P s̃(m) = s(m) s(m) = 0 — the true negative rate

The paper defines judge bias as JB(m):= E(s̃(m) − s(m)) = (1 − q(m))(1 − b(m)) − (1 − p(m))b(m). Whenever JB(m) has large magnitude, the proxy score s̃ misrepresents the performance b(m) of model m, even with large sample sizes.

Proposition 1 shows that when using a classifier m̃ to evaluate a set M of strictly better classifiers (where m̃(x) = y(x) implies mi(x) = y(x) for all mi ∈ M), "E s(mi) > E s(mj) implies E s̃(mi) < E s̃(mj)" — meaning the proxy evaluations fully reverse the correct model ranking.

The paper defines agreement as AG(m):= P s(m) = s̃(m) = b(m)p(m) + (1 − b(m))q(m).

Proposition 2 shows that for agreement AG(m) = r, there can be a judge bias of 1 − r in either direction. This means even if AG(m) = r was constant across models m, we could only reliably rank models for which the true score E s(m) differs by more than 2(1 − r). The authors conclude: without further assumptions an agreement rate above 99% (for each evaluated model!) would be required to ensure (asymptotically) correct rankings.

Proposition 3 states that if all model evaluations are approximately unbiased (E θ̂(m) − E(s(m)) < ϵ/2 for models separated by at least ϵ), then using θ̂ to rank models yields the correct ranking with high probability as variance converges to zero.

The paper follows the PPI framework (Angelopoulos et al., 2023a). The PPI estimator is:

θ̂ PP(x) = (1/N) Σ i=n+1 N+n s̃ i(m) + (1/n) Σ j=1 n (s j(m) − s̃ j(m))

This estimator is unbiased: E θ̂ PP = E s̃(m) + E s(m) − E s̃(m) = E s(m).

The variance is: Var θ̂ PP = (1/N) Var s̃(m) + (1/n) Var(s̃(m) − s(m)).

An interpolated version θ̂ λ PP = λθ̂ PP + (1−λ)θ̂ GT with optimal λ* = Cov(s(m), s̃(m))/(1 + n/N) Var s̃(m) never increases variance compared to the classical ground truth estimator.

The paper defines the sample efficiency factor as τ(θ̂):= Var θ̂ GT / Var θ̂ = 1/r, where r = Var θ̂ / Var θ̂ GT.

Proposition 4 provides an upper bound: τ(θ̂ λ PP*) ≤ 1/(1 − ρ(s(m), s̃(m))2), where ρ2 is the squared Pearson correlation between s and s̃.

Theorem 5 states: "Let Θ be the set of all unbiased estimators for E s(m) that observe n joint IID samples (s(m), s̃(m)) and N IID proxy samples s̃(m) independent of the joint samples. Then, any θ̂ ∈ Θ fulfills the variance bound Var θ̂ ≥ Var θ̂ λ PP*."

This means the sample efficiency factor of θ̂ is bounded by τ(θ̂) ≤ max θ̂∈Θ τ(θ̂) = τ(θ̂ λ PP*) ≤ 1/(1 − ρ(s(m), s̃(m))2).

Theorem 6: Assume 0.5 ≤ AG(m) ≤ b(m). Then, ρ(s(m), s̃(m))2 ≤ 0.5.

Corollary 7: Assume 0.5 ≤ AG(m) ≤ b(m). Then, τ max = max θ̂∈Θ τ(θ̂) ≤ 2.

The authors explain: "Corollary 7 implies that when evaluating state-of-the-art models, the best we can expect from using LLM judges is a factor-two improvement in sample efficiency. This is unless the judge's task is substantially easier than the evaluated model's task."

Proposition 8: "For any agreement rate 0.5 ≤ AG(m) < 1, there exist values of b(m), p(m), q(m) ∈ (0, 1) such that JB(m) = ρ(s(m), s̃(m))2 = 0 and thus τ max = 1."

This shows that in some cases despite access to s̃, no unbiased estimator has less variance than θ̂ GT.

The paper evaluates models on MMLU (Hendrycks et al., 2021) using predictions from the HELM leaderboard. Figure 2 shows that Sample efficiency gains stay below two, unless SOTA models are used to evaluate weak models. Specifically, τ max consistently stays below the value of two suggested by Corollary 7, except when current flagship models are used to judge the significantly worse LLama2-7b on MMLU.

For MT-Bench (Zheng et al., 2024), Figure 3 shows that Sample efficiency get close to two in some cases, but consistently stay below that value.

The authors note: "Interestingly, τ max stays below two in all other cases, even when stronger models like GPT-4 are used to evaluate weaker models like LLama3-70B. This suggests that the upper bound on τ max from Corollary 7 is fairly robust."

The paper relaxes the assumption of binary proxy scores, allowing continuous proxies such as the probability a judge assigns to a model's answer.

Proposition 9: For any binary proxy s̃ such that 0.5 ≤ AG ≤ b, we have 0.5 ≤ SO(R(s̃)) ≤ b, where SO is the soft agreement and R(s̃) is the recalibrated proxy.

Theorem 10: For any proxy score s̃ with SO(R(s̃)) ≤ b, we have ρ2(s, s̃) ≤ 0.5. Correspondingly, the sample efficiency of PPI is bounded: τ(θ̂ λ PP*) ≤ 2.

Experiments using LLama3.1-405B's next-token predictions on MMLU show that "using the non-binary proxy consistently improves sample efficiency. However, as suggested by Theorem 10, the sample efficiency factor τ(θ̂ λ PP*) remains below two when we use LLama3.1-405B to evaluate the stronger Claude 3.5."

Proposition 11: For strictly better classifiers with q(m) = 0, b(m) = x + δ, and p(m) = x/(x+δ), ρ(s(m), s̃(m))2 ≤ 1/9. Correspondingly, the sample efficiency factor is bounded by τ max ≤ 1.125.

Theorem 12: For any value of BA(m), we have: 4b(m)(1−b(m))(2 BA(m) − 1)2 ≤ ρ(s(m), s̃(m))2 ≤ 2 BA(m) − 1, where BA is the balanced agreement.

Corollary 13: Whenever BA(m) ≥ 0.5, we have ρ(s(m), s̃(m))2 ≤ min p(m), q(m).

The authors conclude: "Our results show that for evaluating frontier models, LLM judges might fall short of the promise of largely replacing expert labelers: While doubling the effective sample size can be useful for practitioners, the order of magnitude of required ground truth labels remains the same with and without access to LLM judges."

They note three ways to potentially circumvent the negative results:

1. Approaches like stratified PPI (Fisch et al., 2024) might be able to obtain a somewhat better sample efficiency factor — though Theorem 5 still applies per-stratum and empirical results suggest that at the frontier, stratified PPI rarely improves sample efficiency by more than a factor of two.

2. "Theorem 6 only holds if the proxy s̃ is less capable at predicting s than the evaluated model m is at its task. Thus, for tasks in which evaluation is substantially easier than the task itself... gains in sample efficiency of more than a factor two are possible."

3. Our results do not only apply to LLM judges, but any form of biased evaluators including (poorly instructed) crowdworkers.

The paper situates itself within the LLM-as-a-Judge literature, noting that LLM judges can be biased in numerous ways including self-preferencing (Liu et al., 2023; Panickssery et al., 2024), preference for longer outputs (Dubois et al., 2024b; Wei et al., 2024), and choice-order bias (Dominguez-Olmedo et al., 2023; Wang et al., 2023; Shi et al., 2024).

Regarding debiasing methods, the authors note that prior work (Chaganty et al., 2018) found that their method, which is essentially equivalent to PPI, only improved data efficiency by around 10% using 2018's automated metrics. More recent works (Boyeau et al., 2024; Chatzi et al., 2024; Fisch et al., 2024) consistently show that PPI improves efficiency in terms of ground truth labels, but gains in effective sample size rarely exceed 50% and are always below 100%.

Based on the paper, here are specific improvements for AI systems:

1. Implement Bias-Corrected Evaluation for Model Comparisons

• Improvement: When evaluating a new model against a judge model (e.g., using GPT-4 to judge Claude), do not rely solely on the judge’s proxy scores. Instead, collect a small set of ground-truth labels (n) and apply the Prediction Powered Inference (PPI) estimator: θ̂ PP = (1/N)Σs̃ i + (1/n)Σ(s j - s̃ j). Then optimize the interpolation parameter λ to minimize variance.

• What it can do: Provides unbiased performance estimates, preventing misleading rankings (e.g., avoiding the full reversal shown in Proposition 1). This is critical when the evaluated model may outperform the judge.

2. Enforce a Sample Efficiency Ceiling of 2× at the Frontier

• Improvement: Before deploying an LLM-as-a-judge for a new model, check if AG(m) ≤ b(m) (judge agreement ≤ evaluated model’s true score). If true, cap the expected benefit of debiasing at a factor of 2 in ground-truth sample efficiency (Corollary 7). Do not allocate resources expecting more than a 2× reduction in annotation cost.

• What it can do: Prevents overinvestment in complex debiasing pipelines for frontier models where gains are fundamentally limited. Guides researchers to instead double the ground-truth data collection if more accuracy is needed.

3. Use Non-Binary Proxy Scores with Recalibration for Slightly Better Efficiency

• Improvement: Instead of forcing the judge to output a single binary label, extract the judge’s probability distribution over answers (e.g., next-token probabilities for MMLU options). Recalibrate this proxy R(s̃) = P(s=1s̃) and use it in the PPI estimator.

• What it can do: Achieves sample efficiency factors closer to 2 (and slightly exceeds it for models 5% weaker than the judge), as shown in Figure 4. This is a practical gain over binary proxies, especially when evaluating models slightly below the judge’s capability.

4. Detect and Flag High-Agreement-but-Biased Judges

• Improvement: Do not trust high agreement rates (e.g., 85% with experts) as a proxy for evaluation quality. Instead, compute the balanced agreement BA(m) = (p(m) + q(m))/2. If BA(m) is close to 0.5, the judge provides almost no debiasing value (Theorem 12), even if overall agreement is high.

• What it can do: Identifies judges that appear accurate but are actually useless for ranking (e.g., a judge that always predicts the majority class). Prevents false confidence in evaluation results and prompts the use of ground-truth labels instead.

5. Optimize for Pairwise Ranking Differences, Not Individual Scores

• Improvement: When the goal is ranking models (not absolute scores), jointly optimize the PPI interpolation parameters λ and λ′ for pairs of models (m, m′) to minimize the variance of the difference θ̂(m) - θ̂(m′), rather than optimizing each model’s estimate independently.

• What it can do: Can yield better ranking accuracy for the same annotation budget, especially when proxy scores for different models are correlated. This is a direct improvement over standard per-model debiasing.

6. Implement Stratified PPI with Per-Stratum Ceiling Checks

• Improvement: When using stratified sampling (e.g., by MMLU subtask), apply PPI per stratum, but first verify that the judge is not better than the evaluated model in each stratum. If the judge is better in a stratum, the sample efficiency gain can exceed 2× there; otherwise, cap it.

• What it can do: Maximizes debiasing gains where possible (e.g., in subtasks where the judge excels) while avoiding wasted effort in strata where gains are capped. Empirically, this rarely exceeds 2× at the frontier (Appendix B.3), but it can be exploited when the judge is stronger on specific domains.

7. Avoid Using Judge Scores for Models That Are Strictly Better

• Improvement: If you know a model m is strictly better than the judge (e.g., the judge’s correct predictions are a subset of m’s correct predictions), do not use the judge’s scores even with debiasing. Proposition 11 shows the maximum sample efficiency gain is only 1.125× in this case.

• What it can do: Saves annotation budget by directly using ground-truth labels instead of a debiasing pipeline that offers negligible benefit. This is a clear operational rule for practitioners.

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.