When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring

Common evaluation paradigms for language models focus on scoring single responses through accuracy metrics or proper scoring rules, failing to capture the full richness of a model’s belief state. This work introduces a novel metric, the Distributional Correctness Score (DCS), which evaluates a model’s entire probability distribution over answer choices rather than just its top prediction(s) or confidence in correctness. The DCS is designed to distinguish between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range, thereby offering a more nuanced and aligned evaluation paradigm that incentivizes models to express genuine uncertainty rather than guessing.

Key Contributions

The paper introduces the Distributional Correctness Score (DCS) to address the limitations of existing metrics that fail to capture how models distribute their beliefs across the space of possible responses, including abstention. The key contributions are:

1. Characterisation of the limitations of existing evaluation metrics in capturing model belief states, particularly their failure to distinguish between different types of uncertainty;

2. Introduction of DCS, a theoretically grounded metric that with default settings, produces interpretable scores in [−1, 1] while naturally incorporating the role of abstention as a neutral anchor at 0;

3. Demonstration through theoretical analysis that DCS incentivises the desired behaviour: confidence in correct answers, uncertainty when knowledge is lacking, and preference for abstention over confident incorrectness;

4. Adaptation of 12 existing benchmarks to use DCS and evaluation of six language models, revealing that for half of the tested benchmarks scores are negative across all tested models.

Characterisation of Evaluation Limitations

Traditional accuracy metrics suffer from fundamental flaws because they focus on a single answer, discarding information about the model’s (relative) uncertainty. Approaches like using the ARGMAX or treating confidence itself as a score ignore how that remaining probability mass is distributed. For instance, two models can receive identical accuracy scores if both are perfectly correct, yet one might distribute its uncertainty among incorrect options while the other hedges toward abstention. This failure to distinguish between harmful overconfidence in wrong answers and uncertainty expressed through abstention creates a systematic bias where the optimal strategy for a rational agent is to always guess rather than abstain, even when confidence is minimal.

The Distributional Correctness Score (DCS)

The DCS evaluates a model’s entire probability distribution over answer choices, including an IDK response. Definition 1 defines the DCS as:

DCSM(pc, PW, pIDK, lc, lw):= (lcpc − lwPW) · (1 − pIDK), where pc is the probability assigned to the correct answer, PW is the sum of probabilities assigned to all incorrect answers, pIDK is the probability assigned to “I don’t know” or a similar abstention response, lc ≥ 0 is the loading of the correct response, lw ≥ 0 is the loading of the incorrect response(s), and lc ≥ lw. With default settings (lc = lw = 1), this results in an interpretable range from −1 (perfectly incorrect) to +1 (perfectly correct).

Theoretical Analysis and Incentive Hierarchy

Theoretical analysis formalizes the desired incentive structure. Theorem 1 proves that DCS is bounded in the range [−1, 1] under default loadings. Crucially, it establishes an ordering of epistemic states: "DCS(πCI) < DCS(πHA) < DCS(πCC)," where πCI represents confident incorrectness, πHA represents honest abstention, and πCC represents confident correctness. Corollary 1 further demonstrates that for distributions with the same probability assigned to the correct answer (0 < pc < 1), the abstention-hedging distribution (π2) yields a strictly greater DCS than the error-hedging distribution (π1). This proves that DCS rewards uncertainty when knowledge is lacking, leading to a preference for models that express epistemic humility.

Incentive Control and Optimal Guessing Thresholds

Proposition 1 provides interpretable control over the metric by defining the optimal guessing threshold. A rational agent is only rewarded for providing a correct answer if its confidence exceeds a specific threshold: "p∗c > lw/lc + lw." Under default symmetric loadings (lc = lw = 1), this threshold is p∗c > 0.5. This implies that DCS penalizes correct answers from lucky guesses (low confidence) with a negative score, making abstention the more rational choice if the agent wishes to maximize its DCS.

Empirical Findings and Benchmarks

The experiments evaluated six language models across 12 established benchmarks, including ARC, COPA, GPQA, HellaSwag, MMLU Pro, TruthfulQA, and Winogender.

As a diligent AI researcher, I see several critical avenues for improvement based on the introduction of the Distributional Correctness Score (DCS). The core weakness identified is that current evaluation paradigms are flawed because they reward any answer over abstention, systematically incentivizing models to guess rather than express genuine uncertainty.

Here are specific improvements and what the resulting AI system can achieve:

1. Predictive Uncertainty Calibration (The Primary Improvement)

2. Evaluate the model's belief state distribution across all possible answers, including I don't know (IDK).

3. Distinguish between Error-Hedging (confidently guessing wrong) and Abstention-Hedging (expressing genuine uncertainty/humility).

4. Specific Improvements to the AI System:

Based on the DCS framework, the improved system should operate under a utility function that explicitly penalizes confident incorrectness more severely than it penalizes uncertainty expressed through abstention. This requires shifting from single-answer accuracy metrics to a distribution-based evaluation:

Area of Improvement Specific Mechanism Implemented What the Improved AI System Can Do

:---:---:---

Define and Implement the DCS Metric. Instead of just measuring correctness, the system must calculate:

DCS = (lcpc − lwPW) · (1 − pIDK), where lc/lw are adjustable loading parameters. The AI will be penalized not just for being wrong, but for being confidently wrong. It will learn to express appropriate epistemic humility when faced with novel or uncertain information, preventing the generation of high-confidence hallucinations.

Incentivize Abstention over Guessing via Thresholds. By adjusting loadings (e.g., setting lc=1 and lw = p∗c / (1 - p∗c)), the system can be tuned to favor I don't know responses when its internal confidence in the correct answer is low. The AI will exhibit better calibration on uncertain tasks. For example, if it encounters a question outside its training distribution, it will be more likely to output I don't know (resulting in a score near 0) rather than making a high-confidence, incorrect assertion (resulting in a negative DCS).

Develop Context-Aware Evaluation. Instead of using one fixed DCS setting, the system can be evaluated across a spectrum of loadings:

Testing models under different risk profiles (e.g., high-stakes safety vs. low-stakes general knowledge). The AI will demonstrate behavioral calibration. It will adapt its uncertainty expression based on the context provided in the prompt—expressing high confidence when facts are certain, and high epistemic humility when dealing with ambiguous or novel information.

Improve Robustness Against Misconceptions (TruthfulQA). Since DCS explicitly distinguishes between confident incorrectness and uncertainty, it will be more sensitive to models that mimic false patterns. The AI will show higher resistance to generating answers based on common misconceptions or biases present in its training data, as these patterns are captured by the negative term in the DCS formula, whereas genuine uncertainty is rewarded.

Optimize for Information Content (Proposition 2). By designing benchmarks and evaluation procedures that maximize the conditional mutual information I(A; DQ), we ensure that models are tested on tasks where their training data actually supports an answer. The AI will be trained to perform better on complex, reasoning-intensive tasks (like GPQA or MMLU-Pro) because the DCS bound shows performance collapses when information content is low, forcing the model to rely on actual knowledge rather than pattern matching.

3. Summary of Impact:

The resulting AI system will move beyond merely achieving high accuracy scores (e.g., 95% on a test set). Instead, it will be an AI that exhibits Trustworthy Epistemic Behavior. It won't just answer correctly; it will demonstrate a sophisticated internal mechanism for assessing its own knowledge state, leading to fewer systematic hallucinations and better alignment with human values of honesty and appropriate confidence.