When are likely answers right? On Sequence Probability and Correctness in LLMs
Johannes Zenn, Jonas Geiping
stat.ML, cs.LG
Submitted: 2026-06-25
Comments: 38 pages, including 10 pages of main text and 28 pages of appendix, preprint
License: http://creativecommons.org/licenses/by/4.0/
The gist: Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at
Terminology
Abstract
Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement.
Sources
- Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning
- Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Deep Think with Confidence
- Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- Measuring Mathematical Problem Solving With the MATH Dataset
- Truncation Sampling as Language Model Desmoothing
- Distilling the Knowledge in a Neural Network
- The Curious Case of Neural Text Degeneration
- Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
- Language Models (Mostly) Know What They Know
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty
- Reasoning with Sampling: Your Base Model is Smarter Than You Think
- Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language Models
- Can GRPO Help LLMs Transcend Their Pretraining Origin?
- Outcome-based Exploration for LLM Reasoning
- Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier