Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: 20 pages, 4 figures, and 43 appendix tables. Research paper on language-model interpretability, reasoning evaluation, output-scoring bottlenecks, and label-free calibration. The paper evaluates controlled three-way logical reasoning tasks, ProofWriter, ANLI, and FOLIO using Qwen3.5, OLMo-2-1B, Llama-3.1-8B, and Pythia checkpoints
License: http://creativecommons.org/licenses/by/4.0/
The gist: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability.
Terminology
Abstract
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
Sources
- Understanding intermediate layers using linear classifier probes
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Transformers as Soft Reasoners over Language
- The Llama 3 Herd of Models
- Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
- 2 OLMo 2 Furious
- GPT-4 Technical Report
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection