The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale
Mingguang Chen, Bo Qu, Licheng Wang
cs.CL
Submitted: 2026-08-05
Comments: 36 pages, 5 figures
Code: https://github.com/MBZUAI-CLeaR/IoE-Prompting
License: http://creativecommons.org/licenses/by/4.0/
The gist: Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning.
Terminology
Abstract
Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.
Sources
- Self-Refine: Iterative Refinement with Self-Feedback
- STaR: Bootstrapping Reasoning With Reasoning
- Self-Rewarding Language Models
- When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
- When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysis
- Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
- LLMs cannot find reasoning errors, but can correct them given the error location
- Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models
- Language Models (Mostly) Know What They Know
- When are likely answers right? On Sequence Probability and Correctness in LLMs
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- Qwen3.5-Omni Technical Report
- Qwen3 Technical Report
- Training Verifiers to Solve Math Word Problems
- Measuring Mathematical Problem Solving With the MATH Dataset
- Evaluating Large Language Models Trained on Code
- Program Synthesis with Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering