Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
cs.AI
Submitted: 2026-03-10
Updated: 2026-09-10
Comments: 20 pages
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely
Terminology
Abstract
Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using meta-d'. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Sources
- Measuring Massive Multitask Language Understanding
- Language Models (Mostly) Know What They Know
- Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models
- Teaching Models to Express Their Uncertainty in Words
- Training Verifiers to Solve Math Word Problems
- Qwen3 Technical Report
- Cognitive Effects in Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection