Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
cs.CL
Submitted: 2026-01-12
Updated: 2026-09-23
Code: https://github.com/vllm-project/vllm
Terminology
Sources
- Uncertainty in Natural Language Generation: From Theory to Applications
- Language Model Cascades: Token-level uncertainty and beyond
- Reading Calibrated Uncertainty from Language Model Trajectories
- Language Models (Mostly) Know What They Know
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- 2 OLMo 2 Furious
- Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
- Calibrated Interpretation: Confidence Estimation in Semantic Parsing
- Revisiting Uncertainty Estimation and Calibration of Large Language Models
- Qwen2.5 Technical Report
- Black-box Uncertainty Quantification Method for LLM-as-a-Judge
- Towards Generation-Efficient Uncertainty Estimation in Large Language Models
- Can LLMs Assist Computer Education? an Empirical Case Study of DeepSeek
- Bayesian Reward Models for LLM Alignment
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering