Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs
cs.CL, cs.LG
Submitted: 2025-09-17
Updated: 2026-09-22
Comments: 24 pages, 8 figures. Camera-ready version, published in Transactions on Machine Learning Research (2026). OpenReview: https://openreview.net/forum?id=5UVv7gkgUD
Journal ref: Transactions on Machine Learning Research (2026)
Code: https://github.com/zlin7/UQ-NLG
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models are known to hallucinate, generating linguistically plausible but incorrect answers to questions.
Terminology
Abstract
Large language models are known to hallucinate, generating linguistically plausible but incorrect answers to questions. Uncertainty quantification has been proposed as a strategy to detect such behaviour, but existing methods lack a unified framework to assess reliability at both the prompt and answer level. We introduce a geometric framework which quantifies language model uncertainty at both levels by explicitly modelling a prompt-conditioned semantic distribution in answer embedding space. Our approach is black-box and sampling-based; we generate multiple answers per prompt, and use archetypal analysis to estimate a geometric support for the answer distribution. At the prompt level, we approximate the distribution entropy to quantify uncertainty; for each individual answer, we then use notions of atypicality to assess its reliability relative to the batch. We employ our framework to not only detect hallucinations but correct them, by selecting the batch example deemed most reliable. Experiments show that our framework performs comparably to or better than prior methods on short form question-answering datasets, and achieves superior results on medical datasets where hallucinations carry particularly critical risks. Beyond pure performance, we suggest the theoretical grounding of our work provides support for semantic distributions as useful objects of study for language model uncertainty.
Sources
- Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
- Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis
- A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law
- A Survey on LLM-as-a-Judge
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs
- Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach
- Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
- Uncertainty Estimation in Autoregressive Structured Prediction
- K-QA: A Real-World Medical Q&A Benchmark
- Confidence Improves Self-Consistency in LLMs
- A Survey of Uncertainty Estimation Methods on Large Language Models
- HuatuoGPT, towards Taming Language Model to Be a Doctor
- CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models
- TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering