When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs
cs.AI
Submitted: 2026-02-06
Updated: 2026-09-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are increasingly deployed in high-stakes settings where good decisions require forming beliefs over the probability of unknown outcomes.
Terminology
Abstract
Large language models (LLMs) are increasingly deployed in high-stakes settings where good decisions require forming beliefs over the probability of unknown outcomes. However, it is unclear whether LLMs act as if they hold coherent beliefs when making decisions, or if so, how we could validate models' reports of such beliefs. We propose a decision-theoretic framework that elicits both probability judgments and decisions from an agent and tests their mutual consistency. Formally, our methods characterize whether it is possible for the actions to be produced by a ``near-rational" decision maker who holds the elicited probability as their true belief. We show that, perhaps surprisingly, this formalization implies empirically testable conditions even without any assumption about the agent's utility function. Applying our framework to stylized clinical diagnosis tasks, we find that models' reported beliefs are demonstrably imperfect summaries of the information revealed in their decisions, but that the discrepancies are small for the strongest models.
Sources
- Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs
- Large Language Models Assume People are More Rational than We Really are
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- Training language models to follow instructions with human feedback
- Open Problems in Mechanistic Interpretability
- OpenAI GPT-5 System Card
- Calibrating Verbalized Probabilities for Large Language Models
- Eliciting the Priors of Large Language Models using Iterated In-Context Learning
- Incoherent Probability Judgments in Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection