Can Revealed Preferences Clarify LLM Alignment and Steering?
cs.LG
Submitted: 2026-05-08
Updated: 2026-09-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different
Terminology
Abstract
LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.
Sources
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context
- Inference-Time Reward Hacking in Large Language Models
- Generative Value Conflicts Reveal LLM Priorities
- Large Language Models Assume People are More Rational than We Really are
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- AI as Decision-Maker: Ethics and Risk Preferences of LLMs
- What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
- OpenAI GPT-5 System Card
- When Do LLM Preferences Predict Downstream Behavior?
- When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs
- Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks