KTO: Model Alignment as Prospect Theoretic Optimization
cs.LG, cs.AI
Submitted: 2024-02-02
Updated: 2026-09-08
Comments: ICML 2024
Code: https://github.com/tatsu-lab/alpaca_eval
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Kahneman & Tversky's prospect theory tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse.
Terminology
Abstract
Kahneman & Tversky's prospect theory tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them belonging to a family of loss functions that we call human-aware losses (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach KTO, and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B, despite only learning from a binary signal of whether an output is desirable. More broadly, our work suggests that there is no one HALO that is universally superior; the best loss depends on the inductive biases most appropriate for a given setting, an oft-overlooked consideration.
Sources
- Training Verifiers to Solve Math Word Problems
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Towards Ecologically Valid Research on Language User Interfaces
- The Llama 3 Herd of Models
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Human irrationality: both bad and good for reward inference
- Evaluating Large Language Models Trained on Code
- Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
- ORPO: Monolithic Preference Optimization without Reference Model
- Nash Learning from Human Feedback
- Mistral 7B
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- OpenAssistant Conversations -- Democratizing Large Language Model Alignment
- Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
- Proximal Policy Optimization Algorithms
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback
- Fine-Tuning Language Models from Human Preferences
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks