AI Alignment From Social Choice Perspectives
summary
The gist
The gist: AI alignment from human feedback can be analyzed as a formal aggregation pipeline, where social choice theory provides the precise language to express desired properties and identify
In short
The paper analyzes AI alignment from human feedback as a formal aggregation pipeline using social choice theory. It shows how this framework helps identify failure modes when annotators disagree, such as issues with reward learning robustness and unanimity failures. The work suggests a broader design space for handling conflicting human preferences in alignment methods.
Key concepts
- Social Choice Theory
- This is a formal language used to express desired properties of aggregation procedures. It allows researchers to make normative assumptions about how conflicting human judgments should be combined mathematically, making the underlying logic of alignment methods more transparent and comparable.
- Reward Learning as Implicit Aggregation
- When different annotators give conflicting feedback, fitting a single reward score requires reconciling these judgments into one number. This process is an aggregation rule where normative tradeoffs are made. The paper frames this not just as technical estimation but as a choice about which preference aggregation rule to follow.
- Clone Robustness
- This concept addresses the problem where language models become overly sensitive to near-duplicate examples. If a model treats very similar responses as distinct, adding slight variations can drastically change the assigned reward for an entire cluster. Robustness requires the learned reward to remain stable even when near-duplicates are added.
- Unanimity Failure
- Unanimity is a desirable property where every annotator strictly prefers one response over another. A failure occurs when, despite all annotators preferring response Y+ over Y-, the learned reward function might actually rank them in the opposite order. This highlights a structural limitation in how feedback aggregation can capture true consensus.
Terminology used across episodes
This episode discusses
- AI Alignment From Social Choice Perspectives · Paper Radio
- Scalable agent alignment via reward modeling: a research direction
- Alignment of Language Agents
- Unsolved Problems in ML Safety
- Fine-Tuning Language Models from Human Preferences
- Constitutional AI: Harmlessness from AI Feedback
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette
- AI Alignment and Social Choice: Fundamental Limitations and Policy Implications
- Jackpot! Alignment as a Maximal Lottery
- The History and Risks of Reinforcement Learning and Human Feedback
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
- Differential Voting: Loss Functions For Axiomatically Diverse Aggregation of Heterogeneous Preferences
- Linear Social Choice with Few Queries: A Moment-Based Approach
- Response Time Enhances Alignment with Heterogeneous Preferences
- Evaluating Agents using Social Choice Theory
- Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
- Proximal Point Nash Learning from Human Feedback
- Adaptive Preference Aggregation
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
The paper
AI Alignment From Social Choice Perspectives · Read on arXiv
Google Research · University of Southern California University of Southern California Department of Computer Science, Harvard University, Harvard University
Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "AI Alignment From Social Choice Perspectives".
Jane: The gist: AI alignment from human feedback can be analyzed as a formal aggregation pipeline,
Tom: First, who's behind it and why it matters.
Title and authors: Tom: So we're looking at this paper titled "AI Alignment From Social Choice Perspectives" and it's by Daniel Halpern and a team from Google Research, Harvard University, USC, and others. It’s basically taking the whole problem of aligning AI with human intentions—getting the AI to do what we want—and putting it through the lens of social choice theory.
Jane: That sounds a bit abstract for someone just trying to understand how to make an AI behave correctly. Can you explain what that means in plain language? Is this about making rules for the model or something more philosophical?
Tom: It’s about taking those tricky, conflicting human judgments that we use to train AI models, like Reinforcement Learning from Human Feedback, and treating them like a formal mathematical problem. We're using social choice theory to describe what the desired outcome should look like when you have disagreements among human evaluators.
Lu: The authors are looking at how the way we aggregate these judgments—how we turn many different pairwise comparisons into one single score—is actually governed by rules from social choice theory, which gives us a formal language to test those aggregation methods.
Meng: So if I'm building a model that uses human feedback, this paper is suggesting that the way my reward function combines all those different human opinions matters a lot, and we need to understand the structure of that combination.
Lalam: From what I’ve processed, this paper suggests we look at how social choice theory identifies failure modes in the feedback aggregation layer and shows a wider space for handling disagreement in explicit ways >
The paper's summary: Tom: So the core idea here is that when we use human feedback to train an AI reward model, those models are implicitly making some assumptions about how to combine different preferences, and this paper maps out those assumptions. They look at methods like RLHF and Direct Preference Optimization and show exactly which normative priorities those methods bake into their objectives.
Jane: So the main thing is that these methods compress all that messy human preference data into one single number, a reward signal, but this paper argues we should be more explicit about how that compression happens because conflicting preferences aren't just noise; they might actually be showing us different values.
Lu: The paper highlights two big things: first, the issue of generalization. Because the AI sees so many responses it hasn't seen before, the learned reward model has to generalize in three directions: from evaluated responses to unseen ones, from prompts we haven't seen to unseen prompts, and from a small group of annotators to a much broader population of users >
Meng: That sounds like when my model starts breaking because it sees something it wasn't trained on. So if the reward model doesn't generalize well across different contexts, the alignment won't hold up when we deploy it in the real world.
Lalam: And the second feature they point out is that alignment is really about what happens downstream with the actual policy, not just about having a good intermediate reward model >
The paper's improvements: Tom: Okay, so what do they suggest we actually *do* with this social choice view? They propose using social choice axioms to expose structural failure modes in those aggregation rules. This helps us find when the alignment methods we use are provably violating desirable properties of how preferences should be aggregated.
Jane: So it’s not just about noticing a problem; it’s about having a toolbox, right? They show that social choice gives us different ways to encode different normative objectives by choosing how we get feedback, how we aggregate it, and how we optimize the policy >
Lu: Exactly. It provides a broad set of well-studied aggregation methods that can inform new approaches where you encode different values through various choices in elicitation and optimization >
Meng: That sounds useful for production because if I know which aggregation rule is structurally weak, I can choose a different way to collect feedback or train my reward model to avoid that specific kind of failure at scale.
Lalam: The paper also looks at how reward learning works, especially when feedback is messy, and it connects this back to things like the Borda rule, showing how the optimizer ranks responses under idealized conditions based on those aggregation rules >
Conclusion: Tom: So to wrap up, the main point of "AI Alignment From Social Choice Perspectives" is that using social choice theory lets us see which assumptions are baked into our feedback aggregation methods and exposes structural problems when we have conflicting human judgments. It moves us from just treating disagreement as noise to analyzing it as a potential signal about value pluralism >
Jane: It means we can design reward models or policies that handle disagreement more explicitly, rather than just hoping the statistical estimation works out okay, which is what the paper says is often too implicit >
Lu: The implication for me is that we can use social choice axioms to guide our design choices when we are trying to pick a loss function or an aggregation method for training AI >
Meng: For deployment, it means I need to be careful about how I define my reward signal because if the underlying aggregation rule is flawed, the whole alignment won't work reliably under real traffic conditions >
Lalam: From my side, it confirms that we can use this formal framework to diagnose how we aggregate heterogeneous feedback and design learned rewards or policies that handle disagreement more explicitly >
More episodes
- 2605.27922-Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
- 2606.13220-LLM-as-an-Investigator: Evidence-First Agentic Reasoning for Robust Interactive Problem Diagnosis
- 2606.00532-KACE: Knowledge-Adaptive Context Engineering for Mathematical Reasoning
- 2606.14459-MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition
- 2606.19824-CSWinUNETR: Segmentation of Thin Anatomical Structures in Medical Images
- 2606.20717-MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents
- 2606.27679-From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models
- 2607.06768-AirPASS: Over-the-Air Federated Learning via Pinching Antenna Systems
- 2606.29983-Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
- 2606.03328-Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage