POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
cs.CL, cs.AI
Submitted: 2025-10-17
Updated: 2026-09-21
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users.
Terminology
Abstract
Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- LoRe: Personalizing LLMs via Low-Rank Reward Modeling
- PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
- KTO: Model Alignment as Prospect Theoretic Optimization
- End-to-end Training for Recommendation with Language-based User Profiles
- HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
- SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback
- A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- ORPO: Monolithic Preference Optimization without Reference Model
- RALI@TREC iKAT 2024: Achieving Personalization via Retrieval Fusion in Conversational Search
- Test-Time Alignment via Hypothesis Reweighting
- From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
- Personalized Language Modeling from Personalized Human Feedback
- Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems
- Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
- Transparent and Scrutable Recommendations Using Natural Language User Profiles
- PersonaBOT: Bringing Customer Personas to Life with LLMs and RAG
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering