Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy
cs.AI, cs.CL, cs.HC
Submitted: 2025-05-28
Updated: 2026-09-01
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: AI copilots represent a new generation of AI-powered systems designed to assist users, particularly knowledge workers and developers, in complex, context-rich tasks.
Terminology
Abstract
AI copilots represent a new generation of AI-powered systems designed to assist users, particularly knowledge workers and developers, in complex, context-rich tasks. As these systems become more embedded in daily workflows, personalization has emerged as a critical factor for improving usability, effectiveness, and user satisfaction. Central to this personalization is preference optimization: the system's ability to detect, interpret, and align with individual user preferences. While prior work in intelligent assistants and optimization algorithms is extensive, their intersection within AI copilots remains underexplored. This survey addresses that gap by examining how user preferences are operationalized in AI copilots. We investigate how preference signals are sourced, modeled across different interaction stages, and refined through feedback loops. Building on a comprehensive literature review, we define the concept of an AI copilot and introduce a taxonomy of preference optimization techniques across pre-, mid-, and post-interaction phases. Each technique is evaluated in terms of advantages, limitations, and design implications. By consolidating fragmented efforts across AI personalization, human-AI interaction, and language model adaptation, this work offers both a unified conceptual foundation and a practical design perspective for building user-aligned, persona-aware AI copilots that support end-to-end adaptability and deployment.
Sources
- A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
- Design and evaluation of AI copilots -- case studies of retail copilot templates
- The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems
- "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning
- Learning Retrieval Augmentation for Personalized Dialogue Generation
- PersoBench: Benchmarking Personalized Response Generation in Large Language Models
- Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations
- Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness
- Selective Prompting Tuning for Personalized Conversations with LLMs
- TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
- A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation
- PersonaPKT: Building Personalized Dialogue Agents via Parameter-efficient Knowledge Transfer
- PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
- Towards Persona-Based Empathetic Conversational Models
- Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts
- Fine-Tuning Language Models from Human Preferences
- Fine-Tuning Language Models with Reward Learning on Policy
- Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
- Dense Reward for Free in Reinforcement Learning from Human Feedback
- Confronting Reward Model Overoptimization with Constrained RLHF
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection