Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

arXiv:2604.04144 · cs.CL, cs.AI · Submitted 2026-04-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Many Preferences, Few Policies".

Jane: A principled method for selecting a small portfolio of Large Language Models (LLMs) that captures representative behaviors across heterogeneous user preferences addresses the impracticality of maintaining a separate LLM per…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let’s talk about the title and who came up with this, "Many Preferences, Few Policies." It really captures the essence of what they’re doing—finding a way to have a compact set of policies that handles a lot of different user needs. Jane The authors are from some very respected places, including Harvard and CMU, which tells us this isn't just theoretical work; it’s coming from teams deeply involved in cutting-edge AI research.

Lu: Yes, the collaboration between those institutions suggests a very thorough approach to this problem of aligning LLMs across multiple objectives <ref:2604.04144#pg0>. The title itself points directly to the tension they’re trying to resolve: wanting high personalization while keeping operational costs low.

Meng: I wonder how much of that operational cost reduction they are actually talking about in terms of the hardware they need, because my experience is that smaller models still demand significant GPU resources <ref:2604.04144#pg1>.

Lalam: It’s interesting to hear about this paper because it suggests a path toward truly efficient, generalized AI service delivery where the personalization happens through policy selection rather than massive model duplication <ref:2604.04144#pg2>.

The paper's summary: Tom: Now for the summary of "Many Preferences, Few Policies." Essentially, they describe an algorithm called PALM which takes those different user preference traits and generates a small portfolio of LLMs that can handle any combination of those preferences quite well. Jane They show how this portfolio can approximate the best possible LLM for any given set of weights in the probability simplex, which is pretty powerful mathematically.

Lu: The summary emphasizes that this method provides provable bounds on both the size of the portfolio and how accurately it approximates what an optimal LLM would be for a specific preference setting <ref:2604.04144#pg0>. It’s not just a heuristic guess; they’ve got mathematical proof backing the approach.

Meng: That's the critical part for me; having those guarantees means we aren't guessing about coverage anymore, which makes it much easier to integrate into a production system reliably <ref:2604.04144#pg1>.

Lalam: The summary also highlights that this approach allows them to replace per-user fine-tuning with this fixed portfolio, meaning the system becomes much more robust when dealing with a large and diverse user base <ref:2604.04144#pg2>.

The paper's improvements: Tom: The paper points out that their method is better than just picking weights randomly or using a simple uniform grid to find policies, showing that PALM actually covers the whole preference space much more effectively <ref:2604.04144#pg1>. Jane They also found that when comparing PALM to those simpler methods, it actually achieves the same level of approximation quality with a smaller portfolio in some cases.

Lu: The improvement they highlight is that this technique provides explicit bounds on size and approximation quality, which formalizes the trade-off between how faithful we want the coverage to be versus how small our model set can be <ref:2604.04144#pg2>. It makes the operational simplicity and personalization fidelity trade-off very clear for anyone working in this space.

Meng: I like that they provide that explicit mathematical trade-off; it lets us actually make informed decisions about how much diversity we need versus how much compute we can afford <ref:2604.04144#pg2>. It’s not just a black box solution anymore, which is what I need for deployment.

Lalam: The paper also gives qualitative evidence that their portfolio yields more diverse responses compared to the baselines, meaning users will actually see a wider variety of tones and styles available to them <ref:2604.04144#pg2>. That’s a tangible benefit for the end user experience.

Conclusion: Tom: So, wrapping up on "Many Preferences, Few Policies," we see that this method provides a principled way to select a small portfolio of LLMs that captures representative behaviors across all possible user preferences <ref:2604.04144#pg0>. Jane It fundamentally shifts the approach from trying to train one perfect model for everyone to managing a set of specialized models that collectively cover the entire preference space efficiently.

Lu: The main implication is that scalable personalization isn't about increasing compute endlessly; it’s about using clever portfolio selection algorithms like PALM to ensure we have the right tools available for any user need <ref:2604.04144#pg2>.

Meng: For practical implementation, the result is a much more manageable system where we trade per-user fine-tuning complexity for a fixed set of models that are mathematically guaranteed to work well across the board <ref:2604.04144#pg1>.

Lalam: Ultimately, this means we can offer richer response styles to users while keeping the infrastructure lean, ensuring every user gets an AI that truly reflects their specific needs <ref:2604.04144#pg2>.

Cheol Woo Kim, Jai Moondra, Roozbeh Nahavandi

Harvard University · Carnegie Mellon University · The Ohio State University

cs.CL, cs.AI

Submitted: 2026-04-05

Updated: 2026-10-01

Importance score: 84/100

The gist: A principled method for selecting a small portfolio of Large Language Models (LLMs) that captures representative behaviors across heterogeneous user preferences addresses the impracticality of

Key concepts

Scalarized Objective
This mathematical goal measures how well an LLM performs based on multiple user preferences. It combines different reward functions into a single score using weights. The paper aims to find a small set of models that can achieve near-optimal scores for all possible preference combinations.
Probability Simplex
This is the mathematical space where user preferences are modeled. Each dimension in this space represents a different trait, like safety or humor. A weight vector within this simplex describes how much importance to place on each trait when choosing an LLM.
CONSTRUCTGRID
This phase builds a comprehensive map of all possible user preference combinations (the probability simplex). It uses multiplicative and additive grids to systematically cover the entire space efficiently, ensuring that no important preference combination is missed in the initial selection process.

Terminology

Summary

A principled method for selecting a small portfolio of Large Language Models (LLMs) that captures representative behaviors across heterogeneous user preferences addresses the impracticality of maintaining a separate LLM per user. This work introduces PALM, an algorithm that generates a small set of LLMs such that, for any weight vector, the portfolio contains a near-optimal LLM for the corresponding scalarized objective, providing theoretical guarantees on both size and approximation quality.

Problem Setup

The core challenge is choosing a small set of LLMs to maximize diversity while minimizing operational costs. User preferences are modeled across multiple traits (e.g., safety, humor) through a multidimensional weight vector, denoted as a vector in the probability simplex, where each weight corresponds to a reward function. The objective is to find a policy that optimizes the weighted linear combination of these rewards:

/Jw(π):= Ex∼P[h Ey∼π(x) h d Σ i=1 wi ri(x, y) − f(π)

Here, the scalarized objective is defined as:

/Jw(π):= Ex∼P,y∼π(x)[r i(x, y)] − f i = d Σ i=1 wi · E x ∼ P,y ∼ π(x) [r i(x, y)] - f(π)

The goal is to construct a portfolio of policies that approximates the entire family of optimal LLMs induced by all possible weight vectors in the probability simplex.

The PALM Algorithm

PALM (Portfolio of Aligned LLMs) is designed to produce this small portfolio with provable bounds on size and approximation quality. The algorithm operates in two main phases:

  1. Constructing a grid of weight vectors, denoted as CONSTRUCTGRID, which uses a combination of multiplicative and additive grids to cover the probability simplex effectively. The process involves iterating through each coordinate and selecting remaining coordinates from a 1-dimensional grid defined by parameters µ and α.

  2. Generating candidate policies, P1, by calling an oracle that computes the optimal policy πw for every weight vector w in the constructed set W.

  3. Pruning the initial set of candidate policies P1 to obtain a minimal cover P using a greedy algorithm (PRUNE), which addresses redundancy in the initial set.

Theoretical Guarantees and Approximation

The paper establishes that scalable personalization is possible because the number of policies required is governed by the number of reward functions, independent of the number of users. The algorithm provides explicit theoretical guarantees on portfolio size and approximation quality:

**/Theorem 3.4 states that PALM outputs an (ε, δ)-approximate portfolio P ⊆ Π with size at most P ≤ d · (2 + 2µ / log(1/α) **

/ε ≤ 4µ and δ ≤ 2(dαRmax + µ fmax)

The approximation quality is formalized by Definition 3.1, which defines an (ε, δ)-approximation as a policy π such that its objective value satisfies Jw(π) ≥ (1 − ε) optw − δ. The paper proves Lemma 3.3, which formalizes the intuition that if two weight vectors w and v are close enough in each coordinate, then the optimal policy πw is a suitable approximation for v, establishing the relationship between weight vector proximity and policy approximation quality.

Empirical Validation

Experiments validate the theoretical guarantees across three distinct tasks: RLVR-GSM (two reward functions), Safety Alignment (two reward models), and Helpful Assistants (three reward models). The results consistently show that PALM outperforms standard heuristics—such as uniformly spaced or randomly selected weight vectors—at the same portfolio size in terms of approximation quality. Furthermore, qualitative evidence demonstrates that the PALM portfolio yields more diverse responses compared to baselines, suggesting it captures critical behavioral regimes missed by simpler constructions. Policy usage statistics reveal that the resulting portfolios exhibit more balanced usage across policies, as pruning removes redundant models, ensuring each LLM contributes meaningfully to covering distinct regions of the reward space.

Implementation Details

The implementation details involve fine-tuning policies using Group Relative Policy Optimization (GRPO) and training on specific datasets tailored to the task. For instance, in RLVR-GSM, the policy model is Qwen2.5-3BInstruct, and training utilizes a KL divergence penalty with coefficient β = 0.01 against a frozen reference policy. The evaluation involves sampling a finite set of weight vectors V from a Dirichlet distribution or using Monte Carlo sampling for inference-time methods (MOD) to approximate the objective value Jw(π). The paper details specific parameter choices, such as inputting µ = 8/30 and α = 2/10, which yielded a portfolio size of 13 in the 2-dimensional simplex.

Improvements for AI systems

Here are specific improvements to AI systems based on the PALM (Portfolio of Aligned LLMs) framework, and what these improved systems can achieve:

) 1. Serving Diverse User Preferences via Scalable Portfolios: Instead of training a single, monolithic LLM that must compromise across many user needs, an AI service should deploy a small, fixed portfolio of specialized LLMs (e.g., 3 to 7 models).

  1. Quantifiable Trade-off Management: The system can explicitly manage the trade-off between operational cost and personalization fidelity by tuning the approximation parameters in PALM (the multiplicative tolerance ε and additive tolerance δ). A researcher can choose a portfolio size that guarantees a specific level of behavioral coverage (e.g., I need at least 95% coverage for safety alignment, even if it means accepting a slightly larger portfolio).

  2. Guaranteed Coverage Across Preference Spaces: The system is mathematically guaranteed to provide an LLM in the portfolio that is near-optimal for any possible weight vector of user preferences (e.g., balancing helpfulness, brevity, and factuality). This eliminates the risk of missing critical behavioral regimes that occurs with random or uniform sampling methods.

  3. Enhanced Response Diversity: Because PALM selects policies based on maximizing coverage over the entire preference simplex, the resulting portfolio exhibits significantly greater output diversity than baselines (as seen in Table 2). This ensures that a single user can select from a variety of styles and tones, catering to nuanced preferences that simple prompt engineering cannot capture.

  4. Reduced System Overhead: By replacing per-user fine-tuning with a small, fixed set of policies, the computational and memory overhead associated with hosting thousands of individual LLMs is drastically reduced while maintaining high personalization quality.

  5. Adaptive Service Routing: The system can use the policy usage distribution metric (perplexity) to diagnose redundancy. If a policy in the portfolio is rarely selected, it indicates that its reward profile overlaps with another policy, allowing the system to prune redundant models dynamically and maximize serving capacity efficiency without sacrificing coverage guarantees.

This improved AI system can do the following:

  1. Generate highly tailored responses for a large, heterogeneous user base (e.g., one model for a short-form summary user, another for a highly cautious safety-focused user).

  2. Provide guaranteed performance bounds on personalization accuracy and portfolio size, making the system auditable and reliable for enterprise deployment.

  3. Offer users a richer selection of response styles (e.g., formal vs. casual) while ensuring that every possible combination of their stated objectives is adequately represented by one of the deployed models.

  4. Operate at a fraction of the cost required for per-user fine-tuning, enabling large-scale, scalable personalization without incurring prohibitive infrastructure costs or safety assurance complexities associated with managing numerous specialized models.

Sources

Related papers