Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods are limited because they largely reduce supervision to binary pairwise comparisons. This reduction is problematic because:

1. Training data naturally contains multiple candidate images for the same prompt.

2. Modern reward models can assign continuous scores to each candidate, and reducing this information discards the relative quality of the remaining candidates and ignores the magnitude of reward gaps.

To address these limitations, the authors propose Diffusion LAIR (Listwise Advantage-weighted Implicit Reward), which treats diffusion preference optimization as a listwise, reward-aware learning problem.

The Diffusion LAIR objective generalizes preference optimization from pairwise comparisons to listwise supervision by leveraging continuous reward scores across all candidates for a given prompt.

A. Listwise Supervision and Advantage Weights:

For each prompt c, let x 0 N be a set of N candidates associated with c, where N may vary across prompts. Each candidate has a corresponding reward score r i = r(c, x 0).

The method converts these scores into centered advantage weights (w i) using the softmax function and centering against the uniform baseline:

1. Define pi i = (r i / tau), where tau is a temperature parameter.

2. Calculate the centered advantage weight w i = pi Nc - 1/N.

These weights ensure that samples with higher reward receive positive weight and samples with lower reward receive negative weight.

B. The Reward-Aware Objective:

The objective is defined as an advantage-weighted regression on the implicit reward, incorporating a quadratic penalty for regularization:

L Diffusion-LAIR(theta) = E c, t, epsilon i N sum i=1 Nc w i s theta + lambda sum s squared

Where:

• s theta:= s theta(x 0, x t, t, c, epsilon i is the denoising-loss improvement of the current model over a fixed reference model.

• lambda > 0 controls the strength of regularization.

This construction preserves the reference-based structure of diffusion preference optimization while distributing learning signal across all candidates rather than selected comparison pairs.

The objective is designed to be analytically tractable:

A. Optimal Implicit Reward:

For a fixed listwise weights w i N and prompt c, the objective admits a unique pointwise minimizer:

s* i = wi over 2 lambda

This proves that "samples with relatively higher reward receive positive weights w i > 0 and are assigned positive optimal implicit reward s* i > 0, while samples with relatively lower reward receive negative optimal implicit reward."

B. Zero-Sum Redistribution:

Because the weights are centered (sum w i = 0), the objective enforces zero-sum redistribution within each group: sum s i = 0. This is desirable because "preference optimization is fundamentally a relative problem: for a fixed prompt, the goal is not to make every candidate more preferred, but to shift probability mass toward higher-quality samples and away from lower-quality ones."

C. Surrogate KL Bound:

The analysis provides a surrogate distribution-shift interpretation. The optimal implicit reward S* (x 0, c) is bounded by the finite-list range:

S* - S* = w i - w i over 2 lambda

This suggests that the regularization parameter lambda controls the sharpness of the surrogate preference tilt.

A. Experimental Setup:

The models fine-tune Stable Diffusion 1.5 (SD1.5) and Stable Diffusion XL (SDXL) using the Pick-a-Pic v2 dataset, which contains multiple candidate images per prompt, aggregated into lists x 0 N. The reward scores for these images are obtained offline using the PickScore reward model.

B. Benchmarks:

The models are evaluated across several benchmarks:

1. Text-to-Image Generation (T2I): Using test prompts from the HPD and Parti-prompts datasets.

2. Compositional Generation: Using the GenEval benchmarking suite (Table 5).

3. Image Editing: Using the InstructPix2Pix dataset, measuring win rates against SDXL (Table 2).

C. Performance and Efficiency:

• General Preference Alignment (T2I): In Table 1, Diffusion LAIR consistently outperforms baselines like DSPO and InPO across both SD1.5 and SDXL models on the HPD and Parti-prompt datasets.

• Compositional Generation (GenEval): Table 2 shows that the SDXL model trained with Diff.-LAIR achieves a significantly higher score (e.100) than all baselines, demonstrating consistent compositional gains.

• Image Editing: Table 2b shows that the SDXL model outperforms pairwise baselines in InstructPix2Pix win rates.

• Computational Cost: The method is highly efficient; for SDXL training, it required only 140 A100 GPU hours, which is significantly less than the nearly 5 times more H100 GPU hours required by standard baselines like Diffusion DPO and DSPO.

The authors conclude that Diffusion LAIR provides a broader improvement that extends beyond reward-model preference scores to compositional reasoning, semantic alignment, and instruction-based image editing, proving the value of listwise reward-aware optimization as an effective direction for aligning diffusion models with human preferences.

Based on the principles outlined in Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models, here are the specific, actionable improvements to implement a next-generation AI alignment system.

We transition from a pairwise optimization pipeline to a Listwise Reward-Aware Optimization (LAIR) pipeline.

A. Data Handling and Aggregation:

• Current State: The training loop requires input data structured as (c, x w, x l), where x w is the preferred image and x l is the dispreferred image for prompt c.

• Improved Implementation:

1. Input Structure: The system must ingest a list of candidate images for a single prompt: (c, x i i=1 N). N is the maximum list size (a hyperparameter).

2. Reward Integration: For each image x i in the the list, apply a pre-trained reward model to obtain its continuous score r i = (RewardModel (c, x i)).

3** Advantage Weight Calculation:** Calculate the centered advantage weights (w i) for each candidate:

p i = (r i / tau)

w i = p N - 1/N (where p N is the normalized softmax output, and tau is the temperature parameter).

B. Loss Function Re-engineering:

• Current State: The loss function relies on comparing two discrete points (the log-ratio of preferred vs. dispreferred).

• Improved Implementation: Replace the pairwise objective with an Advantage-Weighted Regression Objective:

L LAIR = sum i=1 N w i times (s theta(x 0, x t, t, c, epsilon i) - s ref(x 0, x t, t, c, epsilon)) squared

• w i: The centered advantage weight for each candidate.

• s theta:The implicit reward (denoising-loss improvement) of the current model theta.

• s ref: The corresponding implicit reward of a fixed reference model ref.

C. Regularization and Stability:

• Current State: Pairwise optimization often leads to unbounded implicit rewards, causing aggressive, unstable policy shifts.

• Improved Implementation: Introduce a quadratic penalty term (lambda) into the objective:

L Total = sum i=1 N w i times squared + lambda times s theta squared

• This explicit regularization controls the magnitude of the implicit reward, ensuring that even highly rewarded samples do not induce extreme, destabilizing updates away from the reference model.

The implementation of Diffusion LAIR enables the improved AI system to achieve capabilities that are fundamentally impossible with existing pairwise alignment methods:

A. Capture and Preserve Ranking Structure:

• Capability: The system learns not just which image is better, but the relative quality spacing between all candidates for a single prompt.

• Mechanism: By assigning positive weights to high-reward samples and negative weights to low-reward samples within the same list, the model is forced to refine its ability to distinguish subtle differences in quality (the ranking), rather than just making one sample win.

B. Robustness Against Noisy Preference Data:

• Capability: The system is inherently more stable and robust when processing large, imperfect datasets common in real-world collection (e.g., Pick-a-Pic).

• Mechanism: Because the LAIR objective utilizes a weighted average over all candidates, it does not require the perfect binary label of a single pair; it leverages the continuous signal from multiple candidates, mitigating errors introduced by noisy human annotators.

C. Efficient and Conservative Alignment:

• Capability: The system achieves superior alignment while maintaining training stability and computational efficiency.

• Mechanism: The closed-form, bounded nature of the optimal implicit reward (due to lambda) provides a clear theoretical guarantee that the model's deviation from the reference model is controlled, preventing catastrophic forgetting or unstable divergence during fine-tuning.

D. Broader Performance Gains (Transfer Learning):

• Capability: The system demonstrates generalization beyond the specific reward model used during training.

• Mechanism: By optimizing for a general listwise advantage rather than a specific pairwise comparison, the resulting alignment (e.g, improvements in ImageReward or Aesthetics) transfers successfully to other evaluation metrics, indicating that the learned feature representation is more robust and semantically aligned with human intent.

Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.