A Zeroth-Order Paradigm for LLM Preference Alignment
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-16
Updated: 2026-09-16
Comments: 39 pages
Code: https://github.com/tatsu-lab/alpaca_eval
License: http://creativecommons.org/licenses/by/4.0/
The gist: Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency.
Terminology
Abstract
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.
Sources
- GPT-4 Technical Report
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- Two-Fidelity Best-Action Identification for Stochastic Minimax Tree
- Direct Language Model Alignment from Online AI Feedback
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- LLaMA: Open and Efficient Foundation Language Models
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- Fine-Tuning Language Models from Human Preferences
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering