Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

arXiv:2604.04410 · cs.LG, cs.AI, cs.CL, stat.ML · Submitted 2026-04-06 · Read on arXiv

cs.LG, cs.AI, cs.CL, stat.ML

Submitted: 2026-04-06

Updated: 2026-09-26

Code: https://github.com/takahashihiroshi/rdro

Terminology

• Alignment — 26× in this paper · explained in 1 episode
• Consistency — 13× in this paper · explained in 3 episodes
• Convergence Guarantees — 8× in this paper · explained in 3 episodes
• Bradley-Terry model — 7× in this paper · explained in 2 episodes
• Alignment Methods — 6× in this paper · explained in 1 episode
• Generalization — 5× in this paper · explained in 13 episodes
• Regularization — 5× in this paper · explained in 3 episodes
• BIG-Bench Hard — 4× in this paper
• Covariate Shift — 4× in this paper · explained in 2 episodes
• Loss Function — 4× in this paper · explained in 2 episodes
• alpha — 4× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Benchmark — 3× in this paper · explained in 2 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Catastrophic Forgetting — 2× in this paper · explained in 5 episodes
• KL Regularization — 2× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Direct Preference Optimization (DPO) — 1× in this paper · explained in 9 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
• Meta-learning — 1× in this paper · explained in 2 episodes
• Nash Equilibrium — 1× in this paper · explained in 2 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Theoretical Guarantees — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• GPT-4 Technical Report
• Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• The Llama 3 Herd of Models
• Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
• Binary Classifier Optimization for Large Language Model Alignment
• Preference Optimization by Estimating the Ratio of the Data Distribution
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Decoupled Weight Decay Regularization
• Nash Learning from Human Feedback
• The Woman Worked as a Babysitter: On Biases in Language Generation
• Qwen2 Technical Report
• Solving math word problems with process- and outcome-based feedback
• Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks