Decoding-time Realignment of Language Models
summary
The gist
The gist: Decoding-time realignment (DeRa) is a simple method that allows users to explore and evaluate different regularization strengths in aligned language models without retraining, enabling
In short
Decoding-time Realignment (DeRa) is a method that allows users to adjust how aligned a language model's output is during generation without retraining. It blends between an unaligned reference model and an aligned model at decoding time using a parameter called lambda (λ). This lets users explore different alignment strengths simply by changing λ, controlling the trade-off between following the original model and adopting desired behaviors.
Key concepts
- Language Model Alignment
- This process aims to make language models behave safely and accurately by reducing factual errors and biases. It involves training models to generate responses that conform to human standards like helpfulness, while ensuring they retain their original knowledge.
- Regularization Strength (KL Divergence)
- Regularization is a term used during training that keeps the aligned model close to the original unaligned model. This closeness is measured by the Kullback-Leibler (KL) divergence. Too strong, and the model loses its original capabilities; too weak, and it might start 'reward hacking' by ignoring safety rules.
- Decoding-time Realignment (DeRa)
- DeRa is a simple technique that modifies how models generate text during inference. Instead of retraining, it blends the output from an unaligned model and an aligned model based on a parameter λ. This enables users to smoothly control the alignment level while generating responses.
- Lambda (λ)
- Lambda is the key variable in DeRa that controls the blend between models during decoding. When λ is 0, the output resembles the original unaligned model, and when λ is 1, it strongly resembles the aligned model. Adjusting this value allows for fine-grained control over how much alignment is applied to a specific response.
Terminology used across episodes
This episode discusses
- Decoding-time Realignment of Language Models · Paper Radio
- Concrete Problems in AI Safety
- PaLM 2 Technical Report
- A General Theoretical Paradigm to Understand Learning from Human Preferences
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Accelerating Large Language Model Decoding with Speculative Sampling
- UltraFeedback: Boosting Language Models with Scaled AI Feedback
- On Using Monolingual Corpora in Neural Machine Translation
- DeAL: Decoding-time Alignment for Large Language Models
- Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
- Mistral 7B
- Tuning Language Models by Proxy
- Scaling Up Models and Data with t5x and seqio
- Proximal Policy Optimization Algorithms
- Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Zephyr: Direct Distillation of LM Alignment
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- Fine-Tuning Language Models from Human Preferences
The paper
Decoding-time Realignment of Language Models · Read on arXiv
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko
Google DeepMind
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Decoding-time Realignment of Language Models".
Jane: The gist: Decoding-time realignment (DeRa) is a simple method that allows users to explore and evaluate different regularization strengths in aligned language models without retraining,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: This paper, "Decoding-time Realignment of Language Models," introduces decoding-time realignment as a simple way to explore and evaluate different regularization strengths in aligned language models without retraining > It tackles the problem where traditional alignment training involves a tradeoff between human preference rewards and a proximity regularization term, which is usually the KL divergence between unaligned and aligned models >
Jane: The core claim here is that this method enables control over the degree of alignment by allowing users to smoothly transition between unaligned and aligned models at decoding time > It points out that insufficient regularization can cause reward hacking, while too much hinders actual alignment, which is a tough balance to strike >
Lu: They show how you can compute this realigned model using Equation three from the paper without needing any retraining of the main model > This involves moving lambda to the exponent in Equation two so that it multiplicatively reweighs each response based on an importance ratio between the aligned and SFT models > <ref:2402.02992#pg1>
Meng: That sounds computationally efficient, but computing that distribution over all possible sequences is usually intractable, so they have to use some approximation techniques > The paper mentions a per-token approximation defined as πbθ(β/λ)(ytx, y1:t−one) > <ref:2402.02992#pg1>
Lalam: So it's not just about training better models; it's about giving users direct control over the model's behavior during inference, which is pretty powerful for immediate testing > This approximation is equivalent to a softmax combination of the logits from the reference model and the aligned model using lambda >
Conclusion: Tom: So wrapping up this discussion on "Decoding-time Realignment of Language Models," we're talking about how this technique lets us explore alignment strengths without retraining, which is a big deal for efficiency > The authors focus on making it a simple modification of the response sampling procedure to blend models at decoding time >
Jane: They show that by adjusting lambda, you can see tangible changes in the output, for example, in a toy summarization problem where lower lambda values give fake plans and higher values give warnings about those actions > This demonstrates how meaningful these adjustments to lambda actually are for controlling the alignment during decoding >
Lu: The implication is that this method could be widely applicable across different alignment approaches, including policy gradient methods and Direct Preference Optimization DPO > It shows it's a flexible tool rather than just one specific trick for one training method >
Meng: From an implementation standpoint, it streamlines hyperparameter selection and cuts down on the cost of retraining models just to test various regularization settings across a wide range of possibilities > It’s about saving compute cycles when you want to fine-tune behavior quickly >
Lalam: Ultimately, this paper suggests that we can tailor the model's adherence to desired behaviors dynamically for individual users or specific tasks right when they need it > It moves alignment control from a static training step into a dynamic generation step >
More episodes
- 2610.10616-When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
- 2610.10655-Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning
- 2610.11031-Language Modeling is Monotone Compression
- 2610.01253-Context-Aware Error Mitigation Orchestration for Hybrid Quantum Reinforcement Learning on NISQ Systems
- 2604.24201-CMGL: Confidence-guided Multi-omics Graph Learning for Cancer Subtype Classification
- 2609.34069-Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization
- 2312.01221-Enabling Quantum Natural Language Processing for Hindi Language
- 2508.08833-An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
- 2405.04118-Policy Learning with a Language Bottleneck
- 2407.14562-Thought-Like-Pro: Enhancing Reasoning of Large Language Models through Self-Bootstrapped Prolog-based Chain-of-Thought