On-Policy Distillation with Negative-Policy Rollouts
summary
The gist
The gist The introduction introduces Negative-Policy Distillation (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy to provide an
In short
Negative-Policy Distillation (NP-OPD) enhances On-Policy Distillation by adding rollouts from a lower-performing negative policy to provide an explicit negative signal. This complements teacher supervision, helping the student model avoid tokens preferred by the negative policy while maintaining positive guidance from the teacher. Experiments show this improves performance across various tasks and scales.
Key concepts
- On-Policy Distillation (OPD)
- OPD trains a student model using its own generated data and token-level supervision from a stronger teacher model. It works best when the student's data distribution closely matches the teacher's, providing positive guidance for learning.
- Negative-Policy Rollouts
- This involves sampling trajectories from an additional, lower-capability negative policy during training. This provides an explicit negative signal to the student, showing it which tokens are preferred by a weaker policy compared to the teacher.
- Negative Policy Suppression Precision (NSP)
- NSP measures how well NP-OPD suppresses tokens favored by the negative policy over the teacher. Higher NSP indicates that NP-OPD is effectively pushing the student away from undesirable outputs generated by the negative policy.
Terminology used across episodes
This episode discusses
- On-Policy Distillation with Negative-Policy Rollouts · Paper Radio
- OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- Weak-to-Strong Generalization via Direct On-Policy Distillation
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- Gemma 4 Technical Report
- The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains
- MiniLLM: On-Policy Distillation of Large Language Models
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
- Measuring Mathematical Problem Solving With the MATH Dataset
- On-Policy Delta Distillation
- Distilling the Knowledge in a Neural Network
- Entropy-Aware On-Policy Distillation of Language Models
- DistiLLM: Towards Streamlined Distillation for Large Language Models
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
- AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
The paper
On-Policy Distillation with Negative-Policy Rollouts · Read on arXiv
Jaehui Hwang, Dongyoon Han, Sangdoo Yun, Byeongho Heo
NAVER AI Lab
On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at https://github.com/naver-ai/np-opd.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "On-Policy Distillation with Negative-Policy Rollouts".
Jane: The gist The introduction introduces Negative-Policy Distillation (NP-OPD), which complements teacher supervision with rollouts from a lower-performing,
Tom: First, who's behind it and why it matters.
Title and authors: Tom: Let's talk about the title and who wrote this. It’s "On-Policy Distillation with Negative-Policy Rollouts", and it’s written by Jaehui Hwang Dongyoon Han Sangdoo Yun Byeongho Heo, along with others from NAVER AI Lab >
Jane: The paper is essentially tackling the limitation of standard on-policy distillation. It shows how you can use a negative policy to help steer the student away from bad choices when the teacher's signal isn't strong enough >
Lu: What’s interesting is that they don't change the core distillation reward formulation, they just modify where those rollouts come from during training >
Meng: So it’s not a whole new training objective, it’s an addition to the rollout distribution itself, which is pretty smart for deployment because it keeps the familiar on-policy setup >
Lalam: It also mentions that this negative policy is selected from the same model family as the student but with lower model capacity and overall reasoning performance >
The paper's summary: Tom: So, to summarize, this work introduces NP-OPD. It complements the positive guidance from a stronger teacher by adding rollouts from a negative policy that shows tokens preferred by that negative policy over the teacher >
Jane: That explicit negative signal is what helps the student model learn away toward things it shouldn't be doing, especially when the distributions of the teacher and student are not very similar >
Lu: They mathematically define this change in gradient contribution. The difference between OPD and NP-OPD is quantified by an expression involving alpha, which mixes the two policies during sampling >
Meng: The paper shows that this means tokens generated by the negative policy have a negative expected OPD reward, so the distillation process trains the student not to generate those tokens >
Lalam: That’s a concrete idea. It suggests that if you sample from this negative policy, the resulting outputs will have a lower expected reward under the on-policy distillation framework >
The paper's improvements: Tom: The main improvement they point to is that NP-OPD consistently improves on standard OPD across different model sizes and generation modes, reasoning domains, and even different variants of OPD >
Jane: They show that NP-OPD effectively suppresses tokens favored by the negative policy over the teacher and moves the student away from those undesirable tokens >
Lu: They quantify this suppression with a metric called Negative-Policy Suppression Precision or NSP, which measures what fraction of suppressed tokens are actually preferred by the negative policy over the teacher >
Meng: They also found that NP-OPD can be used to reuse pre-generated negative-policy rollouts, which is a big practical thing because it reduces the cost of generating those rollouts during training >
Lalam: And they measured a shift in probability gaps toward the teacher’s top prediction relative to the negative policy’s at disagreement positions, showing a mean NOR of zero point zero one nine under NP-OPD compared to zero point zero zero five under OPD >
Conclusion: Tom: So, wrapping this up, NP-OPD is a way to get an explicit negative signal into on-policy distillation by using a lower-capability policy as a reference for the student >
Jane: It seems to be very effective because it works even when the positive guidance from the teacher is weak and it helps the student learn away from unwanted behaviors >
Lu: They also showed that this method can complement existing advanced OPD variants like ExOPD and OPD2 without needing to change their reward functions >
Meng: From a deployment standpoint, reusing pre-generated negative rollouts gives us training efficiency gains, which is something we need when we're running models at scale >
Lalam: Ultimately, this paper on On-Policy Distillation with Negative-Policy Rollouts shows that adding an explicit negative reference can significantly improve the student model's ability to avoid undesired outputs while keeping the core distillation process intact >
More episodes
- 2610.07475-Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity
- 2610.07458-Interpretable Hypergraph Learning via Neural Additive Models
- 2610.07485-Robust Importance Sampling for Rare Events via Constrained Gaussian Mixtures
- 2610.07499-Source-Learned Reliance for Selective Test-Time Adaptation of Multimodal Time Series
- 2610.07602-A Neural JKO Scheme for Hellinger-Kantorovich Gradient Flows via Monge-Growth Pairs
- 2610.07662-MS-ECG-FM: Towards a More Universal Electrocardiogram Foundation Model for Health Monitoring using Multi-source Contrastive Learning
- 2610.07610-Hub for Outliers, Spokes for Inliers: Uniform Latent Space Construction for Dual-Mismatched Semi-Supervised Learning
- 2610.07825-Forecast Accuracy Is Not Trading Profit: Evolving Small Recurrent Networks for Stock Return Prediction
- 2610.07529-Targeted search shows that random-device testing underestimates worst-case error in a simulated wave-based neural operator
- 2610.07559-TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity