Simple-OPD: Demystifying Warm-up for On-policy Distillation

arXiv:2608.06802 · cs.CL · Submitted 2026-08-07 · Read on arXiv

Tao Liu, Taiqiang Wu, Mao Zheng, Xuan Luo, Runming Yang, Xuewei Yang, Junjie Wang, Yujiu Yang

cs.CL

Submitted: 2026-08-07

Updated: 2026-08-10

Code: https://github.com/Utaotao/Simple-OPD

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Distillation — 39× in this paper · explained in 2 episodes
• On-Policy Distillation — 25× in this paper · explained in 3 episodes
• Benchmark — 21× in this paper · explained in 2 episodes
• Generalization — 21× in this paper · explained in 13 episodes
• Adaptation — 18× in this paper · explained in 1 episode
• Verification — 7× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Domain Adaptation — 3× in this paper · explained in 5 episodes
• GRPO — 3× in this paper · explained in 6 episodes
• Low-Rank Adaptation (LoRA) — 3× in this paper · explained in 9 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Self-Distillation — 3× in this paper · explained in 7 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Consistency — 2× in this paper · explained in 3 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning — 1× in this paper
• Thinking mode — 1× in this paper · explained in 2 episodes

Sources

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
• Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
• Evaluating Large Language Models Trained on Code
• Entropy-Aware On-Policy Distillation of Language Models
• Rethinking On-Policy Self-Distillation for Thinking Models
• Weak-to-Strong Generalization via Direct On-Policy Distillation
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
• KL for a KL: On-Policy Distillation with Control Variate Baseline
• Trust-Region Behavior Blending for On-Policy Distillation
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
• MiMo-V2-Flash Technical Report
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
• Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering