Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models

arXiv:2609.38025 · cs.LG, cs.AI, cs.CL, math.OC · Submitted 2026-09-29 · Read on arXiv

cs.LG, cs.AI, cs.CL, math.OC

Submitted: 2026-09-29

Updated: 2026-09-29

Code: https://github.com/zywang0701/Dr-OPD

Terminology

• Distillation — 58× in this paper · explained in 2 episodes
• GRPO — 18× in this paper · explained in 6 episodes
• On-Policy Distillation — 17× in this paper · explained in 3 episodes
• Actor — 15× in this paper · explained in 3 episodes
• Generative AI — 5× in this paper · explained in 2 episodes
• Benchmark — 4× in this paper · explained in 2 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Alignment — 3× in this paper · explained in 1 episode
• Core — 3× in this paper · explained in 2 episodes
• Verifier — 3× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Meta-learning — 2× in this paper · explained in 2 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Reward Function — 2× in this paper · explained in 2 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• GRPO Objective — 1× in this paper · explained in 2 episodes
• Influence Functions — 1× in this paper · explained in 2 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Model-Agnostic — 1× in this paper · explained in 2 episodes
• Self-Distillation — 1× in this paper · explained in 7 episodes

Sources

• Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
• Process Reinforcement through Implicit Rewards
• Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
• Measuring Coding Challenge Competence With APPS
• On-Policy Delta Distillation for Multilingual Math Reasoning
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
• TACO: Topics in Algorithmic COde generation dataset
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• On-policy Distillation with Verifiable Reward
• Extremely Sparse Supervision Incentivizes Reasoning Ability
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
• REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation
• Kimi K3: Open Frontier Intelligence
• MiMo-V2-Flash Technical Report
• On the Position Bias of On-Policy Distillation
• Trust Region On-Policy Distillation
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks