Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

arXiv:2609.33791 · cs.LG, cs.AI · Submitted 2026-09-27 · Read on arXiv

cs.LG, cs.AI

Submitted: 2026-09-27

Updated: 2026-09-27

Code: https://github.com/LeapLabTHU/KL-Free-OPD

Terminology

• Distillation — 72× in this paper · explained in 2 episodes
• On-Policy Distillation — 57× in this paper · explained in 3 episodes
• Verifier — 39× in this paper · explained in 2 episodes
• Accuracy — 26× in this paper · explained in 1 episode
• Benchmark — 13× in this paper · explained in 2 episodes
• Multi-Teacher On-Policy Distillation — 13× in this paper
• Cosine Similarity — 9× in this paper · explained in 3 episodes
• Redundancy — 8× in this paper · explained in 2 episodes
• GRPO — 7× in this paper · explained in 6 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Knowledge Distillation — 3× in this paper · explained in 11 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Distribution Matching — 1× in this paper · explained in 2 episodes
• Self-Distillation — 1× in this paper · explained in 7 episodes

Sources

• Reward-Gated On-Policy Distillation
• Program Synthesis with Large Language Models
• NVIDIA Nemotron 3: Efficient and Open Intelligence
• H$^2$SD: Hybrid Hindsight Self-Distillation
• Evaluating Large Language Models Trained on Code
• Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
• Process Reinforcement through Implicit Rewards
• SAF-OPD: Stable Advantage Fusion for On-Policy Distillation
• Rethinking On-Policy Distillation of Large Language Models II: One Training Example
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
• Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs
• Distilling the Knowledge in a Neural Network
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
• OpenAI o1 System Card
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• On-policy Distillation with Verifiable Reward
• MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks