What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

arXiv:2609.05198 · cs.AI · Submitted 2026-09-04 · Read on arXiv

cs.AI

Submitted: 2026-09-04

Updated: 2026-09-26

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 33× in this paper · explained in 1 episode
• Distillation — 32× in this paper · explained in 2 episodes
• On-Policy Distillation — 17× in this paper · explained in 3 episodes
• Alignment — 11× in this paper · explained in 1 episode
• Benchmark — 7× in this paper · explained in 2 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Exposure Bias — 3× in this paper · explained in 2 episodes
• Verifier — 3× in this paper · explained in 2 episodes
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Structural Reasoning — 2× in this paper · explained in 2 episodes
• Aggregation — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Thinking mode — 1× in this paper · explained in 2 episodes

Sources

• Rethinking On-Policy Distillation of Large Language Models II: One Training Example
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
• Large-Scale Data Selection for Instruction Tuning
• Entropy-Aware On-Policy Distillation of Language Models
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
• MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
• KL for a KL: On-Policy Distillation with Control Variate Baseline
• HybridFlow: A Flexible and Efficient RLHF Framework
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
• Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
• LESS: Selecting Influential Data for Targeted Instruction Tuning
• MiMo-V2-Flash Technical Report
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Qwen3 Technical Report
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
• GLM-5: from Vibe Coding to Agentic Engineering

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection