DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

arXiv:2610.11659 · cs.CL, cs.AI · Submitted 2026-10-08 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/THUDM/slime

Terminology

• Distillation — 43× in this paper · explained in 2 episodes
• Accuracy — 37× in this paper · explained in 1 episode
• On-Policy Distillation — 28× in this paper · explained in 3 episodes
• Benchmark — 13× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Self-distillation — 5× in this paper · explained in 6 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Distribution Matching — 1× in this paper · explained in 2 episodes
• Exposure Bias — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Training Verifiers to Solve Math Word Problems
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
• Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
• Weak-to-Strong Generalization via Direct On-Policy Distillation
• Rethinking On-Policy Distillation of Large Language Models II: One Training Example
• GLM-5: from Vibe Coding to Agentic Engineering
• MiniLLM: On-Policy Distillation of Large Language Models
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Measuring Mathematical Problem Solving With the MATH Dataset
• The Curious Case of Neural Text Degeneration
• DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models
• Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
• When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
• Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
• Entropy-Aware On-Policy Distillation of Language Models
• Kimi K3: Open Frontier Intelligence
• Solving Quantitative Reasoning Problems with Language Models
• CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering