Recovering Off-Policy Supervision for Speculative Decoding

arXiv:2609.38795 · cs.CL, cs.CV, cs.LG · Submitted 2026-09-30 · Read on arXiv

cs.CL, cs.CV, cs.LG

Submitted: 2026-09-30

Updated: 2026-10-04

Code: https://github.com/js-lee-AI/ALR-IRA

Project page: https://math-ai-org.github.io/aime25

Terminology

• Benchmark — 25× in this paper · explained in 2 episodes
• Speculative Decoding — 19× in this paper · explained in 2 episodes
• Distillation — 18× in this paper · explained in 2 episodes
• Verification — 13× in this paper · explained in 2 episodes
• In-Rollout Anchors — 6× in this paper
• Vision-language models — 6× in this paper · explained in 2 episodes
• Knowledge Distillation — 5× in this paper · explained in 11 episodes
• On-Policy Distillation — 3× in this paper · explained in 3 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Anchor-Label Relabelling — 2× in this paper
• Robustness — 2× in this paper · explained in 5 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• American Invitational Mathematics Examination — 1× in this paper
• Block Diffusion — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Covariate Shift — 1× in this paper · explained in 2 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Tokenization — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
• Qwen3-VL Technical Report
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Accelerating Large Language Model Decoding with Speculative Sampling
• ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
• DFlash: Block Diffusion for Flash Speculative Decoding
• Evaluating Large Language Models Trained on Code
• Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs
• MiniLLM: On-Policy Distillation of Large Language Models
• Distilling the Knowledge in a Neural Network
• Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• There is no rose without a thorn: Finding weaknesses on BlenderBot 2.0 in terms of Model, Data and User-Centric Approach
• Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
• Speculative Decoding Reimagined for Multimodal Large Language Models
• Online Speculative Decoding
• Blockwise Parallel Decoding for Deep Autoregressive Models
• SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding
• ParallelSpec: Parallel Drafter for Efficient Speculative Decoding

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering