SR-OPSD: Self-Referenced On-Policy Self-Distillation

SR-OPSD: Self-Referenced On-Policy Self-Distillation proposes a new framework for on-policy self-distillation (OPSD) in large language models. The paper states: "On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the authors identify a core problem: the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration."

To address this, the paper introduces SR-OPSD, which "at fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. The formulation separates where the adaptive target is placed from how the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios."

The main contributions are threefold: "(i) We adapt the reference-regularized on-policy self-distillation formulation to frozen-context self-distillation and construct a normalized target from a self-teacher policy and a reference policy. (ii) We optimize this target using Rényi divergence and derive the resulting logit gradient, showing that Rényi order controls the power weighting of self-teacher-to-student probability ratios without changing the unconstrained target distribution. (iii) We evaluate the resulting method across scientific reasoning, mathematical reasoning, and code generation, and open-source the code which supports multiple hardware platforms, including NVIDIA GPUs."

The method is formalized through two key propositions. Proposition 3.1 establishes a Token-Level Variational Characterization of OPSD, showing that maximizing the objective Jα is equivalent to minimizing the reverse KL from the student to πα⋆, where πα⋆ is a normalized geometric interpolation between the self-teacher and reference policies. Proposition 3.2 extends this to SR-OPSD, showing that maximizing Jα,ρ with respect to θ is equivalent to minimizing the forward Rényi projection from πα⋆ (· hsf, href) to πθ (· h). Proposition 3.3 derives the logit gradient, showing that ρ power-tempers the target-to-student density ratio.

Experiments are conducted across three settings. First, on SciKnowEval science Q&A benchmarks with Qwen3-8B and OLMo3-7B-Instruct, the paper reports: SR-OPSD achieves the best 15-hour Avg@16 accuracy in seven of the ten cases and outperforms both GRPO variants in all ten settings. For Qwen3-8B, SR-OPSD surpasses the strongest competing baseline by 2.3, 8.2, 1.6, and 1.2 percentage points on Chemistry, Physics, Materials, and Tool Use, respectively. Averaged across five domains, SR-OPSD reaches 75.2 Avg@16 at 15 hours, compared with 72.4 for SDPO and 69.2 for GRPO.

Second, on mathematical reasoning benchmarks (AIME 2024, AIME 2025, HMMT 2025, AMC 2023, Minerva) with Qwen3-4B, the paper states: SR-OPSD achieves the strongest overall performance across the five mathematical-reasoning benchmarks. It obtains the best Avg accuracy on AIME25 and Minerva, ties for the best Avg accuracy on HMMT25, and achieves the best or tied-best Pass accuracy on AIME24, AIME25, HMMT25, and AMC23. Compared with GRPO, SR-OPSD improves Pass accuracy by 6.7 points on AIME25 and 10.0 points on HMMT25. Averaged uniformly, SR-OPSD reaches 56.2 Avg accuracy and 78.4 Pass accuracy, compared with 55.0 and 75.2 for GRPO. The advantage over self-distillation baselines is pronounced: SR-OPSD exceeds forward-KL OPSD by 9.4 points in mean Avg accuracy and 11.5 points in mean Pass accuracy, and exceeds reverse-KL SDPO by 14.1 and 8.0 points, respectively.

Third, on code generation with LiveCodeBench v6 across Qwen3 model scales (0.6B, 1.7B, 4B, 8B), the paper reports: SR-OPSD achieves the best performance on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, reaching 32.0, 46.2, and 50.1, respectively. Compared with GRPO, these results correspond to improvements of 2.7, 6.3, and 8.9 percentage points, while the gains over SDPO are 1.3, 1.5, and 1.3 points. On Qwen3-0.6B, SR-OPSD remains competitive but trails GRPO and SDPO by 0.5 and 0.7 points, respectively, though it still nearly doubles the base-model pass-all success rate, improving it from 9.2 to 18.0.

The ablation study on SciKnowEval Physics shows that self-reference does not improve performance when it is combined with the standard JSD or forward-KL objectives. Adding self-reference reduces Avg@16 from 79.4 to 77.2 for JSD and from 79.1 to 76.3 for forward KL. By contrast, "SR-OPSD, which combines self-reference with Rényi projection, achieves the strongest performance. Its Avg@16 score increases from 79.9 at ρ = 0.50 to 80.6 at ρ = 0.70 and 81.1 at ρ = 0.95, outperforming the best non-Rényi variant by 1.9 points."

The paper concludes: "Our results highlight that the effectiveness of self-reference depends critically on the objective through which it is imposed. Reference anchoring alone is not sufficient, and self-reference can even degrade performance under standard forward-KL or JSD objectives. In contrast, coupling the self-referenced target with an appropriate Rényi projection improves performance and training stability across scientific reasoning, mathematical reasoning, and code generation. Limitations acknowledged include: Our theoretical characterization concerns token-level optimization at fixed rollout contexts and does not establish global convergence of the fully coupled student–self-teacher dynamics, and the target-interpolation coefficient, Rényi order, and teacher update rate are fixed and selected empirically."

Improvements to AI Systems:

1. Stabilized On-Policy Self-Distillation for Continuous Learning

• Replace standard forward-KL or JSD objectives with Rényi-divergence-based projection (order ρ > 0.5) when using a self-referenced teacher. This prevents distributional collapse and unstable optimization caused by moving targets.

• The improved system can maintain training stability over long horizons (e.g., 15+ hours) without performance degradation, as demonstrated by consistent gains over GRPO and SDPO across domains.

2. Adaptive Target Interpolation with Reference Anchoring

• Use a normalized geometric interpolation between the self-teacher and a fixed reference policy, with the interpolation coefficient controlling target placement. This separates target selection from projection geometry.

• The improved system can dynamically balance exploration (reference policy) and exploitation (self-teacher) during training, leading to better generalization on unseen reasoning tasks (e.g., +6.7 points on AIME25 Pass accuracy vs. GRPO).

3. Rényi-Order-Controlled Density Ratio Weighting

• Apply ρ power-tempering to self-teacher-to-student probability ratios in the logit gradient, allowing fine-grained control over how much the student trusts the teacher’s token-level feedback.

• The improved system can adjust sensitivity to high-confidence vs. low-confidence teacher predictions, reducing overfitting to noisy or spurious token-level signals. This yields +8.9 points on LiveCodeBench for Qwen3-8B vs. GRPO.

4. Frozen-Context Self-Distillation for Efficient Multi-Task Training

• Fix student-generated contexts during distillation to decouple the teacher’s evolution from the student’s context distribution, avoiding co-adaptation.

• The improved system can train on diverse benchmarks (science, math, code) simultaneously without catastrophic forgetting, achieving best or tied-best accuracy on 7 of 10 SciKnowEval domains and 4 of 5 math benchmarks.

5. Hardware-Agnostic Deployment with Open-Source Implementation

• Use the released codebase supporting NVIDIA GPUs to deploy SR-OPSD across different model scales (0.6B to 8B) with minimal adaptation.

• The improved system can scale from small to large models while retaining performance gains (e.g., +2.7 to +8.9 points over GRPO on code generation), making it suitable for resource-constrained environments.

6. Robustness to Objective Choice in Self-Reference

• Automatically select Rényi order ρ (e.g., 0.70–0.95) based on validation performance, avoiding the performance drop seen with standard JSD or forward-KL when self-reference is added (e.g., 79.4→77.2 for JSD).

• The improved system can reliably incorporate self-reference without manual tuning, achieving up to 81.1 Avg@16 on SciKnowEval Physics—a 1.9-point improvement over the best non-Rényi variant.

What the Improved AI System Can Do:

• Achieve state-of-the-art performance in scientific reasoning (75.2 Avg@16), mathematical reasoning (56.2 Avg, 78.4 Pass), and code generation (50.1 on LiveCodeBench) with faster convergence and better stability than GRPO or SDPO.

• Learn from its own outputs without collapsing into mode-seeking behavior, even when the teacher and student evolve jointly.

• Adapt to new tasks or domains with minimal retraining, thanks to reference-regularized targets and Rényi projection that prevent over-concentration on the teacher’s biases.

• Operate on standard GPU hardware, enabling reproducible and scalable deployment for research and production.

On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed Self-Referenced On-Policy Self-Distillation (SR-OPSD). At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the R'enyi divergence family to generalize the projection geometry. This formulation separates where the adaptive target is placed from how the student is projected toward it: the interpolation coefficient controls underlying target, while the R'enyi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.