Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

SKALD (Skill-Anchored Latent Distillation) is an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model: a question-only student and a teacher conditioned on an abstract, explicit-answer-filtered skill card. The student is trained on its own prefixes, transferring the skill-induced advantage into shared parameters without privileged input at test time. To stabilize context-induced distribution mismatch, SKALD employs an annealed exponentially tilted objective that downweights teacher-preferred tokens with very low student likelihood; as the tilt vanishes, it converges to teacher cross-entropy and recovers the forward-KL student gradient. An empirical gate activates distillation only when verified rollouts estimate a positive teacher advantage. Across five held-out mathematics benchmarks, SKALD improves overall avg@8 over GRPO by +2.46, +4.85, and +12.01 at 0.6B, 1.7B, and 4B, respectively. At 1.7B, zero-variance-only distillation recovers 84.7% of the full gain, while SKALD remains +4.06 above FLOP-matched GRPO and exceeds contextual skill exposure by +3.77. These results show that abstract skills provide dense supervision where group-relative rewards become uninformative.

Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.0–68.0% of groups in the experiments. The central question is what privileged information to provide. A full reference solution exposes the final answer and a particular derivation to the teacher branch; a larger online teacher avoids that input but adds training and deployment dependencies. SKALD instead conditions the self-teacher on a compact skill abstraction—a relevant principle, when it applies, and common mistakes to avoid—while the student sees the question alone. Both branches score the same student-generated prefixes. At test time, neither the skill nor a teacher branch is required. The method does use a larger model offline to construct and screen cards; the no larger teacher claim therefore refers only to online distillation and inference. This design contrasts skill use with skill internalization: a skill in the student prompt acts as a temporary scaffold, whereas teacher-only distillation changes the parameters used by the question-only policy. In the 1.7B no-skill-at-test comparison, GRPO reaches 45.52, training-time contextual exposure reaches 46.60, and SKALD reaches 50.37. Thus SKALD is +3.77 above contextual exposure in this setting; the comparison supports a weight-level transfer effect but does not by itself isolate every training difference.

Direct teacher cross-entropy can be unstable when a privileged context assigns substantial mass to tokens the student currently considers improbable. SKALD uses the annealed objective −τ −1 log Ev∼q [p(v)τ]. At τ > 0, it creates a student-dependent effective target that discounts very low-p teacher tokens and has the sharpened stationary solution p∗ ∝ q 1/(1−τ); annealing τ: 0.8→0 removes this transient mode-seeking bias and recovers teacher cross-entropy, equivalently the forward-KL student gradient under stop-gradient. A fixed empirical gate further filters distillation using an initial sampled accuracy difference; it is a noisy heuristic, not a certificate of teacher superiority throughout training. Across Qwen3-Base models at 0.6B, 1.7B, and 4B, SKALD outperforms GRPO on the problem-weighted average of MATH500, AMC23, AIME24, AIME25, and Minerva, with observed gains of +2.46, +4.85, and +12.01. Three sizes from one model family establish an encouraging monotonic trend, not a general scaling law.

The contributions are: introducing shared-parameter, on-policy skill distillation that transfers an explicit-answer-filtered abstraction into a question-only policy without a larger online teacher or test-time privileged input; characterizing the tilted objective as a Rényi-type cross-entropy with a student-dependent escort target, bounded logit-gradient coordinates, and a cross-entropy limit whose student gradient matches forward KL, with the contribution being its annealed use under privileged on-policy mismatch; and directly localizing the gain to zero-variance rollout groups and validating it with conditional-leakage audits, problem/solution/skill decontamination, hierarchical uncertainty, equal-compute GRPO, gate sensitivity, and a full 4B component ablation.

The method trains a single model through two context views. Skill conditioning can, but need not, improve the model's conditional distribution on a given problem; the empirical gate is designed to filter cases in which a sampled improvement is not observed. A skill-conditioned branch scores the rollouts of a question-only branch, and an annealed cross-entropy objective transfers their distributional difference into the shared weights. For each scale, a single Qwen3-Base checkpoint pθ is instantiated as two context views that share all parameters. The student pS (· x):= pθ (· renderS (x)) sees only the problem x and matches the deployment condition exactly; the self-teacher pT (· x, s):= pθ (· renderT (x, s)) additionally sees an abstract skill s. This construction eliminates the two standing costs of external teachers at once: there is no capacity mismatch, because teacher and student share every parameter, and no serving dependency, because the teacher exists only as an alternative prompt. All trajectories ŷ are sampled from the student. For each position, define pn:= pS (· x, ŷ<n) and qn:= sg[pT (· x, s, ŷ<n)], where sg denotes stop-gradient. This operation freezes the teacher only within one update: because the branches share parameters, the teacher distribution changes after every optimizer step. SKALD is therefore a coupled moving-target self-distillation procedure, not distillation from a fixed teacher. During optimization, the teacher scores student rollouts in one prefill pass; separate teacher rollouts are used only to precompute the usefulness gate. This asymmetry keeps all distillation prefixes on-policy under the student, avoiding supervision on teacher-generated prefix distributions.

The skill is a structured card s = title, principle, usage, common mistakes mined offline from training solutions. The pipeline retains 11,382 cards from 17,384 candidates and merges them into 1,926 reusable skills; a skill is linked to a median of 5 problems (mean 7.4), and 93.1% of training problems receive an assignment. A larger Qwen3-14B model is used only for offline extraction and screening, not online distillation or inference. In a conditional audit, answer recovery is 2.4% from the question alone, 3.7% with a shuffled skill, 5.1% with the matched skill, and 94.2% with the full solution; removing instance-specific numerals, entities, and intermediate expressions changes avg@8 only from 50.37 to 50.24. These controls substantially narrow, but cannot logically eliminate, conditional-leakage concerns. Boxed-answer extraction yields the binary reward R:= correct(x, y). The default gate is precomputed from the initial shared checkpoint using K = 8 rollouts per branch and remains fixed. It is an empirical filter, not a certificate: it covers 61.2% of training problems, and 14.6% of initially positive cases reverse sign by training's end. Nevertheless, hard/soft and fixed/refreshed variants score within 49.98–50.48 at 1.7B.

For the objective, direct teacher cross-entropy can be dominated by positions at which the teacher assigns mass to tokens that the student currently considers highly improbable. Let ln (v):= log pn (v) and dn (v):= −ln (v). For τ > 0, SKALD uses L(τ)n = −(1/τ) log Ev∼q [eτ ln(v)] = −(1/τ) log Σv qn(v)pn(v)τ. L(0)n:= H(qn, pn) by continuity. Thus limτ→0 L(τ)n = H(qn, pn) = KL(qn∥pn) + H(qn). Because qn is stop-gradient within an update, the limit has exactly the same student gradient as forward KL, but its numerical value differs by the teacher entropy H(qn). For small positive τ, the cumulant expansion is L(τ)n = Eqn [dn] − (τ/2) Varqn [dn] + O(τ 2). The negative variance correction is not a variance penalty; it is an optimistic entropic risk for the loss dn: high-disutility tokens are discounted relative to ordinary cross-entropy. Lemma 1 states that if rn(τ)(v):= qn(v)pn(v)τ / Σu qn(u)pn(u)τ, then ∇zn L(τ)n = pn − rn(τ). Consequently, each logit-gradient coordinate lies in [−1, 1]. For 0 < τ < 1, the minimizer over the probability simplex is p∗n(v) = qn(v)1/(1−τ) / Σu qn(u)1/(1−τ). At τ = 0, rn = qn and p∗n = qn. At the initial τ = 0.8, the stationary distribution is proportional to qn5, a strongly sharpened escort rather than the original teacher distribution. This mode-seeking bias is used only transiently: as the student's support improves and τ is annealed to zero, the effective target approaches qn and the objective approaches teacher cross-entropy. The training loss combines the verifiable reward with gated skill distillation: L(θ) = λR LR (θ) + λD Ex [gK (x) Eŷ∼pS (1/ŷ) Σn L(τ)n], where LR is group-normalized REINFORCE (GRPO) on student rollouts and gK (x) = 1[∆̂(0)K (x) > 0], with ∆̂(0)K (x) = (1/K) Σi [R(yiT) − R(yiS)], where yiT ∼ pT (· x, s) and yiS ∼ pS (· x). The superscript (0) denotes the initial shared checkpoint; the resulting gate remains fixed during training. It selects problems for which the sampled initial difference is positive, but finite-K error and later teacher drift can create false positives, false negatives, and sign reversals. On a rollout group with identical rewards, the GRPO term has zero group-relative gradient, whereas distillation can remain nonzero whenever gK (x) = 1 and rn(τ) ≠ pn. Setting s = y∗, τ = 0, g ≡ 1, and λR = 0 yields the OPSD student gradient because H(q, p) and KL(q∥p) differ only by the stop-gradient teacher entropy. Dropping the distillation term yields GRPO.

Skill distillation is most plausible under three measurable desiderata. First, reliability: skill conditioning should improve expected rollout accuracy, ∆∗(x) = EyT∼pT [R(yT)] − EyS∼pS [R(yS)] > 0. The fixed gate estimates this quantity only at initialization and with finite samples. Second, nontriviality: the skill should change the next-token distribution on reasoning-relevant positions rather than only boilerplate. Third, manageable mismatch: the privileged context should avoid an extreme teacher–student support gap. These are diagnostic desiderata, not sufficient conditions for policy improvement, and the shared-parameter teacher may cease to satisfy them as training evolves.

In experiments, the evaluation covers MATH500 (500 problems), AMC23 (40), AIME24 (30), AIME25 (30), and Minerva (272), totaling 872 problems. Each checkpoint uses eight samples at T = 0.7, top-p = 0.95; avg@8 is primary and overall scores are problem-weighted micro-averages. Training uses DAPO-Math-17k and the mathematics split of OpenThoughts. Exact, formula-normalized, and semantic checks flag unions of 3 problem texts, 14 reference solutions, and 10 skill cards against the evaluation suite; every reported model is retrained after removing the union, leaving 23,691 problems. The conditional leakage audit uses a common judge and decoding protocol: question-only, empty-card, random-text, shuffled-skill, matched-skill, and full-solution recovery rates are 2.4, 2.5, 2.6, 3.7, 5.1, and 94.2%, respectively. A three-rater audit finds no full-answer leak in 200 retained cards (κ = 0.83), and the sanitized-card score is 50.24 versus 50.37 for the original. Median/95th-percentile normalized solution–card overlap is 0.03/0.11. At 1.7B, SKALD-shuffled reaches 48.83: +3.31 over GRPO, +0.32 over OPSD, and −1.54 below matched SKALD. Thus matching matters, but generic card exposure or regularization can explain part of the gain. Three training seeds are reported, and a two-level bootstrap first resamples seeds and then evaluation problems within seed; decoding samples are aggregated at the problem level. Compute accounting includes scoring and gate construction, and equal-FLOP controls are reported.

SKALD attains the best overall avg@8 and pass@8 at all three tested scales, with mean gains over GRPO of +2.46, +4.85, and +12.01. At 1.7B, the per-seed differences are +4.70, +4.91, and +4.94; the hierarchical 95% interval is [+3.92, +5.78]. Rollout-matched GRPO uses 88.0 GPU-hours and reaches 45.52; an equal-FLOP/equal-GPU-hour run uses 131.2 GPU-hours and reaches 46.31, versus SKALD's 50.37. At 4B, tuned equal-FLOP GRPO reaches 52.19 ± 0.18, while SKALD reaches 63.38 ± 0.32. The central mechanism is directly visible in the rollout groups. Zero-variance groups account for 68.0%, 63.0%, and 63.9% at 0.6B, 1.7B, and 4B; with scale, their composition shifts from all-wrong (54.2% → 21.8%) toward all-correct (13.8% → 42.1%). At 1.7B, zero-variance-only distillation reaches 49.63 (+4.11 over GRPO), recovering 84.7% of the full gain, while mixed-only distillation reaches 46.31 (+0.79). This supports the claim that SKALD primarily adds signal where group-relative advantages vanish. Against privileged baselines at 1.7B, SKALD is +1.86 above OPSD and +3.04 above EOPD. The full 4B ablation is also monotonic under one addition order: OPSD 55.67, skill abstraction 57.32, tilted objective 61.05, and the gate 63.38. These are ordered marginal effects rather than interaction-free contributions.

For robustness, the conditional audit shows a modest +2.7-point recovery increase from question-only to question+matched-skill, far below the +91.8 increase from a full solution. More importantly, deleting instance-specific numerals, entities, and intermediate expressions preserves the downstream score (50.24 versus 50.37), and replacing each card by a peer card from the same abstract-method cluster changes overall avg@8 by only 0.11 ± 0.19. The median skill is reused by five problems, while problem-, solution-, and skill-level decontamination removes all flagged cross-split overlaps before retraining. These controls do not prove zero information leakage, but jointly make direct answer transfer an implausible explanation for a +4.85-point gain. The equal-compute comparison gives GRPO the entire measured SKALD budget, including the additional scoring and gate costs. Its gain from the extra 181 updates is +0.79, leaving SKALD +4.06 ahead at identical FLOPs and GPU-hours. At 4B, a scale-specific GRPO sweep raises the equal-FLOP baseline to 52.19, but the remaining SKALD margin is still +11.19. Full-solution OPSD also consumes slightly more measured compute than SKALD at 1.7B and reaches 48.51, so the result is not explained by privileged scoring alone. All three paired 1.7B seeds yield gains between +4.70 and +4.94; the seed×problem interval excludes zero without treating the 872 problems as independent training runs. Varying K from 4 to 32, using a soft gate, or refreshing it periodically/with an EMA changes avg@8 by at most 0.39. Thus finite-K noise and teacher drift are measurable—notably, 14.6% of default positive gates reverse sign—but do not account for the headline improvement in the tested configurations.

The ordered component ablation reports marginal changes under one addition order: skill abstraction +0.55, annealed tilting +0.88, and the gate +0.43. The same ordering remains positive at 4B (+1.65, +3.73, and +2.33). For internalization versus context, three 1.7B conditions are compared under an identical no-skill-at-test protocol: GRPO (45.52), GRPO with the skill in the RL context and withdrawn at test (46.60), and SKALD (50.37). This comparison contrasts contextual skill exposure with teacher-only skill distillation under the same no-skill-at-test evaluation, with SKALD improving by +3.77. The context condition remains above GRPO. The supported conclusion is therefore limited to this protocol: teacher-only skill distillation outperforms student-context exposure with later withdrawal. Removing the skill from the teacher context while keeping the same Qwen3-Base checkpoint and shared-parameter training setup reduces the 1.7B result from 50.37 to 48.01. Because the teacher and student use identical parameters, this comparison isolates the eff…

Based on this paper, here are the specific improvements to AI systems and what the improved systems can do:

1. Skill-Anchored Self-Distillation for Reasoning Models

• Improvement: Train a single model with two context views—a question-only student and a teacher conditioned on abstract skill cards (principles, usage, common mistakes). The student learns from its own rollouts scored by the teacher, transferring skill-induced advantages into shared parameters without needing the skill at inference.

• Improved capability: The AI can solve math/reasoning problems more accurately (avg@8 gains of +2.46, +4.85, +12.01 at 0.6B, 1.7B, 4B over GRPO) while requiring no extra model or privileged input at test time, making deployment simpler and cheaper.

2. Annealed Tilted Objective for Stable Distillation

• Improvement: Use an annealed exponentially tilted cross-entropy loss (τ: 0.8→0) that downweights teacher-preferred tokens with very low student likelihood during training, preventing gradient explosions and divergence. As τ anneals to zero, it recovers standard teacher cross-entropy.

• Improved capability: The AI trains stably even when the teacher and student distributions are mismatched (e.g., privileged context assigns high mass to improbable tokens). This avoids training divergence (risk-neutral runs diverged in 2/3 1.7B and 3/3 4B seeds; tilted runs all converged) and maintains low gradient norms (0.3 vs 4.5).

3. Empirical Gating for Selective Distillation

• Improvement: Activate distillation only when initial sampled rollouts show a positive teacher advantage (gate: gK(x) = 1 if teacher accuracy > student accuracy). This filters out cases where skill conditioning doesn’t help.

• Improved capability: The AI avoids harmful or useless distillation updates, focusing compute on problems where the skill provides real signal. This adds +0.43 (1.7B) and +2.33 (4B) over ungated versions, and prevents performance degradation on problems where the teacher isn’t superior.

4. Zero-Variance Group Exploitation

• Improvement: Target training signal specifically on rollout groups where all samples are uniformly correct or wrong (63–68% of groups), where group-relative RL rewards (GRPO) give zero gradient. Distillation provides dense supervision here.

• Improved capability: The AI improves precisely where standard RLVR fails. Zero-variance-only distillation recovers 84.7% of the full gain (+4.11 over GRPO), showing the system can learn from uniformly correct/wrong groups without needing reward differentiation.

5. Skill Abstraction vs. Full Solution Leakage Control

• Improvement: Use compact, abstract skill cards (not full solutions) as teacher context, with explicit filtering to remove instance-specific numerals, entities, and intermediate expressions. This reduces conditional leakage (answer recovery: 5.1% with matched skill vs 94.2% with full solution).

• Improved capability: The AI learns transferable reasoning principles rather than memorizing specific answers. Sanitized cards preserve performance (50.24 vs 50.37), and decontamination ensures generalization to held-out benchmarks without overfitting to leaked content.

6. Shared-Parameter Teacher (No Capacity Mismatch)

• Improvement: Use the same model parameters for both teacher and student branches, differing only in prompt context (with/without skill card). This eliminates external teacher dependencies and capacity gaps.

• Improved capability: The AI can be trained as a single model with no extra serving cost or architecture changes. It avoids the instability of distilling from a larger, fixed teacher and allows the teacher to co-adapt with the student during training.

7. On-Policy Distillation with Student-Generated Prefixes

• Improvement: Train the student on its own generated rollouts (on-policy), scored by the teacher, rather than on teacher-generated data. This avoids distribution shift from off-policy supervision.

• Improved capability: The AI learns to improve its own output distribution directly, making the training more sample-efficient and aligned with deployment conditions. This is critical for autoregressive generation where small errors compound.

8. Rényi-Type Cross-Entropy with Escort Target

• Improvement: Use the objective L(τ) = −(1/τ) log Σ q(v)p(v) τ, which for τ>0 creates a sharpened stationary target p* ∝ q(1/(1−τ)), providing a mode-seeking bias that stabilizes early training and gradually relaxes to teacher cross-entropy.

• Improved capability: The AI avoids being overwhelmed by teacher tokens it can’t yet produce, allowing gradual learning from hard-to-reach targets. This yields bounded logit gradients (each coordinate in [−1,1]) and prevents catastrophic updates.

9. Compute-Aware Training (Equal-FLOP Comparisons)

• Improvement: Ensure all gains are measured against equal-FLOP/equal-GPU-hour baselines, including scoring and gate construction costs.

• Improved capability: The AI system achieves +4.06 (1.7B) and +11.19 (4B) over tuned equal-FLOP GRPO, proving the method is genuinely more compute-efficient, not just better due to more training.

10. Robustness to Skill Mismatch and Noise

• Improvement: The method tolerates imperfect skill cards—shuffled skills still give +3.31 over GRPO, and replacing cards with peer cards changes performance by only 0.11 ± 0.19. The gate is robust to finite-K noise (K=4–32 changes ≤0.39).

• Improved capability: The AI system is practical in real-world settings where skill cards may be imperfectly mined or noisy, still providing substantial gains without requiring perfect skill quality.

What the improved AI system can do:

• Solve complex math/reasoning problems (MATH500, AMC23, AIME24/25, Minerva) with significantly higher accuracy at multiple model scales.

• Train stably and efficiently without divergence, even with privileged context mismatch.

• Learn transferable reasoning principles from abstract skills, generalizing to unseen problems without test-time prompts.

• Operate with a single model (no external teacher at inference), making deployment simple and cost-effective.

• Improve specifically in scenarios where standard RLVR gives no signal (uniformly correct/wrong groups), filling a critical gap in reward-based training.

• Maintain performance under noisy or imperfect skill annotations, making it robust for practical use.