Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

Preprint: arXiv:2608.09263v1 [cs.AI], 10 Aug 2026

The paper addresses a fundamental gap in reasoning model training: Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. However, the authors argue that A token likelihood change, however, is not automatically outcome credit.

The central claim is that treating privileged likelihood scores as token credit requires passing three distinct checks:

• Q1: Does teacher preference track task success? Holding a reasoning state fixed, does the score prefer next tokens that lead to better eventual outcomes?

• Q2: Are rollouts compared under the same scoring rule? Scoring y with c(y) can measure both solution quality and how well y matches its own description.

• Q3: What behavior does the training loss reinforce? Different losses and token-weighting rules can turn the same scores into updates with different strengths and directions.

The paper formalizes the setting: "For a prompt x, let y = (y1,..., yT) ∼ πθ be a rollout, st = (x, y<t), and U(y) its terminal utility." A scorer q sees privileged context c unavailable to the deployed policy. Two common token scores are defined:

• One-sided: dc(st, v) = log q(v st, c) − log πθ(v st)

• Two-sided: d±(st, v) = log[q(v st, c+)/q(v st, c−)]

The authors emphasize: "Both are token likelihood scores: they assign a number to each candidate token v at state s. By construction they measure how the scorer's probability changes when its text context changes. Outcome credit instead requires the action value QπU(st, v): the expected final utility if the next token were set to v and generation then continued with π. Changing the scorer's context and changing the generated token are different interventions."

Proposition 1 (Exact local affine equivalence): "At an interior policy, allowing every local simplex-tangent perturbation h with 1⊤h = 0, d(s, ·) is positive-affine equivalent to QπU(s, ·) if and only if d(s, v) = α(s) + β(s)QπU(s, v) for every v, β(s) > 0."

The paper notes that Correct ranking alone is weaker. For example, Q = (0, 1, 2) and d = (0, 1, 4) rank three actions identically, but weight mixtures of those actions differently.

Proposition 2 (Forward-KL projection identity): ES,C DKL(qC ∥π) = IP⋆(V; C S) + ES DKL(q̄∥π).

The conditional mutual information term is a constant loss floor: it is not an additional expected gradient, and it contains no utility U. The authors stress: A tempting shortcut is to argue that privileged context must be useful because it changes the teacher distribution. The forward-KL projection identity shows exactly why that argument is incomplete.

Proposition 3 (Target-side independence from cross-fitting): The design implies C ⊥ Y tar X, F. Conditional on frozen C, the return-to-go likelihood-ratio estimator is unbiased for the target-rollout gradient of the declared held-out-feedback objective.

The paper clarifies: Cross-fitting therefore removes the target rollout's direct influence on the feedback used to score it; it does not preserve the meaning of own-rollout scoring while merely removing bias. A toy counterexample shows that own-rollout agreement alone cannot validate transferred action value.

Proposition 4 (Two-action frozen-target comparison): "If the student puts 1 − ε on y while fixed full-support q retains mass on ȳ, then under the displayed fixed loss normalizations, the magnitudes ∂F/∂zy for reverse KL, generalized JSD, and a bounded detached reward scale respectively as Θ(ε log(1/ε)), Θ(ε log(1/ε)), and O(ε). Forward KL instead approaches 1 − qy."

Proposition 5 (Scalar invariants do not determine the parameter gradient): The exact expected-gradient difference in (21) is generally nonzero. Positivity, coefficient sign preservation, and mean-one normalization therefore do not conserve the expected policy-gradient estimate.

A concrete counterexample: With uniform weights, two independent coordinates each receive update 1/8. Weights (2 − ϵ, ϵ) remain positive, have mean one, and preserve total scalar credit, but produce ((2 − ϵ)/8, ϵ/8).

The experiments use gpt-oss-20b on AIME 2025. "For each rollout, an external LLM judge reads the problem, reference solution, verifier result, and full reasoning trace. It writes a correction and a plausible but incorrect alternative. A frozen or earlier checkpoint of gpt-oss-20b then measures how those two texts change each rollout token's likelihood."

The authors emphasize: the external judge writes the feedback, while the base model supplies the training score; this is not a policy-as-judge experiment.

Six variants share initialization, data, optimizer, rollout batch, verifier, and evaluation prompts:

1. Outcome-only control: Uses ARi; dense coefficient identically zero

2. Entropy-gated additive: Uses 0.5 tanh(AC/0.5), multiplies it by min(1, Hi,t/0.2), gates it on nonzero ARi, prevents it from reversing the sign of ARi

3. Source-clipped additive: Clips every di,t(v) to [−2, 2] before centering, then uses 0.5 tanh(AC/0.5)

4. Entropy-gated + source-clipped: Combines both modifications

5. Full KL-regularized: Uses (12) with α = β = 1, clips di,t(v) to [−2, 2] before centering, uses no tanh

6. Projected KL correction: Uses (24) with α = β = 1 and κ = 0.25

One-sided experiment: Both outcome buckets have negative residuals at every logged update, with a smaller negative value for correct rollouts. Specifically, correct rollouts had mean residual −0.0334, incorrect −0.0502, with the gap +0.0167 positive across all 18 logged updates.

Two-sided AUC results (Table 2):

Score construction Raw pooled AUC Length-adjusted AUC

Implemented additive 0.505 [0.483, 0.524] 0.474 [0.452, 0.494]

Own-rollout feedback 0.588 [0.500, 0.673] 0.551 [0.463, 0.625]

Held-out masked feedback, mean 0.512 [0.440, 0.579] 0.485 [0.423, 0.548]

Held-out correct-fact feedback, mean 0.416 [0.347, 0.484] 0.446 [0.382, 0.513]

Identical-feedback null 0.503 [0.499, 0.507] 0.662 [0.293, 0.710]†

The paper states: Table 2 answers Q1 negatively for the implemented training signal: it does not reliably rank correct trajectories above incorrect ones, and after removing its length association it slightly favors incorrect trajectories.

"Feedback form changes what is measured. In two audited variants, positive feedback masked values and emphasized formatting, whereas negative feedback supplied explicit near misses. Their likelihood contrasts can therefore reflect wording and length as well as mathematical content."

"Removing self-dependence does not validate the score. For the held-out rows in Table 2, each trajectory uses feedback written about another rollout of the same problem, removing its direct influence on the scoring context. These constructions pass target-exclusion, polarity-reversal, and identical-feedback checks, yet their AUCs remain near chance or change direction across comparisons."

Table 3 (Paired AIME 2025 test-set comparison, avg@4 percentages):

Variant Control Variant Effect [95% CI] (pp) Exact p

Additive + entropy weighting 64.3 33.9 −30.4 [−42.9, −17.9] 1.755×10−4

Additive + bounded score 64.2 27.5 −36.7 [−49.2, −24.2] 1.240×10−5

Additive + both 64.2 30.0 −34.2 [−45.8, −23.3] 7.629×10−6

Full KL regularization 64.2 24.2 −40.0 [−52.5, −27.5] 1.907×10−6

Projected KL regularization 64.2 25.0 −39.2 [−50.8, −27.5] 9.537×10−7

Figure 2 shows that the outcome result is not confined to one checkpoint: the runs begin at similar accuracy, then separate over training.

"In an auxiliary audit, the likelihood score remained active in groups with zero outcome coefficient and concentrated on high-entropy tokens... Across the matched recipes, the additive variants moved toward higher token entropy and reference-policy KL. The KL-regularized variants kept both quantities near the outcome-only range but acquired none of its accuracy gain."

57%–71% of absolute scalar weight fell in the highest-entropy token decile in an ungated additive run.

The paper acknowledges principled cases: Likelihood ratios can be principled credit when their conditioning variables have a defined relationship to reward. Specifically:

• Hindsight Credit Assignment (HCA): compares actions conditioned on future outcomes

• Counterfactual Contribution Analysis (COCOA): conditions on reward-relevant outcome representations to avoid credit for incidental state details

For binary utility, Bayes' rule turns a coherent ratio between successful and unsuccessful action distributions into a logit transform of success probability. The identity: log[q1(vs)/q0(vs)] = logit P(U = 1 s, v) − logit P(U = 1 s).

However: Free-form texts labeled 'helpful' and 'unhelpful' need not be distributions conditioned on success and failure in this sense.

The paper concludes: "A privileged-context likelihood difference is not token value by default: treating it as credit requires checking outcome alignment, feedback assignment, and the behavior induced by the loss. The matched AIME 2025 experiments show why: the implemented score is near chance, and all five token-score recipes trail the outcome-only control."

The authors clarify: This does not imply that all self-distillation fails; successful methods can pass the three checks. The numerical results are setup-specific, while the formal distinctions apply under their stated assumptions.

• The largest matched comparison uses one seed, the AIME 2025 test set, and unequal run lengths

• In the two audited source-clipped and likelihood-magnitude weighting runs, positive/negative templates violated the matched-template requirement

• The outcome-only run was still improving when stopped and is not an empirical ceiling

• The feedback comes from an external LLM judge. The judge saw the reference, verifier result, and full trajectory before writing feedback

• The three logical questions apply broadly, while the stated propositions retain their finite-action, frozen-target, and coordinate-specific assumptions

Improvement 1: Outcome-Conditioned Credit Assignment for Reasoning Models

I will modify the training objective to replace privileged likelihood scores with explicitly outcome-conditioned value estimates. Specifically, I will train a separate value head that predicts terminal utility from partial reasoning states, then use the temporal difference error between consecutive states as token credit. This directly implements the paper's Q1 requirement by ensuring scores track expected future outcomes rather than likelihood changes. The improved system will assign credit proportional to each token's causal contribution to final correctness, preventing the collapse observed when likelihood-based scores (AUC ≈ 0.50) fail to distinguish correct from incorrect trajectories.

Improvement 2: Cross-Fitted Feedback with Explicit Reward Decomposition

I will restructure the feedback generation pipeline to separate mathematical correctness from stylistic/length attributes. Instead of free-form LLM feedback, I will use a structured verifier that outputs: (a) a binary correctness label, (b) a list of specific reasoning errors, and (c) a normalized length control. The training score will be computed as log P(correct state, token) − log P(incorrect state, token) using Bayes' rule, as suggested in the paper's positive case. This addresses Q2 by ensuring feedback measures task success rather than incidental text properties. The improved system will maintain high accuracy even when feedback templates vary, avoiding the 30–40 percentage point drops seen in Table 3.

Improvement 3: Gradient-Sign-Conserving Token Weighting

I will implement a token weighting scheme that preserves the sign and relative magnitude of the outcome-based advantage estimates while preventing entropy exploitation. Specifically, I will clip advantage values to [−1, 1], apply a softmax temperature that decreases with training step, and add a small penalty proportional to the KL divergence from the reference policy only when the advantage is positive. This directly addresses Q3A by avoiding the Θ(ε log(1/ε)) gradient explosion seen with reverse KL and the O(ε) vanishing gradients of bounded rewards. The improved system will maintain stable training dynamics, preventing the entropy increase and reference-policy drift observed in the additive variants.

Improvement 4: Entropy-Balanced Exploration with Outcome-Gated Updates

I will modify the exploration strategy to only update tokens when the terminal outcome is correct AND the token's entropy is below a threshold that decreases over training. This combines the paper's finding that 57–71% of weight falls on high-entropy tokens with the need to avoid rewarding random guesses. The gating function will be: update weight = I(outcome = correct) × I(entropy < τ t) × advantage, where τ t decays from 0.5 to 0.1 over training. The improved system will focus learning on decisive reasoning steps rather than exploratory noise, potentially recovering the 24–40 point gap between the outcome-only control and token-score variants.

Improvement 5: Two-Stage Training with Verifier-Grounded Distillation

I will split training into two phases: (1) outcome-only reinforcement learning until the policy achieves 60%+ accuracy on the validation set, then (2) token-level distillation using only the verifier's binary correctness signal, not likelihood ratios. In phase 2, I will use the HCA-style credit assignment: for each token, compute the expected utility of continuing from that state versus the expected utility of the actual rollout, using a frozen value network trained in phase 1. This directly implements the paper's positive case (COCOA/HCA) and avoids the Q1 failure mode. The improved system will combine the stability of outcome-only training with the sample efficiency of token-level credit, potentially exceeding the 64.3% control accuracy.

Improvement 6: Adversarial Feedback Validation Before Training

I will add a pre-training validation step that checks whether the proposed token scores pass the paper's three checks: (a) compute AUC of score vs. outcome on a held-out set—reject if < 0.55; (b) verify that positive and negative feedback templates differ only in mathematical content, not length or formatting; (c) simulate one gradient step and confirm it increases the probability of correct tokens more than incorrect tokens. This prevents deploying a broken credit assignment scheme (as in the paper's experiments) and saves compute. The improved system will automatically reject any training signal that fails these checks, avoiding the 30–40 point performance drops entirely.

Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2%, versus 24.2%--33.9% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.