When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

A study was conducted to audit whether an authenticity asymmetry, created when synthetic data uses real corpus text as ground truth but generates distractors, translates into measurable shortcut reliance by trained reinforcement learning policies. This research establishes a causal protocol to determine if a detectable statistical signal in the data is actually exploited by the model rather than simply being present.

How it works

The researchers audited GooseReason-0.7M, a corpus of 700,000 items spanning mathematics, code, and general STEM reasoning that was constructed by masking critical spans in real documents and prompting a language model to invent plausible wrong answers. The core asymmetry being investigated is that the correct option is real text [lifted from a real document]. Every distractor was written by a model.

The audit proceeds in two distinct stages to separate the detection of an artifact from its exploitation. Stage 1 focuses on characterizing the presence of the asymmetry using surface statistics alone, without access to meaning. This involves computing five features for every option—"rare-token rate (share of tokens below a Zipf-frequency threshold), word- and sentence-length burstiness (variance-to-mean ratio of per-token and persentence character length), character-level Shannon entropy, and POS-tag KL-divergence against a fixed reference distribution." Two classifiers are trained on these features to predict provenance from the data.

How it works

Stage 2 is a causal intervention designed to test whether the detected asymmetry is exploited. To do this, researchers build a paraphrase-matched control corpus in which every option—gold and distractor alike—passes through the same transformation. This transformation is domain-conditioned: for math and STEM, they apply full semantic paraphrase, whereas for code, they use a restyle-only transform to preserve control flow while altering stylistic cues.

Two policies are then trained under identical settings: a treatment policy on original data (asymmetry present) and a control policy on the paraphrasematched version (asymmetry removed). The evaluation protocol defines the artifact exploitation gap as gap = acc(neutralized) − acc(adversarial), where the adversarial test substitutes an authentic-looking distractor back into the neutralized set. A larger gap in the treatment arm is considered the signature of the hypothesis, indicating that a policy leans harder on authenticity-correlated cues when trained on unmodified data.

How it works

The evaluation protocol involves testing three variants for both policies: "Original keeps the data unmodified, asymmetry present; Neutralized uses the paraphrasematched version, asymmetry removed; Adversarial pairs the neutralized gold option with one unmodified, authenticity-bearing distractor substituted back in. The reward used is rulebased and fully verifiable—an exact match between the emitted option letter and the correct one—so no learned verifier’s biases leak into the causal comparison."

The results show that the exploitation gap does not favour the unmodified-data arm: 0.021 against 0.027 for the control, indicating that a detectable artifact went unexploited under their budget. In other words, a real if weak signal did not translate into measurably greater shortcut reliance at our budget.

How it works

The findings reveal a crucial dissociation between detection and exploitation. Stage 1 successfully characterizes the presence of an asymmetry—a domain-specific, statistically real authenticity signal—but this signal does not show up as a measurable difference in shortcut reliance in Stage 2. This mirrors prior findings regarding verifier error, suggesting that an authenticity asymmetry in the data need not be an exploited one.

The audit revealed a domain-specific construction finding: Code breaks the pattern at 0.416—below chance—which manual audit attributes to a different generation mechanism entirely: its distractors are minimal mutations of the gold code, not independent rewrites. This suggests that a mechanism that varies by domain can produce a qualitatively different failure surface—bug-injection mutation versus free rewriting—that no single aggregate statistic will reveal. The researchers suggest that future audits should pair cheap characterization with a causal intervention before concluding a detected asymmetry needs engineering around.

How it works

The overall conclusion is that an authenticity asymmetry is present in the data, weak and unevenly spread, but under their budget, it did not translate into a larger exploitation gap. The authors conclude that dissociation, along with the domain-specific construction finding, is worth knowing for anyone curating corpora of this kind. They release the audit protocol to allow other researchers to point at other synthetic RLVR corpora and repeat the audit at larger budgets.

The gist

A surface-only classifier finds a domain-specific, statistically real authenticity signal in Stage 1; that signal does not show up as a measurable difference in shortcut reliance in Stage 2.

Here are specific improvements that can be made to AI systems based on the findings of this research:

1. Incorporate a Provenance Auditing Layer into data pipelines that synthesize RLVR training data (e.g., those using masked-span prompting). This layer should utilize a set of cheap, semantics-blind surface statistics (rare-token rate, burstiness, character entropy, POS-tag KL-divergence) to flag potential authenticity asymmetry in the generated distractors versus the gold answer.

2. Develop a Causal Intervention Protocol for evaluating synthetic data quality. When an authenticity asymmetry is detected in Stage 1, systems should be capable of automatically constructing a paraphrase-matched control corpus (using domain-specific paraphrasing for math/STEM and restyle-only transformations for code) to neutralize the provenance signal while preserving the underlying reasoning difficulty.

3. Implement a Dissociation Check during policy training evaluation. Instead of only measuring downstream accuracy, systems should be designed to test whether an exploitation gap (the difference in reliance on authenticity cues) persists when comparing policies trained on unmodified data versus those trained on neutralized data, holding training budget and sample size constant.

4. Improve synthetic data generation reporting by requiring pipelines to report the specific generation mechanism used per domain (e.g., bug-injection mutation vs. free rewriting). This allows downstream researchers to understand if a qualitative difference in failure surface is being inadvertently introduced by the synthesis process, which single aggregate statistics will miss.

5. For code-related reasoning tasks, specifically monitor for and flag one-token mutations of the gold answer as a distinct generation mechanism. This specificity helps distinguish between high-effort rewrites and low-effort structural modifications, which may be exploited differently by models (as seen in the code domain finding).

6. When deploying RLVR policies trained on synthetic data, use the Exploitation Gap metrics to predict shortcut learning potential. If the gap is near zero or negative (as observed in many domains), it suggests that detectable authenticity asymmetries are not being leveraged by the policy for reasoning shortcuts, allowing for a higher confidence assessment of the policy's genuine reasoning capability rather than artifact exploitation.