Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery

Synthetic data generation for low-resource languages like Korean presents a critical challenge because existing corpora are often unavailable for redistribution, making it difficult to measure how much real-data gain synthetic supervision can recover. This research investigates this recovery by analyzing how the coupling between the metric used to score the synthetic output and the construction access within the synthesis pipeline inflates these estimates.

The gist

Synthetic supervision appears to surpass the data it was meant to substitute for when evaluation resources are independent of the generation pipeline, but this recovery figure is strongly dependent on whether a marker-based metric's scoring inventory is contained within the transformation rules.

Key Contributions and Findings

1. KoDialectBench: The authors contribute KoDialectBench, a 1,000-item Korean dialect benchmark released under a license-compatible reconstruction scheme using identifier hashes and scoring codes so users can reconstruct items from their own licensed copies.

2. Recovery Axis Dependency: Recovery is strongly axis-dependent. For instance, the best synthetic arm reaches 91.2% of the real-data gain on region identification but 63.7% on comprehension. On generation, the answer depends on whether the evaluation resource is independent of the generation pipeline.

3. Metric–Construction Coupling Bias: The central finding is that recovery differs sharply by axis, and on the generation axis the answer depends on whether the evaluation resource is independent of the generation pipeline. When it is not independent, synthetic supervision appears to surpass the data it was meant to substitute for.

4. Marker Metrics Confound with Construction Access: The study demonstrates that a lexicon-based dialect metric becomes confounded with construction access when its scoring inventory is contained in the synthesis inventory, suggesting that Shared construction and evaluation inventories can therefore substantially inflate estimates of synthetic-data recovery.

5. Construction-Time Intervention: A crucial intervention involves splitting the marker inventory before synthesis: 80% by stratified sample (region, frequency quartile, purity quartile) drives the transformation rules, and the held-out 20% is used only for scoring. This intervention results in a 76–84 point reduction on the marker metrics without a corresponding decrease on the three pipeline-independent measurements.

Experimental Setup and Methodology

The experiments utilized a base model (Qwen3.8-27B, bf16) and identical LoRA settings across all arms. The primary comparison involves arm R (real-data reference), arm C and SK5/SK4 (synthetic arms), the exactly size-matched construction-disjoint intervention SK12-D9b, and an independently sampled disjoint arm SK11-D9. All training sets were mixed to ensure uniform sampling across regions for T2.

Analysis of Coupling Mechanisms

The researchers tested several mechanisms to isolate the source of recovery differences:

(i) Disjointness Test:

The study performed a disjointness test where the transformation rules were restricted to forms emitted by the synthesis pipeline, and the held-out inventory was scored. The result showed that No scoring marker survives removal of the forms the transformation rules can emit, indicating that while lexical disjointness might be absent, evaluation triggers can still occur through suffix matches.

(ii) Coverage Statistic (MCC):

The authors defined metric–construction coverage as MCC = Veval ∩ Vconstruct and found that measured dialectness recovery increasing monotonically as overlap rises from MCC = 0 to MCC = 1. The coupling is quantified by sweeping MCC, showing that at a full coverage (MCC=1.00), measured recovery reaches 84.8%.

(iii) Construction-Time Intervention:

The intervention was designed to test the effect of construction access directly by withholding 20% of marker types from the transformation rules. This led to a collapse in marker-based recoveries, falling from 91.8% to 8.1% for dialectness and 101.9% to 25.6% for region match on the held-out inventory, while pipeline-independent metrics remained unaffected, moving by +3.3, +2.0 and +2.3 recovery points.

Conclusion on Recovery Structure

The research concludes that Marker-based recovery is dominated by construction-inventory coverage and cannot by itself support claims about broader dialect capability. The coupling mechanism explains the observed inflation: the metric's scoring inventory is often a subset of what the transformer can produce, meaning the metric cannot distinguish producing genuine dialect from producing the forms our rules happen to know. The paper also notes that Meaning preservation rises as transformation decreases, suggesting that restricting transformation rules can improve scores on certain metrics, though this is treated as a guardrail rather than a gain axis.

As a fastidious and diligent researcher, I have analyzed this paper to extract actionable insights for improving AI systems, particularly those operating in low-resource language domains like Korean dialect NLP.

Here are the specific improvements an AI system can undergo based on the findings:

1. [][] Improve synthetic data generation fidelity by explicitly managing the overlap between linguistic features and evaluation metrics.

2. [][] Implement a Construction-Time Intervention layer to decouple feature extraction from scoring, preventing metric inflation due to shared training inventory.

3. [][] Develop a Metric-Construction Coverage (MCC) monitoring system that dynamically adjusts the confidence in synthetic recovery estimates based on the overlap between the model's output space and the evaluation inventory.

4. [][] Design robust evaluation protocols that use disjoint or construction-disjoint arms to establish true baseline performance, isolating capability gains from metric alignment artifacts.

5. [][] Integrate a mechanism to detect and mitigate Goodhart-style proxy failures, where a metric (like a marker lexicon) ceases to track the intended construct (dialect capability) once the data construction process has privileged access to those features.

Here is what the improved AI system can do:

1. [][] The system will provide more reliable and trustworthy recovery estimates for synthetic data by explicitly accounting for how well its generated outputs align with the specific markers used in evaluation, preventing overestimation when training and evaluation inventories overlap.

2. [][] The system will be able to generate synthetic Korean dialect text that maintains high fidelity across multiple, independent evaluation metrics (e.g., dialectness and region match), rather than showing inflated scores on a single metric due to construction access bias.

3. [][] The system can distinguish between genuine linguistic capability improvement (e.g., increased understanding of regional vocabulary) and superficial performance gains resulting from the synthetic pipeline's access to the exact features used for scoring, leading to more accurate assessments of true dialectal generalization.

4. [][] When deployed in a low-resource setting, the system can use construction-disjoint synthesis pipelines to generate data that is guaranteed not to rely on specific patterns learned during training for the sake of satisfying a metric, resulting in synthetic data that generalizes better to unseen linguistic forms.

5. [][] The system's evaluation framework will be more resilient to data contamination and annotation artifacts by utilizing evaluation sets whose feature inventories are demonstrably disjoint from the training process, ensuring that reported performance metrics reflect true model capability rather than dataset leakage.

Korean dialect corpora are available but not redistributable: weights may be released, while reproducing training and evaluation from the underlying data cannot be. We ask how much of that supervision synthetic data recovers, and whether that recovery can be measured independently of the synthesis pipeline. We contribute KoDialectBench, 1,000 items across five regions on three axes, released as identifier hashes and scoring code so users reconstruct the items from their own licensed copy. Recovery is strongly axis-dependent: our best synthetic arm reaches 91.2% of the real-data gain on region identification but 63.7% on comprehension. On generation the answer depends on the metric: the deployed marker lexicon reports 92.3% on dialectness and 119.3% on region match, the latter exceeding the real-data reference, whereas reference-based generation reaches 72.4%. We find the marker metrics' scoring inventory is entirely contained in the inventory our transformation rules can emit. We test the effect of construction access directly with an exact-form construction-disjoint arm that withholds 20% of marker types from the rules. At exactly matched training size (8,600 examples) it reduces dialectness recovery from 91.8% to 8.1% and region-match recovery from 101.9% to 25.6% on the held-out marker inventory, while the three pipeline-independent measurements do not fall at all. A complementary evaluator sweep defines metric-construction coverage (MCC) and finds measured dialectness recovery increasing monotonically as overlap rises from MCC=0 to MCC=1. Shared construction and evaluation inventories can therefore substantially inflate estimates of synthetic-data recovery.