Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR

This paper presents a rigorous, multi-seed evaluation of Automatic Speech Recognition (ASR) for Garhwali, an under-resourced Indo-Aryan language of the central Himalaya, spoken by roughly 2.5 million people. The authors build the first reproducible multi-seed ASR benchmark on the official VAANI splits, with per-seed outputs and significance testing.

Central thesis: "At the corpus sizes typical of low-resource dialects, comparing systems from single training runs is treacherous: per-utterance error variance is high, the choice of data split materially affects the reported number, and a single lucky seed can manufacture an apparent improvement that disappears on replication. The authors argue that multi-seed evaluation on fixed official splits with significance testing is necessary to separate real gains from seed noise."

Data and setup: The study uses the Garhwali subset of the VAANI corpus (5,894 utterances total) with official train/validation/test splits of 4,778/666/450 utterances; the training split comprises 8.8 hours of transcribed field speech. All systems are trained over five random seeds (42, 123, 777, 2025, 1234), with paired Wilcoxon signed-rank tests and Holm–Bonferroni correction. The primary encoder is w2v-BERT 2.0 (580M parameters, 24 Transformer layers), fine-tuned with CTC.

Key results:

1. Standard CTC baseline: Fine-tuning w2v-BERT 2.0 with standard CTC over five seeds reaches 47.0% WER (CER 17.0%), outperforming the larger MMS-1B (49.0% WER) and comparable-scale models (XLS-R at 50.2%, HuBERT at 60.9%). The authors conclude that pretraining design, not parameter count, drives low-resource dialectal performance.

2. Objective-level interventions fail: Neither Focal CTC (47.83% WER) nor a phonologically-motivated matra-weighted objective (47.42% WER) beats standard CTC under seed-level testing. The matra-weighted objective gains nothing on its targeted error category relative to standard CTC (matra error rate 22.31% vs. 22.26% for standard). The authors note that the only seed-consistent gap (standard vs. focal) favours the baseline.

3. Hindi→Garhwali transfer gives no gain: Two-stage transfer through Hindi (FLEURS) yields 47.22% WER, marginally worse than direct fine-tuning (47.02%), with no significant difference (paired Wilcoxon p = 0.81). The per-seed differences change sign—transfer helps on seeds 42 and 2025 but hurts on 123, 777, and 1234.

4. Speed augmentation is the most consistent lever: 3× speed augmentation (0.9×/1.0×/1.1×) lowers mean WER by 1.1–1.5 points under every objective, with 13 of 15 paired seeds improving. It is nonetheless the most dependable lever we test: its mean effect is direction-consistent under all three objectives.

Error analysis: The residual error profile is dominated by dependent signs and conjuncts—virama/halant (30% error rate), nasal marking (26%), and vowel categories (22–23%), while retroflex and aspirated consonants are far more reliable (10–15%). This pattern matches independent analysis by Dhasmana et al. (2026). The authors conclude the bottleneck is representational rather than procedural: "loss design and transfer can only redistribute information the model already has; for Garhwali, the limiting factor appears to be how much disambiguating evidence the acoustic representation and the available data contain."

Power analysis: The post-hoc power analysis shows that even the largest objective gap (standard vs. Focal CTC, dz = 0.98) has only 0.39 achieved power with five seeds at α = 0.05, requiring approximately 11 seeds for 80% power. The standard vs. matra-weighted gap (dz = 0.66) would need roughly 20 seeds. Crucially, even the largest gap is under one WER point in absolute terms, and where it would become significant (standard vs. Focal) the direction favours standard CTC.

Comparison with prior work: The authors compare against Dhasmana et al. (2026), who reported 49.3% WER with the same encoder on a train-internal 87/6/7 split with single runs. The two studies disagree sharply on encoder rankings (XLS-R at 65.0 vs. 50.2; HuBERT at 51.5 vs. 60.9), which are not contradictions but a direct illustration of our central point: at this corpus size, single-run dialectal ASR numbers are highly sensitive to the choice of split, early-stopping metric, and seed.

Main contributions:

• A methodological argument that single-seed comparisons are unreliable at low-resource-dialect corpus sizes

• The first reproducible, multi-seed Garhwali ASR benchmark on official VAANI splits with means, standard deviations, bootstrap intervals, and Holm-corrected significance tests

• Evidence that pretraining design, not scale, drives performance

• A characterisation of the residual error profile, corroborated by independent analysis

Conclusion: "The answer is sobering: under multi-seed testing on the official splits, the objective-level and transfer interventions we examined dissolve into seed noise, and even an objective built specifically to attack the dominant error category fails to move it. What remains robust is mundane—a strong pretrained encoder and simple speed augmentation. The authors conclude that for dialectal ASR at this corpus scale, the limiting factor is the acoustic and data evidence available to the model, not the training objective."

Improvements to AI Systems:

1. Multi-seed evaluation with statistical significance testing as a default training protocol.

• What the improved system does: Automatically trains ASR models over ≥5 fixed seeds, computes per-seed WER/CER distributions, runs paired Wilcoxon signed-rank tests with Holm–Bonferroni correction, and reports confidence intervals. It refuses to flag a model as better unless the improvement is seed-consistent and statistically significant. This eliminates false positives from lucky initializations.

2. Split-robust model selection and early stopping.

• What the improved system does: Instead of relying on a single train/validation split, it evaluates candidate checkpoints across multiple held-out splits (e.g., 5-fold cross-validation on the official train set) and selects the architecture/hyperparameters that maximize mean performance across splits, not just the best single split. This prevents overfitting to a particular data partition and yields models that generalize better to unseen dialectal speech.

3. Pretraining-design-aware model ranking.

• What the improved system does: When choosing a foundation model for a low-resource dialect, it ranks candidates by pretraining architecture and data diversity (e.g., w2v-BERT 2.0 vs. MMS-1B vs. XLS-R) rather than parameter count alone. The system automatically recommends the smallest model with the most relevant pretraining data, reducing compute and latency while improving WER by up to 3 points (as shown: 47.0% vs. 49.0% for MMS-1B).

4. Speed-augmentation as a default data-agnostic lever.

• What the improved system does: Integrates 3× speed perturbation (0.9×/1.0×/1.1×) into the training pipeline for all low-resource ASR tasks. It consistently improves WER by 1.1–1.5 points across objectives and seeds, with 13/15 paired-seed improvements. The system applies this automatically when training data < 10 hours, without requiring task-specific tuning.

5. Representational bottleneck diagnosis and targeted data collection.

• What the improved system does: After training, it runs an automated error-profile analysis (e.g., virama/halant, nasal marking, vowel categories) and identifies which linguistic features are poorly represented in the acoustic model. It then recommends targeted data augmentation or additional data collection for those specific phoneme classes (e.g., more utterances with conjuncts and dependent signs) rather than changing the loss function—since the paper shows loss design cannot overcome representational limits.

6. Seed-consistent transfer learning gating.

• What the improved system does: Before applying transfer learning from a related high-resource language (e.g., Hindi→Garhwali), it runs a multi-seed pilot to check if the transfer direction is consistently beneficial. If per-seed differences change sign (as in the paper: helps on 2 seeds, hurts on 3), the system automatically falls back to direct fine-tuning, saving compute and avoiding degradation.

7. Power-aware experimental design for research systems.

• What the improved system does: For any proposed intervention, it pre-computes the required number of seeds to achieve 80% statistical power at α=0.05, based on the expected effect size. If the effect is <1 WER point (as in the paper), it flags the experiment as underpowered and suggests either more seeds (e.g., 11 for dz=0.98, 20 for dz=0.66) or abandoning the intervention to avoid wasted compute.

8. Reproducible benchmark reporting with per-seed artifacts.

• What the improved system does: Automatically logs and publishes per-seed model weights, training curves, and test-set outputs for every experiment. This allows independent verification, meta-analysis across studies, and direct comparison with prior work (e.g., resolving the XLS-R 65.0 vs. 50.2 WER discrepancy) by re-evaluating on identical splits and seeds.