Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

CULTIVAR is a source-contrastive, locale-oriented translation benchmark designed to address limitations in existing multilingual test suites. The paper argues that current benchmarks, typically sourced in English and translated into other languages, are prone to data contamination and overlook locale and cultural considerations. Cultivar is instantiated as a localised subset of FLORES, enabling locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancies allow probing of data contamination and localisation robustness.

The benchmark is constructed through a three-stage pipeline: 1) named entity-rich instance selection, 2) LLM-assisted paraphrasing, and 3) human post-editing. The pipeline identifies 658 sentences with named entities from the FLORES dev set, selects 200 with at least 2 named entities, and uses GPT-5.5 to generate localised source-English pairs. Human annotators, selected for proficiency in both source and target languages and familiarity with the location, review and edit the LLM paraphrases. The final benchmark consists of 27 language-script-location combinations, with 21 unique languages, 8 scripts, and 21 countries. On average, annotators accept 70% of LLM-generated localisations without modification, post-edit roughly 28%, and revert fewer than 2% to original FLORES entries.

The paper evaluates 32 open-weight models across three groups: pre-trained MT models (NLLB, MADLAD), LLM-based translation models (Seed-X-PPO-7B, Hy-MT2), and general LLMs (Qwen3.5, Llama, Gemma, etc.). Evaluation uses BLEU and chrF metrics, with a primary focus on performance discrepancy (∆BLEU and ∆chrF) between localised and original tests. Key findings include:

1. Ranking consistency: Cultivar and FLORES induce highly consistent model rankings, with Spearman's rs ≥ 0.93 and Kendall's τb ≥ 0.80 for all translation directions.

2. Performance on localised content: Most models show ∆BLEU between −2 and +2 and ∆chrF between −2 and +0.7, with averages of −0.49 and −1.1 respectively. MT-specialised models (NLLB, MADLAD, Hy-MT2, Seed-X-PPO-7B) rank lower by ∆ scores, consistently clustered in the negative region. Model size scaling shows ∆ scores increase with model size for 6 out of 8 families, suggesting FLORES may underestimate true capabilities of stronger models.

3. Potential FLORES contamination: aya-expanse-8b and Seed-X-PPO-7B show strong performance on both tests but exhibit much larger performance declines (∆BLEU < −8) than other models (−2 < ∆BLEU < +2). However, false positive rates for verbatim FLORES word recall remain below 1% for most models, providing no direct evidence of exact text memorisation.

4. Locale difficulty variation: Models perform better on localised plt Latn Madagascar and gom Deva India, while cmn Hans Singapore receives significantly low ∆ scores. The difficulty of test sets can change after localisation.

5. US-centric bias: For Hindi and Chinese localised to different regions, models translate US content better than content from other locales. Positive ∆ for US localisations implies these are easier for models, even when the source language is not de facto in the US. This reflects a US-centric bias in training data.

6. Manual error analysis: A Chinese native speaker inspected Chinese-to-English outputs from 10 selected models. Named entity errors account for >80% of locale-specific errors and >50% of all errors. Pinyin transcription errors are exclusive to the China split, making up 20% of errors there. Number and unit errors show regional variation, with UK and China splits containing 22 and 14 such errors respectively compared to only 5 in the US split. Error counts decrease with model size, and Seed-X-PPO-7B shows a mismatch between error counts and BLEU, hinting at possible FLORES-like data contamination.

The paper concludes that Cultivar demonstrates true multilingual proficiency requires locale-aware evaluation to accurately reflect real-world use cases. The benchmark serves as a diagnostic tool for data contamination and reveals pervasive US-centric bias in translation models.

Based on the paper, here are specific improvements for AI systems:

1. Locale-Aware Translation Training

• Train models on parallel corpora that include multiple regional variants of the same source language (e.g., Hindi for India vs. Mauritius, Chinese for Singapore vs. China) rather than assuming one standard form.

• Add explicit locale tags (e.g., ``) to training data so models learn to condition output on geographic context.

2. Contamination-Resistant Evaluation

• Integrate Cultivar-style localised test sets into the standard evaluation pipeline to detect overfitting to FLORES or similar benchmarks.

• Add a localisation robustness score (∆BLEU/∆chrF) as a mandatory metric alongside BLEU, so model selection penalizes drops on culturally adapted inputs.

3. Named Entity Handling

• Improve named entity translation by training on locale-specific entity lexicons (e.g., place names, personal names, units) and using retrieval-augmented generation to fetch correct local variants.

• Add a post-processing step that checks for pinyin transliteration errors (for Chinese) and locale-specific number/unit formats (e.g., UK vs. US vs. China).

4. US-Centric Bias Mitigation

• Rebalance training data to include more non-US cultural references, even for languages not de facto spoken in the US (e.g., Hindi for Mauritius, Chinese for Singapore).

• Use adversarial debiasing: penalize models that perform better on US-localised inputs than on other locales for the same source language.

5. Model Scaling with Locale Sensitivity

• For larger models, explicitly verify that performance gains on standard benchmarks translate to localised content; if not, add locale-specific fine-tuning stages during scaling.

6. Human-in-the-Loop Post-Editing

• Deploy a hybrid pipeline where LLM-generated translations are automatically flagged for named entity or unit errors, then routed to human annotators familiar with the target locale for correction, reducing manual effort.

The improved AI system can:

• Translate a single source sentence into multiple locale-specific variants (e.g., How much is this? → UK English vs. US English vs. Singapore English) with correct units, idioms, and named entities.

• Detect and report its own contamination risk by comparing performance on localised vs. unlocalised test sets.

• Automatically adapt to a user's geographic context (e.g., via IP or explicit setting) to produce culturally appropriate output.

• Provide confidence scores that reflect locale-specific difficulty, flagging low-confidence outputs for human review.

• Reduce US-centric bias by achieving similar translation quality for non-US locales, even when the source language is not native to the US.