Why Pretraining Fails to Share Cross-Lingual Knowledge

arXiv:2609.19291 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-16 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-16

Updated: 2026-09-16

Code: https://github.com/AdamJaber03/torchtitan-mulitlingual

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages.

Terminology

Abstract

Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency --- 14 times the baseline.

Sources

Related papers