Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions, including whether multilingual acquisition leads to delays in learning and whether there are better and worse ways to structure multilingual input. Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We create matched mono- and bilingual datasets using synthetic data and machine translation to simulate a variety of controlled exposure conditions, pairing English with Spanish, Chinese, and Dutch input. We train GPT-2 models on these datasets and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales, measures, and languages, bilingual models perform similarly to monolingual models in the first language, but show strong performance in the second as well. Control experiments show that our findings hold on real-world language data and even on shuffled second-language input. These results suggest that there are no strong differences between different bilingual exposure regimes, and bilingual input poses no in-principle challenges for agnostic statistical learners.