Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
cs.CL, cs.AI
Submitted: 2026-08-15
Updated: 2026-09-09
Comments: Accepted to EMNLP 2026 (Main Conference). 22 pages, 12 figures, 7 tables
Code: https://github.com/kleines-gespenst/word-order-preferences
License: http://creativecommons.org/licenses/by/4.0/
The gist: We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages.
Terminology
Abstract
We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for right-branching subject-verb-object (SVO) languages emerges while SOV falls behind despite being the most frequent order cross-linguistically. This SVO advantage extends to multilingual models and correlates with language resource level and data quality rather than word order. Thus, the same architecture exhibits opposite preferences on artificial and natural languages, establishing that word order biases observed in practice are data-driven. Since highly-resourced languages are overwhelmingly SVO, these biases risk gradually reducing word order diversity, particularly in languages that productively use multiple word orders, with the widespread adoption of LLMs.
Sources
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
- How LLMs Distort Our Written Language
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
- The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Few-shot Learning with Multilingual Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering