What Language is This? Ask Your Tokenizer

arXiv:2602.17655 · cs.CL · Submitted 2026-02-19 · Read on arXiv

cs.CL

Submitted: 2026-02-19

Updated: 2026-09-10

Comments: In Proceedings of ICML 2026

Code: https://github.com/Ahmetcanyvz/UNILID

Project page: https://cimeister.github.io

License: http://creativecommons.org/publicdomain/zero/1.0/

The gist: Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual

Terminology

Abstract

Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fasttext, GlotLID-M, and CLD3, show that UniLID achieves competitive performance on standard benchmarks, reaches 69% accuracy with five labeled samples per language and 89% with 25, and delivers large gains on fine-grained dialect identification.

Sources

Related papers