Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation

arXiv:2602.04241 · cs.CL · Submitted 2026-02-04 · Read on arXiv

cs.CL

Submitted: 2026-02-04

Updated: 2026-02-14

Journal ref: Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)

DOI: 10.18653/v1/2026.loreslm-1.43

Code: https://github.com/farfromshallow/Uralic-language-NLP

License: http://creativecommons.org/licenses/by/4.0/

The gist: Subword tokenization critically affects Natural Language Processing (NLP) performance, yet its behavior in morphologically rich and low-resource language families remains under-explored.

Terminology

Abstract

Subword tokenization critically affects Natural Language Processing (NLP) performance, yet its behavior in morphologically rich and low-resource language families remains under-explored. This study systematically compares three subword paradigms -- Byte Pair Encoding (BPE), Overlap BPE (OBPE), and Unigram Language Model -- across six Uralic languages with varying resource availability and typological diversity. Using part-of-speech (POS) tagging as a controlled downstream task, we show that OBPE consistently achieves stronger morphological alignment and higher tagging accuracy than conventional methods, particularly within the Latin-script group. These gains arise from reduced fragmentation in open-class categories and a better balance across the frequency spectrum. Transfer efficacy further depends on the downstream tagging architecture, interacting with both training volume and genealogical proximity. Taken together, these findings highlight that morphology-sensitive tokenization is not merely a preprocessing choice but a decisive factor in enabling effective cross-lingual transfer for agglutinative, low-resource languages.

Sources

Related papers