Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

arXiv:2606.18717 · cs.CL, cs.AI · Submitted 2026-06-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish".

Jane: Turkish is an agglutinative language where meaning resides in morphemes, and current subword tokenizers fail to capture this morphology effectively.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish, the authors really argue that unifying tokenization and representation into one model is the way forward for morphologically rich languages like ours. It’s a single neural morphemeboundary model that does exactly that.

Jane: They essentially suggest that because Turkish meaning resides in its morphemes, a tokenizer should be inherently aware of those boundaries, and Morpheus achieves this by using a differentiable program to learn soft morpheme memberships during training. This allows it to produce exact segments at inference without any lossy steps.

Lu: The implication here for the wider research community is that we might see models that intrinsically understand linguistic structure rather than just learning statistical patterns on top of pre-defined structures. This moves the field toward a more structurally informed AI approach for language processing.

Meng: From an engineering perspective, having a model that handles both roles at once, while keeping memory usage down with about nineteen percent less GPU memory compared to some alternatives, makes it very attractive for building robust Turkish NLP pipelines where faithful decoding is critical.

Lalam: I think the cultural impact comes from creating systems that can interpret and generate Turkish with a much deeper fidelity to its actual structure, which could improve how we process and analyze cultural texts and historical documents written in the language.

Tom: That’s a big picture thought, Lalam; Morpheus isn't just about better accuracy on benchmarks like TR-MMLU; it’s about building a fundamentally different kind of AI component for Turkish NLU. It achieves the lowest bits-per-character at one point four two five and maintains a very high frequency-weighted purity of eighty-three point five percent.

Jane: And that high purity, combined with its ability to yield exact segments, really sets it apart from systems that rely on normalization or fixed dictionaries which inherently discard surface information during processing.

Lu: The authors are setting a new standard by showing how this single architecture can simultaneously produce high-quality word embeddings and perfectly invertible tokenization for this specific language type. It shows the power of coupling these two tasks tightly within one neural framework.

Meng: So, the main thing is that for applications demanding precision in Turkish, like sequence labeling or root identification, Morpheus looks like a much more informed default choice because it prioritizes morphological alignment directly in its objective function.

Tom: Absolutely; Morpheus positions itself as the better informed default for Turkish NLU and sequence-labeling where faithful decoding and morphology are paramount. It’s not just another incremental tweak; it’s a new way to think about how we model this language.

Conclusion: Segment: Conclusion**

Tom: So, to wrap things up on Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish, we've seen how this single model tackles the hard problem of handling Turkish morphology by building both a tokenizer and an embedding producer simultaneously.

Jane: That’s right, Tom; the core idea is that you don't need separate tools for segmenting words and representing their meaning when those meanings are deeply tied to their internal structure.

Lu: I think the authors really nailed the mechanism where they use that dynamic program to turn character probabilities into soft morpheme memberships, which gives us a very clean way to map that complex language structure onto a neural network.

Meng: From an engineering standpoint, it's impressive how they managed to keep the model relatively lean while still achieving this high degree of morphological accuracy during inference.

Lalam: For me, the real implication is how much more accurately we can process and generate Turkish texts because the system isn't just guessing word boundaries; it’s respecting the actual linguistic rules embedded in those morphemes.

Tom: Exactly, Lalam; when you think about that level of fidelity in sequence labeling or root identification, it opens up new possibilities for applications where nuance matters a lot.

Jane: It really simplifies the pipeline because you get a perfectly decoded output directly from the input without any messy normalization steps that usually introduce errors.

Lu: The vision here is incredibly exciting; imagine applying this approach to other highly agglutinative languages where current tokenizers just fail spectacularly at capturing meaning.

Meng: I'm wondering if the fertility trade-off, where it might produce a bit more output per input character than some other methods, actually makes it more practical for large-scale production systems.

Lalam: That’s a fair point, Meng; if we can get better structural understanding at that level with less computational overhead overall, it could mean deploying much smarter language tools everywhere.

Tom: So Morpheus isn't just another model; it’s a new blueprint for building language processing systems that are fundamentally aware of the grammar underneath the surface text.

Jane: It shows that combining deep structural awareness directly into the tokenization layer can lead to a representation quality that standard methods just can't reach.

Lu: And this architecture, which couples the segment structure with the embedding generation so tightly, is really pushing how we design multimodal language systems in general.

Meng: I’m still curious about how they plan to scale this specific dynamic program for much larger and more complex languages moving forward.

Lalam: That future work will be crucial because if we can refine this structure learning, it could unlock a level of linguistic understanding that helps us preserve and analyze cultural heritage in ways we haven't thought possible.

S¸akar, Tolga

cs.CL, cs.AI

Submitted: 2026-06-17

Updated: 2026-10-02

Code: https://github.com/lonewolf-rd/TurkishMorpheus

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Turkish is an agglutinative language where meaning resides in morphemes, and current subword tokenizers fail to capture this morphology effectively.

Key concepts

Differentiable Poisson–binomial dynamic program: This mathematical tool is used during training to convert simple character boundary probabilities into soft morpheme memberships. It allows the model to learn where one morpheme ends and another begins by optimizing a loss function, ensuring the resulting segmentation is accurate at inference.
MorphScore macro-F1: This metric evaluates how well Morpheus aligns its tokenization with actual Turkish morphology. A high score indicates that the segments produced by the tokenizer correspond correctly to meaningful morphemes in the language, proving its morphological awareness.
Word Embedding Generation Coupling: Morpheus generates word embeddings by pooling character vectors based on their learned segment membership probabilities. This design ensures that the structure used for tokenization directly informs the final embedding, creating a unified representation where morpheme structure is explicitly captured in the vector.

Terminology

Decode — 26× in this paper · explained in 2 episodes
Tokenization — 24× in this paper · explained in 2 episodes
Alignment — 16× in this paper · explained in 1 episode
Embeddings — 10× in this paper · explained in 2 episodes
AUC — 8× in this paper · explained in 2 episodes
Verification — 8× in this paper · explained in 2 episodes
Fidelity — 7× in this paper · explained in 2 episodes
Analyzer — 5× in this paper · explained in 1 episode
Accuracy — 4× in this paper · explained in 1 episode
Benchmark — 3× in this paper · explained in 2 episodes
Distillation — 3× in this paper · explained in 2 episodes
Absolute Indices — 2× in this paper · explained in 1 episode
RAG — 2× in this paper · explained in 3 episodes
Transformer — 2× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Contrastive Learning — 1× in this paper · explained in 5 episodes
Core — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Knowledge Distillation — 1× in this paper · explained in 11 episodes
Linear probe — 1× in this paper · explained in 2 episodes
Objective Function — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Trace — 1× in this paper · explained in 2 episodes

Summary

Turkish is an agglutinative language where meaning resides in morphemes, and current subword tokenizers fail to capture this morphology effectively. Morpheus introduces a single neural model that simultaneously functions as a lossless, morphology-aware tokenizer and a structured word-embedding producer for Turkish.

How it works

Morpheus utilizes a differentiable Poisson–binomial dynamic program to turn per-character boundary probabilities into soft morpheme memberships during training, which then yields exact segments at inference. This mechanism ensures that decode(encode(w)) = w holds by construction because no string normalization is applied; the emitted pieces are the surface form. The model is trained using a weighted sum of four loss terms: auxiliary boundary BCE against Morfessor labels, skip-gram negative sampling (SGNS), InfoNCE contrastive loss on root identity, and a vocabulary-free character-level reconstruction (MLLM).

Tokenization Mechanism

The architecture involves three stages connected by a differentiable segmentation operator. The character encoder uses multi-scale convolutions and self-attention layers with Rotary Position Embedding (RoPE) to capture local n-grams while injecting relative offsets into the attention dot product, allowing the model to reason about offsets between characters rather than absolute indices. This feeds into a boundary detector that emits inter-character boundary probabilities, which are then processed by the Poisson–binomial dynamic program to generate a soft segmentmembership matrix, M[j, k].

Word Embedding Generation

Each segment is summarized via attentionpooling the character vectors weighted by their membership probability, sk = Pj αjk h k j. The final word embedding (ew) is derived as the mean of these valid segment vectors followed by a two-layer feedforward network with residual LayerNorm. This coupling ensures that the morpheme structure that defines the tokenization is exactly the structure pooled in the embedding, making Morpheus a tokenizer and an embedder at once.

Evaluation and Performance

Morpheus is evaluated across several dimensions, including reversibility, morphological alignment (MorphScore macro-F1), and language modeling efficiency (bits-per-character, BPC). Among reversible tokenizers, Morpheus attains the lowest bits-per-character (1.425) and the highest frequency-weighted purity (83.5%) on TR-MMLU. As an embedder, it leads on lexical retrieval (root-family MAP 0.85) and same-root verification (ROC-AUC 1.00), surpassing contextual encoders like BERTurk and BGE-M3 on root identity tasks, while trailing them on context-dependent tasks like NER and case/number probing.

Trade-offs

The primary trade-off involves fertility versus quality; Morpheus exhibits a higher fertility (∼1.73 vs. ∼1.5 tokens/word), which lowers raw character throughput but yields the lowest BPC among reversible tokenizers and no drop from len to exact in surface fidelity testing, confirming its lossless decoding capability, unlike rule-based systems which suffer from lossy canonicalization. The embedding's strength is rooted in its design: the contrastive objective concentrates root inflections, leading to superior lexical performance but causing it to trail contextual encoders on number/case probing and NER. Morpheus is positioned as the better informed default for Turkish NLU and sequence-labeling where faithful decoding and morphology are paramount.

The gist: Morpheus introduces a single neural model that simultaneously functions as a lossless, morphology-aware tokenizer and a structured word-embedding producer for Turkish. It attains the lowest bits-per-character (1.425), the highest frequency-weighted purity (83.5%), the strongest morphological alignment (MorphScore macro-F1 0.61), 100% reversibility, and ∼19% lower GPU memory while leading on lexical retrieval and same-root verification tasks. It is positioned as the better informed default for Turkish NLU and sequence-labeling where faithful decoding and morphology are paramount.

Improvements for AI systems

Here are the specific improvements that Morpheus offers to AI systems, based on the provided research:

The ability to create a single, unified model that is simultaneously a lossless, morphology-aware tokenizer and a word embedding producer.
For generative language models (LLMs), this system ensures perfect fidelity: decoding the output back to the original text is guaranteed because no string normalization is applied (i.e., it prevents corrupted outputs like those seen in WordPiece or rule-based tokenizers).
For sequence-labeling tasks, such as morphological segmentation and analysis, Morpheus can be used to generate high-quality, morphology-aligned labels (MorphScore macroF1 of 0.61) that are superior to standard frequency-driven subword methods.
For lexical retrieval systems (like RAG), Morpheus provides a root-centric embedding that leads on tasks requiring root matching and same-root verification (MAP 0.85, ROC-AUC 1.00), allowing for much more precise keyword indexing and deduplication than general contextual embeddings like BGE-M3 or BERTurk.
For memory-constrained inference, Morpheus achieves a low bits-per-character (1.425) and uses significantly less GPU memory (19% less) compared to 64K vocabulary subword tokenizers, making it ideal for deployment on devices with limited resources while maintaining high quality.
It enables the creation of specialized, morphology-aware lexical encoders that complement heavy contextual models; these encoders excel at identifying the root and are suitable for building a two-tier retrieval system (lexical index + dense semantic index).

Abstract

Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so decode(encode(w)) = w holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character (1.425), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 0.61 vs. about 0.32), and uses about 19% less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP 0.85) and same-root verification (ROC-AUC 1.00), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.