Inside the LLM Word Factory

arXiv:2606.08562 · cs.CL · Submitted 2026-06-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Inside the LLM Word Factory".

Jane: Detokenization is characterized as an early-layer two-stage mechanism where attention writes a token-specific directional signal from preceding subwords, and the MLP composes it with the local embedding.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're diving into "Inside the LLM Word Factory" today; this paper is looking at how large language models actually turn subword pieces into full words, and it seems like they've found a really specific two-stage process. It claims that this detokenization happens in early to middle layers, but they want to pinpoint exactly what's going on.

Jane: That makes sense, Tom; so the main idea is that models use subwords for input but need to build word-level meaning, and this paper is trying to map out that reconciliation process through activation patching. It suggests a two-stage mechanism where attention and the MLP do different jobs.

Lu: What's really interesting about this paper is how they isolate these components using those paired experiments; they're not just looking at the whole system but trying to see which part handles what, like how attention transmits a specific signal versus how the MLP combines it with local embedding.

Meng: From an engineering standpoint, isolating these parts is crucial because if we don't know where the bottleneck or the primary driver is, it's hard to optimize for better performance in real-world applications. The paper mentions this two-stage structure applies across twelve different transformer language models from eight families.

Lalam: I see a massive potential here for improving how we train and fine-tune models; understanding this factory process could lead us to design architectures that build word meaning more efficiently from the start, which would definitely improve culture by making AI output much more coherent.

Tom: Exactly, Lalam; the paper suggests this process is universal across a lot of different models, which means we can have a much better grasp on how to push them further. They detail how attention sends that token-specific signal and the MLP then mixes it with the local embedding at each step.

Jane: That two-stage division is key, Tom; they differentiate the roles based on whether a pair shares a token; for instance, in Last-Shared-Token pairs, the difference has to come from cross-position information from attention because the local embedding at that last position is identical across runs.

Lu: And in First-Shared-Token pairs, where tokens are shared right at position one, the paper finds that the Layer one attention output is essentially interchangeable; it doesn't carry enough distinguishing information on its own to tell a successful run from a failed one.

Meng: That points directly to the MLP being responsible for making that nonlinear transformation that actually determines if the reconstruction succeeds or fails in those FST settings. It shows the MLP has this crucial role in composing the signal with whatever attention provides.

Paper summary: Lalam: If we can better understand how that composition works, it could mean we can fine-tune models to be more robust against errors when they have to handle slightly unusual word combinations during inference. That level of nuance is what makes a model truly reliable for complex tasks.

Tom: And the paper moves into measuring success using "canonicity," which they define as the cosine similarity at layer n minus two between the residual stream at the last subword position and its canonical single-token representation. High canonicity, they say, indicates a successful reconstruction, while low values signal failure.

Jane: That metric gives us a concrete way to quantify how well detokenization is working within the model's internal computations; it moves beyond just looking at the final output to checking the stream itself at an earlier layer.

Lu: They also used activation patching to localize these components, showing that patching Layer one attention output closed about fifty-three percent of the gap in LST settings, establishing attention as the first mechanism for transfer from position one to position two.

Meng: That tells us attention is definitely involved early on, but it doesn't mean it does all the heavy lifting; we still need to account for what’s happening later in the network structure.

Lalam: So, if attention handles the initial signal transmission, where do we look next to understand how that signal gets fully realized into a proper word? That transition sounds like something worth exploring further for cultural impact on content creation.

Tom: The paper also looked at how depth affects this process depending on positional encoding; they found that RoPE-based models detokenize over one to five layers, whereas learned-absolute models take between five and ten layers.

Jane: And they identified a specific point, called the "gap-closed-eighty percent depth," which is layer l star, where the transition from a successful run to a failed run becomes characterized by that specific depth, and this depth varies based on word length.

Lu: The way this gap-closed-eighty percent depth fluctuates depending on whether you use RoPE or learned positional encoding shows us how different positional encodings affect the overall flow of information during detokenization.

Meng: From a practical standpoint, knowing this depth variation is helpful because it suggests that when we deploy these models, the required precision in capturing this process changes based on the model's underlying architecture.

Paper summary: Lalam: If we can predict how deep we need to look into the network for accurate word reconstruction based on the model type, it could help us build more efficient and less computationally expensive AI systems overall.

Tom: Moving past just the two stages, they looked at intermediate positions relaying information through sequential relays, noting that each intermediate position relays information in a fixed window of two to three layers based on its index.

Jane: And they also quantified the effect of this relaying; corrupting all those intermediate positions simultaneously produced peak drops in success at Layer two which increased significantly with longer word lengths.

Lu: That collective relay contribution growing with word length suggests that as inputs get longer, these intermediate steps become increasingly important for maintaining the integrity of the detokenization signal.

Meng: It makes sense that if you're dealing with a longer sequence, there are more chances for errors to compound through those sequential relays before they get composed properly by the MLP.

Lalam: This idea of sequential relaying implies that we might need to focus our efforts on stabilizing these intermediate layers rather than just focusing solely on the input or output stages.

Tom: So, to wrap up this look at "Inside the LLM Word Factory," the authors are showing us a detailed two-stage process where attention provides a directional signal and the MLP composes it with local embedding to form words. The paper establishes that this mechanism is universal but its depth depends on positional encoding.

Jane: And they provide metrics like canonicity and activation patching to show exactly where these mechanisms operate within the transformer blocks, confirming attention's role in LST pairs and the MLP's role in FST pairs.

Lu: It’s a very precise decomposition of a complex process that was previously just an observed phenomenon; pinning down this mechanism gives us a much clearer blueprint for how these models function at their core.

Meng: Knowing the exact layer depth needed for success, like that gap-closed-eighty percent depth, helps engineers set realistic expectations when building systems that rely on these models to handle specific data formats.

Lalam: The implication is that we can design AI systems with better introspection capabilities, allowing us to diagnose exactly where a failure in word generation might be originating in the model's internal structure.

Tom: It’s clear this paper gives us a detailed roadmap for understanding how these models handle the translation from subwords to full words, and it sets up some really interesting avenues for future research into model efficiency.

Conclusion: Tom: So we've been looking at how these massive language models actually build words from subwords, and now we're getting to the wrap-up of this paper, "Inside the LLM Word Factory."

Jane: That paper really maps out that complex process of detokenization by breaking it down into two distinct stages handled by attention and the MLP.

Lu: I think what’s most striking is how they use activation patching to show exactly where those different components are working within the transformer layers.

Meng: From a practical standpoint, understanding this factory process gives us a much clearer picture of where errors might be creeping in during inference.

Lalam: If we can precisely locate these stages, it opens up huge possibilities for building more robust and reliable language models for real-world applications.

Tom: Exactly! And looking at the title and authors, it’s clear this research is about deconstructing the internal workings of how AI converts raw tokens into coherent text.

Jane: The authors have done a really good job of presenting this complex mechanism in a way that’s easy to follow, especially when you break down the roles attention and MLP play.

Lu: It's fascinating how they show that this process isn't just happening randomly across layers but follows specific rules tied to positional encoding.

Meng: I wonder how this structural understanding will change the way we design new AI architectures moving forward; knowing these constraints upfront is invaluable for engineering decisions.

Lalam: For culture, this means we can potentially train models that are far less likely to generate nonsensical or corrupted text during complex tasks.

Tom: That’s a big vision, Lalam; it suggests a level of internal control over the AI's generation process that we haven't fully realized yet.

Jane: And the implications for how we evaluate model success are huge, because they introduce a new metric called canonicity to measure how well that reconstruction actually worked.

Lu: Canonicity seems like a very powerful way to quantify success within the internal workings of the model, giving us a hard number on performance.

Meng: It moves us away from just looking at final output quality and gives us deep insight into the fidelity of the process itself.

Lalam: That level of detail is what we need to improve AI's ability to handle nuanced and complex human language accurately in every context.

Tom: So, in a nutshell, this paper is giving us a blueprint for understanding the inner mechanics of how language models actually form words.

Jane: And it’s showing that this process isn't monolithic; it’s a carefully orchestrated two-stage effort between different parts of the network.

Lu: It really helps paint a picture of the entire transformation from input tokens to meaningful output in these large systems.

Meng: Knowing these specific layer dependencies means we can start thinking about optimizing training methods around these known operational constraints instead of just blindly scaling up.

Lalam: This deep insight is vital because it’s how we build an AI that truly understands the structure of language, which will ultimately enhance how we interact with and create content with AI tools.

Benzi Busigin Yuval Pinter

Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev

cs.CL

Submitted: 2026-06-07

Updated: 2026-09-28

Code: https://github.com/Benzi-Busigin/Inside_The_

Importance score: 92/100

The gist: Detokenization is characterized as an early-layer two-stage mechanism where attention writes a token-specific directional signal from preceding subwords, and the MLP composes it with the local

Key concepts

Detokenization
An early-layer two-stage mechanism where attention writes a token-specific directional signal from preceding subwords, which the MLP then combines with the local embedding to reconstruct the original token. This process is universal across twelve transformer language models.
Last-Shared-Token (LST) Pairs
In LST pairs, success depends on cross-position information transmitted by attention from a differing first token. The local embedding at the last position is identical across runs, meaning the difference in outcome must come from this attention signal.
First-Shared-Token (FST) Pairs
For FST pairs where tokens are shared at position 1, Layer 1 attention output is 'genuinely interchangeable' and carries no distinguishing information. Instead, the MLP's nonlinear transformation is what determines whether the reconstruction is successful or failed.
Canonicity
The primary success metric is 'canonicity,' measured as cosine similarity at layer n-2 between the residual stream at the last subword position and its canonical single-token representation. High canonicity indicates a successful reconstruction, while low values indicate failure.

Terminology

Summary

Detokenization is characterized as an early-layer two-stage mechanism where attention writes a token-specific directional signal from preceding subwords, and the MLP composes it with the local embedding. This mechanism is universal across twelve transformer language models and its depth is governed by positional encoding, allowing for a linear prediction of detokenization success from early layers.

How it works

The two-stage process begins in Layer 1, where attention transmits a signal from nonfinal subwords using sequential relays if necessary, while the MLP composes this signal with the local embedding. This division of labor holds across word lengths ranging from two to six subword tokens, with composition occurring at the last position and intermediate positions serving as relays.

The specific roles of these components are differentiated based on whether a pair shares a token:

  1. In Last-Shared-Token (LST) pairs, the difference in outcome must arise from cross-position information that attention transmits from the differing first token, as the local embedding at the last position is identical across runs.

  2. In First-Shared-Token (FST) pairs, where tokens are shared at position 1, Layer 1 attention output is genuinely interchangeable, meaning it carries no information that distinguishes successful from failed composition; in this case, the MLP's nonlinear transformation is what turns the local input into a successful (or failed) reconstruction.

Measuring Success and Localization

The primary success metric introduced is canonicity, defined as the cosine similarity at layer n − 2 between the residual stream at the last subword position of a split input and its canonical single-token representation. High canonicity indicates successful reconstruction, while low values indicate failure. Activation patching is used to localize these components:

  1. Patching Layer 1 attention output closes 53% of the gap in LST settings, establishing it as the first and primary mechanism for information transfer from position 1 to position 2.

  2. In FST settings, patching Layer 1 MLP output closes 53% of the gap, demonstrating that while attention is uninformative in this setting, the MLP is responsible for composing the signal with the local embedding.

Mechanism Depth and Scaling

The depth over which detokenization takes place depends on positional encoding:

  1. RoPE-based models detokenize over 1 to 5 layers.

  2. Learned-absolute models take 5 to 10 layers.

  3. The transition from a successful run to a failed run is characterized by the gap-closed-80% depth (denoted as layer l∗), which varies with word length (k). For example, for RoPE models, this depth stays in single digits across all word lengths (5–6 layers at k=4), while for learned-PE models, it ranges from 11 to 17 layers at k=4.

Behavioral Readability and Generalization

The outcome of detokenization is linearly readable from early layers:

  1. A class-mean-difference probe fits on the residual stream at layer l∗ (the gap-closed-80% depth) to predict success or failure, achieving high AUROC scores (e.g., 0.94 for Isolated and 0.97 In-context on Llama2-7B).

  2. This isolated direction transfers to natural text, where the probe achieves AUROC of 0.91 on in-context activations within 0.03 of its own isolated test-set AUROC (0.94).

  3. The two regimes—concentrated and distributed—are partitioned by positional encoding: concentrated models complete both stages within the first 5 layers, whereas distributed models spread both stages across layers 5–12. Bloom-7B1 (ALiBi) sits between these poles, tracking the concentrated regime for 2-token words but drifting toward a distributed profile as token count scales up.

Intermediate Position Relay

Intermediate positions actively contribute to detokenization through sequential relaying:

  1. Each intermediate position relays first-token information in a fixed 2–3 layer window whose timing is determined by the position's index, not total word length.

  2. The collective relay contribution grows with word length; corrupting all intermediate positions simultaneously produces peak drops at Layer 2 (9% in 3-token words, rising to 21% in 5-token words), exceeding the sum of individual drops by an increasing factor (1.4× for 4-token, 1.6× for 5-token).

Conclusion

The mechanism is a two-stage process: attention writes a small, token-specific directional signal, and the MLP applies a continuous transformation to compose it with the local embedding into the canonical representation.

Improvements for AI systems

Based on the provided research, here are specific improvements that can be made to AI systems (specifically Transformer language models) derived from this work, categorized by impact:

)Improved System Capabilities: Mechanistic Debugging and Detokenization Failure Detection

The core contribution is identifying the precise location (Layer 1 for Llama2-7B) and mechanism (Attention + MLP interaction) responsible for mapping subword fragments back to word-level semantics. This allows for the creation of systems capable of diagnosing internal representation failures.

  1. Mechanistic Failure Localization:

Deep inspection tools can be developed that use activation patching to pinpoint which specific layer and component (Attention vs. MLP) is failing during token-to-word reconstruction for a given input fragment.

  1. Proactive Artifact Detection:

An inference pipeline can be designed to calculate Canonicity Scores in real-time for generated text or incoming prompts. If the score drops below a critical threshold, the system flags the output as potentially suffering from tokenization artifacts (e.g., under-trained tokens, poor arithmetic reasoning).

  1. Targeted Model Retraining:

Instead of retraining the entire model on noisy data, researchers can use these localized insights to create detokenization-aware fine-tuning objectives that specifically target the identified layer(s) and components (e.g., focusing on refining the MLP's transformation in Layer 1 or tuning specific attention heads).

)Improved System Capabilities: Enhanced Robustness and Generalization

The paper establishes a universal two-stage mechanism governed by positional encoding, providing a roadmap for architectural design and scaling.

  1. Positional Encoding Optimization:

Architectural design can be guided by the required detokenization depth. If the goal is high robustness across all word lengths (k=2 to 6), models should employ positional encodings that favor a concentrated regime (like RoPE) if computational efficiency is prioritized, or distributed regimes (like ALiBi) if maximizing generalization across varying context lengths is the priority.

  1. Context-Aware Inference Scheduling:

Since the paper shows that context accumulation in deep layers causes a divergence from early-layer signals, inference engines can be optimized to halt or re-initialize detokenization checks at specific layer depths (e.g., Layer 1 or 2) when processing prompts where word length is highly variable, thereby reducing computational overhead while maintaining high fidelity for critical early semantic mapping.

)Improved System Capabilities: Predictive Modeling and Evaluation Metrics

The finding that early-layer activations linearly predict success offers a powerful new evaluation paradigm.

  1. Early-Layer Success Predictors (ELS):

A lightweight probe module can be trained on the first few layers of a model to act as a fast, cheap filter for input quality. This ELS would output a probability of successful detokenization before the full generation process begins, significantly speeding up pre-processing for high-throughput applications.

  1. Tokenization Metric Replacement:

The system can move beyond behavioral metrics (like next-token prediction agreement) as the sole measure of tokenizer quality. A new metric based on the linear readability of early-layer residuals (as established by the AUROC probe in Section 6) provides a cleaner, model-internal measure of whether the tokenizer is successfully feeding coherent semantic concepts to the core transformer computation.

)Improved System Capabilities: Handling Complex Inputs (K-Token Words)

The findings on scaling with token count inform how systems handle complex or rare vocabulary.

  1. Adaptive Relay Management:

For inputs that are not standard two-token words (e.g., technical jargon or compound terms), the system can dynamically adjust its internal relay mechanism, allocating more computational resources to intermediate layers (Position 2, 3, etc.) when the input is long and complex, effectively managing the non-uniform cost of detokenization identified in Section 4.

Sources

Related papers