Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)

arXiv:2605.27268 · cs.CL, cs.AI · Submitted 2026-08-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)".

Jane: The paper was written by the authors from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary of Findings: Jane: Right, so we’ve talked about *how* they measure it with the Word Coverage Score, but now we need to talk about what they actually showed us when they ran these tests. The summary part of "Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)" highlights some really striking differences.

Tom: What struck me immediately when looking at the examples—like comparing 'supposedly' versus 'profitable'—is how massive that range of mean reachability is. It's not just a matter of being rare or common; it’s about the *consistency* of accessibility.

Lu: I noticed the disparity between words like 'exceptionally' and 'disadvantage.' One gets a mean reachability of zero point one zero seven, while the other hits zero point four nine four in the first table's examples, suggesting an inherent structural difference in how the model treats those concepts.

Meng: And that contrast is critical for engineers because it tells us if we’re fighting a low-level tokenization problem or something deeper related to semantic context within the training data that makes certain concepts harder to stabilize.

Lalam: Thinking about implication, these results show that surface-level word frequency, which is what many previous metrics relied on, completely misses this deeper layer of reliable usage that WCS captures.

Jane: So, if we look at the second set of examples—the 'Easier to reach' words—they tend to cluster around a higher mean reachability score, suggesting they are robustly embedded in the model’s active vocabulary.

Tom: But Jane, you have to point out that even within those "easier" words, there’s still variation. We can’t just assume that because 'bedside' scored zero point five zero one and 'strangers' scored zero point four seven seven, they are equally useful in a real-world application.

Lu: That variation is the signal! It implies that the context surrounding the prompt matters immensely, and WCS is giving us a quantitative proxy for that contextual stability across different sampling regimes.

Meng: From a practical standpoint, if we were building an application where accurate recall of specific domain jargon was paramount—say, medical or legal text—we’d need to target the low-scoring words identified here first.

Lalam: This underscores how culture transmits language; highly reachable words are the ones that become stable and easily transferable through communication, while the low-reachability ones might be niche or evolving concepts.

Tom: So, we're seeing a clear delineation between what's just *present* in the data versus what's *accessible* on demand. This leads perfectly into how the authors suggest we can actually improve upon these scores. Jane?

Suggested Improvements: Jane: Moving on to the suggested improvements, "Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)" isn't just pointing out flaws; it's offering paths forward. The authors are suggesting ways to make these scores more informative and actionable.

Tom: I remember reading about them focusing on moving beyond a simple mean across all parameters, which is a huge step in precision, Jane. It’s like saying, "Don't just tell me the average score; tell me *under what conditions* this score holds."

Lu: The suggestion to incorporate more granular analysis of the sampling process itself—for instance, analyzing reachability specifically when switching between temperature levels—that’s where the real breakthroughs will happen for model interpretability.

Meng: What I liked about their proposed improvements is that they sound engineering-adjacent. They aren't asking for magical fixes; they are asking for better measurement tools and more controlled testing environments to diagnose the underlying failure mechanism.

Lalam: If we incorporate this idea of diagnostic testing into how AI is used culturally, it changes the expectation from "It knows" to "Show me *how* you know it." That level of accountability is vital for trust.

Jane: It’s about making the model's decision-making process less of a black box and more like a transparent machine that can show its work when asked about word retrieval.

Tom: Exactly! And Lu, building

Paper discussion segment 3: Tom: So, if we’re taking away one thing from this analysis of word reachability, it's that simply knowing how often a word appears in the corpus isn't enough to guarantee an AI can actually use it.

Jane: Exactly. The WCS really highlights that even if a word is statistically common, the mechanics of generating it—the specific context or the sampling setting—can make it surprisingly difficult for the model to hit.

Meng: That’s where the actionable research comes in, right? If we know *that* low reachability exists, then we need concrete methods to improve it before deploying these systems widely.

Lu: And those methods can't just be brute force; the paper suggests that adjusting the sampling process itself—maybe making it context-aware or dynamically altering temperature based on predicted difficulty—is key to overcoming these bottlenecks.

Tom: So, you're saying we need a smarter way to sample? Not just picking randomly, but actively trying to reach those tricky words like "exceptionally" or "profitable"?

Jane: It’s about making the generation process less random and more targeted toward maintaining lexical coverage across the board, rather than optimizing purely for fluency or statistical probability.

Lalam: Thinking about this capability improvement—the ability to reliably generate any word needed—it profoundly impacts how we build cultural continuity in our digital tools.

Meng: From an engineering standpoint, that means building a robust layer on top of the LLM that constantly monitors the WCS *during* generation, not just after. We need real-time feedback loops.

Lu: That opens up possibilities for specialized domain models; imagine an AI writing historical fiction where it *must* flawlessly reproduce archaic vocabulary because we’ve trained it on a limited, high-value corpus.

Tom: It makes the whole system feel less like magic and more like predictable, controllable machinery that we can actually debug!

Jane: I think the biggest implication is that this sets a new standard for evaluating LLMs—we can't just grade them on coherence; we have to grade them on their lexical stability.

Lalam: If AI models are guaranteed to preserve the full spectrum of human language, then they become true partners in knowledge creation, not just sophisticated echo chambers repeating common phrases.

Meng: So, practically speaking, implementing this could mean a noticeable increase in the diversity and precision of AI-generated text across professional fields like legal writing or specialized scientific reporting.

Lu: And it pushes us toward multimodal models that can cross-reference linguistic difficulty with other sensory data—like how hard it is to describe an obscure color, for example.

Tom: It really shifts the goalposts, doesn't it? We’re moving beyond just asking "Can it talk?" to asking "Can it say *exactly* what we need it to say?"

Jane: And that level of precise control is going to unlock so many complex applications that we haven't even thought about yet.

Lalam: This focus on comprehensive linguistic access signals a maturity in AI, moving us closer to tools that truly augment the breadth of human thought and culture.

Conclusion: Tom: So, looking back at everything we covered today on "Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score," it really hammers home that just knowing how often a word appears in the training data isn't enough to predict if an AI will actually use it.

Jane: Exactly, Tom; what the authors showed us is that reachability is this deep, complex decoding property—it involves things like how the tokenizer breaks up words or even the specific temperature settings you’re running at.

Lu: And I keep thinking about how revolutionary it is that we can quantify this failure mode! If we can map out which vocabulary items are structurally hard for an LLM to access, we're talking about a whole new frontier in model design, maybe even dynamic context injection systems.

Meng: But Lu, from an implementation standpoint, if the goal is just making the model *better*, doesn't this all boil down to just tweaking the sampling parameters until those scores look high enough for critical terms?

Lalam: I think that overlooks something deeper than mere optimization, Meng. If we can reliably boost reachability for specific lexicon sets, we’re not just improving text generation; we’re improving the richness of collective thought itself.

Tom: You've got a point there, Lalam; it suggests that if we want the AI to adopt certain cultural idioms or niche vocabulary, we might need to engineer the sampling process specifically for those semantic domains.

Jane: Right? It means we can’t just treat language like a smooth, continuous surface; it’s more like this incredibly intricate scaffolding where some words are easier to build with than others.

Lu: Exactly! We could design models that actively prioritize paths through the vocabulary space that have proven high WCS scores for historically important or underrepresented lexicons.

Meng: That brings me back to practicality: if we use this score to guide training or decoding, what kind of computational overhead are we adding? Are we just calculating a metric, or is it slowing down inference significantly?

Lalam: It’s about valuing the *potential* for culture, Meng. If this research helps us build systems that consistently preserve less common but vital words—like *sylvan* or *bridle*—we're ensuring that the breadth of human experience remains audible through the AI.

Tom: So, while the technical details of WCS are fascinating, what it ultimately signals is that we need to be far more thoughtful about how we guide these powerful systems when they generate text.

Jane: It’s a reminder that behind every coherent sentence there's a whole lot of structural probability working its magic, and sometimes that magic gets stuck.

Lu: I just hope this work inspires us to see LLMs less as prediction machines and more as incredibly detailed, but fragile, linguistic mirrors.

Meng: Fingers crossed the next iteration of these models can handle the practical implications of this research without becoming computationally prohibitive for general use.

Lalam: Keeping the full scope of "Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)" in mind, it points toward a future where AI collaboration elevates our shared vocabulary.

Tom: We've certainly got a lot to chew on here; next up, we’re switching gears entirely and looking at some really wild developments in multimodal data fusion...

cs.CL, cs.AI

Submitted: 2026-08-21

Updated: 2026-08-24

Code: https://github.com/WordsGPT/WCS

Project page: https://wordsgpt.github.io/WCS

Importance score: 83/100

The gist: The paper assesses lexical reachability in Large Language Models (LLMs) using the Word Coverage Score (WCS).

Key concepts

Word Coverage Score (WCS)
A metric introduced in the paper to assess lexical reachability in LLMs. It measures how consistently and reliably a model can generate specific words, going beyond simple word frequency to capture structural accessibility.
Lexical Reachability
The concept that measures an LLM's ability to access and use specific vocabulary items. The hosts emphasize that this is a deeper, more reliable measure of usage than just knowing if a word exists in the training data.
Sampling Regimes
The various settings or methods used when an LLM generates text (e.g., temperature levels). The discussion highlights that the context and specific sampling regime significantly impact how easily a model can access certain words.
Tokenization Problem
A potential failure mechanism in LLMs where the way words are broken down into tokens makes certain concepts harder to stabilize or generate. Engineers must diagnose if low reachability is due to this or a deeper semantic issue.

Terminology

Summary

The paper assesses lexical reachability in Large Language Models (LLMs) using the Word Coverage Score (WCS). The WCS is defined as the average fraction of evaluated model/sampler/context conditions in which the word remained reachable, providing a word-level view of which selected targets were consistently preserved by the decoding process and which were more often pruned.

A primary focus of the analysis was determining if a word's source frequency correlates with its reachability. The findings indicate that frequency alone is insufficient to explain lexical reachability. Specifically, The scatter shows that frequency alone does not explain reachability: some comparatively frequent words remain difficult to reach, while some lower-frequency words are preserved more often. Quantitatively, "Using log-transformed source frequency, the Pearson correlation with mean word-level WCS is weakly positive (r = 0.29), indicating that corpus frequency alone explains only a limited portion of the observed variation in lexical reachability."

The study provided detailed insights into both low and high reachability words. Regarding difficulty, The hardest words to reach were not simply the rarest words in the selected band. Instead, several low-reachability targets were found across various frequency ranges. Examples of such difficult words included those from the middle or upper parts of the sampled frequency range, such as supposedly (rank 14,095; mean reachability 0.076), exceptionally (rank 16,242; 0.107), and acknowledges (rank 16,036; 0.119). The lowest-reachability group also contained rarer words like sylvan (rank 27,065; 0.079), saddened (rank 39,755; 0.082), and precipitated (rank 39,823; 0.087). This leads to the conclusion that lexical reachability is not determined by corpus frequency alone; it also depends on tokenizer segmentation, context, and model-specific probability structure.

Conversely, the analysis of easy-to-reach words showed a mix of frequencies. The highest-scoring targets were diverse, including relatively frequent words such as profitable (rank 10,437; mean reachability 0.537), disadvantage (rank 15,341; 0.526), bedside (rank 24,974; 0.501), and offenders (rank 10,104; 0.494). Furthermore, the easiest group included several lower-frequency targets, such as volley (rank 37,393; 0.446), feces (rank 34,197; 0.444), bridle (rank 37,428; 0.434), and workmen (rank 38,637; 0.389). This pattern reinforces that the WCS captures a decoding-level accessibility property rather than merely reproducing word-frequency rank.

Improvements for AI systems

This paper introduces a critical diagnostic concept—Per-word reachability—that fundamentally challenges the assumption that source word frequency dictates generation difficulty. Since this measure captures an inherent decoding-level accessibility property, it provides a powerful new axis for model evaluation and control.

Given the high stakes (millions of dollars), I propose three interconnected improvements to AI systems: a novel diagnostic module, an enhanced decoding mechanism, and a specialized fine-tuning objective.

We must build a dedicated, offline module that quantifies per-word reachability before deployment or task execution. This moves beyond simple Perplexity or BLEU scores.

What the improved system can do:

The diagnostic module accepts a corpus of target words and simulates their generation across various candidate decoding paths (e.g., beam search, top- k sampling, nucleus sampling) under controlled conditions (fixed temperature, fixed context window).

  1. Quantify Word Accessibility: For every target word (W t), it calculates the mean reachability score (WCS mean) by tracking the fraction of successful paths that include W t.

  2. Identify Bottlenecks: It generates a comprehensive Reachability Heatmap for a given domain or task, pinpointing specific words that are structurally difficult for the model to maintain in context, regardless of their corpus frequency (e.g., identifying why sylvan is problematic even if the model has seen it many times).

  3. Predictive Failure Analysis: It can predict which complex terminology or domain-specific vocabulary will likely be pruned during generation, allowing human reviewers to preemptively flag high-risk outputs before they reach the end-user.

The most impactful improvement is integrating the calculated reachability score directly into the decoding process. Instead of only maximizing probability (P(W t C)), the system must maximize a weighted combination of probability and accessibility.

  1. Modified Sampling: When selecting the next token or word, the system calculates:

RAL Score(W t) = (P(W t C)) + lambda times WCS mean(W t)

Where lambda is a tunable hyperparameter that controls the weight given to accessibility versus raw probability.

  1. Constrained Generation: If the system detects that the current generation path is trending toward a low-reachability word (e.g., predicting supposedly when context strongly suggests something else), it can dynamically adjust its sampling parameters (e.g., reducing temperature or widening k) to force exploration of higher-reachability alternatives, thereby mitigating pruning failure modes.

  2. Guaranteed Inclusion: For mission-critical outputs (e.g., names, technical jargon), the system can operate in a Forced Reachability Mode, where it biases the sampling towards known high-WCS targets, even if they carry a slightly lower immediate probability score, guaranteeing their inclusion when necessary.

To make the model intrinsically better at maintaining reachability, we must introduce a novel loss function during fine-tuning or pre-training.

  1. Training Data Augmentation: The training data is augmented with Reachability Pairs: (Context C, Target Word W t, WCS target).

  2. Loss Function Modification: The overall loss function (L total) is modified:

L total = L NLL + beta times sum i Predicted Reachability(W t) - WCS target(W t)

Where L NLL is the standard Negative Log-Likelihood loss, and beta controls the strength of the reachability penalty.

  1. Structural Improvement: By minimizing this loss, the model is forced to learn internal representations that do not just predict high probabilities, but also structurally support those tokens across diverse contexts and decoding methods. This fundamentally improves the robustness and consistency of complex vocabulary generation, making the system less prone to contextual drift or token pruning.

Sources

Related papers