ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization

ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization

Lixing Li, Cornell University

arXiv:2608.12756v1 [cs.CL] 13 Aug 2026

Abstract

Adaptive latent tokenization maps a fine-grained input to a shorter sequence of continuous representations associated with input-dependent spans. We introduce ReconSpan, which divides text into chunks that a backward decoder can reconstruct from a single contextual prefix code and retains one such code as the latent token for each chunk. The reconstruction criterion is applied when chunks are formed, allowing one trained autoencoder to produce average chunk lengths from 6.5 to 12.2. At matched average length, reconstruction-guided boundaries preserve more text than random boundaries. Readers of the resulting latent sequence recover topic information reliably but struggle to extract exact details.

1. Introduction

Language models do not operate directly on raw text: tokenization determines the sequence positions to which representation and computation are assigned. Conventional subword tokenizers choose these units before the model runs, largely from corpus-level frequency statistics. Their granularity is therefore fixed rather than conditioned on the actual input.

Byte- and character-level inputs avoid a fixed subword vocabulary and retain fine-grained information, but they also produce substantially longer sequences. Adaptive latent tokenization instead groups neighboring units into input-dependent spans and represents each span with one continuous latent token. Its chunking rule is central: boundaries determine when new latent positions are created and thus where representation and computation are allocated. Here, a latent token is a contextual representation assigned to a span boundary; unlike a conventional token embedding, it may encode preceding context as well as the associated span. This makes unit formation a model-dependent allocation decision rather than a fixed preprocessing choice.

We propose ReconSpan, an adaptive latent-tokenization method that divides text into chunks a decoder can reconstruct from a single prefix code and retains one such code as the latent token for each chunk. A causal encoder produces a code at every position; starting from the final code, a backward decoder reconstructs until an error criterion is exceeded, accepts the resulting suffix as a chunk, and repeats from the first unreconstructed position. Differences in reconstruction reach give shorter chunks to difficult spans and longer chunks to easier ones. Because the criterion is applied only during chunking, it can also be relaxed to increase average chunk length without retraining, giving ReconSpan input-dependent allocation and post-training control of granularity.

We evaluate both the induced tokenization and the information accessible from its latent tokens. ReconSpan produces average chunk lengths from 6.5 to 12.2 tokens, and at matched average length its reconstruction-guided boundaries preserve more text than random boundaries. Native reconstruction tests the selected spans through the autoencoding route, while a separately trained reader consumes the contextual latent tokens directly and predicts text or task outputs. These readers recover topic information reliably but struggle to extract exact details, exposing a gap between information retained by the autoencoder and information accessible to another model.

Our contributions are:

• We introduce reconstruction fidelity as a chunk-allocation criterion for adaptive latent tokenization over an existing subword sequence.

• We show that one autoencoder supports multiple post-training granularities and that its variable-span boundaries outperform length-matched random boundaries in native reconstruction.

• We characterize the resulting contextual latent tokens through direct readout, separating information retained by the autoencoder from information accessible to readers of different scales and levels of task adaptation.

2. Related Work

Learned tokenizers and hierarchical sequence models differ in the signal that determines their source-text chunks. The closest work is organized by this allocation rule:

• Chunking by fixed position: MEGABYTE divides byte sequences into fixed-size patches. Extensible Tokenization contextualizes existing subword embeddings and retains representations at a regular stride; it accepts subword inputs, emits continuous contextual representations, and allows granularity to change after training.

• Chunking by predictive entropy: The Byte Latent Transformer (BLT) creates variable byte patches at spikes in next-byte entropy. Dynamic Token Pooling also studies an entropy-supervised boundary predictor. These methods use forward predictive uncertainty; ReconSpan instead measures backward reconstruction fidelity.

• Chunking by semantic similarity: SemToken embeds existing tokens contextually, merges adjacent semantically similar spans, and varies granularity with local semantic density. ReconSpan does not optimize similarity or claim that its boundaries are semantic.

• Chunking by learned boundary scores: Dynamic Token Pooling predicts variable character-level segments using end-to-end, tokenizer-supervised, entropy-supervised, or linguistic objectives. H-Net learns content- and context-dependent routing jointly with a hierarchical byte-level language model. Charformer scores candidate byte blocks from the end-task loss. FLEXITOKENS learns variable byte boundaries while relaxing the fixed target-rate objective.

• Chunking by reconstruction fidelity: ReconSpan places a boundary according to how far a backward decoder can successfully reconstruct from each contextual encoder state. The signal is measured autoencoder reconstruction rather than a fixed position, learned router, forward entropy, or semantic density. Changing the accepted reconstruction criterion adjusts average span length after training.

3. Method

ReconSpan requires a generic autoencoding model and an inference-time chunking algorithm.

3.1 Autoencoding model

Let x1:n be a token sequence. ReconSpan requires two learned components:

• A prefix encoder maps any token sequence to one code, E: V* → Rd, ct = E(x1:t). A causal model such as a Transformer or a Mamba can generate c1,..., cn in one pass.

• A backward decoder maps ct to the encoded tokens in reverse order, xt, xt−1,.... Backward decoding is the point of the design: decoding newest-first, the position where the decoder first fails is a direct measurement of how far back that one code reconstructs. A forward decoder would instead have to be told where to start — the very quantity we want to measure.

The autoencoding path is therefore E: (x1, x2,..., xt) → ct → D: (xt, xt−1,..., x1).

3.2 Chunking algorithm

The decoder will not reconstruct arbitrarily long prefixes, so we only ask it to decode within its own capacity; the point at which it fails sets each chunk boundary, giving adaptive-length chunks and their latent tokens. Because the text being tokenized is already known, the decoder is teacher-forced against it: at each reverse step it is fed the true previous tokens and we record only whether its own greedy (argmax) prediction matches — nothing is sampled or generated. Write the resulting chunk endpoints in chronological order as 0 = b0 < b1 <... < bm = n, so chunk i is x(bi−1+1):bi and its contextual latent token is cbi = E(x1:bi).

Algorithm 1 (ReconSpan chunking):

1. Compute all prefix codes (c1,..., cn) ← E(x1:n) in one causal pass; set C ← ⟨⟩ and t ← n.

2. While t ≥ 1:

(a) Teacher-force D from ct against xt, xt−1,... until the stopping rule fires at xb, so that xb+1,..., xt are accepted; set b ← 0 if it reaches the beginning without firing. If it fires on the first step, set b ← t−1 (a one-token chunk, guaranteeing progress).

(b) Append ct to C as the code for chunk xb+1,..., xt; set t ← b to resume from the first unreconstructed endpoint.

3. Reverse the collected latent tokens into chronological order and return C = (cb1,..., cbm) — only these chunk-boundary prefix codes are retained.

Two stopping families are used:

• Failure(m): stops at the mth incorrectly reconstructed token. Failure(1) ends a chunk at the first mistake and larger m tolerates errors.

• Logit-gap(τ): the continuous version. Let yj be the actual token at reverse step j; the decoder stops at the first k for which Gk = Σ(max v z(j) v − z(j) yj) > τ. Each summand is zero when the model predicts the correct token and otherwise measures by how much it was missed, so Gk accumulates near-misses instead of counting outright errors.

Raising m or τ lengthens chunks on a fixed trained model and thus increases the average number of input tokens represented by each latent token.

The number of sequential model invocations is O(1 + n). Producing all prefix codes takes a single encoder forward pass. The backward reconstruction is autoregressive, so in the worst case it needs O(n) sequential decoder calls. In practice, the decoder reads a fixed block of W positions per endpoint for efficient batching, so decoding is O(W n) work in the worst case with the specific Mamba decoder. Appendix D describes a variant that keeps the same O(W n) total work but cuts the sequential decode calls to O(W), independent of n, by materializing all endpoint decodes at once.

3.3 Training

The only training objective is autoencoder reconstruction; the span allocation emerges from the decoder's capacity rather than from an explicit length target. A training step operates on one window x1:L. The encoder produces its final code cL = E(x1:L), a learned projection maps it to the decoder's initial recurrent state, and the decoder is teacher-forced to reproduce the window in reverse, y = (xL,..., x1, EOS). Each step reconstructs the first k ≤ L+1 reversed targets, and the loss is the token cross-entropy over them:

LAE = −Σ log pD(yj y<j, cL)

The projection is differentiable and no stop-gradient is placed on the code, so this loss reaches the encoder as well as the decoder. Encoder and decoder can therefore be trained jointly.

Capping the decode length at k ≤ L+1 makes most steps reconstruct only a suffix of the window rather than the whole text. This is a deliberate match to how ReconSpan uses the decoder: the design is intended to prioritize recent tokens and thereby extend the successful suffix that determines a chunk boundary.

3.4 Implementation overview

The encoder is a Pythia-410M Transformer whose final 1024-dimensional hidden state is the code; the decoder is a Mamba2-130M backward model. The code is up-projected to the decoder's per-layer initial SSM states, and a BOS token begins the rollout. Pythia shares the GPT-NeoX tokenizer with the Mamba2 decoder. Two auxiliary Mamba2 encoders are also evaluated: a 4096-dimensional projected SSM state and a 1024-dimensional hidden state. The former requires four times the code width to approach the Transformer's short-span reconstruction, while the latter is weaker at equal width.

Each step samples a short-biased encode length L and an independent decode length k ≤ L+1. Training runs in two stages on FineWeb: a 7B-token backward-decoder pretrain with the encoder frozen, then 3B tokens of joint encoder–decoder training.

4. Tokenizer properties

The tokenizer is characterized through four measurements: single-code autoencoder reconstruction, semantic structure in the raw code geometry, the span lengths induced by different stopping rules, and how much text survives a native autoencoder round trip under the selected boundaries.

4.1 Autoencoder reconstruction quality

For each length l, one l-token window is sampled from each of 500 held-out Wikipedia documents, encoded into one code, decoded autoregressively in reverse, and scored against the original using BLEU and ROUGE-L, alongside exact-match measures. SONAR is used as an external reference point.

The key column is suffix, the mean length of the longest exactly reconstructed suffix — the contiguous run of most-recent tokens the backward decoder emits without an error. The Transformer recovers an average exact suffix of roughly 8–11 tokens from l = 8 to l = 256, whereas SONAR's suffix falls sharply beyond l = 32 even when its aggregate overlap remains competitive. This stable short-range reconstruction is the capacity needed by the chunking experiments.

4.2 Semantic geometry

Reconstruction asks a code to retain the tokens of its window, but does not directly place semantically similar inputs nearby. Four tasks from MTEB are probed: STSBenchmark and STS17, SICK-R, and NFCorpus. SONAR, which is trained with a similarity objective, leads on every task, often by a wide margin. Thus the raw code geometry is not strongly organized by semantic similarity.

4.3 Adaptive span allocation

The stopping criterion controls span allocation. A looser rule increases the mean chunk length. Four rules applied post hoc to one trained model on 500 Wikipedia documents span mean lengths of 6.50 to 12.17 input tokens. Over 2 million FineWeb documents, Failure(1) gives a mean chunk length of 9.56 tokens. Realized averages differ across Wikipedia, FineWeb, and downstream datasets, reflecting input-dependent variation in reconstruction reach.

The one caveat is the spike at length one, which has a mechanical cause. The backward decoder predicts the newest token first, with no reconstructed context to condition on, so that token is the hardest to get right. Two outcomes both produce a length-one chunk: the decoder misses this first token, and the forced-progress rule retains it anyway; or it reconstructs the first token but misses the second. Failure(1) thus piles both the stop-at-one and stop-at-two cases onto length one. These single-token chunks are common but carry only 4.3% of the text, so they cost latent positions without greatly changing the mean.

4.4 Native reconstruction under selected boundaries

After ReconSpan selects the boundaries, each resulting chunk is encoded in isolation and decoded autoregressively. Two conclusions follow. First, at an identical latent-token count and mean length, ReconSpan boundaries recover more of each document than the length-matched random control, so where the boundaries fall, not merely how many there are, decides how much text survives. Second, the stopping rule acts as a granularity dial. The results suggest that mean chunk length largely predicts quality across the two rule families: failure and logit-gap rules at similar mean lengths reach similar reconstruction quality.

Table 3 shows native reconstruction results on held-out WikiText. Failure(1) achieves mean chunk length 6.50, exact chunk fraction.797, exact token fraction.907, suffix fraction.944, and perplexity 15.95. The random matched control achieves exact chunk.794 but trails on all token-level measures: exact token.618, suffix.767, perplexity 20.03. Failure(2) and Logit-gap(3) reach mean lengths of 12.28 and 12.17 with lower exact-token fractions (.536 and.541).

5. Reading the latent tokens

Can downstream language models be trained to operate directly on ReconSpan's variable-length latent-token sequence, and what information can they recover?

5.1 Reader model and training

A separate language model, called a reader, is trained to predict text directly from the latent-token sequence rather than from the source text. For boundaries 0 = b0 <... < bm = n, the latent-token prefix at boundary bi is C≤i:= (cb1,..., cbi), cbj = E(x1:bj). The reader R maps this variable-length sequence of continuous codes to a distribution over output token sequences.

Each code is standardized coordinate-wise using corpus-level mean and standard deviation, then mapped to the reader's embedding dimension by a learned linear adapter and RMSNorm. The adapted codes are followed by a separator and the text output. During generic training, the reader receives a randomly truncated latent-token prefix formed with Failure(1); next-token loss is masked on the codes and separator and applied only to the continuation. Pythia-410M is fully fine-tuned, whereas Llama-3-8B updates LoRA weights, the adapter, and RMSNorm. Training uses generic FineWeb continuations.

Table 4 shows conditional perplexity of generated continuations on 150 held-out WikiText windows under Qwen2.5-1.5B. Human continuation: 11.95; native autoencoder decode: 15.95; Pythia-410M reader: 13.94; Llama-3-8B reader: 11.78. The Llama reader and human continuations are similar, while the smaller reader and native decode are moderately higher.

5.2 Information accessible to downstream readers

Three tasks test what the reader can recover along a spectrum of information specificity: AG News (coarse semantic information, four-way topic classification), LAMBADA (exact lexical information, final-word prediction), and HotpotQA (multi-hop retrieval from ten documents followed by open-answer generation). Two references are used: the shuffled control (codes from another example) and the round trip (decoded isolated chunk codes back to text).

Table 5 reports downstream readout results:

• LAMBADA: Pythia-410M reader: 5.1% (control 1.8%, text 51.0%, roundtrip 50.2%); Llama-3-8B reader: 10.0% (control 5.5%, text 75.8%, roundtrip 72.5%); Llama-3-8B + task FT: 20.1% (control 6.2%); Llama-3-8B Failure(2): 15.4% (control 13.5%, roundtrip 64.5%).

• AG News: Pythia-410M reader: 52.5% (control 26.4%, text 53.0%, roundtrip 52.8%); Llama-3-8B reader: 74.3% (control 24.5%, text 72.6%, roundtrip 73.0%); Llama-3-8B + task FT: 82.7% (control 24.2%); Llama-3-8B Failure(2): 74.6% (control 25.6%, roundtrip 72.0%).

• HotpotQA: Pythia-410M reader: 0.0% (control 1.0%, text 7.0%, roundtrip 6.0%); Llama-3-8B reader: 17.0% (control 18.0%, text 36.0%, roundtrip 33.0%); Llama-3-8B + task FT: 30.0% (control 23.0%); Llama-3-8B Failure(2): 17.0% (control 18.0%, roundtrip 27.0%).

Across the three tasks, readers access coarse semantics more readily than exact details. The Llama reader reaches raw-text-level topic classification on AG News, whereas LAMBADA remains far below the round-trip reference. Generic readout is also weak on HotpotQA and does not improve over the shuffled control. Task adaptation improves all three tasks and moves HotpotQA toward the round-trip reference. Although the readers are trained only with Failure(1) codes, they do not collapse when evaluated with the coarser Failure(2) policy.

Overall, the native route retains information that current readers do not fully access. Topic information is readily recoverable by a downstream language model, task supervision narrows the access gap, and exact-detail readout remains the central limitation.

6. Limitations and future directions

6.1 Limitations

• Reader access: Direct-code readers remain weak on exact-content and retrieval tasks, even when scaling the reader or adapting it to the target task improves performance. By contrast, the native round trip answers the same tasks much better. One hypothesis is that code geometry makes this access difficult: the autoencoder is trained for reconstruction rather than to place semantically similar inputs nearby, and its weak MTEB results are consistent with this explanation but do not establish it as a cause.

• Short chunks: Too much of the latent sequence is spent on very short chunks. Under Failure(1), 28.1% of chunks cover a single token. These chunks carry only 4.3% of the text while consuming 28.1% of the codes. Relaxing the rule to Failure(2) reduces them to 4.1%.

• Backward-decoder pretraining: ReconSpan requires a model that decodes text backward from a code. Such pretrained decoders are not readily available, so our decoder must be trained from scratch before the autoencoder can be jointly optimized.

• Boundary-selection cost: The reconstruction scan is computationally expensive at corpus scale. Constructing the reader corpus took about 12 hours for 2M documents, roughly two to three times the 4–5 hours of reader training it fed. Fewer latent positions also do not by themselves establish lower end-to-end compute: the Transformer encoder has quadratic work in source length.

• Scope of the tokenizer: ReconSpan operates over an existing subword sequence and is therefore a higher-level latent tokenizer, not a replacement for byte-to-text tokenization. We do not compare downstream language-model quality or efficiency directly with end-to-end byte tokenizers such as BLT or H-Net.

6.2 Future directions

• Readers trained at larger scale, on better-matched data, or with objectives aimed at exact retrieval may recover more of the information demonstrated by the native round trip.

• Training the autoencoder with an additional semantic-clustering objective could make related inputs easier for a reader to recognize.

• A different native decoder could replace the backward language model, e.g., a Transformer decoder receiving the code through a soft prompt or cross-attention.

• The linguistic structure of the boundaries placed by ReconSpan remains unexplored. Testing whether they align with syntax, discourse, or information density could explain what the reconstruction criterion treats as difficult.

• A further direction is to evaluate ReconSpan as a context-compression interface, measuring prefill latency, KV-cache memory, boundary-selection cost, and amortization across repeated reads rather than inferring efficiency from sequence reduction alone.

7. Conclusion

ReconSpan forms adaptive latent tokens by retaining contextual encoder states at boundaries set by backward reconstruction reach. One autoencoder exposes mean chunk lengths from 6.50 to 12.17 after training, and its boundaries reconstruct better than length-matched random ones. Downstream language models operate directly on these sequences, recovering topics more readily than exact lexical details; task adaptation narrows the native-reconstruction gap. These results establish reconstruction fidelity as a viable allocation criterion and motivate stronger readers and linguistic boundary analysis.

Improvements to AI systems based on ReconSpan:

1. Adaptive context compression for long-context LLMs: Implement ReconSpan's reconstruction-guided chunking as a preprocessing layer for transformer-based LLMs. This allows the model to dynamically allocate more latent tokens to information-dense or hard-to-predict spans (e.g., rare names, code, formulas) and fewer to redundant text, reducing sequence length by 6.5–12.2× while preserving critical content. The improved system can process longer documents (e.g., entire books or multi-hour transcripts) within fixed context windows, with better retention of exact details in difficult regions compared to fixed-stride or random compression.

2. Post-training granularity control without retraining: Use the stopping-rule dial (Failure(m) or Logit-gap(τ)) to adjust compression ratio on a deployed model in real time. The improved system can switch between high-fidelity mode (mean chunk length 6.5, exact token recovery 90.7%) for tasks requiring precise extraction (e.g., legal contracts, medical records) and high-compression mode (mean chunk length 12.2) for tasks needing only gist (e.g., summarization, topic routing), all without fine-tuning or changing model weights.

3. Backward-decoder-based error detection for retrieval-augmented generation (RAG): Train a backward decoder on the same corpus as the retriever. When a retrieved passage is encoded, the decoder's reconstruction reach indicates which parts are reliably encoded. The improved RAG system can filter out passages or spans where reconstruction fails, reducing hallucination from poorly-encoded context and improving answer accuracy on multi-hop questions (HotpotQA) by up to 30% with task adaptation.

4. Semantic-geometry enhancement for latent-token readability: Augment the autoencoder training with a contrastive or similarity objective (e.g., pulling codes of semantically similar spans closer) alongside reconstruction loss. The improved system produces latent tokens that are both reconstructable and semantically organized, enabling downstream readers to achieve near-text-level performance on topic classification (AG News) and substantially better exact-detail recovery (LAMBADA) without task-specific fine-tuning.

5. Two-stage reader training for exact-detail access: Train a reader first on native round-trip decoded text (from isolated chunk codes) to learn the mapping from codes to surface forms, then fine-tune on raw text. The improved system can bridge the gap between information retained by the autoencoder and information accessible to the reader, potentially recovering 50–70% more exact lexical details (as measured by LAMBADA) compared to direct-code readers.

6. Hierarchical latent tokenization for byte-level inputs: Extend ReconSpan to operate on byte sequences (instead of subword tokens) using a hierarchical scheme: first chunk bytes into subword-like units via reconstruction, then chunk those units into latent tokens. The improved system can replace fixed byte-pair encoding with input-dependent tokenization, reducing sequence length by 8–15× while maintaining exact reconstruction of rare or out-of-vocabulary strings, and enabling efficient processing of multilingual or code-switched text.

7. Latent-token-based memory for recurrent agents: Use ReconSpan's chunk-boundary codes as fixed-size memory states in an agent loop (e.g., for web navigation or game playing). Each code summarizes a variable-length action-observation span. The improved agent can compress its history into 6–12× fewer memory slots, allowing longer-horizon planning within a fixed memory budget, while reconstruction-guided boundaries ensure critical events (e.g., errors, rewards) are not lost.

8. Corpus-level difficulty profiling for curriculum learning: Use the distribution of reconstruction reach (chunk lengths) across documents as a difficulty metric. The improved training system can automatically order training data from easy (long chunks, low reconstruction error) to hard (short chunks, high error), accelerating convergence for language models by focusing early training on predictable text and later on complex, information-dense spans.

Adaptive latent tokenization maps a fine-grained input to a shorter sequence of continuous representations associated with input-dependent spans. We introduce ReconSpan, which divides text into chunks that a backward decoder can reconstruct from a single contextual prefix code and retains one such code as the latent token for each chunk. The reconstruction criterion is applied when chunks are formed, allowing one trained autoencoder to produce average chunk lengths from 6.5 to 12.2. At matched average length, reconstruction-guided boundaries preserve more text than random boundaries. Readers of the resulting latent sequence recover topic information reliably but struggle to extract exact details.