MDL-Calibrated Significance-Gain Pair Encoding: Replication-Aware Automatic Stopping for Subword Tokenization

arXiv:2609.31705 · cs.CV, cs.CL · Submitted 2026-09-20 · Read on arXiv

cs.CV, cs.CL

Submitted: 2026-09-20

Updated: 2026-09-20

Terminology

• Tokenization — 7× in this paper · explained in 2 episodes
• Byte-Pair Encoding — 4× in this paper
• Minimum Description Length — 4× in this paper
• Perplexity — 2× in this paper · explained in 3 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• Pointer Sentinel Mixture Models
• An MLP Baseline for Handwriting Recognition Using Planar Curvature and Gradient Orientation
• Critical Exponent Elliptic Equations on the Half-Space: Uniqueness and Explicit Solutions
• A Sobel-Gradient MLP Baseline for Handwritten Character Recognition
• Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging
• StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models