Counting and Min-Cost Encoding for Tokenization in Large Language Models

arXiv:2610.01127 · cs.CL · Submitted 2026-10-01 · Read on arXiv

cs.CL

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/openai/tiktoken

Terminology

• Tokenization — 26× in this paper · explained in 2 episodes
• Benchmark — 16× in this paper · explained in 2 episodes
• Min-Cost Encoding — 7× in this paper
• Alignment — 4× in this paper · explained in 1 episode
• Accuracy — 3× in this paper · explained in 1 episode
• Byte Pair Encoding — 3× in this paper
• Consistency — 3× in this paper · explained in 3 episodes
• Fidelity — 3× in this paper · explained in 2 episodes
• Pruning — 3× in this paper · explained in 4 episodes
• Iterative Refinement — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Massive Multitask Language Understanding — 2× in this paper
• Perplexity — 2× in this paper · explained in 3 episodes
• Aggregation — 1× in this paper · explained in 1 episode
• Contextual Understanding — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Physical Interaction Question Answering — 1× in this paper
• Regularization — 1× in this paper · explained in 3 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
• Evaluating Large Language Models Trained on Code
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Training Verifiers to Solve Math Word Problems
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Measuring Massive Multitask Language Understanding
• MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
• SuperBPE: Space Travel for Language Models
• CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models
• StarCoder 2 and The Stack v2: The Next Generation
• Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
• TokEval: A Tokenizer Evaluation Suite
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
• No Language Left Behind: Scaling Human-Centered Machine Translation
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
• Mangosteen: An Open Thai Corpus for Language Model Pretraining
• Qwen2.5 Technical Report
• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
• Olmo 3

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering