Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

arXiv:2505.09738 · cs.CL, cs.AI · Submitted 2025-05-14 · Read on arXiv

cs.CL, cs.AI

Submitted: 2025-05-14

Updated: 2026-09-24

Terminology

• Tokenization — 21× in this paper · explained in 2 episodes
• Embeddings — 20× in this paper · explained in 2 episodes
• Adaptation — 14× in this paper · explained in 1 episode
• Perplexity — 12× in this paper · explained in 3 episodes
• Core — 10× in this paper · explained in 2 episodes
• Large Language Model — 5× in this paper
• Alignment — 4× in this paper · explained in 1 episode
• Fidelity — 4× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Model-Agnostic — 4× in this paper · explained in 2 episodes
• Hypernetwork — 3× in this paper · explained in 3 episodes
• Cross-lingual Transfer — 2× in this paper · explained in 2 episodes
• Hybrid Approach — 2× in this paper · explained in 4 episodes
• Semantic Similarity — 2× in this paper · explained in 2 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Continued Pre-Training — 1× in this paper
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Language-Adaptive Pre-Training — 1× in this paper
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes
• Zero-Shot Transfer — 1× in this paper · explained in 4 episodes

Sources

• Entanglement generation in capacitively coupled Transmon-cavity system
• Tamil-Llama: A New Tamil Language Model Based on Llama 2
• ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
• An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
• Airavata: Introducing Hindi Instruction-tuned LLM
• Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
• SuperBPE: Space Travel for Language Models
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
• Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
• The ultraspherical rectangular collocation method and its convergence
• Attention Is All You Need
• Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering