Frequency-Ordered Tokenization for Better Text Compression

arXiv:2602.22958 · cs.IT, cs.CL, math.IT · Submitted 2026-02-26 · Read on arXiv

cs.IT, cs.CL, math.IT

Submitted: 2026-02-26

Updated: 2026-10-10

Terminology

• Tokenization — 2× in this paper · explained in 2 episodes

Related papers

• Language Modeling is Monotone Compression
• Clipped Affine Policy: Low-Complexity Near-Optimal Online Power Control for Energy Harvesting Communications over Fading Channels
• Discrepancy for Random Linear Codes
• A New Approach to Code Smoothing Bounds
• Contextual Memory-Enhanced Source Coding for Low-SNR Communications
• Symmetry-Enforced Quadratic Approximate-Degradability Bounds for Noisy Landau-Streater Channels