From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

arXiv:2512.13491 · cs.IT, cs.LG, math.IT, math.ST, stat.TH · Submitted 2025-12-15 · Read on arXiv

cs.IT, cs.LG, math.IT, math.ST, stat.TH

Submitted: 2025-12-15

Updated: 2026-06-22

Comments: 32 pages, no figures

Journal ref: Journal of Machine Learning Research, vol. 27, no. 194, pp. 1-35, 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Scaling Laws — 18× in this paper · explained in 4 episodes
• Foundation Model — 12× in this paper · explained in 3 episodes
• Foundation Models — 6× in this paper · explained in 7 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Embeddings — 2× in this paper · explained in 2 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Exchangeability — 1× in this paper · explained in 2 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• Explaining Neural Scaling Laws
• On the origin of neural scaling laws: from random graphs to natural language
• Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
• Scaling Laws for Associative Memories
• Deriving Neural Scaling Laws from the statistics of natural language
• Heaps' law and Heaps functions in tagged texts: Evidences of their linguistic relevance
• PaLM: Scaling Language Modeling with Pathways
• Multiperiodic Processes: Ergodic Sources with a Sublinear Entropy
• How Linguistics Learned to Stop Worrying and Love the Language Models
• Scaling Laws for Autoregressive Generative Modeling
• Scaling Laws for Transfer
• Deep Learning Scaling is Predictable, Empirically
• Training Compute-Optimal Large Language Models
• Learning Curve Theory
• Scaling Laws for Neural Language Models
• Heaps' Law in GPT-Neo Large Language Model Emulated Corpora
• Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
• Superposition Yields Robust Neural Scaling
• A Solvable Model of Neural Scaling Laws
• The Quantization Model of Neural Scaling

Related papers

• Clipped Affine Policy: Low-Complexity Near-Optimal Online Power Control for Energy Harvesting Communications over Fading Channels
• Discrepancy for Random Linear Codes
• A New Approach to Code Smoothing Bounds
• Contextual Memory-Enhanced Source Coding for Low-SNR Communications
• Symmetry-Enforced Quadratic Approximate-Degradability Bounds for Noisy Landau-Streater Channels
• Anonymous Shamir's Secret Sharing via Reed-Solomon Codes Against Permutations, Insertions, and Deletions