How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
cs.LG
Submitted: 2026-09-16
Updated: 2026-09-17
Comments: 44 pages. Code: https://github.com/qlabs-eng/scaling-exponents
Code: https://github.com/qlabs-eng/scaling-exponents
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Attention Residuals
- Net2Net: Accelerating Learning via Knowledge Transfer
- The Llama 3 Herd of Models
- Deep Learning Scaling is Predictable, Empirically
- Less is More: Recursive Reasoning with Tiny Networks
- Scaling Laws for Neural Language Models
- Scaling Laws for Fine-Grained Mixture of Experts
- Reusing Overtrained Language Models Saturates Scaling
- Muon is Scalable for LLM Training
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Prescriptive Scaling Laws for Data Constrained Training
- Parcae: Scaling Laws For Stable Looped Language Models
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
- Will we run out of data? Limits of LLM scaling based on human-generated data
- HRM-Text: Efficient Pretraining Beyond Scaling
- Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
- SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks