SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
cs.LG
Submitted: 2026-09-01
Updated: 2026-09-11
Comments: 35 pages, 25 figures
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
- Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- AI capabilities can be significantly improved without expensive retraining
- Loop the Loopies!
- Measuring the Algorithmic Efficiency of Neural Networks
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- From Growing to Looping: A Unified View of Iterative Computation in LLMs
- Scaling Laws for Neural Language Models
- Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
- The Remarkable Robustness of LLMs: Stages of Inference?
- Sparse Layers are Critical to Scaling Looped Language Models
- Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- Parcae: Scaling Laws For Stable Looped Language Models
- How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
- Massive Activations in Large Language Models
- On the Residual Scaling of Looped Transformers: Stability and Transferability
- Scaling Latent Reasoning via Looped Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks