Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
summary
The gist
The gist The proposed framework establishes a reusable, cost-efficient experimental paradigm for systematically studying MoE architectural properties at scale by employing a mathematical decoupling
In short
The framework optimizes Mixture-of-Experts (MoE) architectures by systematically studying their properties at scale. It addresses the inadequacy of using only FLOPs per token by employing a mathematical decoupling strategy to reduce the complex search space from O(n^16) to a manageable two-phase process of O(n^3) and O(n^2). This provides a reusable, cost-efficient method for finding optimal MoE designs.
Key concepts
- FLOPs per token (M)
- This metric measures the computational work done for each token processed by the model. The paper argues that relying solely on M is flawed because different layers (like Attention vs. FFN) have different computational densities, leading to misleading performance gains when only M is considered.
- Constraint Triad
- This involves simultaneously constraining the architectural search space using three key parameters: FLOPs per token (M), active parameters (Na), and total parameters (N). This triad is necessary because MoE models can inflate N and Na while keeping M constant, introducing bias if only M is used for optimization.
- Dimension Decomposition
- This strategy transforms the extremely large search space into a manageable problem by mathematically decoupling it. It locks macroscopic parameters (M, Na, N) first and then uses rank-preserving properties to factorize the global optimization into two sequential phases: one O(n^3) phase and one O(n^2) phase.
- Two-Phase Search Strategy
- The optimization is split into two distinct stages. Phase 1 evaluates optimal scaling behaviors for total parameters (Nopt_a(M)) using a proxy search costing O(n^3). Phase 2 then determines the optimal scaling law for the hidden dimension (d) given fixed constraints, requiring only an additional O(n^2) runs.
Terminology used across episodes
This episode discusses
- Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization · Paper Radio
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
- GPT-4 Technical Report
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Scaling Laws for Predicting Downstream Performance in LLMs
- Language models scale reliably with over-training and on downstream tasks
- Scaling Laws for Autoregressive Generative Modeling
- Deep Learning Scaling is Predictable, Empirically
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- dots.llm1 Technical Report
- OpenAI o1 System Card
- Mixture of Parrots: Experts improve memorization more than reasoning
- Scaling Laws for Neural Language Models
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- DeepSeek-V3 Technical Report
- Decoupled Weight Decay Regularization
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
- OLMoE: Open Mixture-of-Experts Language Models
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- GLU Variants Improve Transformer
The paper
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization · Read on arXiv
Weilin Wan, Jingtao Han, Weizhong Zhang, *Cheng Jin
College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · School of Data Science, Fudan University
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization".
Jane: The gist The proposed framework establishes a reusable,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So, looking at "Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization," the authors point out that just relying on FLOPs per token isn't enough because of how the attention and feed forward networks work together.
Jane: They argue that you get misleading performance gains from parameter inflation when you only look at M, which means you need to consider other things too.
Lu: To fix this, they use a mathematical decoupling strategy that takes into account inherent structural constraints and a property of the hidden dimension called rank preservation to factorize the big optimization problem.
Meng: It seems like they are trying to move away from just fitting scaling laws directly across all these variables because that leads to a massive experimental cost of O(n sixteen) <ref:2603.21862#pg3,fitting scaling laws directly across>.
Lalam: They propose locking in some main parameters first and then tackling the rest in stages, which is a clever way to manage that complexity.
Conclusion: Tom: So, looking at "Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization," the authors are really pushing for a general-purpose framework that can find the complete optimal allocation of M, D, Na, N, and d from just a given compute budget C.
Jane: What this means in simpler terms is they've created a deterministic pipeline that takes your budget and gives you the best possible MoE architecture directly.
Lu: It provides a way to get the optimal Mopt and Dopt through these derived scaling laws, which is something we needed for better structural optimization.
Meng: The implication for engineers is that they can now use this framework to make real infrastructure decisions based on compute budget instead of just following some heuristic recommendations.
Lalam: It helps improve culture by giving us a consistent way to evaluate different architectural choices fairly across a wide range of scales.
More episodes
- 2512.03974-Refining Machine Learning Potentials through Thermodynamic Theory of Phase Transitions
- 2512.03923-Quantum-Classical Physics-Informed Neural Networks for Solving Reservoir Seepage Equations
- 2601.07674-Self-Creating Random Walks for Decentralized Learning under Pac-Man Attacks
- 2602.03945-Grables: When Graph Structure Helps Tabular Prediction
- 2601.22371-FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models
- 2602.00872-Learning Heat-based Equations in Self-similar variables
- 2602.01581-Towards Optimality in Active Preference Learning
- 2602.06138-Flow Matching for Offline Reinforcement Learning with Discrete Actions
- 2603.03172-Less Noise, Same Certificate: Retain Sensitivity for Unlearning
- 2602.07496-A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning