Towards a Statistical Understanding of Mixture-of-Experts

arXiv:2609.03501 · stat.ML, cs.LG, math.ST, stat.TH · Submitted 2026-09-03 · Read on arXiv

stat.ML, cs.LG, math.ST, stat.TH

Submitted: 2026-09-03

Updated: 2026-09-03

Terminology

• Aggregation — 65× in this paper · explained in 1 episode
• Calibration — 13× in this paper · explained in 7 episodes
• Benchmark — 12× in this paper · explained in 2 episodes
• Accuracy — 11× in this paper · explained in 1 episode
• Alignment — 7× in this paper · explained in 1 episode
• Sparsity — 7× in this paper · explained in 4 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• Approximation Bound — 5× in this paper · explained in 1 episode
• Mixture of Experts — 3× in this paper · explained in 2 episodes
• RAG — 3× in this paper · explained in 3 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Adaptive Selection — 2× in this paper · explained in 1 episode
• Consistency — 2× in this paper · explained in 3 episodes
• Geometric Mismatch — 2× in this paper · explained in 2 episodes
• Mixture-of-Experts (MoE) — 2× in this paper · explained in 7 episodes
• Residualization — 2× in this paper · explained in 2 episodes
• Adaptive Partitioning — 1× in this paper · explained in 1 episode
• Architectural Sparsity — 1× in this paper · explained in 1 episode
• Communication Overhead — 1× in this paper · explained in 2 episodes
• Degeneracy — 1× in this paper · explained in 3 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes

Sources

• DeepSeek-V3 Technical Report
• Combining pre-trained models via localized model averaging
• Mixture of A Million Experts
• Mixtral of Experts
• Kimi K3: Open Frontier Intelligence
• On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
• GPT-4 Technical Report
• Route Experts by Sequence, not by Token
• When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
• Toward Inference-optimal Mixture-of-Expert Large Language Models

Related papers

• Behavior of prediction performance metrics with rare events
• Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
• A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
• One Permutation Is All You Need: Fast, Deterministic Feature Importance and Model Stress-Testing
• Online Conformal Prediction for Non-Exchangeable Panel Data
• Deep Time-Series Forecasting in 10 Years: A Survey