Convergence guarantees for Muon: New parameter regimes and generalizations

arXiv:2609.30546 · math.NA, cs.AI, cs.NA, math.OC · Submitted 2026-09-24 · Read on arXiv

math.NA, cs.AI, cs.NA, math.OC

Submitted: 2026-09-24

Updated: 2026-09-24

Code: https://github.com/kellerjordan/modded-nanogpt

Terminology

• Regularization — 21× in this paper · explained in 3 episodes
• Convergence Guarantees — 13× in this paper · explained in 3 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• Lyapunov function — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Benchmark — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Transformer Models — 2× in this paper · explained in 2 episodes
• Singular Value Decomposition (SVD) — 1× in this paper · explained in 2 episodes

Sources

• The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
• Modular Duality in Deep Learning
• On the Convergence of Muon and Beyond
• Convergence of Muon with Newton-Schulz
• A Note on the Convergence of Muon
• Muon is Scalable for LLM Training
• Convergence Bound and Critical Batch Size of Muon Optimizer
• On the Convergence Analysis of Muon

Related papers

• Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm
• A Neural-preconditioned Poisson Solver for Mixed Dirichlet and Neumann Boundary Conditions
• Second-order consistency for learning chaotic dynamics via randomized Jacobian matching
• Windowed thinning and query complexity for the bouncy particle and Zigzag samplers
• Data-efficient Kernel Methods for Learning Hamiltonian Systems
• Adjoint Method versus Physics-Informed Neural Networks in PDE-Constrained Inverse Problems