On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

arXiv:2608.04607 · math.OC, cs.LG · Submitted 2026-08-05 · Read on arXiv

Thang Do, Steffen Dereich, Arnulf Jentzen

math.OC, cs.LG

Submitted: 2026-08-05

Comments: 82 pages

Code: https://github.com/KellerJordan/modded-nanogpt

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Adam — 35× in this paper · explained in 2 episodes
• Objective Function — 8× in this paper · explained in 2 episodes
• Artificial Intelligence (AI) — 3× in this paper · explained in 2 episodes
• Stochastic Gradient Descent (SGD) — 3× in this paper · explained in 5 episodes
• Core — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Generalized Newton-Schulz — 2× in this paper
• Hilbert Space — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Loss Function — 2× in this paper · explained in 2 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Society for Industrial and Applied Mathematics — 1× in this paper
• Trace — 1× in this paper · explained in 2 episodes

Sources

• The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
• Old Optimizer, New Norm: An Anthology
• signSGD: Compressed Optimisation for Non-Convex Problems
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• On the Convergence of Muon and Beyond
• Muon Optimizes Under Spectral Norm Constraints
• Robustness to Unbounded Smoothness of Generalized SignSGD
• Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
• Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates
• Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods
• Convergence rates for the Adam optimizer
• ODE approximation for the Adam algorithm: General and overparametrized setting
• Sharp higher order convergence rates for the Adam optimizer
• Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks
• Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks
• Handbook of Convergence Theorems for (Stochastic) Gradient Methods
• Insights on Muon from Simple Quadratics
• Convergence of Muon with Newton-Schulz
• On the Convergence of Adam, Revisited
• Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

Related papers

• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Incremental Learning in Mirror Flows
• Online Control via Counterfactual Tracking
• Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability
• Petrov-Galerkin operator inference with application to stability-encouraging identification