LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Arman Bolatov, Artem Riabinin, Nikita Kornilov, Andrey Veprikov, Samuel Horváth, Martin Takáč, Aleksandr Beznosikov
cs.LG
Submitted: 2026-08-19
Updated: 2026-08-20
Code: https://github.com/brain-lab-research/lion-muon
Terminology
Sources
- Training Compute-Optimal Large Language Models
- Kimi K2: Open Agentic Intelligence
- Benchmarking Neural Network Training Algorithms
- Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
- Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
- HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
- Dion: Distributed Orthonormalized Updates
- LiMuon: Light and Fast Muon Optimizer for Large Models
- OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Improving Generalization Performance by Switching from Adam to SGD
- On the Convergence Analysis of Muon
- Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization
- Muon Converges under Heavy-Tailed Noise: Nonconvex H\"{o}lder-Smooth Empirical Risk Minimization
- Benchmarking Optimizers for Large Language Model Pretraining
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks