Second-Moment Stochastic Approximation Methods

arXiv:2609.36600 · math.OC, cs.AI, stat.ML · Submitted 2026-09-29 · Read on arXiv

math.OC, cs.AI, stat.ML

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• Adam — 28× in this paper · explained in 2 episodes
• Regularization — 6× in this paper · explained in 3 episodes
• Generalization — 4× in this paper · explained in 13 episodes
• Loss Function — 3× in this paper · explained in 2 episodes
• Two-Stage Framework — 3× in this paper · explained in 2 episodes
• Adaptive Structured Gradient Optimization — 2× in this paper
• Alignment — 2× in this paper · explained in 1 episode
• Embeddings — 2× in this paper · explained in 2 episodes
• Additivity — 1× in this paper · explained in 1 episode
• Exponential Moving Average (EMA) — 1× in this paper · explained in 2 episodes
• Lyapunov function — 1× in this paper · explained in 2 episodes
• Proceedings of International Conference on Learning Representations — 1× in this paper
• Trace — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• Adam: A Method for Stochastic Optimization
• ASGO: Adaptive Structured Gradient Optimization
• Adam-mini: Use Fewer Learning Rates To Gain More
• The AdEMAMix Optimizer: Better, Faster, Older
• Quasi-hyperbolic momentum and Adam for deep learning
• MARS: Unleashing the Power of Variance Reduction for Training Large Models
• Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization
• Stochastic Approximation with Block Coordinate Optimal Stepsizes
• Muon is Scalable for LLM Training
• The Llama 3 Herd of Models
• An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
• Why Gradients Rapidly Increase Near the End of Training
• Scale Weight Decay and Train Better
• The Newton-Muon Optimizer

Related papers

• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Incremental Learning in Mirror Flows
• Online Control via Counterfactual Tracking
• Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability
• Petrov-Galerkin operator inference with application to stability-encouraging identification