On the O(sqrt d over K 1/4) Convergence Rate of AdamW Measured by 1 Norm
cs.LG, math.OC
Submitted: 2025-05-17
Updated: 2026-09-27
Code: https://github.com/adonis-dym/Convergence-Rate-AdamW
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- A Novel Convergence Analysis for Algorithms of the Adam Family
- High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise
- Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization
- On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm
- AdaGrad under Anisotropic Smoothness
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Reevaluating Theoretical Analysis Methods for Optimization in Deep Learning
- On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
- Fantastic Pretraining Optimizers and Where to Find Them
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks