Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness
cs.LG, cs.SY, eess.SY
Submitted: 2026-08-12
Updated: 2026-08-12
Terminology
Sources
- A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
- A Simple Convergence Proof of Adam and Adagrad
- Can SGD Handle Heavy-Tailed Noise?
- Revisiting Convergence of AdaGrad with Relaxed Assumptions
- From Gradient Clipping to Normalization for Heavy Tailed SGD
- High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize
- Adam: A Method for Stochastic Optimization
- SP2: A Second Order Stochastic Polyak Method
- A High Probability Analysis of Adaptive SGD with Momentum
- Enhancing Policy Gradient with the Polyak Step-Size Adaption
- AdaGrad under Anisotropic Smoothness
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- On the Convergence of Adam and Beyond
- MoMo: Momentum Models for Adaptive Learning Rates
- AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods
- Less Regret via Online Conditioning
- On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
- Mirror Descent Under Generalized Smoothness
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks