Exact information accounting for SGD methods
cs.LG, cs.IT, math.IT, math.OC, stat.ML
Submitted: 2026-09-30
Updated: 2026-09-30
Terminology
Sources
- Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequences
- Beyond Optimal Rates in Stochastic Optimization: Trajectory-Adaptive Stopping Rules
- Adaptive Bayes exactly tracks information over intrinsic time
- Potential-Function Proofs for First-Order Methods
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It
- The CMA Evolution Strategy: A Tutorial
- Adam: A Method for Stochastic Optimization
- The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives
- Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Flat Minima and Generalization: Insights from Stochastic Convex Optimization
- On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
- Cyclical Learning Rates for Training Neural Networks
- Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
- Less Regret via Online Conditioning
- SOAP: Improving and Stabilizing Shampoo using Adam
- Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks