A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioning

arXiv:2606.17364 · math.ST, math.OC, stat.ML · Submitted 2026-06-15 · Read on arXiv

Sunyoung An, Xiaoming Huo

math.ST, math.OC, stat.ML

Submitted: 2026-06-15

Comments: 44 pages, 3 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Adam — 107× in this paper · explained in 2 episodes
• Verification — 10× in this paper · explained in 2 episodes
• Spectral Gap — 3× in this paper · explained in 5 episodes
• Adaptivity — 2× in this paper · explained in 1 episode
• Conference on Learning Theory — 2× in this paper
• Lyapunov function — 2× in this paper · explained in 2 episodes
• Regularization — 2× in this paper · explained in 3 episodes
• Stochastic Gradient Descent (SGD) — 2× in this paper · explained in 5 episodes
• Adaptive Mechanism — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Degeneracy — 1× in this paper · explained in 3 episodes
• Monte Carlo — 1× in this paper
• Non-asymptotic Analysis — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• When Does Dynamic Preconditioning Preserve the Polyak-Ruppert CLT? A Stabilization Threshold
• Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise
• SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration
• Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality
• Inference of Online Newton Methods with Nesterov's Accelerated Sketching
• Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality

Related papers

• Conformal Prediction for Dyadic Regression Under Complex Missingness
• Bentkus-type asymptotic e-values
• High-Dimensional Asymptotics of Differentially Private PCA
• KL Convergence Guarantees for Score diffusion models under minimal data assumptions
• Geometric bias in eigenspace perturbation under random heterogeneous noise
• On the Asymptotic Inadmissibility of Double Machine Learning Estimators Under Structure-Agnostic Models