The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
cs.LG, math.OC, stat.ML
Submitted: 2026-09-01
Updated: 2026-09-01
Terminology
Sources
- Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network
- Gradient Descent Happens in a Tiny Subspace
- The Origin of Edge of Stability
- FOCUS: First Order Concentrated Updating Scheme
- Neural Thermodynamic Laws for Large Language Model Training
- Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes
- Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
- Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability
- Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- A Walk with SGD
- Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks