Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

arXiv:2510.25060 · math.OC, cs.LG, math.DS · Submitted 2025-10-29 · Read on arXiv

math.OC, cs.LG, math.DS

Submitted: 2025-10-29

Updated: 2026-09-01

Terminology

• Loss Function — 12× in this paper · explained in 2 episodes
• Degeneracy — 3× in this paper · explained in 3 episodes
• alpha — 3× in this paper · explained in 1 episode
• Additivity — 2× in this paper · explained in 1 episode
• Symmetry Breaking — 2× in this paper · explained in 3 episodes
• Activation Function — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Hysteresis — 1× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Global Bifurcation In Four-Component Bose-Einstein Condensates In Space

Related papers

• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Incremental Learning in Mirror Flows
• Online Control via Counterfactual Tracking
• Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability
• Petrov-Galerkin operator inference with application to stability-encouraging identification