Sharp First-Order Lower Bounds for Higher-Order Smooth Nonconvex Optimization
cs.LG, math.OC
Submitted: 2026-06-03
Updated: 2026-09-21
Comments: 24 pages, 1 table. Version 2 adds explicit numerical constants to all theorems
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: We study the deterministic first-order oracle complexity of finding ε-stationary points in smooth nonconvex optimization when the objective satisfies higher-order smoothness assumptions.
Terminology
Abstract
We study the deterministic first-order oracle complexity of finding ε-stationary points in smooth nonconvex optimization when the objective satisfies higher-order smoothness assumptions. While the classical ε-2 rate is optimal under only Lipschitz gradients, higher-order smoothness leads to accelerated first-order upper bounds, most notably the ε-7/4 rate under Lipschitz Hessians and the ε-5/3 rate under Lipschitz third derivatives. The matching lower bounds, however, have remained open. We resolve this gap by proving a new dimension-free first-order lower bound for higher-order smooth nonconvex functions, valid for every finite smoothness order. In particular, our construction gives a matching Ω(ε-7/4) lower bound in the Hessian-Lipschitz case and a matching Ω(ε-5/3) lower bound in the third-order-smooth regime. The hard instance is based on a block-chain mechanism that enforces blockwise oracle revelation while preserving the smoothness structure needed for the scalar hard instance. The lower-bound construction was discovered with the assistance of ChatGPT 5.5 Pro and subsequently verified by the authors.
Sources
- Worst-case evaluation complexity and optimality of second-order methods for nonconvex smooth optimization
- On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
- NEON+: Accelerated Gradient Methods for Extracting Negative Curvature for Non-Convex Optimization
- Finding Local Minima via Stochastic Nested Variance Reduction
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks