Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates

arXiv:2607.07623 · cs.LG, cs.NA, math.NA, physics.chem-ph, physics.comp-ph, physics.data-an · Submitted 2026-07-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates".

Jane: Nonlinear least-squares optimization is central to many computational problems,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, let's talk about the title of this paper, "Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates." It sounds very technical, but the core idea is about making our optimization steps more geometrically accurate.

Jane: Exactly. The authors are focusing on how to move past the straight-line updates that standard Levenberg–Marquardt does by incorporating higher-order corrections using something called Riemann normal coordinates.

Lu: The authors explain that standard LM just takes a tangent-space direction, but this direction doesn't stay consistent over finite steps because the way we change parameters affects the geometry of the problem itself.

Meng: So, if it’s not following the true geometry of what we’re optimizing, then our model might be taking steps that look good locally but lead us astray globally. That's a practical concern for deployment.

Lalam: It really speaks to how we structure our learning processes; instead of just chasing the gradient in a flat coordinate system, this method tries to follow the actual path dictated by the physics or data manifold.

Tom: And that’s what they aim to solve with RNC-LM: extending second-order geodesic acceleration to arbitrary orders so that we get updates that are consistently accurate even when we take finite steps.

The paper's summary: Jane: To summarize the core of "Higher-Order Geometric Updates for Levenberg–Marquardt Method via Riemann Normal Coordinates," the authors are reformulating the geodesic equation into a first-kind residual condition along a specific trajectory.

Lu: Instead of just using coordinates, they define these Riemann normal coordinates where the path is expanded as a finite-order series starting with theta mu(t) = theta p + t v mu + t two/two c mu squared +.

Tom: The main goal here is to recursively determine those higher-order coefficients, c two c three and so on, ensuring the resulting update curve adheres to the geodesic condition of their damped metric G mu nu = J mu alpha J nu beta + lambda D mu nu.

Meng: They are essentially building a correction term that ensures the "curve itself" is corrected to represent the finite-step geometry accurately, which sounds like a lot of bookkeeping for an optimizer.

Lalam: It’s about making sure that when we move from point A to point B in our parameter space, we aren't just taking a straight line, but rather following the natural curvature imposed by the model structure itself.

Jane: So they take the geodesic equation and turn it into a residual condition A(t):= J(theta(t)) (t) + lambda (t) = zero and enforce that its Taylor coefficients vanish at zero to find those higher-order corrections.

Tom: That’s the mathematical engine driving the entire method, which lets them reuse the same left-hand matrix G from standard LM, only needing extra right-hand sides generated by automatic differentiation along a one-dimensional trial curve.

The paper's improvements: Lu: The key improvement they suggest is that RNC-LM achieves finite-step updates with progressively higher reparameterization consistency, which means the accuracy of the path we take doesn't degrade as our step size increases.

Tom: They’re moving beyond just second-order corrections to handle arbitrary orders, which is a big deal because it means the method is more robust across different problem complexities.

Jane: The authors also introduce a two-level control mechanism where the damping parameter lambda controls the local geometry and "the shape of the RNC curve itself," while t, the scalar curve parameter, determines how far along that constructed geometric curve we move.

Meng: That nested control structure is smart; it lets us manage how aggressively we explore versus how strictly we follow a geometrically valid path, which sounds like a sophisticated way to handle uncertainty during training.

Lalam: This allows for a trust-region ratio check rho(t) to decide when to commit to an update, ensuring that the inner search for t only uses loss evaluations and checks on that ratio.

Tom: The empirical validation shows this is effective; for instance, on the generalized Rosenbrock problem, they found that second-order RNC-LM works for n=two but higher orders are required when dealing with three dimensions because the geometry demands cubic bending.

Conclusion: Jane: So, to wrap up this discussion on "Higher-Order Geometric Updates for Levenberg–Marquardt Method via Riemann Normal Coordinates," it seems the paper successfully extends optimization methods to handle complex parameter spaces by incorporating higher-order geometric information.

Tom: The main implication is that we can get more geometrically consistent finite-step updates, which should lead to faster and more reliable convergence in many nonlinear least-squares tasks.

Lu: It shows that by explicitly constructing and factorizing the damped metric, the recursion only needs repeated solutions of linear systems with the same metric, which makes it computationally viable.

Meng: For practical application in areas like potential energy surface fitting, this means we can achieve a thirty-four times speedup over standard LM while maintaining high accuracy for complex molecular systems.

Lalam: This work suggests that by respecting the manifold structure, we can build AI models and optimization routines that are inherently more stable and less dependent on how we initially choose to label our parameters.

Jane: It's a solid piece of work that turns geometric insight into a practical way to handle finite steps in optimization. We're excited to see where this line of thinking goes next.

State Key Laboratory of Chemical Reaction Dynamics and Department of Chemical Physics, University of Science and Technology of China · State Key Laboratory of Chemical Reaction Dynamics, Dalian Institute of Chemical Physics, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Hefei National Laboratory

cs.LG, cs.NA, math.NA, physics.chem-ph, physics.comp-ph, physics.data-an

Submitted: 2026-07-08

Updated: 2026-09-28

Comments: 16 pages, 6 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Nonlinear least-squares optimization is central to many computational problems, and this work introduces a method that improves geometric consistency for finite steps in optimization by using

Key concepts

Damped Model-Graph Metric (Gµν): This metric defines the geometric structure of nonlinear least-squares problems. It combines a standard term with a damping factor ($\lambda$) to guide the optimization process along paths that account for the local curvature and constraints of the problem.
Riemann Normal Coordinates (RNC): RNC are a coordinate system used to represent geodesics near a point. They allow complex curved paths, like those in optimization, to be expressed as a simple polynomial expansion (up to order K) involving the initial direction and higher-order correction coefficients.
First-Kind Geodesic Residual: This is a computationally convenient way to express the geodesic equation. By setting this residual to zero, the method ensures that each step taken along the trial curve matches the actual geometric path defined by the damped metric, enforcing consistency at higher orders.
Trust-Region Ratio Control: This mechanism manages how far to take a step ($t$) along the constructed RNC curve. It compares actual objective function reduction against a linear prediction. If the reduction is poor, it adjusts the damping ($\lambda$) and rebuilds the curve; if good, it accepts the update.

Terminology

Benchmark — 26× in this paper · explained in 2 episodes
Accuracy — 8× in this paper · explained in 1 episode
Consistency — 7× in this paper · explained in 3 episodes
Failure Modes — 6× in this paper · explained in 3 episodes
Core — 3× in this paper · explained in 2 episodes
Robustness — 3× in this paper · explained in 5 episodes
Adam — 2× in this paper · explained in 2 episodes
Fisher Information — 2× in this paper · explained in 3 episodes
Objective Function — 2× in this paper · explained in 2 episodes
PDE residual — 2× in this paper · explained in 2 episodes
Ab Initio — 1× in this paper · explained in 1 episode
Anisotropy — 1× in this paper · explained in 3 episodes
Ensemble Learning — 1× in this paper · explained in 2 episodes
Gradient Flow — 1× in this paper · explained in 2 episodes
Meta-learning — 1× in this paper · explained in 2 episodes
Physics-Informed Neural Networks (PINNs) — 1× in this paper · explained in 6 episodes
Trace — 1× in this paper · explained in 2 episodes
Universality Class — 1× in this paper · explained in 2 episodes
alpha — 1× in this paper · explained in 1 episode

Summary

Nonlinear least-squares optimization is central to many computational problems, and this work introduces a method that improves geometric consistency for finite steps in optimization by using higher-order corrections derived from Riemann normal coordinates.

The gist: RNC-LM extends second-order geodesic acceleration to arbitrary orders by reformulating the geodesic equation as a first-kind residual condition along the residual trajectory, achieving finite-step updates with progressively higher reparameterization consistency.

Geometric Foundation and Initial Setup

The method begins by defining the geometric structure of nonlinear least-squares problems using a damped model-graph metric, denoted as Gµν = JmµJmν + λDµν. The key is to move beyond the coordinate-straight line update used by standard Levenberg–Marquardt (LM) and instead follow the geodesic equation defined by this metric. The paper establishes that while LM determines a tangent-space direction, it fails to realize this direction consistently over finite steps because straight lines in one parameter chart do not remain straight in another under nonlinear reparameterization.

Riemann Normal Coordinates and Finite-Order Expansion

To address this mismatch, the authors introduce Riemann normal coordinates (RNC). In these coordinates, the geodesic initialized by the LM direction is represented as a finite-order expansion: θµ(t) = θp + tvµ + t2/2cµ2 + t3/6cµ3 + ···. The goal is to determine the higher-order coefficients c2, c3,... recursively so that the resulting update curve satisfies the geodesic condition induced by Gµν to a prescribed order K. This construction ensures that the curve itself is corrected to represent the finite-step geometry accurately.

Recursive Construction of High-Order Updates

The core computational mechanism involves reformulating the geodesic equation into a computationally convenient form called the first-kind geodesic residual, A(t):= J(θ(t))⊤R¨(t) + λ¨θ(t) = 0. The coefficients c2,..., cK are then determined by enforcing that the Taylor coefficients of this residual vanish order by order at t=0. Specifically, the recursion is governed by the central algebraic structure: Gcn = −bn, where G is the damped metric and bn is a defect term derived from differentiating truncated curves. This allows higher-order corrections to reuse the same left-hand matrix (G) as the standard LM system, requiring only additional right-hand sides generated by automatic differentiation along a one-dimensional trial curve.

Trust-Region Ratio Control

The method employs a two-level control mechanism: the damping parameter λ controls the local geometry and the shape of the RNC curve itself, while the scalar curve parameter t determines how far along that constructed curve to move. This is managed by evaluating a trust-region ratio, ρ(t), which compares the actual reduction in objective function to its linear prediction. If no acceptable t is found, λ is increased and the RNC curve is rebuilt; if an acceptable tacc is found, the update occurs at θp+1 = θK(tacc). This structure ensures that the inner search over t requires only additional loss evaluations and evaluations of ρ(t) and naturally limits extrapolation beyond the effective radius of the finite-order RNC expansion.

Empirical Validation and Performance

Experiments demonstrate substantial improvements across various benchmarks. On the generalized Rosenbrock problem, second-order RNC-LM is adequate for n=2 (overshoot correction via t), while higher orders (K=3, 4) are necessary for n=3 because the geometry requires cubic bending. On the MGH10 benchmark, RNC-LM outperforms LM and LM-GA by avoiding collocation-overfitting failure mode and effectively handling near-rank deficiency. In large-scale potential-energy surface fitting, higher orders (up to fourth order) achieve a 34× speedup over standard LM while maintaining relative L2 errors on the order of 10−3 for reaction–diffusion PINNs, suggesting the geometric correction is effective beyond low-dimensional settings.

Future Directions

The authors suggest several open avenues for future research. One direction is to extend the construction from the pullback geometry of nonlinear least squares to more general statistical manifolds, such as those defined by Fisher information metrics used in natural-gradient methods and stochastic reconfiguration. Furthermore, the implementation relies on explicitly constructing and factorizing the damped metric; however, because the recursion only requires repeated solutions of linear systems with the same metric, it is compatible with approximate metric representations like K-FAC or PCG. This approach turns geometric insight into a practical finite-step optimization method that enhances robustness and efficiency.

How it works

Improvements for AI systems

Here are the specific improvements that can be made to AI systems by implementing the RNC-LM method, along with what those improved systems will be able to do:

The implementation of RNC-LM introduces a geometrically consistent finite-step update mechanism for nonlinear least-squares optimization, fundamentally changing how models are refined during training or fitting. The resulting improvements apply to AI systems relying on complex parameter spaces, such as Physics-Informed Neural Networks (PINNs) and potential energy surface fitting.

Here are the specific improvements and capabilities:

Higher-order Geometric Consistency in Optimization:

RNC-LM replaces the standard Levenberg–Marquardt (LM) coordinate-straight update with a finite-order Riemann normal coordinate (RNC) expansion of the geodesic flow. This ensures that the parameter update trajectory follows a path that satisfies the geodesic equation up to an arbitrary order, rather than just at the base point.

Enhanced Robustness to Parameterization Artifacts:

The method explicitly addresses parameter-effect curvature—the distortion induced by an arbitrary coordinate system—by eliminating its tangential component of residual acceleration order by order in the moving tangent frame.

Improved Convergence in Complex Loss Landscapes (e.g., Curved Valleys):

For problems with highly anisotropic or sloppy loss landscapes (like those found in generalized Rosenbrock tests or the MGH10 benchmark), RNC-LM provides a geometrically informed descent direction that avoids coordinate-dependent pitfalls, leading to significantly faster convergence than standard LM and LM-GA.

Increased Accuracy in Physics-Informed Models (PINNs):

When applied to PINN loss functions involving coupled reaction–diffusion equations, RNC-LM maintains the physically meaningful solution (recovering relative L2 errors around 10−3) instead of converging to a function with low collocation residual but high physical error. This directly mitigates the failure mode of collocation-overfitting.

Optimized Computational Efficiency via Adaptive Control:

RNC-LM integrates two nested control mechanisms:

a) The damping parameter (λ) controls the local metric and curve shape.

b) The curve parameter (t), selected via a line search criterion based on the trust-region ratio, controls the finite displacement along that constructed geometric curve. This prevents unnecessary large steps while ensuring progress along a geometrically valid path.

Significant Wall-Clock Time Reduction:

In large-scale tasks like high-accuracy Born–Oppenheimer potential energy surface fitting, RNC-LM demonstrates substantial speedup (up to 34× over standard LM), allowing these models to reach target accuracy in drastically reduced wall-clock time, making them feasible for larger molecular systems.

The resulting AI system will be significantly more reliable and efficient in the following domains:

High-Fidelity Scientific Modeling (e.g., Molecular Dynamics/Potential Fitting):

The system will be able to fit complex, high-dimensional potential energy surfaces with high precision (relative L2 errors of 10−3) and speed, enabling the simulation of larger molecular systems that are currently computationally prohibitive due to optimization bottlenecks.

Physics-Informed Neural Networks (PINNs) for Complex PDEs:

The system will be capable of training PINNs to solve challenging reaction–diffusion problems accurately by avoiding the collocation-overfitting failure mode, leading to solutions that are both numerically accurate and physically consistent across a wide range of diffusion coefficients.

Robust Optimization in Ill-Conditioned Spaces:

The system will handle optimization tasks characterized by severe ill-conditioning, near-rank deficiency (plateaus), and high parameter-effect geometry (sloppiness) with superior robustness, ensuring that the optimization process does not get stuck or converge to sub-optimal local minima caused by coordinate distortion.