Adaptive Extremum Seeking Control via the RMSprop Optimizer

arXiv:2409.12290 · math.OC, cs.SY, eess.SY · Submitted 2024-09-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Adaptive Extremum Seeking Control via the RMSprop Optimizer".

Dev: Extremum Seeking Control (ESC) is a family of continuous time algorithms for model-free optimization of a cost function, used in applications such as variable cam timing engine operation

9: ,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: We’re starting with the title and authors of "Adaptive Extremum Seeking Control via the RMSprop Optimizer," Patrick McNamee and Zahra Nili Ahmadabadi, and I want to get a simple breakdown of what this actually means for us.

Dev: I think the title tells us right away that they've taken a known method, Extremum Seeking Control, and improved it by swapping out the basic optimizer for something more robust.

Taro: It suggests that instead of relying on just standard gradient information, they are incorporating an adaptive scaling mechanism to handle situations where our cost function isn't perfectly behaved.

Rosa: So, in simple terms, they are taking a control strategy designed to find the best point in a system and making it smarter about how fast it searches for that best point.

Dev: That means the core idea is to make sure the system doesn't get stuck or move too slowly just because the shape of our cost function changes unexpectedly.

Taro: It seems like they are addressing a known weakness in previous ESC approaches by making the convergence speed less dependent on unknown local curvature information, which is a significant area for autonomy research.

Rosa: That’s right, and it opens up possibilities for applying this to optimization problems where we don't have an explicit model of how the system behaves.

Dev: It’s about moving away from algorithms that might get bogged down if the second or third derivatives are either way too small or way too large.

The paper's summary: Rosa: Now, let's look at the actual summary of "Adaptive Extremum Seeking Control via the RMSprop Optimizer" to see what they’ve actually done in terms of methodology and the results they claim.

Dev: The authors present a continuous time algorithm called RMSpESC, which is defined by a set of variable-wise differential equations involving gradient estimates and filter states like i and v i.

Taro: They use sinusoidal dither signals, m i(t) = 2a i (omega rit), to probe the cost function, which is a common technique in this field.

Rosa: The key takeaway from their summary is that they propose using the RMSprop optimizer because it adapts its gradient scaling to normalize convergence speed across all parameters.

Dev: They then show that this approach leads to semiglobal practical uniform asymptotic stability, or sGPUAS, for the average system dynamics of RMSpESC under certain assumptions.

Taro: The proof they provide uses a Lyapunov function based on observed contracting attractive sets, which is a strong tool for rigorously analyzing these interconnected systems.

Rosa: So, they're claiming that this method provides practical stability in real-world scenarios for minimizing a cost function without needing perfect knowledge of the cost function’s second derivatives.

The paper's improvements: Dev: The most significant improvement they point out is mitigating the dependency on higher-order derivatives, which is a big deal when we can't calculate those things easily.

Rosa: That directly addresses my concern about convergence rates; standard Gradient-based Extremum Seeking Control often struggles because the convergence speed changes depending on the unknown Hessian eigenvalues.

Taro: I see that as a way to make the system more robust against poorly conditioned optimization landscapes, which is exactly what we need when dealing with unpredictable external interactions.

Dev: By using RMSprop’s adaptive scaling, they claim they achieve a normalized convergence rate in all parameter directions, meaning it should perform consistently no matter how curved the cost function is locally.

Rosa: That normalization idea sounds very powerful for applications where the optimization landscape might be highly non-linear or even discontinuous in certain regions.

Taro: Also, the paper suggests that this framework can be applied to interconnected systems through their Lyapunov function design, which means we could potentially use it to manage multiple interacting AI components safely.

Conclusion: Rosa: To wrap up, what's the final word on the implications of "Adaptive Extremum Seeking Control via the RMSprop Optimizer"? I want a quick summary of why this work matters.

Dev: Essentially, this paper gives us a way to build model-free optimization systems that are more reliable because they don't rely on knowing precise second-order derivatives for stability guarantees.

Taro: For autonomy, it means we can design systems that maintain reasonable performance even when the environment throws unpredictable challenges at them because the convergence isn't overly sensitive to local variations in the cost function.

Rosa: It sounds like a step toward making AI controllers more resilient when deployed outside of controlled lab settings, and I’m excited to see how this plays out in those real-world tests.

Dev: From an engineering standpoint, the stability proof they offer suggests that we can design tighter constraints on our loop rates while still expecting practical convergence towards the optimum.

Taro: I'm just looking forward to seeing how they extend this concept when we move beyond simple scalar functions to more complex, multi-variable optimization problems in dynamic environments.

Department of Mechanical Engineering, San Diego State University

math.OC, cs.SY, eess.SY

Submitted: 2024-09-18

Updated: 2026-09-29

Comments: `0 pages, 4 figure

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: Extremum Seeking Control (ESC) is a family of continuous time algorithms for model-free optimization of a cost function, used in applications such as variable cam timing engine operation [9], ABS

Key concepts

Extremum Seeking Control (ESC)
A family of continuous time algorithms used for model-free optimization of a cost function. It is applied in areas like variable cam timing engine operation and aims to find the best point in a system.
RMSprop Optimizer
An adaptive scaling mechanism incorporated into ESC. It adapts its gradient scaling to normalize convergence speed across all parameters, helping the system perform consistently regardless of how curved the cost function is locally.
Semiglobal Practical Uniform Asymptotic Stability (sGPUAS)
The stability result claimed for the RMSpESC algorithm under certain assumptions. This suggests that the average system dynamics of this method achieve practical stability in real-world scenarios for minimizing a cost function.
Higher-order derivatives
These are derivatives beyond the first derivative, such as second or third derivatives of a cost function. The paper's improvement is mitigating dependency on these, which is crucial when they cannot be easily calculated.

Terminology

Summary

Extremum Seeking Control (ESC) is a family of continuous time algorithms for model-free optimization of a cost function, used in applications such as variable cam timing engine operation [9], ABS braking [3], and speed control on sailing yachts [12]. The most basic ESC is the Gradient-based Extremum Seeking Control (GESC), which attempts to mimic a gradient descent algorithm. This baseline approach has local convergence rates towards extremums that are dependent on the eigenvalues of an unknown Hessian of the cost function at the extrema, which could be arbitrarily small or large, potentially leading to impractical implementation in real-world scenarios. Another ESC is the Newton-based ESC (NESC) [5], which attempts to correct for the unknown Hessian by estimating its inverse; however, these algorithms require that the Hessian be invertible for convergence and necessitate estimating more derivatives of the cost function.

To mitigate this dependency on higher-order derivatives, the authors propose using the RMSprop optimizer for ESCs because RMSprop is an adaptive gradient-based optimizer which attempts to have a normalized convergence rate in all parameters. The proposed algorithm is termed RMSpESC.

The problem statement involves practically minimizing a sensor output y using model-free methods, where the sensor output is generally considered as y = J(θ) where the cost function J: R n → R depends on the parameter vector θ ∈ R n and is minimized by some optimal parameter θ∗. This work uses standard additive sinusoidal dither signals such as those in [5] to decompose the parameter into θ = θˆ + s(t), where θˆ is the estimate of θ∗ and s is the dither signal, defined as a vector whose elements are si(t) = aisin(ωri t) where ri ∈ Z̸=0 is a non-zero integer and the dither amplitude ai ̸= 0. The overall magnitude of s can be scaled by adjusting a0, where a0 = q ∑ n i=1 a squared i and treated as a small parameter so that the individual dither signal amplitudes have fixed ratios ai/a0 but the overall magnitude of s can be scaled.

The cost function assumptions made for this work are:

Assumption 1: The cost function J: R n → R is continuously differentiable (J ∈ C 1) on the domain θ ∈ R n.

Assumption 2: The cost function J has a unique minimum θ∗ such that ∀ θ ∈ R n, J(θ∗) < J(θ) if θ ≠ θ∗.

Assumption 3: The gradient vector ∇J(θ) = 0 if and only if θ = θ∗.

Assumption 4: The cost function J is a radially unbounded function.

The continuous time differential equations defining the RMSpESC are given by the following variable-wise equations for a parameter space of dimension n:

dˆθ i = -k gˆ i(t,ˆθ,ξ) / √v̂ i + ε (1)

d v̂ i = ωl i gˆ 2 i(t,ˆθ,ξ) / 2 - v̂ i (2)

d ξ = ωξ (y−ξ) (3)

where ˆg it is the ith element of the gradient estimate defined as:

gˆ it,ˆθ,ξ = m i(t) J(θ̂ + s(t)) - ξ (4), and m i is a sinusoidal signal defined as m i(t) = 2 a i sin(ωr it).

The corresponding average system for the RMSpESC is autonomous and globally uniformly asymptotically stable (GUAS). The average system dynamics are defined by:

dˆθ i = -k gˆ i p / v̂ i + ε (5)

d v̂ i = ωl i gˆ 2 i / 2 - v̂ i (6)

d ξ = ωξ J(ˆθ) - ξ (7)

where J(ˆθ)=1/T ∫[t+T,t] dτ J(ˆθ + s(t+τ)).

The main theorem of the work states: Theorem 1 (RMSpropESC is sGPUAS): For a cost function J: R n → R satisfying the Assumptions 1-4, the RMSpESC defined by Eqs (1)-(3) is sGPUAS to the point (θ∗,0, J(θ∗)) with the small parameter vector a0,ω−1.

To illustrate local stability for a scalar cost function J(θ) = J∗ + 1/2 H θ squared, the average gradient is found to be gˆ i p = H θ̂.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems by applying the concepts from this research:

  1. Superior Optimization for Model-Free Control Systems: The RMSpESC (RMSprop Extremum Seeking Control) framework provides a method for model-free optimization of cost functions in continuous time. This is particularly valuable for AI systems operating in environments where an explicit mathematical model of the system dynamics is unavailable or too complex to derive.

  2. Robust Convergence to Unknown Extrema: The key improvement offered by using the RMSprop optimizer over standard gradient methods (like GESC) is achieving a uniform convergence speed in all parameter directions. This means the AI system's parameters will converge toward an optimal configuration regardless of the local curvature (Hessian eigenvalues) of the unknown cost function.

  3. Mitigation of Local Convergence Rate Dependency: Traditional methods suffer when the second-order derivatives are arbitrarily small or large. The RMSpESC, by adapting its gradient scaling based on past gradient magnitudes (the RMSprop mechanism), ensures that convergence rates are less sensitive to these unknown higher-order derivative properties, leading to more reliable performance in complex optimization landscapes.

  4. Practical Stability and Guaranteed Performance: The paper proves the semiglobal practical stability (sGPUAS) of the RMSpESC. This means the AI system is not only theoretically stable but is guaranteed to converge practically (i.e., within a certain bounded error region) under realistic conditions, which is crucial for real-world deployment where perfect theoretical convergence is unattainable.

  5. Enhanced Stability via Lyapunov Function Design: The development of a specific Lyapunov function based on contracting attractive sets provides a powerful tool for analyzing the stability of the interconnected systems that these AI controllers govern. This allows researchers to rigorously prove stability properties across complex, multi-component AI subsystems.

  6. Improved Trajectory Control in High-Oscillation Environments: The numerical example suggests that RMSpESC avoids the severe, high-frequency oscillations seen in standard GESC when initial conditions are poor. This translates to AI systems that can maintain a more moderate and predictable convergence rate toward the optimum, leading to smoother, less erratic control outputs.

  7. Adaptive Learning for Parameter Estimation: The mechanism where the gradient estimate converges to the true gradient as dither amplitude approaches zero (Corollary 1) implies an inherent adaptive quality. The system can effectively learn the cost function landscape even when only noisy or perturbed measurements are available, making it suitable for online, real-time parameter tuning in dynamic AI models.

In summary, applying this research allows for the creation of AI systems that are more robust to model uncertainty, converge faster and more predictably across diverse optimization landscapes (avoiding pathological convergence issues), and offer rigorous proofs of practical stability for deployment in complex, interconnected environments.

Sources

Related papers