An Introduction to Zero-Order Optimization Techniques for Robotics

summary

Video file (mp4)

The gist

The gist The authors propose a mathematical tutorial on random search to provide a simple and unifying perspective for understanding various zero-order optimization techniques used in robotics, such

In short

The paper introduces random search as a unifying mathematical framework for zero-order optimization techniques used in robotics, such as trajectory optimization and reinforcement learning. It explains how sampling-based methods, including various gradient approximation techniques and stochastic dynamics like Simulated Annealing, can effectively handle non-differentiable functions and escape local minima.

Key concepts

Zero-Order Optimization
These are derivative-free methods used in robotics to optimize problems where the objective function is not smooth or differentiable. They work by evaluating the function directly rather than using gradients, which makes them suitable for complex tasks like locomotion and manipulation where simulators provide noisy or non-smooth feedback.
Random Search
The core idea is to perform a random search by generating samples at random points in the parameter space. The algorithm updates its estimate of the best solution based on how well each randomly sampled point performs, offering a simple way to explore complex spaces without needing gradient information.
Gradient Approximation Methods
Since true gradients are often unavailable, these techniques estimate them using only function evaluations. Examples include forward finite difference and Simultaneous Perturbation Stochastic Approximation (SPSA), which allow optimization algorithms to proceed by approximating the direction of steepest descent through clever sampling.
Simulated Annealing
This method uses additive Gaussian noise to explore the search space. By defining a temperature parameter ($\lambda$), it can be used to sample from a distribution related to the objective function, providing theoretical guarantees for finding global solutions in non-convex settings by allowing temporary uphill moves.

Terminology used across episodes

This episode discusses

The paper

An Introduction to Zero-Order Optimization Techniques for Robotics · Read on arXiv

Armand Jordana, Jianghan Zhang, Joseph Amigo, Ludovic Righetti

New York University

Zero-order optimization techniques are becoming increasingly popular in robotics due to their ability to handle non-differentiable functions and escape local minima. These advantages make them particularly useful for trajectory optimization and policy optimization. In this work, we propose a mathematical tutorial on random search. It offers a simple and unifying perspective for understanding a wide range of algorithms commonly used in robotics. Leveraging this viewpoint, we classify many trajectory optimization methods under a common framework and derive novel competitive RL algorithms.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "An Introduction to Zero-Order Optimization Techniques for Robotics".

Dev: The gist The authors propose a mathematical tutorial on random search to provide a simple and unifying perspective for understanding various zero-order optimization techniques used in robotics,

Rosa: First, who's behind it and why it matters.

Paper summary: Dev: So, to wrap up, this paper "An Introduction to Zero-Order Optimization Techniques for Robotics" provides a mathematical tutorial on random search to give us a unified perspective on zero-order optimization techniques used in robotics.

Rosa: They show how this single viewpoint allows them to classify things like Model Predictive Path Integral and Covariance Matrix Adaptation under the same framework, and they derive some novel competitive RL algorithms as a result <ref:2506.22087#pg3>.

Taro: The paper explains the theoretical concepts behind why stochastic algorithms are suited to avoid getting stuck in local minima, which is something we need when we are testing real world failure modes <ref:2506.22087#pg3>.

Dev: Overall, it connects policy gradient techniques and Nesterov’s random search through the study of how these stochastic algorithms succeed in avoiding poor local minima <ref:2506.22087#pg3>.

Rosa: The title itself is a mathematical introduction to zero-order optimization techniques for robotics Armand Jordana, Jianghan Zhang, Joseph Amigo and Ludovic Righetti one <ref:2506.22087#pg1,introduction to zero-order optimization techniques for robotics Armand Jordana, Jianghan Zhang>.

Taro: It’s about using sampling-based methods to understand the success of popular RL algorithms <ref:2506.22087#pg2>.

Dev: This paper is really about giving us a broad perspective connecting gradient-free approaches in trajectory optimization and reinforcement learning <ref:2506.22087#pg3>.

Rosa: It’s not about some magic trick, it’s about showing how random search provides a unifying perspective on zero-order algorithms commonly used in robotics one <ref:2506.22087#pg1,algorithms commonly used in robotics>.

Taro: It opens up the potential for novel RL algorithms that stem naturally from this unified viewpoint <ref:2506.22087#pg3>.

Conclusion: Rosa: So this paper is introducing zero-order optimization, which is basically derivative-free optimization used in robotics, and they’re calling it an introduction to random search.

Dev: Right, so we’re talking about how these methods handle functions that aren't smooth and how they escape those local traps without needing gradients.

Taro: It seems like the core idea here is using random sampling because deterministic gradient methods get stuck easily in non-smooth problems like contact physics.

Rosa: Exactly, and they show this random search idea as a way to unify so many different techniques we use for trajectory optimization and policy optimization.

Dev: They're classifying things like Model Predictive Path Integral under this one framework, which is really neat because it makes comparing them easier for real-time loops.

Taro: And the part about stochasticity helping escape local minima through things like Langevin dynamics, that’s important when we think about how a system recovers from an unexpected situation in the real world.

Rosa: It connects directly to policy optimization too, where they use Gaussian smoothing to estimate gradients and avoid those bad local minima when training a neural network.

Dev: So what this means for us on the ground is that we might be able to design more robust controllers because we have a clearer mathematical way to understand why some sampling approaches work better than others.

Taro: It points toward novel reinforcement learning algorithms that leverage this sampling idea, which could lead to smarter AI agents in complex physical environments.

More episodes

← Home