Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces

summary

Video file (mp4)

The gist

The gist: This work proposes a fully data-driven approach for optimal control of nonlinear controlaffine systems represented by a stochastic diffusion by embedding state probability densities into a

In short

The work proposes a data-driven method to find optimal control for nonlinear systems where dynamics and costs are unknown, using only control penalties. It embeds state probabilities into a Reproducing Kernel Hilbert Space (RKHS) to learn Markov transition operators from data. This allows the use of scalable operator-theoretic Hamilton-Jacobi-Bellman recursions to solve complex problems like autonomous underwater vehicle depth regulation.

Key concepts

Stochastic Optimal Control Problem
This is a mathematical challenge where you need to find the best sequence of control actions over time to minimize an expected cost, given that the system's movement is governed by random processes (stochastic diffusion). The goal is to determine the optimal feedback control law.
Reproducing Kernel Hilbert Space (RKHS)
An RKHS is a mathematical space used to embed complex data, like probability densities of a system's state, into a higher-dimensional space where functions can be represented efficiently. This embedding allows the method to learn and approximate the system's nonlinear dynamics using kernel regression techniques.
Markov Transition Operator
This operator mathematically describes how the probability density of a system's state evolves from one time step to the next under a specific control strategy. By learning this operator from data, researchers can capture the underlying nonlinear dynamics of the system in a compact, linear form.

Terminology used across episodes

This episode discusses

The paper

Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces · Read on arXiv

Chair of Information-oriented Control, TU Munich, Germany · School of Information Science and Technology, ShanghaiTech University, China

This paper proposes a fully data-driven approach for optimal control of nonlinear control-affine systems represented by a stochastic diffusion. The focus is on the scenario where both the nonlinear dynamics and stage cost functions are unknown, while only a control penalty function and constraints are provided. To this end, we embed state probability densities into a reproducing kernel Hilbert space (RKHS) to leverage recent advances in operator regression, thereby identifying Markov transition operators associated with controlled diffusion processes. This operator learning approach integrates naturally with convex operator-theoretic Hamilton-Jacobi-Bellman recursions that scale linearly with state dimensionality, effectively solving a wide range of nonlinear optimal control problems. Numerical results demonstrate its ability to address diverse nonlinear control tasks, including the depth regulation of an autonomous underwater vehicle.

DOI: 10.1016/j.engappai.2026.115698

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's start with the title itself, "Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces." It sounds very technical, but what it really means is they are using data to find optimal control when the system is nonlinear and you don't have all the model details.

Jane: Exactly. They are focusing on situations where both the dynamics and the stage cost functions are unknown, relying only on a penalty function and some constraints to guide the learning process.

Lu: The authors, Hoischena, Bevandaa, Sosnowskia, Hirchea, and Houskab from TU Munich and ShanghaiTech University are bringing together operator theory with modern data science techniques here.

Meng: What does that actually mean for a control engineer? It means we don't need to spend months building a precise mathematical model of the system before we can even start designing the feedback law.

Lalam: It shifts the focus from pure model-based design to learning from experience, which is something our large models are getting really good at doing anyway when given enough data.

The paper's summary: Tom: The core idea of this work is that instead of trying to solve the traditional Bellman equations in the state space, which gets impossible quickly, they reformulate the problem into a convex PDE-constrained optimization problem first.

Jane: That reformulation allows them to define a unique minimizer for the system and then use that structure to guide their data-driven learning process through an RKHS embedding.

Lu: They create empirical embeddings from training samples, say D n = (x(i), u(i), x(i)+, (x(i))), and then they learn the Markov transition operators by minimizing the squared RKHS norm error.

Meng: So, instead of solving a massive equation in high-dimensional state space, they are learning approximations of the operators that govern how the system moves from one state to another based on what we’ve observed.

Lalam: It’s like training a very sophisticated pattern recognition system to predict the future behavior of a complex physical process without ever being explicitly told the exact mathematical rules governing it.

The paper's improvements: Tom: The paper highlights two big improvements over previous methods. First, they develop a nonparametric framework to learn finite-rank approximations of those Markov transition operators using kernel regression techniques.

Jane: That’s key because it means the complexity of the learning process actually scales linearly with the number of states and controls, which is much better than what we usually see in these types of approaches.

Lu: The second major improvement is proposing this RKHS embedded dynamic programming recursion, which uses those learned Markov operator estimates to compute a data-driven approximation of the global minimizer.

Meng: So, they are using the data they collected to approximate the globally optimal solution in a way that is computationally simple enough for real-world use.

Lalam: This is powerful because it lets us bypass some of those strict requirements you see in other works, like not needing specific control excitations to learn infinitesimal generators; this method just requires real system data.

Conclusion: Tom: So, to wrap up the Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces paper, it proposes a way to use kernel regression to learn transition operators and then feed those into a recursion that approximates the global minimizer of the optimal control problem.

Jane: The results show that this approach recovers optimal control laws reliably across various nonlinear benchmarks, with numerical examples showing a mean cost reduction of five point zero nine percent and a standard deviation reduction of thirty point one one percent compared to an LQR baseline for something like depth regulation in an autonomous underwater vehicle.

Lu: The real implication here is that this framework offers a data-driven solution that exploits the full nonlinear dynamics without requiring prior knowledge of system parameters or models, which is really cool given the complexity of these problems.

Meng: For engineers, it means we can design robust controllers for systems we’ve only partially observed or where the physics are too complex to model perfectly from scratch.

Lalam: It suggests that future AI systems will get better at handling uncertainty by learning directly from messy, real-world data rather than relying on perfect mathematical specifications.

Tom: That’s what this paper is all about—moving toward a control method that learns directly from the environment instead of being rigidly programmed with every detail upfront.

Jane: It's a solid contribution to operator theory and control design, showing how learning methods can integrate naturally with stochastic optimal control theory through these RKHS embeddings.

Lu: We’re definitely watching how this kernel-based system identification method interacts with other advanced machine learning architectures in the coming years.

Meng: From an engineering standpoint, the linear scaling is what makes it viable for deployment, which is a big deal when you're dealing with real-time control loops.

Lalam: It really shows how AI can help us navigate problems that are fundamentally too complex for traditional, fully model-based approaches.

More episodes

← Home