Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control

arXiv:2609.38655 · eess.SY, cs.SY · Submitted 2026-09-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control".

Rosa: Safety is embedded directly into the policy class by constructing a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set…

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So, we're looking at the paper titled "Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control," and it’s really interesting because it talks about how to embed safety right into the policy class instead of adding external checks later.

Dev: I agree, Rosa; the title makes it sound like they're redefining how we approach safety in control systems, especially with reinforcement learning where things can get unpredictable.

Taro: From an autonomy research standpoint, I’m curious how this handles the inevitable surprises when the world doesn't behave exactly as expected during deployment.

Rosa: Exactly what I mean; it seems like they’re trying to build a system that is inherently safe because of how its actions are selected, not by constantly watching for errors after the fact.

Dev: It’s about shifting the burden from runtime safety filtering to a design problem upfront, which is something we always look for in control engineering.

Taro: If this holds up when things get messy outside the simulation environment, that would be a big deal for real-world autonomous systems.

The paper's summary: Rosa: The paper explains that they construct a finite library of feedback controllers that all share one common Lyapunov certificate, and this certificate guarantees forward invariance of a specific set under any switching sequence.

Dev: That’s the core mechanism; having one shared Lyapunov function, V(z) = z P z, where P is positive definite, ensures that the system stays within bounds even when we switch controllers arbitrarily.

Taro: So, if you have this certificate established for the dynamics of a quadcopter hover regulation, it means no matter which controller from the library you pick next, the state won't leave that safe region.

Rosa: Right; they reformulate safe learning as an MDP where safety is guaranteed by design before any learning even starts, which is quite a departure from typical Safe RL methods.

Dev: It simplifies the optimization problem significantly because instead of worrying about constraints during the reward maximization, the policy only needs to choose an action from this pre-certified set.

Taro: That seems like a solid theoretical foundation for tackling complex control problems where precise safety margins are hard to maintain dynamically.

The paper's improvements: Rosa: What they propose is a two-pronged approach: first, designing an action space where the dynamics satisfy F(xi, a) in X for every possible state and action, and second, learning a policy over that invariant set.

Dev: That means Problem one is about creating this finite library of admissible gain sets so that they are guaranteed to preserve forward invariance of the tracking-error set X, which is crucial for the control engineers on our side.

Taro: The improvement there is formalizing the constraints into the structure of the action space itself, rather than treating them as hard penalties in an optimization loop.

Rosa: And Problem two then focuses purely on maximizing expected reward J pi(xi zero) over this policy class, meaning we can focus on performance metrics without worrying about violating those safety boundaries during training.

Dev: This is powerful because it directly addresses the issues of runtime projection or barrier filtering that we usually have to implement in Safe RL setups, which often introduces its own latency and complexity.

Taro: It’s interesting how they connect the action space design directly to the policy optimization; it makes the safety guarantee intrinsic to whatever learning algorithm we choose.

Conclusion: Rosa: To wrap up, this paper successfully separates safety certification from policy optimization by embedding safety directly into the action space, giving us a policy-independent safety certificate for multikopter control.

Dev: I think the main implication is that we can achieve superior performance compared to non-adaptive tuning while maintaining a formal guarantee of forward invariance under arbitrary switching, which is something we’ve been chasing.

Taro: For autonomy research, this suggests that we can develop agents that are inherently safe across a whole range of possible control adjustments without needing complex runtime safety mechanisms to manage uncertainty.

Rosa: It really shows that the performance gains aren't just coming from better rewards; they are coming from a fundamentally safer policy structure defined by the common Lyapunov certificate.

Dev: We’re looking at systems where, instead of running a safety filter every time, we just ensure the action itself is safe because it belongs to this certified library.

Taro: I think if we can extend this concept beyond quadcopters to more complex aerial systems, the impact on deployment will be significant because it removes a major hurdle in trusting these learning agents in sensitive environments.

Chieh Tsai, Muhammad Junayed Hasan Zahed, Jinzhi Shen, Yi Xie, Ruoshan Lan, Majed Obaid, Salim Hariri

Department of Electrical and Computer Engineering, University of Arizona Department of Aerospace and Mechanical Engineering, University of Arizona Department of Computer Science, University of Arizona Community Medicine and Primary Care Department

eess.SY, cs.SY

Submitted: 2026-09-29

Updated: 2026-09-29

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: Safety is embedded directly into the policy class by constructing a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed

Key concepts

Forward-Invariant Action Space Design
This involves creating a limited set of control actions and feedback parameters such that any combination of these actions will always keep the vehicle's dynamics inside a specific safe region, regardless of which controller is active. This ensures safety is guaranteed by the structure of the learning problem itself.
Common Quadratic Lyapunov Function
This is a mathematical function used to prove stability. The paper finds one single quadratic function that works for every controller in the library. This function proves that the system's energy (related to tracking error) always decreases, no matter which controller is applied next.
Certified Tracking-Error Admissibility
This defines a specific boundary around the desired hover state where the vehicle must stay. By defining a safe region based on these bounds, and using the Lyapunov function, the paper proves that if you start inside this region, you will never leave it under any valid control sequence.

Terminology

Summary

Safety is embedded directly into the policy class by constructing a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set under arbitrary switching.

How it works

The framework reformulates safe learning for dynamical systems as an MDP where safety is guaranteed by design, independent of the learning process. This is achieved through two main problems:

  1. Forward-Invariant Action Space Design: Constructing a feedback parameterization and a finite action set such that the closed-loop dynamics satisfy the condition that F(ξ, a) ∈ X, ∀(ξ, a) ∈ X, ensuring that X is forward invariant under every admissible action sequence.

  2. Learning over an Invariant Policy Class: Determining a policy π: X → A that maximizes the expected reward Jπ(ξ0), where the action set A is derived from Problem 1. Because every admissible action preserves X, learning reduces to policy optimization over a forward-invariant policy class.

Key Theoretical Guarantees

The core of the safety guarantee lies in Theorem 1, which establishes stability and forward invariance for the switched closed-loop system. The paper constructs a common quadratic Lyapunov function, V (z) = z⊤Pz, where P is positive definite. The crucial finding is that the common quadratic Lyapunov function V (z) = z⊤Pz satisfies, for every admissible controller Ki, V˙ (z) ≤ −α∥z∥2 + β∥r(4)d(t)∥2, where α and β are common to all controllers in the library. This ensures that the dissipation inequality holds regardless of the switching sequence.

Safety Certification and State Space Definition

The paper defines a specific safety criterion related to tracking error bounds. The Certified tracking-error admissibility is defined by a set Zsafe where zl ≤ z¯l, l = 1,..., 14, specifying admissible bounds on the components of the external tracking-error state z. By defining a specific Lyapunov sublevel value ρsafe based on these bounds, the paper establishes a certified state space X:= Ωρ. Corollary 1 then proves that z(0) ∈ omegaρ ⇒ z(t) ∈ Zsafe, ∀t ≥ 0, meaning every trajectory initialized in this certified region remains within the prescribed tracking-error admissible set under any switching policy whose actions belong to the certified action set A.

Application to Multicopter Control

The framework is instantiated for quadcopter hover regulation, considering control-affine dynamics (5). The external error dynamics are modeled as z˙ = AEXTz + BEXTs − BEXTr(4)d(t), allowing the mapping of feedback parameters k to the closed-loop dynamics Acl(k). The RL component is a DQN that schedules among a finite library of controllers, where each action selects a controller from this library. The stage reward (35) penalizes tracking error and residual motion while discouraging aggressive vehicle motion via terms like − wu∥u∥2 − ws1[ak ≠ ak−1].

Empirical Evaluation

The experimental evaluation demonstrates the practical benefits of the framework. In nominal tests, the proposed DQN outperforms a fixed baseline by showing improvements in tracking performance, such as reducing hover RMSE from.013326 m to.010937 m. Furthermore, robustness tests show that while both policies remain safe under nominal conditions with a common certificate, the uncertified DQN fails safety and deadline satisfaction under nonideal conditions like wind or sensing delay. The dwell-time ablation study empirically exposes an explicit execution-level tracking–switching trade-off, showing that increasing the minimum hold time reduces switching but can diminish the tracking margin. This confirms that the common certificate itself does not rely on dwell time.

Conclusion and Significance

The paper successfully separates safety certification from policy optimization by embedding safety directly into the action space, yielding a policy-independent safety certificate. This approach eliminates the need for runtime safety filtering or action projection during deployment. The results illustrate that this framework can achieve superior performance compared to non-adaptive nominal tuning while maintaining a formal guarantee of forward invariance under arbitrary switching among certified controllers. Future work aims to investigate less conservative invariance certificates and extensions to more complex autonomous driving constraints.

The gist

Safety is embedded directly into the policy class by constructing a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set under arbitrary switching.

TABLE II

HELD-OUT NOMINAL AND NONIDEAL SAFETY/ARRIVAL PERFORMANCE. ARROWS INDICATE THE PREFERRED DIRECTION: HIGHER SAFETY AND DEADLINE RATES (↑), AND LOWER CONDITIONAL POST-ARRIVAL RMSE (↓).

Method Nominal Wind 2 m/s Model mismatch Sensor noise Delay 60 ms Joint Safe Dead. RMSE Safe Dead. RMSE Safe Dead.

Improvements for AI systems

Based on the provided research paper, here are specific improvements to AI systems derived from this framework, and what those improved systems can achieve:

The core contribution of this paper is shifting safety from an external constraint (runtime filtering) to an intrinsic property of the policy class itself (forward invariance).

Here are the specific improvements and capabilities:

  1. A reinforcement learning (RL) agent can be trained to optimize closed-loop performance without any need for runtime safety shields, penalty functions, or action projection mechanisms.

  2. The system's action space is restricted to a finite library of pre-certified feedback controllers (e.g., gain schedules). This means the RL policy only selects from these safe actions during training and deployment.

  3. The resulting autonomous system exhibits a formal, policy-independent safety guarantee: any trajectory initialized within a certified safe set will remain within that set for all future time, regardless of which certified controller is chosen at any decision instant (arbitrary switching).

Specific Improvements and Capabilities:

  1. A DQN-based controller can be used for complex, nonlinear control tasks (like quadcopter hover regulation) where traditional model-based controllers require conservative, fixed gains.

  2. The improved system can perform Safe Gain Scheduling by selecting between a finite set of stabilizing feedback laws based on the current state, effectively adapting the controller aggressiveness in real-time while maintaining formal stability guarantees across all switching sequences.

  3. The RL agent learns an optimal scheduling policy (mapping state to certified controller index) that maximizes performance metrics (e.g., tracking accuracy, hover RMSE) specifically within the bounds guaranteed by the common Lyapunov certificate, rather than optimizing performance subject to a potentially violated safety constraint.

Specific System Capabilities:

  1. A drone or robotic platform can execute complex maneuvers (like hovering and trajectory tracking) that are more aggressive and perform better than fixed-gain controllers, because the RL component optimizes the gain selection across a certified library.

  2. The system achieves superior performance compared to traditional methods (like median/nominal fixed controllers) by leveraging state-dependent scheduling, leading to measurable reductions in tracking error (e.g., up to 18% reduction in hover RMSE shown empirically).

  3. The system is robust against uncertainties encountered during operation, such as wind disturbances, model mismatches, sensor noise, and sensing delays (up to 60ms), because the safety certificate is designed based on the nominal model and verified against these perturbations.

  4. The AI system can operate with high confidence in its safety guarantees: because the safety property holds for every action in the learned policy class, there is no need for complex, computationally expensive runtime checks or projections to ensure physical constraints are met during operation.

Related papers