Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators

arXiv:2407.15003 · cs.CR · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators".

Jane: The paper was written by Kyo Hyun Kim, Tejaaswini Narendran, Bijan Mehralizadeh, Zaid Abu-Abbas, Denizhan Kara et al. from University of Illinois Urbana-Champaign and George Washington University and Boeing Research and Technology.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we're digging into a paper that has both of us genuinely excited, and it's called "Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators." Jane, that title alone got my heart racing.

Jane: Mine too, Tom. And I love it because it's basically saying that the very thing that makes these drone systems safe and reliable is the same thing that makes them hackable. That's a bold claim, and the authors from UIUC, GWU, and Boeing Research and Technology are backing it up with real experiments.

Tom: Right, and for our listeners who might not be deep in the weeds here, let's break this down. We're talking about drones, UAVs, and the software that helps them figure out where they are in the world. That's the state estimator.

Jane: Exactly. And the paper's central idea is that these estimators are designed to be predictable. That's a feature. You want to know exactly how the drone will react to sensor data. But the researchers realized that if a machine learning model can learn that predictable behavior just by watching inputs and outputs, then an attacker can use that same learned model to fool the drone.

Tom: So they built a framework called Requiem that does exactly that. It watches the drone's state estimator, learns its behavior, and then generates fake sensor data that looks totally normal to the drone but actually causes it to drift off course.

Jane: And here's the kicker, Tom. They tested this on real hardware, actual quadrotors flying in a lab, and on high-fidelity simulations. And they got the drones to deviate by tens of meters while the drone's own systems thought everything was perfectly fine.

Tom: Tens of meters. That's not a rounding error. That's the difference between a drone hovering over a target and a drone drifting into a no-fly zone. And the drone has no idea it's happening.

Jane: The implications here are huge. We're talking about delivery drones, surveillance drones, search and rescue operations. If an attacker can silently redirect these vehicles, the consequences are pretty serious.

Tom: And the scary part is, the attack doesn't require root access or any kind of privileged control. It just needs to observe the system and inject data. We'll get into how they actually pull that off, but first, let's talk about why this "learnability" is such a fundamental issue.

Jane: Good plan. Because that's the real heart of the paper, and it's what makes this more than just another hack.

Paper discussion segment 2: Tom: So we're back, still talking about "Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators." And Jane, I want to get into the mechanics of how Requiem actually works.

Jane: Let's do it. So the key insight is that a state estimator, like the Extended Kalman Filter that most drones use, has a very specific job. It takes in sensor data, like GPS position and velocity, and it blends that with its own predictions to produce a best guess of where the drone is.

Tom: And that process is deterministic. Same inputs, same outputs. That's what makes it analyzable and certifiable. But it also means it can be learned.

Jane: Exactly. So the team built what they call a surrogate model, which is a deep neural network that mimics the behavior of the drone's state estimator. They trained it by feeding it lots of input-output pairs, basically watching the drone fly around in simulation and recording what the estimator did.

Tom: And once you have that surrogate, you can use it to train a spoofer. The spoofer is another neural network that generates fake GPS values. The goal is to make the residual, which is the difference between what the drone expects and what the sensors report, stay small enough to avoid triggering any alarms.

Jane: Right. Because the anomaly detector on the drone is looking for big residuals. If the sensor data is way off from what the estimator predicted, it raises a flag. So the spoofer has to be subtle. It has to nudge the drone off course without ever creating a suspiciously large error.

Tom: And they had two strategies for that. One is called "No Correction," where the spoofer basically makes the sensor data match the drone's prediction perfectly, so the drone never corrects its course. It just keeps going in the wrong direction, thinking it's on track.

Jane: And the other is "Direction Bias," where the spoofer trades a little bit of stealth to actively push the drone in a specific direction, like north or south, while still keeping the residuals under the detection threshold.

Tom: The results are pretty wild. In their circle mission, the Direction Bias attack caused a twenty-three-meter deviation while the drone's reported position stayed right on the nominal path. And the No Correction attack managed over one hundred ten meters before it finally tripped an alarm.

Jane: And they didn't just do this in simulation. They flew actual drones in a lab with a motion capture system, and they got similar results. The drone physically moved off its planned path, and its internal systems had no clue.

Tom: What really gets me is that they even went up against a state-of-the-art anomaly detector called SAVIOR, and they still managed to cause a two point five-meter deviation while staying stealthy. That's not just breaking a toy system. That's breaking a system that was specifically designed to catch this kind of attack.

Jane: So the attack is real, it's effective, and it's hard to detect. But the question is, what can we actually do about it? That's what we need to talk about next.

Paper discussion segment 3: Tom: We're back, still on "Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators." And Jane, we've established that the attack works. Now we have to ask the hard question: how do we defend against it?

Jane: That's the tricky part, Tom. And the paper is pretty honest about this. Because the attack exploits the fundamental properties of the estimator, there's no simple patch. You can't just update the firmware and make it go away.

Tom: So what are the options? The paper suggests a few directions. One is sensor redundancy. If you have multiple sensors providing overlapping information, like GPS and a camera or an IMU, then an attacker has to spoof all of them consistently, which is much harder.

Jane: That makes sense. But the paper also points out that this isn't always practical. Cameras don't work in fog or at night. And adding more sensors means more weight, more cost, more complexity.

Tom: Another idea is randomization. If you make the state estimator's behavior less deterministic, it becomes harder to learn. But that's a tough sell in safety-critical systems, because you want predictable behavior for certification and testing.

Jane: And that's the real tension the paper highlights. The determinism that makes these systems safe and certifiable is exactly what makes them vulnerable. You can't have one without the other, at least not with current approaches.

Tom: I want to bring in Lu and Meng here, because this is where the practical and the theoretical really collide. Lu, from a research perspective, do you see a way out of this bind?

Lu: Honestly, Tom, I think the paper is pointing at something deeper. It's not just about drones. Any system that relies on a learned or learnable model of the physical world is vulnerable to this kind of attack. Self-driving cars, industrial robots, even medical devices. The question is whether we can design systems that are both predictable and robust to adversarial learning.

Meng: And from an engineering standpoint, that's a huge challenge. You're asking for a system that behaves exactly as expected under normal conditions but is also unpredictable to an attacker. Those goals are fundamentally in tension. I think the more immediate solution is better intrusion detection at the system level, not just at the sensor level.

Jane: So we're not going to see a quick fix. But the paper is valuable because it forces us to think about these trade-offs now, before these systems become even more widespread.

Tom: And that's the takeaway for me. This isn't a doomsday paper. It's a wake-up call. It's saying, "Hey, we have a problem, and we need to start thinking about it seriously."

Jane: Well said, Tom. And I think that's a perfect segue into wrapping up our discussion.

Conclusion: Tom: Alright, we're in the final stretch of our discussion on "Predictable by Design, Vulnerable by Nature: Security Consequences of Learnability in UAV State Estimators." And Jane, I think we've covered a lot of ground.

Jane: We have. We've talked about how the paper shows that the very predictability that makes drone state estimators reliable also makes them learnable, and therefore attackable. The Requiem framework demonstrates that an attacker can learn the estimator's behavior and generate stealthy sensor spoofs that cause real physical deviations.

Tom: And we've seen the numbers. Over a hundred meters of deviation in some cases, all while the drone's own systems reported everything was nominal. They even beat a state-of-the-art anomaly detector.

Jane: But we also talked about the defense side. There's no easy fix. Sensor redundancy helps, but it's not always practical. Randomization could work, but it conflicts with the need for certifiable, predictable behavior.

Tom: So the paper leaves us with a challenge. We need to rethink how we design safety-critical autonomous systems. We can't just assume that predictability is always a good thing.

Jane: Exactly. And that's why this paper is so important. It's not just a hack. It's a fundamental insight into the security properties of these systems. It's a call to action for researchers, engineers, and policymakers.

Tom: And with that, we're going to say goodbye to this paper and get ready for the next one. Thanks for joining us, everyone.

Jane: Stay curious, stay safe, and we'll see you next time.

Kyo Hyun Kim, Tejaaswini Narendran, Bijan Mehralizadeh, Zaid Abu-Abbas, Denizhan Kara, Vineetha Paruchuri, Sibin Mohan, Greg Kimberly, Jae Kim, Josh Eckhardt

University of Illinois Urbana-Champaign · George Washington University · Boeing Research and Technology

cs.CR

Submitted: 2026-08-07

Code: https://github.com/dji-sdk/Onboard-SDK

Project page: https://projrequiem.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 44/100

Key concepts

State Estimator
This is the software component in a UAV (drone that determines its location and movement). It takes raw sensor data, such as GPS position and velocity, and blends it with internal predictions to create a best guess of where the drone actually is.
Learnability
The concept that because state estimators are designed to be predictable (same inputs yield same outputs), an attacker can use a surrogate model (a deep neural network) to learn how the system behaves. This learned model can then be used to generate deceptive data.
Spoofing/Spoofer
A mechanism used by an attacker, often involving a neural network, that generates fake sensor values. The goal is to nudge the drone off course while ensuring the resulting error (residual) remains small enough not to trigger the drone's internal anomaly detection systems.

Terminology

Summary

Summary

This paper investigates a novel security vulnerability in Unmanned Aerial Vehicles (UAVs) that stems from the inherent learnability of state estimation algorithms, specifically Extended Kalman Filters (EKFs). The authors propose that if a mathematical function can be learned from its input/output behavior alone, an adversary can exploit this property to compromise the system. The central thesis is that the very properties making state estimators reliable may constitute a security liability, motivating investigation into 'learnability' as an attack surface in safety-critical systems.

The paper introduces R EQUIEM, a machine-learning-based framework designed to exploit this vulnerability. The framework's objective is to cause stealthy physical deviations in a UAV's trajectory—where the drone physically moves away from its mission path while its onboard systems and operators believe it is operating nominally. The attack works by generating spoofed sensor values that lie within the nominal operational region, making them undetectable by standard anomaly detectors. As the paper states, if the 'attack values' (aka spoofed sensor values) lie within the nominal region... then the state estimation algorithms as well as the anomaly detectors are unable to detect the false values.

The R EQUIEM framework operates through a three-stage pipeline: (1) data collection, (2) surrogate model training, and (3) spoofer training. In the first stage, data is collected from flight controller simulation interfaces, using a random walk mission to capture diverse vehicle kinematics. The second stage involves creating a surrogate model—a deep neural network (DNN)—that emulates the target state estimation function based on observed input/output behavior. The paper notes that the use of DNNs allows us to develop a 'spoofer' by optimizing against the surrogate similar to generative adversarial networks (GANs). The third stage trains a spoofer model against the surrogate to generate malicious sensor inputs that transfer to real drones.

The threat model assumes a realistic and somewhat restrictive security architecture where tasks are isolated in user space, communicating only via middleware, and critical functions are protected using techniques like triple redundancy or trusted execution environments. The attacker does not require root access but can observe and inject inputs. The paper emphasizes that our model anticipates a security architecture where critical flight logic function is sandboxed or isolated, but I/O remains accessible.

The paper addresses three key design questions: how to effectively collect data for the surrogate (DQ1), how to prevent overfitting and underfitting (DQ2), and how to optimize the spoofer for effectiveness (DQ3). For DQ2, the authors use data augmentation via Monte Carlo Arithmetic and slicing to reduce the complexity of the function the surrogate must emulate. For DQ3, they propose two attack strategies: No Correction (NC) and Direction Bias (DB). The NC attack prevents the estimator from realizing the existence of errors by injecting sensor values that match the estimator's expected sensor value, effectively inducing dead reckoning. The DB attack trades off some residual (i.e., stealthiness) to bias the direction of the estimation error by exploiting knowledge of the anomaly threshold.

The evaluation was conducted against both standard PX4 controllers and the state-of-the-art SAVIOR anomaly detector, using real-world quadrotor experiments and high-fidelity Gazebo/PX4 simulations. The results demonstrate significant deviations from planned mission paths while evading anomaly detection. For the circle mission, R EQUIEM's DB attack achieved up to 23.78 meters of stealthy deviation, and the NC attack achieved 110.62 meters before becoming overt. In contrast, baseline attacks (constant offset, random offset, and boiling frog) all failed to remain stealthy, with the best baseline (position boiling frog) achieving only 0.111 meters of stealthy deviation.

The paper evaluates the attack's performance against two types of anomaly detectors: the onboard χ2 detector and a more stringent threshold-based detector (τ-AD). R EQUIEM successfully bypassed both, with the paper noting that R EQUIEM is able to evade the state-of-the-art EKF-based anomaly detector (SAVIOR); our approach succeeds in stealthy attack demonstrate up to 2.5m of deviation in this case.

Real-world experiments were conducted using a quadrotor with a CubeOrange+ flight controller running PX4 v1.15.4, using a Vicon motion capture system instead of GNSS due to indoor testing constraints. The results showed that "R EQUIEM caused significant deviations (i.e., reached the net after about 1m of deviation) that we had to kill the quadrotor before it was able to finish the mission, thus demonstrating the attack's effectiveness in real-world." The paper also demonstrates that simulation results transfer to real-world scenarios when configurations closely mirror each other.

The paper also explores how environmental factors, such as wind, affect the attack. Interestingly, the attack exploits the wind by tricking the controller into 'over-correcting' against a non-existent force, effectively using the drone's own stabilization logic to drive it off course. The deviation direction is opposite to the wind direction.

Regarding the learnability of the target function (RQ6), the paper concludes that "the surrogate models managed to learn the target function since important functions in safety critical systems, such as state estimator, are designed to be predictable, especially w.r.t. the correlation between their ins and outs."

The paper's contributions are summarized as: (1) a novel, stealthy attack approach and the R EQUIEM framework, (2) demonstration of attacks on the widely-used PX4 flight controller in both real hardware and simulation, (3) successful attacks against the SAVIOR anomaly detector, and (4) insights into when deterministic safety critical systems become learnable and thus vulnerable to ML-driven attacks. The authors conclude that we highlight a fundamental tension viz. the determinism required for robust modeling, safety and certification is precisely what enables learnability, i.e., makes our attack possible.

Improvements for AI systems

Based on the paper, here are specific improvements I can make to AI systems:

Improvement: Build a validation module that tests any AI-based state estimator or anomaly detector against learnability-based attacks before deployment.

Capability: The improved system can automatically generate stealthy sensor spoofing inputs (using surrogate DNN models of the target estimator) to expose vulnerabilities in production autonomous systems—quantifying maximum deviation achievable before detection.

Improvement: Implement a data-efficient training framework that: (a) uses random-walk missions to collect diverse I/O pairs, (b) applies Monte Carlo Arithmetic for data augmentation, and (c) uses slicing to reduce target function complexity.

Capability: The system can learn accurate approximations of any black-box state estimation function (e.g., EKF variants) using only observed inputs/outputs, achieving near-zero validation loss—enabling transferable attack generation without internal knowledge.

Improvement: Train a spoofer DNN using a combined loss function: min (residual)2 − (deviation budget × deviation direction), where residual loss ensures stealth and budget loss respects anomaly thresholds.

Capability: The optimized spoofer can generate sensor injection values that: (a) keep residuals within strict anomaly bounds (τ-AD), (b) cause directional bias (e.g., 23m deviation in circle missions), and (c) remain undetected by both χ2 and SAVIOR detectors.

Improvement: Create a lightweight (3-layer, 21KB) inference module that runs onboard flight controllers, using branchless ReLU and pre-loaded weights from SD card.

Capability: The system can execute attacks in real-time on physical drones (demonstrated on PX4/CubeOrange), causing 1m+ deviations in constrained indoor arenas while maintaining computational overhead that doesn't degrade flight quality.

Improvement: Develop a framework that transfers attacks trained on GNSS spoofing to other sensors (e.g., Vicon motion capture) by retraining the spoofer on the new sensor's I/O characteristics.

Capability: The improved system can adapt to different sensor types and data rates (10Hz Vicon vs 5Hz GNSS) with centimeter accuracy, demonstrating generalizability across hardware configurations.

Improvement: Add a module that predicts how environmental factors (e.g., wind) affect attack effectiveness, using the insight that EKF fights wind, making suppression attacks cause over-correction.

Capability: The system can forecast that: (a) wind increases attack deviation (up to 289m in simulations), (b) attack direction reverses relative to wind, and (c) it can exploit the drone's own stabilization logic to amplify deviations.

Improvement: Implement a benchmark that evaluates anomaly detectors (χ2, τ-AD, SAVIOR) against learnability-based attacks, with metrics for stealth duration and max stealthy deviation.

Capability: The tool can quantify that: (a) baseline attacks (constant/random/boiling-frog) fail within seconds, (b) R EQUIEM achieves 110m+ deviations undetected, and (c) it can identify detector brittleness (e.g., SAVIOR's numerical instability) before real-world deployment.

Improvement: Build a scanner that checks if any safety-critical function (state estimator, controller) satisfies k-Lipschitz continuity—the property that makes it learnable.

Capability: The system can flag systems as vulnerable by design if they exhibit bounded continuity, enabling security audits to prioritize functions that are mathematically predictable and thus exploitable via ML.

Sources

Related papers