Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation

arXiv:2609.39611 · cs.RO · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Towards Agile Vision-Based Multi-UAV Flight".

Rosa: Agile multi-UAV flight requires accurate and low-latency onboard estimation of neighboring UAVs' kinematic states for critical tasks like collision avoidance and motion coordination.

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: The paper focuses on the title and authors of "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," and it immediately signals that the core issue they're addressing is how we estimate the kinematics of neighboring UAVs when we rely only on position data.

Dev: They bring in a new way to look at this by proposing integrating tilt measurements, which they say are provided by a state-of-the-art visual detector, to get information about the thrust direction of co-planar multirotor UAVs.

Taro: That tilt information is key because it gives them that extra constraint they need beyond just where the object is located in space.

The paper's summary: Rosa: Basically, the summary explains that most vision-based methods only use position measurements, which means velocity and acceleration have to be inferred indirectly from displacement, and this introduces a fixed structural delay in estimating those higher-order states.

Dev: They benchmarked four position-only estimators against five pose-aware estimators, including a new formulation of a linear thrust-constraining Kalman Filter.

Taro: The main point they pull out is that those pose-aware methods consistently reduce the average velocity and acceleration estimation errors by forty percent and fifty-seven percent across the three datasets they tested.

The paper's improvements: Rosa: What really stands out about their suggested improvements is how tilt-constrained estimators operate almost at the physical response limit given by the camera frame rate, because they observe the change in thrust direction before a lot of displacement accumulates.

Dev: That contrasts sharply with the position-only filters, which they found exhibit a constant around three hundred milliseconds delay in their acceleration step response that doesn't change regardless of how agile the UAV is.

Taro: That fixed delay is what limits the achievable agility, and by using tilt measurements to constrain that thrust direction, you get much better performance when things start moving fast.

Conclusion: Rosa: So wrapping up this discussion on "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," it seems the main implication is that integrating these tilt measurements is necessary to move from just tracking positions to achieving true agile motion coordination in multi-UAV setups.

Dev: I think the practical impact here is significant because the Z-KF formulation they propose, which uses a Degenerate Kalman Filter framework, allows them to fuse those affine subspace measurements into a standard linear KF form effectively.

Taro: For me, it means that when things get misbehave in the real world—like an unexpected gust of wind or another drone suddenly changing course—this estimation system is far better equipped to handle those rapid changes than what was previously possible with only position data.

Rosa: We've just covered how this paper tackles the title and authors, focusing on the initial problem they set up regarding state estimation accuracy for multi-UAV flight.

Dev: And that leads us into the summary of what they actually proposed in terms of their methodology and findings for "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation."

Taro: I'm curious about how deep their analysis went when they looked at the performance across different agility levels.

Rosa: The authors summarize that the core idea is to move past relying only on position measurements by incorporating tilt data to constrain the thrust direction of co-planar multirotor UAVs.

Dev: They compare four position-only estimators with five pose-aware ones, and they found that those pose-aware estimators consistently reduced average velocity and acceleration estimation errors by forty percent to fifty-seven percent.

Taro: That reduction in error is substantial; it means the AI system is much more reliable when the environment gets busy.

Rosa: One major improvement they highlight is that pose-aware filters are not only more accurate but also operate near the physical response limit of what's possible based on the camera frame rate because they capture thrust direction changes before displacement builds up.

Dev: That’s a big deal for latency, because the position-only methods suffer from a fixed structural delay of about three hundred milliseconds in their acceleration step response, which is independent of agility.

Taro: So essentially, they're trading that fixed structural bottleneck for something that reacts much more quickly to actual changes in motion.

Rosa: To conclude this discussion on "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," the key implication is that pose-aware estimation isn't just a minor tweak; it’s what unlocks the capability for truly agile multiUAV motion coordination.

Dev: I think the practical impact is seen in how this improved estimation quality directly translates into closed-loop control, where position-only tracking fails to allow a follower to stably hover during complex lateral maneuvers.

Wrap-up: Taro: And that ties back to why pose-aware relative state estimation is necessary for realizing multiUAV motion coordination approaching the dynamic limits of individual UAVs.

Rosa: We've just summarized the paper's summary, focusing on what they found regarding the error reduction achieved by using tilt measurements in "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation."

Dev: Now we need to discuss what they actually suggested as improvements to their existing estimation techniques.

Taro: I'm thinking about how this new formulation, the Z-KF, fits into the bigger picture of robust estimation.

Rosa: The paper suggests that a major improvement is incorporating tilt measurements by representing them as the z-axis vector of the UAV’s body frame to constrain thrust acceleration direction.

Dev: They then use this constraint within a modified measurement model, which they then fuse using the Degenerate Kalman Filter framework to create something called the Z-KF.

Taro: It's smart how they used that DKF framework because it lets them fuse measurements that don't usually work together in a conventional KF structure, which is what makes the Z-KF so effective.

Rosa: Another key improvement is the performance across different agility regimes, where pose-aware filters consistently outperform position-only variants at every level with a smaller estimation error and error variance.

Dev: Specifically, they showed that the Z-KF+BDC estimator achieves the overall lowest velocity and acceleration errors on the Unreal dataset, reducing MEN by forty-two percent and fifty-eight percent relative to the best position-only variant.

Taro: That kind of quantified performance difference is what really tells you how much better their method actually is when you're looking at hard data comparisons.

Rosa: So, to wrap up this segment on the improvements in "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," the takeaway is that the system needs to incorporate tilt measurements and a sophisticated filter like the Z-KF for superior state estimation.

Dev: I think this means we should focus our development efforts on developing that specific filtering architecture, as it’s what delivers the most significant performance gains in terms of error reduction.

Taro: From an autonomy standpoint, this gives us a more reliable foundation to plan complex maneuvers where things might not be behaving perfectly as expected.

Rosa: We've moved on to discussing the improvements suggested by the authors in "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," specifically focusing on how they suggest enhancing their estimation techniques.

Dev: Now that we know about the Z-KF, we should talk about how this new formulation fits into the broader context of state estimation research.

Taro: I'm wondering if these methods are going to be practical for deployment outside of a controlled lab setting or if they have real-world limitations on how long they can reliably work.

Wrap-up: Rosa: The paper points out the suggested improvement is using the Z-KF formulation, which represents tilt as the z-axis vector of the UAV’s body frame to constrain thrust acceleration direction.

Dev: They then use this constraint within a modified measurement model, which they then fuse using the Degenerate Kalman Filter framework to produce a standard linear KF form for implementation.

Taro: This fusion step is crucial because it bridges the gap between non-linear measurements and the standard linear KF structure, allowing them to handle those difficult measurements in a way that was previously hard.

Rosa: Another improvement they detail is the consistent performance across different agility regimes, where pose-aware filters consistently outperform position-only variants at every level with a smaller estimation error and error variance.

Dev: They also showed that the Z-KF+BDC estimator achieves the overall lowest velocity and acceleration errors on the Unreal dataset, reducing MEN by forty-two percent and fifty-eight percent relative to the best position-only variant.

Taro: Those specific numbers are what give us a concrete measure of how much better their method performs compared to existing systems like those they benchmarked.

Rosa: So, wrapping up this segment on the improvements in "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," the main improvement is clearly moving toward a system that uses tilt measurements and the Z-KF to get superior, low-latency state estimation.

Dev: I think this means our immediate development focus should be on implementing that specific filtering architecture because it's what yields the most significant performance gains in error reduction.

Taro: For autonomy, this gives us a more reliable foundation to plan complex maneuvers where things might not be behaving perfectly as expected.

Rosa: We've just discussed the improvements suggested by the authors in "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," and we've looked at how these enhancements translate into better estimation performance.

Dev: Now we need to look at what they say about the practical application of this work, especially concerning real-world deployment and potential limitations or limitations of the system itself.

Taro: I'm eager to hear your thoughts on where this research might actually be useful outside of a perfect simulation environment.

Rosa: Regarding practicality, the paper implies that this approach is crucial because it addresses the need for low-latency onboard estimation for collision avoidance and motion coordination in real multi-UAV flight.

Dev: They emphasize that position-only estimation introduces a constant overhead of about three hundred milliseconds above the physical onset, which they call a structural bottleneck in position-based estimation.

Taro: That means if we can get that latency down to near-instantaneous response, it becomes viable for actual flight scenarios.

Rosa: The improvement they suggest is that the Z-KF+BDC estimator corresponds to a near-instant response time, responding near-instantly because thrust-direction sensing allows for faster state inference.

Wrap-up: Dev: That low latency is critical because it’s what makes the system viable for maintaining stability during high-agility maneuvers in formation control.

Taro: If the paper states its limitations, one thing they flag is that the position-only estimation of the leader's state fails to facilitate stable hovering of a follower, which shows where this new method excels.

Rosa: In summary, "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation" suggests that incorporating tilt measurements and advanced filtering like the Z-KF is the path to achieving stable and agile motion coordination for multi-UAV systems.

Dev: I think the main implication is that we need to prioritize latency reduction because position data alone just isn't cutting it for high-speed coordination tasks.

Taro: So, this research gives us a solid direction on how to build more capable estimation systems for complex aerial environments.

Rosa: We've just discussed the paper and its improvements, focusing on the practical aspects of "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation."

Dev: Now we need to discuss the limitations they actually state regarding deployment conditions and whether this sophisticated estimation system is ready for real flight.

Taro: I'm thinking about how long this setup would last before we can trust it in the field.

Rosa: The paper suggests that this approach is crucial because it directly addresses the need for low-latency onboard estimation specifically for collision avoidance and motion coordination in real multi-UAV flight.

Dev: They point out that position-only estimation introduces a constant overhead of about three hundred milliseconds above the physical onset, which they call a structural bottleneck in position-based estimation.

Taro: That means if we can get that latency down to near-instantaneous response, it becomes viable for actual flight scenarios.

Dev: That low latency is critical because it’s what makes the system viable for maintaining stability during high-agility maneuvers in formation control.

Taro: If the paper states its limitations, one thing they flag is that position-only estimation of the leader's state fails to facilitate stable hovering of a follower, which shows where this new method excels.

Rosa: So, to wrap up this segment on "Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation," the main improvement is clearly moving toward a system that uses tilt measurements and advanced filtering like the Z-KF to get superior, low-latency state estimation.

Dev: I think the main implication is that we need to prioritize latency reduction because position data alone just isn't cutting it for high-speed coordination tasks.

Taro: So, this research gives us a solid direction on how to build more capable estimation systems for complex aerial environments.

Michal Pliska, Matous Vrba, ůˇrej Vľta, Martin Jirousek, Viktor Walter, Ř Martin Saska

Czech Technical University in Prague

cs.RO

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: This work has been accepted to the IEEE for possible publication (IROS 2026)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: Agile multi-UAV flight requires accurate and low-latency onboard estimation of neighboring UAVs' kinematic states for critical tasks like collision avoidance and motion coordination.

Key concepts

Pose-Aware Estimation: This method uses tilt measurements from the camera to estimate a UAV's full 6D pose (position and orientation). By incorporating tilt data, the system gains crucial information about thrust direction, which constrains movement and leads to much more accurate state predictions compared to methods that only use position.
Position-Only Estimators: These traditional methods estimate a UAV's location using only its position coordinates. The paper found these estimators suffer from a constant delay in acceleration response, regardless of how agile the UAV is. This limitation prevents them from accurately tracking fast movements or coordinating complex maneuvers.
Z-Axis Measurement Kalman Filter (Z-KF): The Z-KF is a novel estimation technique derived using the Degenerate Kalman Filter framework. It specifically uses the tilt measurement to constrain thrust direction, allowing the filter to fuse measurements that are otherwise incompatible. This results in a faster, near-instantaneous response time for state inference compared to other methods.

Terminology

Kalman Filter — 19× in this paper
Fidelity — 9× in this paper · explained in 2 episodes
Accuracy — 8× in this paper · explained in 1 episode
Degenerate Kalman Filter — 7× in this paper
Benchmark — 3× in this paper · explained in 2 episodes
Benchmarking — 3× in this paper · explained in 3 episodes
Closed-Loop Control — 3× in this paper · explained in 3 episodes
Core — 3× in this paper · explained in 2 episodes
Unmanned Aerial Vehicle — 3× in this paper
Adaptation — 2× in this paper · explained in 1 episode
Constant Velocity — 2× in this paper
Mean Error Norm — 2× in this paper
Active Vision — 1× in this paper · explained in 1 episode
Constant Acceleration — 1× in this paper
Domain Adaptation — 1× in this paper · explained in 5 episodes
Loss Function — 1× in this paper · explained in 2 episodes
Non-linear Model Predictive Control — 1× in this paper
Pseudo-Linear Kalman Filter — 1× in this paper
Regularization — 1× in this paper · explained in 3 episodes

Summary

Agile multi-UAV flight requires accurate and low-latency onboard estimation of neighboring UAVs' kinematic states for critical tasks like collision avoidance and motion coordination. This work presents a systematic comparison between position-only and pose-aware vision-based state estimation methods for co-planar multirotor UAVs across various agility levels, demonstrating that integrating tilt measurements significantly improves estimation accuracy and enables higher degrees of maneuverability necessary for truly agile flight.

Key Findings on Estimation Methods

The research systematically compares four position-only estimators and five pose-aware estimators, including a novel formulation of a linear thrust-constraining Kalman Filter (KF). The core finding is that pose-aware estimation consistently reduces the average velocity and acceleration estimation errors by 40% and 57% across the three tested datasets. Specifically, Position-only filters exhibit a constant ∼300 ms delay in acceleration step response independent of agility, whereas tilt-constrained estimators operate near the physical response limit given by the camera frame-rate by observing the change in thrust direction before the displacement accumulates.

Methodology for State Estimation

The paper evaluates state estimation using a full monocular perception-to-estimation pipeline, which involves:

Utilizing deep learning architectures, specifically YOLOv5-6D, to obtain full 6D poses of the detected objects and PnP for pose extraction.
Employing different Kalman Filter variants, including the baseline Point Mass-Model KF (CV and CA), a PoseKF estimator incorporating tilt measurements into the process noise covariance matrix, and a novel Z-Axis Measurement Kalman Filter (Z-KF) formulated using the Degenerate Kalman Filter (DKF) framework.
Benchmarking these estimators across different agility regimes, ranging from 3 to 21 m s−2 acceleration levels, using real-world datasets (Phantom 4 and Mavic 2) and a high-fidelity simulated dataset (FlightForge).

The Proposed Z-KF Formulation

The Z-KF is derived by representing the tilt measurement as the z-axis vector of the UAV's body frame, which constrains the direction of thrust acceleration. This constraint is incorporated into a modified measurement model:

(Equation 9)

(Equation 10)

The paper introduces the Degenerate Kalman Filter (DKF) framework to fuse these affine subspace measurements by transforming them into a standard linear KF form. The DKF framework allows the fusion of measurements that are not directly compatible with conventional KF structures, leading to the Z-KF, which is shown to outperform other state-of-the-art methods.

Performance Across Agility and Scenarios

The performance evaluation is conducted using a Mean Error Norm (MEN) metric across position, velocity, and acceleration states.

(Table II)

The results show that while position accuracy remains relatively constant across all methods (0.10 m), the relative improvement of pose-aware estimators over position-only variants ranges from 3% to 60% depending on the dataset and state being measured.

(Figure 3)

In the performance over agility section, Pose-aware filters outperform position-only variants at every agility level with a consistently smaller estimation error and error variance. The Z-KF+BDC estimator achieves the overall lowest velocity and acceleration errors on the Unreal dataset, reducing MEN by 42% and 58% relative to the best position-only variant.

Impact on Closed-Loop Control

The study validates the estimation quality in a closed-loop leader–follower simulated experiment using Non-linear Model Predictive Control (NMPC). The results demonstrate that position-only estimation of the leader’s state fails to facilitate stable hovering of the follower, whereas the proposed estimator enables tracking of lateral maneuvers exceeding 2 g of acceleration. This confirms that pose-aware relative state estimation is a necessary element for realizing truly agile multiUAV motion coordination approaching the dynamic limits of individual UAVs.

Estimation Latency Analysis

The paper quantifies the impact on low-latency requirements by comparing the reaction time of estimators. The CA-KF+BDC introduces an approximately constant overhead of ∼300 ms (corresponding to ∼7–8 frames) above the physical onset, independent of agility level, indicating a structural bottleneck in position-based estimation. In contrast, the Z-KF+BDC responds near-instantly, showing that thrust-direction sensing allows for faster state inference. This low latency is critical for maintaining stability during high-agility maneuvers in formation control.

Conclusion and Contributions

The work provides the first systematic comparison of pose-aware and position-only multirotor UAV state estimation across different agility levels. The key contributions include:

Providing the first systematic comparison of pose-aware versus position-only vision-based state estimation.

Improvements for AI systems

Based on the provided scientific paper, here are specific improvements that can be made to AI systems, categorized by where they apply:

) Improved AI System Capabilities:

The core improvement is moving from position-only state estimation to pose-aware state estimation using tilt measurements. This enables the AI system to achieve significantly higher performance in dynamic, multi-agent scenarios. Specifically, the improved AI system can perform the following:

Achieve True Agile Multi-UAV Motion Coordination: The system can successfully track and coordinate neighboring UAVs during aggressive maneuvers (lateral accelerations exceeding 2g) without lag or instability.
Enable Stable Cooperative Flight in Communication-Free Formations: The system can maintain stable hovering and accurately track leader trajectories through complex lateral maneuvers, which was impossible with the position-only estimator.
Reduce State Estimation Errors by Significant Margins: The AI system can achieve a 40% to 60% reduction in average velocity and acceleration estimation errors compared to position-only methods across various agility levels.
Operate Near Physical Response Limits Latency-Free: The system can operate with an estimation delay near the physical frame-rate limit of the camera, allowing for near real-time reaction to sudden changes in thrust direction, rather than suffering from a fixed structural delay (e.g., 300 ms) inherent in position-only filters.
Improve Control Loop Stability: In closed-loop leader–follower systems, the improved estimation allows the follower's NMPC control to function effectively, preventing instability and enabling the follower to execute maneuvers exceeding 2g of acceleration.

) Specific Technical Improvements for AI/Control Architectures:

Implement Z-KF (Degenerate Kalman Filter) for Pose Estimation: The system should adopt the proposed Z-KF framework, which uses a specific formulation derived from the DKF to handle affine subspace measurements (like thrust direction). This replaces standard position-only KFs and provides a mathematically rigorous way to fuse positional data with directional constraints.
Integrate Tilt Measurement as a Constraint in State Estimation: The AI pipeline should explicitly use the tilt measurement (thrust direction) as an input constraint within the state estimator (e.g., via the DKF formulation). This directly informs the estimation of higher-order states (velocity and acceleration) that are otherwise inferred indirectly from displacement.
Develop a Robust 6D Pose Detection Pipeline: The vision component should utilize architectures like YOLOv5-6D combined with PnP to extract full 6D poses, ensuring the input to the state estimator is as rich as possible (position and orientation).
Utilize Adaptive Process Noise Modeling (BDC): The Kalman Filter structure should incorporate Block-Diagonal Covariance (BDC) process noise models. This allows the AI system to be more robust against unmodeled dynamics and cross-correlations between position, velocity, and acceleration states, leading to better performance under challenging conditions.
Implement Automated Parameter Tuning (CMA-ES): The initialization of the filters should use advanced optimization techniques like CMA-ES on dedicated tuning subsets for each dataset to ensure that the resulting estimator parameters are optimally tuned for a specific agility regime, rather than relying on empirical, fixed tuning procedures.

) Summary of System Impact:

The improved AI system transitions from a reactive, lag-prone estimator to a high-fidelity, predictive state estimator capable of supporting autonomous agents operating at the dynamic limits of physical UAV platforms. This shift allows for the deployment of communication-free multi-UAV systems that exhibit unprecedented agility and coordination.