RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control
Di Lu, Bo Zhang, Xiyuan Li, Yongzhi Liao, Xuewen Dong, Yulong Shen, Zhiquan Liu, Jianfeng Ma
cs.CR, cs.AI, cs.RO, eess.SY
Submitted: 2026-07-20
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural
Terminology
Abstract
Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed agent iterations are often acceptable. UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic reasoning from onboard execution and security/safety enforcement. Slow cloud or edge reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated UAV actions.
Sources
- Taking Flight with Dialogue: Enabling Natural Language Control for PX4-based Drone Agent
- Next-Generation LLM for UAV: From Natural Language to Autonomous Flight
- A Prompt-driven Task Planning Method for Multi-drones based on Large Language Model
- A Universal Large Language Model -- Drone Command and Control Interface
- Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorial
- UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
- When Large Language Models Meet UAV Projects: An Empirical Study from Developers' Perspective
- TypeFly: Flying Drones with Large Language Model
- AerialClaw: An Open-Source Framework for LLM-Driven Autonomous Aerial Agents
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
- Code as Policies: Language Model Programs for Embodied Control
- ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- ChatGPT for Robotics: Design Principles and Model Abilities
- PaLM-E: An Embodied Multimodal Language Model
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs