Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction

arXiv:2609.25769 · cs.AI · Submitted 2026-09-22 · Read on arXiv

cs.AI

Submitted: 2026-09-22

Updated: 2026-09-22

Comments: Accepted to EMNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 15× in this paper · explained in 2 episodes
• Data Heterogeneity — 12× in this paper · explained in 3 episodes
• Alignment — 11× in this paper · explained in 1 episode
• Core — 9× in this paper · explained in 2 episodes
• Grounding — 9× in this paper · explained in 3 episodes
• Cosine Similarity — 7× in this paper · explained in 3 episodes
• Unified Framework — 6× in this paper · explained in 5 episodes
• Generalization — 5× in this paper · explained in 13 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Action Space — 4× in this paper · explained in 1 episode
• Accuracy — 3× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Consistency — 3× in this paper · explained in 3 episodes
• Graphical User Interface — 3× in this paper
• Trace — 3× in this paper · explained in 2 episodes
• Ablation Studies — 2× in this paper · explained in 1 episode
• Reasoning Traces — 2× in this paper · explained in 3 episodes
• Success Rate — 2× in this paper
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Self-distillation — 1× in this paper · explained in 6 episodes
• Structured Reasoning — 1× in this paper · explained in 3 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen2.5-VL Technical Report
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
• DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
• InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization
• Decoupled Weight Decay Regularization
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
• GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
• GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
• Falcon-UI: Understanding GUI Before Following User Instructions
• OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• GUI Agents with Foundation Models: A Comprehensive Survey
• MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
• TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents
• Android in the Zoo: Chain-of-Action-Thought for GUI Agents

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection