Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization

arXiv:2609.02309 · cs.CL · Submitted 2026-09-02 · Read on arXiv

cs.CL

Submitted: 2026-09-02

Updated: 2026-09-02

Terminology

• Grounding — 30× in this paper · explained in 3 episodes
• Pruning — 19× in this paper · explained in 4 episodes
• Verifier — 17× in this paper · explained in 2 episodes
• Verification — 14× in this paper · explained in 2 episodes
• Abstraction — 13× in this paper · explained in 1 episode
• Benchmark — 13× in this paper · explained in 2 episodes
• Core — 12× in this paper · explained in 2 episodes
• Visual Grounding — 8× in this paper · explained in 2 episodes
• Actor — 5× in this paper · explained in 3 episodes
• Benchmarking — 4× in this paper · explained in 3 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Decode — 3× in this paper · explained in 2 episodes
• Structured Reasoning — 3× in this paper · explained in 3 episodes
• Agentic System — 2× in this paper · explained in 2 episodes
• Embeddings — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• KV cache — 2× in this paper · explained in 3 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Actionability — 1× in this paper · explained in 3 episodes
• Agentic Framework — 1× in this paper · explained in 6 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Amortization — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Continual Learning — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Hybrid System — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Test-time Scaling — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
• Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
• Agent S: An Open Agentic Framework that Uses Computers Like a Human
• Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
• ScreenAI: A Vision-Language Model for UI and Infographics Understanding
• Local Information Matters: Inference Acceleration For Grounded Conversation Generation Models Through Adaptive Local-Aware Token Pruning
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
• Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment
• Mind2Web: Towards a Generalist Agent for the Web
• Read More, Think More: Revisiting Observation Reduction for Web Agents
• Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
• OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• Efficient Multimodal Large Language Models: A Survey
• FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents
• LineRetriever: Planning-Aware Observation Reduction for Web Agents
• Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
• The BrowserGym Ecosystem for Web Agent Research

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering