ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

arXiv:2608.26991 · cs.AI · Submitted 2026-08-27 · Read on arXiv

cs.AI

Submitted: 2026-08-27

Updated: 2026-08-27

Project page: https://sharryxr.github.io/ASIL

Terminology

• Benchmark — 49× in this paper · explained in 2 episodes
• Trace — 22× in this paper · explained in 2 episodes
• Grounding — 11× in this paper · explained in 3 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Action Space — 5× in this paper · explained in 1 episode
• Actor — 3× in this paper · explained in 3 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Verifier — 2× in this paper · explained in 2 episodes
• Visual Grounding — 2× in this paper · explained in 2 episodes
• Action Primitives — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Agentic Framework — 1× in this paper · explained in 6 episodes
• Aggregation — 1× in this paper · explained in 1 episode
• Annotation Errors — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 4 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Semantic Trajectories — 1× in this paper · explained in 2 episodes

Sources

• Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• Executable Code Actions Elicit Better LLM Agents
• OpenCUA: Open Foundations for Computer-Use Agents
• OmniParser for Pure Vision Based GUI Agent
• From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents
• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
• GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• CLI-Anything: Towards Agent-Native Computer Use
• UFO2: The Desktop AgentOS
• UFO: A UI-Focused Agent for Windows OS Interaction

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection