Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
cs.LG, cs.AI
Submitted: 2026-09-15
Updated: 2026-09-15
Comments: Project Page: https://zju-real.github.io/EvoSkill-GUI/ Code: https://github.com/ZJU-REAL/EvoSkill-GUI
Code: https://github.com/ZJU-REAL/EvoSkill-GUI
Project page: https://zju-real.github.io/EvoSkill-GUI
License: http://creativecommons.org/licenses/by/4.0/
The gist: GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution.
Terminology
Abstract
GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a reflect-revise-reuse loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.
Sources
- Experiential Reflective Learning for Self-Improving LLM Agents
- Qwen3-VL Technical Report
- CUA-Skill: Develop Skills for Computer Using Agent
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Memp: Exploring Agent Procedural Memory
- WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model
- XSkill: Continual Learning from Experience and Skills in Multimodal Agents
- MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
- SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
- MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
- GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
- UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
- A Survey on (M)LLM-Based GUI Agents
- UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning
- Self-Distilled Agentic Reinforcement Learning
- Dynamic Dual-Granularity Skill Bank for Agentic RL
- OpenCUA: Open Foundations for Computer-Use Agents
- SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks