OSWorld-Pro: Process-based Evaluation for Computer Use Agents

arXiv:2609.24890 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-21 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-21

Updated: 2026-09-21

Comments: 27 pages, 7 figures

Code: https://github.com/xlang-ai/osworld

Project page: https://osworld-v1.xlang.ai

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Benchmark — 12× in this paper · explained in 2 episodes
LLM judge — 7× in this paper · explained in 2 episodes
Robustness — 7× in this paper · explained in 5 episodes
Verification — 7× in this paper · explained in 2 episodes
Adam — 5× in this paper · explained in 2 episodes
Benchmarking — 5× in this paper · explained in 3 episodes
Generative AI — 4× in this paper · explained in 2 episodes
Verifier — 4× in this paper · explained in 2 episodes
Failure Modes — 3× in this paper · explained in 3 episodes
Generalization — 3× in this paper · explained in 13 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Latent Reasoning — 1× in this paper · explained in 2 episodes
Reward Hacking — 1× in this paper · explained in 8 episodes
Trace — 1× in this paper · explained in 2 episodes

Sources

Related papers