cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Computer use agents (CUAs), which utilize graphical user interfaces to complete tasks, have recently surpassed human performance on many standard benchmarks, but their speed and cost remain significant barriers to widespread deployment. This work introduces cua-speedrun, a standardized platform that addresses the reproducibility crisis in CUA benchmarking by providing uniform infrastructure and task sets for evaluating agent speed and efficiency across different models.

The gist

cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks.

Standardization of Evaluation

The framework standardizes evaluations across four dimensions: (1) the agent driven by a large language model; (2) the benchmark defining the task; (3) the environment infrastructure managing VMs and action/observation contracts; and (4) the agent loop determining interaction. To ensure decoupling, comparisons fix two components while varying one, allowing for clean ablation. Speed is measured by recording time from when instructions are given until termination or limit is reached, separating agent time from environment operations.

Representative Task Selection

To minimize evaluation cost and time while preserving statistical power, the authors propose an energy minimization framework to select a minimal representative subset of tasks. This involves selecting a subset K that minimizes the energy distance between the distributions of agent scores on the full benchmark and the subset, aiming to minimize end-to-end evaluation time while preserving the signal of the results. They determine an optimal size K by ensuring Spearman correlation thresholds are met for (K-1), (K), and (K+1) to prevent selection based on chance.

Infrastructure and Agent Interface

The platform utilizes a serverless cloud-based provider with a uniform CUA evaluation infrastructure, ensuring all evaluations run on the exact same execution pipeline and the same virtual machine setup. A common agent interface is implemented to allow for portable agent harnesses and implementations, enabling a single implementation to operate across different benchmarks. The system supports both hosted evaluations via Modal sandboxes and local evaluations using self-hosted vLLM inference servers on fixed L40S GPUs.

Key Findings on Performance Trade-offs

The analysis reveals that no single model family is optimal for all three [speed, cost, capability], as the Pareto frontier shifts across benchmarks. Key insights include:

1. More reasoning effort can reduce both task time and model cost for CUAs because it avoids repeated unsuccessful actions.

2. Faster environment I/O can paradoxically slow down overall task completion time because models are not trained to interact with desktops in different I/O speed configurations.

3. The optimal reasoning setting is context-dependent: additional reasoning can be valuable on one (harder) benchmark and unnecessary on another (simpler) benchmark.

4. Batching actions can reduce task time, but this benefit depends on how well a model plans multi-action sequences; batching alone does not make an agent fast.

Practical Suggestions for Building CUAs

The research suggests practical improvements for developing efficient agents:

Choose a frontier configuration for the required performance and budget.

Tune reasoning effort rather than setting it to an extreme. Too little reasoning can lengthen trajectories, while too much can add time and cost without improving performance.

Batch actions when intermediate screenshots are unnecessary.

The paper concludes that the community should focus on turning our attention beyond performance and success rates, emphasizing the need to study reasoning and interaction together when developing faster agents.

Evaluation Metrics

The evaluation reports mean verifier score, mean task time, and mean model cost per task. The framework also tracks detailed interaction metrics: Interaction turns, generated tokens, and exact completion are defined in Appendix D. Token rates are calculated as generated-token count divided by total agent time, which includes the time spent on model requests and agent-side processing.

Benchmark Coverage

The platform evaluates agents across four different CUA benchmarks: OSWorld-Verified, OSWorld 2.0, CUA-World, and MyPCBench, covering long-horizon tasks and personalized computer use. The selection process ensures that the chosen task sets preserve model rankings by maintaining Spearman correlation thresholds of at least 0.95 for (K-1), (K), and (K+1) task counts.

Validation of Infrastructure

The infrastructure includes CUAAutoDebug, an end-to-end test suite to distinguish between model errors and harness errors, ensuring that the evaluation accurately reflects agent capability rather than execution failures. FastCUA is a new I/O system designed to reduce action-to-observation latency from seconds down to milliseconds.

Model Cost and Token Accounting

Model cost is computed by summing the cost of each task’s model requests, excluding environment hosting and setup time.

Based on the findings of CUA-SPEEDRUN, here are specific, actionable improvements for developing Computer Use Agents (CUAs) and what these enhanced systems can accomplish:

1. leungh and reasoning effort optimization:

A CUA should not blindly set reasoning effort to maximum. Instead, use the paper's insight that raising reasoning effort can improve benchmark scores while reducing task completion time.

• Improvement: Implement an adaptive reasoning setting mechanism where the agent starts at a low or medium setting and dynamically increases it only when performance plateaus or when the model is struggling with repeated unsuccessful actions.

• Improved System Capability: CUAs will complete complex, long-horizon tasks (e.g., multi-step software configuration) significantly faster and with higher success rates by strategically investing computational effort where it yields the highest return on time reduction, rather than just maximizing token generation.

2. Intelligent Action Batching:

The paper shows that batching actions is beneficial when intermediate screenshots are unnecessary, reducing total task time and token consumption.

• Improvement: Develop an action-batching heuristic that analyzes the current task state and environment feedback to decide whether to execute multiple related actions in a single request or wait for an observation. This should be prioritized over maximizing the token rate per step.

• Improved System Capability: CUAs will exhibit significantly reduced wall-time (up to 50% faster on OSWorld) by minimizing inefficient round trips between the agent and the environment, leading to more practical, real-time interaction capabilities for desktop automation.

3. Harness and Model Pairing Optimization:

The optimal model/harness combination is not universal; it depends on the benchmark and reasoning setting (e.g., GPT-6 Astra performs better with a direct API harness at high effort).

• Improvement: Create a dynamic Model Configuration Selector that uses the current benchmark profile to recommend the best harness (Direct API vs. Codex) for any given model and reasoning level, based on pre-computed performance trade-offs.

• Improved System Capability: The agent will achieve peak efficiency by leveraging specialized infrastructure; for example, when tackling a highly visual task (like those on OSWorld), it will automatically switch to the harness proven to handle that specific environment's I/O latency most effectively.

4. Environment Latency Mitigation (Fast CUA):

The paper demonstrates that optimizing I/O latency (using FastCUA) can sometimes be detrimental if not handled by the model; current models don't always wait optimally for UI updates.

• Improvement: Integrate a Fast I/O Mode that aggressively minimizes the time between action and observation, while simultaneously using model-specific predictive modeling to estimate necessary UI update delays.

• Improved System Capability: CUAs will be highly responsive in dynamic desktop environments, reducing perceived lag during tasks like real-time data monitoring or rapid application switching.

5. Task Selection via Energy Minimization:

Instead of running entire benchmarks, use the proposed energy minimization framework to select only the most representative subset of tasks for evaluation.

• Improvement: Implement an automated pre-processing step that uses the energy distance metric (Equation 2) to select a minimal set of tasks that preserves the performance ranking and statistical power across a full benchmark.

• Improved System Capability: The deployment pipeline will drastically reduce evaluation costs and time by focusing resources on tasks that provide the most informative signal, enabling faster iteration cycles for deploying new CUA models.

6. Robust Error Detection (CUA-AutoDebug):

Since agent failures can be model errors or harness errors, a robust verification layer is critical.

• Improvement: Integrate the CUA-AutoDebug testing suite directly into the agent's execution loop to distinguish between incorrect model actions and incorrect environmental execution (harness) in real-time.

• Improved System Capability: The system will achieve much higher reliability by instantly flagging when an action fails due to a flawed instruction (model error) versus a flawed translation of that instruction into a physical mouse click or keystroke (harness error), leading to faster debugging and more stable task completion.

Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.