VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

The gist The VISTA benchmark evaluates end-to-end web-app generation capabilities of LLM-based agents by targeting realistic UI-centric development from underspecified inputs.

Benchmark Design and Conditions

VISTA defines five prompt-information conditions that vary along two axes: visual/structural fidelity and stack constraint The five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint are (1) text only with free stack choice, (2) text with reference screenshots under three specified stacks, (3) text with reference screenshots under free stack choice, (4) text with screenshots and pruned Figma structure under a single specified stack, and (5) text with screenshots and pruned Figma structure under free stack choice. Table 1 shows that visual fidelity increases from C0 to C3/C4 (Figma structure), while Stack constraint varies independently.

Evaluation Methodology

To enable robust evaluation, each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points addressing the well-known limitations of script-based testing tools such as Playwright in openended code generation settings. Evaluation combines DOM-grounded reference matching, behavior-specific browser tests, and CLIP-based visual similarity jointly measuring structural alignment, behavioral completeness, and overall visual fidelity. The DOM-grounded evaluator jointly measures whether the generated interface preserves the reference structure and whether the matched elements implement the expected behavior.

Evaluation Metrics

The evaluation utilizes several metrics to assess agent performance. The structure-function score S is defined as S = 1/N Sum X N i=1 Li · Bi, where N is the number of critical annotated interactions. Visual fidelity is assessed using CLIP image similarity as a complementary page-level metric rather than the sole evidence of design fidelity. Furthermore, agent trajectory and efficiency are analyzed through command usage, file edits, tool calls, skill usage, build attempts, test attempts, and failure-repair cycles. The Surgical Diff Score quantifies how much of an agent’s editing work is delivered through localized patches rather than full-file rewrites.

Key Findings on Fidelity and Correctness

The results indicate that visual fidelity and functional correctness are partially decoupled across both input conditions and agents. Adding screenshots (C0 → C1) raises mean localization from 0.489 to 0.577 (+18% relative) but slightly drops Combined (0.242 → 0.229). Removing the stack constraint while keeping the same visual inputs (C1 → C2) raises Combined from 0.229 to 0.264, the largest single jump in the table. The two strongest conditions are therefore the free-stack ones, C2 (best mean Combined) and C4 (best Combined median and CLIP, with Combined within 0.003 of C2).

Agent Workflow Analysis

The analysis of agent workflows suggests a shared macro-pattern where agents spend more of the early trajectory inspecting context, concentrate writing in the middle of the task, and increase verification toward the end. Claude-family models show stronger inspect→inspect transitions and substantially higher failure→inspect probabilities consistent with more repeated context gathering and a clearer return-to-diagnosis pattern after errors. Conversely, GPT-family models show more dispersed transitions after failure and verification suggesting a more heterogeneous repair loop or more frequent movement through auxiliary workflow states. The Surgical Diff Score has a weak negative Pearson correlation with the Combined structure-function score (ρ = −0.145).

Editing Style and Task Quality

The relationship between editing style and task success is largely orthogonal across models. Opus has the highest Combined (0.261) but the lowest Surgical Diff Score (22.4) and a rewrite share of 0.988 indicating that its strong outputs are produced almost entirely through rewrites. GPT-5.5 shows the opposite profile with the lowest Combined (0.223) but highest Surgical (33.6) and Strict (9.0) Diff Scores. This suggests that the model-level inversion reflects different editing strategies aligned with harness versions rather than a general dependence between surgical editing and task quality.

Limitations

The benchmark covers ten application categories, 128 pages, 3,253 interactive annotations, and four agent systems drawn from two harnesses. The three tech stacks evaluated under C1 and the single stack reused in C3 are also proposed by an LLM rather than a human panel which biases the benchmark toward stacks that LLMs find plausible. The DOM-grounded evaluator relies on a per-axis affine alignment, IoU and center-distance tiers, and interaction-specific behavior probes so we treat S as a robust ranking signal rather than an absolute measure of correctness. CLIP captures high-level visual resemblance but does not verify exact spacing, typography, color tokens, or accessibility properties. The Surgical Diff Score is computed from harnesslevel file-mutation events making it an agent-system-level property rather than a claim about model-internal preferences. All four values are well below the threshold at which one would expect editing style to predict task success.

References

The references for this paper are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents. The paper also cites foundational work on visual grounding and design-to-code benchmarks. The references are listed in the final section of the document. The references include work on program synthesis with large language models, agent-computer interfaces, and multimodal agents.

1. Improve agent decision-making by incorporating explicit state management tools for task tracking, as demonstrated by Claude Code's stateful TodoWrite tool. This allows agents to maintain a structured checklist of progress (Explore inputs and choose stack, Scaffold Next.js app with TypeScript), leading to more deliberate and less exploratory workflows than those relying solely on natural-language agent message events.

2. Enhance the ability of agents to produce high-quality, executable front-end code by systematically optimizing for structural alignment alongside functional correctness. This is achieved by jointly measuring DOM-grounded reference matching (measuring if controls exist as real DOM elements and appear near the corresponding reference locations) and behavior-specific browser tests, resulting in a higher structure-function score S.

3. Improve visual fidelity in generated applications by utilizing CLIP-based visual similarity as a complementary metric to DOM grounding, ensuring agents prioritize overall visual resemblance while simultaneously verifying that annotated reference controls are present, grounded in the DOM, and structurally aligned with the mockup.

4. Refine agent editing style by training models on minimizing rewrite-heavy workflows and maximizing localized modifications. This is achieved by using the Surgical Diff Score, which quantifies how much of an agent’s editing work is delivered through localized patches rather than full-file rewrites, aiming to reduce high rewrite shares observed in top-performing models.

5. Improve stack selection accuracy by training agents to choose appropriate technologies based on application type and expected interaction patterns, as the benchmark uses LLMs to propose three suitable tech stacks per task. This moves agents beyond simply following a fixed stack template (C3) to making context-aware, task-dependent framework decisions.

Coding agents can now build applications from design mockups, but a screen that looks right is not an application that works. Existing benchmarks either score visual reconstruction without a backend or test full-stack functionality from textual requirements, and few tie scores to the individual components a design requires. We introduce VISTA (VIsual Spec-To-App), an end-to-end benchmark in which coding agents turn multi-page design handoffs (Figma renders, structure, and textual requirements) into runnable full-stack Web and Mobile (Android) applications. VISTA provides 8,487 human-annotated interactive components across 18 applications (126 Web pages and 576 Android screens) and an executable evaluator that locates annotated components in the running application and probes their interactions, yielding localization, behavior, and joint scores traceable to individual design requirements. Across 14 deployed coding-agent systems, the best joint score is 0.553 on Web and 0.378 on Mobile, and 13 of 14 systems score lower on behavior than on localization on Web. Component probes detect unresponsive controls in 90.1% of audited Web deliveries. Development traces show where the process departs from the task: 44.5% of trajectories never open the design screenshots, 73.0% omit the prescribed self-audit, and 8.3% report success after a failed check. VISTA makes the gap between rendering a design and delivering a working application measurable at the level of individual components. Code is available at.