DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation

DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation introduces a benchmark for evaluating large language models on the open-ended generation of interactive analytic dashboards. The paper states: We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards, together with datasets, task documents, and anonymized human baselines.

The benchmark's key innovation is requiring each system to generate both a dashboard and a replayable interaction trajectory: "Its key innovation is to require each system to generate both a dashboard and a replayable interaction trajectory. A browser executor replays the trajectory and turns the system's intended analytical workflow into reproducible visual and execution evidence. A VLM judge compares candidates using this evidence, and Bradley–Terry aggregation produces the leaderboard."

The paper explains the motivation: "Evaluating this capability is difficult because dashboard generation is open-ended. Multiple layouts, chart selections, and interaction designs may validly support the same analytical goal, so matching a single human reference or a fixed component checklist can penalize reasonable alternatives. Static screenshots reveal visual quality but not whether interactions work. Conversely, programmatic execution checks can detect broken actions but cannot determine whether a functional dashboard answers the task well."

The benchmark construction process is described: "We use high-quality human-authored dashboards as task seeds and ask a VLM to observe their pages, data schema, and interaction features extracted from code, then express the underlying user context, analytical goals, and desired interactions as a natural-language task. The task describes the intended analysis instead of requiring reproduction of the original dashboard. The final benchmark contains 234 tasks spanning 14 topic clusters and a broad range of structural complexity."

For candidate generation, Each model receives the task document and data schema and must return one complete HTML document. For comparable browser evaluation, we require ECharts and stable data-test-id attributes on interactive elements. The model then produces a structured interaction trajectory describing a meaningful analytical walkthrough in a second turn.

The evaluation protocol uses pairwise judgments: "We use pairwise judgments rather than assign each candidate an absolute score. A pointwise scale requires globally calibrated anchors... Pairwise evaluation conditions on the same task and data and asks which candidate better supports the requested analysis, avoiding cross-task scale drift. The VLM judge compares two anonymous candidates for the same task. Its input contains the task instructions, dataset schema, page-level screenshots, and trajectory execution reports."

The paper distills the judge: We then finetune DashJudge-8B, a VLM judge based on Qwen3-VL-8B-Instruct, using 255 independent teacher-labeled pairs with a data augmentation strategy that swaps candidate order and reverses the label. Training details: We train for two epochs with LoRA rank 16, scale 32, dropout 0.05, learning rate 10−4, effective batch size 8, and a cosine schedule. It takes approximately 50 minutes to complete training on a single 80 GB A100 GPU.

Human validation results: we invited six independent annotators to label 100 stratified candidate pairs, each with three annotations. On the 99 evaluable pairs, DashJudge-8B obtains 79.8% human agreement and κ = 0.600. The paper reports: DashJudge-8B improves over its base model by 9.1 agreement points and raises Cohen's κ from 0.419 to 0.600, while its human agreement is comparable to the proprietary teacher.

Ablation results show the importance of interaction evidence: "Ablating trajectories and their execution reports lowers agreement to 71.7%, while deterministic execution rules alone reach only 42.4%, showing that interaction evidence is useful but insufficient without multimodal semantic judgment. The paper states: Renderability and replayability checks alone predict a tie for 89 of 99 pairs and obtain only 8.1% human agreement. The full rule reaches 42.4% agreement and κ = 0.095, far below DashJudge-8B's 79.8% and κ = 0.600 on the same 99 pairs."

The main leaderboard evaluates seven models: We evaluate GPT-5.5, Claude Opus 4.6, GLM-5.2, Gemini 3.5 Flash, DeepSeek V4 Pro, Kimi K2.7 Code, and Grok 4.3, with one candidate per model and task. Results: "GPT-5.5 ranks first, although its CI overlaps the human baseline's. The human baseline and GLM-5.2 occupy the next two positions with overlapping uncertainty, while Claude, DeepSeek, Gemini, and Kimi form a broad middle tier. Grok 4.3 is clearly last."

Deterministic reliability measures show: "No model exceeds an 86% render rate or a 74% replay rate, showing that high pairwise quality does not eliminate basic runtime and interaction failures. GPT-5.5 leads both measures, while GLM's high preference rank coexists with relatively low functional reliability." Specific rates: GPT-5.5 has 85.8% render and 74.2% replay; GLM-5.2 has 82.5% render and 63.3% replay; Claude Opus 4.6 has 83.3% render and 54.2% replay; DeepSeek V4 Pro has 84.2% render and 72.5% replay; Gemini 3.5 Flash has 73.3% render and 54.2% replay; Kimi K2.7 Code has 62.5% render and 47.5% replay; Grok 4.3 has 67.5% render and 54.2% replay.

A trajectory-coverage audit over 100 candidates found: Valid trajectory targets 99.8%, Per-page control coverage 80.8%, All pages visited 96.0%, Authored-control downstream response 90.1%, Unauthored-control downstream response 86.0%. The paper notes: "The gap between the 80.8% per-page coverage and a theoretical 100% is largely due to the single-trajectory length limit and the common practice of generating multiple parallel controls (e.g., several side-by-side filters) but using only one in the trajectory."

A qualitative failure audit of 100 candidates found: 67 of 70 (95.7%) signal-enriched candidates contain a confirmed model-attributed defect and 21 of 30 execution-clean candidates contain semantic, analytical, interaction, or visual defects that deterministic browser checks are not designed to detect. The paper explains: "For example, these candidates concatenate overlapping components, treat a filtered-away comparison group as zero and report a false value, or replace a requested county choropleth with a non-geographic rectangular heatmap."

Robustness checks show: Replacing the logistic link with the Gaussian link of Thurstone–Mosteller, or simply ranking models by their observed average win rates, produces exactly the same eight-model order as BT. Also: Giving every task equal weight preserves the full ordering, while equal topic weighting swaps only the overlapping DeepSeek and Gemini positions.

The paper acknowledges limitations: "First, Tableau Public carries platform-specific biases, leaving private and other visualization ecosystems outside the current scope. Second, model-authored trajectories enable deterministic replay without relying on an exploratory agent, but restrict direct evidence to the submitted walkthrough. Third, budget constraints limit our experiments to model-only, single-conversation generation rather than iterative coding-agent harnesses."

The contributions are summarized as: "We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards, together with datasets, task documents, and anonymized human baselines. We develop an interaction-grounded evaluation methodology that turns generator-authored, replayable interaction trajectories into execution evidence for multimodal pairwise judgment. We release the distilled open-weight judge, DashJudge-8B, as a reproducible and efficient implementation of this methodology. We conduct a comprehensive empirical study of seven frontier models and the human baseline, supported by a human calibration study, deterministic reliability metrics, aggregation diagnostics, robustness checks, and a 100-case failure audit."

Based on the paper, here are the specific improvements I can make to AI systems:

1. Multi-turn generation with execution-aware self-correction

I can generate a dashboard and a replayable interaction trajectory in two turns, then use the execution reports from replaying that trajectory to identify and fix broken interactions or rendering failures before final submission. This would improve render and replay rates beyond the current 86% and 74% ceilings.

2. Interaction-evidence-grounded evaluation

I can learn to judge candidate outputs not just from static screenshots but from replayable trajectory execution reports, combining deterministic checks (e.g., whether controls respond) with multimodal semantic judgment. This lets me distinguish a functional dashboard from a visually appealing but broken one, and catch semantic defects like treating a filtered-away group as zero or replacing a choropleth with a non-geographic heatmap.

3. Pairwise preference learning with label augmentation

I can train on pairwise comparisons with swapped-order and reversed-label augmentation to learn robust relative quality judgments, avoiding the need for globally calibrated absolute scores. This yields higher human agreement (79.8% vs. 42.4% for rule-based checks) and better calibration (κ=0.600 vs. 0.095).

4. Trajectory-coverage optimization

I can generate interaction trajectories that maximize per-page control coverage (currently 80.8%) by explicitly planning to visit all pages and exercise multiple parallel controls (e.g., side-by-side filters) rather than only one, while staying within length limits. This improves the completeness of the analytical walkthrough.

5. Functional reliability prioritization

I can balance pairwise quality with deterministic reliability, since high preference rank (e.g., GLM-5.2) can coexist with low replay rates (63.3%). I can add a post-generation pass that checks renderability and replayability, then repair failures to ensure at least 86% render and 74% replay rates before evaluation.

6. Failure-mode-aware generation

I can avoid common model-attributed defects found in the audit: overlapping components, false values from filtered-away comparison groups, and replacing geographic visualizations with non-geographic alternatives. I can add explicit constraints for spatial data types and aggregation logic during generation.

7. Robust aggregation and uncertainty handling

I can use Bradley–Terry aggregation with confidence intervals to report rankings that account for overlap (e.g., GPT-5.5 vs. human baseline), and verify robustness across link functions (logistic vs. Gaussian) and weighting schemes (task vs. topic) to avoid overclaiming rank differences.

8. Open-ended task interpretation from code-derived features

I can infer user context, analytical goals, and desired interactions from data schemas and code-extracted interaction features, enabling me to generate dashboards that support the intended analysis rather than reproducing a specific reference, thus avoiding penalization of reasonable alternative layouts.

The improved AI system can:

• Generate interactive analytic dashboards with replayable walkthroughs that reliably render and respond to user actions.

• Self-audit its own output for semantic, interaction, and visual defects before deployment.

• Rank and select among multiple candidate solutions using interaction-grounded, human-aligned judgment.

• Produce dashboards that correctly handle filtered data, geographic mappings, and multi-control interactions.

• Provide uncertainty-aware performance estimates and robust leaderboard rankings across diverse tasks and topics.