InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

arXiv:2609.01383 · cs.CL, cs.CV, cs.HC · Submitted 2026-09-01 · Read on arXiv

cs.CL, cs.CV, cs.HC

Submitted: 2026-09-01

Updated: 2026-09-01

Comments: To be presented at EMNLP Main Conference

Code: https://github.com/maevehutch/insight

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of

Terminology

Abstract

Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.

Sources

Related papers