CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

The gist The CiteVQA benchmark introduces an evaluation framework that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly.

How it works

1. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document.

2. The core evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct.

3. The benchmark mandates that models provide the precise PDF source supporting their answer at the granularity of element-level bounding-box citations, ensuring every generated claim is visually verifiable by human users.

4. A scalable, automated annotation pipeline was developed to generate ground-truth citations by identifying crucial evidence via masking ablation and validating them through expert review.

Key Findings on Model Performance

**- Auditing 20 MLLMs reveals a pervasive Attribution Hallucination, where models frequently produce the right answer while citing the wrong region. The strongest system, Gemini-3.1-Pro-Preview, achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. This exposes a reliability gap that answer-only evaluations overlook. The performance disparity shows that while models possess the perceptual capacity to extract information for a correct answer, they lack the ability to precisely link that information to its specific spatial source within the document. 5.3 Case Study To intuitively illustrate the disparity between linguistic performance and attribution accuracy—specifically why some powerful MLLMs achieve high Ans. but low SAA scores—we provide a case study in Figure 7. While Qwen3-VL-235B-A22B answers correctly (Ans.=1), it yields SAA=0 because its evidence crops are either blank or incomplete. 5.4 Impact of Multi-document Complexity The challenge of attribution is significantly exacerbated as the environment shifts from Single-Doc to Multi-Doc (N-Gold) settings. For instance, Gemini-3.1-Pro’s Recall drops from 68.9 in Single-Doc tasks to 55.3 in Multi (N-Gold) scenarios. This multi-gold setting consistently yields the lowest SAA scores across the board. 6 Conclusion We introduced CiteVQA, a benchmark designed to advance trustworthy document intelligence by requiring models to provide element-level visual citations alongside answers. By exposing these hidden hallucinations, CiteVQA establishes a rigorous standard for developing interpretable and reliable multimodal systems in high-stakes domains. 3. Contributions Our main contributions are threefold: • The CiteVQA Benchmark and Traceability Metrics: We introduce an evaluation framework that transitions Doc-VQA from answer-only scoring to joint evidence-answer verification, anchored by the Strict Attributed Accuracy (SAA) metric. • Scalable High-Fidelity Dataset Construction: We design an automated data generation pipeline that resolves the cost and consistency bottlenecks of granular visual annotation. • Discovery of the Attribution Hallucination Phenomenon: Through a comprehensive audit of 20 leading MLLMs, we expose a critical vulnerability where models frequently output correct text while grounding it in entirely incorrect visual evidence. The paper concludes that CiteVQA provides the instrumentation needed to close the reliability gap that answer-only evaluations overlook. 3.4 Dataset Overview and Analysis As summarized in Table 2 and Figures 3-4, CiteVQA is a diverse benchmark comprising 711 documents across 7 macro-domains, with a realistic average length of 40.6 pages. The questions cover varied scenarios including single-doc (52.0%), multi-doc with one gold document (25.7%), and multi-doc with multiple gold documents (22.3%). Evidence is uniformly distributed across document positions and often spans multiple pages, demanding robust long-context aggregation. 4 Evaluation 4.1 Evaluation Metrics To evaluate evidence attribution, we introduce a novel set of metrics assessing both answer correctness and trustworthiness in grounding predictions on verifiable evidence. The key metrics include Recall (Rec.), Relevance (Rel.), Answer Correctness (Ans.), and Strict Attributed Accuracy (SAA), defined as SAA = 1(Ans.≥4∧(Rel.≥4∨Rec.≥0.6)). 4.3 Main Results Table 3 presents a comprehensive evaluation of state-of-the-art MLLMs on CiteVQA. The analysis reveals several critical insights into the current state of faithful evidence attribution. The Attribution Hallucination Phenomenon A pervasive gap exists between answer accuracy (Ans.) and Strict Attributed Accuracy (SAA) across all tested models. Notably, while GPT-5.4 and Gemini-3-Flash achieve high answer scores (87.1 and 84.5), their SAA scores drop significantly to 59.0 and 65.4, respectively. Performance Disparity across Model Tiers There is a stark performance hierarchy among different model categories. Closed-source MLLMs dominate the benchmark, with Gemini-3.1-Pro-Preview leading at an Overall SAA of 76.0. In contrast, a significant cliff exists for Open-source Models, where the strongest (Qwen3-VL-235B) achieves an SAA of only 22.5. 4.4 More Results of Experiments Widespread Deficiency in Coarse-grained Attribution A striking observation from Table 12 is that Page-level Recall (Page.) remains remarkably low for the vast majority of models. This indicates that the failure in evidence attribution is not merely a consequence of weak fine-grained grounding, but a more fundamental inability to navigate to the correct document page. Impact of Multi-document Complexity The challenge of attribution is significantly exacerbated as the environment shifts from Single-Doc to Multi-Doc (N-Gold) settings. In these high-density scenarios, even top-tier models exhibit a sharp performance collapse. 5. Analysis & Discussion 5.1 Fine-grained Results Question Type Results show a significant performance gap between question types. Models excel in Quantitative Reasoning (e.g., Gemini-3.1-Pro-Preview at 82.6) because numerical computations rely on objective logic and offer clear alignment between evidence and answers. In contrast, the newly introduced Multimodal Parsing task remains a major bottleneck; this category requires models to locate specific document elements based on descriptive cues and subsequently parse the content, leading to substantial difficulties in both precise evidence attribution and final answer generation. 5.2 Further Analysis of Evidence Attribution Beyond the initial identification of attribution fallacies, we seek to further explore the nuanced relationship between Attribution and Accuracy Synergy between Attribution and Accuracy Beyond serving as a metric for trustworthiness, faithful attribution appears to be positively correlated with the model’s reasoning success.

1. Evaluate models using Strict Attributed Accuracy (SAA) to ensure predictions are grounded in correct evidence, as models frequently produce the right answer while citing the wrong region. This directly addresses Attribution Hallucination by rewarding models only when the answer and the cited region are both correct, which is critical for high-stakes domains.

2. Implement a scalable, automated annotation pipeline to generate high-fidelity, element-level bounding-box citations, as manual annotation is prohibitively expensive and prone to inconsistencies. This allows for the creation of a robust dataset comprising 1,897 complex queries across 711 multi-page, multi-domain PDFs, which improves the fidelity of document understanding.

3. Develop an inference system that utilizes element-level citation rules, requiring evidence to be at the element level: a complete paragraph, a complete table, a complete image, or a complete note. This prevents models from selecting partial text and ensures that evidence is precisely bounding box, which is vital for verifiable reasoning.

4. Introduce Traceability Metrics like Page-level Recall (Page.) to assess coarse-grained localization ability, as the paper notes that Page-level Recall (Page.) remains remarkably low for the vast majority of models. This helps diagnose whether failures are due to weak fine-grained grounding or a fundamental inability to navigate to the correct document page.

5. Design a multi-agent framework, inspired by DocDancer and AgenticOCR, that can perform element-level bijective function fmap mapping between synthetic layouts and original files, ensuring that every byte of synthesized evidence is traceable to its original page. This eliminates visual hallucinations by enforcing absolute fidelity in citation annotations.