ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models

arXiv:2605.23694 · cs.CL · Submitted 2026-08-15 · Read on arXiv

Fen Wang, Zekai Shao, Qiman Kang, Chunran Hu, Zhixuan Zhang, Lexu Xie, Chao Liu, Siming Chen

cs.CL

Submitted: 2026-08-15

Updated: 2026-08-18

Code: https://github.com/wangfen01/ChartFI

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 46× in this paper · explained in 2 episodes
• Verification — 22× in this paper · explained in 2 episodes
• Hallucination — 19× in this paper · explained in 3 episodes
• Accuracy — 12× in this paper · explained in 1 episode
• Alignment — 10× in this paper · explained in 1 episode
• Core — 8× in this paper · explained in 2 episodes
• Consistency — 7× in this paper · explained in 3 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Cognitive Load — 3× in this paper · explained in 2 episodes
• Grounding — 3× in this paper · explained in 3 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Annotation Ambiguity — 1× in this paper · explained in 1 episode
• Calibration — 1× in this paper · explained in 7 episodes
• Causal Reasoning — 1× in this paper · explained in 2 episodes
• Cognitive Depth — 1× in this paper · explained in 2 episodes
• Evaluation Dimensions — 1× in this paper · explained in 3 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Multi-agent Collaboration — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Semantic Gap — 1× in this paper · explained in 2 episodes
• Semantic Similarity — 1× in this paper · explained in 2 episodes
• Systematic Bias — 1× in this paper · explained in 3 episodes
• Verifier — 1× in this paper · explained in 2 episodes
• Visual Consistency Score — 1× in this paper
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization
• Figure Captioning with Reasoning and Sequence-Level Training
• On the Use of ArXiv as a Dataset
• Benchmarking and Improving Detail Image Caption
• ChartLlama: A Multimodal LLM for Chart Understanding and Generation
• Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate
• The Semantic Scholar Open Data Platform
• ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
• ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning
• Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts
• BERTScore: Evaluating Text Generation with BERT

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering