TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

arXiv:2608.24145 · cs.CL, cs.SE · Submitted 2026-08-25 · Read on arXiv

cs.CL, cs.SE

Submitted: 2026-08-25

Updated: 2026-08-25

Comments: Code&Data: https://github.com/Skyorca/TrustDABench

Code: https://github.com/Skyorca/TrustDABench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Robustness — 124× in this paper · explained in 5 episodes
• Benchmark — 42× in this paper · explained in 2 episodes
• Text-to-SQL — 17× in this paper · explained in 3 episodes
• Abstention — 10× in this paper · explained in 1 episode
• Core — 10× in this paper · explained in 2 episodes
• Evidence Conflict — 10× in this paper
• Verification — 7× in this paper · explained in 2 episodes
• Aggregation — 6× in this paper · explained in 1 episode
• Benchmarking — 5× in this paper · explained in 4 episodes
• Consistency — 5× in this paper · explained in 3 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Header Conflict — 4× in this paper
• Provenance — 4× in this paper · explained in 4 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Deep Analysis Missing — 2× in this paper
• Hallucination — 2× in this paper · explained in 4 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Agentic Reasoning — 1× in this paper · explained in 1 episode
• Data Missing — 1× in this paper
• Evaluation Dimensions — 1× in this paper · explained in 3 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Feature Engineering — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Full Refusal Rate — 1× in this paper
• Grounding — 1× in this paper · explained in 3 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Partial Refusal Rate — 1× in this paper
• Representation Learning — 1× in this paper · explained in 2 episodes
• Robustness Accuracy Drop — 1× in this paper
• Structural Context Missing — 1× in this paper

Sources

• The Mighty ToRR: A Benchmark for Table Reasoning and Robustness
• Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis
• Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness
• PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries
• Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries
• Towards Robustness of Text-to-SQL Models against Synonym Substitution
• InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
• Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL
• Towards Fair In-Context Learning with Tabular Foundation Models
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
• Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systems
• TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring
• Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents
• Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attacks
• SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces
• SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
• Confidence Estimation for Text-to-SQL in Large Language Models
• Interpretable LLM-based Table Question Answering
• Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering