EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

arXiv:2609.17081 · cs.CL · Submitted 2026-09-15 · Read on arXiv

cs.CL

Submitted: 2026-09-15

Updated: 2026-09-15

Comments: Accepted as an archival short paper in GroundLM Findings at EMNLP 2026; to appear in the GroundLM 2026 workshop proceedings in the ACL Anthology

License: http://creativecommons.org/licenses/by/4.0/

The gist: Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or

Terminology

Abstract

Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory. We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, distracting, or contradicting its evidence. EviScope-v1.1 contains 40 four-condition quartets with repaired counterfactual claims and span-level support labels for automatic evaluation. Across 960 gold-blind generations from Qwen2.5-7B, Llama 3.1 8B, and Gemini 3.5 Flash, paired metrics expose model-dependent grounding behavior that answer accuracy hides. On two local open models, an explicit evidence-action gate underperforms vanilla RAG on QCS: 0.15 vs. 0.50 for Qwen and 0.10 vs. 0.375 for Llama. Gemini reaches 0.944 joint success under both prompts, yet still answers 5% of conflict cases after contradiction insertion. EviScope therefore distinguishes unsupported answering, conflict blindness, and wrong non-answer actions rather than scoring answers alone.

Sources

Related papers