Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

arXiv:2609.18204 · cs.CL, cs.AI, cs.CY, cs.HC · Submitted 2026-09-16 · Read on arXiv

cs.CL, cs.AI, cs.CY, cs.HC

Submitted: 2026-09-16

Updated: 2026-09-16

Comments: 11 pages, 4 figures, 3 tables. Accepted at the 60th Hawaii International Conference on System Sciences (HICSS)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Trace — 109× in this paper · explained in 2 episodes
Accuracy — 20× in this paper · explained in 1 episode
Verification — 17× in this paper · explained in 2 episodes
LLM-as-a-Judge — 7× in this paper · explained in 3 episodes
Calibration — 4× in this paper · explained in 7 episodes
LLM judge — 3× in this paper · explained in 2 episodes
Robustness — 3× in this paper · explained in 5 episodes
Sycophancy — 3× in this paper · explained in 4 episodes
Verifiability — 3× in this paper · explained in 2 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
Access Regimes — 1× in this paper · explained in 1 episode
Accountability — 1× in this paper · explained in 2 episodes
Agentic Retrieval — 1× in this paper · explained in 1 episode
Agentic System — 1× in this paper · explained in 2 episodes
Annotator Agreement — 1× in this paper · explained in 1 episode
Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
Clustering — 1× in this paper · explained in 2 episodes
Explainability — 1× in this paper · explained in 3 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Provenance — 1× in this paper · explained in 3 episodes
RAG — 1× in this paper · explained in 3 episodes
Verifier — 1× in this paper · explained in 2 episodes

Sources

Related papers