Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes

arXiv:2606.07834 · cs.SE, cs.AI, cs.CL, cs.MA · Submitted 2026-06-05 · Read on arXiv

cs.SE, cs.AI, cs.CL, cs.MA

Submitted: 2026-06-05

Updated: 2026-09-12

Comments: Substantially revised and extended. The failure covers both non-directional verdicts the contract authorizes, not conflict alone. New probes separate recognition from commitment and bound the rates from below. The planned two-reviewer audit was not completed; all rates stay dataset-defined. 4 judges, 2 substrates. Code: https://github.com/HrxuAlbert/cherry-pick-override

Code: https://github.com/HrxuAlbert/cherry-pick-override

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Verification — 21× in this paper · explained in 2 episodes
• Abstention — 14× in this paper · explained in 1 episode
• Accuracy — 13× in this paper · explained in 1 episode
• Aggregation — 13× in this paper · explained in 1 episode
• Calibration — 11× in this paper · explained in 7 episodes
• Cherry-pick Override — 10× in this paper
• LLM judge — 8× in this paper · explained in 2 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Conformal Prediction — 5× in this paper · explained in 5 episodes
• Hallucination — 4× in this paper · explained in 3 episodes
• Empirical Methods in Natural Language Processing — 3× in this paper
• Trace — 3× in this paper · explained in 2 episodes
• Core — 2× in this paper · explained in 2 episodes
• Downstream Tasks — 2× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Verifier — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Accountability — 1× in this paper · explained in 2 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Homogeneity — 1× in this paper · explained in 2 episodes
• In International Conference on Machine Learning — 1× in this paper
• International Conference on Learning Representations — 1× in this paper
• Self-consistency — 1× in this paper · explained in 2 episodes
• Test-time Scaling — 1× in this paper · explained in 2 episodes
• Uncertainty Quantification — 1× in this paper · explained in 8 episodes

Sources

• A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queries
• No Certificate, No Categorical Speech Act: A Brouwerian Assertibility Constraint for Public Reason
• Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
• Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web
• Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties