Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline

arXiv:2609.30290 · cs.CL, cs.LG · Submitted 2026-09-09 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-09

Updated: 2026-09-09

Code: https://github.com/JamesL404/synca-audit

Terminology

• Hallucination — 17× in this paper · explained in 3 episodes
• Calibration — 15× in this paper · explained in 7 episodes
• Benchmark — 13× in this paper · explained in 2 episodes
• Text-to-SQL — 12× in this paper · explained in 3 episodes
• LLM judge — 6× in this paper · explained in 2 episodes
• Accuracy — 5× in this paper · explained in 1 episode
• Ablation — 4× in this paper · explained in 1 episode
• LLM-as-a-Judge — 4× in this paper · explained in 3 episodes
• Provenance — 4× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Abstention — 2× in this paper · explained in 1 episode
• Quantization — 2× in this paper · explained in 12 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• AUC — 1× in this paper · explained in 2 episodes
• Aggregation — 1× in this paper · explained in 1 episode
• Grounding — 1× in this paper · explained in 3 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• Model Drift — 1× in this paper · explained in 4 episodes
• Prompt Engineering — 1× in this paper · explained in 4 episodes

Sources

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
• To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation
• Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect
• Can LLM be a Personalized Judge?
• Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
• A Survey on LLM-as-a-Judge
• Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
• Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
• No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
• Auto-Prompt Ensemble for LLM Judge
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
• Semantic Agreement Enables Efficient Open-Ended LLM Cascades
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• Large Language Models are not Fair Evaluators
• Self-Instruct: Aligning Language Models with Self-Generated Instructions

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering