DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

arXiv:2609.31215 · cs.AI, stat.AP, stat.ME, stat.ML · Submitted 2026-09-25 · Read on arXiv

cs.AI, stat.AP, stat.ME, stat.ML

Submitted: 2026-09-25

Updated: 2026-09-25

Code: https://github.com/JinHongDu-Lab/DIAL

Terminology

• Calibration — 105× in this paper · explained in 7 episodes
• LLM judge — 19× in this paper · explained in 2 episodes
• Benchmark — 13× in this paper · explained in 2 episodes
• Consistency — 12× in this paper · explained in 3 episodes
• Alignment — 10× in this paper · explained in 1 episode
• Robustness — 10× in this paper · explained in 5 episodes
• Adaptive Weighting — 7× in this paper · explained in 1 episode
• Aggregation — 7× in this paper · explained in 1 episode
• LLM-as-a-Judge — 5× in this paper · explained in 3 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Post-Selection — 3× in this paper · explained in 2 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Uncertainty Quantification — 3× in this paper · explained in 8 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Fisher Information — 2× in this paper · explained in 3 episodes
• Unified Framework — 2× in this paper · explained in 5 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Failure Modes — 1× in this paper · explained in 3 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Longitudinal Data — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection