Accounting for Bias Enables Sustainable LLM Evaluation

arXiv:2609.31184 · cs.AI, cs.LG, stat.AP, stat.ME · Submitted 2026-09-25 · Read on arXiv

cs.AI, cs.LG, stat.AP, stat.ME

Submitted: 2026-09-25

Updated: 2026-09-25

Terminology

• Benchmark — 7× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 6× in this paper · explained in 3 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• LLM judge — 3× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Adaptive Sampling — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Fidelity — 1× in this paper · explained in 2 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Improving LLM Leaderboards with Psychometrical Methodology
• Revealing the structure of language model capabilities

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection