LLJ Cards: Best practices for the Use of LLMs as Judges

arXiv:2609.24516 · cs.CL · Submitted 2026-09-21 · Read on arXiv

cs.CL

Submitted: 2026-09-21

Updated: 2026-09-21

Comments: Prepared for conference submission

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Alignment — 11× in this paper · explained in 1 episode
• Benchmark — 9× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 7× in this paper · explained in 3 episodes
• Accountability — 5× in this paper · explained in 2 episodes
• Generalizability — 5× in this paper · explained in 6 episodes
• Grounding — 3× in this paper · explained in 3 episodes
• LLM judge — 3× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Prompt Engineering — 3× in this paper · explained in 4 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• Linguistic Annotation Workshop — 1× in this paper
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Training-Free Framework — 1× in this paper · explained in 2 episodes

Sources

• FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Constitutional AI: Harmlessness from AI Feedback
• Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
• Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
• Are We on the Right Way to Assessing LLM-as-a-Judge?
• Datasheets for Datasets
• A Survey on LLM-as-a-Judge
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Preference Leakage: A Contamination Problem in LLM-as-a-judge
• CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
• Red Teaming AI Red Teaming
• Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
• Evaluating Generative AI Systems is a Social Science Measurement Challenge
• A Red Teaming Roadmap Towards System-Level Safety

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering