Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
cs.CL, cs.LG
Submitted: 2026-05-15
Updated: 2026-09-23
Comments: 50 pages
Code: https://github.com/nfelnlp/JuCi
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs.
Terminology
Abstract
LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs. a True/False label). Existing diagnoses of these format-induced inconsistencies stop at the input-output level. Using Position-aware Edge Attribution Patching (PEAP), we causally investigate the internal mechanism in five open-weight instruction-tuned models (Gemma-3, Qwen2.5, Llama-3.1) across five judgment tasks. We find that judgments across structured understanding and open-ended preference tasks share a sparse Latent Evaluator sub-graph in the mid-to-late multi-layer perceptrons (MLPs); zero-ablating it collapses judgment while preserving performance on our knowledge probes in architecturally modular models. By structurally decoupling abstract judging from output formatting, we provide a mechanistic account of format-induced inconsistency on the open-weight models we study: a continuous judgment signal computed in the shared trunk is mapped through fragile, format-specific terminal branches. The judgment itself can therefore be read out independently of the requested output format. Our findings imply that benchmark comparisons of judge reliability across formats partly measure the fragile formatting stage, and can understate the quality of the underlying evaluation.
Sources
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Reference-Free Rating of LLM Responses via Latent Information
- The Llama 3 Herd of Models
- RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Qwen2.5 Technical Report
- Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Path
- Gemma 3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering