Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges

Graph-Structured Rubrics (GSR) compiles a rubric into a response-independent typed evaluation graph before observing candidate responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62–6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.

The paper introduces the first response-independent compilation framework that turns a rubric specification into a typed cross-criterion evaluation DAG with rubric-derived criterion nodes, operator nodes, named ports, and a unique sink. It defines execution semantics in which criterion-level judgments flow through deterministic TRANSFORM, REDUCE, and GATE operators in topological order. Static validation rejects malformed graphs. Task-specific Readout produces pointwise or pairwise decisions, while audit traces enable deterministic replay of composition.

The framework is demonstrated across four pointwise datasets and two pairwise preference benchmarks. Under GPT-OSS-120B, GSR achieves the numerically highest exact score agreement on all four pointwise datasets and the numerically highest end-to-end pairwise accuracy on both pairwise datasets; controlled ablations further show higher exact score agreement than direct scoring and weighted aggregation on every pointwise dataset.

GSR turns a hierarchical rubric policy into a graph program. Criterion nodes produce semantic judgments, operator nodes encode reductions and constraints, and edges fix their dependencies. Deterministic topological execution applies the declared rules in order rather than asking the LLM to reconstruct that order during every decision. For example, factuality and completeness can be reduced before a safety flag gates the result, preventing later positive evidence from overriding the cap. Pointwise scoring and pairwise preference share these node and operator semantics and differ only in task-specific Readout.

The compile-time specification S = (x, r, z, τ) contains the task input x, rubric r, optional fixed reference material z, and task contract τ, but excludes candidate responses and gold labels. The contract fixes the number of candidates m, a closed internal quality-score interval Uτ ⊂ R, output space Aτ, and quantization and tie policies. The execution stage subsequently receives X = (x, Y), where Y = (y1,..., ym) is an ordered candidate tuple with stable identifiers. This partition is enforced at the invocation boundary: compilation cannot access Y, while execution cannot alter the accepted program.

Compile uses an LLM to synthesize a declarative program from S under Λ. The model is restricted to catalogued types, criterion procedures, and operators, and emits nodes, named-port edges, parameters, and Rτ. A deterministic validator V checks JSON parsing, acyclicity, sink reachability, ports, arity, routing type compatibility, and readout compatibility. The first structurally valid program is accepted; no manual semantic screening or selection among valid candidates is performed. If none is valid after the fixed repair budget Krep, the instance is recorded as a compilation failure.

The graph G = (V, E, ρ) contains criterion nodes C, operator nodes O, type-checked routing edges E, port names P, and a unique sink ρ. An edge (u, p, l, ω) routes the judgment produced by predecessor node u to slot l of named port p of operator node ω. The graph permits criterion-to-operator and operator-to-operator edges; criterion nodes have no incoming judgment-flow edges. Compilation rejects cycles, missing ports, arity mismatches, and type-incompatible routes.

Criterion-node semantics are task-invariant. Compile instantiates one node for each semantic criterion selected from the rubric, and every criterion node returns candidate-aligned judgments for all m candidates. Pointwise evaluation sets m = 1, whereas pairwise evaluation sets m = 2. Within F, operators are grouped by the kind of composition they perform: a TRANSFORM operator maps Ja to Jb, a REDUCE operator maps J1 ×... × Jk to J, and a GATE operator maps J × B to J by applying a declared cap, mask, or veto.

Execute applies the fixed graph to X without changing its nodes, edges, or parameters. Each criterion node c ∈ C produces a candidate-aligned judgment vector jc = ((id(yi), jc,i))mi=1, jc,i ∈ Judg(c). Criterion procedures include LLM calls, rule checks, verifiers, or human annotations, but their outputs must conform to the criterion node's declared judgment type. After the criterion-level judgments are available, operators run in a topological order of G. The deterministic function fω receives a named, slot-ordered sequence at each port and applies the fixed parameters θω.

Readout is fixed during Compile and makes no additional LLM call. It receives sρ = ((id(yi), sρ,i))mi=1, checks identifier coverage, finiteness, and domain membership, and applies the deterministic task policy in Rτ. For pointwise evaluation (m = 1), dτ returns the native score or ordinal label obtained by the deterministic conversion and quantization policies in τ. For pairwise evaluation (m = 2), dτ(sρ) returns id(y1) if sρ,1 > sρ,2 + ϵτ, id(y2) if sρ,2 > sρ,1 + ϵτ, or resolveτ(id(y1), id(y2)) if sρ,1 − sρ,2 ≤ ϵτ, where ϵτ ≥ 0 and resolveτ are fixed by the task contract.

Pointwise datasets are UltraFeedback–TruthfulQA (3,244 responses), HelpSteer2 validation (1,038), SummEval Relevance (1,600), and BiGGen (2,776); their heterogeneous 1–5 targets are termed reference scores. Pairwise datasets are MT-Bench (2,575 labeled pairs) and RubricBench (1,147). Unless noted, GPT-OSS-120B is the judge. Pointwise baselines are controlled, same-backbone adaptations of Prometheus, G-Eval, and FLASK; pairwise baselines add OpenRubric, TICK, and CheckEval.

For pointwise scoring, GSR gives the best Exact Agreement, Within-1 Accuracy, and MAE in every dataset block. Relative to the strongest baseline by Exact Agreement, its gains are 5.44 points on UF–TruthfulQA, 4.78 on HelpSteer2, 0.62 on SummEval Relevance, and 0.99 on BiGGen. The corresponding Within-1 gains are 3.77, 3.43, 3.02, and 0.56 points, while MAE falls by 0.102, 0.083, 0.042, and 0.021. Across the four GSR rows, the six-run standard deviations for Exact Agreement are 0.45–0.84 points.

For pairwise evaluation, GSR attains the highest end-to-end Pairwise Accuracy at 99.87–99.91% coverage; TICK and CheckEval have higher valid-only accuracy but 19.57–34.66% tie rates, which are counted as non-decisions. The GSR Pairwise Accuracy standard deviation is 0.30 points on MT-Bench and 0.51 points on RubricBench.

In the ablation study, GSR improves Exact Agreement over Direct by 3.51–9.25 points. Against weighted aggregation, the gains are 2.60 points on BiGGen, 5.79 on UF–TruthfulQA, 3.26 on HelpSteer2, and 0.36 on SummEval Relevance. Because the latter comparison reuses the same criterion-level traces, these differences isolate composition and Readout rather than criterion elicitation. The flat weighted variant still has better Within-1 Accuracy on BiGGen and HelpSteer2, a marginally lower MAE on HelpSteer2, and sometimes stronger correlation, so GSR improves exact ordinal selection here but does not dominate a flat rule on all secondary metrics.

In cross-model sensitivity on HelpSteer2 with Qwen3.5-35B-A3B, GSR exceeds Prometheus-style by 0.25 points in Exact Agreement and 0.72 points in Within-1 Accuracy, while reducing MAE by 0.035. With GLM-4.7, its Exact Agreement is 1.04 points lower, but its Within-1 Accuracy is 3.58 points higher and its MAE is lower by 0.071. The graph interface therefore transfers structurally, but its metric profile is not backbone-invariant. The corresponding GSR Exact Agreement standard deviations are 0.74 points for Qwen3.5-35B-A3B and 0.87 points for GLM-4.7.

GSR composes criterion-level judgments through a typed graph shared by pointwise and pairwise evaluation. Under GPT-OSS-120B, GSR is numerically strongest on the primary end-to-end metric in all six evaluated datasets, while ablations support the value of graph composition and cross-model results bound that value by backbone dependence. GSR offers a programmable alternative whose composition step is trace-replayable.

Improvements to AI Systems:

1. Deterministic Rubric Execution: Replace implicit, in-context rubric following with a compiled, typed evaluation graph. The AI system pre-compiles any rubric into a fixed DAG of criterion nodes and operators (TRANSFORM, REDUCE, GATE) before seeing responses, then executes it in topological order. This eliminates the LLM’s need to reconstruct rubric logic during scoring, reducing variance and improving exact score agreement by 0.62–6.75 percentage points.

2. Static Validation of Evaluation Pipelines: The system rejects malformed or type-incompatible graphs at compile time (cycles, missing ports, arity mismatches, routing type errors). This ensures that any evaluation program is structurally sound before deployment, preventing silent failures or hallucinated scoring logic.

3. Unified Pointwise and Pairwise Evaluation: Use the same compiled graph for both single-response scoring and pairwise preference decisions, differing only in a task-specific Readout. The system handles ties and abstentions via a fixed threshold ϵτ and resolution policy, achieving 99.87–99.91% decision coverage on preference benchmarks (vs. 19–35% tie rates for baselines).

4. Trace-Replayable Audits: Every evaluation produces an audit trail of criterion-level judgments and operator outputs. The system can deterministically replay the entire composition, enabling verification, debugging, and human review of why a score or preference was assigned.

5. Composition Over Weighted Averaging: Replace flat weighted sums with typed operators that encode constraints (e.g., a safety GATE caps the final score regardless of later positive evidence). This improves exact ordinal selection by 3.51–9.25 points over direct scoring and by 0.36–5.79 points over weighted aggregation, while preserving Within-1 accuracy.

6. Backbone-Agnostic Interface: The graph program transfers across different LLM backbones (GPT-OSS-120B, Qwen3.5-35B-A3B, GLM-4.7) without recompilation, though metric profiles vary. The system can thus be deployed on cheaper or faster models while retaining structural guarantees.

What the Improved AI System Can Do:

• Reliable Grading: Score essays, summaries, or code against multi-criteria rubrics with higher exact-match accuracy and lower mean absolute error than direct prompting or weighted scoring.

• Safe Preference Selection: Choose between two responses with near-zero abstention, while enforcing hard constraints (e.g., veto unsafe content) via gating operators.

• Explainable Decisions: Provide a full, replayable trace of how each criterion judgment flowed through operators to produce the final score or preference.

• Flexible Policy Integration: Support custom operators (caps, masks, vetoes, reductions) and task contracts (quantization, tie-breaking) without retraining the LLM.

• Cross-Model Deployment: Run the same compiled rubric on different LLM backbones, trading cost/latency for accuracy, with predictable structural behavior.