Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies

summary

The gist

The gist The proposed VERT metric improves correlation with radiologists' judgments by up to 11.7% relative to GREEN, demonstrating that reliable evaluation can be achieved with lightweight

In short

The study evaluated existing LLM metrics for radiology report evaluation across different imaging types and body parts, not just chest X-rays. The proposed VERT metric showed improved correlation with expert judgments compared to current methods like GREEN, demonstrating that general-purpose LLMs can be effective judges when combined with light adaptation techniques.

Key concepts

LLM Judges: These are large language models used to automatically score or evaluate radiology reports based on expert standards. The research tests if these models can reliably judge reports from various medical images and body regions, moving beyond just chest X-rays.
VERT Metric: This is a newly proposed metric designed to better align LLM scores with human radiologist judgments. It was shown to improve correlation with expert ratings by up to 11.7% compared to the existing GREEN metric, suggesting a more accurate way to evaluate reports.
Lightweight Adaptation: This refers to small changes made to an LLM, such as ensembling scores or using parameter-efficient fine-tuning (LoRA). The research found that these simple adjustments significantly boost the performance of the LLM judges without requiring massive computational resources.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies · Read on arXiv

Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha

Cornell University · Microsoft Healthcare & Life Sciences

Current work on radiology report evaluation has focused primarily on LLM-based metrics for chest X-rays (CXR), leaving open whether such approaches remain robust across other modalities and anatomies, and which model and prompt configurations are best suited to serve as LLM judges. We present a comprehensive evaluation of LLM judges for radiology report evaluation across diverse model families, prompting strategies, and adaptation methods, leveraging two expert-annotated datasets that span multiple imaging modalities and anatomical regions. Specifically, we compare four existing metrics (RadFact, GREEN, FineRadScore, and CRIMSON) with our proposed metric, VERT, across open- and closed-source models of varying sizes and reasoning capabilities. Among existing metrics, GREEN proves the most robust, outperforming more recent CXR-specific approaches. VERT further improves Kendall's correlation with radiologist judgment by up to 26% over GREEN, while ensembling and lightweight fine-tuning of Qwen3-30B yield gains of up to 9.9% and 25%, respectively, of which the latter also has a latency reduction of 37.2x. A systematic error analysis further shows that, while these metrics correlate with expert ratings overall, they tend to miscount errors such as omissions. Overall, our results show that carefully selected general-purpose LLM judges provide strong radiology report evaluation across modalities, with lightweight adaptation offering complementary gains when annotated data is available.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies".

Jane: The gist The proposed VERT metric improves correlation with radiologists' judgments by up to 11.7% relative to GREEN,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title itself, "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies," tells you they are tackling a problem where the evaluation methods aren't general enough anymore. Jane, what do the authors actually want to achieve with this investigation?

Jane: They want to figure out which model and which prompting setups are the best choices when you need an AI judge for radiology reports that go beyond just chest X-rays <ref:2604.03376#pg1>.

Tom: So they're not just looking at one specific setup, but they’re doing a comprehensive study comparing open- and closed-source models of different sizes, including reasoning and nonreasoning variants <ref:2604.03376#pg3>.

Lu: They are testing zero-shot prompting using things like GPT-four point one mini and Gemini two point five flash, as well as formula-based prompts and rubric-based prompts to see what works best <ref:2604.03376#pg5>.

Meng: I’m curious about the cost here; running those different sized models and testing all those prompting strategies across multiple modalities sounds like it would be computationally expensive for a production system <ref:2604.03376#pg3>.

Lalam: The authors are also testing their own proposed metric, VERT, to see if it actually surpasses the existing methods they’re comparing it against <ref:2604.03376#pg2>.

Tom: Right, and that leads us into the summary of what they actually did in this paper about their results.

The paper's summary: Jane: Okay, so the summary points out that they conducted a thorough correlation analysis between expert and LLM-based ratings across two datasets called RadEval and RaTE-Eval <ref:2604.03376#pg3>.

Tom: And they compared the three existing metrics—RadFact, GREEN, and FineRadScore—against their new one, VERT <ref:2604.03376#pg1>.

Lu: The key finding mentioned is that VERT surpasses existing methods on RadEval and RaTE-Eval <ref:2604.03376#pg2>.

Jane: They also looked at how to improve the scoring process by investigating ensembling strategies, like averaging scores or using linear regression on inferences from five hundred samples <ref:2604.03376#pg6>.

Meng: Ensembling sounds like a way to get a more stable score when you’re dealing with inherently noisy LLM outputs, which is relevant for real-world deployment <ref:2604.03376#pg6>.

Lalam: They also looked at parameter-efficient fine-tuning using LoRA, specifically fine-tuning Qwen3 30B on the RaTE-Eval training set <ref:2604.03376#pg6>.

Tom: And they found that both ensembling and that LoRA fine-tuning showed improvements in correlation with expert judgments <ref:2604.03376#pg2>.

The paper's improvements: Tom: So what are the specific ways they suggest we can improve things? They point to these lightweight adaptation techniques as a way forward, even if prompt-based evaluation with generalist LLMs is still the starting point <ref:2604.03376#pg2>.

Jane: The authors suggest that you don't need massive retraining; you can get better results by using methods like ensembling and LoRA fine-tuning <ref:2604.03376#pg3>.

Lu: They also performed a systematic error injection study to see where the metrics align with human judgments, which helps identify areas of both high and low agreement <ref:2604.03376#pg11>.

Meng: That error detection part is crucial for production because it tells you exactly what kind of errors the AI judge is missing or misclassifying, like they did with error type c through f <ref:2604.03376#pg11>.

Lalam: For example, on the RaTE-Eval dataset, zero-shot and few-shot approaches got an F1 score of zero point nine zero to zero point nine two for finding errors of type a, but only zero point six nine to zero point seven two for type b, which is missing findings <ref:2604.03376#pg12>.

Tom: That gap between error types shows that even with good correlation scores, the metrics aren't perfectly balanced across all error categories <ref:2604.03376#pg12>.

Conclusion: Jane: So to wrap up, the main implication is that we can achieve reliable evaluation by combining prompt-based evaluation with these lightweight adaptation techniques like ensembling and LoRA fine-tuning <ref:2604.03376#pg3>.

Tom: They found that VERT improves correlation with radiologist judgments by up to eleven point seven percent relative to GREEN <ref:2604.03376#pg6>.

Lu: And the fine-tuning results for Qwen3 30B were quite strong, showing gains of up to twenty-five percent using only a thousand three hundred training samples <ref:2604.03376#pg2>.

Meng: I’m still focused on what this means for scaling; they also noted that the inference time was reduced up to thirty-seven point two times with that fine-tuning setup <ref:2604.03376#pg2>.

Lalam: The paper shows that while prompt-based evaluation should be prioritized, you can get better performance through these lightweight adaptation techniques <ref:2604.03376#pg3>.

Tom: We’re done with this one today. The paper "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies" shows us that getting reliable evaluation doesn't always require building a massive system; sometimes it just needs smart, lightweight adaptation techniques <ref:2604.03376#pg2>.

Jane: It certainly gives us more concrete tools to try when we’re looking at reports from different types of scans and anatomy <ref:2604.03376#pg1>.

More episodes

← Home