Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting

arXiv:2603.11938 · cs.AI, cs.CV, cs.LG · Submitted 2026-03-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting".

Tom: Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, summarizing what we just touched on, ProtoSR claims that by using an automatic extraction pipeline to build a multimodal knowledge base aligned with a structured template, they can create visual prototypes for each answer option. This knowledge base then feeds into a late fusion module that retrieves relevant prototypes to generate a support bias correction.

Jane: It’s important to stress that the paper isn't just about building the extraction part; it's about using those prototypes in the late fusion layer to generate a scaled static residual logit correction, which is then combined with the base model’s output for the final prediction.

Lu: The authors are tackling a specific limitation in automated structured reporting where fine-grained templates include many rare attributes but structured datasets are limited in size, providing sparse supervision for those attributes.

Meng: I'm interested in how they handle the integration of that knowledge; they state that this mechanism preserves the backbone decision pathway while enabling targeted corrections specifically where prototype evidence is informative.

Lalam: This really speaks to improving our AI culture because it shows we can create a feedback loop where unstructured clinical context actively guides structured outputs, making the whole system more context-aware.

Tom: It’s about transforming routine free-text reports from just descriptive text into an active signal that influences discrete per-field answer selection based on visual evidence.

Jane: And the results they show indicate that this integration yields consistent gains across different question levels, but the most notable improvements are seen at Level three which deals with those detailed attribute questions <ref:2603.11938#pg2>.

Lu: They tested this against a benchmark called Rad-ReStruct, which features three question levels: L1 for coarse abnormality existence, L2 for specific findings, and L3 for fine-grained attributes. The paper shows how the system performs across these hierarchical levels.

Meng: From an engineering view, the paper also points out that they treat this prototype bank as external memory and use periodic updates of prototype vectors using the current image encoder to maintain alignment with a continuously fine-tuned encoder.

Lalam: That continuous update aspect is very important; it suggests the knowledge base isn't static but evolves with our vision model, which keeps the system relevant in real-world medical imaging scenarios.

Tom: So, we’re seeing a system that learns to use prior clinical language context to sharpen its focus on those subtle details in complex diagnostic reports.

Jane: And this moves us closer to systems that can handle the complexity of real patient data by leveraging the vast amounts of descriptive text available implicitly in routine care.

Conclusion: Tom: So looking at the authors and the title again, ProtoSR is essentially a framework that uses extracted knowledge from clinical reports to guide a prediction model toward better fine-grained answers in radiology. It’s about using those rich descriptions as a kind of learned second opinion during the final decision making stage.

Jane: In simple terms, this means if the model is struggling with something subtle, like distinguishing between two very similar findings described differently in text, ProtoSR can pull in examples from past reports to help it pick the most accurate label.

Lu: The implications for future research are huge because it shows that we don't have to rely solely on having massive, perfectly labeled structured datasets when dealing with rare or complex attributes; we can augment our structured training with this kind of domain-specific knowledge infusion.

Meng: I think the practical impact is in creating more reliable AI tools for clinical workflows where precision matters, especially in areas where missing a single fine detail could have serious consequences.

Lalam: For our AI culture, this paper suggests that we can build systems that are more context-aware and less brittle when faced with the ambiguity inherent in real medical imaging descriptions.

Tom: Exactly; the paper demonstrates that leveraging existing, massive streams of free text data can provide a powerful, targeted correction mechanism for models struggling with long-tail decisions in structured tasks.

Jane: And while it’s an impressive technical achievement, we have to remember what the authors noted about their limitations: they focused on improving fine-grained attributes based on the Rad-ReStruct benchmark and didn't explicitly detail how this generalizes to entirely new domains without retraining.

Lu: That limitation is acknowledged; the method’s effectiveness seems tied closely to the specific knowledge base constructed for that task, so adapting it requires building new knowledge from scratch for different types of radiology.

Meng: From an engineering standpoint, if we want this to be widely adopted, we need better ways to automate that initial pipeline construction so it doesn't require manual setup every time.

Lalam: It’s a direction for us; the goal is to make this knowledge injection process more automated and less dependent on bespoke knowledge bases for every new application.

Tom: So, the big idea here is taking descriptive text and making it an active ingredient in the prediction pipeline to boost performance specifically on those hard, nuanced decisions that used to be impossible for current structured models.

Computer Aided Medical Procedures, Technische Universität München

cs.AI, cs.CV, cs.LG

Submitted: 2026-03-12

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decisions about rare

Key concepts

Knowledge Base Construction
This step automatically extracts structured information from free-text radiology reports. It involves expanding terminology using an LLM, extracting template-aligned findings, and filtering out inconsistent data to create a reliable pool of examples linked to specific image studies.
Prototype Bank Generation
For every category of finding, the system samples images from the knowledge base and creates a single prototype embedding. This is done by using element-wise max pooling on the sampled image encoder embeddings, effectively summarizing visual evidence into a representative vector for each label.
Knowledge-Enhanced Late Fusion Architecture
This module combines a base model's prediction with retrieved prototypes. It uses prototypes to summarize visual evidence and aggregate potential answers. This fused information is then used to predict a support bias, which is added to the base prediction via a learned scaling vector, allowing targeted corrections.
Late Fusion
Instead of merging predictions early, this framework waits until the main model makes an initial decision before incorporating knowledge from prototypes. The prototypes act as external memory that guides the final decision-making process, ensuring that specific visual evidence influences the output for complex questions.

Terminology

Summary

Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decisions about rare findings and attributes from limited structured supervision. ProtoSR proposes a prototype-conditioned late-fusion framework that leverages information extracted from routine free-text reports to improve fine-grained structured radiology reporting by injecting data-driven second opinions into the prediction pipeline.

Knowledge Base Construction

The approach begins with an automatic extraction pipeline designed to mine large volumes of free text and build a multimodal knowledge base aligned with a structured reporting template. This process involves several key steps:

  1. Terminology expansion: An instruction-tuned LLM is used to propose synonyms, abbreviations, and alternative phrasings, yielding a dictionary that maps description variants to the canonical label.

  2. Template-aligned extraction: The pipeline queries the LLM to decide if a finding corresponding to a template label is present in a report. If present, it extracts the corresponding attribute value(s) specified by the template, utilizing constrained decoding where only template-aligned answers are kept.

  3. Post-processing and Knowledge Base assembly: Rule-based filters are applied to reduce noise and enforce consistency with A’s ontology. Uncertain extractions are discarded, and hierarchical constraints are enforced by removing positive parent labels when none of their child labels are supported. Each retained tuple is then linked to its imaging study, yielding an example pool for each l ∈ L.

Prototype Bank Generation

Once the knowledge base is constructed, a prototype bank is created to represent visual evidence. For each label-specific example pool, the process involves:

"uniformly sample up to K images from each label-specific example pool and aggregate their image-encoder embeddings into a single prototype using element-wise max pooling, preserving the strongest signals across the sampled images."

Knowledge-Enhanced Late Fusion Architecture

The core of ProtoSR is a late-fusion module that augments the base structured reporting model. The process involves:

  1. Retrieval: Given an image and question, prototypes are retrieved from the knowledge base. The system calculates cosine similarity weights α between the projected fused representation (S) and prototype embeddings (P), considering only prototypes whose labels correspond to valid answer options for the current question.

  2. Evidence Summarization: This retrieval yields two vectors: a prototype feature vector (v), which summarizes retrieved visual evidence as a weighted average in the prototype embedding space, and an answer vector (u), which aggregates the corresponding one-hot prototype labels into a support vector with soft scores for all labels in the answer space Y.

  3. Bias Prediction: These two vectors are concatenated and transformed through an MLP to create a support bias (bsup):

bsup = MLP([v; u]) ∈ R Y

Late Fusion and Training

The final prediction is achieved by combining the backbone prediction with the knowledge-derived bias. This is done via a learned scaling vector s:

zfinal = zbase + s ⊙ bsup

This design preserves the backbone decision pathway and enables targeted corrections where prototype evidence is informative. The full model is trained end-to-end using the same multi-label objective as RadReStruct, treating the prototype bank as external memory, with periodic updates of prototype vectors using the current image encoder to maintain alignment with the continuously fine-tuned encoder.

Experimental Validation and Results

The method was evaluated on Rad-ReStruct, a benchmark featuring three question levels (L1 for coarse abnormality existence, L2 for specific findings, and L3 for fine-grained attributes). The extraction pipeline was compared across three instruction-tuned LLMs (Mistral 8B, Llama 3.1 8B, Qwen2.5 7B) with and without terminology expansion; the Qwen2.5-7B-Instruct model with terminology expansion achieved the strongest correctness results. The final ProtoSR model showed consistent gains across levels, with the strongest improvements on detailed attribute questions (L3), demonstrating that routine free-text reports can be leveraged as a knowledge signal to improve fine-grained understanding. Specifically, prototype-guided late fusion yielded a relative improvement of +72.1% at Level 3 compared to the base model without knowledge integration. The ablation study confirmed that the performance gains come from the content of retrieved prototypes rather than added fusion capacity, as replacing prototypes with Gaussian noise caused performance to fall back to baseline levels.

The gist: ProtoSR proposes a prototype-conditioned late-fusion framework that leverages information extracted from routine free-text reports to improve fine-grained structured radiology reporting by injecting data-driven second opinions into the prediction pipeline. This approach transforms paired images and free-text reports into an explicit prototype memory and learns to retrieve prototypes that influence discrete per-field answer selection, enabling targeted corrections of long-tail decisions.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems based on this scientific paper, along with what these improved systems can achieve:

  1. The core improvement is a shift from relying solely on large-scale, unstructured free text (like MIMIC-CXR reports) for knowledge injection to building a structured, prototype-aligned knowledge base.

  2. This enables the creation of a Prototype Knowledge Base where every fine-grained reporting option is associated with visual prototypes derived from real clinical reports.

  3. The system implements an LLM-driven mining pipeline (Terminology Expansion, Template-Aligned Extraction) to systematically convert raw free text into structured, label-linked examples aligned with the target reporting template.

  4. The resulting knowledge base allows the system to perform targeted retrieval of visually similar examples (prototypes) relevant to a specific image and question pair from this bank.

  5. The improved AI system incorporates a Prototype-Conditioned Late Fusion Module into its structured reporting backbone (Rad-ReStruct architecture). This module uses cosine similarity between the current image/question representation and the retrieved prototypes to generate an answer-aligned correction signal.

  6. The final prediction is not just a base model output, but a fused output where the prototype evidence is injected as a residual logit correction:

zfinal = zbase + s ⊙ bsup

  1. This allows the AI system to selectively correct long-tail or rare attribute decisions (like fine-grained location or appearance) by leveraging diverse, high-quality signals from the free text, without corrupting the overall prediction made by the base model.

  2. The improved system can achieve state-of-the-art performance on fine-grained attribute questions (L3), which are typically underrepresented and sparse in standard supervised datasets. This directly addresses the challenge of handling rare findings accurately and consistently during structured reporting.

  3. The system is robust against noise, as demonstrated by an ablation study showing that replacing prototypes with Gaussian noise causes performance to fall back to the baseline, confirming that only meaningful prototype structures are utilized for correction.

  4. The overall improved AI system can produce highly consistent, complete, and accurate structured radiology reports by systematically leveraging the implicit fine-grained knowledge encoded in millions of routine free-text clinical narratives.

Abstract

Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decisions about rare findings and attributes from limited structured supervision. In contrast, free-text reports are produced at scale in routine care and implicitly encode fine-grained, image-linked information through detailed descriptions. To leverage this unstructured knowledge, we propose ProtoSR, an approach for injecting free-text information into structured report population. First, we introduce an automatic extraction pipeline that uses an instruction-tuned LLM to mine 80k+ MIMIC-CXR studies and build a multimodal knowledge base aligned with a structured reporting template, representing each answer option with a visual prototype. Using this knowledge base, ProtoSR is trained to retrieve prototypes relevant for the current image-question pair and augment the model predictions through a prototype-conditioned residual, providing a data-driven second opinion that selectively corrects predictions. On the Rad-ReStruct benchmark, ProtoSR achieves state-of-the-art results, with the largest improvements on detailed attribute questions, demonstrating the value of integrating free-text derived signal for fine-grained image understanding.

Sources

Related papers