Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection

arXiv:2608.00442 · cs.CV, cs.AI · Submitted 2026-08-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection".

Jane: The paper was written by Yibo Wan, Jinyu Cai and See-Kiong Ng from National University of Singapore.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back, everyone. We're digging into a fresh one from the arXiv this week, and the title alone is a mouthful — "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection." Jane, I need you to decode that for me.

Jane: Happy to, Tom. So "medical anomaly detection" is basically teaching a computer to spot what's wrong in a medical image — a tumor in a brain scan, a lesion in a liver CT, something off in a chest X-ray. And the big challenge is that you rarely have enough labeled examples of "wrong" to train on.

Tom: Right, and that's where the "static anchors" part comes in. Most existing methods use a fixed reference — like a text prompt saying "this is normal" or "this is abnormal" — and they compare every new image against that same reference.

Jane: Exactly. And the paper's core argument is that a fixed reference is fragile. A brain MRI looks completely different from a liver CT, so a single "normal" anchor trained on one domain might be way off for another. The authors propose something they call ReCAP, which adapts the reference for each individual image.

Tom: So instead of one-size-fits-all, it's like having a tailor who adjusts the measuring tape for every patient?

Jane: That's the gist. They call it "bounded prototype conditioning" — the system learns a normal and an abnormal prototype, then nudges those prototypes based on the current image's context. But here's the clever part: they deliberately limit how much the prototypes can move.

Tom: Why limit it? Wouldn't you want maximum flexibility?

Jane: Because the conditioning signal comes from the image itself. If the image contains a lesion, that lesion contaminates the context. An unconstrained model could cheat by shifting the "normal" prototype toward the lesion, making the anomaly look normal. The bound prevents that collapse.

Tom: That's a really subtle point. So they're saying, "adapt, but don't adapt so much that you explain away the very thing you're looking for."

Jane: Precisely. And the results back it up — they report the best zero-shot image-level AUROC on all six medical datasets they tested, and they cut inference time by over seventy percent compared to the fastest baseline.

Tom: Seventy percent faster and more accurate? That's the kind of headline that gets clinicians' attention.

Jane: It should. And the fact that it's language-free means you don't need to craft the perfect text prompt — which, as anyone who's worked with CLIP knows, is its own headache.

Tom: So we've got a method that's faster, more accurate, and doesn't rely on fragile text descriptions. I'm already curious about how they actually build these adaptive prototypes. Let's get into the summary next.

Summary: Tom: Back to "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection." Jane, give us the big-picture summary — what did these authors actually build?

Jane: So ReCAP is a framework that works in two settings. In zero-shot, you train on several medical domains, then test on a completely unseen one. In few-shot, you get a handful of target-domain examples — say, two or four normal and abnormal images — and you use those to adapt.

Tom: And the key innovation is replacing text prompts with visual prototypes, right?

Jane: Yes, but not just any visual prototypes. They're input-conditioned. For each query image, the model pools its global appearance — the modality, the dominant anatomy — and uses that to re-center the normal and abnormal prototypes. The re-centering is gated and bounded, so it adapts to the domain without drifting into the lesion.

Tom: So the prototypes are like adjustable anchors that get tuned per image. But they also have a second branch for few-shot, don't they?

Jane: They do. They build a non-parametric memory bank from the normal support images. Each query patch is compared against its nearest neighbor in that memory. If it's far from all normal patches, that's a signal of abnormality. So you have two complementary sources of evidence: the adaptive prototype boundary and the instance-level memory.

Tom: Two different ways of saying "this doesn't look normal" — one is a learned boundary, the other is a direct comparison to known normals.

Jane: Exactly. And they fuse those signals across multiple layers of the CLIP vision transformer. Different layers capture different levels of detail — early layers see edges and textures, later layers see more semantic structure. The fusion weights are learned, so the model figures out which layers matter most.

Tom: And the results? I remember they claimed best-in-class across the board.

Jane: On zero-shot, they got the best image-level AUROC on all six datasets — brain MRI, liver CT, retinal OCT, chest X-ray, histopathology. On pixel-level segmentation, which is localizing the anomaly on the image, they got the best AUROC on all three datasets that have those annotations. In few-shot, they won twenty-three out of twenty-four image-level settings.

Tom: Twenty-three out of twenty-four. That's not a fluke, that's a pattern.

Jane: And it's robust — they ran multiple seeds and the standard deviations were tiny, under half a percentage point. So the improvements aren't just luck of the draw.

Tom: So the architecture is: frozen CLIP encoder, lightweight adapters, conditional prototypes, a memory bank, and learned layer fusion. That's a lot of moving parts, but the efficiency numbers suggest it all runs fast.

Jane: Seventeen milliseconds per image in zero-shot. That's roughly four times faster than the language-free baseline they compared against.

Tom: Four times faster and more accurate. I want to know how they got there — let's look at the specific improvements they're claiming.

Improvements: Tom: Continuing with "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection." Jane, the paper lists three main contributions. Walk us through them.

Jane: The first is identifying the problem itself — that static anomaly anchors are a key limitation in cross-domain medical settings. That might sound obvious in hindsight, but the paper does a nice job showing why it matters. If your "normal" reference is fixed, and you move from brain MRI to liver CT, that reference is just wrong.

Tom: So the diagnosis comes first, then the treatment. What's the second contribution?

Jane: The bounded gated modulation. This is the heart of the method. They learn normal and abnormal prototypes, then modulate them based on the input image's context. The modulation is gated by a sigmoid and bounded by a tanh, so the prototypes can only move so far from their base position.

Tom: And that bound is what prevents the model from cheating by absorbing the lesion into the "normal" prototype.

Jane: Right. They actually ablate this — they tested an unbounded version and a bounded version without the gate. The unbounded version hurt performance, especially in zero-shot. The bounded gated version was the best. So the constraint isn't just a nice safety feature; it's essential for the method to work.

Tom: So the bound is doing real work, not just theoretical hand-waving. And the third contribution?

Jane: The normal-reference memory for few-shot. When you have a few target-domain support images, they build a memory bank of normal patches. Each query patch is scored by its nearest-neighbor distance to that memory. This preserves instance-level variation — the memory keeps many different normal patterns, not just one averaged prototype.

Tom: So the prototype gives you a compact boundary, and the memory gives you the full spread of what normal looks like in this specific domain.

Jane: Exactly. And they fuse both with learnable layer weights. The ablation shows that removing any of these three components — memory, conditioning, or layer fusion — hurts performance. The biggest drop came from removing layer fusion on the liver CT dataset, which makes sense because different layers capture different anomaly cues.

Tom: So each piece is pulling its weight. But I'm curious — how does this actually compare to the baselines in practice? We saw the numbers, but what's the qualitative story?

Jane: The paper includes visualizations. The competing methods produce diffuse heatmaps that spill into normal tissue, or they highlight the wrong structures entirely. ReCAP produces compact, well-aligned anomaly maps that match the ground truth much more closely.

Tom: So it's not just scoring better on a metric — it's actually localizing the lesion where the doctor would point.

Jane: That's the practical payoff. And it does it without text prompts, which means no sensitivity to wording, and without test-time gradient updates, which means it's fast.

Tom: Fast, accurate, and robust. Let's get into the first page of the paper itself — the abstract and the framing.

First Page: Tom: We're on "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection." Jane, we've covered the method and the results. Let's zoom into the first page — the abstract and the problem framing.

Jane: The abstract sets up the core tension really well. Medical anomalies are rare, diverse, and expensive to annotate. So you need a detector that works with scarce supervision and transfers across organs and modalities. The paper argues that existing CLIP-based methods, whether they use text prompts or learned visual tokens, share a common weakness: the reference is fixed after training.

Tom: Fixed reference, reused for every test image. And that's brittle when the target domain shifts.

Jane: Exactly. They list three specific limitations. First, text prompts like "normal" and "abnormal" are coarse and wording-sensitive. Medical lesions are subtle and hard to describe with generic language. Second, removing text doesn't solve cross-domain miscalibration — learned visual anchors can still define a boundary that's misplaced for an unseen target. Third, compact global anchors can't preserve the diverse instance-level normal patterns you need for fine-grained localization.

Tom: So the problem isn't just "text is bad" — it's that the whole paradigm of a single static reference is fundamentally limited.

Jane: Right. And that's why they propose ReCAP. The central idea is to replace a static reference with an input-conditioned visual prototype boundary. The prototypes re-center for each image based on its global visual context, producing an image-specific boundary without text prompts or test-time gradient updates.

Tom: And the "bounded" part is the safety mechanism. They say the conditioning signal is pooled from the input image itself, so it may be contaminated by the lesion. The bound prevents the boundary from shifting toward anomalous content and explaining it away.

Jane: That's the key insight on page one. They also introduce the few-shot complement: a non-parametric normal-reference memory that preserves instance-level target-domain variation. So you have the adaptive prototype branch and the memory branch working together.

Tom: And the efficiency claim — they mention over seventy percent reduction in inference latency compared to the fastest baseline. That's a bold claim for page one.

Jane: It is, and they back it up in the experiments. Seventeen milliseconds per image in zero-shot. That's fast enough for real-time screening workflows.

Tom: So the first page sets up the problem, the solution, and the headline results. It's a tight framing. But I want to hear what our other hosts think about the bigger implications.

Jane: Good idea. Let's bring in Lu and Meng.

Lu: I'll jump in. The thing that excites me most is the amortized adaptation. Instead of test-time optimization or fine-tuning per domain, they learn a single forward-pass mechanism that adapts the boundary. That's a design pattern that could generalize beyond medical imaging — any field with distribution shift and scarce labels.

Meng: And from an engineering standpoint, the efficiency is the story. Seventeen milliseconds means you could deploy this on edge devices in clinics, not just in server rooms. The fact that it's language-free also removes a whole class of failure modes — no prompt engineering, no text encoder to maintain.

Tom: So we've got a method that's theoretically interesting and practically deployable. That's a rare combination.

Jane: It really is. Let's wrap this up in the conclusion.

Conclusion: Tom: We're closing out our discussion of "Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection." Jane, give us the final summary.

Jane: So ReCAP replaces static anomaly anchors with input-conditioned visual prototypes. The prototypes are re-centered per image using a bounded gated modulation, which adapts to the domain without collapsing onto the lesion. In few-shot, a normal-reference memory adds instance-level detail, and learnable layer fusion combines evidence across the CLIP transformer.

Tom: And the results — best zero-shot image-level AUROC on all six datasets, best pixel-level on all three segmentation datasets, and twenty-three out of twenty-four few-shot wins.

Jane: Plus a seventy percent reduction in inference latency. It's faster, more accurate, and doesn't need text prompts.

Lu: The broader lesson for me is that adaptation doesn't have to be expensive. A bounded, amortized conditioning mechanism can capture domain shifts in a single forward pass. That's a template for other low-resource vision tasks.

Meng: And from deployment, the speed and the lack of text dependencies make it practical. You could see this in clinical decision support tools without a huge infrastructure investment.

Tom: The paper does note some limitations — it's focused on 2D images, and extending to volumetric or multimodal data is future work.

Jane: Right. But for what it sets out to do — language-free, adaptive, efficient medical anomaly detection — it delivers. We'll be watching for the volumetric extension.

Tom: That's a wrap on "Beyond Static Anchors." Thanks for joining us, and we'll see you for the next paper.

Jane: Take care, everyone.

Yibo Wan, Jinyu Cai, See-Kiong Ng

National University of Singapore

cs.CV, cs.AI

Submitted: 2026-08-10

Updated: 2026-08-11

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 69/100

The gist: The paper proposes ReCAP (Re-Centered Anomaly Prototypes), a language-free framework for zero- and few-shot medical anomaly detection.

Key concepts

Static Anchors
Existing methods use fixed references, such as text prompts, to compare new images against a single 'normal' or 'abnormal' example. The paper argues this is fragile because a reference trained on one medical domain may be incorrect for another domain.
Bounded Prototype Conditioning
This technique learns normal and abnormal prototypes and then adjusts them based on the current image's context. The adjustment is limited by a bound to ensure the prototypes do not move too far, preventing them from absorbing the anomaly into the 'normal' representation.
Normal-Reference Memory
For few-shot learning, this component builds a memory bank of normal patches from support images. A query patch is then compared to its nearest neighbor in this memory; if it is far from all normal patches, it signals an abnormality.

Terminology

Summary

The paper proposes ReCAP (Re-Centered Anomaly Prototypes), a language-free framework for zero- and few-shot medical anomaly detection. The authors identify that existing CLIP-based anomaly detection methods rely on static anomaly anchors—whether text prompts or learned visual tokens—that remain fixed after training and reused for all test images. They argue that such static scoring reference can be brittle in cross-domain medical AD, where the feature-space location of normality may vary sharply across target domains.

The paper identifies three limitations of the static-anchor formulation: (1) text prompts such as normal and abnormal provide coarse semantic supervision and are sensitive to wording, whereas medical lesions are often subtle, localized, and difficult to describe with generic language; (2) removing text does not solve cross-domain miscalibration because learned visual anchors can still define a fixed anomaly boundary that is misplaced for an unseen target; and (3) compact global anchors cannot preserve the diverse instance-level normal patterns needed for fine-grained localization.

The central idea of ReCAP is to replace a static anomaly reference with an input-conditioned visual prototype boundary. Specifically, ReCAP re-centers a learned and separated prototype pair for each image according to its global visual context, producing an image-specific boundary without text prompts or test-time gradient updates. This re-centering is deliberately bounded to mitigate excessive prototype drift, because the conditioning signal is pooled from the input image itself and may be contaminated by the very lesion we aim to detect.

The method uses a frozen CLIP visual encoder with lightweight segmentation and detection adapters inserted into selected transformer layers (layers 6, 12, 18, 24 for ViT-L/14). Each branch and layer maintains a normal prototype and abnormal prototype. The conditional modulation module computes an input context descriptor by average-pooling patch tokens, then applies bounded gated modulation via the formula: q̃ = Norm(q + σ(β) · η · tanh(W·c(x))), where the learnable gate σ(β) ∈ (0,1) and bounded tanh keep the residual small, so that the conditioned anchor stays close to the learned base prototype.

For few-shot detection, ReCAP builds a non-parametric normal-reference memory from normal support images, scoring each query patch by its nearest-neighbor cosine distance. Multi-layer fusion combines evidence from different CLIP layers using learnable layer weights, and the prototype and memory branches are fused with a weight λ (set to 0.5 by default). The training objective contains three terms: a pixel-level localization loss (focal-plus-Dice), an image-level detection loss (binary cross-entropy), and a prototype separation regularizer that enforces a margin between normal and abnormal prototypes.

Experiments are conducted on six medical datasets spanning five imaging domains: BrainMRI, LiverCT, RESC, OCT17, ChestXray, and HIS. Results show that ReCAP achieves the best image-level AUROC on all six datasets in the zero-shot setting and the best image-level AUROC on all 23 of 24 few-shot settings. For pixel-level segmentation, ReCAP achieves the best zero-shot pixel-level AUROC on all three segmentation datasets and the best pixel-level F1-max on all three segmentation datasets across few-shot settings. Notably, ReCAP reduces inference latency by over 70% compared to the fastest baseline, requiring only 17.4 ms per image in zero-shot and 19.0 ms in four-shot settings, while using fewer method-specific parameters (22.0M vs. 31.5M) than the comparable language-free baseline VisualAD.

Ablation studies confirm the contributions of the normal memory bank, context-conditioning module, and learnable layer fusion. The authors also demonstrate that bounded gated residual modulation outperforms both static prototypes and unbounded residual variants, supporting the use of controlled query-conditioned modulation rather than either static or unconstrained anomaly prototypes. The paper concludes that controlled visual prototype adaptation as an effective alternative to text-based anomaly modeling, while noting that extending ReCAP to volumetric and multimodal clinical data remains an important direction.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in an AI system:

Improvement: Instead of using fixed text prompts or static visual tokens for normal/abnormal classification, implement a bounded gated modulation mechanism that re-centers normal/abnormal prototypes per input image based on its global visual context.

What the improved system can do:

  • Adapt anomaly boundaries to unseen medical domains (e.g., brain MRI → liver CT) without retraining

  • Avoid the brittleness of fixed text prompts like normal and abnormal that fail on subtle, localized lesions

  • Prevent prototype drift toward lesion-contaminated context via the sigmoid gate and tanh bound

Improvement: Build a memory bank from normal support patch tokens and score each query patch by nearest-neighbor cosine distance, then fuse with the prototype branch using a learnable weight λ.

Improvement: Aggregate patch-level anomaly probabilities across multiple CLIP transformer layers (e.g., layers 6, 12, 18, 24) using learnable layer weights and top-k pooling.

Improvement: Add a margin-based regularizer that keeps base normal/abnormal prototypes discriminative before input-conditioned modulation, coupled with the bounded residual.

Improvement: Replace token–patch interactions and spatial recalibration modules with lightweight bounded prototype modulation and direct patch–prototype cosine scoring.

Improvement: Remove the text encoder entirely and learn normal/abnormal visual prototypes directly in CLIP visual space, with support-based prototype initialization for few-shot targets.

Improvement: Use a fixed modulation strength η (default 0.05) with a learnable gate to bound the context-induced prototype shift.

Abstract

Medical anomaly detection identifies abnormal images and localizes lesions under scarce supervision while generalizing across organs and modalities. Existing CLIP-based methods reduce annotation requirements through vision--language alignment, but their normal and abnormal references, whether text prompts or learned visual tokens, remain fixed across test images. Such static references may not transfer reliably to unseen targets in a cross-domain medical imaging scenario. To address this, we propose ReCAP, a language-free framework that replaces static anchors with input-conditioned visual prototypes. ReCAP re-centers separated normal and abnormal prototypes for each image through a bounded gated modulation, enabling query-adaptive anomaly scoring while constraining context-induced prototype drift. For the few-shot setting, we introduce a non-parametric normal-reference memory to preserve instance-level target-domain variation and complement the conditional prototype branch. Across six medical benchmarks, ReCAP achieves the best image-level AUROC on all zero-shot and 23 of 24 few-shot settings, and the best zero-shot pixel-level AUROC on all three segmentation datasets. Particularly, it reduces inference latency by over 70% compared to the fastest baseline, without text prompts or test-time gradient updates.

Sources

Related papers