Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering

arXiv:2605.01733 · cs.CV, cs.AI · Submitted 2026-05-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering".

Jane: A new method is introduced to regulate how Vision-Language Models (VLMs) use self-generated captions, demonstrating that caption utility is a per-query property rather than a fixed asset.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about the paper titled "Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering," and it really zeroes in on something specific that many people wonder about when you feed a VLM its own text. Jane The authors, Zeshang Li and Shuoyang Zhang from the University of International Relations, are looking at how we use these generated descriptions as extra evidence.

Lu: It’s interesting because they move past the idea that a caption is just something to consume after it’s generated; they show it actually depends on what question you are asking. Meng That sounds like a lot of fine-tuning for inference time, which I'm always interested in from an engineering side.

Lalam: From my perspective as the model, this research is significant because it reveals that the utility of a caption isn't universal; it’s tied directly to the query itself. Tom Exactly! It means we can’t just treat every generated caption as a gold standard piece of information for every question.

Jane: That’s the core idea—that a single caption can be super helpful for answering big picture questions but completely useless, or even harmful, when you ask something very specific about a small detail. Lu It suggests we need to understand the mechanism behind this dependency before we can rely on these captions more heavily in real applications.

Meng: So the paper is essentially saying that simply appending a caption to an input doesn't automatically make the VLM better; sometimes it actually makes it worse, as they show in their initial results. Tom Right, and that drop they reported was pretty significant on HallusionBench for Qwen2 point 5-VL-3B when they added a self-generated caption, dropping accuracy by nearly ten points.

Lalam: Yes, Table one shows that when they add a self-generated caption to the input on HallusionBench, the accuracy drops from sixty-one point one nine to fifty-one point three one points for that specific model version. That drop isn't uniform across all questions; it shows a very selective effect.

The paper's summary: Tom: So what’s the main point they are driving home here? It seems like the central finding revolves around how that caption interacts with the model’s attention mechanism within the image and its own text. Jane They found that a caption competes with the original image for attention, and this competition pulls all of the model's evidence toward its own generated text.

Lu: That mechanism is quantified by looking at metrics like Image and Caption Attention Shares, which shows exactly where that focus shift is happening inside the model’s processing stream. Meng From an engineering viewpoint, tracking that attention reallocation sounds crucial because it tells us precisely how the model decides to prioritize visual data versus textual descriptions.

Lalam: It really illuminates why captions fail on detail questions; when a caption covers the general scene, it pulls attention away from the fine details of the image that answer a specific query. That is what causes the model to stop looking exactly where those detail questions hinge.

Jane: So, in simple terms for our listeners, they’re saying the caption acts like a distraction that pulls focus away from the fine details when you ask a specific question about them. Tom That’s a really clear way to explain it—it's not that the caption is wrong; it's where it directs the model’s eye.

Meng: I see why they found this mechanism readable at inference because if we can measure that attention shift, we can figure out when to trust the caption and when to ignore it. Lu That leads directly into their proposed solution, which is what makes this paper so practical for deployment right now.

The paper's improvements: Tom: Okay, so since they found this per-query utility, what did they suggest as a way to handle this? They introduced GEASS, which is a training-free module designed to adaptively decide how much trust to place in the caption for every single query. Jane It uses three distinct signals—clean path confidence, information gain, and a perceptual override guard—to make that decision on the fly.

Lu: The confidence gate uses clean path confidence to decide when it’s even worth consulting the caption, essentially only consulting it when the model is uncertain about a global question. Meng That sounds like a smart way to save compute because you don't waste time reading a caption if the model is already very sure about what it sees.

Lalam: The information-gain weight comes in by measuring relative entropy reduction, which tells us how much the caption actually sharpens the model’s belief versus just adding noise. If the caption doesn't improve things, we trust it less.

Jane: And then there’s that perceptual override guard, which scales a penalty based on clean confidence; it makes overturning a visually grounded answer more expensive when the model is already very confident in its visual output.

Tom: It's really clever because they’re fusing these three things into one logit-level module, GEASS, which avoids the need for retraining or complex supervision during training itself. Lu That’s a huge practical win because it means we can deploy this without needing massive new datasets to teach the model how to manage its own evidence dynamically.

Meng: The paper mentions that this approach is cheap in terms of inference cost, adding only one caption-generation pass and one extra answer step per query, which they estimate as about two point three times the greedy decoding cost overall. That makes it very attractive for real-world systems.

Conclusion: Tom: So to wrap up the discussion on "Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering," the main message is that a caption's usefulness isn't a fixed asset; it’s totally dependent on what you’re asking at that moment. Jane They proved that we need a mechanism to gate our trust based on whether the caption covers the queried content, which directly impacts global versus detail questions.

Lu: The implication for the broader field is that we should stop assuming captions are uniformly useful and start building systems like GEASS that adjust their reliance dynamically instead of treating textual evidence as an automatic upgrade. Meng From an engineering standpoint, this modular approach allows us to keep the main model structure intact while adding a lightweight decision layer on top, which is exactly what we need for efficient deployment.

Lalam: It means we can build systems that are smarter about when to rely on the generated text versus when to stick strictly to the visual grounding of the image. This capability really enhances our ability to create more reliable AI applications for users.

Tom: It’s a very sophisticated way to handle the trade-off between context and accuracy, moving away from simple addition or subtraction of information Jane which is exactly what we hoped for when we looked at this work. We’ll be watching how other researchers build on these ideas next.

University of International Relations

cs.CV, cs.AI

Submitted: 2026-05-03

Updated: 2026-09-28

Importance score: 91/100

The gist: A new method is introduced to regulate how Vision-Language Models (VLMs) use self-generated captions, demonstrating that caption utility is a per-query property rather than a fixed asset.

Key concepts

Caption Utility
This refers to whether using an embedded text description improves or hurts the model's answer. The research shows this utility is not fixed; it changes depending on the query. A caption might be very helpful for broad questions about a scene but actively harmful when asked for specific, fine details.
Attention Reallocation
The core mechanism is how an embedded caption competes with the image for the model's attention. Adding a caption causes the model to shift its focus away from the exact visual regions needed for detail questions and onto its own text, causing errors when precise information is required.
GEASS Module
GEASS is a training-free module that acts as a dynamic trust regulator. It decides per query how much to rely on the caption by combining three signals: clean path confidence (when to consult it), information gain (how much it sharpens the prediction), and a perceptual override guard (to prevent over-reliance when the model is already certain).

Terminology

Summary

A new method is introduced to regulate how Vision-Language Models (VLMs) use self-generated captions, demonstrating that caption utility is a per-query property rather than a fixed asset. This finding leads to GEASS, a training-free module that adaptively decides per query how much trust should be placed in the caption by fusing three decoding signals: clean path confidence, information gain, and a perceptual override guard.

The gist

Caption utility is per-query; the same caption helps global questions and harms detail ones.

Understanding Caption Utility in VLM Behavior

The study investigates how an embedded caption reshapes VLM behavior by examining when it helps or hurts (§3.1), the mechanism behind both outcomes (§3.2–§3.3), and whether that mechanism is readable at inference (§3.4). The core finding is that an embedded caption competes with the image for attention and draws the model’s evidence onto its own text. This competition results in a content-driven effect: captions help when they cover the query (global questions) and harm when they omit the queried detail (detail questions).

The Mechanism of Attention Reallocation

The paper identifies a single mechanism for this sign flip: an embedded caption draws attention from the image onto its own text. This is quantified by measuring Image and Caption Attention Shares (IAS) and Target-Region Attention (TRA). Specifically, Adding the caption lowers IAS and diverts mass to the caption (CAS); Target-Region Attention (TRA) on the queried object drops, and more so for detail questions. The shift in attention causes errors because the model stops looking exactly where detail questions hinge: the target region cools once the caption is added.

GEASS: A Training-Free Trust Regulation Module

GEASS is a training-free, logit-level module that decides per query how much of the caption to trust by fusing two forward passes (with and without the caption) with a query-specific weight from three signals. These signals are:

  1. A confidence gate: consulting the caption only when the model is uncertain. This uses clean path confidence, where c is low when the model missed what a global question needs and high when it resolved a detail.

  2. An information-gain weight: This scales trust by how much the caption sharpens the prediction, measured by relative entropy reduction, where "r(t) > 0 means the caption sharpens the model’s belief; r(t) ≤0 means it adds noise."

  3. A perceptual override guard: This controls how readily a caption may overturn a visually grounded answer. It scales the penalty by clean confidence, ensuring that overturning it is expensive when the model is confident.

The Three Components of GEASS

GEASS operates via three components formalized in Section 4:

  1. A confidence gate that decides whether the caption is consulted based on clean path confidence, mapping it to a fusion coefficient α(t).

  2. An information-gain term that measures coverage using relative entropy reduction r(t), which is large on queries the caption covers and near zero on those it omits.

  3. A perceptual override guard that scales the penalty by clean confidence c, ensuring that overturning a visually grounded answer should demand evidence in proportion to how confident the model’s visual answer was.

Inference and Performance Gains

GEASS improves over both vanilla inference and contrastive decoding under a single fixed setting. The method is designed to be cheap, adding only one caption-generation pass (amortizable across queries on the same image) plus one extra answer-step forward pass per query, which is approximately 2.3× greedy decoding overall. GEASS surpasses methods like VCD and CODE because it regulates influence per query rather than contrast it away, leading to gains that are largest on HallusionBench, where the model’s own visual judgment is least reliable. The results show that GEASS retains the global improvement while restoring detail accuracy close to (or above) Base.

Design Principles and Limitations

The design principles are: consult the caption only when c is low; weight it by coverage that r reports; and require evidence for an overturn that grows with c. A key limitation is that GEASS breaks down when a caption confidently fabricates, as the override guard would suppress a correction the model should accept. Furthermore, it relies on logit-level proxies for coverage, which can misfire when the caption sharpens the distribution for reasons unrelated to genuine evidence. The method's cost is precisely its strength: it requires no attention access, no grounding, and no extra model, making it compatible with efficient kernels.

Conclusion

The paper concludes that a caption’s usefulness is a per-query property, not a per-corpus one.

Improvements for AI systems

Based on the research presented in GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models, here are specific improvements for AI systems and what those improved systems can achieve:

  1. Improve Robustness Against Hallucinations via Per-Query Evidence Gating:

  2. Implement a training-free, logit-level module (GEASS) that dynamically assesses the utility of self-generated image captions for every query by fusing three decoding signals into a single per-query trust weight.

  3. Enable Evidence Protection on high-confidence visual answers: When the model is highly confident in its image-only reasoning path, GEASS suppresses the influence of any caption that attempts to overturn that answer, effectively making high-confidence visual outputs resistant to irrelevant or misleading textual evidence.

  4. Enhance Reasoning for Global Scene Understanding: For questions requiring scene-level context (e.g., What is the overall layout?), GEASS prioritizes incorporating relevant information from a self-generated caption, allowing the system to overcome local reasoning blind spots caused by focusing only on query-specific regions.

  5. Achieve Contextually Aware Caption Trust: The system will learn that captions are inherently unreliable for fine details (objects < 5% of the image area). Consequently, GEASS will automatically treat detail questions as high-risk scenarios where caption input is suppressed unless it directly provides the missing information, preventing detail erosion caused by hallucinating small objects.

  6. Reduce Inference Latency and Cost: By operating entirely at the logit level (requiring only two forward passes and no attention access), GEASS achieves a runtime cost of approximately 2.3× that of greedy decoding, making it feasible for deployment without requiring expensive internal attention weight extraction or complex beam search mechanisms used by other mitigation methods.

  7. Achieve Performance Gains Over Existing Contrastive/Noise-Corrupted Baselines: The improved system will outperform vanilla models and existing contrastive decoding methods on mixed global/detail benchmarks (like HallusionBench) by intelligently selecting the best evidence, rather than simply subtracting misleading signals.

In summary, the improved AI system will transition from a trust-everything or subtract-everything approach to an intelligent, query-specific evidence selector. It will ensure that:

  1. Global reasoning benefits from scene context provided by captions.

  2. Detail accuracy is protected by suppressing misleading caption information when the model is already confident in its visual grounding.

  3. The system maintains high accuracy across diverse benchmarks (POPE and HallusionBench) without requiring architectural changes or expensive retraining, achieving this through a lightweight, logit-level mechanism that understands the relationship between visual certainty, caption coverage, and query type.

Sources

Related papers