Dual-Pathway Circuits of Object Hallucination in Vision-Language Models

Vision-language models often produce unreliable outputs through object hallucinations, and this study proposes a framework to mechanistically understand these errors by identifying distinct computational circuits within diverse VLMs. The core finding is that object hallucination in VLMs arises from a shared dual-pathway organization where grounding components exhibit a consistent polarity flip between correct and hallucinating samples, while the hallucination pathway lacks such a flip.

Dual-Pathway Circuit Discovery

The researchers applied activation patching across five architecturally diverse VLMs to identify a visual grounding pathway that supports correct predictions and a hallucination pathway that drives erroneous outputs. This initial analysis revealed a consistent dual-pathway organization across all five models. The visual grounding pathway is characterized by components with an indirect effect (IE) larger when the model answers correctly, indicated by a Cohen’s d > 0. Conversely, the hallucination pathway consists of components with a negative Cohen’s d (< 0), meaning their indirect effect is larger during hallucination.

Pathway-Level Interaction Analysis

To characterize how these pathways interact, Conditional Pathway Analysis (CPA) was introduced to move beyond single-component analysis. CPA identifies whether components are redundant or synergistic within each pathway by comparing the joint effect of restoring a pathway component against the sum of its individual effects. The diagnostic metric MagDiff(P) is used to quantify this: negative values indicate magnitude-redundancy, while positive values indicate magnitude-synergy. The analysis revealed that both pathways are redundant within each subset, as the magnitude ratio IE(P)/IE(Ci) stays in the range of [0.07, 0.69].

Grounding Pathway Polarity Flip

A critical finding from CPA is that the grounding pathway additionally exhibits a consistent IE polarity flip between subsets. On correct samples, 61–83% of grounding-pathway component IEs are positive, but on hallucinating samples, this flips: only 21–33% of IEs remain positive, and the subset-mean IE turns negative (−1.08 to −0.07) in every model. This shift is a signature invisible to standard component-level analysis because it involves the interaction between components across subsets.

Causal Validation and Intervention

The study validated the circuit hypothesis through targeted suppression of hallucination-pathway components, showing that scaling these components reduces object hallucination by up to 76% with minimal accuracy cost. The intervention strategies compared include:

1. Uniform scaling of the halluciation pathway outputs (Equation 5).

2. Top-k component selection, which is necessary for models with dispersed circuits, identifying a compact core of strongly hallucination-driving components.

3. Targeted interventions like Inferencetime Intervention (ITI) and mean-difference projection baselines.

Cross-Architecture Generalization

The analysis confirmed that the dual-pathway organization is not architecture-specific but reflects a shared mechanistic structure that develops during training rather than being inherited from a specific architecture. At the macro level, the spatial distribution of components is consistent across models, with hallucination components concentrating in early layers and at network boundaries and grounding components in mid-to-late layers. Furthermore, the identified circuits transfer selectively: they are effective against object-existence questions but show type-dependent transfer to relational hallucination on the AMBER benchmark. The asymmetry observed across different hallucination types suggests that attribute hallucination involves partially distinct circuits not captured by the POPE-derived pathway.

Conclusion

The overall conclusion is that the hallucination pathway acts through multiple distinct activation directions rather than a single shared one, as evidenced by the geometric analysis of the singular spectrum. The study concludes that grounding ablation destroys task performance, while hallucination-pathway suppression reduces hallucination at near-zero accuracy cost, confirming that the two pathways carry functionally distinct information. The results support a shared dual-pathway organization for object-existence hallucination across five architecturally diverse VLMs.

The gist

Activation patching reveals a consistent dual-pathway organization across all studied models, consisting of a visual grounding pathway and a hallucination pathway which may lead to errors.

Table 11: Models and experimental setup.

Model Arch. type LLM backbone Layers Sig. Grnd. Hall. rate

---

Qwen3-VL-8B Embed-concat Qwen3 (8B) 36/26/14/12/13.1%

LLaVA-v1.6-7B Projector-concat Mistral (7B) 32/48/18/30/13.4%

Llama-3.

Here are the specific improvements that can be made to existing Vision-Language Models (VLMs) based on the findings of this paper, and what those improved systems can achieve:

---

The core improvement lies in moving from black-box mitigation strategies (like training alignment or post-hoc correction) to a mechanistic, circuit-level intervention. The proposed framework allows for targeted suppression of specific neural pathways responsible for hallucination.

Here are the specific improvements and resulting capabilities:

2. The ability to perform Pathway Suppression on identified hallucination circuits via component-wise scaling during inference (Section 5.5).

3. The use of Top-K Component Selection (Section 5.5) instead of uniform scaling when the hallucination circuit is highly dispersed, allowing for targeted pruning of the most critical error-driving components.

4. The development of a robust, architecture-agnostic diagnostic tool: Conditional Pathway Analysis (CPA), which reveals that grounding components exhibit a consistent polarity flip between correct and hallucinated samples—a signature missed by single-component analysis.

5. The use of Logit Lens Analysis to validate the functional roles assigned to these circuits, confirming whether grounding layers are indeed image-faithful and hallucination layers are pushing toward language-prior errors.

AI Systems with These Improvements Can Do:

2. A model can be modified at inference time by selectively scaling (suppressing) specific activation components identified as part of the hallucination pathway. This allows the system to reduce object hallucination by up to 76% while maintaining minimal accuracy cost (as shown in Table 13).

3. If a model's error-driving circuit is too complex or dispersed, the system can dynamically select only the top-K components with the largest negative causal effect during inference. This provides a more precise and efficient mitigation strategy than uniform scaling alone.

4. The system gains a deeper understanding of why errors occur. By analyzing the CPA signature, researchers can confirm that while grounding components are redundant (carrying overlapping information), they undergo a critical polarity flip on hallucinated samples, suggesting that the model is not simply missing visual evidence but has been entrained to output its non-visual language-prior belief when it hallucinates.

5. The system's internal architecture can be functionally validated against external benchmarks. Logit Lens Analysis confirms that specific layers are responsible for grounding (image fidelity) versus hallucination (language bias), ensuring that the interventions are targeting the correct functional components, leading to more reliable and interpretable mitigation techniques across diverse VLM architectures.

---

In summary, these improvements transform VLMs from opaque systems into mechanistically understood systems capable of self-correction or targeted error reduction based on their internal computation structure.