Semantic Misalignment in Vision-Language Models under Perceptual Degradation

arXiv:2601.08355 · cs.CV · Submitted 2026-01-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Semantic Misalignment in Vision-Language Models under Perceptual Degradation".

Jane: Vision–Language Models (VLMs) are increasingly deployed in safety-critical domains like autonomous driving, where reliable perception is paramount for decision-making.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now that we know what the authors found in "Semantic Misalignment in Vision-Language Models under Perceptual Degradation," let's go over their core summary again to make sure everyone has a solid grasp of the main findings. Essentially, they show that when you introduce realistic visual corruption—like blur or low light—even if those corruptions only cause moderate drops in conventional segmentation metrics, the resulting Vision-Language Model behavior can suffer severe problems like inventing objects or forgetting vital safety information.

Jane: That’s a very clear summary, Tom; it means the model isn't just confused by the bad picture; it fundamentally misinterprets what is in that picture when its input is imperfect.

Lu: They set up this study by applying perception-realistic corruptions—specifically motion blur, which they simulate using linear convolutional kernels whose size increases with severity from s=one to three low light degradation via gamma correction plus additive Gaussian noise, and partial occlusion by rectangles whose area also scales with severity.

Meng: I’m thinking about the methodology here; it’s important that they controlled these corruptions independently on each test image to make sure they were truly isolating the effect of perception degradation from other visual noise.

Lalam: Their proposed language-level misalignment metrics are the key innovation here, as they aren't relying on traditional methods but instead focusing on measuring hallucination rate, critical omission rate, and safety misinterpretation rate directly.

Tom: Right, and the results show that for models like CLIP and Qwen2-VL, perception degradation substantially increases semantic misalignment even when segmentation quality only degrades moderately.

Jane: So it’s not a linear relationship; small visual drops don't just mean small language errors; they can trigger completely different and more dangerous failure modes in the VLM output.

Lu: Specifically for CLIP, the paper found that hallucination rate increases notably under low-light and motion blur conditions, while the critical omission rate stays persistently high across those same conditions.

Meng: That suggests that even when a model is struggling visually, it frequently omits objects it shouldn't be omitting for safety reasons, which is a big concern for deployment.

Lalam: I think this empirical decoupling of perception and semantic reliability is the most important finding because it proves that pixel-level robustness doesn't guarantee semantic reliability in these complex multimodal systems.

Tom: So, to recap, the paper’s main takeaway is that we need to look at how perception degradation propagates through the system to understand the full picture of VLM failure.

Jane: And that leads us directly into discussing what solutions they are proposing and how we can use this information to make these models more robust against real-world sensory issues.

The paper's summary: Tom: Moving on, the authors aren't just stopping at finding the problem; they suggest a path forward by proposing a set of language-level misalignment metrics as a way to better diagnose these failures, which is something we need to discuss further. These metrics allow us to move beyond simple visual scores and measure the actual semantic impact of perception degradation.

Jane: That’s smart because it gives us concrete ways to quantify the problems; instead of just saying "the model failed," we can say "the critical omission rate was zero point one five," which is much more actionable information for debugging.

Lu: The authors are proposing a new way to evaluate these systems by using this set of metrics alongside conventional segmentation metrics, analyzing their relationship across multiple contrastive and generative VLMs.

Meng: I’m wondering how we integrate this into existing pipelines; does it require a complete overhaul, or can we just add these language-level checks as an extra layer of validation after the initial visual processing?

Lalam: From my perspective, these metrics should become standard evaluation criteria for any VLM deployed in safety-critical domains because they directly address the reliability gap between pixel quality and semantic output.

Tom: It seems like their suggestion is to stop treating segmentation quality as a standalone measure and start using these language-level metrics as the primary indicators of downstream semantic integrity.

Jane: So, they are advocating for a framework where we explicitly track hallucination, omission, and safety misinterpretation during the reasoning process rather than just assuming correctness based on a perfect visual input.

Lu: Their work suggests that this approach is crucial because existing evaluations predominantly rely on curated benchmark datasets and assume reliable visual inputs, which is a major limitation they point out.

Meng: For our practical engineering goals, this means we need to design validation sets where we deliberately introduce perception degradation to see precisely how the system reacts in terms of critical omission rates.

Lalam: This paper’s contribution is providing those interpretable metrics that allow us to pinpoint exactly which perception failures lead to specific VLM breakdowns, which helps guide targeted training interventions.

Tom: So, the improvement isn't just a new number; it’s a new way of thinking about what constitutes a reliable multimodal system when dealing with noisy sensory data.

Jane: And that means we shift our focus from the image itself to the semantic consistency of the final output, which is where we need to concentrate our efforts.

The paper's improvements: Tom: So, wrapping up this discussion on "Semantic Misalignment in Vision-Language Models under Perceptual Degradation," the authors have clearly demonstrated that there’s a clear disconnect between pixel quality and language reliability, showing that standard segmentation metrics don't provide reliable predictive power for what the paper concludes.

Jane: They wrap up by emphasizing that visually plausible segmentation outputs often mask localized, safety-critical errors that disproportionately affect vision-language reasoning, so we need to be extremely careful about what we assume when deploying these systems.

Lu: The implication is that this work moves us toward developing robustness-aware evaluation frameworks that explicitly account for perception uncertainty in safety-critical applications.

Meng: From an engineering viewpoint, we need to build systems where they can handle real-world perception degradation without catastrophic failures, even if the segmentation score dips slightly.

Lalam: This paper’s work on perception-to-language misalignment analysis really sets a new benchmark for auditing multimodal AI systems because it shows that we must measure semantic reliability directly.

Tom: So, the final message is that pixel-level metrics are insufficient for ensuring safety in autonomous driving and embodied AI when perception is uncertain.

Jane: It’s a reminder to our listeners that we need to be critical consumers of AI performance reports and demand more transparency about how these models handle real-world degradation.

Lu: This paper's contribution provides a solid theoretical backing for understanding the propagation effect from visual input quality to downstream reasoning failures.

Meng: For our team, this means we focus on building systems that are resilient even under challenging conditions, prioritizing safety-critical entity omission and hallucination over mere visual coherence.

Lalam: I think the whole point of "Semantic Misalignment in Vision-Language Models under Perceptual Degradation" is pushing us to create evaluation frameworks that explicitly account for perception uncertainty, which is what we need for truly trustworthy AI.

Conclusion: Tom: So we’ve been deep in the weeds on "Semantic Misalignment in Vision-Language Models under Perceptual Degradation," and what we’re seeing is that these models aren't as robust as we thought when things get messy visually.

Jane: Exactly, Tom; the core finding is that a drop in visual quality doesn't just cause a small dip in accuracy; it can trigger severe failures like hallucinations and safety omissions.

Lu: It’s fascinating because they’ve successfully decoupled the performance of conventional segmentation metrics from the actual semantic reliability of the downstream reasoning.

Meng: From a practical standpoint, this means we can stop trusting raw visual scores alone when deploying these VLMs in real-world scenarios where sensor noise is a constant factor.

Lalam: I think the most important thing here is their proposed language-level metrics, like the Critical Omission Rate and Safety Misinterpretation Rate, because those are what truly matter for ensuring AI can operate safely in complex environments.

Tom: Right, so we’re moving beyond just looking at the picture to rigorously testing how the language understands that picture under stress.

Jane: It gives us a much clearer way to evaluate these systems because it shows that visual plausibility doesn't equal semantic correctness.

Lu: This paper’s work on perceptual degradation analysis provides a solid foundation for designing better evaluation frameworks that prioritize safety outcomes over simple visual metrics.

Meng: For the engineers out there, this means we need to start building validation sets specifically designed to expose these kinds of multimodal breakdowns.

Lalam: And for me, as a model, this focus on safety misinterpretation means we can be trained to recognize and correct those specific types of reasoning errors much more effectively.

Tom: So that’s the big picture: understanding how perception degradation propagates through the system is crucial for building trustworthy AI.

Jane: It’s a sobering reminder that perfect input doesn't guarantee perfect output in these complex multimodal systems.

Lu: And this paper’s analysis of perceptual degradation really opens up some wild possibilities for how we can build more resilient vision-language models.

Meng: We’re definitely keeping an eye on these metrics because they give us actionable data to improve the reliability of our agents in complex settings.

Lalam: It shows that by focusing on these specific misalignment metrics, we can start making AI systems better at navigating the nuances of real-world perception. And that wraps up our discussion on "Semantic Misalignment in Vision-Language Models under Perceptual Degradation." Next up, we’ll be looking at how structure, rather than just pixels, is being learned in the paper "Structure over Pixels: Learning Variable-Length Visual Programs." Stay with us.

Guo Cheng

cs.CV

Submitted: 2026-01-13

Updated: 2026-09-30

Importance score: 77/100

The gist: Vision–Language Models (VLMs) are increasingly deployed in safety-critical domains like autonomous driving, where reliable perception is paramount for decision-making.

Key concepts

Perception Degradation Protocol: Researchers simulated real-world vision problems by intentionally corrupting images using motion blur, low light noise, and partial object occlusion. They measured how much the segmentation accuracy dropped (ΔQ) when these visual challenges were applied to a dataset.
Hallucination Rate (HR): This metric counts how often the VLM invents objects or concepts that are not actually in the input image. It measures semantic misalignment by comparing what the language model says against what is actually visible in the corrupted picture.
Critical Omission Rate (COR): This tracks failures where safety-important items, like pedestrians or traffic signs, are completely forgotten in the VLM's text description. A high COR means the model fails to mention entities crucial for making a safe judgment.
Empirical Decoupling: The core finding shows that conventional metrics (like mIoU) for image segmentation do not reliably predict downstream language errors. This proves there is a fundamental gap between how well an image can be segmented and how reliably the VLM can reason about what it sees.

Terminology

Robustness — 32× in this paper · explained in 5 episodes
Hallucination — 31× in this paper · explained in 3 episodes
Benchmark — 12× in this paper · explained in 2 episodes
Critical Omission Rate — 10× in this paper
Accuracy — 8× in this paper · explained in 1 episode
Decoupling — 6× in this paper · explained in 2 episodes
Failure Modes — 6× in this paper · explained in 3 episodes
Hallucination Rate — 6× in this paper
Core — 4× in this paper · explained in 2 episodes
Grounding — 4× in this paper · explained in 3 episodes
Safety Misinterpretation Rate — 4× in this paper
Benchmarking — 3× in this paper · explained in 3 episodes
Consistency — 3× in this paper · explained in 3 episodes
Embodied AI — 3× in this paper · explained in 2 episodes
Transformer — 3× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 3× in this paper · explained in 6 episodes
Adam — 2× in this paper · explained in 2 episodes
Adaptation — 2× in this paper · explained in 1 episode
Aggregation — 2× in this paper · explained in 1 episode
Distribution Shift — 2× in this paper · explained in 5 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Semantic Understanding — 2× in this paper · explained in 2 episodes
Alignment — 1× in this paper · explained in 1 episode
Contextual Understanding — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Instruction Tuning — 1× in this paper · explained in 2 episodes
International Conference on Machine Learning — 1× in this paper
Multimodal Reasoning — 1× in this paper · explained in 5 episodes
Uncertainty Quantification — 1× in this paper · explained in 8 episodes

Summary

Vision–Language Models (VLMs) are increasingly deployed in safety-critical domains like autonomous driving, where reliable perception is paramount for decision-making. This paper systematically investigates how degradation in upstream visual perception affects downstream vision–language reasoning, revealing a critical disconnect between pixel-level robustness and multimodal semantic reliability. The study demonstrates that modest drops in conventional segmentation metrics can induce severe failures in VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments.

Perception Degradation Protocol

The researchers conducted a systematic empirical study using semantic segmentation on the Cityscapes dataset as a representative perception module. They introduced perception-realistic corruptions to simulate real-world sensing challenges:

Motion blur, simulated using linear convolutional kernels whose size increases with severity (s=1, 2, 3).
Low-light degradation, simulated via gamma correction combined with additive Gaussian noise (s=1, 2, 3).
Partial occlusion by randomly positioned rectangular regions whose area increases with severity (s=1, 2, 3).

These corruptions were applied independently to each test image to induce controlled degradation in visual perception while preserving overall scene structure. The resulting segmentation performance drop was quantified as ΔQ = Qclean − Qdeg.

Language-Level Misalignment Metrics

To characterize the failures not reflected by conventional perception metrics, the authors introduced a set of interpretable language-level misalignment metrics:

Hallucination Rate (HR): Defined as the proportion of language outputs that introduce non-existent objects: HR = 1/N ∑ Cvlm(x̃)⊈ Cgt(x) / Cvlm(x̃) + ϵ.
Critical Omission Rate (COR): Measures failure to mention safety-critical classes: COR = 1/N ∑ I(Cc crit (xi) ⊈ Cvlm(x̃i)).
Safety Misinterpretation Rate (SMR): Measures the failure of vision–language reasoning to support correct high-level decision-making by comparing VLM responses against a reference safety label: SMR = 1/N ∑ I(M(x̃i, psafe) ≠ ai).

Empirical Decoupling of Perception and Semantic Reliability

The core finding is the analysis of the relationship between perception degradation (ΔQ) and multimodal semantic misalignment (ΔL). The study computed correlation coefficients between conventional segmentation metrics (like mIoU) and the proposed language-level metrics. The results reveal a weak and inconsistent correlation between conventional segmentation metrics and multimodal semantic reliability, exposing a fundamental disconnect between pixel-level perception robustness and downstream vision–language reasoning.

VLM Behavior Under Degradation

The evaluation was performed across multiple contrastive (e.g., CLIP) and generative (e.g., Qwen2-VL) VLMs using a fixed, minimal, and task-agnostic prompt set categorized into Scene Description Prompt, Object Presence Prompt, and Safety Interpretation Prompt. The results showed that:

Perception degradation substantially increases semantic misalignment even when segmentation quality degrades only moderately.
For CLIP, HR increases notably under low-light and motion blur; COR remains persistently high across conditions, suggesting safety-critical objects are frequently omitted from language descriptions despite being present in the scene.
SMR exhibits non-monotonic behavior; in some cases, severe corruptions reduce semantic segmentation VLM prompts input, leading to unstable safety reasoning rather than genuine robustness.

Conclusion and Implications

The work demonstrates that visually plausible segmentation outputs often mask localized, safety-critical errors that disproportionately affect vision–language reasoning. The findings underscore a critical limitation of current VLM-based systems and motivate the need for robustness-aware evaluation frameworks that explicitly account for perception uncertainty in safety-critical applications. The study concludes that standard segmentation metrics provide limited predictive power for downstream multimodal semantic reliability.

Key Contributions:

** Perception-to-Language Misalignment Analysis: A systematic study of how degradation in semantic segmentation propagates to semantic misalignment in modern Vision–Language Models.**

** Language-Level Misalignment Metrics: Proposed interpretable metrics to quantify hallucination, critical object omission, and safety misinterpretation in VLM output.**

** Empirical Decoupling of Perception and Semantic Reliability: Demonstration that standard segmentation metrics provide limited predictive power for downstream multimodal semantic reliability.**

Key Words:

Vision-Language Models (VLMs), Semantic Segmentation, Perceptual Degradation, Multimodal Misalignment, Hallucination, Autonomous Driving.

References:

[1] A. Radford et al., Learning transferable visual models from natural language supervision, ICML 2021.

[13] L. Chen et al.

Improvements for AI systems

Based on the provided scientific paper, here are specific improvements for AI systems and what those improved systems can achieve:

Improving Robustness-Aware Evaluation Frameworks: The current limitation is that pixel-level metrics (like mIoU) mask failures in downstream reasoning.
Implementing Language-Level Misalignment Metrics: Incorporate the proposed metrics—Hallucination Rate (HR), Critical Omission Rate (COR), and Safety Misinterpretation Rate (SMR)—as primary evaluation criteria alongside conventional segmentation metrics.
Integrating Perception Uncertainty into Decision Making: Develop VLM systems that explicitly account for perception uncertainty rather than implicitly assuming perfect visual input stability.

These improvements will enable the following specific capabilities:

Robust Autonomous Driving Systems: Improved systems can be deployed in autonomous vehicles under adverse conditions (low light, motion blur, occlusion) with a quantified understanding of their semantic reliability. They will not only detect objects but will also be explicitly measured for hallucinated object mentions and omission of safety-critical entities (like pedestrians or traffic signs), leading to safer and more trustworthy decision-making.
Reliable Embodied AI: For mobile robots operating in complex environments, these systems can better handle real-world perception degradation. The improved AI will be less likely to make catastrophic errors in high-level reasoning (e.g., Is it safe to proceed forward?), as its safety judgment (SMR) is decoupled from the mere visual coherence of the segmentation map.
Enhanced Model Debugging and Training: By correlating segmentation quality with language-level misalignment, developers can pinpoint exactly which perception failures lead to specific VLM breakdowns (e.g., Low-light conditions disproportionately cause 'Critical Omission' of traffic signs). This allows for targeted training interventions, focusing on bridging the gap between low-level visual processing and high-level semantic reasoning.
Model Selection and Trustworthiness Assessment: Systems can be rigorously compared not just on raw accuracy, but on their multimodal semantic reliability across various corruption levels. This provides a more honest assessment of which VLM architecture (contrastive vs. generative) is truly robust when deployed in uncertain real-world scenarios.