Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

arXiv:2608.29996 · cs.CV, cs.AI, cs.LG · Submitted 2026-08-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models".

Tom: This paper introduces PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework designed to mitigate spurious object-background correlations in Large Vision-Language Models (LVLMs).

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, let's talk about the specifics of this paper. The title is "Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models," and it’s authored by Aditi Sarker, Nazreen Shah, Rafi Ibn Sultan, Rhongho Jang, Dongxiao Zhu, and Prashant Khanduri.

Jane: Those authors are clearly experts in the multimodal space. The title points out that the main issue they're addressing is how Large Vision-Language Models often exploit spurious object-background correlations instead of looking at the actual object evidence.

Lu: What’s interesting about their approach, based on what I see here, is that they aren't just suggesting a fix for inference; they are proposing a framework for constructing, benchmarking, and mitigating these correlations right within the training or fine-tuning process.

Meng: So instead of just tweaking prompts during use to stop errors when they happen, this paper seems to suggest we can actually modify how the model learns those associations based on structured data separation. That implies a more fundamental change in the learning objective.

Lalam: That makes sense because if we can control *how* the model learns to separate what's object-related from what's just background noise, it should lead to much more stable and trustworthy outputs for our applications.

The paper's summary: Tom: Moving on, the core summary of "Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models" is that they tackle hallucination by separating object-grounded examples from shortcut-prone ones using structured data partitioning.

Jane: Essentially, they argue that we need to create specific retain and forget partitions to get a controlled measure of how much the model is leaning on background cues versus actual object evidence. They propose three complementary ways to build these partitions: PURGE-D for data-level disentanglement, PURGE-H for hybrid behavior-driven partitioning, and PURGE-M for model-informed semantic partitioning.

Lu: That separation idea is key; by isolating the samples where contextual cues are present but the object is absent, they get a very specific set of examples to target during unlearning. It’s a very structured way to expose these spurious correlations that we usually just see as general errors.

Meng: So, it sounds like they aren't just looking at raw data; they are using model predictions and counterfactual inputs alongside ground-truth annotations to identify exactly where the model is failing due to those shortcuts. That’s a lot of engineering work.

Lalam: When you break it down, it means they are moving away from general debiasing methods that don't specify *why* a prediction failed, and instead focusing on isolating the specific association we want to remove.

The paper's improvements: Tom: Now for the improvements they suggest in "Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models." They introduce partition-aware unlearning that aims to suppress predictions driven by background cues while keeping the object reasoning intact.

Jane: The mathematical goal they set is quite specific: they want the updated model parameters, theta', to satisfy P theta' (c z) to zero for the forget set, but still maintain P theta' (c o) about P theta (c o) for the retain samples. That’s a precise way of saying we want to purge the background reliance while keeping object-based reasoning stable.

Lu: The authors implement this using parameter-efficient fine-tuning with LoRA and three different unlearning objectives: Gradient Ascent, KL-regularized unlearning, and Negative Preference Optimization. This is a sophisticated mathematical tool applied directly to the model weights rather than just changing the input or output settings.

Meng: Applying that kind of targeted weight modification while keeping the overall model close to its original state on certain samples seems like a very clever way to achieve that balance between cleaning up errors and avoiding catastrophic forgetting on important object knowledge.

Lalam: The choice of those three unlearning objectives suggests they are hedging their bets, trying different mathematical levers to ensure they don't accidentally erase crucial object-based knowledge while successfully suppressing the unwanted background dependencies.

Conclusion: Tom: So we’ve covered the title, the summary of how it works, and those specific unlearning objectives. The overall conclusion of "Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models" is that their framework consistently reduces hallucinations and spurious correlation errors across a diverse set of benchmarks while maintaining or improving overall performance in most settings.

Jane: Essentially, they show that by structuring the data separation and applying targeted unlearning, we can effectively reduce those background-driven predictions without significantly harming the model's ability to reason about objects. It’s a very practical result for making multimodal models more dependable.

Lu: The implication here is that we can start systematically diagnosing *where* a model is failing—is it missing the object, or is it just getting distracted by the scenery—and then surgically fix that specific dependency through partition-aware unlearning. That opens up new avenues for causal learning in vision and language.

Meng: From an engineering side, this means we can build iterative testing pipelines where we can specifically target these spurious correlations identified by the partitioning strategies, rather than relying on broad accuracy metrics alone to catch them later.

Lalam: I think the big picture is that this research gives us a pathway toward models that are not just smart pattern recognizers but are actually more robust reasoners grounded in verifiable visual evidence, which is exactly what we need for safe deployment.

Aditi Sarker, Nazreen Shah, Rafi Ibn Sultan, Rhongho Jang, Dongxiao Zhu, Prashant Khanduri

Department of Computer Science, Wayne State University · Department of Electronics & Communications Engineering, Indian Institute of Information Technology Delhi (IIIT Delhi) · Institute for AI and Data Science at Wayne State University (AIDaS)

cs.CV, cs.AI, cs.LG

Submitted: 2026-08-30

Updated: 2026-08-30

Comments: 35 pages, 11 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: This paper introduces PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework designed to mitigate spurious object-background correlations in Large

Key concepts

Spurious Object-Background Correlations
This refers to when large vision-language models rely on irrelevant background cues instead of focusing on the actual object evidence. The paper addresses how these shortcuts lead to errors in the model's predictions.
Partition-Aware Unlearning
This is a framework designed to mitigate spurious correlations by separating data into retain and forget partitions. The goal is to suppress predictions driven by background cues while maintaining the model's ability to reason about objects.
Purge Methods (PURGE-D, PURGE-H, PURGE-M)
These are three complementary ways proposed to build the necessary data partitions. They include data-level disentanglement (PURGE-D), hybrid behavior-driven partitioning (PURGE-H), and model-informed semantic partitioning (PURGE-M).
Unlearning Objectives
These are mathematical tools used during parameter fine-tuning to remove unwanted knowledge. The authors use Gradient Ascent, KL-regularized unlearning, and Negative Preference Optimization to target specific dependencies in the model weights.

Terminology

Summary

This paper introduces PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework designed to mitigate spurious object-background correlations in Large Vision-Language Models (LVLMs). By addressing the tendency of models to exploit spurious object-background correlations and rely on contextual shortcuts rather than object-relevant visual evidence, PURGE provides a method to reduce hallucinations and improve the reliability of multimodal reasoning in safety-critical applications.

The challenge of spurious correlations

LVLMs are increasingly prone to hallucination, a failure mode where models generate or predict objects that are not present in the image. A primary driver of these failures is the presence of spurious correlations in training data, where objects frequently co-occur with specific backgrounds or contextual cues. This leads to models relying on background shortcuts even when the target object evidence is absent, ambiguous, or contradicted.

Existing mitigation strategies, such as training-time debiasing or inference-time decoding, often lack the ability to explicitly identify and remove these learned associations. Furthermore, current evaluations often measure hallucination at an aggregate level without systematically disentangling whether a prediction is supported by object-relevant visual evidence or induced by correlated background cues.

Structured dataset construction

To address these gaps, PURGE utilizes a two-stage process beginning with the construction of structured retain/forget partitions that separate object-grounded evidence from shortcut-prone cues. The framework employs three complementary data construction strategies:

  • PURGE-D (Data-level disentanglement): This strategy separates object and background information via masking (or inpainting) to create object-focused retain sets and context-only forget sets.

  • PURGE-H (Hybrid behavior-driven partitioning): This method combines ground-truth annotations, model predictions, and counterfactual inputs to identify failure cases, such as missed detections and hallucinations.

  • PURGE-M (Model-informed semantic partitioning): This strategy uses cue scoring to partition examples by object presence and contextual cues, specifically isolating hallucination-prone samples where contextual cues are present despite object absence.

Partition-aware unlearning

Once the partitions are constructed, the framework applies partition-aware unlearning to suppress background-driven predictions while preserving object-based reasoning. The mathematical goal is to ensure that the updated model parameters theta' satisfy P theta' (c z) to 0, while maintaining P theta' (c o) about P theta (c o). The authors implement this using parameter-efficient fine-tuning (LoRA) and three distinct unlearning objectives:

  1. Gradient Ascent (GA): An aggressive approach that reduces the likelihood of spurious labels on the forget set.

  2. KL-regularized unlearning (KL): A method that keeps the updated model close to a reference model to improve stability and reduce drift on retain samples.

  3. Negative Preference Optimization (NPO): A targeted approach that penalizes spurious predictions relative to the reference model, focusing on forget labels that maintain high confidence.

Experimental validation

The PURGE framework was evaluated across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, using a diverse suite of benchmarks such as CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. The results demonstrate that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings. Notably, the PURGE-M-NPO variant showed significant success in reducing object hallucination on the CHAIR benchmark and improving robustness to spurious correlations on the Waterbirds and MM-SpuBench datasets.

Improvements for AI systems

Based on this research, the core improvements must focus on transforming large, knowledge-rich models from merely associative pattern matchers into causally robust and visually grounded reasoning engines. The gains are not linear; they require a synergistic combination of mechanisms.

Here are the specific architectural and methodological improvements required for an AI system:

Improvement: Integrate a dedicated, differentiable unlearning module that operates on the model's parameter space. This module must be specifically designed to identify and mitigate knowledge derived from spurious correlations or non-causal dependencies learned during pretraining (e.g., relying on background context rather than the primary object for classification).

Technical Specification:

  1. The unlearning process must be partition-aware, meaning it selectively targets specific weight partitions responsible for hallucination pathways, rather than simply applying global regularization.

  2. It must operate in a fine-tuning loop, systematically purging the model's memory of statistically common but factually incorrect associations (the source of high CH and Hal).

What the Improved System Can Do:

  • Achieve Factual Consistency: The system will exhibit a measurable, quantitative reduction in hallucination rates (e.g., reducing CH from 5.9 to 2.8 on Qwen3-VL benchmarks).

  • Improve Reasoning Robustness: It will resist misleading inputs or contextual noise by focusing its reasoning solely on the necessary, causal links between visual evidence and text prompts, significantly increasing reliability in high-stakes applications (e.g., medical diagnosis support, legal document analysis).

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (artition-aware nlearning for emoving spurious-correlation enerated rrors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of: -- (1) Structured dataset construction wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and -- (2) Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs.

Sources

Related papers