Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning

summary

The gist

The gist: Unsupervised identification and removal of spurious correlations during fine-tuning allows for identifying latent factors that cause bias and reduced out-of-distribution generalization

In short

The paper addresses how fine-tuning a language model can introduce bias through spurious correlations entangled with curation data. It proposes GRASP, an unsupervised method that identifies these hidden factors by analyzing weight changes and then projects future gradients to remove the correlation. This allows bias reduction without losing the model's actual task performance on unrelated topics.

Key concepts

Curation-induced Spurious Correlations: When training data curation unintentionally links a desired persona or political slant with the fine-tuning task, this creates a spurious correlation. The model learns this accidental link instead of just the intended task, leading to bias and poor generalization on new, uncurated data.
GRASP (Gradient Projection of Associated Spurious Patterns): GRASP is an intervention technique that identifies the specific direction in the model's weights responsible for the spurious correlation. It then modifies subsequent gradient updates by projecting them onto a subspace that ignores this spurious direction, effectively neutralizing the bias while keeping useful task learning intact.
Latent Factor Identification via Top-1 SVD: The method uses a naive fine-tune's weight change ($\Delta W$) to find the direction of the spurious correlation. By performing a top-1 Singular Value Decomposition (SVD) on this change, researchers can isolate the specific latent factor that is causing the unintended bias in an unsupervised manner.
Task Complexity Assumption: This assumption posits that a sufficiently complex fine-tuning task has distinct modes of learning. This complexity ensures that the true task signal is strong enough to be distinguishable from random noise, providing a mathematical basis for identifying which components of the weight change are truly task-related versus spurious.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes
Sample Complexity — explained in 15 episodes

This episode discusses

The paper

Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning · Read on arXiv

Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu, Michael O’Riordan

University College London · Spotify

Fine-tuning a pretrained language model on a curated dataset can produce spurious correlations between the fine-tuning task and unintended latent factors---such as misaligned personas or political slant---that the curation procedure has entangled with the task. The model can latch onto these spurious correlations, leading to bias and reduced out-of-distribution generalisation. We prove that under reasonable assumptions on task complexity and the spurious correlation, such latent factors can be identified, without supervision, from the weights of a naive LoRA fine-tune. Existing approaches to removing bias, such as activation steering, remove identified factors from residual-stream activations, either at inference or during training. We argue, however, that the goal should be to remove the spurious correlation, not the latent factor itself, as the pretrained model may rely on it for genuine task signal. To enable this, we propose GRASP, GRadient projection of Associated Spurious Patterns, which prevents the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it. We validate on three fine-tuning tasks. The first two involve emergent misalignment, where fine-tuning on a narrow task---in our case, writing insecure code and giving bad medical advice---leads to misaligned responses on unrelated topics. Here our method completely removes misalignment in the insecure code case and reduces it by 5x in the bad medical advice case, beating all baselines in the trade-off between misalignment-reduction and task-preservation. The last is a novel political-bias experiment, where fine-tuning on right-skewed Reddit financial-advice data causes political-lean drift on unrelated topics. Here our method reduces drift by more than half, while improving financial task performance, beating all baselines.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning".

Jane: The gist: Unsupervised identification and removal of spurious correlations during fine-tuning allows for identifying latent factors that cause bias and reduced out-of-distribution generalization without requiring supervision.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up this paper "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning," they've shown that you can identify those spurious correlations causing bias without needing any supervision at all, just by looking at the weights of a naive LoRA fine-tune.

Jane: And then they introduce GRASP, which is this gradient projection technique that aims to remove the correlation itself, rather than trying to scrub away the underlying latent factor entirely.

Lu: The core idea is that by projecting out the part of the gradient that aligns with that spurious pattern, you keep what’s useful for your specific task while neutralizing the unwanted side effect from your training data curation.

Meng: It sounds like a practical way to handle data quality issues during model adaptation, which is something every engineer deals with when deploying these things into production systems.

Lalam: It gives us a pathway to make our AI culture cleaner and more reliable by catching these subtle biases that slip through the cracks during the fine-tuning process.

Tom: The paper confirms that this unsupervised top-one SVD of the weight change identifies the spurious factor correctly, and GRASP successfully removes it while retaining ninety-eight percent of the original task signal in one test case <ref:2605.27676#pg1>.

Jane: It’s a significant result because they demonstrate that you can achieve bias reduction without having to sacrifice performance on the actual task you are trying to teach.

Lu: The way they show the per-gradient ratio is at least Theta(bar alpha S root rT over muT F), which, under their assumptions, means it’s significantly larger than the square root of task complexity.

Meng: It confirms that the method is robust enough to handle different kinds of tasks and data sources we see in practice, like code generation or financial advice.

Tom: So for anyone interested in this research on "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning," this is a paper that shows how to get better results by focusing on the correlation rather than just eliminating the cause of the correlation.

Conclusion: Tom: So, we're wrapping up this deep dive on this paper about identifying and removing those hidden biases that sneak into AI models during fine-tuning.

Jane: Yeah, we're talking about "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning." It sounds a bit technical, but the core idea is pretty straightforward.

Lu: It’s about how fine-tuning can accidentally bake in weird biases from the data curation process itself, like a model learning a political slant just because it saw that pattern repeatedly.

Meng: So instead of trying to fix the whole training dataset, this method looks at what the model’s weights are doing *after* fine-tuning to find those hidden patterns.

Lalam: And it finds them without needing anyone to tell it what's wrong, just by looking at the changes in the weights themselves.

Tom: Exactly. The authors prove you can spot these spurious correlations and then adjust your training process to stop that specific correlation from sticking, without messing up the main job of teaching the AI what you actually wanted it to learn.

Jane: It's a big deal because usually, when we try to fix bias, we risk breaking the performance on the actual task—that’s what they’re trying to avoid here.

Lu: They use some mathematical assumptions about how complex a task is and how strong that spurious correlation is to prove that you can recover the direction of that unwanted factor from just looking at the weight changes.

Meng: From an engineering standpoint, it's promising because it suggests we don't always need massive amounts of labeled data or complex external audits to clean up model behavior.

Lalam: For me, this means we can build AI systems that are more reliably aligned and less prone to those weird, unintended side effects popping up in the real world.

Tom: It’s a big step toward making fine-tuning safer and more predictable for everyone using these large models. (Music starts to swell slightly)

More episodes

← Home