The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs

summary

The gist

The alignment training process in large language models often suppresses cultural knowledge through an inequitable safety gate, which requires distinct interventions depending on whether the failure

In short

Alignment training often suppresses cultural knowledge via an inequitable safety gate, which is termed the alignment veto. This research shows that when models refuse sensitive questions, their internal thoughts align more with human values than their free answers. It distinguishes this suppression from representational bias and suggests prompt engineering can partially address suppression failures.

Key concepts

Alignment Veto: This term describes a phenomenon where a model's internal decision-making process (logit distribution) when it refuses to answer a culturally sensitive question matches human survey data more closely than the answers it generates freely. It signifies that safety training actively blocks knowledge aligned with human values.
Suppression Failures: These occur when a model's internal distribution aligns reasonably well with human data, but a safety gate prevents it from expressing that knowledge in its output. These failures are partially fixable using prompt engineering interventions.
Representational Bias Failures: This failure happens when the model's core encoding itself is fundamentally misaligned with human values, requiring changes during the training phase. Unlike suppression failures, these are not easily fixed by simple framing changes.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes
Sample Complexity — explained in 15 episodes

This episode discusses

The paper

The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs · Read on arXiv

Pardis Sadat Zahraei†, Gokhan Tur†, Dilek Hakkani-Tur¨, Ehsaneddin Asgari‡

University of Illinois Urbana-Champaign · Qatar Computing Research Institute

What happens inside a language model when alignment training conflicts with a cultural value it encodes? Across 16 MENA countries, 26 models, and 1.53M human survey responses, we show the answer is suppression, not erasure: at the moment of refusal, a model's internal logit distribution correlates with human survey data more strongly than its freely generated answers. We call this the alignment veto. We distinguish suppression failures (accurate internal distributions blocked at output) from representational bias failures (the encoding itself diverges from human values), and show the two require different interventions. The gate is inequitable: the safety tax reaches 37.6%, with a 19.8% alignment-quality gap between best- and worst-served nations, and native-language prompting widens rather than closes it. Sparse autoencoder analysis corroborated by comparisons across alignment stages identifies a candidate DPO-stage feature mediating suppression in Tulu-3-8B. The gate is real, its costs are unequal, and deciding what it should protect is not a technical question.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "The Alignment Veto".

Jane: The alignment training process in large language models often suppresses cultural knowledge through an inequitable safety gate,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve been looking at this paper, "The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs," and what it really boils down to is that when you tune an AI for safety, it can end up actively pushing out cultural information instead of just deleting it.

Jane: Right. The main thing they show is that the internal math of a model when it’s refusing something culturally sensitive lines up way better with human survey data than what it actually spits out on other stuff. That correlation is what they call the alignment veto.

Lu: It’s wild because this isn't just happening in one spot; they used data from sixteen MENA countries, twenty-six different models, and a million human survey responses to back up this idea of suppression being widespread.

Meng: So it means we have to figure out if the AI is failing because its inner workings are fighting the safety rules, or if the whole way it was trained just got skewed away from what people actually value in a specific region.

Lalam: From my side, this suggests that simply slapping on a safety filter isn't enough; we have to look at both how the model is being gated and how its core encoding itself is shaped during training.

Tom: Exactly. The paper points out that suppression failures can sometimes be fixed with just prompt engineering, but those representational bias failures, where the model’s core understanding is wrong, they need changes in the actual training process.

Jane: And they even found some technical proof for this in a specific model called Tulu-three-8B <ref:2510.13154#pg3>. They identified a particular feature from the DPO stage that seems to be causing this suppression, showing it’s T3 selective and doesn't affect neutral content.

Lu: That suggests we can start looking for these specific features using sparse autoencoder analysis, which is a kind of deep dive into how the model compresses information.

Meng: It changes things because if we can spot these features, maybe we aren't just guessing at why an AI is refusing something; we get a better handle on the mechanism behind it.

Lalam: And that means addressing cultural alignment failures isn't just about a single technical fix anymore; it needs to be a nuanced approach that looks at both the gating and the encoding issues.

Tom: So, what this all means for us is that deciding what cultural values an AI should represent isn't really a simple technical question anymore; it demands these transparent, participatory frameworks.

Jane: That’s the big picture they’re driving toward. We’re moving away from just fixing errors in a box and toward creating systems where the community has a say in what those boxes actually contain.

Tom: And that leads right into how we can start building those kinds of frameworks, which is what we'll be talking about next.

Conclusion: Tom: So we’re wrapping up this look at "The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs." Basically, this paper shows that safety training can end up actively suppressing cultural knowledge instead of just removing it entirely.

Jane: Right. The authors show that when a model refuses something culturally sensitive, its internal logit distribution lines up better with human survey data than what the model actually writes on questions it did answer freely. That’s the alignment veto concept they introduced, which is pretty important to understand.

Lu: They used data from sixteen MENA countries and one million human responses to show this isn't an isolated incident; it happens everywhere. It highlights how widespread this suppression issue is across different regions and models.

Meng: So it means we have to figure out if the AI is failing because its internal math is clashing with the safety rules, or if the whole training process itself has just shaped what people value differently in a specific area.

Lalam: From my side, this suggests that just putting on a safety filter isn't enough; we actually need to look at both how the AI is being gated and how its core encoding was shaped during training.

Tom: Exactly. The paper points out that some of these suppression failures can sometimes be fixed with prompt engineering, but the ones where the model’s core understanding is just wrong need changes in the actual training process.

Jane: And they found some technical proof for this in a specific model called Tulu-three-8B. They identified a feature from the DPO stage that seems to be causing this suppression, and it’s selective for T3 content, but not for neutral stuff.

Lu: That suggests we can start looking for these specific features using sparse autoencoder analysis, which is a way to dig deep into how the model compresses information.

Meng: It changes things because if we can spot those features, maybe we aren't just guessing why an AI is refusing something; we get a better handle on the mechanism behind it.

Lalam: And that means addressing cultural alignment failures isn't just about one technical fix anymore; it needs to be a nuanced approach that looks at both the gating and the encoding issues.

Tom: So, what this all means is that deciding what cultural values an AI should represent isn't just a simple technical question anymore; it demands transparent, participatory frameworks.

Jane: That’s the big picture they’re driving toward. We’re moving away from just fixing errors in a box and toward building systems where the community has a say in what those boxes actually contain.

Tom: And that leads right into how we can start building those kinds of frameworks, which is what we'll be talking about next.

More episodes

← Home