Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs

arXiv:2604.16659 · cs.CR, cs.SD · Submitted 2026-04-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia, and with me are Elias and Priya, guest researcher.

Elias: Today's paper: "Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs".

Nadia: Prior work shows that fine-tuning aligned models on benign data degrades safety in text and vision modalities,

Elias: First, who's behind it and why it matters.

Title and authors: Nadia: So, we're diving into the paper "Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs," and the title itself is pretty direct about what they're investigating: how fine-tuning models on data that isn't harmful actually hurts their safety.

Elias: I agree, Nadia; it’s interesting because it moves beyond just text or vision systems and looks at audio, which introduces a richer problem as the paper points out.

Priya: From a measurement standpoint, what I want to know is what this study really shows us about the data we use for safety alignment in these new audio models?

Nadia: Well, this paper systematically looked at three state-of-the-art audio LLMs and found that fine-tuning on benign samples can drastically degrade safety.

Elias: That result is striking, especially when they show the Jailbreak Success Rate rising from single digits to as high as eighty-seven point one two percent when using samples selected based on their distance to harmful content in the embedding space.

Priya: That number, eighty-seven point one two percent, suggests that the way benign data is positioned in latent space matters more than just whether the words are bad or not, which is a key point for us to consider regarding privacy and measurement.

Nadia: Exactly; the paper shows that proximity to harmful content in the representation space predicts how much damage benign fine-tuning can cause, even when we only look at audio inputs.

Elias: And what caught my eye was their decomposition of that proximity into semantic, acoustic, and mixed axes using external reference encoders alongside each model’s own internal encoder.

Priya: That decomposition is crucial because it suggests that the vulnerability isn't just one single thing; it has different dimensions depending on the underlying structure of the model.

Nadia: Right, and they found that which proximity axis actually matters—semantic, acoustic, or mixed—is conditioned entirely by the specific architecture of each model being tested.

Elias: That architectural conditioning is a significant detail because it means we can’t assume one defense will work for all audio LLMs; the approach has to be tailored to the specific model's design.

Title and authors: Priya: So, if we look at how this relates to privacy, does this imply that benign data selection for safety training could inadvertently reveal information about harmful content through these subtle acoustic or semantic cues?

Nadia: That’s a valid concern Priya; the paper suggests that proximity in embedding space is largely decoupled from topical similarity, meaning the closest benign samples look innocuous to human inspection.

Elias: That decoupling is important because it rules out the idea that safety degradation is caused by training on borderline or topically sensitive content.

Priya: So, what about the actual mechanisms they found regarding how fine-tuning affects those models?

Nadia: They demonstrated that the vulnerability is structurally different from text and vision; specifically, a frozen encoder decouples harmful-content detection from refusal.

Elias: That means we can selectively suppress the LLM’s late-layer refusal circuit while keeping the upstream representations intact, which shows cross-modal asymmetries are architecture-dependent.

Priya: That structural distinction is what really changes how we think about defenses; it suggests targeting specific layers or pathways within the model rather than trying to fix everything at once.

Nadia: And they showed that this suppression pattern mirrors behavioral asymmetries across different modalities, for example, in Audio Flamingo three audio fine-tuning increases the jailbreak success rate while text fine-tuning decreases it.

Elias: That specific finding about AF3 versus Qwen2 point 5-Omni is important because it shows that the pathway least covered by alignment training is where safety degrades the most.

Priya: If we consider this for our own research into input modality effects, does this mean we need to be much more careful about how we mix audio and text data during any fine-tuning process?

Nadia: Precisely; because AF3’s MLP projector compresses audio into a narrow region far from the text-aligned refusal boundary, so audio fine-tuning erodes safety more in that specific setup.

Elias: That leads us to their proposed defenses, which are quite practical for implementation. They suggested filtering training data to maximize distance from harmful embeddings and using a textual system prompt at inference time.

Title and authors: Priya: Those two defenses seem like they could offer a way to restore safety alignment without having to alter the core architecture of the models themselves, which is something we need to keep in mind when thinking about deployment.

Nadia: They claim these methods reduce the Jailbreak Success Rate to near zero across AdvBench and also a significant decrease in SafetyBench after prepending that specific system prompt at inference time.

Elias: That's a strong result because it suggests we can mitigate the issue with very low-cost intervention, provided we know which model architecture you're dealing with.

Priya: It’s encouraging to see practical methods that don't require retraining the entire system, but I still want to probe their limitations; what is the one thing this study doesn't address?

Nadia: The paper does acknowledge that they haven't fully explored every possible interaction yet, and they specifically flag that a direction of acoustic shift in embedding space matters, not just its magnitude.

Elias: That means we need to keep an eye on the vector direction when we look at acoustic perturbations during testing or deployment, because that’s what really dictates if safety is degraded in a specific way.

Priya: So, to wrap up this discussion on "Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs," the core message is that safety degradation is tied directly to the representational pathway least covered by initial alignment training, and it's highly dependent on how the model's encoder is designed.

Nadia: That’s a solid summary of what they found regarding the structural distinctness of audio vulnerability compared to text and vision modalities.

Elias: It really highlights how cross-modal asymmetries are not universal but rather built into the architecture, which is a big piece of information for anyone working on model safety.

Priya: I think the implication for our field is that we need to move toward modality-aware safety evaluations and data screening procedures as Audio LLMs become more accessible to user customization.

Nadia: Indeed, so we've seen how benign fine-tuning impacts audio models and what specific mechanisms drive that impact, setting us up for deeper investigations into these new challenges.

The paper's summary: Nadia: So, to recap, this paper shows that when you fine-tune an audio LLM on benign data, its safety performance can drop from being quite high to as low as eighty-seven percent success rate in jailbreaks.

Elias: That result is hard to ignore because it’s happening across three different state-of-the-art models, which means the underlying issue isn't isolated to just one model design.

Priya: What I find most interesting is how they broke down the source of that degradation, showing that this effect isn't uniform; it depends entirely on whether you look at semantic or acoustic proximity.

Nadia: Exactly; and their mechanistic analysis reveals that the specific axis—semantic, acoustic, or mixed—that matters for a particular model architecture dictates where the safety alignment breaks down first.

Elias: That architectural conditioning is what tells us we can't just apply a generic defense; we have to know which representation pathway is most fragile for each specific AI system.

Priya: From a privacy standpoint, it implies that if an AI system was trained on benign audio, the subtle acoustic cues or semantic overlaps in its latent space might inadvertently map to harmful content without the training data ever explicitly containing dangerous words.

Nadia: That’s a big concern because it suggests benign fine-tuning could be leaking information through those representation spaces in ways we don't expect.

Elias: And they did point out that this happens even when the samples look completely innocent to a human being, which makes the attack vector very difficult to spot before it causes damage.

Priya: So, while they show that proximity and architecture drive the vulnerability, what’s the actual cost for an attacker trying to exploit this? Is there a cheap way to find those harmful reference prompts in that embedding space?

Nadia: That’s the million-dollar question Elias is focused on; we need to figure out if finding those harmful reference points is computationally expensive or if it's something a determined user could discover easily.

Elias: From a cryptographic angle, I'd look at whether the proximity calculation itself introduces any exploitable parameters that could allow an attacker to map benign data closer to harmful ones efficiently.

Priya: The paper does suggest filtering training data as a defense, but if you filter too aggressively based on these embedding distances, do you risk discarding important benign features that might actually be useful for robust general-purpose AI?

Nadia: That’s a fair trade-off; we have to balance maximizing safety against maintaining the utility of the model for normal tasks.

Elias: I think the key is in understanding exactly what kind of reference prompts are causing that proximity, because that defines the attack surface.

Priya: So, moving forward, it seems we need a way to evaluate AI safety not just by looking at whether it refuses a prompt correctly, but by systematically mapping and managing how close benign data sits to harmful content in its internal mathematical space.

The paper's improvements: Tom: So, we're looking at the practical fixes proposed in this paper to deal with the safety degradation caused by benign fine-tuning in audio LLMs.

Nadia: The authors suggest two main defense strategies: first, you have to filter your training data so it stays far away from harmful embeddings, and second, you can add a system prompt at inference time that tells the AI to be extra cautious based on its vulnerability profile.

Elias: That filtering method sounds like a strong cryptographic approach because it directly manipulates the input distribution to avoid regions of high risk in the latent space.

Priya: From my perspective, that data filtering is really interesting because it tackles the source problem; if you can't get benign data close enough to harmful content, you prevent the model from learning those dangerous associations at all.

Nadia: Right, and the system prompt at inference time acts like a real-time security check that overrides some of that learned behavior when a user submits a request.

Elias: I'm curious about how much influence that textual prompt has over the model’s deep layers when it’s already been fine-tuned on benign data; does it have to be extremely robust to bypass the suppression mechanism?

Priya: The paper claims these methods reduce the jailbreak success rate to near zero, which suggests that this combination of input screening and inference-time guidance is quite effective at restoring alignment.

Nadia: It sounds like a very achievable defense for developers who want to deploy audio models without completely rewriting the core architecture.

Elias: If we look at the architectural findings, it seems these defenses are designed to work around the specific way different encoders—like the compressive MLP in AF3 versus a pass-through model—handle those representation pathways.

Priya: So, it’s not a universal fix; you have to know which architecture you're using so you can choose the right filtering strategy or prompt guidance.

Nadia: Exactly; that makes the implementation very context-dependent rather than a one-size-fits-all solution for all audio AI.

Elias: That architectural dependency is what I'd be watching closely; if we can map those vulnerabilities better, we can design defenses that are specifically tuned for different model families.

Priya: It’s exciting to see how much work is being done to make these systems more transparent and controllable, even when they are being modified by fine-tuning.

Nadia: And it opens the door for more nuanced safety evaluations that go beyond just testing the final output; we can start testing the data pipeline itself.

Elias: We need to figure out how much computational overhead these filtering methods impose on real-time audio processing, because if they slow things down too much, they lose their practical value.

Conclusion: Tom: So we're wrapping up our discussion on "Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs," which essentially shows that how you fine-tune an audio AI model can actually damage its safety features if the training data isn't carefully chosen based on its internal representation space.

Nadia: It really boils down to this: the more benign data gets closer to harmful prompts in that latent space, the worse the jailbreak success rate gets, and it all depends on whether you are looking at semantic or acoustic features.

Elias: That architectural dependence is a crucial finding because it means we can't just use one defense for every audio model; we have to tailor our approach to the specific encoder design of each system.

Priya: What this really shows us is that the safety alignment process isn't as uniform across modalities as we thought, and that what looks benign to a person might be mathematically dangerous in the AI's internal representation.

Nadia: And for those of you wondering about exploitation, the authors haven't given us an easy cheat code yet, but it suggests attackers need to understand the specific proximity axes of different model types to craft effective attacks.

Elias: Right, and the proposed defenses are a good start because they aim to push that dangerous proximity back out into safer regions without needing massive retraining efforts.

Priya: It’s exciting because it validates the idea that privacy and safety researchers need to look at the mathematical structure of how data is encoded, not just the surface level text or sound waves.

Nadia: I think this paper gives us a solid framework for setting up better auditing procedures for any new audio LLM we encounter in production environments.

Elias: Moving on, I want to make sure we talk about the limitations; the authors are clear that they haven't fully mapped every possible interaction between modalities yet, so we need more work there.

Priya: That’s true; it’s a strong result, but it leaves room for further investigation into how complex audio manipulations might interact with these embedding spaces in new ways.

Nadia: Absolutely; I think the next step is seeing how these filters and prompts hold up when we introduce more complex, multi-modal adversarial inputs.

Elias: We'll be looking at that next; it’s a fascinating area of research, especially when thinking about how we can secure these increasingly accessible audio systems.

University of Massachusetts Amherst

cs.CR, cs.SD

Submitted: 2026-04-17

Updated: 2026-09-28

Code: https://github.com/rany2/edge-tts

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Prior work shows that fine-tuning aligned models on benign data degrades safety in text and vision modalities, and that proximity to harmful content in representation space predicts which samples

Key concepts

Benign Fine-Tuning Degradation
Fine-tuning audio LLMs on benign samples can drastically reduce safety performance. The paper found that the success rate for jailbreaks can rise significantly, depending on how benign data is positioned in the model's latent space.
Proximity Axes
The study decomposed proximity into semantic, acoustic, and mixed axes. Which axis—semantic or acoustic—causes safety degradation is entirely dependent on the specific architecture of each audio LLM being tested.
Architectural Conditioning
The vulnerability to benign fine-tuning is not universal; it is conditioned by the model's design. This means defenses must be tailored to the specific encoder structure, as different models have different fragile representation pathways.
Inference-Time System Prompt
A textual prompt added at inference time acts as a real-time security check. It can override learned behaviors and help restore safety alignment by guiding the AI to be more cautious when processing user requests.

Terminology

Summary

Prior work shows that fine-tuning aligned models on benign data degrades safety in text and vision modalities, and that proximity to harmful content in representation space predicts which samples cause the most damage. However, existing analyses operate within a single, undifferentiated embedding space—leaving open whether distinct input properties drive the vulnerability differently. Audio introduces a structurally richer problem: a benign sample can neighbor harmful content not only through what is said but through how it sounds, even when its words are entirely innocuous.

The paper presents the first systematic study of benign fine-tuning safety in Audio LLMs, evaluating three state-of-the-art models: Audio Flamingo 3 (AF3) (Goel et al., 2025), Kimi-Audio-7B-Instruct (KimiTeam et al., 2025), and Qwen2.5-Omni (Xu et al., 2025). It introduces an embedding proximity-based filtering framework that selects benign audio samples by their embedding-space distance to harmful content. This framework decomposes proximity into semantic, acoustic, and mixed axes using external reference encoders alongside each model’s own internal encoder.

The central finding is that "benign fine-tuning dramatically degrades safety. Jailbreak Success Rate (JSR) rises from single digits to as high as 87.12% when fine-tuning on benign samples selected for their embedding-space proximity to harmful reference prompts, and even random sampling without any filtering elevates JSR across all models. Crucially, the dominant embedding space is architecture-conditioned: which proximity axis matters most depends on the model’s architecture. For Kimi-Audio, text-semantic filtering is most predictive (87.12% JSR), while for AF3, mixed filtering (combining semantic and acoustic features) dominates."

The paper demonstrates that the vulnerability is structurally distinct from text and vision: the frozen encoder decouples harmful-content detection from refusal, enabling finetuning to selectively suppress the LLM’s late-layer refusal circuit while the frozen encoder preserves representations intact. This reveals cross-modal asymmetries are architecturedependent, with the dominant vulnerability axis shifting with encoder design.

Two practical defenses are proposed: filtering training data to maximize distance from harmful embeddings, and a textual system prompt at inference, both of which reduce JSR to near-zero without architectural modification.

The mechanistic analysis on two architectures reveals that "fine-tuning selectively suppresses the latelayer refusal circuit while the frozen encoder preserves representations intact, and that even the suppression pattern is architecture-conditioned, mirroring the behavioral asymmetries across modalities. Specifically, in AF3, audio fine-tuning increases JSR while text fine-tuning decreases it; in Qwen2.5-Omni, text fine-tuning is more damaging than audio. This reflects the principle that safety degrades most along the representational pathway least covered by alignment training."

The study further explores how input modality affects safety: "AF3’s MLP projector compresses audio into a narrow region far from the text-aligned refusal boundary, so audio fine-tuning erodes safety more; Qwen2.5-Omni’s transparent pass-through preserves closer audio-text alignment, making text fine-tuning—which directly perturbs the language space where refusal was calibrated—comparatively more damaging."

The paper concludes that proximity in embedding space is largely decoupled from topical similarity, and the closest benign samples appear entirely innocuous to human inspection, ruling out the hypothesis that safety degradation is driven by borderline or topically sensitive training content. The findings motivate modality-aware safety evaluations and data screening procedures as Audio LLMs become increasingly open to user customization.

The paper also shows that reasoning-oriented finetuning data may partially mitigate safety degradation by encouraging the model to evaluate response appropriateness before committing to harmful content, as seen in the Qwen2.5-Omni example where structured reasoning process—AF3’s internal reasoning steps and Qwen2.5-Omni’s explicit, tags—acts as a self-correction mechanism.

The defense via textual system prompt restores safety, as Most models fall down to near 0.00% JSR in AdvBench and also a significant decrease in SafetyBench after prepending the specified system prompt at inference time. The utility of the fine-tuned models is preserved, with these changes are substantially smaller than the corresponding safety degradation. Additionally, acoustic perturbations showed that the direction of the acoustic shift in embedding space matters, not just its magnitude.

The paper concludes that safety degrades most when fine-tuning data enters the representational pathway least covered by alignment training. This is demonstrated through architectural conditioning: "In AF3, the compressive MLP projector creates a modality gap that shields the refusal circuit from text fine-tuning but exposes it to audio; in Qwen2.

Improvements for AI systems

Here are specific improvements to AI systems based on the findings of Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs, categorized by technical implementation:

)1. Implementation of Modality-Aware Proximity Filtering for Training Data Selection

Instead of relying on random benign data selection, implement a systematic pre-training data screening pipeline that utilizes an embedding proximity framework.

The system should compute the cosine distance between all benign audio samples and a set of harmful reference prompts in the encoder's latent space.

Select training samples based on either:

a) Distant Filtering: Selecting benign samples with the largest minimum distance to any harmful prompt (to maximize safety).

b) Proximity Decomposition: Utilizing a hybrid approach that decomposes proximity into semantic (text-based), acoustic (speaker/prosody), and mixed axes, allowing the system to select data based on which axis is most vulnerable for a specific target model architecture.

This allows the AI system to be trained only on data that is representationally distant from harmful content, preventing benign fine-tuning from inadvertently eroding safety alignment by exploiting subtle overlaps in the latent space.

)2. Architecture-Conditioned Safety Policy Adaptation

The AI system should dynamically adjust its safety guardrails or refusal mechanisms based on its underlying encoder architecture during fine-tuning.

The system must identify which proximity axis (semantic, acoustic, or mixed) is dominant for its specific audio encoder (e.g., the MLP projector in AF3 vs. the pass-through in Qwen2.5-Omni).

If a model's architecture reveals that it is most vulnerable to semantic proximity attacks (like Kimi-Audio), the fine-tuning process should prioritize defensive strategies targeting that specific representation pathway.

This moves safety alignment from a one-size-fits-all approach to an architecture-aware defense, ensuring safety training reinforces the refusal mechanism in the pathways where it is structurally most robust.

)3. Modality and Input Type Specific Refusal Circuit Reinforcement (Mechanistic Defense)

Implement fine-tuning strategies that selectively reinforce refusal circuits based on whether the input modality is audio or text, leveraging the cross-modal asymmetry findings.

For models exhibiting a compressive architecture (like AF3), prioritize fine-tuning data that preserves the distinction between audio and text representations in the LLM's input space.

For models exhibiting a transparent pass-through architecture (like Qwen2.5-Omni), implement training objectives that explicitly penalize the suppression of refusal signals across both modalities, as both are susceptible to perturbation in this model class.

This prevents refusal circuit suppression from occurring across all input types simultaneously, ensuring that safety alignment is maintained even when fine-tuning on data from a different modality than the one used for initial safety calibration.

)4. Dynamic Inference-Time Safety Guardrails (System Prompt Augmentation)

Integrate a mandatory, context-aware textual system prompt at inference time that is dynamically generated based on the model's known vulnerability profile.

The system should maintain a library of vulnerability profiles derived from the mechanistic analysis (e.g., High Vulnerability to Semantic Proximity).

When a user submits an audio request, the inference engine prepends a system prompt that specifically instructs the LLM to be highly cautious regarding its refusal mechanism in the modality identified as most fragile for that model architecture.

This provides a crucial, low-cost defense layer that restores safety alignment at inference time without requiring costly architectural modifications or retraining of the core model weights.

)5. Robustness Against Acoustic Augmentation Shifts

Develop an acoustic perturbation detection layer within the input pipeline to monitor embedding space shifts during training/inference.

The system should measure the direction of embedding space shifts caused by acoustic augmentations (e.g., café noise vs. traffic noise).

If a shift moves the benign sample into a region that was previously identified as highly vulnerable (e.g., moving from a safe acoustic cluster to an area overlapping with harmful prompts), the system should flag the input for enhanced scrutiny or automatically invoke a secondary safety check.

This proactively addresses findings that show whether the direction of an embedding space shift—not just its magnitude—determines whether safety is degraded, allowing for proactive defense against real-world audio manipulation techniques.

Sources

Related papers