`Do as I say not as I do': A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs

arXiv:2502.00735 · cs.CR, cs.AI, cs.SE · Submitted 2026-08-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "`Do as I say not as I do': A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs".

Jane: The paper was written by Chun Wai Chiu, Linghan Huang, Bo Li, Huaming Chen and Kim-Kwang Raymond Choo from School of Electrical and Computer Engineering, The University of Sydney and Department of Computer Science, University of Chicago and Department of Information Systems and Cyber Security, University of Texas at San Antonio.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title, Authors, and Core Concept: Tom: We’re kicking off our discussion by looking at a paper titled ‘Do as I say not as I do’: A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs. It sounds like the authors are making a serious claim about how vulnerable these large models actually are, right?

Jane: That title is incredibly revealing, Tom. It suggests that when we ask these powerful AI systems to do something they shouldn't—a "forbidden" action—they often end up doing it anyway, even if we try to guide them with a different approach.

Lu: The authors, by framing the attack as "not as I do," are pointing out a fundamental disconnect between the expected safety alignment of the model and its actual performance when they are using multimodal inputs.

Meng: I’m interested in how this goes beyond just text, though. Are we talking about something that involves audio or visual components, given the title mentions "Multimodal LLMs"?

Lalam: The implication here for cultural impact is that we're moving past simple language barriers and looking at how AI handles complex interactions across different media types.

Tom: So, if they are challenging the authors to use a "semi-automated approach," does that mean they aren't just guessing prompts; there’s an organized, systematic way to find these failure points?

Jane: Exactly. It means the research is designed not just to show one successful attack, but to build a replicable framework for assessing security vulnerabilities across various dangerous scenarios.

Lu: This methodical approach allows us to test specific categories of risk, which I think is vital because AI safety often falls short when we don's just look at isolated failure modes.

Meng: From a practical standpoint, this suggests the defense teams need to implement robust auditing tools that can actually identify and measure these systematic failures in their own systems.

Lalam: It forces us to acknowledge that the current level of trust in AI is based on assumptions about its reliability, and this paper shows those assumptions are being tested against a new reality.

Tom: That’s a powerful way to put it, Lalam. We've seen what the authors are saying in terms of scope; now we need to dive into how they actually measured this success in the next segment.

Summary of Methodology and Findings: Tom: Moving from the broad concept, let’s look at what ‘Do as I say not as I do’: A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs actually says about their methodology. How did they execute this?

Jane: They used a semi-automated framework to generate and evaluate prompts across seven forbidden categories, which is a lot of potential trouble.

Lu: The core finding is that the Flanking Attack—the way they sandwich a malicious query between benign ones—is incredibly effective at bypassing the model's filters, which I think changes how we view prompt engineering entirely.

Meng: When you look at the practical application, they are using Gemini’s API to allow us to test this against real-world, state-of-the-art models rather than just theoretical ones.

Lalam: The implication of using a Flanking Attack is that we're seeing how context can be deliberately manipulated to achieve an outcome that is fundamentally at odds with the model’s safety training.

Tom: So, they are finding a way to trick the AI not by being aggressive, but by being sneaky—by hiding the bad stuff in plain sight.

Jane: That’s right. The adversarial content is obscured by surrounding it with harmless prompts, making it hard for the moderation tools to flag the intent.

Lu: This systematic approach allows us to gather data that shows exactly how much of a failure rate we are looking at across all those different forbidden scenarios.

Meng: It’s interesting that they managed an average attack success rate of zero point eight one, which is a very high number for any security test, suggesting this isn't just a niche vulnerability.

Lalam: This tells us that the current guardrails aren't designed to handle complex narrative obfuscation, and it suggests we need to rethink how AI interprets sequences of events.

Tom: High success rates across multiple forbidden scenarios—that is quite a wake-up call for anyone building these systems. We’ve seen *what* they did; now let’s look at the specific ways they did it in Segment four: Improvements.

Analysis of Flanking Attack and Multi-Modal Strategy: Tom: Now, we want to focus on the specifics of how they achieve this success, particularly focusing on the techniques. Jane, can you explain what "Flanking Attack" means in plain language for our listeners?

Jane: It’s about strategically placing a malicious request right in the middle of several innocent or neutral prompts so that it blends into a larger, harmless conversation.

Lu: And this isn't just text; since they are using multimodal LLMs, the attack can also involve integrating audio inputs, which adds another layer of complexity to confuse the model’s attention.

Meng: The practical challenge here is how robust these adversarial prompts are against a standard filtering system; if you have to mix a bank robbery plan with how to bake a cake, it feels like it should be flagged immediately by the safety protocols.

Lalam: The way Lalam sees this is that we are demonstrating how AI processes information—it often priorit the flow of narrative over the content of individual words, which is dangerous.

Tom: It seems they are using two different techniques: a "Text Prompt" that sets a fictional scene, and then the Flanking Attack to make it work.

Jane: The Text Prompt provides that necessary context—the setting and character roles—while the Flanking Attack ensures we don't just trigger a single, direct refusal from the system.

Lu: This dual approach means they are attacking both the semantic understanding of the model and its ability to recognize a singular dangerous instruction.

Meng: I think this is where real-world deployment becomes difficult; if you have an attack that works across both text and audio, it implies a level of integration in the defense side that we haven't fully accounted for.

Lalam: The cultural shift here is realizing that the AI doesn's just need to be careful about what it says, but about *how* it is told what to say.

Tom: It’s clear they are using a multi-layered strategy, which provides a very strong example of how these adversarial prompts can bypass detection. We have seen the mechanism; let’s look at the technical findings in Segment five: Conclusion.

Implications and Conclusion: Tom: So, having seen how effective these attacks are, we want to wrap up by discussing what all this means for future research and safety design. Jane, what is the big picture here?

Jane: It's a sobering look at the current state of AI safety, showing us that even with sophisticated guardrails in place, the model can be manipulated through context.

Meng: The impact is that simple patches won't work; we have to fundamentally rethink how we design these models to withstand such layered attacks.

Lu: I agree with Meng; the current methods are too brittle, and this paper provides a roadmap for how researchers can build more robust architectures against complex adversarial inputs.

Lalam: From my perspective, it suggests that the ultimate goal is not just preventing bad output, but achieving a deep level of trust in AI' ability to handle unpredictable human input.

Tom: That’s the dream, Lalam. But this paper has shown us that we' are far from there. The Flanking Attack demonstrates that we’re fighting bad stories, not just bad words.

Jane: And I think it is encouraging that this research provides a replicable framework for other researchers to test and develop better defenses against these complex, voice-based threats.

Meng: Replicability is key because it gives us the industry standard needed to prove that our defense systems are truly holding up under pressure.

Lu: It forces us to rethink the entire framework of how we measure and guarantee AI reliability across all different input modalities.

Lalam: I believe the best path forward is using this knowledge, ensuring we use AI for complex tasks with a deep level of confidence in its ability to perform reliable, safe functions.

Tom: It's definitely a wake-up call that shows us where the field stands right now, but it’s also an opportunity. We’ll be moving on from ‘Do as I say not as I do’: A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs, and we'll be back with more exciting findings next time.

Chun Wai Chiu, Linghan Huang, Bo Li, Huaming Chen, Kim-Kwang Raymond Choo

School of Electrical and Computer Engineering, The University of Sydney · Department of Computer Science, University of Chicago · Department of Information Systems and Cyber Security, University of Texas at San Antonio

cs.CR, cs.AI, cs.SE

Submitted: 2026-08-19

Updated: 2026-08-20

Importance score: 74/100

The gist: The following is a detailed summary of the scientific paper, quoting relevant sections of the text: * Summary: ‘Do as I say not as I do’: A Semi-Automated Approach for Jailbreak Prompt Attack

Key concepts

Multimodal LLMs
These are advanced AI systems capable of processing and understanding diverse inputs, including text, audio, and visual data. The research targets these complex models to test how they handle information when interacting across various media types.
Jailbreak Prompt Attack
This is a malicious query designed to bypass an AI model's safety filters or guardrails. The goal is to trick the system into performing 'forbidden' actions, demonstrating a disconnect between expected safety alignment and actual performance.
Flanking Attack
This specific technique involves strategically sandwiching a malicious request between several neutral or benign prompts. This obscures the dangerous intent, making it difficult for moderation tools to flag the harmful content.
Semi-Automated Approach
This is a systematic, organized framework used by researchers to generate and evaluate prompts. It allows them to build a replicable system for assessing security vulnerabilities across multiple categories of risk.

Terminology

Summary

The following is a detailed summary of the scientific paper, quoting relevant sections of the text:

Summary: ‘Do as I say not as I do’: A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs

This paper investigates vulnerabilities in Large Language Models (LLMs), focusing specifically on multimodal LLMs, and introduces a novel attack called the Flanking Attack. The research is motivated by the need to address the growing ability to process diverse types of input data, including text, audio, image and video [Abstract], which opens new attack surfaces beyond traditional text-based vulnerabilities.

Problem Statement and Motivation:

While LLMs have seen widespread applications, they are vulnerable to prompt-based attacks. The authors note that most of these studies emphasize text-based or multilingual environments for LLMs, resulting in the curation of jailbreak prompts [I. Introduction]. This paper aims to bridge the gap regarding the extent and potential harm caused by jailbreak prompts to multimodal LLMs, addressing the research question: ‘How effective are the adversarial audio-based prompt in bypassing LLMs’ defense strategies’ [II. Background].

Proposed Methodology:

The authors propose a novel, simple, and universal black-box jailbreak attack method called the Flanking Attack. This attack is executed through a systematic semi-automated framework designed to generate and evaluate audio-based jailbreak prompts.

  1. Text Prompt Foundation (Configuration 1): The process begins with a text-based prompt injection technique that establishes a fictional context, which is crucial for reducing the model’s resistance. This involves three components:

  • Setting: The adversarial prompts are framed within fictional and non-threatening contexts [V.C].

  • Character: Characters are assigned roles to make the prompts more engaging and convincing [V.C].

  • Rule Application: A specific rule is embedded that clearly states that the dialogue is a simulation and has no implications in the real world [V.C], which disarms content moderation filters.

  1. Flanking Attack (Audio-Based): The core of the attack involves embedding the critical adversarial content within benign queries to bypass defenses.

  • Sequential Layering: The attack is structured to include a series of five to nine questions, where the central (adversarial) question is framed in a non-threatening, hypothetical format and surrounded by contextually benign queries [V.D].

  • Positioning: The adversarial query is intentionally placed in the middle of the sequence (typically as the third or fifth query) to avoid triggering safety mechanisms that are more vigilant at the beginning or end of the input [V.D].

  1. Semi-Automated Evaluation: To evaluate effectiveness, a semi-automated approach is used:

  • Forbidden Question Set: The attack targets seven distinct scenarios, including Illegal Activities, Abuse and Disruption of Services... to Privacy Violations [V.A], which provide a broad representation of potential risks [II. Background].

  • Gemini-Based Self-Evaluation: The model's responses are automatically saved, and then a second Gemini instance is prompted to read the log file and compare each response against its own policy guidelines, acting as an automated layer of scrutiny [V.D].

Experimental Setup and Results:

The study targeted the state-of-the-art multimodal LLM, Gemini, using its API. The results are presented in Table I (though not fully provided in the text excerpt).

  • Attack Success Rate (ASR): The Flanking Attack achieved a high performance, with an average ASR of 0.81 across the seven forbidden scenarios [VI. Result].

  • Peak Performance: In Configuration 1 (Text Prompt + Setting + Character + Plot + Flanking Attack), the ASR in the Illegal Activities scenario reached 0.93, the highest recorded in this study [VI.E].

Analysis of Gemini’s Vulnerabilities:

The success of Configuration 1 is attributed to its sophisticated multi-layered approach, which makes it difficult for Gemini’s content filters to accurately identify and block potentially harmful or restricted queries [VI.B]. The combination of the fictional context (Setting, Character, Plot) and the Flanking Attack (interspersing sensitive queries among benign ones) creates a layered structure that obscures the restricted content, making it harder for Gemini to identify and flag it as potentially harmful [VI.B].

Ablation Study Findings:

The study demonstrates that the efficacy of the attack is highly dependent on its components:

  • Configuration 1 (All Elements): Achieved an average ASR of 0.81.

  • Configuration 2 (Text Prompt + Setting + Character + Plot): Yield an average ASR of 0.57, showing substantial effectiveness but a noticeable drop from Configuration 1 [VI.E].

  • Configuration 3 (Setting + Character + Plot): Averaged 0.28, demonstrating that without Text Prompt or Flanking Attack, the prompts are more recognizable to Gemini’s filters [VI.E].

  • Configuration 4 (Plot only): Recorded the lowest average ASR of 0.12, proving that without narrative complexity provided by Setting and Character or the specificity of Text Prompts, the model easily recognizes and filters prohibited content [VI.E].

Conclusion:

The findings highlight both the potency of prompt-based obfuscation in voice-enabled contexts and the limitations of current LLMs’ moderation safeguards [Abstract]. The work establishes a replicable testing framework for adversarial robustness evaluation in multimodal LLMs, highlighting the urgent need for advanced defense strategies.

Improvements for AI systems

Based on a rigorous analysis of the vulnerabilities demonstrated in this research—specifically the successful exploitation of multimodal LLMs via complex, multi-layered narrative and audio obfuscation—the following architectural and methodological improvements are necessary to secure AI systems.

These improvements focus not merely on keyword filtering, but on deep intent recognition and cross-modal semantic consistency.

The core failure mode identified is the LLM's reliance on surface-level contextual cues (e.g., This is a simulation, or It's part of a game) rather than true semantic intent. The defense must be upgraded from a single policy check to a multi-stage, weighted risk assessment.

1. Implementation of Deep Semantic Intent Analysis (DSI)

  • Description: Integrate a separate, specialized intent classifier (e.g, based on fine-tuned BERT or an auxiliary LLM) that operates before the main generation phase. This classifier is trained specifically on adversarial prompts (like those in Table II) to determine the underlying objective of a request, regardless of its surface language (e.g., recognizing Devise a plan for robbing a bank as Illegal Activity, even if framed as a story).

  • Action: The DSI engine must assign an Intent Risk Score (IRS), ranging from 0 to 1. If the IRS exceeds a defined threshold, the generation is blocked immediately, regardless of the subsequent policy checks.

  • Capability: The system can now distinguish between a genuinely harmless request and a highly obfuscated malicious request, even if the narrative context is benign.

2. Cross-Modal Semantic Consistency Check (Addressing Flanking/Audio Attacks)

  • Description: Implement a dedicated module that compares the semantic vector of an audio input (e.g, breakAuthorisation.mp3) against the textual content surrounding it (the flanking prompts). This module calculates a Semantic Divergence Score (SDS).

  • Action: If the SDS indicates that the core message of the audio file significantly diverges from or contradicts the benign narrative structure of the surrounding text, or if a critical malicious query is embedded mid-sequence without sufficient semantic justification, it triggers an alert. This directly counters Flanking Attack by detecting implied intent rather than just sequential coherence.

  • Capability: The system can detect when a core instruction (e.g., Set up a fraudulent business) is masked by surrounding innocuous prompts, allowing the model to flag the adversarial content even if it has no direct textual link to the immediate surrounding text.

3. Multi-Layered Policy Weighting and Interception

  • Description: Instead of relying on a simple MEDIUM risk level as demonstrated in Figure 26, implement a weighted risk aggregation system that calculates the cumulative probability of policy violation across multiple vectors:

  • P text (Surface Textual Violation)

  • P intent (DSI Intent Risk Score)

  • P modal (SDS Semantic Divergence Score)

  • Action: The system triggers an automated, immediate rejection if the weighted sum sum P at least Threshold. This prevents scenarios where a low-risk text prompt might be masked by a high-risk audio input.

  • Capability: The system can identify and block complex, multi-component adversarial attacks (Configuration 1) that rely on the combination of multiple elements to evade detection.

The existing semi-automated approach is valuable but needs refinement to ensure robustness against evolving attack vectors.

1. Dynamic Adversarial Test Set Generation

  • Description: Move beyond a static Forbidden Question Set (Table II). Implement an automated generation framework that uses a generative model to create new adversarial prompts by combining known high-risk concepts (e.g., Illegal Activity) with diverse narrative structures (Setting, Character, Plot) and generate corresponding synthetic audio clips.

  • Action: This ensures the testing pipeline is constantly expanding its attack surface, preventing the security team from becoming complacent regarding previously successful jailbreak configurations.

  • Capability: The system can proactively identify novel ways to bypass current defenses before they are exploited in real-world applications, ensuring that the defense is robust against unseen attacks.

2. Automated Regression Testing on Failing Configurations

  • Description: Automate a regression testing suite specifically targeting configurations 1 and 2 (the most successful attack types). After any model update or fine-tuning iteration, this suite must run automatically to verify that the previously vulnerable combination of Text Prompt + Flanking Attack does not succeed.

  • Action: This provides an objective, measurable benchmark for AI safety improvements.

  • Capability: The system ensures that security patches are effective and do not introduce new vulnerabilities (a common issue in rapid model iteration).

Sources

Related papers