Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

arXiv:2412.16555 · cs.CL · Submitted 2024-12-21 · Read on arXiv

cs.CL

Submitted: 2024-12-21

Updated: 2026-09-16

Code: https://github.com/metallama/PurpleLlama

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Ablation — 14× in this paper · explained in 1 episode
• Alternating Translation — 10× in this paper
• Word Encryption — 10× in this paper
• Benchmark — 9× in this paper · explained in 2 episodes
• Feature Collapse — 9× in this paper
• Low-Resource Languages — 7× in this paper · explained in 4 episodes
• Ablation study — 6× in this paper · explained in 1 episode
• Transformer — 5× in this paper · explained in 2 episodes
• Adversarial Attacks — 3× in this paper · explained in 1 episode
• Alignment — 3× in this paper · explained in 1 episode
• Generative Model — 3× in this paper · explained in 2 episodes
• Generative Models — 3× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Attack Success Rate (ASR) — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Iterative Refinement — 2× in this paper · explained in 3 episodes
• Latent Space — 2× in this paper · explained in 7 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Adversarial Examples — 1× in this paper · explained in 2 episodes
• Adversarial Robustness — 1× in this paper · explained in 3 episodes
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Guided attention — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Multimodal Dataset — 1× in this paper · explained in 2 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• Jailbreaking Black Box Large Language Models in Twenty Queries
• StruQ: Defending Against Prompt Injection with Structured Queries
• SecAlign: Defending Against Prompt Injection with Preference Optimization
• MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
• Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character
• GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction
• Are You Human? An Adversarial Benchmark to Expose LLMs
• Qwen2.5-Coder Technical Report
• Jailbreaking Attack against Multimodal Large Language Model
• A Cross-Language Investigation into Jailbreak Attacks in Large Language Models
• From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
• Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step
• Voice Jailbreak Attacks Against GPT-4o
• ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
• Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering