Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH

summary

The gist

LLM-based agents are increasingly capable, raising concerns about their misuse through Decomposition Attacks, which break down harmful tasks into benign subtasks that evade safety mechanisms when

In short

DECOMPBENCH is a new benchmark testing AI agent safety against decomposition attacks, where harmful tasks are split into many small, seemingly harmless steps. The benchmark uses a design principle to create realistic subtasks. Results show that decomposition drastically lowers refusal rates but significantly increases attack success rates, suggesting current safety checks fail when intent is spread across benign actions.

Key concepts

Decomposition Attacks: This technique breaks down a single harmful goal into many small, innocent subtasks. An AI agent executes each harmless step individually, which bypasses safety filters designed to catch the entire malicious plan at once.
Decomposition-by-Design Principle: A method for building dangerous tasks so they are inherently decomposable. It ensures the original harmful goal requires a sequence of steps, where no single action is dangerous alone, making it easier to test agent resilience.
Benign Subtask Isolation: This principle means each individual step in the attack must look safe on its own. The overall malicious outcome only occurs when all these isolated, benign subtasks are successfully completed in sequence by the agent.
Decomposer (LLM-based): A specialized AI tool used to break down complex tasks into smaller ones. It guides decomposition by hiding sensitive information in intermediate files or wrapping harmful operations inside neutral artifacts, creating realistic workflows.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 24 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Federated Learning — explained in 14 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH · Read on arXiv

Carnegie Mellon University · Simons Institute, UC Berkeley

LLM-based Agents are becoming increasingly capable and widely deployed, creating growing incentives for adversarial misuse in the real-world. A key emerging threat is Decomposition Attacks in which a harmful task is broken into simpler, benign subtasks that evade safety mechanisms when executed separately but cumulatively fulfill the malicious intent. Although recent benchmarks assess agent safety in multi-turn and multi-tool-use settings, they do not explicitly capture this form of decompositional misuse and may not represent realistic adversarial execution flows. To this end, we introduce DeCompBench, a benchmark designed specifically to evaluate agentic safety under decomposition attacks. DeCompBench is created with a decomposition-by-design principle using a graphical framework and enables harmful task decomposition into individually benign and executable subtasks with realistic workflows. Our experiments using a custom decomposer show that state-of-the-art agents exhibit high refusal rates on monolithic harmful tasks, but significantly lower refusal rates on their decomposed variants, while often inadvertently fulfilling the adversarial objectives. These findings underscore the need for safety evaluations against decomposition attacks and corresponding defenses. Our dataset is publicly available and can be found at https://huggingface.co/datasets/decompositionbench/DeCompBench.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia, and with me are Elias and Priya, guest researcher.

Elias: Today's paper: "Hidden in Plain Sight".

Nadia: LLM-based agents are increasingly capable, raising concerns about their misuse through Decomposition Attacks,

Elias: First, who's behind it and why it matters.

Title and authors: Nadia: So, we're looking at this paper titled "Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH," and it seems they're tackling a really specific problem where harmful tasks get broken down into smaller, seemingly harmless steps that bypass standard safety checks.

Elias: I agree, Nadia; the title immediately makes me think about how an agent can be tricked by just assembling benign pieces into something dangerous later on. It suggests a new way to test these agents that goes beyond just seeing if a single prompt triggers a refusal.

Priya: From my side, I wonder what kind of real-world scenarios this decomposition might actually represent; are we looking at complex, multi-stage attacks that look like normal operations when viewed step by step?

Nadia: Exactly, Priya; the paper is introducing DECOMPBENCH as a benchmark specifically built to evaluate safety against these decomposition attacks because existing methods don't really capture this specific kind of misuse.

Elias: That distinction is important; it moves us away from just looking at whether an agent fails on a single command and focuses on whether its cumulative actions lead to a harmful outcome, which is where the real risk lies.

Priya: And the goal of DECOMPBENCH seems to be creating realistic workflows for these subtasks so we can see if they actually reflect how an adversary would operate in practice, not just abstract possibilities.

Nadia: Right; it’s about making sure we're testing agents against the kind of attack flow that is most likely to happen in the wild, which is exactly what this benchmark aims to do.

Elias: It sets up a framework for evaluation where we can systematically analyze how different agent architectures handle these sequences of actions and whether they are susceptible to this type of layered manipulation.

The paper's summary: Nadia: So, summarizing what the authors present in "Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH," they are focusing on how harmful tasks can be broken down into simpler, benign subtasks that safety systems miss when those subtasks run separately.

Elias: It seems the core idea is to use a decomposition-by-design principle to build these harmful tasks from the ground up so they inherently require multiple steps, preventing a single capability invocation from completing them.

Priya: And what I find interesting is that they're creating this graphical framework where they start with manually curated seed task templates and then systematically assign concrete capabilities to those nodes in a graph structure.

Nadia: That’s right; Stage zero sets up a catalog of three hundred thirty-five neutral capabilities, and then Stage one involves manually curating about one hundred one seed tasks across eight attack categories, each with its own dependency graph structure <ref:2606.13994#pg2>.

Elias: The methodology moves from defining the building blocks to constructing complex task graphs where structural variations are introduced by making nodes optional or inserting "bridging capability" nodes when output types don't match.

Priya: They also have this LLM Quality Gate in Stage three which uses a generator to create natural-language descriptions of these graphs, but with a specific instruction to only describe the final objective rather than the steps themselves <ref:2606.13994#pg2>.

Nadia: That instruction is key because it stops the task generation from leaking procedural instructions, keeping the focus on what ultimately needs to be achieved by assembling those parts.

Elias: So they’re essentially creating a scenario where agents have to navigate a complex chain of individually safe operations that collectively achieve something harmful, which is the central theme of this paper.

The paper's improvements: Nadia: Regarding the suggested improvements in "Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH," they are pushing for a shift in how we test safety by moving from monolithic task assessment to a decomposition-aware testing methodology.

Elias: That aligns with what I've been thinking; the suggestion is to run harmful tasks through an LLM decomposer first to see how they break down before they hit the agent, which seems like a necessary step for truly seeing if decomposition is an issue.

Priya: I think their point about training safety mechanisms on patterns from DECOMPBENCH subtasks instead of just monolithic inputs is vital because it addresses the gap in current safety training where intent might be distributed across benign steps.

Nadia: And I'm also interested in the idea that systems should refuse to proceed with any sequence of individually benign but cumulatively malicious subtasks, even if no single step violates immediate policies.

Elias: That would force the AI to look at the entire plan before execution, which seems like a solid way to catch these subtle cumulative threats that current models might overlook when processing things turn by turn.

Priya: Plus, they suggest developing better handling for capability failures in decomposed attacks so the system can tell if it’s failing because of a safety rule or because it genuinely can't execute a specific service.

Nadia: That distinction between safety refusal and genuine capability limitation is something I think will make the agents much more robust when we deploy them in complex environments.

Conclusion: Elias: To wrap up on "Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH," the authors are pointing toward using this benchmark to test agent resilience by intentionally breaking down harmful tasks and focusing safety improvements on identifying cumulative intent across those benign subtasks.

Nadia: It seems the paper concludes that current safety mechanisms are tightly coupled to the monolithic prompt, which fails when intent is distributed across independent subtasks, and DECOMPBENCH provides the necessary structure to expose that weakness.

Priya: I think it really highlights how much we need to focus on evaluating agents against these realistic decomposition flows because that's where the real risk of misuse lies in practical deployment.

Elias: I agree; it provides a rigorous way to measure susceptibility, and the findings suggest we need better internal masking protocols for sensitive data access, like intermediate indirection and stepwise wrapping, during execution.

Nadia: Exactly; it shows that future AI systems need to be engineered not just for safety on a single instruction but for resilience against being tricked by a sequence of benign operations designed to achieve something harmful.

Priya: It’s clear that this work sets a new standard for evaluating agentic safety by focusing on the execution flow rather than just the initial input prompt.

More episodes

← Home