IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
Ankur Singh, Jinqiu Yang, Tse-Hsun Chen
cs.CR, cs.AI, cs.SE
Submitted: 2026-07-22
Comments: 10 pages, 4 figures, 4 tables
Code: https://github.com/sympy/sympy
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools.
Terminology
Abstract
AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.
Sources
- When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- Agentic Much? Adoption of Coding Agents on GitHub
- MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants
- DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions
- A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
- Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems
- "Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors
- Defending Against Indirect Prompt Injection Attacks With Spotlighting
- Defense Against Indirect Prompt Injection via Tool Result Parsing
- Defeating Prompt Injections by Design
- LlamaFirewall: An open source guardrail system for building secure AI agents
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
- Design Patterns for Securing LLM Agents against Prompt Injections
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs