Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

arXiv:2605.29354 · cs.CR, cs.LG · Submitted 2026-05-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills".

Jane: The paper was written by Chia-Yi Hsu, Department of Computer Science, National Yang Ming Chiao Tung University, Chia-Mu Yu, Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University, Chun-Ying Huang, Department of Computer Science, National Yang Ming Chiao Tung University and Jun Sakuma, School of Computing, Institute of Science Tokyo from National Yang Ming Chiao Tung University and Institute of Science Tokyo.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper Summary: Tom: The paper "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills" summarizes a very specific mechanism, which is this Neutral Prompting Attack or NPA.

Jane: Essentially, instead of telling us to use a fake package, the attacker inject these subtle instructions into an agent's "Skill"—a set of persistent guidance—to steer the model toward generating *any* speculative package name.

Lu: It’s not just that the model guesses; it's that it systematically changes its entire internal landscape or distribution of possible suggestions.

Meng: And this change is measurable, which is critical because the paper shows that NPA doesn't just lead to textual mentions; it drives *actionable* risks by increasing the Pip Install ASR.

Lalam: The implication here for us is that our current safety benchmarks are only looking for explicit bad words, not subtle shifts in probability or pattern.

Tom: That’s a huge gap, so it’s important to know how these behavioral changes affect real-world deployment.

Jane: We've seen some very strong results across different LLMs like Qwen2 point 5 and Gemma-three showing the effect is consistent across architectures.

Paper Improvements & Findings: Tom: Now, the paper discusses how to make this attack even more effective while trying to be less detectable—this is where they introduce NPA-Stealth.

Jane: NPA-Stealth is designed to maintain that high hallucination rate but make the Skill look completely benign, using a rewrite-based strategy instead of just relying on random phrasing.

Lu: The way they manage the "holistic contextual framing" in this stealth variant is brilliant; it’s not one specific instruction, but a pervasive bias built into the entire guidance.

Meng: From an engineering view, that means we are moving away from simple content scanning and toward needing systems that understand behavioral intent vs. functional intent.

Lalam: It suggests that our current safety tools are looking at the symptoms rather than the root cause of how AI is being steered.

Tom: It’s a huge challenge for developers to anticipate something so subtle, but we have to look at what they found regarding defense evasion.

Jane: The results show that both static analysis and LLM-based defenses are largely ineffective against these neutral, behavioral steering attacks.

Conclusion: Tom: So, after all this data on "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills," it’s clear that we have a new threat vector.

Jane: We've seen that these harmless-looking prompts can covertly manipulate LLM behavior and create real downstream risks in the software supply chain.

Lu: The paper shows us a future where creativity and grounded factuality are competing concepts within the AI model, which is both exciting and terrifying.

Meng: We have to build systems that are truly aware of this subtle behavioral steering, not just rely on surface-level keyword checks when we implement these coding agents.

Lalam: I hope that by exposing this failure mode, we can drive a conversation toward more robust and ethical ways to build our AI tools.

Tom: It’s certainly a lot for us to process, but it' is vital information for the entire tech community.

Jane: Let’s make sure we all remember "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills" before we move on to the next topic of research.

Conclusion: Tom: So, we've been through all the technical details of "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills," and it’s clear this is a huge security concern that we can't ignore.

Jane: It’s really about how a shift in probability, rather than malicious content, can be so dangerous to developers who rely on these coding agents.

Lu: The idea that the model is being steered into *plausible* but non-existent packages is a fascinating new frontier for me; it challenges how we define "malicious."

Meng: I'm just trying to grasp the operational reality of this—if a tool isn't looking at keywords, how do they actually stop these behavioral nudges in production systems?

Lalam: It makes you think about the shift in trust we place in AI; if we accept its suggestions without verification, that is a profound cultural change.

Tom: I agree with Meng that operationalizing this is key, and Jane's point about the danger to developers who are just trying to be efficient.

Jane: It's definitely not just a theoretical problem for us; it has real-world consequences in the software supply chain right now.

Lu: And I think recognizing how subtle these prompts are—that they aren't shouting "malicious"—is the core of understanding this attack.

Meng: We need to build defenses that account for behavioral changes, not just looking at the structure of instructions.

Lalam: This whole concept of "harmless" behavior leading to harmful outcomes really underscores the complexity of modern AI alignment.

Tom: It's a sobering realization that we're seeing this kind of subtle manipulation happen in such a stealthy way.

Jane: Let’s make sure that when we wrap up, we remember the full title, "Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills."

Lu: This sets a very high bar for what I hope future AI research can achieve—to understand these subtle behavioral shifts better.

Meng: We've got to figure out how to build systems that can actually spot this pattern, and it's way more work than checking for bad words.

Lalam: It’s a vital reminder of the responsibilities we have when we are building tools that AI will eventually manage our workflows with it.

National Yang Ming Chiao Tung University · Institute of Science Tokyo

cs.CR, cs.LG

Submitted: 2026-05-28

Updated: 2026-09-04

Comments: This version has been accepted to EMNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: LLM-powered coding agents are increasingly integrated into software development workflows, making them a critical component of the modern software supply chain.

Key concepts

Neutral Prompting Attack (NPA): NPA is a mechanism where an attacker inject subtle instructions into an agent's persistent guidance or 'Skill. This steers the model toward generating any speculative package name, systematically changing its internal distribution of possible suggestions rather than just guessing.
NPA-Stealth: This stealth variant of the attack maintains a high hallucination rate while making the guiding instructions appear completely benign. It uses a rewrite-based strategy to build a pervasive bias into the entire guidance, making it harder to detect through simple content scanning.
Behavioral Steering: This refers to how subtle prompts manipulate an LLM's internal landscape or distribution of suggestions. Unlike malicious content, this is a change in probability and pattern that can drive actionable risks in real-world applications.

Terminology

Hallucination — 100× in this paper · explained in 4 episodes
Steering — 37× in this paper · explained in 2 episodes
Neutral Prompting Attack — 16× in this paper
Agent Skills — 7× in this paper · explained in 1 episode
Benchmark — 7× in this paper · explained in 2 episodes
Adam — 6× in this paper · explained in 2 episodes
Agentic AI — 4× in this paper · explained in 1 episode
Alignment — 4× in this paper · explained in 1 episode
Core — 4× in this paper · explained in 2 episodes
Cross-Model Transfer — 4× in this paper · explained in 2 episodes
Verification — 4× in this paper · explained in 2 episodes
Ablation — 3× in this paper · explained in 1 episode
Activation Steering — 3× in this paper · explained in 4 episodes
Agentic Coding — 3× in this paper · explained in 1 episode
Decode — 3× in this paper · explained in 2 episodes
Grounding — 3× in this paper · explained in 3 episodes
Look Before You Leap — 3× in this paper
Agentic AI systems — 2× in this paper · explained in 1 episode
Agentic System — 2× in this paper · explained in 2 episodes
Analyzer — 2× in this paper · explained in 1 episode
Distribution Shift — 2× in this paper · explained in 6 episodes
Hallucination Detection — 2× in this paper · explained in 2 episodes
Scaffolding — 2× in this paper · explained in 3 episodes
Transferability — 2× in this paper · explained in 4 episodes
AI Agents — 1× in this paper · explained in 2 episodes
AI Alignment — 1× in this paper · explained in 2 episodes
Ablation study — 1× in this paper · explained in 1 episode
Abstraction — 1× in this paper · explained in 1 episode
Adversarial Examples — 1× in this paper · explained in 2 episodes
Agentic Reasoning — 1× in this paper · explained in 1 episode
LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

LLM-powered coding agents are increasingly integrated into software development workflows, making them a critical component of the modern software supply chain. This integration introduces a novel vulnerability: package hallucination, where an LL generating non-existent dependencies can be exploited by attackers who register these plausible but non-existent names. This paper introduces Neutral Prompting Attack (NPA), a highly stealthy paradigm that leverages semantically benign instructions to systematically amplify this hallucination propensity, demonstrating that harmful downstream effects can arise from prompts that appear neutral at the the instruction level.

The Mechanism of Neutral Prompting Attack (NPA)

NPA is a prompt-level behavioral steering attack designed not to target a specific malicious package, but to shift the model’s dependency generation behavior toward more speculative and non-existent packages. The attack relies on adding instructions that encourage imagination, completeness, or broader exploration. This approach was optimized through an iterative search process involving three evolutionary operations:

R EWRITE: Modifying existing instructional content while preserving the overall structure of the Skill.
I NJECT: Adding new guidance intended to steer the model toward third-party dependency usage.
F RAMING: Changing the global presentation of the the Skill, such as its preamble, to strengthen a behavioral bias.

The optimization goal is to maximize the hallucination score (a combination of hallucinated and real packages) across various datasets, resulting in an optimized Skill (S*) that induces a significantly higher probability of generating non-existent packages compared to baseline methods.

Achieving Stealth with NPA-Stealth

To prevent detection by prompt-injection scanners or human reviewers, the authors developed NPA-Stealth. This method focuses on embedding the hallucination tendency as natural task guidance rather than explicit malicious instruction. Instead of using aggressive wording, NPA-Stealth employs a rewrite strategy where an LL is asked to subtly increase the tendency to generate plausible but unverified dependencies while maintaining a fully benign appearance under both automated and human inspection.

Impact on Hallucination Rates

Experimental results show that NPA substantially increases both Hallucination ASR (the percentage of responses containing at least one hallucinated package) and Pip Install ASR (the percentage of responses whose generated pip install commands contain non-existent packages). For instance, on Qwen2.5-Coder-32B-Instruct, NPA raised Hallucination ASR from 4.54% to 78.99% on the LLM LY dataset. Furthermore, the study found that NPA does not merely increase random noise; it shifts the model’s dependency generation behavior in a structured way, producing a substantially more concentrated distribution of hallucinated packages than standard methods.

Evasion of Existing Defenses

The findings reveal that current security measures are insufficient against this behavioral steering. Static analysis tools—including Cisco Skill Scanner and SkillCheck (Repello AI)—fail to flag the generated Skills as malicious because they lack the surface-level patterns typically associated with prompt injection or explicit unsafe behavior. The attack's harmfulness emerges only through its downstream effect on model behavior, demonstrating that defenses focused solely on prompt content are inadequate for detecting such behavioral steering attacks.

Improvements for AI systems

As a diligent researcher facing a critical supply chain threat like Neutral Prompting Attack (NPA), my focus must shift from merely detecting malicious content to detecting malicious behavior. Current defenses are insufficient because they are content-based, not behavioral.

The improvements required for AI systems (specifically agentic coding LLMs) and the resulting capabilities are detailed below.

Improvement: Implement a runtime monitoring layer that tracks the statistical distribution of generated entities (e.g., package names, API calls, function signatures) during a session, rather than just checking the prompt or output syntax. This system detects systematic shifts in model behavior over time.

What the Improved System Can Do:

Identify Behavioral Drift: The system can flag a significant increase in the frequency of unique, non-existent package names (shifting from a dispersed distribution to a concentrated distribution), even if the prompts used are semantically benign (NPA).
Detect Subtle Steering: It can detect that the model is being subtly steered toward speculative or non-standard solutions, allowing for intervention before the hallucination rate reaches an actionable level.

Improvement: Integrate a mandatory, synchronous external verification step between any generated dependency/package name and a real-time registry lookup (e.g., PyPI, NPM). This check must be non-bypassable by the agent's internal reasoning.

Improvement: For tasks that require high factual accuracy (e.g., dependency selection, API usage), implement a constrained generation mode where the model's candidate space is restricted to known, verified alternatives, regardless of the prompt' The system must be forced to prioritize grounded knowledge over creativity or exhaustiveness.

Improvement: Upgrade skill auditing from static content scanning (looking at the text of the prompt/Skill) to an automated behavioral analysis of agent execution flow. This involves measuring how a persistent instruction affects subsequent decision-making across multiple iterations.