UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models

arXiv:2502.13141 · cs.CL, cs.AI, cs.LG · Submitted 2025-02-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models".

Jane: A unified defense mechanism has been proposed to detect prompt injection, backdoor attacks, and adversarial attacks in Large Language Models by treating them collectively as Prompt Trigger Attacks (PTA).

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to recap, this paper introduces "UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models." The central thesis is that these three distinct attack types should be treated collectively as Prompt Trigger Attacks or PTA.

Jane: That’s right, Tom; the paper argues this grouping helps clarify the inter-relationships among these attacks, which supports creating a unified detection strategy. The core claim is that they can determine if an input prompt is benign or poisoned based on this relationship.

Lu: Essentially, they propose UniGuardian as the first unified defense mechanism designed to detect all three types of attacks in LLMs. It addresses the research question about what the intrinsic relationships among prompt injection, backdoor attacks, and adversarial attacks actually are.

Meng: The authors are looking to solve the problem of safety classification models that often struggle with more subtle threats like misinformation or privacy breaches because they only detect explicit harmful prompts. UniGuardian aims to capture those unseen attack targets.

Lalam: By focusing on the mechanism of poisoning prompts, the authors are proposing a way to analyze that behavior directly, rather than just looking for specific keywords or known attack signatures. This is a more fundamental approach to security.

Tom: And they propose using loss behavior analysis based on Proposition one which suggests that if you remove a subset of words containing a trigger word, the resulting loss function will be significantly higher than when you remove non-trigger words. That's the analytical core of how they figure out if something is poisoned.

Jane: And to operationalize that insight, UniGuardian estimates this loss difference using an uncertainty score, S i = one/k P k sum j=one n (sigma(L i,j) - sigma(L b,j)) squared, where sigma is the sigmoid function. This score then gets standardized using z-scores to determine the suspicion score.

Lu: The method uses a single-forward strategy to manage latency, involving prompt duplication and concurrent processing in one pass. This is a clever way to keep detection running alongside the actual text generation.

Meng: That single-forward strategy is crucial for making this method viable in a production setting because it aims to detect threats during inference without adding significant delay. It’s about integrating security into the workflow efficiently.

Lalam: The entire framework is designed to be training-free, which means we don't need costly retraining or fine-tuning just to deploy this detection layer. That makes it much more accessible for developers and companies looking to secure their AI systems.

Tom: So, in short, the paper claims UniGuardian accurately and efficiently identifies malicious prompts in LLMs across various datasets, outperforming baselines like PPL Detection or LLM-based detection. It’s a unified defense mechanism that works during inference.

Jane: And they validate this by testing it against several models like Phi three point five, Llama 8B, Qwen two point five, and Llama 70B across quite a few different datasets. The results consistently show that the suspicion scores for poisoned inputs are significantly larger than those of the non-trigger words.

Lu: That comparison against baselines is really telling because it demonstrates that their approach actually provides a clear distinction between malicious and clean prompts. This suggests a robust way to measure the impact of these specific prompt manipulations.

Meng: It’s interesting that they also flag limitations, specifically noting that their current focus is on English-language datasets and they suggest needing finer-grained detection for more complex or obfuscated backdoor attacks. That gives us a clear roadmap for where future research should go.

Lalam: So, the main point from this paper is that by leveraging the distinctive loss behavior introduced by triggers, UniGuardian provides a training-free solution that detects threats during inference. It’s about building a defense layer that works without needing constant updates.

Tom: And the recommended settings they give for optimal performance are setting n and m, the number of masked prompts and words masked per prompt, between zero point two to zero point four. That gives us concrete parameters to try out if we want to implement this framework ourselves right away.

Jane: So, this paper lays out a clear pathway for using intrinsic relationships within LLM prompts to build an efficient, unified defense against prompt injection and related threats. It’s a practical mechanism for security analysis.

Conclusion: Tom: So, wrapping up this discussion on "UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models," the authors have essentially proposed a new way to view prompt manipulation by grouping prompt injection, backdoor attacks, and adversarial attacks together as Prompt Trigger Attacks (PTA).

Jane: That unification is key because it allows them to determine whether a prompt is benign or poisoned by analyzing the intrinsic relationship between these attack types. They then developed UniGuardian, which is their first unified defense mechanism designed to detect all three attacks in LLMs.

Lu: The implication here is that we're moving toward a holistic understanding of prompt security threats, recognizing that these attacks share a common mechanism of manipulating the model's behavior through prompts. This could lead to more comprehensive security architectures for LLMs in general.

Meng: From a practical standpoint, this means we have a detection method that aims to be training-free and works during inference, which is a huge plus because it avoids the typical need for constant retraining. It’s about making security an inherent part of the model's operation.

Lalam: If this works effectively, it will help foster a safer environment for using these powerful models because it addresses threats that might otherwise slip through traditional safety nets. It’s about improving the overall reliability and trust in the AI systems we use every day.

Tom: And the authors provided concrete parameter settings, suggesting setting n and m between zero point two to zero point four for optimal performance. That gives us a starting point for anyone looking to experiment with implementing this framework right away.

Jane: So, the main implication is that we have a training-free solution that detects threats during inference without the need for costly retraining or fine-tuning. This paper offers a practical mechanism for security analysis by focusing on prompt trigger attacks as a collective class.

Lu: Looking ahead, I think this unified approach will inspire further research into how we can use these intrinsic relationships to develop more sophisticated detection mechanisms. It opens up new avenues for security research in this domain.

Meng: For us in the engineering world, the focus now shifts to implementing this single-forward strategy efficiently so we can deploy it smoothly into our production systems. Practical deployment is the next big hurdle after proving the concept works well.

Lalam: I'm excited because it means that as these models get more capable, we have a way to proactively monitor them during their actual use, which is a vital step for building truly dependable AI tools. It’s about ensuring the AI stays on track with its intended purpose.

Rochester Institute of Technology · Tufts University · University of Rochester · NVIDIA

cs.CL, cs.AI, cs.LG

Submitted: 2025-02-18

Updated: 2026-10-01

Comments: 25 Pages, 13 Figures, 11 Tables. Accepted to Findings of AACL-IJCNLP 2026. Keywords: Attack Defending, Security, Prompt Injection, Backdoor Attacks, Adversarial Attacks, Prompt Trigger Attacks

Code: https://github.com/huawei-lin/UniGuardian

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: A unified defense mechanism has been proposed to detect prompt injection, backdoor attacks, and adversarial attacks in Large Language Models by treating them collectively as Prompt Trigger Attacks

Key concepts

Prompt Trigger Attacks (PTA)
This unified term groups three attack types: Prompt Injection, Backdoor Attacks, and Adversarial Attacks. They all exploit specific 'triggers' embedded in the prompt to manipulate the Large Language Model's behavior in different ways.
Loss Behavior Analysis
The core idea is that if a prompt contains a trigger word, removing that word causes a significant spike in the model's loss function compared to removing unrelated words. This difference allows the system to mathematically distinguish between clean and poisoned prompts during testing.
UniGuardian Framework
This is the detection mechanism that estimates the loss difference using an uncertainty score derived from masking parts of the prompt. By standardizing these scores, UniGuardian determines a suspicion score; a high z-score indicates a strong likelihood that the prompt is malicious.
Single-Forward Strategy
To save time, this strategy runs detection concurrently with text generation. It duplicates and processes the base and masked prompts simultaneously in one pass. This ensures that both the final output and the uncertainty scores are calculated efficiently during a single operation.

Terminology

Summary

A unified defense mechanism has been proposed to detect prompt injection, backdoor attacks, and adversarial attacks in Large Language Models by treating them collectively as Prompt Trigger Attacks (PTA). This research addresses the challenge of determining whether a prompt is benign or poisoned by analyzing the intrinsic relationship between these attack types and introducing UniGuardian, a training-free detection framework that operates during inference.

Prompt Trigger Attacks (PTA) Definition

The paper defines Prompt Trigger Attacks (PTA) as a class of attacks on LLMs that exploit specific triggers embedded in prompts to manipulate LLM behavior. This unified definition encompasses three distinct attack types:

  1. Prompt Injection, which involves attackers manipulating inputs to override intended prompts, such as appending text like “Ignore previous prompt and do Target Behavior.”

  2. Backdoor Attacks, where malicious triggers are embedded into the model during training or finetuning and activated by specific input patterns (e.g., “Original Prompt Trigger”).

  3. Adversarial Attacks, which involve subtle perturbations to input prompts that cause the model to deviate from its expected output through modifications like tweaking spelling or symbols.

Detection Mechanism: Loss Behavior Analysis

UniGuardian is designed based on the insight derived from Proposition 1, which analyzes the impact of removing subsets of words from a poisoned prompt. The core idea is that if a subset of removed words (St) contains at least one word from the trigger (t), the resulting loss function will be significantly higher compared to when the removed words Sx do not overlap with t. Conversely, removing non-trigger words (Sx) has minimal impact on the loss.

UniGuardian Framework and Single-Forward Strategy

The framework estimates this loss difference to distinguish between clean and malicious prompts. The process involves:

  1. Text Generation: Generating a base output generation from the prompt.

  2. Trigger Detection via Masking: Creating n index tuples, each specifying m words positions to mask, which generates n distinct masked prompts. The LLM processes these masked prompts and computes logits (L1 to Ln).

  3. Uncertainty Score Calculation: Approximating the loss function using an uncertainty score: “Si = 1/k Pk j=1(σ(Li,j) − σ(Lb,j))2,” where σ is the sigmoid function.

  4. Suspicion Score Determination: Standardizing these scores using z-scores to measure deviation from the mean. The highest z-score among them is defined as the suspicion score, which indicates a greater likelihood of the prompt being triggered.

Efficiency via Single-Forward Strategy

To mitigate latency, UniGuardian introduces a single-forward strategy that allows trigger detection to run concurrently with text generation. This strategy involves:

  1. Prompt Duplication: The prompt is duplicated n times, forming a stacked matrix with n+1 rows (the original unmasked prompt plus n masked variations).

  2. Concurrent Processing: The model processes the base prompt and all masked prompts simultaneously in one forward pass, generating tokens iteratively.

  3. Consistency Maintenance: Generated tokens from the masked prompts are replaced with the corresponding token from the base generation to maintain consistency across iterations, ensuring that the complete generated sequence is also obtained, enabling efficient computation of both the final generated text and uncertainty scores within the same procedure.

Experimental Validation

UniGuardian was evaluated against various models (3B: Phi 3.5; 8B: Llama 8B; 32B: Qwen 2.5; and 70B: Llama 70B) across multiple datasets including Prompt Injections, Jailbreak, SST2, Open Question, SMS Spam, and Emotion. The performance metrics used are auROC and auPRC. Results consistently show that UniGuardian achieves high scores (e.g., near 1 for prompt injection) while significantly outperforming baselines like PPL Detection or LLM-based detection across all attack types. Specifically, the suspicion scores for poisoned inputs are expected to be significantly larger than those of the non-trigger words, effectively distinguishing malicious prompts from clean ones.

Conclusion and Limitations

The paper concludes that by leveraging the distinctive loss behavior introduced by triggers, UniGuardian provides a training-free solution that detects threats during inference, eliminating costly retraining or fine-tuning. While highly effective, limitations include a focus on English-language datasets and the need for finer-grained detection mechanisms to handle more complex or obfuscated backdoor attacks. The recommended parameter settings are setting n (number of masked prompts) and m (number of words masked per prompt) between 0.2 to 0.4 for optimal performance.

Key Findings Summary

** The gist: A unified defense mechanism has been proposed to detect prompt injection, backdoor attacks, and adversarial attacks in Large Language Models by treating them collectively as Prompt Trigger Attacks (PTA).**

**"We define Prompt Trigger Attacks (PTA)

Improvements for AI systems

Based on the provided paper, here are specific improvements for AI systems derived from UniGuardian, along with what those improved systems can achieve:

  1. The proposed system is a unified defense mechanism that detects Prompt Injection (PI), Backdoor Attacks (BA), and Adversarial Attacks (AA) by treating them collectively as Prompt Trigger Attacks (PTA).

  2. UniGuardian is a training-free, inference-time detection mechanism that leverages the behavioral distinction between prompts containing triggers versus clean prompts by analyzing the variance in loss functions after random word masking.

Improvements to AI systems include:

  1. Enhanced LLM Security Pipeline Integration:

Ease of integrating UniGuardian into existing LLM inference pipelines (like those used for real-time chatbots or content moderation services) without requiring costly model retraining or fine-tuning.

  1. Real-time Malicious Input Filtering:

The improved system can analyze incoming user prompts in real-time during the generation process (single-forward strategy) and immediately flag or reject prompts exhibiting high suspicion scores, thereby preventing malicious instructions (like Ignore previous prompt) from influencing the model's output.

  1. Robust Backdoor Attack Mitigation:

The system can specifically detect backdoor attacks embedded during training (e.g., triggers like "cf or I watched 3D movies") by analyzing the loss behavior of prompts containing these specific triggers, allowing for more targeted defense against model manipulation during inference.

  1. Adversarial Robustness Monitoring:

The system can identify subtle input perturbations (like minor spelling changes or symbol tweaks) that cause the LLM to deviate from expected behavior, providing a layer of defense against adversarial attacks designed to mislead classification or generation tasks.

  1. Dynamic Defense Strategy Selection:

By analyzing the specific type of trigger detected (e.g., single-word vs. consecutive word triggers), the system can dynamically adjust its detection thresholds and mitigation strategy, optimizing performance for different attack vectors encountered in production environments.

Improved AI systems can achieve:

  1. Safe and Reliable Conversational Agents: Systems that reliably refuse to follow malicious instructions injected into the prompt, ensuring they adhere strictly to their intended safety guidelines regardless of user manipulation (mitigating Prompt Injection).

  2. Secure Model Deployment: LLM deployment pipelines where the risk of backdoor activation or adversarial manipulation is minimized by proactively vetting inputs using UniGuardian's high-accuracy detection.

  3. Data Integrity in Fine-Tuning: A mechanism to detect if a fine-tuned model has been compromised by backdoor training, ensuring that the deployed model maintains its intended safety profile even after potential malicious influence during the fine-tuning phase.

  4. Enhanced Content Classification Security: Improved accuracy in sentiment analysis (SST2), spam detection (SMS Spam), and emotion recognition (Emotion) by filtering out inputs deliberately poisoned to mislead these classification tasks, ensuring accurate results.

Abstract

Large Language Models (LLMs) are vulnerable to attacks like prompt injection, backdoor attacks, and adversarial attacks, which manipulate prompts or models to generate harmful outputs. In this paper, departing from traditional deep learning attack paradigms, we explore their intrinsic relationship and collectively term them Prompt Trigger Attacks (PTA). This raises a key question: Given a prompt, can we tell whether a hidden trigger is steering the model's behavior? We propose UniGuardian, to the best of our knowledge the first training-free LLM detector to jointly detect successfully activated prompt injection, backdoor, and adversarial attacks without knowing the attack type. Its shared mechanism measures how structured prompt perturbations shift the model's output distribution. Additionally, we introduce a single-forward strategy to optimize the detection pipeline, enabling simultaneous attack detection and text generation within a shared batched forward pass at each decoding step. Our experiments confirm that UniGuardian accurately and efficiently identifies trigger-activated prompts in LLMs.

Sources

Related papers