Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
cs.LG
Submitted: 2025-12-03
Updated: 2026-09-01
Comments: Accepted to the findings of EMNLP 2026
Code: https://github.com/FujitsuResearch/LLM-policy-violation-detection
License: http://creativecommons.org/licenses/by/4.0/
The gist: As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority.
Terminology
Abstract
As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we derive policy-violation scores directly from normalized representations of LLM hidden activations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks. It achieves up to 86.0% F1, outperforming both fine-tuned and LLM-as-a-judge baselines while requiring substantially less computation. Our code is publicly available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection
Sources
- Whitened CLIP as a Likelihood Surrogate of Images and Captions
- Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Features
- Using LLMs for Automated Privacy Policy Analysis: Prompt Engineering, Fine-Tuning and Explainability
- JavelinGuard: Low-Cost Transformer Architectures for LLM Security
- Inside-Out: Hidden Factual Knowledge in LLMs
- AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
- A Survey on LLM-as-a-Judge
- SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Pretrained Transformers Improve Out-of-Distribution Robustness
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- The Widespread Adoption of Large Language Model-Assisted Writing Across Society
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- On Automating Security Policies with Contemporary LLMs
- LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey
- LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliance
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks