GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners

arXiv:2509.24418 · cs.CR · Submitted 2025-09-29 · Read on arXiv

cs.CR

Submitted: 2025-09-29

Updated: 2026-08-28

Comments: Accepted by EMNLP 2026

Code: https://github.com/meta-llama/llama3

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 32× in this paper · explained in 2 episodes
• Alignment — 21× in this paper · explained in 1 episode
• GRPO — 17× in this paper · explained in 6 episodes
• Accuracy — 11× in this paper · explained in 1 episode
• Generalization — 11× in this paper · explained in 13 episodes
• Trace — 9× in this paper · explained in 2 episodes
• Reasoning Traces — 8× in this paper · explained in 3 episodes
• Explainability — 7× in this paper · explained in 3 episodes
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Core — 5× in this paper · explained in 2 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Robustness — 4× in this paper · explained in 5 episodes
• Adam — 3× in this paper · explained in 2 episodes
• F1 Score — 3× in this paper · explained in 3 episodes
• Group Relative Policy Optimization — 3× in this paper
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• Group Relative Policy Optimization (GRPO) — 2× in this paper · explained in 5 episodes
• Information Leakage — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Ablation Studies — 1× in this paper · explained in 1 episode
• Actor — 1× in this paper · explained in 3 episodes
• Alignment Tax — 1× in this paper · explained in 4 episodes
• Computational overhead — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• GRPO Objective — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hierarchical Classification — 1× in this paper · explained in 2 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Proximal Policy Optimization — 1× in this paper
• Proximal Policy Optimization (PPO) — 1× in this paper · explained in 4 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Synthetic Data Generation — 1× in this paper · explained in 5 episodes
• Verifiable Rewards — 1× in this paper · explained in 2 episodes

Sources

• BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger
• OR-Bench: An Over-Refusal Benchmark for Large Language Models
• Privacy Side Channels in Machine Learning Systems
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
• Mistral 7B
• MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol
• $R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning
• Poly-Guard: Massive Multi-Domain Safety Policy-Grounded Guardrail Dataset
• GuardReasoner: Towards Reasoning-based LLM Safeguards
• Ignore Previous Prompt: Attack Techniques For Language Models
• Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs