DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
cs.LG, cs.AI
Submitted: 2026-06-04
Updated: 2026-09-02
Comments: Accepted by 2026 EMNLP (Conference on Empirical Methods in Natural Language Processing)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Safety alignment for large language models relies on preference data, but current pipelines often train on large, redundant datasets.
Terminology
Abstract
Safety alignment for large language models relies on preference data, but current pipelines often train on large, redundant datasets. Existing data selection methods typically score each preference pair independently, collapsing directional preference information into scalar quality or diversity scores. This sample-centric view is especially limiting in multi-dataset settings, where shared safety directions coexist with dataset-specific residual risks. We propose DOG-DPO, a training-free data selection framework that treats preference pairs as structured geometric signals. DOG-DPO first represents each preference pair as a direction in model representation space. It then decomposes multi-dataset preference geometry into a global anchor subspace and dataset-specific residual subspaces. Finally, it selects subsets by maximizing diversity-based coverage, encouraging broad, non-redundant coverage of alignment directions before DPO training. Across six safety benchmarks and two model backbones, DOG-DPO achieves a strong utility-robustness trade-off using only 11% of the preference pairs. It recovers most of the safety gains of full-data training while remaining entirely teacher-free, training-free, and substantially faster than representative selection baselines.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
- SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Less is More: High-value Data Selection for Visual Instruction Tuning
- Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
- Reinforcement Learning for LLM Post-Training: A Survey
- CoAct: Co-Active LLM Preference Learning with Human-AI Synergy
- Cat-DPO: Category-Adaptive Safety Alignment
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks