GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models
cs.CY, cs.CL, cs.CV
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: Accepted to EMNLP 2026
Code: https://github.com/dukesun99/GGSS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Generative vision-language models (VLMs) are increasingly used in human-centered settings, yet they can produce demographically biased outputs even when images differ only in controlled attributes
Terminology
Abstract
Generative vision-language models (VLMs) are increasingly used in human-centered settings, yet they can produce demographically biased outputs even when images differ only in controlled attributes such as perceived race or gender. However, existing inference-time debiasers were largely designed for static embeddings or CLIP-like models rather than generative VLMs. We propose GGSS---Geodesic-Gated Spherical Steering---a norm-preserving intervention that discovers a counterfactual bias subspace on the unit hypersphere, steers visual tokens along geodesic arcs, and uses an adaptive gate to focus correction on tokens that carry stronger demographic signal. We evaluate four generative VLMs against ten adapted inference-time debiasing baselines and prompt-based mitigation under a single operating-point protocol across categorical, pairwise, and occupation-gender bias tests, while also measuring general visual-language capability. GGSS achieves the lowest average bias on all four models, significant on three of four backbones under paired permutation tests, while preserving MMStar accuracy within +/- 0.6 p.p. of the unsteered baseline. Code is available at https://github.com/dukesun99/GGSS.
Sources
- Pixtral 12B
- Interpretable Debiasing of Vision-Language Models for Social Fairness
- Qwen3-VL Technical Report
- Learning To Guide Human Decision Makers With Vision-Language Models
- Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Debiasing Vision-Language Models via Biased Prompts
- BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations
- Do Prompts Guarantee Safety? Mitigating Toxicity from LLM Generations through Subspace Intervention
- One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces
- Steering Language Models With Activation Engineering
- Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMs
- VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus
- Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot
- PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks
- What is an intelligent system?
- AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study
- Generative AI Use in Entrepreneurship: An Integrative Review and an Empowerment-Entrapment Framework