Vision-Language Models Suppress Female Representations Under Ambiguous Input

arXiv:2605.31556 · cs.CV, cs.AI, cs.CL, cs.CY, cs.HC · Submitted 2026-05-29 · Read on arXiv

cs.CV, cs.AI, cs.CL, cs.CY, cs.HC

Submitted: 2026-05-29

Updated: 2026-08-31

Comments: Accepted at EMNLP 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: Alignment teaches vision-language models (VLMs) to avoid expressing demographic biases, and when gender is clearly visible they largely succeed.

Terminology

Abstract

Alignment teaches vision-language models (VLMs) to avoid expressing demographic biases, and when gender is clearly visible they largely succeed. Far less is known about ambiguous inputs (a worker in full gear, a figure seen from behind), cases common in practice yet rarely studied. We find that minimal prompting pressure exposes occupation-gender defaults when prompting ambiguous input images, with models collapsing to male even for strongly female-stereotyped occupations. But do these outputs reflect what models actually encode internally? We introduce LALS (Latent Association Leaning Score), a zero-shot metric that projects visual-token activations into the model's text-embedding space to measure concept associations per token and layer. Across 15 occupations, over 800 gender-ambiguous images, and four VLMs, internal representations and outputs often become systematically decoupled: models often encode a female association internally yet output male. Layer-wise analysis reveals an asymmetric filter: male signal amplifies end-to-end while female signal peaks mid-network and is suppressed before generation. A color ablation shows that culturally loaded visual cues such as clothing color further modulate these internal associations.

Sources

Related papers