Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection

arXiv:2608.03627 · cs.AI · Submitted 2026-08-04 · Read on arXiv

cs.AI

Submitted: 2026-08-04

Updated: 2026-08-30

Comments: Accepted to the 6th Workshop on Bias and Fairness in AI at ECML PKDD 2026. Dataset available at https://github.com/raziehch/GenderedLIARDataset

Code: https://github.com/raziehch/GenderedLIARDataset

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their susceptibility to gender bias in this context remains underexplored.

Terminology

Abstract

Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their susceptibility to gender bias in this context remains underexplored. This study presents the first systematic investigation of gender bias in LLM-based fake news detection using real-world data. We augment the LIAR benchmark with three gender variants of speaker job titles (Neutral, Male, Female) for each statement to test whether veracity judgments vary solely based on gender presentation. Six state-of-the-art LLMs are evaluated across multiple bias and fairness metrics. All models exhibit gender sensitivity: 9.79%-35.13% of statements receive inconsistent labels across the three variants, with Male-Female comparisons showing 6.5%-23.6% flip rates. Two primary bias manifestations are identified: instability (inconsistent judgments) and directionality (systematic favoritism). Five models show statistically significant directional effects, with the strongest effects displaying male-skeptic patterns. These findings demonstrate that gender bias undermines both reliability and fairness in LLM-based fake news detection, highlighting the need for bias-aware evaluation and mitigation strategies. The augmented dataset is publicly released to support future research.

Sources

Related papers