Social Pressure Breaks Majority Voting in LLM Safety Panels
Yibo Hu, Jiaming Qu
cs.CL
Submitted: 2026-08-05
Code: https://github.com/yibo-hu-lab/llm-safety-panel-conformity
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are increasingly used to detect unsafe content.
Terminology
Abstract
Large language models (LLMs) are increasingly used to detect unsafe content. A common approach is to combine judgments from a panel of models to correct individual mistakes, but this benefit may disappear when every model sees the same misleading context before voting. We study this risk in a controlled two-round experiment. Each model first judges an item alone, then judges it again after six simulated peers either assert the wrong label or abstain. We combine the final judgments by majority vote. Across six open-weight LLMs and six datasets, we find that the wrong-label peer message raises the average reviewer false-alarm rate from 56.5% under silent peers to 87.5%, and majority voting raises the panel false-alarm rate to 100%. Without an asserted label, the same panel outperforms its average member. The effect is strongly asymmetric: reviewers follow pushes toward "unsafe" far more than pushes toward "safe" (about 75% versus 17%), so the panel's false-alarm rate rises sharply while its harmful-miss rate changes little. The proprietary-model probe shows substantial variation across models. These results identify susceptibility to shared social cues as a failure mode of safety panels and provide a simple pre-deployment diagnostic.
Sources
- Large Language Models Exhibit Normative Conformity
- ELEPHANT: Measuring and understanding social sycophancy in LLMs
- Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems
- Belief in Authority: Impact of Authority in Multi-Agent Evaluation Framework
- Conformity Generates Collective Misalignment in AI Agents Societies
- Gemma 2: Improving Open Language Models at a Practical Size
- AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
- Conformity Dynamics in LLM Multi-Agent Systems: The Roles of Topology and Self-Social Weighting
- When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMs
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Qwen2.5 Technical Report
- Mistral 7B
- 2 OLMo 2 Furious
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- The Llama 3 Herd of Models
- Do as We Do, Not as You Think: the Conformity of Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering