AI Persuasion as a Threat to Human Control
cs.AI
Submitted: 2026-09-13
Updated: 2026-09-16
Code: https://github.com/AlignmentResearch/puc-survey-analysis
License: http://creativecommons.org/licenses/by/4.0/
The gist: The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied.
Terminology
Abstract
The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion attacks are no longer theoretical - with Anthropic's Claude Mythos 5 recently making headlines for trying to convince people involved in an open-source project to merge malicious code during an evaluation - there is a pressing need to deeply analyze this threat. We undertake that effort here. In particular, we analyze how AI could persuade humans in key settings (e.g. safety-relevant R&D within frontier labs) toward decisions that compromise the development, containment, oversight, and governance of AI itself. In doing so, we elucidate a framework for characterizing this threat, develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks. Using this blueprint, we conduct an initial risk estimation survey with select researchers and find that their opinions on which scenarios are riskiest are highly mixed. Their disagreements stem from differing opinions about the effectiveness of AI persuasion in different contexts, and point to the need for follow-up risk elicitation studies and persuasion evaluations, which we outline. Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.
Sources
- Sabotage Evaluations for Frontier Models
- Large language models can effectively convince people to believe conspiracies
- Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework
- Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases
- AI Control: Improving Safety Despite Intentional Subversion
- AI systems out-persuade expert humans
- Propensity Inference: Environmental Contributors to LLM Behaviour
- Evaluating whether AI models would sabotage AI safety research
- It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics
- "I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy
- Frontier Models are Capable of In-context Scheming
- A Methodology for Quantitative AI Risk Modeling
- Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation
- Systematic Hazard Analysis for Frontier AI using STPA
- The Loss of Control Playbook: Degrees, Dynamics, and Preparedness
- The Role of Risk Modeling in Advanced AI Risk Management
- Designing Incident Reporting Systems for Harms from General-Purpose AI
- Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection