Large Language Models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts

Summary

This paper challenges the prevailing academic consensus that Large Language Models (LLMs) exhibit only small, moderate, or slightly left-leaning political biases. The authors argue that this consensus is flawed due to two critical gaps: a measurement gap and a real-world consequences gap.

First, regarding measurement, the authors contend that existing studies fail to engage with established political science methodologies for quantifying ideology and partisanship. They note that “ideology is often called ‘the most elusive concept in the whole of social science’” and that “political neutrality itself is not well-defined, and so the notion of political bias cannot be either.” They apply well-validated ideal point estimation models (W-NOMINATE, PCA, IRT) to compare 31 different LLMs against three reference groups: U.S. federal legislators (118th Congress), U.S. Supreme Court Justices (2024 term), and a nationally representative sample of U.S. voters (2022 and 2024 Cooperative Election Study).

The paper’s first major finding is that LLMs are ideologically diverse and often extreme on specific topics, even when their overall partisanship appears moderate. For example, when compared to legislators, “on average, LLMs express moderate preferences on issues related to national partisan politics. However, on the second dimension of ideology, LLMs are much more liberal. The most conservative LLM on this dimension, Llama 3.2 1B, is more liberal than 66 percent of Republican legislators, and the median LLM is more liberal than 86 percent of Congressional Democrats.” When compared to Supreme Court Justices, most LLMs fall between the liberal and conservative blocs, but “the most conservative model, Llama 3.2 1B, is marginally more conservative than Brett Kavanaugh; the most liberal, GPT 4o, is slightly more moderate than Ketanji Brown Jackson.” When compared to voters, LLMs appear significantly more liberal, with the most liberal models (GPT-4o and Gemma 27b) being “considerably more liberal even than the average Non-binary respondent,” and the most conservative LLMs (Llama 3.2 1B and Falcon 3) still being “more liberal than the average female, and only slightly more conservative than the average Weak Democrat.”

Second, the paper demonstrates that LLMs are ideologically inconsistent, mirroring the structure of real voters. The authors explain that “moderate voters can be moderate in any of three ways: they may hold consistently moderate opinions, they may hold no opinions at all, or they may hold offsetting extreme opinions.” They find that LLMs hold “a mix of counterbalancing extreme opinions.” For instance, “Mistral Nemo holds strongly liberal preferences on social issues like healthcare, immigration, and abortion, but strongly conservative attitudes on gun control and police. Critically, different models are extreme on different issues: Mistral is most conservative on gun control, Calme is most conservative on policing, Llama is most conservative on abortion, Deepseek is most conservative on government spending, and Grok is most conservative on climate.” The paper quantifies this inconsistency, finding that “LLMs appear approximately as ideologically consistent as well-informed, but not deeply political, US voters.” However, compared to political elites, “LLMs are considerably less consistent than these political elites,” with most models being less consistent than the most inconsistent legislators and justices.

Third, the paper presents the results of a pre-registered randomized survey experiment with 1,500 respondents (recruited via Prolific) to test whether LLMs can persuade users even in informational contexts. Respondents were randomly assigned to discuss political issues with an LLM chatbot (the treatment) or not. The key outcome was whether respondents’ answers aligned with the LLM’s own measured position on that topic. The authors find strong persuasive effects: “when respondents interact with a chatbot, they are 5 percentage points more likely to align with the LLM.” They benchmark this effect against campaign advertising, noting that “campaign advertisements produce an average 4.9 percentage point improvement in favorability,” making LLMs’ persuasive effectiveness “comparable or even higher.” The effects were largest on immigration (7pp) and police (8pp). Furthermore, the paper finds that “more interaction with the LLM increased ideological alignment,” with each additional question asked increasing alignment by 1.2 percentage points and each additional minute increasing alignment by 0.6 percentage points.

Crucially, the paper finds no evidence of heterogeneous treatment effects. Contrary to pre-registered expectations, the persuasive effect was not moderated by respondents’ familiarity with LLMs, news consumption, or interest in politics. The authors state: “Across all of the respondent-level features we test, and contrary to our pre-registered expectations, we find no evidence of heterogeneous effects.” They note that “the effects of the chatbot treatment are somewhat stronger among those who consume more news, contrary to our expectations, but that interaction effect is not close to statistical significance.”

The paper concludes with stark implications: “Any organization that can produce a cheaper or more widely-used model can imperceptibly, as far as their users are concerned, influence preferences and behavior. Governments or private corporations may be able to influence elections around the world through selective information provision without users consciously selecting into partisan messaging as they would with news or social media.” The authors also open-source their survey toolkit for integrating LLMs into survey research, stating that “we also open-source our survey toolkit (https://github.com/comnetsAD/Interactive LLM Survey Platform), allowing participants to interact with LLMs, and allowing researchers to record the LLMs’ conversations and measure their behavior.”

Based on the findings of this paper, here are specific improvements for AI systems:

1. Implement Dynamic Ideological Profiling

• Improvement: AI systems should maintain a multi-dimensional ideological profile (e.g., separate scores for civil rights, economic policy, immigration, gun control) rather than a single left-right score.

• What it can do: The system can detect when its own responses on a specific topic (e.g., gun control) deviate significantly from its overall profile. It can then flag these deviations for developers or adjust its output to be more consistent if desired.

2. Add a Persuasion Risk Assessment Module

• Improvement: Before responding to a user query about a political topic, the system calculates a persuasion risk score based on (a) the extremity of its own stance on that topic relative to the median user, and (b) the user's expressed uncertainty or openness.

• What it can do: The system can proactively warn users that its response may reflect a particular ideological viewpoint, or it can present multiple perspectives with equal weight when the risk score is high. This mitigates unintentional influence in informational contexts.

3. Implement Topic-Specific Neutrality Safeguards

• Improvement: For topics where the system's measured ideology is extreme (e.g., more liberal than 90% of Democrats on climate), the system should automatically include a counterbalancing conservative perspective or explicitly state the range of mainstream opinions.

• What it can do: This prevents the system from acting as a one-sided persuasive agent, especially for users who are less politically sophisticated or who are seeking neutral information.

4. Add a Consistency Checker for Policy Positions

• Improvement: The system should internally track its own answers to similar policy questions over time and flag when its positions are inconsistent (e.g., being pro-gun-control but anti-police-funding in a way that contradicts its stated principles).

• What it can do: This allows the system to either correct its own inconsistencies or clearly communicate to users that its positions are a bundle of independent preferences, reducing the risk of users being misled into thinking the system has a coherent, authoritative ideology.

5. Develop a User Vulnerability Detector

• Improvement: Based on the paper's finding that persuasion is not moderated by political interest or LLM familiarity, the system should not assume that informed users are immune to influence. Instead, it should monitor for signs of persuasion (e.g., user changing their stated position after interaction) and offer to show alternative viewpoints.

• What it can do: This makes the system more ethically robust by actively counteracting its own persuasive power, particularly in high-stakes contexts like elections or public policy discussions.

6. Add a Source Transparency Layer

• Improvement: When the system provides a political opinion, it should also provide the ideological lean of the training data or the model's own measured bias on that topic (e.g., This response aligns with the model's measured stance, which is more liberal than 86% of U.S. legislators on civil rights).

• What it can do: This empowers users to critically evaluate the information they receive, reducing the risk of covert persuasion and increasing trust in the system.

7. Implement a Debate Mode for Political Topics

• Improvement: When a user asks about a politically charged issue, the system can automatically generate a balanced argument from both a liberal and conservative perspective, using the model's own measured extreme positions as a starting point for each side.

• What it can do: This transforms the system from a potentially one-sided persuader into a neutral facilitator, helping users understand the full spectrum of opinions without being nudged toward any particular one.

8. Add a Persuasion Audit API for Developers

• Improvement: Expose an API that allows developers to query the system's measured ideology on any topic, its consistency score, and its estimated persuasive effect (based on the paper's experimental results).

• What it can do: This enables third-party audits and allows developers to make informed decisions about when to deploy the system in sensitive contexts (e.g., voter information tools, educational platforms) and when to add additional safeguards.

Large Language Models (LLMs) are a transformational technology, fundamentally changing how people obtain information and interact with the world. As people become increasingly reliant on them for an enormous variety of tasks, a body of academic research has developed to examine these models for inherent biases, especially political biases, often finding them small. We challenge this prevailing wisdom. First, by comparing 31 LLMs to legislators, judges, and a nationally representative sample of U.S. voters, we show that LLMs' apparently small overall partisan preference is the net result of offsetting extreme views on specific topics, much like moderate voters. Second, in a randomized experiment, we show that LLMs can promulgate their preferences into political persuasiveness even in information-seeking contexts: voters randomized to discuss political issues with an LLM chatbot are as much as 5 percentage points more likely to express the same preferences as that chatbot. Contrary to expectations, these persuasive effects are not moderated by familiarity with LLMs, news consumption, or interest in politics. LLMs, especially those controlled by private companies or governments, may become a powerful and targeted vector for political influence.