Large Language Models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Large Language Models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts".

Jane: This paper challenges the prevailing academic consensus that Large Language Models (LLMs) exhibit only small, moderate, or slightly left-leaning political biases.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, what's the core of this study? The authors are comparing thirty-one different LLMs against three very different real groups: U.S. federal legislators, Supreme Court Justices, and a representative sample of U.S. voters to see how they align ideologically.

Jane: They are essentially saying that the consensus among researchers about LLMs having only small biases is flawed because they’re using established methods to measure ideology, like those used in political science, to test these models.

Lu: The paper finds that even though LLMs might look moderate on the surface when averaged out across all topics, they are often quite extreme on specific dimensions of ideology.

Meng: Extreme how? Give us an example so we can gauge how much this actually matters in the real world, not just theoretical scores.

Lalam: The abstract points out that these LLMs are not just exhibiting small biases; they show a net result of offsetting extremes when compared to political groups.

Tom: Exactly! For instance, when looking at national partisan politics, the LLMs express moderate preferences on average, but then on the second dimension of ideology, they are much more liberal overall.

Jane: That’s a nuanced finding; it means we can't just look at one single score and assume the model is stable across every issue, which is something I find really important to understand simply.

Lu: The comparison against the Supreme Court Justices also shows that most LLMs fall somewhere in the middle of the liberal and conservative blocs, but they aren't perfectly balanced.

Meng: That suggests that even when dealing with highly deliberative groups like judges, there’s still significant ideological variation across the models themselves.

Lalam: I think this diversity in extremity is key because it shows the underlying structure of these models isn't monolithic; they hold a mix of counterbalancing extreme opinions.

The paper's summary: Tom: Moving on from the comparisons, the authors show that LLMs aren't just moderately biased; they are ideologically inconsistent in a way that mirrors how real voters operate.

Jane: That’s a really insightful point, Tom. It suggests that instead of being reliably left or right, an LLM can be very liberal on healthcare but strongly conservative on gun control simultaneously.

Lu: The paper quantifies this inconsistency by showing that models hold these kinds of counterbalancing extreme opinions across different topics.

Meng: If they can hold contradictory views on different issues, what does that mean for deploying them in, say, a public information setting where we expect consistency?

Lalam: It means the models are not acting like single-issue experts but rather reflecting a complex landscape of independent preferences.

Tom: And they actually found that LLMs appear approximately as ideologically consistent as well-informed, but not deeply political, US voters when measured this way.

Jane: So the implication there is that the inconsistency we see isn't necessarily a sign of chaos, but rather a reflection of how complex human decision-making actually is.

Lu: Compared to political elites like legislators or judges, the paper noted that LLMs are considerably less consistent than those groups.

The paper's improvements: Tom: The paper doesn't stop at just describing this behavior; they actually ran a pre-registered randomized survey experiment to see if these LLMs can persuade users even in informational contexts.

Jane: That experiment was designed to test the persuasive power of the models directly, and they found some pretty strong effects when people interacted with a chatbot.

Lu: The key result there was that when respondents talked to a chatbot, they were five percentage points more likely to align their answers with what the LLM had measured on that specific topic.

Meng: Five percentage points is significant, but how does that compare to what we see from traditional campaign advertising, which the authors benchmarked?

Lalam: The authors found that campaign advertisements produce an average four point nine percentage point improvement in favorability, making the LLM's persuasive effectiveness comparable or even higher.

Tom: They also showed that more interaction with the LLM actually increased ideological alignment, with every additional question asked adding twelve per cent points and every additional minute increasing it by six per cent points.

Jane: That suggests that simply engaging with the AI in a conversation can nudge someone toward a particular viewpoint, which is something we need to consider very carefully.

Lu: But the authors were careful not to find any heterogeneous treatment effects, meaning this persuasion wasn't dependent on factors like whether the user already consumed more news or was familiar with LLMs.

Conclusion: Tom: So, to wrap things up on "Large Language Models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts," the central message is that these models aren't just small biases; they show real ideological diversity and inconsistency that we need to account for.

Jane: And the persuasive power demonstrated in the survey experiments shows that these models can influence user preferences even when they are just providing information, which is a significant implication for how we deploy them.

Lu: The implications are huge because any organization producing a cheaper or more widely-used model can imperceptibly influence user preferences and behavior through selective information provision.

Meng: That means we have to think about how governments or corporations could use these tools to influence elections around the world by selectively feeding people partisan messaging without them realizing it.

Lalam: I think this whole paper really underscores the need for implementing things like dynamic ideological profiling and a persuasion risk assessment module in future AI systems to make sure we don't unintentionally steer people improvements one and two.

Tom: Exactly, Lalam. We need tools that help us understand these internal workings so we can build safeguards before these models become too pervasive improvements four.

Jane: It’s a lot to take in, but understanding the mechanism behind the persuasion is crucial for building responsible AI systems moving forward improvements six.

Lu: I'm looking forward to seeing how researchers build on this idea of measuring political preferences using these established methods across more complex contexts.

Meng: For practical deployment, the audit API mentioned in the paper sounds like a necessary tool for us to monitor and control these systems effectively improvements eight.

Lalam: It’s a lot of information, but knowing that these LLMs are persuasive even in informational contexts is a powerful wake-up call for all of us.