Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation

arXiv:2608.05155 · cs.CL, cs.AI · Submitted 2026-05-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation".

Tom: The paper investigates whether traditional sentiment analysis models adequately capture the rhetorical and ideological dimensions of political discourse and proposes an LLM-based multi-dimensional framework as a more epistemologically adequate tool…

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this new paper called "Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation." Essentially, they are asking if the standard sentiment analysis models we use today actually capture what makes political talking work in the social sciences.

Jane: That’s right, Tom. The core idea is that traditional sentiment analysis models, like RoBERTa, are good at just picking a positive or negative label for a text, but they totally miss the bigger picture of how politics actually gets communicated through framing and ideology.

Lu: What they claim is that these older models have this major limitation called neutral collapse, where they flatten rich political content into just 'neutral' because the writing style—like hedging or balanced sourcing—looks like no strong feeling is present to the machine <ref:2608.05155#pg4>.

Meng: So, if you only look at those basic scores, you might miss all the nuance in a complex news story?

Tom: Exactly. And they show that this means seventy percent of their articles get classified as neutral by RoBERTa, which basically makes the whole political content analytically useless for deep research <ref:2608.05155#pg0>.

Jane: But they’re not just stopping there; they’re comparing that old method against a newer LLM-based platform that breaks things down into four different dimensions: political bias, sensationalism, emotional appeal, and political framing <ref:2608.05155#pg4>.

Lalam: That LLM approach lets us see how much bias there is—like left or right—and how exaggerated the language is through sensationalism and emotional appeals <ref:2608.05155#pg4>.

Lu: It’s interesting because this new method analyzes the full text, not just a short snippet, which they say is important because political framing often relies on the whole article structure <ref:2608.05155#pg4>.

Tom: So what does this mean for us? It suggests that we can't just rely on a single polarity score anymore; we need to look at how things are framed and who is being targeted, which is what the LLM captures <ref:2608.05155#pg4>.

Jane: And they show that the LLM platform gives us scores across four continuous dimensions plus one categorical bias direction, which lines up really well with established social science frameworks <ref:2608.05155#pg4>.

Meng: From an engineering side, it’s cool that this new platform captures things like political framing through things like fear appeals and scapegoating, not just word choice <ref:2608.05155#pg4>.

Tom: And they found that the LLM approach captures rhetorical strategies that run independently of simple sentiment polarity because sensationalism comes from narrative structure rather than just looking for specific emotional words <ref:2608.05155#pg4>.

Lu: It really pushes the idea that we need AI tools configured for social science reasoning instead of just repurposing them for standard NLP benchmarks, which Bail argued is important <ref:2608.05155#pg2>.

Jane: They also point out a specific issue with the old RoBERTa method: eight out of the thirty-five neutral-classified articles have negative probability scores above zero point three zero, meaning the model sees negative content but gets overruled by that neutral label <ref:2608.05155#pg4>.

Paper summary: Tom: That’s a big warning sign, Jane; it shows how easily substantive political content gets mislabeled when you use those simpler models <ref:2608.05155#pg4>.

Meng: So the practical implication is that if you rely only on sentiment analysis for political research, you're likely missing the majority of the actual political signal <ref:2608.05155#pg4>.

Lalam: And Lalam thinks this means that using an LLM approach gives researchers a much deeper view of what’s happening in the discourse, because it preserves that full argumentative structure from processing every single word <ref:2608.05155#pg4>.

Tom: So, to summarize this paper, "Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation," the authors present a direct comparison between RoBERTa and an LLM platform on fifty political articles from seventeen outlets <ref:2608.05155#pg1>.

Jane: The main point is that traditional sentiment analysis struggles to capture the rhetorical and ideological dimensions central to social science, so they introduce an LLM-based framework that measures bias, sensationalism, emotional appeal, and framing <ref:2608.05155#pg0>.

Lu: This study identifies neutral collapse as a key limitation of SA models, showing how they systematically misclassify substantial political writing as neutral because journalistic language uses features like hedging and balanced sourcing <ref:2608.05155#pg4>.

Tom: And the paper argues that the LLM approach provides a more structured comparison, giving researchers something complementary instead of just replacing one tool with another <ref:2608.05155#pg1>.

Jane: Ultimately, the authors suggest that political news is characterized more by framing and rhetorical strategy than by simple emotional polarity, which is exactly what the multi-dimensional analysis captures <ref:2608.05155#pg4>.

Meng: So for people interested in applying this, it suggests using sentiment analysis as a first pass screen, but then using that LLM platform to get the deeper analytical layer aligned with social science needs <ref:2608.05155#pg7>.

Tom: That’s what we’re getting into, guys. It moves the focus from just 'is this positive or negative' to understanding the actual political strategy behind the words <ref:2608.05155#pg4>.

Jane: And while they acknowledge challenges like the opacity of LLM reasoning and higher computational costs, they push for evaluation frameworks that measure tool utility by research relevance instead of just raw NLP benchmark performance <ref:2608.05155#pg7>.

Lu: It’s a call to build tools that are specifically trained for socialscientific reasoning, not just general text analysis benchmarks, which is what Bail suggested earlier in the literature <ref:2608.05155#pg2>.

Tom: And for those of us listening, this paper suggests that political coverage isn't defined by lexical emotion at all; it's about framing and ideological orientation, which is exactly the kind of nuanced data social science researchers are looking for <ref:2608.05155#pg4>.

Jane: So to wrap up this look at "Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation," it’s about showing that LLMs can bridge the gap between simple sentiment and the deeper, structural analysis required in political discourse <ref:2608.05155#pg1>.

Conclusion: Tom: So we’re wrapping up our look at "Beyond Sentiment: Comparing Traditional NLP and LLM-Based Multi-Dimensional Analysis for Political News Evaluation." It’s really about showing that just looking for positive or negative words isn't enough to understand how politics actually works in the media.

Jane: Right. The paper takes this old method, sentiment analysis, and compares it directly to a new approach using a large language model that looks at framing and bias instead of just word choice.

Lu: They point out that traditional models have this neutral collapse where they just dump rich political content into a neutral bucket because the writing style uses hedging and balanced sourcing.

Meng: From an engineering standpoint, I’m interested in how the LLM handles the full text without cutting it off at five hundred tokens. That preservation of structure seems important for capturing framing devices.

Lalam: I think what they show is that political news isn't actually about feeling good or bad; it’s really about who is being targeted and how a story is structured, which the LLM gets better at seeing through those four dimensions.

Tom: So, the big idea here seems to be that for social science research into politics, researchers should probably stop relying solely on those simple polarity scores.

Jane: Exactly. The implication for you listening is that if you’re analyzing political coverage, just getting a positive or negative number isn't going to give you the full picture of the argument being made.

Lu: They argue that this multi-dimensional analysis provides a more epistemologically adequate tool for social science research because it captures rhetorical strategies that operate independently of simple sentiment polarity.

Meng: I’m curious about the practical use case; if we move toward this, what does that look like in a real system? Does it just mean more complex scoring outputs?

Lalam: It means we can start looking at things like political framing and emotional appeals as distinct analytical categories rather than just noise in a sentiment score.

Tom: And the authors themselves suggest that the best way forward is to use sentiment analysis as a quick first pass screen, and then use this LLM approach for the deeper, more aligned analytical layer.

Jane: That’s the complementary framework they propose; using both tools at different stages of research to get a richer understanding of political discourse.

Lu: They also acknowledge that we have to be careful because LLMs can sometimes be opaque in their reasoning, and there are still computational costs involved when you process full articles.

Meng: That’s fair; opacity is always a concern with these kinds of models, and running full text analysis definitely takes more power than a quick sentiment check.

Tom: So the authors aren't pretending this LLM method is perfect or easy for everyone to use immediately; they’re laying out the actual trade-offs we have to consider.

Jane: And ultimately, they are calling on researchers and tool developers to start measuring these tools by how relevant they are to social science questions, not just by how well they score against some general NLP benchmark.

Tom: It shifts the focus away from chasing the highest numbers on a simple metric toward actually getting the deep, structural understanding of political communication that matters for research.

Maryam Fooladi, Federico Bottino

Kakashi Ventures Accelerator (KVA) / Newjee

cs.CL, cs.AI

Submitted: 2026-05-22

Updated: 2026-05-22

Comments: Accepted at PoliticalNLP 2026, the 3rd Workshop on Natural Language Processing for Political Sciences, co-located with LREC 2026. 10 pages, 3 figures

Journal ref: 2026.politicalnlp-1.17

DOI: 10.63317/2wbwmwq3jwfg

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 79/100

The gist: The paper investigates whether traditional sentiment analysis models adequately capture the rhetorical and ideological dimensions of political discourse and proposes an LLM-based multi-dimensional

Key concepts

Neutral Collapse
This is the critical limitation where sentiment analysis models systematically classify politically complex articles as 'neutral.' This happens because journalistic language often uses balanced phrasing and formal registers that standard models interpret as lacking strong positive or negative emotional valence, effectively flattening substantive content.
RoBERTa-based Sentiment Analysis
This is a traditional NLP method that classifies text into three simple categories: positive, neutral, or negative. It works by truncating the full article to 512 tokens and calculating probability scores for each class. The study found this method is insufficient for political discourse.
LLM-Based Multi-Dimensional Analysis
This advanced platform processes the entire text without truncation, generating four continuous scores (Bias, Sensationalism, Emotional Appeal, Framing) plus a bias direction. It captures rhetorical strategies like framing and emotional appeals that traditional polarity models miss because they operate differently than simple word sentiment.
Political Framing
This refers to how political content is structured using specific narrative devices such as fear appeals, scapegoating, or 'us-vs-them' dichotomies. The LLM approach specifically measures the intensity of these devices, which are crucial for understanding political discourse beyond simple word choice.

Terminology

Summary

The paper investigates whether traditional sentiment analysis models adequately capture the rhetorical and ideological dimensions of political discourse and proposes an LLM-based multi-dimensional framework as a more epistemologically adequate tool for social science research. This study addresses the limitations of polarity classification by introducing a comparison between RoBERTa-based sentiment analysis and an LLM platform designed to capture framing, bias, and sensationalism.

The gist: The results reveal a critical limitation we term neutral collapse: RoBERTa classifies 70% of articles as neutral, effectively flattening substantively rich political content into an analytically uninformative category <ref:2608.05155#pg5>.

How it works

The study compares two distinct analytical approaches applied to a corpus of 50 political news articles from 17 international media outlets. The first approach utilizes RoBERTa-based sentiment analysis, which classifies text into a three-class polarity judgment: positive, neutral, or negative <ref:2608.05155#pg4>. This model truncates the full text to 512 tokens and produces probability scores for each class <ref:2608.05155#pg4>.

The second approach employs an LLM-based multi-dimensional analysis platform that processes full article texts without truncation <ref:2608.05155#pg5>. This platform generates multiple analytical outputs corresponding to established Social Sciences and Humanities (SSH) categories, including:

  1. Political Bias Assessment, which assigns a directional label (left, neutral, right) accompanied by a continuous intensity score (0–100) <ref:2608.05155#pg4>.

  2. Sensationalism, measured by a continuous score (0–100) quantifying the degree of emotional exaggeration, clickbait patterns, hyperbolic language, and excessive rhetorical appeals <ref:2608.05155#pg5>.

  3. Emotional Appeal, quantified as a continuous score (0–100) measuring the extent to which the article employs emotional rather than rational argumentation strategies <ref:2608.05155#pg5>.

  4. Political Framing, a composite score (0–100) indicating the overall intensity of political framing devices, including fear appeal, scapegoating, us-vs-them dichotomies, and victim/hero narratives <ref:2608.05155#pg5>.

Key Findings on Traditional Sentiment Analysis

The application of RoBERTa sentiment analysis to the corpus revealed a phenomenon termed neutral collapse, which is defined as the systematic tendency of SA models to classify substantively rich political content as 'neutral' <ref:2608.05155#pg5>. This collapse occurs because journalistic writing uses features like hedging, attribution, balanced sourcing, and formal register—linguistic features that sentiment models interpret as absence of valence <ref:2608.05155#pg5>.

Quantitatively, the neutral collapse is characterized by two findings:

  1. 8 of the 35 neutral-classified articles (23%) have negative probability scores above 0.30, indicating that the model detects substantial negative content but is narrowly overruled by the neutral probability <ref:2608.05155#pg5>.

  2. Low classification confidence is evident, with instances where classification confidence is often low: 3 articles (6%) have a margin of less than 0.10 between the top two classes <ref:2608.05155#pg5>.

Furthermore, cross-outlet patterns showed that RoBERTa reveals limited but notable patterns in the negative dimension, with European outlets scoring highest in P(neg) but a dominance of neutral (grey) is uniform across all outlets <ref:2608.05155#pg5>. This suggests that RoBERTa provides essentially one useful dimension of variation—negative intensity—while the neutral and positive categories offer minimal analytical discriminability for political text <ref:2608.05155#pg5>.

Advantages of the LLM-Based Approach

The LLM-based platform offers outputs that are analytically differentiated across four continuous dimensions plus a categorical bias direction, which directly corresponds to established SSH analytical frameworks <ref:2608.05155#pg5>. This approach captures rhetorical strategies that operate orthogonally to sentiment polarity because sensationalism is achieved through narrative structure and topic selection rather than through lexical polarity cues <ref:2608.05155#pg5>.

The LLM platform preserves the full argumentative structure by processing full article texts (without truncation), which is crucial because Political framing often operates through cumulative article structure, which is preserved only by full-text processing <ref:2608.05155#pg5>. This capacity to analyze the complete text allows for a deeper understanding of political discourse than the truncated RoBERTa output permits <ref:2608.05155#pg5>.

Implications for Research and Future Directions

The findings suggest that "political news articles are not primarily characterised by lexical emotional polarity. They are characterised by framing, ideological orientation, and rhetorical strategy—precisely the dimensions the LLM-based approach captures <ref:2608.05155#pg5>. This implies that researchers relying solely on SA risk drawing conclusions from only the most extreme 30% of their corpus, missing the nuanced framing that characterises the majority of political coverage" <ref:2608.05155#pg5>.

The authors argue for a complementary framework: SA provides a first-pass polarity screen, and LLM-based multi-dimensional analysis provides the deeper, SSH-aligned analytical layer <ref:2608.05155#pg8>. While acknowledging challenges such as the opacity of LLM reasoning and higher computational costs, the paper advocates for developing SSH-aligned evaluation frameworks—measuring tool utility by research relevance rather than NLP benchmark performance <ref:2608.05155#pg8>. Future work is planned to extend this comparison to larger corpora and incorporate expert evaluation by political communication scholars <ref:2608.05155#pg8>.

REFERENCES

[1] Abercrombie, G. and Batista-Navarro, R. (2020). Sentiment and position-taking analysis of parliamentary debates: A systematic literature review. Journal of Computational Social Science, 3(1):245–270

[2] Baden, C., Pipal, C., Schoonvelde, M., and van der Velden, M. A. (2022). Three gaps in computational text analysis methods for social sciences. Communication Methods and Measures, 16(1):1–18 <ref:2608.05155#pg4>

[3] Bail, C. A. (2024). Can generative AI improve social science? Proceedings of the National Academy of Sciences, 121(21):e23140211 <ref:2608.05155#pg8>

[4] Barbieri, F., Camacho-Collados, J., Neves, L., and Espinosa-Anke, L. (2020). TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of EMNLP 2020, pages 1644–1650

[5] Budiharto, W. and Meiliana, M. (2018). Prediction and analysis of Indonesia presidential election from Twitter using sentiment analysis. Journal of Big Data, 5(1):1–10

[6] Card, D., Boydstun, A. E., Gross, J. H., Resnik, P., and Smith, N. A. (2015). The media frames corpus: Annotations of frames across issues. In Proceedings of ACL 2015, pages 438–444

[7] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171–4186 <ref:2608.05155#pg4>

[8] Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51–58 <ref:2608.05155#pg3>

[9] Field, A., Kliger, D., Wintner, S., Pan, J., Jurafsky, D., and Tsvetkov, Y. (2018).

Improvements for AI systems

  1. Bold header: Neutral Collapse Mitigation Strategy

Neutral collapse is addressed by replacing single-dimension polarity with a multi-dimensional framework that captures political bias direction and intensity, sensationalism, emotional appeal, and political framing. This allows the improved system to provide multi-dimensional analytical outputs aligned with SSH epistemologies, moving beyond the limitations of a three-class polarity judgment.

  1. Bold header: Structural Preservation via Full-Text Processing

The LLM platform processes full article texts (without truncation) and analyses each article across multiple dimensions. This directly counters the limitation of RoBERTa, which suffers from 512-token truncation, ensuring that cumulative political framing structures are preserved for analysis.

  1. Bold header: SSH-Aligned Utility Evaluation Framework

The system should adopt an evaluation framework that assesses tools not by NLP benchmark accuracy but by SSH research utility. This shifts validation criteria to include (1) dimensional coverage of SSH-relevant categories, (2) sensitivity to framing and rhetorical features, (3) robustness across journalistic registers, and (4) alignment with expert SSH judgments.

Abstract

Traditional sentiment analysis (SA) models, while effective for polarity classification, provide limited insight into the rhetorical, ideological, and framing dimensions of political discourse -- dimensions that are central to research in the social sciences and humanities (SSH). In this paper, we present a comparative study of RoBERTa-based sentiment analysis and an LLM-based multi-dimensional framing analysis platform applied to a corpus of 50 political news articles from 17 international media outlets. The results reveal a critical limitation we term neutral collapse: RoBERTa classifies 70% of articles as neutral, effectively flattening substantively rich political content into an analytically uninformative category. We find that 23% of neutral-classified articles exhibit negative probability scores above 0.30. By contrast, the LLM-based approach captures political bias direction and intensity, sensationalism, emotional appeal, and political framing -- yielding multi-dimensional analytical outputs aligned with SSH epistemologies. We argue that for political media analysis, traditional SA alone is insufficient, and that LLM-based multi-dimensional frameworks offer a more epistemologically adequate computational lens for SSH research needs.

Sources

Related papers