XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection".

Jane: The gist The XED dataset introduces a multilingual fine-grained emotion dataset consisting of human-annotated Finnish and English sentences, along with projected annotations for 30 additional languages,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection," and what this paper is really doing is creating a foundation for emotion work that spans many languages. It’s not just focusing on one language; it’s about expanding the scope significantly across different tongues.

Jane: I get why that matters, because we know that most existing datasets are heavily biased towards English or just a few specific languages. This paper seems to be trying to fix that by offering something usable across a much wider range of languages.

Lu: They are using movie subtitle data from OPUS as their source material, which is interesting because it provides parallel sentences across different languages at the same level of granularity. This structure helps them evaluate things consistently.

Meng: I wonder about the scale of this projection—how many new languages are we talking about? Because if you're projecting annotations for thirty others, that’s a lot of data to manage and test at scale.

Lalam: They specifically mention human-annotated Finnish sentences are twenty-five thousand and English sentences are thirty thousand plus those projected annotations across thirty more languages. That’s the core composition of XED.

The paper's summary: Tom: So to summarize what they found in "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection," they created this dataset using Plutchik’s Wheel of Emotions—anger, anticipation, disgust, fear, joy, sadness, surprise and trust—plus neutral. They used this to build a multilabel multiclass system.

Jane: The key finding is that when they test this dataset with language-specific models like BERT and SVMs against existing benchmarks like GoEmotions by Demszky et al., two thousand twenty XED performs on par with them, which is a pretty strong result for a cross-lingual resource <ref:2011.01612#pg1>.

Lu: They found that the human-annotated Finnish data gets a macro F1 score of zero point five one when evaluated with FinBERT, and the projected Finnish data has an accuracy of zero point four five in the same evaluation scenario, showing how close they are to human annotation quality for that language.

Meng: So it’s not just that they built a dataset; it’s about proving that these resources are useful tools for sentiment analysis and emotion detection, which is what we need to know if we want to use this in production.

Lalam: They also noted some interesting label combinations, like anger with disgust or joy with trust being the most common beyond single labels, which gives us insight into how people actually express emotions in these texts.

The paper's improvements: Tom: The authors highlight a few improvements, mainly focusing on how they handle the projection aspect of the data. They show that using unique alignment IDs for parallel sentences helped guide the English annotations onto the Finnish sentences, which resulted in an average annotation correlation using Cohen’s kappa of zero point four four.

Jane: That correlation number is interesting because it’s not perfect, but they point out that accuracy by percentage was over ninety percent for that projection, and highest for joy at zero point six five, suggesting the projection method is quite effective even if it differs from human annotation slightly.

Lu: The paper notes that this difference in annotation projection is similar to how human annotations differ from each other, which suggests the projected data isn't completely divorced from reality but rather represents a similar degree of variance.

Meng: If you’re deploying a model, you need to know what the limitations are. They mention that for binary classification—mapping emotions onto positive and negative categories—their model gets an accuracy of zero point eight four zero, which is solid, but they also point out confusion between anger and disgust in the confusion matrix.

Lalam: They also show results with a linear SVM classifier where one class is fear or trust getting a per-class F1 score as high as zero point eight eight three two, which shows that for certain emotions, their method is quite good at distinguishing them.

Conclusion: Tom: So to wrap up the XED paper, it establishes XED as a useful multilingual fine-grained emotion dataset by combining human Finnish and English data with projected annotations for thirty other languages. It shows that this resource performs on par with other state-of-the-art datasets when evaluated by models like BERT and SVMs.

Jane: The implication for us is that we have a new tool to tackle emotion detection in many low-resource languages, moving beyond just English or Finnish data. It’s a real step toward cross-lingual understanding.

Lu: And the fact that they used parallel movie subtitle data to create these datasets means this methodology could be applied to other domains where parallel text exists, which is ambitious for representation learning.

Meng: For deployment, the focus should be on how easily these projected annotations can be scaled up without losing that quality we saw in the correlation scores. We need to see if it translates smoothly from research data to actual live traffic scenarios.

Lalam: Lalam thinks the biggest contribution is opening up emotion datasets for under-resourced languages, giving them a chance to build models and apply sentiment analysis where they previously had very little material.

Tom: So that’s the XED paper. It gives us a resource to work with when building classifiers, especially in those less represented linguistic areas. We'll keep an eye on how the community builds on this foundation for emotion detection in the future.