Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Quality and Agreement in Multilabel Emotion Annotation".

Jane: The gist The paper presents a multilabel emotion annotation case study and uses it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So to get into it, the authors are Emily Ohman and Anna Koufakou from Waseda University and Florida Gulf Coast University in Tokyo. They’ve put this case study on emotion annotation in text, specifically using Hemingway’s The Old Man and the Sea as their example. Jane That sounds like a very specific setup for studying how subjective things like emotions are handled by technology.

Lu: It’s an interesting choice because it gives a concrete text to work with, and they use an expanded version of Plutchik’s wheel of emotions, which includes anger, anticipation, disgust, fear, joy, sadness, surprise and trust for the core set. Meng So they aren't just looking at one simple emotion; they're looking at a whole spectrum.

Lalam: And they also let annotators add free-text notes for any other emotions that pop up as "other" emotions or labels. Tom That adds another layer of complexity because it means the labels aren't just simple yes or no for those core feelings.

Jane: It suggests that just picking one main emotion isn't enough to capture what a piece of text might actually convey, and they are trying to figure out how to handle that richness. Lu Exactly. This paper is looking at how we can move beyond just finding the majority vote and start seeing the actual range of interpretations people have.

The paper's summary: Tom: They set up three annotators, A1, A2, and A3, and they look at how much they agree on different emotion labels across that whole dataset. Jane They found that preliminary agreement scores using Krippendorff’s alpha were low across the board when looking at the full set of labels.

Meng: That suggests that if we just take a single hard label, we’re going to get pretty noisy results because people aren't totally aligned on what they feel. Tom And then they introduce some things to make it more nuanced than just counting how many people picked a label.

Lu: They define targets as "hard labels," which is just one way to think about it, where a hard label is one if at least one annotator gave a positive score, and zero otherwise <ref:2606.21069#pg1>. But they also introduce something called an "intensity-weighted soft target" for modeling purposes. Jane That’s where things get interesting because they modulate the vote shares by those intensity scores that the annotators provide, scaling them from zero to one.

Lalam: These soft targets are interpreted as per-label Bernoulli probabilities, which means they give us a way to represent uncertainty rather than just a single guess. Tom That’s smart because it allows the downstream models to see the entire distribution of opinions, not just the loudest one.

The paper's improvements: Jane: The paper suggests that we shouldn't treat those hard labels as gospel for everything, especially when building classifiers. Tom They show that hard-label training can actually give you the highest F1 score in some scenarios, like when you include a specific annotator they flagged as reliable.

Lu: But the authors found that using those hard labels sometimes masks real uncertainty about what the annotators are thinking. They also introduce a whole evaluation pipeline designed to test this trade-off directly. Jane And that pipeline includes comparing traditional thresholded metrics, like macro-F1, against these probabilistic alignment metrics, which is called Bernoulli cross-entropy SoftBCE.

Meng: The results showed that for BERT under different conditions—using all annotators versus just the most reliable ones—the hard labels gave a macro-F1 of zero point six one six when Annotator two was included, but the soft labels actually yielded higher macro- and micro-F1 scores in both settings. Tom And they specifically pointed out that SoftBCE, which looks at the full predicted probability vector against those soft targets without thresholding, produced lower SoftBCE while still being competitive in F1.

Lalam: So the message is that the benefits of using soft supervision really depend on what you are trying to optimize for in your model. Jane It moves us away from just picking one 'gold' label and treating it as perfect, which is a big shift in how we approach training these systems.

Conclusion: Tom: So, to wrap up the main finding of this paper on "Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework," they are showing that disagreement is structured and leaves measurable traces in model behavior. Jane They found that hard labels might maximize F1 metrics, but soft supervision often yields predictions that better reflect the actual variance in annotator opinions.

Lu: The authors also point out a limitation, which they admit is that one annotator exhibited systematically sparse labeling behavior, which shows the need for clearer guidelines when we are looking at annotation workflows. Meng And another thing they flag is that intensity ratings aren't automatically helpful; their impact depends on how consistently annotators use the scale. Lalam It’s a methodological contribution because they use downstream models as a diagnostic tool to surface those design trade-offs between annotation choices and modeling choices. Jane That really frames annotation and modeling as two coupled decisions, not just sequential steps.

Tom: So we can see that treating majority-voted labels as perfect gold can actually mask uncertainty and overstate how reliable a model is. Lu It’s about using the paper "Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework" to show how to distinguish ambiguity in the data from behavior specific to the annotators.