Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild

arXiv:2407.12927 · cs.CV · Submitted 2024-07-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild".

Jane: Textualization of modalities augments data with emotional cues to help large language models (LLMs) encode interconnections between all modalities in a shared text space,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about who wrote this. The paper is "Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild," and it lists Nicolas Richet, Soufiane Belharbi, Haseeb Aslam, Meike Emilie Schadt, Manuela González-González, Gustave Cortal, Alessandro Lameiras Koerich, Marco Pedersoli, Alain Finkel., Simon Bacon. It shows a really collaborative effort from a group of researchers.

Jane: It’s impressive that so many people contributed to this work; having such a diverse team behind the research usually means they’ve been looking at the problem from many angles.

Lu: The team is clearly tackling this problem from multiple directions, which is exactly what you need when dealing with something as complex as compound emotion in the wild. They're bringing together different expertise to build this system.

Meng: I wonder how their diverse backgrounds helped them decide between the textualized and feature-based approaches they are comparing in the paper. That kind of varied input must have shaped their methodology significantly.

Lalam: Having multiple perspectives on data processing is crucial; it means they aren't just relying on one way to turn audio or visual signals into something useful for emotion recognition.

The paper's summary: Tom: So, the main idea of the "Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild" paper is that traditional methods struggle with compound emotions because they often focus on just one basic emotion at a time, but this new approach adds textual descriptions to boost accuracy.

Jane: They are proposing two main ways to do this: one where they use existing features from audio and video along with text transcripts, and another where they actively convert the non-verbal cues into descriptive text first.

Lu: The core idea is that by adding these textual descriptions—like describing an action unit's intensity or the tone of a voice—they can give the system richer information to understand those complex emotions that happen in real life.

Meng: So, instead of just feeding raw visual and audio features into a fusion module, they are augmenting those features with descriptive text derived from analyzing the source modalities themselves. That adds a layer of interpretation before the final classification happens.

Lalam: Exactly; this textualization acts like an expert-based data augmentation process where they use prior knowledge about emotion cues to create new, descriptive text that helps the AI make a better call on compound emotions.

The paper's improvements: Tom: When we look at what they suggest as improvements for this approach, they point out that using textualized models can achieve better results when you have rich transcripts available. They found this was particularly true when testing on the MELD dataset for basic emotion recognition.

Jane: That makes sense; if you have good text context, the system seems to perform better at understanding those subtle emotional blends compared to relying solely on features extracted from the video and audio streams.

Lu: The paper suggests that their textualized approach can yield performance about four percent higher than the feature-based method specifically on the MELD dataset, which is a solid metric to compare against standard setups <ref:2407.12927#pg0>.

Meng: That improvement comes directly from how they handle the transcript quality; if you have high-quality transcripts, you get more detailed input for that textual modeling pathway. I'm curious how robust their system remains when those transcripts aren't available in the wild, though.

Lalam: The paper suggests a clear path forward: it’s best to use this textualized approach only when you have rich transcripts because that’s where the benefit really shines for compound emotion recognition.

Conclusion: Tom: So, to wrap up on the "Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild," they confirm that textualization of modalities significantly boosts performance when rich transcripts are present, especially for those tricky compound emotions.

Jane: They are essentially showing us that augmenting raw data with meaningful text descriptions is a viable strategy for improving multimodal emotion recognition in real-world videos, provided we have access to good transcripts.

Lu: It opens up a new avenue where the AI doesn't just look at the signals but interprets them through a textual lens, which has huge potential for understanding human behavior more deeply.

Meng: From an engineering standpoint, it tells us that our next steps should probably focus on building pipelines that can reliably generate these descriptive texts efficiently so we can actually use this method in production systems.

Lalam: I think the biggest implication is that by encoding multimodal inputs into a shared text space, we give the AI a unified language to reason about complex emotional interactions across sight, sound, and language simultaneously.

Tom: That’s right; it really pushes us toward more sophisticated ways of modeling these interactions. So that’s our rundown on this paper for now.

Jane: We've covered a lot about how textualization helps unlock better understanding of compound emotions in video data.

Lu: It’s a very promising direction for the future of how we interpret human behavior through AI systems.

Meng: Hopefully, we see more practical applications soon where these contextual cues are easily accessible during operation.

Lalam: We'll keep an eye on how this textualized modeling influences the next generation of multimodal understanding systems.

Nicolas Richet, Soufiane Belharbi, Haseeb Aslam, Meike Emilie Schadt, Manuela González-González, Gustave Cortal, Alessandro Lameiras Koerich, Marco Pedersoli, Alain Finkel, Simon Bacon, Eric Granger

LIVIA, ILLS, Department of Systems Engineering, ETS Montreal, Canada · Department of Health, Kinesiology & Applied Physiology, Concordia University, Montreal, Canada · Montreal Behavioural Medicine Centre CIUSSS Nord-de-l’Ile-de-Montréal CIUSSS Nord-de-l’Ile-de-Montréal CIUSSS Nord de l'Île de Montréal · Université Paris-Saclay CNRS ENS Paris Saclay LMF, 91190, Gif sur Yvette, France · Institut Universitaire de France, France

cs.CV

Submitted: 2024-07-17

Updated: 2026-10-02

Code: https://github.com/nicolas-richet/featurevs-text-compound-emotion

Importance score: 70/100

The gist: Textualization of modalities augments data with emotional cues to help large language models (LLMs) encode interconnections between all modalities in a shared text space, offering an alternative to

Key concepts

Feature-based Modeling
This approach extracts separate features from different data types—like facial movements from video or vocal characteristics from audio—using specialized models for each. These extracted features are then combined using techniques such as LSTMs or attention mechanisms to try and understand the overall emotion in a video sequence.
Text-based Modeling
This method creates textual descriptions from the audio (analyzing tone) and visual data (extracting action units). These descriptions, along with existing text transcripts, are combined into a single prompt fed into an LLM to classify the final emotion. This leverages the LLM's ability to process complex, combined textual information.
Textualization
This technique adds emotional cues by converting raw data from modalities (audio/video) into descriptive text. This text is then used alongside transcripts and fed into an LLM, allowing the model to better encode how different sensory inputs relate to each other in a shared textual space.
Compound Emotion Recognition (CER)
This is the task of identifying multiple emotions present simultaneously within a video. The paper tests different models against this challenge, comparing how well feature extraction versus textual description methods can accurately identify these complex emotional states.

Terminology

Summary

Textualization of modalities augments data with emotional cues to help large language models (LLMs) encode interconnections between all modalities in a shared text space, offering an alternative to traditional feature-based models for compound emotion recognition in real-world videos.

The gist

Multimodal textualization provides lower accuracy than feature-based models on C-EXPR-DB where text transcripts are captured in the wild, but higher accuracy can be achieved when the video data has rich transcripts.

Feature-based Modeling

This approach extracts features from audio (vocal) and video (facial) modalities, and text transcripts for multimodal CER in videos. The general motivation behind combining these modalities is to leverage their complementary information over a video sequence. Each modality typically employs a dedicated pre-trained feature extractor, such as ResNet [18] for visual modality or VGGish [19] for audio modality. Multiple text feature extractors are available, including BERT [10] and RoBERTa [43].

Feature-Level Fusion involves combining these extracted features. Different methods rely on temporal models to combine features from over a video, such as LSTMs [7, 54, 48], or rely on simple concatenation. Recent works focus more on self- and cross-modal attention and transformers [59] to perform attention-based fusion, which can capture interand intra-modality relationships. In one specific experiment for CER in videos, the method used ResNet50 [18] for visual modality, VGGish [19] for audio modality, BERT [10] over text modality, and a temporal convolutional network (TCN) [2]. A co-attention block is employed to attend to features from different modalities. This builds a single embedding per frame while leveraging a contextual window.

Text-based Modeling

This approach extracts textual descriptions from audio (vocal) and visual (facial) modalities for multimodal CER in videos, and combines them with text transcripts. The process involves several steps:

  1. Audio Text Description: The API of Hume Inc. is employed over a sliding window to analyze the tone of the audio, selecting the top 10 tone characteristics based on scores like confusion or anxiety, which are then used to describe the tone textually. A fine-tuned Wav2Vec 2.0 model [61] predicts scores for arousal, valence, and dominance for each audio clip, categorized as Low or High using a threshold. These textual descriptions are concatenated.

  2. Visual Text Description: Face cropping and alignment are performed at each frame using RetinaFace [8]. The Py-feat library is then used to extract action units (AUs) intensity [12, 14] along with basic emotion probabilities. AUs codebook [4] maps facial expressions to action units, and the text description concatenates the names of selected AUs or the top 3 basic emotions.

  3. Combination of Transcripts with Audio and Visual Texts: Text transcripts are generated using Whisper-large-v2 [52] when unavailable. All textual descriptions are combined into a single prompt template fed to an LLM, such as LLaMA-3 [47]. The prompt structure includes speech transcription, visual AUs text, visual emotions text, and prosody characteristics. A fully connected layer (FC) follows the LLM for classification.

Comparison and Results

Experiments were conducted on the challenging C-EXPR-DB dataset for CER and the MELD dataset for basic ER. Ablation studies showed that on MELD, the text-based approach yielded the best performance with approximately 4% above feature-based approach, mainly due to high transcript quality. On C-EXPR-DB, where transcripts are limited in most videos, the feature-based method was ahead of the text-based method. Specifically for C-EXPR-DB, visual modality seemed to be the strongest in the feature-based method (F1 score of 53.46%), while audio seemed strongest in the text-based method (F1 score of 50.96%). The paper concludes that textualization is most beneficial when rich transcripts are available, suggesting one should attempt to use textualization over feature-based methods only when rich transcripts are available.

Challenges of Textualizing Modalities

The process of textualization faces several limitations:

  1. Need for domain experts: There is a need for domain experts to select the cues to be extracted from each modality and how they should be textualized, as handcrafted cues may not provide optimum textualization and require new expertise for different applications.

Improvements for AI systems

Here are the specific improvements to AI systems derived from this research, categorized by approach:

) Improvements for Feature-Based Modeling Systems:

  1. Enhanced Spatio-Temporal Fusion via Co-Attention (LFAN): Integrate the proposed co-attention block (LFAN) into existing feature fusion modules.

  2. Improved Feature Extraction Pipelines: Utilize specialized, pre-trained backbones like ResNet50 (pre-trained on MSCELEB1M/FER+) for vision, VGGish/Wav2Vec 2.0 for audio, and BERT/RoBERTa for text, ensuring feature extractors are frozen while subsequent modules are fine-tuned.

  3. Refined Temporal Modeling: Employ Temporal Convolutional Networks (TCN) after each feature extractor to better leverage spatio-temporal dependencies in video data during the fusion process.

The improved system will perform more robust and contextually aware emotion classification in videos by explicitly learning inter-modal relationships through attention mechanisms, leading to higher accuracy on complex compound emotions.

) Improvements for Text-Based Modeling Systems:

  1. Advanced Modality Textualization: Implement a rigorous, expert-guided process for textualizing non-verbal cues (e.g., using Hume API for tone characteristics, Py-feat for Action Unit intensity).

  2. Contextual Prompt Engineering with LLMs: Develop sophisticated prompt templates that synthesize heterogeneous textual inputs (transcripts, AU descriptions, prosodic features) into a single, rich context window for the Large Language Model (LLM).

  3. Efficient LLM Fine-Tuning: Employ Parameter-Efficient Fine-Tuning techniques like QLoRA when fine-tuning models like LLaMA-3 8B to make them computationally feasible for downstream tasks.

The improved system will translate diverse non-verbal cues into a unified textual representation, allowing powerful LLMs to encode the complex interconnections between all modalities in a shared text space, potentially achieving higher accuracy when rich transcripts are available.

) Improvements for Overall System Architecture:

  1. Hybrid Fusion Strategies (Ensembling): Implement frame-level ensembling across multiple model outputs (feature-based, text-based, and zero-shot MLLM) using majority voting or average logits to produce a final, more reliable prediction for each frame.

  2. Adaptive Modality Weighting: Develop mechanisms to dynamically assess the reliability of different modalities (visual vs. audio vs. text) for specific content segments within a video sequence, allowing the system to leverage the strongest modality for that moment (e.g., prioritizing visual cues when audio is noisy).

  3. Zero-Shot MLLM Integration: Integrate state-of-the-art Zero-Shot Multimodal Large Language Models (like LLaVa-NextVideo) as a powerful baseline, utilizing constrained prompting to test the limits of multimodal reasoning before relying on supervised fine-tuning.

The overall improved system will be more resilient to modality conflicts and data scarcity by employing hybrid fusion strategies and leveraging the reasoning capabilities of LLMs, resulting in superior performance for both basic and compound emotion recognition in real-world video environments.

Sources

Related papers