Rationale-Guided Learning for Multimodal Emotion Recognition

RATIONALE-GUIDED LEARNING FOR MULTIMODAL EMOTION RECOGNITION

The paper proposes Rationale-Guided Learning (RGL), a novel framework for multimodal emotion recognition in conversation (MERC). The authors argue that existing approaches treat emotion recognition as a direct input-output mapping problem, overlooking the causal reasoning humans use when interpreting emotions. To address this, RGL transforms MERC into a cognitively-inspired reasoning task based on dual-process theory, decomposing emotional reasoning into three facets: Intuitive (System 1, immediate perception), Contextual (System 2, situational analysis), and Integrative (synthesis of both).

The framework uses a Multimodal Large Language Model (MLLM, specifically GPT-4o) to generate structured rationales offline during training preparation. These rationales are encoded into three rationale banks (Intuitive, Contextual, Integrative) using a pre-trained text embedder (BGE-large-en-v1.5). The final model operates without any MLLM overhead at inference time.

The trainable model consists of unimodal encoders (ViT-base for visual, RoBERTa-large for text, HuBERT-base for audio) with a dual-head architecture for visual and textual encoders, outputting both main features for emotion prediction and rationale features for alignment with rationale banks. A multimodal fusion module uses Transformer encoder layers with attention pooling. Training aligns the model's rationale features with corresponding rationale vectors via contrastive learning with hard negative mining (top-K=128 most similar negatives), using a temperature parameter τ=0.07 and balancing hyperparameter λ=0.3.

The three alignment objectives are: (1) aligning visual features with Intuitive rationale, (2) aligning textual features with Contextual rationale, and (3) aligning fused features with Integrative rationale. The total loss combines cross-entropy loss with the three rationale-guided alignment losses.

Experiments on IEMOCAP and MELD benchmarks show state-of-the-art performance. On IEMOCAP, RGL achieves W-F1 of 73.68 and Acc of 73.51, outperforming previous best methods (e.g., MAGTKD with 72.91 W-F1). On MELD, RGL achieves W-F1 of 67.43 and Acc of 68.31, also surpassing prior state-of-the-art. Ablation studies reveal that removing all rationale losses causes the most significant performance drop (W-F1 drops from 73.68 to 68.01), with the Contextual loss being the most critical component (removing it drops W-F1 to 68.78), followed by Integrative (72.44) and Intuitive (72.70) losses.

For interpretation, the authors demonstrate that the model's internal features can retrieve semantically correct rationales from the banks for unseen test samples, validating its rationale reasoning capabilities. The paper concludes that emulating cognitive reasoning through rationale-guided learning is an effective approach for advancing multimodal emotion recognition.

Improvements to AI Systems:

1. Add a rationale-alignment training objective to multimodal encoders.

Instead of training encoders solely for final label prediction, add auxiliary contrastive losses that align intermediate visual, textual, and fused representations with pre-generated, structured rationales (intuitive, contextual, integrative). This forces the model to learn causally meaningful features rather than superficial correlations.

2. Implement a dual-head encoder architecture for visual and text modalities.

Each encoder outputs two feature vectors: one for the main prediction task and one for rationale alignment. This separation prevents rationale learning from degrading task-specific feature quality, while still injecting reasoning-based inductive biases.

3. Use hard negative mining (top-K=128) in contrastive learning for rationale banks.

During training, sample the most similar but incorrect rationale vectors as negatives. This sharpens the model’s ability to distinguish between subtle emotional contexts, improving generalization to unseen conversational scenarios.

4. Decouple inference from large language model overhead.

Generate rationales offline using an MLLM (e.g., GPT-4o) and encode them into fixed banks. At inference, the deployed model uses only lightweight unimodal encoders and a transformer fusion module—no MLLM calls—making it fast, cost-effective, and deployable in real-time systems.

5. Prioritize contextual (System 2) reasoning in the loss weighting.

Since ablation shows the contextual rationale loss is most critical (removing it drops W-F1 by 5 points), design the training loss to weight contextual alignment higher (e.g., λ contextual > λ intuitive, λ integrative). This improves performance on emotion recognition tasks requiring situational understanding.

---

What the Improved AI System Can Do:

• Recognize emotions in conversations with higher accuracy and robustness, especially in ambiguous or context-dependent utterances, by leveraging causally grounded reasoning rather than surface-level patterns.

• Operate in real-time (e.g., live customer service, mental health monitoring) without needing a large language model at inference, while still benefiting from LLM-generated reasoning during training.

• Generalize better to new speakers and unseen emotional expressions because the rationale banks provide a structured, semantic prior that reduces overfitting to dataset-specific biases.

• Provide interpretable behavior: the model’s internal features can retrieve the most relevant rationale (e.g., speaker is sarcastic due to previous disagreement), enabling explainable AI outputs for debugging or user trust.

• Adapt to low-resource settings by transferring rationale-guided representations to new emotion domains or languages, as the alignment losses teach transferable reasoning patterns rather than task-specific shortcuts.