Emotion Recognition in Sign Language Conversation

arXiv:2605.23328 · cs.CL · Submitted 2026-05-22 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Emotion Recognition in Sign Language Conversation".

Tom: Emotion Recognition in Sign Language Conversation addresses a critical gap in affective computing by introducing a new task and dataset to move beyond isolated utterances.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, to get into the nitty-gritty, this paper introduces a task called Emotion Recognition in Conversation for sign language video analysis and proposes something called the eJSL Dialog dataset to solve the limitations of existing datasets.

Jane: It’s fascinating how they specifically constructed this dataset using dialogue scripts from the STUDIES corpus to give models more than just single utterances; they provide historical context.

Lu: The authors are clearly focused on bridging that structural gap, moving past isolated sentences that generic models struggle with because they can't use the flow of conversation to predict emotion.

Meng: I’m interested in how they structured the data collection, since building a dataset this large and contextually rich sounds like a huge undertaking for any engineering team.

Lalam: It's about giving the AI something meaningful to learn from beyond just one sign; it's about training it to understand social interaction patterns.

The paper's summary: Tom: The main finding they present is that generic multimodal conversational emotion recognition models simply fail when applied directly to sign language data because they don't have the necessary context-aware visual extractors.

Jane: That’s a crucial point, Tom; it shows that just combining vision and text isn't enough if the visual part doesn't understand the specific nuances of sign language gestures in sequence.

Lu: They show a clear "domain gap," meaning the models trained on general data don't transfer well to this specialized domain without tailored visual understanding.

Meng: So, they’re essentially saying that we need visual tools built specifically for sign language, not just off-the-shelf ones that work for hearing people.

Lalam: This suggests that the next big step in AI isn't just bigger models, but models with specialized vision components tuned precisely to how sign language is made.

The paper's improvements: Tom: One of the main contributions they highlight is formally defining the ERC task for sign language video analysis, which gives researchers a solid objective benchmark for evaluating bidirectional interaction scenarios.

Jane: That formal definition helps standardize how we measure success when trying to build systems that can track emotional history across turns in a conversation.

Lu: They also construct and release the eJSL Dialog dataset itself, which is a big improvement because it directly addresses the structural limitation of relying only on isolated utterance datasets.

Meng: The paper notes that they conduct systematic benchmarking using five different baseline models, ranging from purely visual networks to multimodal graph convolutional networks, which gives us a good idea of what we’re up against.

Lalam: It’s about providing a concrete resource, the dataset, so other researchers can actually build and test these more context-aware systems without starting from scratch every time.

Conclusion: Tom: So to wrap things up, the paper confirms that visual models without contextual awareness struggle with dynamic emotional transitions in sign language, and they emphasize that expanding the scale of conversational datasets is necessary for large-scale pre-training.

Jane: It boils down to this: we need specialized visual extractors and massive amounts of conversational data to build truly empathetic dialogue systems for sign language users.

Lu: The implication is that understanding emotion in sign language isn't just a recognition problem anymore; it’s about modeling complex, multi-turn social dynamics.

Meng: Practically speaking, this means we have a clearer roadmap for where the engineering efforts should go next if we want to build functional systems for real conversations.

Lalam: I think this work sets up the foundation for AI that can genuinely track and respond to the emotional history of deaf users, which is essential for meaningful human-computer interaction.

Institute of Science Tokyo

cs.CL

Submitted: 2026-05-22

Updated: 2026-10-06

Importance score: 55/100

The gist: Emotion Recognition in Sign Language Conversation addresses a critical gap in affective computing by introducing a new task and dataset to move beyond isolated utterances.

Key concepts

eJSL Dialog Dataset: This dataset was built from Japanese empathetic dialogue scripts, containing 1,920 video samples across 480 dialogues. It provides conversational history by including four consecutive utterances per dialogue to allow models to understand emotional transitions over time.
Emotion Recognition in Conversation (ERC) Task: The goal is to predict an emotion label based on the current multimodal input, historical context, and a translation of spoken language. The formula is yt = f(Vt, Tt, C), meaning the prediction depends on the visual sign language (Vt), text/gloss (Tt), and context (C).
Domain Gap: This refers to the performance drop when using generic emotion models on sign language data. It shows that models trained on other modalities cannot handle sign language because they lack the specific visual features needed, highlighting a need for modality-specific visual extractors.

Terminology

Benchmark — 13× in this paper · explained in 2 episodes
F1 Score — 11× in this paper · explained in 3 episodes
Actor — 9× in this paper · explained in 3 episodes
Affective Computing — 8× in this paper · explained in 1 episode
Emotion Recognition in Conversation — 8× in this paper
Benchmarking — 7× in this paper · explained in 3 episodes
Multimodal Fusion — 5× in this paper · explained in 3 episodes
Contextual Awareness — 3× in this paper · explained in 2 episodes
Core — 3× in this paper · explained in 2 episodes
Distillation — 3× in this paper · explained in 2 episodes
Generalization — 3× in this paper · explained in 13 episodes
Feature Extraction — 2× in this paper · explained in 2 episodes
Adaptation — 1× in this paper · explained in 1 episode
Aggregation — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Japanese Sign Language — 1× in this paper
Multimodal Dataset — 1× in this paper · explained in 2 episodes
Representation Learning — 1× in this paper · explained in 2 episodes

Summary

Emotion Recognition in Sign Language Conversation addresses a critical gap in affective computing by introducing a new task and dataset to move beyond isolated utterances. The eJSL Dialog dataset and subsequent benchmarking reveal that generic multimodal conversational emotion recognition models fail when applied to sign language, explicitly demonstrating the need for context-aware visual extractors specific to this modality.

The gist

The eJSL Dialog dataset, constructed from the STUDIES corpus, is used to benchmark models and reveals a domain gap when applying generic multimodal conversational emotion recognition models to sign language, indicating an explicit need for context-aware visual extractors specific to sign language.

Dataset Construction and Scope

The paper introduces the Emotion Recognition in Conversation (ERC) task for sign language video analysis and proposes the eJSL Dialog dataset to address the structural limitation of existing isolated utterance datasets. This dataset is constructed using dialogue scripts from the STUDIES Japanese Empathetic Dialogue Speech Corpus, containing 1,920 video samples organized into 480 unique dialogues centered around teacher and student interactions. Each dialogue consists of four consecutive utterances, providing sufficient conversational history to model emotional transitions.

Formal Task Definition

The emotion recognition in sign conversation task is formally defined as learning a mapping function that predicts the emotion label based on the current multimodal input and historical context. The task is expressed as:

yt = f(Vt, Tt, C) where Vt conveys original sign language information, Tt is an utterance-level spoken language translation or gloss, and C represents the historical context set of previous utterances. This formulation highlights that complete prediction relies on the current multimodal input and the historical context set.

Baseline Model Evaluation

Systematic benchmarking was conducted using five baseline models: EmoAffectNet (purely visual), EANwH (extended visual with LSTM for temporal dynamics), TelME (conversational, cross-modal distillation), EmoTrans (transition-based ERC model), and MMGCN (multimodal graph convolutional network). The evaluation utilized a weighted F1 score as the primary overall metric due to the unequal distribution of emotion categories.

Key Findings from Benchmarking

The results confirm that visual models lacking contextual awareness fail to capture dynamic emotional transitions. Furthermore, generic multimodal conversational emotion recognition models exhibit performance degradation; for instance, TelME achieved a weighted F1 score of 10.46 and MMGCN recorded 8.50 when applied to sign language data. This suggests that generic text-visual fusion fails because cross-domain visual features interfere with the text modality when sign-specific cues are not explicitly disentangled. Case studies illustrate this, showing that contextual reasoning and textual semantics are critical for modeling emotion dynamics, as evidenced by EmoTrans and TelME correctly predicting Joy in context where visual models fail.

Conclusion and Future Directions

The research concludes that the eJSL Dialog dataset addresses the limitations of isolated sign language emotion recognition by providing multiple turn dialogue history. The findings expose the need for developing visual extractors specific to sign language and constructing large-scale conversational datasets to support large-scale pre-training. Future work will focus on scaling the dataset, collecting longer dialogue sessions, and integrating large language models to enhance contextual modeling of emotional transitions across extended conversations.

Limitations

Limitations identified include the small data scope (two actors in a controlled environment), which may affect generalization to diverse signer demographics and lighting conditions. Additionally, the emotion categories are constrained by the original STUDIES corpus, and the current benchmark does not explicitly disentangle linguistic markers from emotional states within visual cues. The paper notes that models trained on non-signing datasets struggle because sign language visual features differ from those of hearing individuals.

Impact

The impact lies in providing a benchmark for bidirectional interactions, supporting the future development of empathetic dialogue systems capable of tracking and responding to the emotional history of deaf users, which is essential for creating context-aware human-computer interaction. The paper establishes three complementary factors in sign language emotion understanding: conversational context, textual semantics, and visual cues.

References

[1] R. W. Picard, Affective computing. MIT press, 2000.

[2] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information fusion, vol. 37, pp. 98–125, 2017.

[3] Y. Wu, Q. Mi, and T. Gao, “A comprehensive review of multimodal emotion recognition: Techniques, challenges, and future directions,” Biomimetics, vol. 10, no. 7, p. 418, 2025.

[4] O. Koller, “Quantitative survey of the state of the art in sign language recognition,” arXiv preprint arXiv:2008.09918, 2020.

[5] M.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems based on the findings of this research, and what those improved systems could achieve:

Improved AI Systems: Develop context-aware visual extractors specifically tuned for sign language (JSL).
System Capability: The improved system will be able to accurately distinguish between grammatical features encoded in hand signs and affective states (emotions) within a single sign language utterance, mitigating the inherent ambiguity that plagues generic models.
Improved AI Systems: Construct and utilize large-scale conversational datasets of sign language video samples (like the proposed eJSL Dialog dataset) for robust pre-training.
System Capability: The improved system will be capable of modeling dynamic emotional evolution across multiple turns in a conversation, allowing it to understand how an individual's emotional state shifts based on their interlocutor's responses and historical dialogue flow, moving beyond simple isolated utterance recognition.
Improved AI Systems: Integrate multimodal architectures (combining visual sign language features with textual/linguistic context) specifically designed to disentangle linguistic markers from emotional cues in sign language.
System Capability: The improved system will achieve superior performance in complex conversational scenarios, successfully resolving cases where factual statements (described via text) are misinterpreted as neutral by purely linguistic models, and correctly identifying emotion based on visual signs when textual modifiers are absent.
Improved AI Systems: Implement transition-based models (like EmoTrans) that explicitly model emotional transitions between consecutive turns in a dialogue sequence.
System Capability: The improved system will accurately predict the change in emotional state from utterance to utterance, providing a richer, more temporally sensitive understanding of the affective dynamics within a conversation compared to models relying only on the current frame or isolated utterances.
Improved AI Systems: Develop specialized visual feature extractors (like those proposed for EANwH) that effectively fuse spatial features (facial expressions/body movements) with temporal information (LSTM networks) specifically trained on sign language kinematics.
System Capability: The improved system will be highly effective at capturing the subtle, non-verbal emotional signals embedded in sign language video, even when facial expressions are restrained or ambiguous, leading to higher accuracy in detecting emotion categories like 'Sad' or 'Joy' where visual cues dominate over linguistic context.