EMMR: Emotion-Mediated Multimodal Reasoning for Personality Assessment in Asynchronous Video Interviews

arXiv:2608.07512 · cs.HC, cs.AI · Submitted 2026-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "EMMR: Emotion-Mediated Multimodal Reasoning for Personality Assessment in Asynchronous Video Interviews".

Jane: The paper was written by Dongsheng Hu, Yuan Zong, Tianyi Zhang, Yong Li, Chuang Liu et al. from Hangzhou Normal University and Southeast University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Hey everyone, welcome back to the show. Today we're digging into a paper that's got a mouthful of a title: "EMMR: Emotion-Mediated Multimodal Reasoning for Personality Assessment in Synchronous Video Interviews."

Jane: And Tom, I have to say, the title actually tells you a lot once you unpack it. We're talking about job interviews that happen over video, and the paper is trying to figure out someone's personality from how they answer questions.

Tom: Right, and the key word there is "emotion-mediated." That's the twist. Most personality assessment from interviews just looks at what people say, the words. This paper says, hey, you're missing half the picture if you ignore how they say it.

Jane: Exactly. So the authors are from Hangzhou Normal University and Southeast University, and they've built a system that watches the video, listens to the audio, and reads the transcript, then uses emotion as the bridge to figure out personality traits.

Tom: And that's the part that gets me excited. We've all seen those video interviews where someone says all the right things but sounds like a robot, or looks terrified. If you only read the transcript, you'd think they're confident. The paper is basically saying that's a blind spot.

Jane: It really is. And the implications go beyond just hiring. Think about any situation where you're trying to read someone through a screen, telehealth appointments, online teaching, customer service calls. If we can teach machines to pick up on emotional cues, we can make those interactions smarter.

Tom: So this isn't just an academic exercise. This could change how companies screen candidates, and maybe even how we think about what "personality" means when we're not in the same room.

Jane: And that's the big question we'll dig into. Can a machine actually understand emotion well enough to make a judgment call about who you are as a person?

Tom: Stay tuned, because we're about to find out how they pulled it off.

Summary: Tom: So Jane, we've got the title unpacked. Now let's talk about what the paper actually does. The summary here is pretty dense, but the core idea is actually simple.

Jane: It really is. They call it EMMR, and it works in two stages. First, they extract emotion cues from the video, the audio, and the text. Then they feed those cues into a large language model to make the personality judgment.

Tom: And the clever part is that they don't just say "this person looks happy." They turn the emotion into a written description, like "the candidate smiled while discussing teamwork, suggesting genuine enthusiasm." That way the language model can reason about it.

Jane: Right, because these models are really good at reading text. So if you can translate a facial expression into a sentence, the model can actually use that information. It's like giving the AI a translator for body language.

Tom: And they tested this on two datasets, OPVA and AVI-six which are collections of actual video interviews. The results show their method beats the baseline models on almost every personality trait they measured.

Jane: The numbers are pretty striking. On the OPVA dataset, they got the error down to zero point two eight seven two for Conscientiousness, which is a big improvement over just using text. And the correlation with human ratings went up to zero point eight three for Extraversion.

Tom: That correlation number is huge. It means their predictions are lining up with what human experts would say, not just getting close on average but actually ranking people in the right order.

Jane: And what's really interesting is that they tested this in a zero-shot setting. That means the model wasn't trained on these specific interviews. It just looked at the videos and made its best guess, and it still outperformed models that were trained on the data.

Tom: So this isn't a case of the model memorizing patterns. It's actually understanding something about how emotions connect to personality.

Jane: Exactly. And that's what makes this paper exciting. It's not just a better algorithm. It's evidence that emotions are a legitimate signal for personality assessment, and that we can teach machines to use that signal.

Tom: So next we should talk about what this means for the real world, because I think there are some big implications here.

Improvements: Tom: So Jane, we've covered what EMMR does and how well it works. But what does this paper actually improve over what was already out there?

Jane: That's the key question. And the answer is that previous approaches were mostly text-only. You'd transcribe the interview, feed the words to a language model, and ask it to score personality traits. The paper's big improvement is adding the emotional layer.

Tom: And they show that just adding the raw video or audio doesn't help much. In fact, some baseline models that used video performed terribly, with correlation scores near zero. But when you extract the emotion and turn it into text, suddenly the model can use it.

Jane: Right, that's the "emotion-mediated" part. The emotion isn't just extra data. It's the bridge that connects what you see and hear to what the language model can understand. They did an ablation study where they removed different parts of the system, and every time they removed the emotion cues, performance dropped.

Tom: So it's not just a nice bonus. The emotion cues are doing real work. But here's what I found interesting, they also tested different types of emotion descriptions. Just saying "happy" or "sad" helped a little, but describing why, like "smiling while discussing a past achievement," helped a lot more.

Jane: That makes sense. A label like "happy" is vague. But a reasoning-based description gives the model context. It can connect the emotion to the situation, which is closer to how a human interviewer would think.

Tom: And that's the real improvement here. It's not just about adding more data. It's about adding the right kind of data, structured in a way that the model can actually reason with.

Jane: There's also a practical improvement in stability. Some of the baseline models were really inconsistent. They'd do great on one personality trait and terribly on another. EMMR is much more even across the board.

Tom: So the paper is saying, look, if you want reliable personality assessment from video interviews, you need to pay attention to emotions, and you need to describe them in a way that makes sense to the model.

Jane: And that's a lesson that could apply beyond just interviews. Any time you're trying to get a language model to understand human behavior, you need to give it the right kind of input.

Tom: Alright, so we've got the improvements. Now let's look at the actual first page of the paper and see what the authors are really arguing.

First Page: Tom: So we're looking at the first page of "EMMR: Emotion-Mediated Multimodal Reasoning for Personality Assessment in Synchronous Video Interviews," and the opening is actually a pretty strong argument.

Jane: It is. The authors start by pointing out that asynchronous video interviews, or AVIs, have become really common, especially after the pandemic. Companies are using them to assess personality, which they see as a key indicator of job fit.

Tom: And then they drop this important point. They say that most AI systems for this task rely only on the transcribed responses. But they cite something called Brunswik's Lens Model, which basically says personality judgments are formed through multiple observable cues.

Jane: So the argument is that if you only look at words, you're missing the facial expressions, the tone of voice, the hesitation, all those non-verbal signals that actually carry a lot of information about who someone is.

Tom: And they give a great example. Someone might describe themselves as confident and proactive, but if their voice is hesitant and their face looks tense, a human interviewer would pick up on that contradiction. A text-only system would just take the words at face value.

Jane: That's the core problem they're trying to solve. And it's not just about being more accurate. It's about avoiding misinterpretation. If someone says they're outgoing but looks anxious, a text-only system might overestimate their Extraversion.

Tom: And the authors back this up with psychological research showing that emotions are closely linked to personality traits. People who score high on Extraversion tend to show more positive emotions, for example.

Jane: So the paper is grounded in established psychology, not just machine learning. That gives it a lot more credibility. They're not just throwing data at a model and hoping it works. They're building on what we know about how humans express personality.

Tom: And that's what makes this paper stand out. It's not just a technical trick. It's an argument that personality assessment needs to be multimodal, and that emotion is the key to making that work.

Jane: Right. And the first page sets up that argument really clearly. You know exactly what problem they're solving and why it matters.

Tom: So let's wrap this up and think about what this means for the future.

Conclusion: Tom: Alright, let's bring it home. We've been talking about "EMMR: Emotion-Mediated Multimodal Reasoning for Personality Assessment in Synchronous Video Interviews," and I think we've got a clear picture now.

Jane: We do. The paper proposes a two-stage framework that extracts emotion cues from video, audio, and text, then uses those cues to help a language model assess personality traits. And it works, beating text-only baselines on accuracy and consistency.

Tom: The key insight is that emotions are the bridge between raw behavioral signals and personality. You can't just feed a model a video and expect it to understand. You have to translate those signals into something the model can reason about.

Jane: And the implications are pretty big. This could change how companies do remote hiring, making it fairer and more accurate. But it also raises questions about privacy and bias that we'll need to think about carefully.

Tom: Yeah, the authors themselves acknowledge that the system depends on the quality of the emotion extraction. If the audio model misreads a neutral tone as sad, that error can propagate into the personality score. So there's still work to do.

Jane: But that's what makes this exciting. It's a real step forward, and it opens up a lot of avenues for future research. Better emotion extraction, more sophisticated fusion, maybe even fine-tuning the model for this specific task.

Tom: So we'll say goodbye to this paper, but I think the ideas in it are going to stick around. Emotion-aware personality assessment feels like the direction the field is heading.

Jane: Absolutely. And with that, we'll wrap up our discussion of EMMR. Thanks for listening, and we'll see you next time with another paper from the arXiv.

Tom: Take care, everyone.

Dongsheng Hu, Yuan Zong, Tianyi Zhang, Yong Li, Chuang Liu, Wenming Zheng, XiuXiu Zhan

Hangzhou Normal University · Southeast University

cs.HC, cs.AI

Submitted: 2026-06-30

Updated: 2026-08-11

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 58/100

The gist: The paper proposes EMMR (Emotion-Mediated Multimodal Reasoning), a two-stage framework for MLLMs-based personality assessment in Asynchronous Video Interviews (AVIs).

Key concepts

EMMR
Emotion-Mediated Multimodal Reasoning for Personality Assessment in Synchronous Video Interviews. It is a system that extracts emotion cues from video, audio, and text to help a language model make personality judgments.
Multimodal Reasoning
The ability of an AI system to reason by processing multiple types of data simultaneously. In this paper, it means combining visual cues (video), auditory cues (audio), and textual responses to understand personality.
Emotion-Mediated
The core idea that emotions serve as the bridge between raw behavioral signals and personality traits. The system translates emotional states into written descriptions so the language model can reason about them.

Terminology

Summary

The paper proposes EMMR (Emotion-Mediated Multimodal Reasoning), a two-stage framework for MLLMs-based personality assessment in Asynchronous Video Interviews (AVIs). The authors state: "we propose EMMR (Emotion-Mediated Multimodal Reasoning), a two-stage framework for MLLMs-based personality assessment for AVIs. EMMR extracts emotion-related cues from multimodal interview data and incorporates them into personality assessment through structured reasoning as auxiliary social and behavioral evidence."

The motivation is that "text-centered methods may overlook non-verbal behavioral cues conveyed through visual and audio modalities, even though such cues are highly relevant to personality assessment. In particular, emotion-related cues provide important social and affective evidence for understanding candidates' behavior related to personality traits. The authors note that relying only on transcribed responses may lead to misinterpretation when verbal content conflicts with non-verbal behavioral cues."

The framework consists of two modules: (1) Emotion cue extraction module which "processes visual and audio modalities to extract modality-specific emotion cues, including facial expressions and facial dynamics from visual inputs, as well as prosodic features such as tone fluctuations and speech rhythm from audio inputs, organizing them into semantic descriptions, providing interpretable evidence across modalities." (2) Emotion cue fusion and personality assessment module in which the extracted emotion cues are integrated with interview responses and incorporated into MLLMs-based personality assessment through structured reasoning.

The emotion cue extraction stage uses a structured prompting framework that emphasizes interpretability and stepwise analysis with five components: Role Definition, Task Objective, Instructions, Data Input, and Answer Template. The instructions use an explicit chain-of-thought strategy with three steps: identifying modality type and timestamp information, analyzing dynamic changes within each modality (e.g., observe facial action units in regions like eyebrows, eyes, and mouth for visual; examine prosodic features such as pitch variation and speech rhythm for auditory), and generating emotion state with traceable explanations. After initial extraction, an LLM (Gemma2) is used to perform structured filtering on the raw emotion cues to identify duplicated or semantically overlapping cues and remove vague or uncertain judgments.

The personality assessment stage uses a prompt with seven components: Role Definition, Task Objective, Emotion Cue Label, Constraints, Instructions, Data Input, and Answer Template. The model is assigned the role of a cognitive psychologist in personality assessment and requires the model to assign a continuous numeric score (ranging from 1.0 to 5.0) for each trait. The model "first identifies the target personality trait associated with each question, then analyzes the participant's responses together with multimodal emotion cues to extract relevant behavioral evidence, and finally assigns a continuous score with justification grounded in the observed cues."

Experiments were conducted on two datasets: OPVA (685 candidates, eight interview questions targeting eXtraversion and Conscientiousness) and AVI-6 (646 candidates, questions covering Honesty-Humility, eXtraversion, Agreeableness, and Conscientiousness). The backbone model was Qwen2.5-VL (7B), with Kimi-Audio (7B) for audio emotion analysis and Gemma2 (9B) for cue integration.

Results show that EMMR achieves an MAE of 0.3686 for eXtraversion (X) and 0.2872 for Conscientiousness (C), alongside MSE of 0.2109 and 0.1310 respectively, on the OPVA dataset. On AVI-6, EMMR yields an MSE of 0.1606 for Honesty-Humility (H), 0.1501 for Agreeableness (A), 0.1533 for X, and 0.0928 for C, culminating in a low average MSE of 0.1392. EMMR achieves a high PCC of 0.8338 for X and 0.7432 for C on OPVA.

Compared to baselines (Qwen2.5, Kimi-audio, Kimi-VL, Qwen2-VL, Qwen2.5-VL, DAN, VAT, Swin-transformer), EMMR consistently outperforms all baseline models across evaluation metrics and datasets. Traditional models exhibit remarkably weak PCC. Their PCC values are confined to a narrow range between –0.1359 and 0.0950.

Ablation studies showed that visual, audio, and emotion cues each play essential and complementary roles. Removing visual modality increased average MSE from 0.1710 to 0.2464 on OPVA and from 0.1503 to 0.2168 on AVI-6. Removing audio modality increased MSE to 0.3080 on OPVA and 0.2963 on AVI-6. Removing emotion cues resulted in average MSE increases of 0.2498 on OPVA and 0.2375 on AVI-6. Removing transcribed text caused a substantial degradation, with average MSE surging to 0.4776 on OPVA and 0.5688 on AVI-6.

The discussion addresses three research questions. First, regarding which modality is most suitable: "there is no absolute 'optimal' auxiliary modality in isolation; rather, their suitability depends heavily on the fusion strategy. While transcribed text serves as the indispensable semantic backbone, raw non-verbal signals introduce high variance when used independently. However, when volatile non-verbal behaviors are transformed into structured semantic descriptions of emotion cues and fused alongside the text, visual information becomes the most effective complementary modality."

Second, regarding whether semantic descriptions of emotion cues help: "transforming non-verbal emotion information into appropriate semantic spaces can serve as a beneficial complement to pure dialogue transcripts. Among the different semantic representation strategies, emotion reasoning semantics seem to be the most compatible with the reasoning mechanisms of LLMs. Coarse-grained labels offer limited descriptive richness, whereas fine-grained feature descriptions might introduce modality-specific fragmentation and semantic inconsistencies."

Third, regarding why audio emotion cues increase variability in Extraversion assessment: the audio model generated a substantial proportion of negative emotions, including sadness (30%), contempt (14.3%), boredom (9.4%), and anxiety (4.1%), neutral emotions account for 32.9%. The authors found that "the candidates' actual emotional states were predominantly neutral. It appears that acoustic characteristics such as slower speech rates and lower pitches were frequently misclassified by the audio extraction models as negative emotions." This caused prediction errors, with anxiety (MSE=0.8035) and sadness (MSE=0.6577) showing notably larger errors compared to neutral emotions (MSE=0.2920).

The authors acknowledge limitations: the framework relies entirely on carefully designed prompts to guide MLLMs, bypassing task-specific fine-tuning, our approach primarily integrates information at the text semantic level, and the method's effectiveness is heavily dependent on the stability and reliability of the emotion cues extracted from audio and video inputs.

Future work directions include exploring parameter-efficient fine-tuning strategies, developing more sophisticated multimodal fusion mechanisms with adaptive fusion architectures that dynamically assign weights to visual, audio, and textual streams based on their contextual reliability, and mitigating inherent biases within MLLMs and improving model calibration.

The authors conclude: "Experimental indicate that EMMR generally improves zero-shot assessment performance across the evaluated HEXACO dimensions. Further analyses suggest that structured emotion semantics effectively complement textual data, aligning with psychological frameworks. Overall, this study provides empirical insights suggesting that fusing explicitly modeled multimodal behavioral evidence with language-based reasoning represents a meaningful direction for future research in personality assessment."

Improvements for AI systems

Based on the paper, here are the specific improvements I can make to AI systems and the resulting capabilities:

Improvement: Implement a two-stage framework where emotion cues are first extracted from visual, audio, and text modalities, then fused with interview responses before personality assessment.

Capability: The AI system can now:

  • Decompose video inputs into synchronized visual and audio streams

  • Extract emotion cues (facial expressions, vocal prosody, linguistic sentiment) with timestamp alignment

  • Generate structured semantic descriptions of emotions rather than raw labels

  • Jointly reason over verbal content and non-verbal behavioral evidence

Improvement: Implement a five-component prompt (Role Definition, Task Objective, Instructions, Data Input, Answer Template) that guides the model through stepwise emotion reasoning: identify modality → analyze dynamic changes → generate emotion state with traceable explanation.

Improvement: Transform heterogeneous non-verbal signals into unified text-based semantic descriptions before fusion, rather than attempting raw feature-level fusion.

Improvement: Implement a post-extraction filtering step using an LLM (e.g., Gemma2) to remove duplicate, overlapping, or vague emotion descriptions while preserving modality tags.

Improvement: Associate each interview question with its target HEXACO trait and explicitly link extracted emotion cues to that trait during assessment.

Metric Baseline (Qwen2.5-VL) EMMR Improvement

MAE (OPVA avg) 0.5369 0.3114 42% reduction

MSE (AVI-6 avg) 0.4493 0.1392 69% reduction

PCC (OPVA X) 0.0269 0.8338 Significant correlation gain

  1. Conduct robust personality assessment from asynchronous video interviews with accuracy comparable to human raters, even with sparse contextual information (single question per trait).

  2. Detect and reconcile verbal-nonverbal conflicts—for example, identifying when a candidate claims confidence but exhibits anxious vocal patterns or tense facial expressions.

  3. Provide interpretable assessment rationales by tracing personality scores back to specific observable emotion cues (e.g., raised eyebrows or rising intonation).

  4. Maintain stability across diverse traits and contexts—avoiding the extreme performance fluctuations seen in baseline MLLMs (which ranged from MAE 0.23 to 0.81 across traits).

  5. Adapt to varying data richness—performing well both when multiple questions per trait are available (OPVA) and when only single questions exist (AVI-6).

  6. Filter noisy emotion signals to prevent misclassification errors (e.g., neutral speech mislabeled as sad) from propagating into personality judgments.

Abstract

Asynchronous Video Interviews (AVIs) have become increasingly popular for personality assessment. Recent large language models (LLMs) have shown potential for personality assessment from transcribed interview responses. However, text-centered methods may overlook non-verbal behavioral cues conveyed through visual and audio modalities, even though such cues are highly relevant to personality assessment. In particular, emotion-related cues provide important social and affective evidence for understanding candidates' behavior related to personality traits. Thus, we propose EMMR (Emotion-Mediated Multimodal Reasoning), a two-stage framework for MLLMs-based personality assessment for AVIs. EMMR extracts emotion-related cues from multimodal interview data and incorporates them into personality assessment through structured reasoning as auxiliary social and behavioral evidence. Experiments on two AVIs datasets, OPVA and AVI-6, show that EMMR improves MAE, MSE, and PCC compared with baselines. Further analysis indicates that semantic descriptions of emotion cues enhance personality assessment, while their quality affects personality assessment reliability. These results suggest that integrating emotion-related cues into multimodal reasoning is a promising direction for more interpretable MLLMs-based personality assessment in AVIs.

Sources

Related papers