Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

Using behavioral science, health interventions focus on behavior change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes.

The gist: Existing models show limited performance for A/H recognition, suggesting that more adapted multi-modal models are required for accurate A/H recognition.

Problem Statement

Ambivalence and hesitancy (A/H) play a primary role for individuals delaying, avoiding, or abandoning health interventions. These are described as subtle and conflicting emotions that place a person in a state between positive and negative evaluations of a behaviour, or between acceptance and refusal to engage in it. A/H manifests as affective inconsistency across modalities or within a modality, such as language, facial, vocal expressions, and body language. While experts can be trained to recognize A/H in person, integrating them into digital health interventions is costly and less effective, making automatic recognition critical for personalization.

Methodological Framework

The study explores the application of deep learning models for A/H recognition in videos using three distinct learning setups:

1. Supervised learning.

2. Unsupervised domain adaptation for personalization.

3. Zero-shot inference via large language models (LLMs).

The experiments are conducted on the unique and recently published BAH video dataset, which is composed of videos from 300 participants answering seven questions, providing three modalities: visual, audio, and transcript (text), annotated at frame- and video-level. The dataset contains a total of 1,427 videos and 10.60 hours of recording.

Modality Pre-processing

The pre-processing involves extracting features for each modality:

(Visual)

Frames are extracted at 24 fps, faces are cropped and aligned using the RetinaFace model, resized to 256x256 RGB images. Visual backbones explored include ResNet architectures (ResNet18, 34, 50, 101, and 152) and ViT-based models.

(Audio)

Videos are converted to mono audio channels at a 16 kHz sampling rate in WAV format. Log Melspectrogram features are extracted using the VGGish model.

(Text)

Audio transcripts are generated using the Whisper model, and word-level features are extracted using the BERT Base Uncased model. A word-level feature is repeated per its timestamp for all corresponding frames to synchronize with other modalities.

Learning Scenarios and Findings

The paper investigates performance across different learning scenarios:

(Supervised Learning)

Models are trained on a basic emotion recognition task (e.g., 'happy', 'sad') using large mixed datasets before being fine-tuned on the BAH train set for A/H recognition, with under-sampling of the negative class to account for class imbalance. Frame-level classification performance shows that context modeling improves results, as temporal dependency between frames is leveraged using a Temporal Convolutional Network (TCN).

(Multimodal Baselines)

When combining modalities, text modality alone yields better performance than visual or audio modalities for frame-level classification. The impact of feature fusion techniques is examined, showing that LFAN and CAN fusion lead over both metrics. Combining all three modalities slightly reduces performance in terms of AP, suggesting the need for more adapted fusion techniques.

(Video-level Classification)

Experiments using Video-FocalNet show that a specialized model yields better performance on average over both metrics compared to models trained on basic emotion understanding. The Video-FocalNet base-case yields an AP of 0.6732 and AVGF1 of 0.5663, leading the board.

(Personalization using Domain Adaptation)

Domain adaptation is explored where each participant in the test set is considered an independent target domain for frame-level classification. The results indicate that ACPL [110] achieves the best performance, with an AVGF1 of 0.5417 and an AP of 0.3739, highlighting the benefit of combining pseudo-labeling with subject-aware alignment over methods like MMD or SHOT.

(Zero-shot Inference using MLLMs)

Zero-shot inference using Multimodal Large Language Models (MLLMs) is tested by varying the query prompt. The best results for frame-level prediction are obtained using the Transcript + Def 1 prompt, where the actual transcript of the video is also provided, suggesting that the textual modality serves a significant role in the identification of the A/H. The performance is substantially influenced by the query prompt, with adding both definition and transcript improving results.

Recommendations for Future Work

The paper recommends a two-level framework for more interpretable A/H recognition systems:

1. Modeling affect per modality independently (e.g.

Here are specific improvements to existing AI systems derived from the findings in this paper, along with what those improved systems can achieve:

1. Improve Video-Based Ambivalence/Hesitancy (A/H) Recognition Systems by Implementing a Dedicated Conflict Detection Module.

2. Enhance Personalization Capabilities Using Subject-Based Domain Adaptation and Test-Time Adaptation (TTA) Strategies for Digital Health Interventions.

3. Develop Multimodal Large Language Models (MLLMs) for Zero-Shot A/H Inference by Optimizing Prompt Engineering with Transcript Integration.

4. Implement a Two-Level Framework for A/H Recognition: Modality-Specific Modeling followed by a Dedicated Cross-Modal Conflict Assessment Module.

5. The improved system can move beyond simple positive/negative emotion classification to explicitly detect the state where an individual exhibits contradictory cues (e.g., smiling while speaking with a tone indicating reluctance, or a positive facial expression paired with negative body language). This allows for the detection of subtle, high-risk emotional states that standard models miss.

6. By integrating Subject-Based Domain Adaptation (like ACPL) and Test-Time Adaptation (TTA), the AI system can adapt its recognition model in real-time or per user session using minimal, personalized data. This enables a digital health intervention to instantly adjust its support strategy—for example, shifting from encouraging physical activity cues to offering motivational dialogue—based on the user's momentary hesitation captured in their video feed.

7. The enhanced MLLM system can perform accurate A/H recognition even when only given a text transcript and a definition of ambivalence (Zero-Shot Inference). This allows for rapid prototyping of new digital health features without requiring massive, task-specific labeled datasets for every new emotional nuance. Furthermore, by using the transcript in the prompt, the model can leverage linguistic context to better disambiguate visual cues related to A/H.

8. The proposed Two-Level Framework will allow researchers to separate affect per modality (e.g., classifying facial expression as 'happy' or 'sad') from cross-modal conflict assessment. This modularity allows for interpretable decision-making; the system can first identify a conflict between the audio tone and the visual expression, confirming that A/H is present, leading to more robust and explainable digital health interventions.

Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes. In-person interventions are costly and difficult to scale, especially in resource-limited regions. Digital health interventions offer a cost-effective approach, potentially supporting independent living and self-management. Automating such interventions, especially through machine learning, has recently gained considerable attention. Ambivalence and hesitancy (A/H) play a primary role for individuals to delay, avoid, or abandon health interventions. A/H are subtle and conflicting emotions that place a person in a state between positive and negative evaluations of a behaviour, or between acceptance and refusal to engage in it. They manifest as affective inconsistency across modalities or within a modality, such as language, facial, vocal expressions, and body language. While experts can be trained to recognize A/H, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition is therefore critical for the personalization and cost-effectiveness of digital health interventions. Here, we explore the application of deep learning models for A/H recognition in videos, a multi-modal task by nature. In particular, this paper covers three learning setups: supervised learning, unsupervised domain adaptation for personalization, and zero-shot inference via large language models (LLMs). Our experiments are conducted on the unique and recently published BAH video dataset for A/H recognition. Our results show limited performance, suggesting that more adapted multi-modal models are required for accurate A/H recognition. Better methods for modeling spatio-temporal and multimodal fusion are necessary to leverage conflicts within/across modalities.