Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations

summary

The gist

The gist The proposed model, AliAd, integrates multimodal multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in

In short

AliAd integrates contrastive learning and a mixture-of-experts module to handle human activity recognition with any number of available views, including missing ones. It uses an adjusted center contrastive loss to align views based on quality, reducing complexity from O(V^2) to O(V). This allows the model to flexibly fuse diverse modalities without needing to reconstruct absent data.

Key concepts

Adjusted Center Contrastive Loss (LAC): This loss function aligns views in a hyperspherical latent space while incorporating view quality. It pulls all views toward higher-quality ones, effectively mitigating the impact of missing views on the fusion process by weighting them based on their perceived quality.
Mixture of Experts (MoE) Module: A sparse MoE architecture is used for classification to handle discrepancies between different view combinations. It uses a load balancing loss to ensure the gating mechanism distributes input tokens evenly among experts, adapting to arbitrary view sets.
Attention-based View Fusion: The model uses an attention module to dynamically capture data quality and task relevance across views. Fusion is achieved by taking a weighted sum of views, where weights are computed based on the latent representation's relationship with high-quality views on the hypersphere.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations · Paper Radio

The paper

Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations · Read on arXiv

University College Dublin

Multimodal multiview learning seeks to integrate information from diverse sources to enhance task performance. Existing approaches often struggle with flexible view configurations, including arbitrary view combinations, numbers of views, and heterogeneous modalities. Focusing on the context of human activity recognition, we propose AliAd, a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on multiview fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from O(V 2) to O(V), where V is the number of views. To address residual discrepancies not captured by contrastive learning, we employ a mixture-of-experts module with a specialized load balancing strategy, tasked with adapting to arbitrary view combinations. We highlight the geometric relationship among components in our model and how they combine well in the latent space. AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.

DOI: 10.1109/JSEN.2026.3718255

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Align and Adapt".

Jane: The gist The proposed model, AliAd, integrates multimodal multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in human activity recognition.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Looking at this paper, "Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations," the core contribution is AliAd, which manages arbitrary view availability during both training and inference without needing to reconstruct missing views.

Jane: The authors show that using an adjusted center contrastive loss allows the model to align views in a hyperspherical latent space while factoring in view quality, which helps mitigate the impact of missing data on the fusion process >

Lu: They also use a mixture-of-experts module with a specialized load balancing strategy to address those remaining discrepancies between different view combinations that standard contrastive learning misses >

Meng: The paper proves that this approach works well across inertial and human pose datasets, handling view counts from three to nine, which suggests it has good practical performance and robustness when you run into missing views during real-world scenarios >

Tom: So, for the listener, this means we can build recognition systems that are much more flexible about what data they receive—they don't have to be perfect combinations of views to work reliably >

Jane: It moves the focus away from trying to perfectly fill in gaps and instead focuses on aligning what you *do* have, using view weights computed by an attention module for dynamic fusion >

Lu: The geometric relationship they highlight among the components in their model shows how these different parts combine effectively within that latent space when things are missing >

Meng: It’s a robust way to deploy multimodal AI because it doesn't require perfect, complete data sets to function reliably in production environments >

Conclusion: Tom: So we’ve seen how AliAd handles missing views and those weird view combinations, but let's talk about the actual title now, "Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations."

Jane: It sounds a little technical for a driver or someone cooking dinner. What does that actually mean in plain English?

Lu: It means the model is designed to handle any situation where you don't have all the data you want at once. If you only have three views instead of nine, it still works well because it adapts.

Meng: From a deployment standpoint, I worry about how much extra complexity this adds when we are running this on our servers. Does "Adapt" mean we need a massive GPU cluster just to handle the shifting view weights?

Lalam: The paper says the adjustment is done through an adjusted center contrastive loss that puts the center closer to higher quality views, which means it’s smart about which data it trusts.

Jane: So, instead of trying to perfectly rebuild what's missing data we can use a weighted sum based on view quality. That sounds like a practical way to make sense of incomplete information.

Tom: Exactly. The authors aren't trying to reconstruct the missing views, which saves us from needing extra complex reconstruction modules we’d have to train separately for every possible gap.

Lu: And they also use this mixture of experts part specifically to fill in the gaps where the contrastive learning isn't picking up any signal from a specific view combination.

Meng: I'm still thinking about the cost and complexity tradeoff here. If we switch from O(V squared) to O(V) for that loss, it’s good for training speed but does it keep inference fast enough when we have nine views?

Lalam: The computational complexity drops from O of V squared to O of V, meaning it scales linearly with the number of views. That should help keep things manageable on the edge devices.

Jane: It sounds like they’ve managed to make a very flexible system that doesn't break just because some sensors fail or a view is missing. What does this mean for actually building real-world activity recognition apps?

Tom: It means we can deploy these models in environments where sensor data is never perfect, which is most of the time. This flexibility opens up applications like monitoring people in messy homes or on uneven terrain.

Lu: The implication is that multimodal learning doesn't have to be a rigid pipeline anymore; it can be more resilient to real-world noise and incompleteness.

Meng: So for production, this suggests we don't need perfect, synchronized data streams to get high accuracy in activity recognition systems. That’s a big win for reliability.

Lalam: I'm confident the method described here offers a solid way to handle these variable inputs reliably across different modalities.

More episodes

← Home