SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning

summary

Video file (mp4)

The gist

The gist: This Systematisation of Knowledge (SoK) paper comprehensively systematises factors leading to inaccuracy in complex Human Activity Recognition (HAR) using deep learning, focusing on sensor,

In short

This Systematisation of Knowledge (SoK) paper analyzes why deep learning models struggle with complex Human Activity Recognition (HAR). It systematically examines challenges across sensor types, data variability, model capacity, and evaluation methods. The findings guide future research toward better domain generalization and multimodal fusion to improve accuracy.

Key concepts

Categorisation by Complexity
Activities are grouped into simple actions with clear patterns and complex actions involving dissimilar or varied movements. This distinction is crucial because the difficulty in recognition scales with the complexity of the activity being analyzed, requiring different model approaches for each type.
Factors Affecting Accuracy
Inaccuracy stems from four main areas: Sensor issues (placement and perspective), Data issues (variability between training and real-world data), Algorithm factors (computational cost vs. precision), and Evaluation flaws. Understanding these specific points helps pinpoint where deep learning models fail in complex scenarios.
Domain Generalisation
This research direction focuses on making models robust to new environments or data distributions. Techniques like using GANs to simulate unseen scenarios or Gradient Reversal Layers aim to train models that work well across different contexts, rather than just memorizing the training data.
Multimodal Approaches
This involves combining information from different sensing sources (like wearable and camera) through sensor fusion. Methods such as contrastive learning help the model learn meaningful correlations between these diverse data types while still allowing it to make reliable predictions from any single source.

Terminology used across episodes

This episode discusses

The paper

SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning · Read on arXiv

Duc-Anh Nguyen, Nhien-An Le-Khac

University College Dublin

Human Activity Recognition (HAR) is a well-studied field with research dating back to the 1980s. Over time, HAR technologies have evolved significantly from manual feature extraction, rule-based algorithms, and simple machine learning models to powerful deep learning models, from one sensor type to a diverse array of sensing modalities. The scope has also expanded from recognising a limited set of activities to encompassing a larger variety of both simple and complex activities. However, there still exist many challenges that hinder advancement in complex activity recognition using modern deep learning methods. In this paper, we comprehensively systematise factors leading to inaccuracy in complex HAR, such as data variety and model capacity. Among many sensor types, we give more attention to wearable and camera due to their prevalence. Through this Systematisation of Knowledge (SoK) paper, readers can gain a solid understanding of the development history and existing challenges of HAR, different categorisations of activities, obstacles in deep learning-based complex HAR that impact accuracy, and potential research directions.

DOI: 10.1109/IJCNN60899.2024.10650322

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning".

Jane: The gist: This Systematisation of Knowledge (SoK) paper comprehensively systematises factors leading to inaccuracy in complex Human Activity Recognition (HAR) using deep learning, focusing on sensor, data, algorithm,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re moving into summarizing "SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning." The main point here is that deep learning methods still struggle when trying to recognize complex activities because there are several factors causing inaccuracy >

Jane: It claims that this paper systematically organizes these factors, focusing on data variety and model capacity as the key hurdles in advancing complex HAR using modern deep learning methods >

Lu: It points out that they give extra attention to wearable and camera sensors because of their prevalence, which is an important context for anyone working in this area >

Meng: So it’s not just about the model being too big; it’s about the entire pipeline—the sensor input, the data quality, and how you set up your evaluation that matters >

Lalam: The paper is a knowledge systematisation effort designed to give readers a solid understanding of the development history of HAR and all those existing challenges >

Tom: Basically, it takes all that evolution from manual features to deep learning models and identifies where the modern deep learning approach hits roadblocks when dealing with complexity >

Jane: It helps us understand the different ways activities can be categorized—by complexity, structure, or interaction—to avoid confusion between different terms like composite versus concurrent activities >

Lu: So understanding these categorisations is key because it helps researchers focus their efforts on the right type of data and model architecture for that specific activity type >

Meng: If you don't categorize correctly, you’re just guessing, which means wasted training time and money on models that aren't suited for the task >

Lalam: It also lays out potential research directions based on these problems, like domain generalisation using data augmentation or multimodal approaches to better capture correlations between different sensors >

Tom: So what it matters for the world is that this paper gives us a structured framework to tackle the inaccuracies in complex HAR, moving away from just trial and error with massive models >

Jane: It shifts the focus toward being more intentional about sensor selection, data quality, and testing methods when you’re working on these kinds of problems >

Conclusion: Tom: So we’ve gone through the whole discussion on "SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning." The paper covers a lot, from sensor issues to evaluation metrics >

Jane: The authors are trying to show that if we want better complex HAR, we need to scale up our understanding of these problems rather than just scaling up the network size >

Lu: They suggest that future research should be very focused on learning how to make models robust against changes in data and environment through techniques like domain augmentation or using gradient reversal layers >

Meng: From a production standpoint, it means we need to prioritize creating smaller, efficient models that can handle those real-world deployment constraints without needing massive computational resources >

Lalam: The implication is that we need better ways to leverage the vast amounts of unlabelled data available by using self-supervised learning techniques to get better representations for complex actions >

Tom: So the big picture is that we have a roadmap now for tackling complexity in HAR, moving toward more intelligent systems that are robust against real-world messy inputs >

Jane: It’s about making the AI more intentional about what it's seeing and how it’s measuring success, which is where the next wave of improvement in this field will likely come from >

More episodes

← Home