Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization

arXiv:2510.06842 · cs.CV · Submitted 2025-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization".

Jane: Action Quality Assessment (AQA) evaluates human actions in videos, and this work introduces Continual AQA (CAQA),

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've gone through the details of "Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization," starting with what it claims and moving into the implications of its authors’ work. The paper, by Kanglei Zhou, Qingyi Pan, Xingxing Zhang, Hubert P. H. Shum, Senior Member IEEE, Frederick W. B. Li, Xiaohui Liang and Liyuan Wang provides a novel framework for Continual AQA that manages the tension between adaptation and stability in non-stationary scoring environments.

Jane: Essentially, they tackle the problem where standard continual learning methods fail because they can't handle evolving quality distributions without causing severe overfitting or forgetting past information. The MAGR++ framework, with its layer-adaptive fine-tuning and feature rectification pipeline, is designed to solve that by intelligently constraining shallow layers while allowing deeper ones to specialize.

Lu: The implication here is that for any AI system dealing with dynamic human performance metrics, we need architectures that are inherently aware of their own adaptation state and the underlying data manifold they're learning on. It moves AQA from a static evaluation to a dynamic process.

Meng: From my perspective at the startup, this suggests we could build more robust simulation environments or training pipelines for complex skills because these models would adapt better during long training cycles without needing constant, massive retraining cycles just to keep up with small skill improvements.

Lalam: For me, the impact is cultural; imagine personalized AI coaching systems that genuinely understand your evolving technique over months, adjusting their feedback based on the subtle shifts in your motion quality in a way that doesn't forget what you learned earlier. That level of continuous understanding is something we need to explore deeply.

Tom: It really boils down to moving toward AI agents that aren't just trained once and then forgotten, but rather systems capable of lifelong learning on their own, even when the underlying performance criteria are constantly changing. The success with those three point six percent offline and twelve point two percent online gains shows the practical viability of this approach for complex video analysis tasks.

Jane: That's a fair summary, Tom; it’s a method focused on maintaining relevance over time in dynamic evaluation settings through careful structural regularization rather than just brute-force parameter updates. It gives us a better tool for tracking skill progression accurately.

Conclusion: Tom: So, we've been diving deep into how this paper tackles that tricky problem of tracking skill progress in AI action assessments over time using Continual AQA via MAGR++.

Jane: Exactly, and the title itself tells us a lot about what they’re trying to achieve—they’re building a system that keeps learning while staying stable. The authors are Zhou, Pan, Zhang, Shum, Li, Liang and Wang.

Lu: What's fascinating is how they structure the core idea around this Manifold-Aligned Graph Regularization; it sounds like a very sophisticated way to manage the relationship between old skills and new ones in the feature space.

Meng: From a practical standpoint, I'm really interested in how they handle those non-stationary score distributions; if the AI gets confused by shifting quality standards, it’s useless in real-world deployment scenarios.

Lalam: I see this as a huge step toward building truly resilient AI that can grow with human expertise rather than getting stuck at a single point of performance. This suggests a cultural shift where skill acquisition is continuous and adaptive across different users.

Tom: Right, it’s about making sure the AI doesn't just memorize old stuff but actually learns new nuances as the environment keeps changing, which is what this framework aims to do.

Jane: It really boils down to giving the AI a smart way to adjust its internal representations so it can adapt gracefully without losing everything it already knows.

Lu: The authors propose a layer-adaptive fine-tuning strategy which seems clever for balancing stability in the early layers while letting deeper ones capture the specific variations of different sessions.

Meng: I wonder if this means we could deploy more personalized AI trainers that track an individual's progress much more accurately over long training periods than we can now.

Lalam: That level of continuous, nuanced understanding could profoundly impact education and personal development tools, allowing them to truly mirror a human mentor’s adaptive feedback style.

Tom: So, in short, this paper introduces a method that uses layered fine-tuning and graph regularization to achieve better long-term performance in dynamic skill assessment tasks.

Jane: It gives us a clearer picture of how we can design AI that learns and evolves robustly even when the evaluation criteria are constantly shifting.

Lu: And the results, with those impressive correlation gains, show that their complex methodology actually translates into solid real-world predictive power across different datasets.

Meng: The practical impact is clear: better models for robotics or complex human-AI interaction where performance metrics aren't static.

Lalam: This research opens the door to AI systems that don't just perform a task once, but genuinely evolve their capabilities alongside the user or environment they operate in.

Tsinghua University

cs.CV

Submitted: 2025-10-08

Updated: 2026-10-05

Comments: Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

Journal ref: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

DOI: 10.1109/TPAMI.2026.3741078

Code: https://github.com/ZhouKanglei/MAGRPP

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Action Quality Assessment (AQA) evaluates human actions in videos, and this work introduces Continual AQA (CAQA), a novel setting that extends Continual Learning to AQA tasks by addressing the

Key concepts

Non-stationary Quality Distributions
This refers to the problem where the scores or quality metrics used to evaluate actions are not fixed. In real scenarios, these distributions shift as skills improve or groups change, making it hard for models trained on old data to perform well on new data.
Layer-Adaptive Fine-Tuning
This strategy fine-tunes different layers of the neural network at different rates. It constrains shallower layers from drifting while allowing deeper layers to adapt fully to session-specific variations, ensuring stability in early features while capturing complex, session-dependent details.
Manifold Projector (MP)
The MP is a component trained to map old, deviated features into the current representation space. It minimizes the difference between historical and updated features, effectively aligning past representations with the new distribution manifold to prevent catastrophic forgetting.
Intra-Inter-Joint Graph Regularization (IIJGR)
This technique enforces consistency between feature representations and quality scores by analyzing relationships using angular distances on a hypersphere. It ensures that the geometric structure of learned features aligns with the expected patterns in the action quality scores across different sessions.

Terminology

Summary

Action Quality Assessment (AQA) evaluates human actions in videos, and this work introduces Continual AQA (CAQA), a novel setting that extends Continual Learning to AQA tasks by addressing the dilemma between capturing fine-grained motion cues through continual adaptation and maintaining stability under non-stationary score distributions. The paper proposes Adaptive Manifold-Aligned Graph Regularization (MAGR++), an innovative framework that couples backbone fine-tuning with a two-step feature rectification pipeline to achieve state-of-the-art performance in CAQA benchmarks.

The gist

MAGR++ achieves state-of-the-art performance, with average correlation gains of 3.6% offline and 12.2% online over the strongest baseline, confirming its robustness and effectiveness.

Motivation and Challenges for CAQA

A major challenge in AQA is the non-stationary nature of quality distributions in real-world scenarios, which limits conventional methods' generalization ability. Existing Pretrained Models (PTMs) decline when distributions shift, as scoring patterns evolve with individual skill progression and differ across groups. The complexity of CAQA poses unique challenges that render even strong Continual Learning (CL) baselines ineffective when applied directly due to the dilemma between capturing fine-grained motion cues through continual adaptation and maintaining stability under non-stationary score distributions. Existing PTM-based CL methods typically follow two paradigms: (i) extensive base-session adaptation followed by feature freezing, or (ii) Parameter Efficient Fine-Tuning (PEFT), which are insufficient for CAQA because AQA relies on subtle motion cues, meaning frozen features without adequate adaptation fail to generalize across evolving distributions.

Proposed MAGR++ Framework

MAGR++ introduces a two-step feature rectification pipeline coupled with layer-adaptive fine-tuning to address overfitting and distribution shift. The overall framework involves:

  1. A layer-adaptive fine-tuning strategy that constrains shallow layers from drifting while fully tuning deeper ones to embrace session-specific variations. This boundary is determined adaptively by identifying the optimal layer, defined as the layer where the abstraction ratio, calculated using the Davies–Bouldin index between frozen and fine-tuned backbones, exceeds a margin:

Lopt = min l ∈ 1... L r l > 1 + ε.

  1. A Manifold Projector (MP) trained to translate deviated historical features into the current representation space by minimizing the discrepancy between predicted and actual updated features: Lproj = 1 / Nt Σ X j h t j − hˆ t j22.

  2. An Intra-Inter-Joint Graph Regularization (IIJGR) that enforces both local (intra-session) and global (inter-session) consistency between the feature and score spaces by leveraging angular distances on a unit hypersphere and a distance matrix partitioning strategy.

Stability–Adaptability Balance

The framework balances stability and adaptability through several mechanisms. The layer-adaptive FPFT constrains shallow layers below Lopt using a feature-matching loss: Ltune = 1 / Nt Σ X l<Lopt z t,l i − z t−1,l i22. This soft constraint still permits shallow layers to undergo limited adaptation while preventing uncontrolled drift. Furthermore, the Manifold Projector (MP) aligns old features with the updated manifold via h s i ← h s i + p(h s i), ensuring stable replay that alleviates catastrophic forgetting. Finally, the IIJGR enforces alignment between feature geometry and quality scores by minimizing Lreg = A − S22 + Σ X 2 i=1 X 2 j=1 Aij − Sij22, where A is the angular distance matrix and S is the score distance matrix.

Experimental Validation

The method was evaluated across four CAQA benchmarks derived from three datasets, including MTL-AQA, FineDiving, and UNLV-Dive. The performance metrics used are:

  1. Overall Correlation (ρavg): Aggregates predictions from all sessions into a single unified estimate.

  2. Average Forgetting (ρaft): Measures the degradation on previous tasks.

  3. Forward Transfer (ρfwt): Measures the ability to generalize to new domains.

Experiments show that MAGR++ consistently achieves state-of-the-art performance, surpassing the strongest baseline by 1.6%–6.5% offline and 4.0%–21.8% online, with average gains of 3.6% and 12.2%, respectively across the tested settings (Table I and II).

Improvements for AI systems

Here are the specific improvements for AI systems based on the Continual AQA (CAQA) framework proposed by MAGR++, and what these improved systems can achieve:

The core improvement is a novel continual learning architecture, MAGR++, which extends standard Action Quality Assessment (AQA) models to handle non-stationary quality distributions in real-world scenarios without catastrophic forgetting.

Here are the specific enhancements and capabilities:

  1. Building a Robust Continual AQA System using MAGR++:

  2. Handling Non-Stationary Skill Evolution: The system can continuously learn and assess human actions across evolving skill levels, user populations, or environmental conditions (e.g., in sports scoring or rehabilitation) without forgetting previously learned assessment patterns.

  3. Mitigating Catastrophic Forgetting via Layer-Adaptive Fine-Tuning: By constraining updates to shallow layers (capturing stable low-level visual cues) while fully fine-tuning deeper layers (modeling abstract execution quality), the system achieves a principled balance between stability and adaptability during continual adaptation.

  4. Correcting Feature Manifold Shifts via Manifold Projector (MP): The system explicitly estimates the shift between previous and current feature representations and translates old stored features into the current representation space, ensuring that historical data remains relevant for accurate score regression.

  5. Enforcing Feature-Score Alignment via Intra-Inter-Joint Graph Regularization (IIJ-GR): The system enforces consistency between feature geometry and quality score relationships by using angular distances on a unit hypersphere and distance matrix partitioning. This ensures that features from different sessions maintain their semantic meaning relative to the quality scores, preventing distortion in the regression head.

  6. Achieving State-of-the-Art Performance: The resulting system consistently achieves superior performance (e.g., up to 30% gains offline and 12% online over strong baselines) across diverse benchmarks like MTL-AQA, FineDiving, and UNLV-Dive, even under severe domain shifts.

This improved AI system can perform the following specific tasks:

  1. Assess the quality of a new dive or athletic movement immediately after a user has learned it in a session (e.g., How well did they execute this jump compared to their previous best?).

  2. Maintain high accuracy when scoring actions across different competition venues or skill progression levels (e.g., ensuring the system correctly scores a beginner's technique vs. an expert's).

  3. Adapt efficiently in real-time environments (online setting) where the underlying distribution of actions changes rapidly, such as monitoring a user’s performance during rehabilitation exercises where their execution quality evolves over time.

  4. Provide reliable and stable performance even when the model is continually updated with new task data, ensuring that past evaluations remain accurate and not corrupted by recent learning.

Sources

Related papers