Autonomous Telerehabilitation via Skeletal Motion Prediction and Joint-Level Performance Assessment

This paper presents a telerehabilitation pipeline that integrates skeleton-based exercise quality assessment and short-term motion prediction into a two-module system operating on marker-free RGB video. A self-attentive Bidirectional LSTM performs exercise quality classification using MMD-NCA metric learning, while a graph-based motion prediction module computes per-joint position errors between predicted and observed poses, generating spatially localized deviation signals. Each module is evaluated independently on established benchmarks: the classifier achieves 96.45% mean-class accuracy on squat sequences from the PROZIS dataset, and the adopted STARS predictor achieves a mean MPJPE of 75.8 mm at 560 ms on Human3.6M, outperforming graph and recurrent baselines across all prediction horizons. The framework is designed for eventual deployment in assistive robotics and home-based rehabilitation contexts; end-to-end integration and clinical validation are important directions for future work. By combining motion recognition and prediction in a single system, this work contributes a step toward autonomous, feedback-driven telerehabilitation, for more accessible and scalable rehabilitation solutions.

The proposed approach consists of three main modules. An off-the-shelf human pose estimator (MediaPipe or OpenPose) converts an input RGB sequence into a time series of 3D skeleton joint coordinates. The skeleton is normalized relative to the hip joint at each frame, removing absolute position and making the representation invariant to subject placement within the camera frame. Variable-length repetition sequences are either subsampled to a maximum length or zero-padded to a minimum length before being passed to the two processing modules. Interpolation was also evaluated but yielded marginally inferior results (≤2 percentage points in mean-class accuracy) compared to zero-padding.

The normalized skeleton sequence is processed by two parallel modules. The exercise quality classifier operates on full-repetition sequences and produces a binary holistic quality label (correct or incorrect) for each repetition. The motion predictor uses a short context window of 10 frames (400 ms) and predicts the next 5–25 frames (200–1000 ms). The predicted joint positions are compared to the observed ground truth using a per-joint Euclidean error, generating a spatially localized error map. This map is thresholded into four color categories: green (low error), yellow (moderate), orange (elevated), and red (high deviation), and rendered as an overlay on a 3D skeleton visualization. Joints irrelevant to the exercise under assessment (e.g., arms during a gait task) are masked to focus feedback on clinically relevant body parts.

A key design requirement of the proposed system is robustness to variations in exercise tempo, which are common in real-world rehabilitation scenarios, especially among older adults and users with limited mobility. To ensure consistent performance across different movement speeds, the BiLSTM module uses a self-attention mechanism that aggregates variable-length sequences into a fixed-length representation, enabling tempo-invariant assessment. In parallel, the graph-based prediction models use trainable temporal adjacency matrices to implicitly learn speed-adaptive motion patterns directly from data, eliminating the need for explicit temporal normalization or dynamic time warping.

For exercise quality classification, the paper adopts Coskun et al.'s approach using a BiLSTM with a self-attention mechanism. The core building block is a layer-normalized LSTM (LNLSTM) that performs forward and backward passes over the input pose sequence. Layer normalization replaces batch normalization inside the LSTM cell, which is known to degrade performance in standard LSTM implementations. The forward and backward LNLSTM outputs are concatenated at each timestep. A self-attention mechanism assigns a scalar importance weight to each timestep, prioritizing the most relevant postures. The attended, fixed-length sequence embedding is then passed to a classification head consisting of multiple fully connected layers with dropout, batch normalization, and ReLU activations.

The architecture uses a variation of the NCA, based on the Maximum Mean Discrepancy (MMD). The MMD-NCA loss operates at the distributional level, measuring divergence between class embedding distributions via the MMD. This function effectively reduces the overlap between category distributions in the embedding space while ensuring that samples from the same distribution remain as close as possible. The model is trained with a batch size of 32 using the Adagrad optimizer and an initial learning rate of 10−3. A learning rate scheduler is applied, reducing the rate by a factor of 0.8 with a patience of 7 epochs across approximately 1000 training sequences. Binary cross-entropy serves as the classification objective alongside the MMD-NCA embedding loss.

For short-term motion prediction, two graph-based networks were chosen. STS-GCN encodes the full observation as a spatio-temporal graph, factorizing the full spatio-temporal adjacency matrix into separate spatial and temporal components. This factorization limits joint-time cross-talk, reducing the number of learnable parameters while preserving the expressiveness of the full space-time representation. The STARS framework extends the STS-GCN backbone with a generative model that decomposes the latent code into a stochastic component and a set of K learnable deterministic anchors. Anchors are further factorized into spatial anchors (controlling movement direction and posture) and temporal anchors (controlling frequency and speed), yielding compositional predictions. The backbone operates in the DCT frequency domain using 8 STS-GCN layers with alternating channel dimensions. Three loss terms are combined during training: a reconstruction loss, a diversity-enhancing loss, and motion constraint losses.

To generate spatially and temporally localized feedback, the paper computes a per-joint, per-frame position error (JPE). By omitting the averaging over both joints and time present in standard MPJPE, the system isolates the specific error for each articulation at every individual time-step. This yields a scalar error for each joint in each predicted frame, which is then mapped to four color thresholds for visualization.

The evaluation uses multiple datasets. Human3.6M is a large-scale 3D human motion capture dataset recorded at 50 Hz with a Vicon system, comprising 7 subjects performing 15 everyday actions. A 22-joint subset is used. Two parameterizations are evaluated: 3D Cartesian coordinates and 3D Euler angles converted to exponential map format. The PROZIS Challenge dataset is a proprietary collection of annotated fitness exercise videos covering five exercise types: squats, sit-ups, push-ups, jumping jacks, and burpees, performed by participants with varying fitness levels. Each repetition is labeled as correctly or incorrectly executed. The squat class has the largest sample count (approximately 950 sequences), while other exercise types range from about 70 to 300 sequences. The CMU Graphics Lab Motion Capture Database is used to benchmark the action recognition and metric learning baselines.

For the exercise quality classification on the PROZIS dataset, the model achieves 96.45% mean-class accuracy for the squat class, with training and validation loss curves converging smoothly before epoch 200 and no signs of overfitting. The self-attention mechanism focuses on the most discriminative phases of the repetition, particularly the lowest point of the squat descent. For the remaining exercises (sit-ups, push-ups, and jumping jacks), the sample counts varied significantly, ranging from 70 to 300 sequences. Sit-ups showed mild underfitting and unstable training curves, while push-ups and jumping jacks failed to converge meaningfully, with validation accuracy collapsing to the class prior. These failures are due to data scarcity and class imbalance rather than architectural limitations. Empirically, reliable convergence appears to require at least several hundred balanced sequences per exercise class. Inference per repetition takes approximately 20ms after extraction.

For motion prediction, STARS consistently outperforms all baselines at every prediction horizon. For short-term prediction, STARS achieves 56.9 mm compared to 65.8 mm for STS-GCN, an improvement of 8.9 mm. For long-term prediction, STARS achieves 108.4 mm compared to 117.0 mm. Notably, STARS also outperforms LTD-50-25, which uses 50 context frames, while the proposed method uses only 10 frames, demonstrating the advantage of the anchor-based generative formulation for long-horizon prediction. The choice of input parameterization significantly affects prediction accuracy: using raw 3D Cartesian coordinates instead of the exponential map representation reduces short-term accuracy by approximately 23% and long-term accuracy by approximately 34%. For the rehabilitation application, where prediction horizons are intentionally short and feedback resolution is at the joint rather than the sub-centimeter level, the Cartesian representation is a pragmatically acceptable trade-off.

The per-joint error signal was qualitatively evaluated on a walking sequence from Human3.6M, using STS-GCN with 3D Cartesian input and a 25-frame prediction horizon. During correctly performed linear walking, leg joints remained mostly in the green and yellow error bands for the first 15 frames, with error accumulating in the final frames as expected due to the open-loop nature of the prediction. Robustness to execution speed was also evaluated by subsampling the same walking sequence to 50% and 25% of its original frame rate. The predictor adapted its output to the altered temporal scale without retraining or explicit dynamic time warping. Context quality is essential: sequences that begin with a static pose result in significantly higher joint errors in the initial frames, because the predictor lacks an active motion signal to condition on and defaults to a near-stationary prediction. This motivates buffering at least 10 frames (400 ms) of active movement before activating the feedback overlay. End-to-end inference time from preprocessed skeleton input to rendered 3D overlay is 2–5 s in Python on an RTX 2060, making the system suitable for per-repetition rather than per-frame feedback.

The paper identifies four limitations. First, the two modules are trained and evaluated on separate datasets, and no end-to-end evaluation of the complete recognition–prediction–feedback loop has been conducted. This is in part a current data constraint: PROZIS lacks the kinematic-tree information required by the graph-based predictor, and Human3.6M contains no rehabilitation exercises or quality annotations. Second, the joint-level error signal is derived from motion-prediction error rather than therapist-annotated deviation ground truth. Third, end-to-end inference takes 2–5 seconds, restricting the system to per-repetition rather than frame-level or continuous feedback. Fourth, the study establishes the feasibility of the system by validating its individual components—a necessary first step before testing the complete setup with users, therapists, and robotic hardware. Future work will focus on end-to-end validation for rehabilitation exercises, improving anatomical accuracy by integrating kinematic constraints, validating joint-level feedback against physiotherapist-annotated data, and prospective evaluation with users and clinicians in a human-robot interaction setting.

Improvements to AI Systems:

1. Tempo-Invariant Motion Assessment: Implement a self-attentive Bidirectional LSTM with MMD-NCA metric learning to classify exercise quality regardless of movement speed, eliminating the need for explicit temporal normalization or dynamic time warping. This enables robust evaluation of rehabilitation exercises performed at varying paces, particularly for older adults or users with limited mobility.

2. Spatially Localized Feedback via Graph-Based Prediction: Use a graph-based motion predictor (e.g., STARS with factorized spatial and temporal anchors) to compute per-joint, per-frame position errors between predicted and observed poses. This generates a color-coded 3D overlay (green/yellow/orange/red) that pinpoints exactly which joints deviate from expected motion, allowing targeted feedback on clinically relevant body parts while masking irrelevant ones.

3. Adaptive Short-Horizon Prediction Without Retraining: Deploy a predictor with trainable temporal adjacency matrices that implicitly learns speed-adaptive motion patterns. This allows the system to handle altered execution speeds (e.g., 50% or 25% slower) without retraining or dynamic time warping, ensuring consistent feedback across real-world tempo variations.

4. Data-Efficient Quality Classification: Train the classifier with a combination of binary cross-entropy and MMD-NCA embedding loss to reduce overlap between class distributions. This improves convergence with limited data, but requires at least several hundred balanced sequences per exercise class to avoid underfitting or collapse to class priors.

5. Context-Aware Feedback Activation: Buffer at least 10 frames (400 ms) of active movement before activating the feedback overlay, as static initial poses lead to high prediction errors. This prevents misleading feedback during the start of a motion sequence.

6. Compositional Motion Prediction: Use anchor-based generative models (STARS) that decompose latent codes into spatial anchors (controlling direction/posture) and temporal anchors (controlling frequency/speed). This improves long-horizon prediction accuracy (e.g., 108.4 mm vs. 117.0 mm at 1000 ms) while using fewer context frames (10 vs. 50), making it suitable for real-time per-repetition feedback.

What the Improved AI System Can Do:

• Automatically assess the correctness of each exercise repetition (e.g., squat, sit-up) from a single RGB video, with 96.45% mean-class accuracy on squat sequences, and provide a binary correct/incorrect label.

• Predict future joint positions 200–1000 ms ahead and generate a real-time, spatially localized error map, highlighting specific joints (e.g., knee, hip) that deviate from expected motion, with color-coded severity levels.

• Operate robustly across different exercise speeds and user mobility levels without manual recalibration, making it suitable for home-based rehabilitation.

• Provide per-repetition feedback (2–5 s inference time) rather than continuous frame-level feedback, enabling interactive coaching sessions with a human or robotic assistant.

• Mask irrelevant joints (e.g., arms during gait) to focus user attention on clinically meaningful deviations, reducing cognitive load and improving rehabilitation outcomes.

• Serve as a foundation for autonomous, feedback-driven telerehabilitation systems, with future integration into assistive robotics for real-time guidance and correction.