Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations

Published: arXiv:2608.07385v1 [cs.LG], 7 Aug 2026, IEEE Transactions and Journals Template

The paper introduces Omni-modal Variational Decomposition Autoencoders (OmniDecVAEs), a framework for learning multi-purpose representations from arbitrarily many modalities. The authors state: "Learning disentangled representations is a key requirement for developing versatile, general-purpose, and sustainable models in multi-modal wearable computing. However, existing approaches do not operate as full-stack wearable processors, i.e., they do not simultaneously address task-specific classification performance, disentangled and interpretable representation learning, fusion, and generative modeling of highly heterogeneous multi-modal time series. OmniDecVAEs extend DecVAEs by learning modality-conditioned time-frequency latent subspaces through a multi-view self-supervised decomposition loss and a shared asymmetric autoencoder (AE) architecture. On a challenging omni-modal human activity recognition (HAR) setting with up to thirty modalities, the method demonstrates full-stack representation ability: OmniDecVAEs full-stack disentangled representation properties lead to accuracy improvements of 1.01% and 6.75% in activity and identity recognition, respectively, compared to transformer-based and VAE-based methods. Additionally, OmniDecVAEs synthesize realistic omni-modal time-frequency data that manifest with enhanced reconstructions (mean absolute error improves by 76.84%) and distributional similarity between real and synthetic data (maximum mean discrepancy improves by 13.85%). The framework is described as a lightweight model suitable for intelligent edge wearables and clinical healthcare, unifying processing requirements and abilities in a single model, through its enhanced representational capacity, modality-invariant spatial complexity (4.1M parameters), and real-time latency."

The authors motivate the work by noting that "Wearable sensing systems are increasingly generating large-scale, heterogeneous multi-modal data, requiring robust and expressive representation learning for applications in healthcare, human-computer interaction, and biometric security. They describe a transition from conventional multi-modality to omni-modality, defined as the unification of arbitrarily many heterogeneous data streams into universal and expressive representations through a single model."

They argue that existing supervised multi-modal approaches require large amounts of labeled data, often leveraging transformer or state-space models to capture long-range dependencies, but fusing heterogeneous streams such as inertial, physiological, and behavioral data requires significant per-modality design considerations. Self-supervised learning (SSL) has emerged as an alternative, but existing approaches primarily focus on task-specific performance and do not jointly address disentanglement and generative modeling. The key gap: To the best of our knowledge, no existing approach jointly addresses scalable multi-modal fusion, disentanglement, and generation within a single unified framework.

The OmniDecVAE model processes up to thirty modalities in our experiments—including inertial, physiological, and audio signals, all transformed into the time-frequency (TF) domain using the short-time Fourier transform (STFT) and Mel-scale representations for audio. A "shared convolutional encoder learns modality-specific latent subspaces, while a shared fully-connected (FC) decoder establishes sample-level correspondence and reconstructs multi-modal signals. The variational formulation enforces a Gaussian latent structure, enabling stochastic generation of unseen samples. The model is evaluated on the HARWE dataset, outperforming supervised transformer-based fusion methods, VAE-based models, and classical approaches such as Independent Component Analysis (ICA) and Principal Component Analysis (PCA)."

The contributions are: (1) a novel multi-modal SSL objective that enables scalable fusion of arbitrarily many modalities within a structured latent space, using a modality-agnostic architecture; (2) a unified architecture that scales to a large number of modalities through a shared encoder, without relying on transformer-based designs, and enables multi-modal TF data generation through a shared decoder; and (3) a disentangled latent representation where modality, activity, and identity factors are well structured, leading to markedly improved performance in downstream recognition tasks.

The authors review multi-modal fusion in wearable computing, noting that existing fusion methods do not naturally scale to an arbitrary number of modalities without architectural modifications, limiting their applicability in omni-modal wearable settings. In SSL, they observe that in most existing methods, SSL objectives primarily enhance intra-modal representations, while cross-modal fusion is handled through architectural design rather than the learning objective itself. For disentangled representation learning and generation, they note that previous AE-based and GAN-based methods achieve high-quality signal synthesis but often do not evaluate the utility of the learned latent representations for downstream tasks.

The paper defines a multivariate time-series dataset X = x1,..., xN ∈ R T×M, where each observation is generated by a three-step generative process θ, involving a global latent variable z and a set of modality-specific latent variables z1, z2,..., zM that act as components of z. The latent variable z is interpreted as a sensing event, while each zm represents a modality-specific view of that event. The latent components are assumed orthogonal: ⟨zi, zj⟩ = Σ zi zj* = 0 for all i ≠ j, and z = Σ zi. In the multi-modal setting, the dataset is already decomposed into its constituent modalities, so instead of performing decomposition, the objective becomes to learn a composition model that reconstructs how the latent event z is expressed in the input space.

The framework operates as follows: (1) the anchor x0 = Σ xi is constructed; (2) multi-modal observations and the anchor are fed to a shared encoder fϕ to obtain latent representations hi and h0; (3) intermediate representations are mapped to modality-specific latent variables zi ∼ qϕ(zi x, z); (4) the self-supervised decomposition objective enforces alignment of hi with h0 via Llat recon and orthogonality via Lortho using omni-modal weighting; (5) modality-specific subspaces are combined into z = [z0, z1, z2,..., zM]; (6) latent variables are sampled via reparameterization and reconstruct inputs using the conditional decoder pθ(x z, c); (7) the model is trained using the conditional DELBO objective.

The SSLDec loss extends DecVAEs to the multi-modal setting. The latent reconstruction term is:

Llat recon = Σi wi pos D KL(D JS(hi, h0) ∥ p)

which minimizes the divergence between each modality-specific representation hi and an anchor representation h0, encouraging alignment across modalities. The orthogonality term is:

Lortho = Σi Σⱼ w(i,j) neg D KL(D JS(hi, hⱼ) ∥ n)

which maximizes the divergence between all pairs of modality representations, promoting separation and reducing redundancy. The use of KLD formulates this adversarial objective into stable cross-entropy terms and mitigates representation collapse. The anchor is defined in input space as x0 = Σi xi.

To handle heterogeneous relationships between channels and modalities, the authors extend SSLDec with an asymmetric contrastive formulation using omni-modal vector VOM pos and matrix WOM neg, with imbalance factors IFpos = 0.25 and IFneg = 0.4. Since the anchor representation is constructed as the sum of all input signals, modalities with higher signal complexity or dimensionality, such as audio, tend to exert a stronger influence on the anchor, so stronger weights are assigned to such anchor–modality interactions. For negative interactions, "smaller weights are assigned to pairs of channels that are expected to be similar, such as physiological signals (e.g., BVP, EDA, temperature), corresponding axes of inertial sensors, and channels originating from the same sensor, while dissimilar modalities are assigned stronger repulsive weights. This formulation can scale to an arbitrarily large number of channels and modalities... without requiring modifications to the underlying model architecture."

The encoder-only objective is:

LDELBOEnc = Llat recon − Lortho − βLprior + const.

The encoder-decoder objective is:

LDELBOEnc−Dec = E[log pθ(xnzn)] − Llat recon − Lortho − βLprior

where β follows the β-VAE formulation.

The architecture uses a seven-layer one-dimensional CNN based on Wav2Vec2, equipped with LayerNorm normalization and GELU activations, followed by a shared FC projection head mapping into a latent interaction space where SSLDec is applied. Modality-specific latent subspaces Zi are obtained via M + 1 FC projection layers that output the mean µ and variance logvar parameters of Gaussian distributions. An aggregation function concatenates subspaces into the final disentangled representation. "A key property of OmniDecVAE is that multi-modality is handled entirely through the SSLDec objective, allowing the network architecture to remain simple and modality-agnostic. No modality-specific encoder branches are required."

The decoder is a shared four-layer FC network with LayerNorm normalization and GELU activations, with no non-linearity at the output layer. The anchor latent z0 is not provided to the decoder. Reparameterization uses a scaling factor τ in the range [0.1, 1.0], where smaller values of τ... lead to more stable training as the number of modalities increases. A conditional embedding c encodes the channel–modality combination, and the decoder is optimized with a smoothed mean absolute error (sMAE) loss.

The HARWE dataset consists of video, audio, inertial and physiological recordings from thirty-five participants performing nine different daily activities in work environments. The video modality is not used. The modalities include: audio sampled at 16kHz decomposed via Filter Decomposition into six frequency band-delimited OCs; physiology including BVP (64Hz), EDA (4Hz), and Temperature (64Hz); and seven inertial measurements (ACC, GYR, GRA, OR, RO, MG, LACC), each tri-axial, totaling 21 inertial channels. The total number of modality channels is 30. Two partitioning schemes are used: Easy (subject-dependent, 80/20 split) and Difficult (subject-independent, 70/30 split).

All physiological and inertial modalities are transformed via a 512-point STFT in the frequency domain with a 0.75 s Hamming window, with an overlap of 75%, resulting in a spectrogram of size 64 × 3 that is flattened to a final size of 192 × 1. Audio uses a Mel spectrogram with 64 Mel scales and 3 hops. The anchor signal is computed by superposing all modalities in the time domain before taking a TF-domain transform... we take the Mel transform for the anchor, resulting in a cumulative 'sonified' Mel representation. Models are trained on a single NVIDIA RTX 6000 Ada with PyTorch, using Adam, for approximately 120 epochs. The learning scheme consists of pre-training with one of Eqs. (14), (15), or (17), followed by post-training with a simple SVM classifier as a classification oracle.

In the subject-dependent scenario, the β-OmniDecVAEEnc−Dec variant provides the best performance in terms of HAR and IR, achieving 84.56% HAR accuracy and 88.97% IR accuracy, closely followed by VAE-based methods. Supervised transformer-based methods have subpar performance and manifest with a much higher sensitivity to the random seed. In the subject-independent scenario, the β-MMVAE variant outperforms other alternatives, signifying that the multi-branch architecture is slightly more adaptable to unseen subjects compared to the proposed OmniDecVae.

TSNE visualizations (Fig. 3) show that "β-OmniDecVAE achieves very clear separation between different modality modes... in contrast to β-MMVAE, attributed to the LDELBOEnc−Dec objective... this disentangled modality structured latent space, renders the representation more informative to the downstream tasks of HAR and IR."

OmniDecVAEs are characterized by enhanced reconstruction quality as evident by the MSE and MAE metrics. The β-OmniDecVAE achieves MSE of 0.116 vs. 4.358 for MMVAE, and MAE of 0.236 vs. 1.019 for MMVAE. Additionally, OmniDecVAE models manifest with a higher distributional similarity to real data. Namely, a lower MK-MMD and DivScore suggest that the distribution learned by OmniDecVAE is more 'realistic', closer to real data (MK-MMD 0.181 vs. 0.210; DivScore 4.094 vs. 15.468). Figure 4 visually confirms that OmniDecVAEs provide much more accurate reconstructions w.r.t. the amplitude and position of TF events, whereas they capture better the morphology of narrow-band frequency phenomena such as the decomposed audio events.

Modality ablation (Tables IV, V) shows that "the full-modal scenario with C = 30 channels does not provide the best performance in the SD scenario, but rather the combinations with less modalities. Specifically, combinations that contain the inertial sensors demonstrate superior performance... In the SI scenario though, the full-modal scenario provides the best results overall. Figure 5 shows β-OmniDecVAE demonstrates a more robust behavior, less influenced by the number of input modality channels, compared to β-MMVAE."

Loss component ablation (Table VI) reveals that "utilizing the SSLDec terms alongside a supervised loss, gives the highest performance while maintaining disentanglement between activity and identity. It becomes evident that SSLDec and the decoder are the main drivers of disentanglement."

The β ablation (Fig. 6) shows that for the full model, "higher β values result in a collapse of the disentangled structure. We also see that the absence of the Lprior through β = 0 is not detrimental to the downstream performance, mainly attributed to the disentanglement mechanisms of SSLDec. A different behavior is evident in the decoder-less variant... in the absence of a decoder, a higher β is needed for a disentangled representation."

OmniDecVAE provides the lighter alternative in terms of SoD and number of parameters (15.82MB and 4.11M parameters, versus 336.46MB and 88.09M for MMVAE), while MMVAE presents increasing storage requirements due to the high number of modalities it accommodates as separate branches. Figure 7 shows the spatial complexity of MMVAEs increase with the number of modalities; on the other hand OmniDecVAEs are invariant to the number of modalities for that matter. Although OmniDecVAE requires more FLOPs (85.85 GFLOPs) and higher latency (4.00ms) than compared methods, it still lie[s] within the real-time processing window for edge applications, with a latency of 4ms per sample.

The authors note that "even though our OmniDecVAE approach does not utilize the video modality, one of the most expressive modalities in HAR, it performs on par with a supervised transformer-based method in the HARWE Difficult SI scenario that utilizes video (Acc. of 68.5% in SI HAR). They emphasize that our omni-modal AE networks performed better under SSL pre-training on both HAR and IR, compared to supervised transformer-based alternatives... and VAE-based methods and other benchmarks. The structure-informed objective is fully explainable and transparent, and the complexity-friendly design is exemplified when the number of modalities aggressively scales; architectures with separate branches per modality quickly become unsustainable in real-life edge deployment scenarios."

Future work includes generating synthetic data in the more raw format of time domain, cross-modal inference and missing modality scenarios, and accommodating other types of modalities beyond time series, such as video or text.

The conclusion states: "OmniDecVAEs simultaneously enhance HAR and IR accuracy by 6.75%, and 1.01%, multi-modal data synthesis MAE and MK-MMD by 13.85% and 76.84%, respectively, along with modality-invariant storage requirements of 15MB or 4.1M parameters. These results underscore OmniDecVAEs as a foundational paradigm for next-generation full-stack models in wearable AI."

Based on the paper, OmniDecVAEs enable the following concrete improvements to AI systems:

1. Arbitrary-modal scalability without architectural redesign

• The shared encoder–decoder architecture replaces per-modality branches, so AI systems can ingest new sensor streams (e.g., adding a new physiological or inertial channel) without changing the model structure.

• Spatial complexity remains constant at 4.1M parameters / 15MB regardless of modality count, enabling deployment on edge wearables instead of cloud servers.

2. Disentangled, interpretable latent representations via self-supervised decomposition

• A new SSL objective (SSLDec) explicitly decomposes the latent space into orthogonal modality-specific subspaces plus a shared anchor subspace.

• The improved AI system can separate latent factors for modality, activity, and identity, making representations more informative for downstream tasks:

• Human activity recognition accuracy increases by 1.01% (84.56% on HARWE)

• Identity recognition accuracy increases by 6.75% (88.97%)

• T-SNE visualizations show clear separation between modality modes, so latent structure is more explainable and transparent.

3. Stronger generative modeling of multi-modal time–frequency signals

• The variational decoder enables stochastic generation of synthetic omni-modal data.

• Reconstruction quality improves dramatically compared to MMVAE:

• Mean squared error drops from 4.358 to 0.116

• Mean absolute error improves by 76.84%

• Distributional realism improves:

• Maximum mean discrepancy improves by 13.85%

• Divergence score drops from 15.468 to 4.094

• The system can therefore synthesize realistic multi-modal sensor data for data augmentation, simulation, and privacy-preserving data sharing.

4. Omni-modal weighting for heterogeneous channel relationships

• Asymmetric contrastive weights handle imbalance: high-complexity modalities (e.g., audio) get stronger positive alignment with the anchor; similar channels get weaker negative repulsion; dissimilar channels get stronger repulsion.

• This allows the AI system to manage 30 heterogeneous channels (audio, BVP, EDA, temperature, 21 inertial channels) in one unified model without manual per-channel feature engineering.

5. Stable training via variational decomposition and reparameterization

• Replacing adversarial objectives with KLD-based cross-entropy terms mitigates representation collapse.

• A scaling factor τ in reparameterization stabilizes training as modality count increases.

• Pre-training with SSL + post-training with a simple SVM classifier outperforms supervised transformer-based fusion, reducing dependence on large labeled datasets.

6. Real-time edge inference

• The system achieves 4ms latency per sample and 85.85 GFLOPs, fitting within real-time processing windows for wearable healthcare and biometric security applications.

7. Robustness and generalizability

• Modality ablation shows the model is less sensitive to the number of input channels than multi-branch VAE models.

• Subject-independent evaluation shows it can generalize to unseen individuals, and even without video input it matches a supervised transformer method that uses video, demonstrating robustness to missing modalities.