2608.07385-Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations".

Jane: The paper was written by Ioannis N. Ziogas, Ensieh Khazaei, Bilal Taha, Aamna Al Shehhi, Ahsan H. Khandoker et al. from Khalifa University and University of Toronto and MIT Media Lab and Aristotle University of Thessaloniki.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Jane: Today's paper comes from a collaboration across four institutions — Khalifa University, the University of Toronto, MIT Media Lab, and Aristotle University of Thessaloniki. The authors are Ziogas, Khazaei, Taha, Al Shehhi, Khandoker, Hadjileontiadis, and Hatzinakos. It's a serious group, and the title they chose is a mouthful.

Tom: Every word in that title carries weight, honestly. Omni-modal, decomposition autoencoders, full-stack, disentangled — put together, it promises that one model can do everything a wearable sensing system needs.

Jane: "Omni-modal" is the word I want to sit with. We usually say "multi-modal" when there are two or three sensor types involved. These authors mean something bigger — arbitrarily many data streams, unified in a single model.

Lu: They actually test that ambition with thirty channels of data at once. That's inertial sensors, physiological signals, and audio — six audio components, three physiological streams, and twenty-one inertial channels. It's the kind of information flood that a modern smartwatch and phone can produce together.

Meng: That flood connects to something I see in the author list. The disciplinary mix is striking — biomedical engineering researchers like Khandoker and Hadjileontiadis working directly with signal processing people like Hatzinakos. The funding comes from Khalifa University's Healthcare Engineering Innovation Group, which fits that clinical angle.

Tom: Exactly. "Full-stack" is their way of saying the model should handle classification, fusion, disentangled representation learning, and generation in one place, instead of four separate systems each trained for a single job.

Jane: Those goals usually pull against each other. A model trained to classify activities tends to produce representations that are hard to interpret or generate from, and generative models often don't classify well. So the paper is making a strong claim.

Lalam: It is a strong claim, and it's worth reading precisely because of that. Getting one architecture to hold all of those abilities without collapsing into a jumble is a genuine open problem in representation learning. And whether they succeed matters beyond this one architecture, because wearable sensing is exactly where the data complexity is growing fastest.

Jane: Then that structure is exactly what we should look at next.

Summary: Tom: We've sized up the team and the ambition behind this paper, so let's get concrete about the architecture. On the surface it's simple: one shared convolutional encoder, one shared decoder, and no modality-specific branches. The encoder is a seven-layer convolutional network modeled on Wav2Vec2, while the decoder is a four-layer fully connected network.

Jane: Everything flows through the same pipeline. Each input becomes a time-frequency representation, and they build an anchor signal by literally summing all the modality channels together before the transform. For mixed acoustic and inertial signals, that anchor becomes a Mel spectrogram — the paper describes it as a sonified representation.

Lu: The thirty channels break down into six audio components from a filter decomposition, three physiological signals from a smartwatch, and twenty-one channels from seven tri-axial inertial sensors. Then the self-supervised loss takes over, and it does the real work. One term pulls each modality's latent representation toward the anchor, enforcing that all sensors describe the same underlying event, while a second term pushes modalities apart so they don't become redundant copies.

Meng: So alignment and separation happen at the same time. Each modality keeps its own subspace in the latent space, but everything is anchored to a shared context. That's how activity and identity get structured as separate factors, rather than tangled together.

Tom: On the HARWE dataset — thirty-five participants, nine daily activities, all thirty channels — the best variant reaches 84 point 56 percent accuracy for activity recognition and 88 point 97 percent for identity recognition in the subject-dependent setting. Those numbers beat both the transformer-based fusion baselines and the VAE-based alternatives.

Jane: For me the generative results are even more striking. The reconstruction mean absolute error improves by more than 76 percent relative to a multi-modal VAE baseline, and the distributional distance between real and synthetic data improves by about 14 percent. So the same latent space that recognizes activities can also synthesize believable sensor signals.

Lu: That's the full-stack claim made concrete. And the model stays at 4 point 1 million parameters whether you give it three channels or thirty, because the architecture itself doesn't grow with the modality count.

Meng: The subject-independent setting deserves a mention too, because there the results are more mixed. The multi-branch VAE baseline reaches 70 point 91 percent for activity recognition, while OmniDecVAE gets 64 point 56 percent. That suggests shared encoders may generalize a bit less to people never seen in training.

Jane: Still, the overall package is unusual — classification, generation, disentanglement, and a constant memory footprint in a single model. That's a real step beyond the typical task-specific wearable system.

Tom: Which raises the natural next question: what does that combination of abilities actually buy you in practice?

Improvements: Jane: We've seen what the model achieves on benchmarks, so let's think about the practical improvements it suggests. The first is architectural: because the encoder is shared and the modality structure lives in the loss, the parameter count stays flat at 4 point 1 million no matter how many sensors arrive. There are no per-modality branches to multiply.

Tom: That flat footprint is genuinely rare. Their multi-modal VAE comparison grows to 88 million parameters once you add all thirty channels, since every modality gets a dedicated branch. OmniDecVAEs avoid that scaling problem entirely, and that changes what you can deploy.

Lu: The paper puts real numbers on deployment too — about four milliseconds of inference latency per sample and roughly 15 megabytes on disk. That fits comfortably inside the real-time window for edge devices like smartwatches and clinical monitors, which is where wearable eye has to live.

Meng: On the generative side, the improvements open up applications that classification-only systems can't touch. You could reconstruct a sensor that failed mid-recording, synthesize extra training data for other models, or reduce the number of physical sensors by generating the modalities you stopped collecting. All of that comes from the same learned representation.

Jane: The privacy angle is the one that pulls me in. Because identity is disentangled into its own subspace, you could anonymize a recording by manipulating that subspace directly while preserving the activity content. In patient-centric healthcare, you could also keep identity available when clinicians genuinely need it — and block it when they don't.

Tom: There's a surprising video result as well. OmniDecVAE never sees video, yet in the subject-independent scenario it reaches 64 point 56 percent activity accuracy, close to the supervised transformer baseline that does use video and gets 68 point 5 percent. Video is normally the most informative modality in human activity recognition, so that gap is remarkable.

Lalam: The paper is also honest about what still needs improving — generating raw time-domain signals instead of time-frequency representations, because that would give downstream processing more freedom, plus handling missing modalities explicitly and moving beyond time series into video and text. Those are natural next steps, and the authors name them directly. That honesty makes the current claims easier to trust.

Jane: So the improvements aren't just incremental accuracy gains. The paper is proposing a different shape for wearable eye, where a single model carries recognition, generation, and privacy controls together. That's a bigger statement than any single benchmark result.

Tom: That framing goes straight back to the opening pages, where the problem is first laid out. Let's look at that framing next.

First page: Tom: We've covered what the model does and what it enables, so let's go back to the opening pages, because the framing there matters. The abstract makes a pointed claim: no existing approach operates as a full-stack wearable processor. None of them simultaneously handle classification, disentangled representation learning, fusion, and generation.

Jane: That's the gap the paper is filling. And it sits on top of a generative story — a latent sensing event, decomposed into modality-specific latent variables, which in turn produce the sensor measurements you observe. That three-step process is the conceptual backbone.

Lu: The clever part is that the data arrives pre-decomposed. You already have separate sensor streams, so instead of learning to decompose a single signal, the model learns to compose — reconstructing the underlying latent event from those separate views. That inversion is what makes the shared encoder viable.

Meng: Composition is a nice way to put it. And that decomposition loss comes with a weighting scheme which deserves more attention than it usually gets. The authors don't treat all sensor pairs equally — audio, being complex and high-dimensional, gets a stronger pull toward the anchor, while closely related channels like BVP and EDA get weaker repulsion in the orthogonality term.

Jane: Those weighting factors — 0 point 25 for the positive interactions and 0 point 4 for the negative ones — encode domain knowledge about which sensors should behave similarly. That's how the disentangled structure ends up matching the physical reality of the wearable setup. It's structure with a purpose, not structure for its own sake.

Tom: The contributions listed on that page are worth holding onto. Scalable fusion without transformer-style architectures, multi-modal time-frequency generation through a shared decoder, and a latent space where modality, activity, and identity are clearly structured — with the downstream results to back it up. The paper reports accuracy improvements of 1 point 01 percent in activity recognition and 6 point 75 percent in identity recognition over the comparison methods.

Lu: The clinical motivation is right there on page one as well. Hospitals upgrade equipment constantly, and a modality-invariant model can adapt to sensor replacements without full retraining. Combined with identity isolation for patient privacy and generation for sensor reduction, that becomes a sustainability story for healthcare eye.

Lalam: So the first page is really the blueprint for the whole paper — problem statement, generative assumption, method outline, contributions, and the healthcare motivation tying it together. Everything in the experiments follows from that opening framing. That's good scientific writing, honestly.

Jane: Which brings us to the closing question: what does this all amount to as a research direction?

Conclusion: Tom: Let's pull it all together. This paper makes a strong case that wearable eye doesn't need separate systems for recognition, generation, fusion, and privacy. One model with a shared encoder and a carefully structured loss can carry all of those, and the evidence is in the tables.

Jane: The evidence is concrete. The best variant reaches 84 point 56 percent activity accuracy and 88 point 97 percent identity accuracy in the subject-dependent setting. Reconstruction mean absolute error improves by 76 point 84 percent, distributional similarity improves by 13 point 85 percent, and the whole model runs at 4 point 1 million parameters with about four milliseconds per sample.

Lu: We should keep the caveat we discussed earlier in view. In the subject-independent setting, the multi-branch baseline edges it out on activity recognition — 70 point 91 percent versus 64 point 56 percent. So the shared-encoder design has a trade-off, and the authors acknowledge it directly.

Meng: On the other side of that caveat, they also lay out clear next steps. Raw time-domain generation, cross-modal inference with missing sensors, and expanding beyond time series are all flagged as future work. Those don't take away from the current results — they show where the framework needs to go.

Lalam: The bigger picture, from where I sit, is that structure can live in the learning objective instead of the architecture. You keep the network simple and push the domain knowledge into the loss, and that's an idea that transfers well beyond wearables. Representation learning papers often point this direction, but few demonstrate it this cleanly on real physiological and inertial data.

Jane: And the practical implications stack up — biometric security, anonymization through the identity subspace, sensor reduction, synthetic signal generation for healthcare, all in a lightweight edge-ready package. Those are exactly the problems next-generation wearable systems need to solve.

Tom: That's a lot of value from one framework, and a good note to end on. Thanks to the authors, and to everyone listening. We're ready to take the next paper from the pile.

Jane: Until next time — keep listening, and keep asking what your models are actually learning. That question is what drives work like this.