CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

CardioState-JEPA is a cardiac foundation model that learns a single shared representation jointly across electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG), built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time.

The paper states: "We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance."

The model is motivated by the observation that ECG, PPG, and PCG provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. The paper notes that "Electrical activation initiates contraction, valve motion then produces the heart sounds, and blood ejection appears later as a peripheral pulse, so the three modalities observe the same beat at different physiological times."

The architecture consists of modality-specific tokenizers that map each signal into a common token space, followed by a single shared Transformer encoder. The paper explains: "Given xm ∈ R Cm ×Tm, the tokenizer produces hm = fm (xm) ∈ R N ×d: a strided convolution whose stride is set per modality so all three emit tokens at a comparable rate, a multi-scale depthwise block that adds context around short events such as the QRS complex, PPG upstroke, and first heart sound, and a linear projection, followed by sinusoidal positional and modality embeddings. A single shared Transformer encoder then processes these tokens with no modality-specific layers, so ECG, PPG, and PCG update the same parameters."

The training uses a two-stage curriculum. Stage I learns within-modality structure from abundant unimodal data through intra-modal masked latent prediction. Stage II uses paired recordings to align modalities through delay-aware cross-modal prediction. The paper describes: Stage I learns each modality's structure from abundant unimodal data, and Stage II aligns modalities from scarce paired data through a learned delay.

For the intra-modal objective, the paper states: "We predict masked regions in latent space rather than reconstructing the waveform, because reconstruction rewards fitting sensor noise and fine morphology, exactly the nuisance we want the code to forget. Following the joint-embedding predictive principle, we mask a large fraction of the tokens in contiguous blocks, encode the visible context, and let a light predictor q theta match the momentum encoder's code at the masked positions."

For the cross-modal objective, the paper explains: "Aligning tokens at equal timestamps would match an electrical event to a mechanical or hemodynamic one and teach surface appearance rather than shared state, so we predict through a learned delay. A delay head h delta maps the source code zim and a target-modality embedding en to a bounded per-token offset. The target is gathered with a Gaussian kernel centered at the shifted time, and the delay is supervised with physiological anchors: beat detection gives a reference offset per pair, the R-peak to first-heart-sound interval for ECG–PCG and the pulse arrival time for ECG–PPG."

The full objective combines several terms: L = lambdaintra Lintra + lambdacross Lcross + lambdadelay Ldelay-sup + lambdastate Lstate + lambdaphase Lphase. The paper notes: In Stage I, each sample is a single modality, so only intra-modal prediction and the phase term are active; the cross-modal, delay, and state terms require paired inputs and switch on in Stage II.

The pretraining data includes MIMIC-IV-ECG with 800K 12-lead ECG recordings at 500 Hz, PPG-EXT as a large-scale PPG corpus at 125 Hz, and BMD-HS for PCG recordings at 4000 Hz. For cross-modal supervision, the paper uses PPG-EXT and VitalDB for synchronous ECG-PPG pairs, EPHNOGRAM for ECG-PCG pairs, and SensSmartTech for trimodal recordings.

The model is evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG. The paper reports: "our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks."

Specifically, on PPG tasks, CardioState-JEPA improves the average classification AUROC from 72.2 to 80.4 compared with the strongest baseline, while also reducing the average regression MAE from 10.9 to 9.1. For ECG, CardioState-JEPA improves the average AUROC across all 18 settings by 15.5 points over the strongest self-supervised baseline (MoCo-v3; 84.1 vs. 68.5). For PCG, the model achieves the best results on both CirCor murmur detection and CinC2016 abnormal heart-sound detection under the linear probing protocol.

The paper also visualizes the shared space using t-SNE: "after Stage I the samples form clear modality-specific clusters, since each modality has only been trained in isolation. After Stage II the same samples become thoroughly interleaved across modalities, indicating that cross-modal training reshapes the space from a sensor-driven layout into a modality-invariant one. The modality silhouette falls from 0.121 after Stage I to −0.006 after Stage II."

Ablation studies show that the full trimodal setting gives the best overall balance, masked latent prediction gives the strongest results compared to SimCLR, BYOL, BarlowTwins, and MAE, and that cross-modal prediction, state alignment, delay modeling, and phase supervision provide complementary benefits.

The paper concludes: "We introduced CardioState-JEPA, a physiology-aware joint embedding predictive framework that learns from ECG, PPG, and PCG through intra-modal latent prediction and delay-aware cross-modal alignment. By predicting latent cardiac states and aligning modalities at the corresponding cardiac time, CardioState-JEPA enables electrical, hemodynamic, and acoustic signals to supervise one another. Our experiments show that the unified model transfers across modalities and surpasses strong self-supervised signal baselines on representative tasks."

Improvements to AI Systems:

1. Cross-Modal Latent State Prediction with Delay-Aware Alignment

• Implement a joint-embedding predictive architecture that maps heterogeneous sensor inputs (e.g., ECG, PPG, PCG) into a shared token space, then predicts masked latent cardiac states rather than raw waveforms.

• Use a learned delay aligner to match tokens at corresponding physiological times (e.g., R-peak to pulse arrival), avoiding naive timestamp alignment that would mix electrical, mechanical, and hemodynamic events.

• Resulting capability: An AI system that can fuse asynchronous multi-sensor data (e.g., wearable ECG + PPG + microphone) into a single physiology-aware representation, enabling robust cardiac monitoring even when sensors are not perfectly synchronized.

2. Two-Stage Curriculum Learning for Scarce Paired Data

• First pretrain on abundant unimodal data using intra-modal masked latent prediction (Stage I), then fine-tune on limited paired multi-modal data using delay-aware cross-modal prediction (Stage II).

• This reduces reliance on expensive synchronized recordings while still achieving modality-invariant representations.

• Resulting capability: An AI system that can be trained on large-scale single-modality datasets (e.g., millions of ECGs) and then adapted to new modalities (e.g., PPG or PCG) with only a small paired dataset, dramatically lowering data collection costs for multi-modal health monitoring.

3. Physiology-Anchored Delay Supervision

• Supervise cross-modal delay predictions using known physiological intervals (e.g., R-peak to first heart sound for ECG-PCG, pulse arrival time for ECG-PPG) derived from beat detection.

• This grounds the model in cardiac physiology rather than arbitrary signal alignment.

• Resulting capability: An AI system that can automatically estimate physiological timing parameters (e.g., pulse transit time, electromechanical delay) from raw waveforms, enabling non-invasive hemodynamic and cardiac function assessment without manual feature engineering.

4. Modality-Invariant Representation via Shared Transformer Encoder

• Use a single shared Transformer encoder with no modality-specific layers, after modality-specific tokenizers that normalize sampling rates and token densities.

• This forces the model to learn a common cardiac state representation across sensors.

• Resulting capability: An AI system that can perform zero-shot or few-shot transfer across modalities—e.g., a model trained on ECG can directly classify PPG or PCG signals, or a system can switch sensors (e.g., from chest patch to wristband) without retraining.

5. Masked Latent Prediction to Ignore Sensor Noise

• Replace waveform reconstruction with latent-space prediction of masked tokens, using a momentum encoder for target codes.

• This discards sensor-specific noise and fine morphology, focusing on shared physiological state.

• Resulting capability: An AI system that is robust to sensor artifacts, baseline wander, and noise, improving performance on real-world wearable data where signal quality varies.

6. Multi-Scale Tokenization for Short Cardiac Events

• Use strided convolutions with per-modality stride and multi-scale depthwise blocks to capture short events (QRS complex, PPG upstroke, S1 heart sound) while maintaining comparable token rates across modalities.

• Resulting capability: An AI system that can accurately detect and classify brief, transient cardiac events (e.g., arrhythmias, murmurs) even when they occur at different temporal resolutions across sensors.

7. Joint State and Phase Supervision

• Incorporate auxiliary losses for cardiac state (e.g., beat phase) and phase alignment, in addition to intra- and cross-modal prediction.

• This provides explicit physiological supervision during pretraining.

• Resulting capability: An AI system that can segment cardiac cycles into phases (systole, diastole) and estimate cardiac timing intervals directly from raw signals, enabling automated echocardiography-like measurements from simple wearables.

8. Frozen Encoder Transfer Across 25 Downstream Tasks

• Evaluate the pretrained encoder without fine-tuning on diverse tasks (classification, regression) across ECG, PPG, and PCG.

• This demonstrates that the learned representation is general-purpose and task-agnostic.

• Resulting capability: An AI system that can be deployed as a universal cardiac feature extractor, enabling rapid development of new diagnostic tools (e.g., detecting hypertension, sleep apnea, or heart failure) by simply adding a linear classifier on top of the frozen encoder, without task-specific pretraining.

What the Improved AI System Can Do:

• Monitor cardiac health continuously from a single wearable device that combines ECG, PPG, and acoustic sensors, fusing them into a unified physiological state representation.

• Detect and classify a wide range of cardiac conditions (arrhythmias, murmurs, abnormal heart sounds) with higher accuracy than single-modality systems, even in noisy, real-world conditions.

• Estimate physiological parameters (pulse transit time, electromechanical delay, cardiac phase) non-invasively and in real-time.

• Transfer learned knowledge across sensor types and data distributions, reducing the need for large annotated multi-modal datasets.

• Provide robust performance on unseen devices or recording conditions due to noise-invariant latent representations.

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.