Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning

arXiv:2507.02915 · cs.SD, cs.AI, cs.LG, eess.AS, eess.SP · Submitted 2025-06-25 · Read on arXiv

cs.SD, cs.AI, cs.LG, eess.AS, eess.SP

Submitted: 2025-06-25

Updated: 2026-09-26

Code: https://github.com/LudovicTuncay/Audio-JEPA

Terminology

• Embeddings — 13× in this paper · explained in 2 episodes
• Latent Representation — 8× in this paper · explained in 2 episodes
• Latent Representations — 8× in this paper · explained in 3 episodes
• Self-Supervised Learning — 8× in this paper · explained in 8 episodes
• Representation Learning — 7× in this paper · explained in 2 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Benchmark — 4× in this paper · explained in 2 episodes
• Adaptation — 3× in this paper · explained in 1 episode
• Building on the Joint-Embedding Predictive Architecture — 2× in this paper
• Data Augmentation — 2× in this paper · explained in 7 episodes
• Exponential Moving Average — 2× in this paper
• Latent Space — 2× in this paper · explained in 7 episodes
• Linear Probing — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Actor — 1× in this paper · explained in 3 episodes
• Affective Computing — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Exponential Moving Average (EMA) — 1× in this paper · explained in 2 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Predictive Coding — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes

Sources

• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• CED: Consistent ensemble distillation for audio tagging
• Scaling up masked audio encoder learning for general audio classification
• Masked Modeling Duo: Towards a Universal Audio Pre-training Framework
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Revisiting Feature Prediction for Learning Visual Representations from Video
• Masked Autoencoders that Listen
• TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems
• GeoJEPA: Towards Eliminating Augmentation- and Sampling Bias in Multimodal Geospatial Learning
• Decoupled Weight Decay Regularization
• Clotho: An Audio Captioning Dataset
• FMA: A Dataset For Music Analysis
• General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline
• FSD50K: An Open Dataset of Human-Labeled Sound Events
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation
• MetaFormer Baselines for Vision

Related papers

• Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
• Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
• AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
• SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
• WASIL: In-the-Wild Arabic Spoken Interactions with LLMs
• Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment