MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables

arXiv:2605.29859 · eess.AS, cs.CL · Submitted 2026-05-28 · Read on arXiv

eess.AS, cs.CL

Submitted: 2026-05-28

Updated: 2026-09-01

Code: https://github.com/descriptinc/descript-audio-codec

Terminology

• Transformer — 17× in this paper · explained in 2 episodes
• Latent Space — 14× in this paper · explained in 7 episodes
• Quantization — 11× in this paper · explained in 12 episodes
• Embeddings — 5× in this paper · explained in 2 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• Downstream Tasks — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Tokenization — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Activation Function — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Data Augmentation — 1× in this paper · explained in 7 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Two-Stage Training Strategy — 1× in this paper · explained in 3 episodes
• Two-stage training — 1× in this paper · explained in 2 episodes
• Variational Autoencoder (VAE) — 1× in this paper · explained in 3 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes

Sources

• On The Landscape of Spoken Language Models: A Comprehensive Survey
• dMel: Speech Tokenization made Simple
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• Moshi: a speech-text foundation model for real-time dialogue
• Multimodal Latent Language Modeling with Next-Token Diffusion
• Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions