LoSATok: Low-Dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation

arXiv:2605.27840 · eess.AS, cs.AI, cs.SD · Submitted 2026-05-27 · Read on arXiv

eess.AS, cs.AI, cs.SD

Submitted: 2026-05-27

Updated: 2026-09-12

Comments: Accepted to EMNLP 2026 Findings

Code: https://github.com/wxzyd123/LoSATok

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Alignment — 8× in this paper · explained in 1 episode
• Latent Space — 8× in this paper · explained in 7 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• Benchmark — 6× in this paper · explained in 2 episodes
• Redundancy — 5× in this paper · explained in 2 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Semantic Alignment — 4× in this paper · explained in 3 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Fidelity — 3× in this paper · explained in 2 episodes
• Generative Model — 3× in this paper · explained in 2 episodes
• Ablation study — 2× in this paper · explained in 1 episode
• Adversarial Training — 2× in this paper · explained in 1 episode
• Distillation — 2× in this paper · explained in 2 episodes
• Generative Modeling — 2× in this paper · explained in 3 episodes
• Generative Models — 2× in this paper · explained in 2 episodes
• Latent Representation — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Adaptive Weighting — 1× in this paper · explained in 1 episode
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Flow Matching — 1× in this paper · explained in 8 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
• Linear Probing — 1× in this paper · explained in 2 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Principal Component Analysis — 1× in this paper
• Principal Component Analysis (PCA) — 1× in this paper · explained in 4 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Semantic Understanding — 1× in this paper · explained in 2 episodes
• Short-Time Objective Intelligibility — 1× in this paper
• Twelfth International Conference on Learning Representations — 1× in this paper
• Verification — 1× in this paper · explained in 2 episodes
• Word Error Rate — 1× in this paper

Sources

• On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation
• MiDashengLM: Efficient Audio Understanding with General Audio Captions
• DashengTokenizer: One layer is enough for unified audio understanding and generation
• LP-MusicCaps: LLM-Based Pseudo Music Captioning
• MusicLM: Generating Music From Text
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs
• SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents
• UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities
• Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis
• Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation
• WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
• Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions