DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generation
cs.SD, cs.AI
Submitted: 2025-11-25
Updated: 2026-09-13
Comments: Accepted at ACM Multimedia 2026 (MM '26). Updated to the accepted version, with author-version formatting. 9 pages, 4 figures, 6 tables. Project page: https://eps-acoustic-revolution-lab.github.io/DUO_TOK/
Project page: https://eps-acoustic-revolution-lab.github.io/DUO_TOK
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Multi-track music generation requires tokens that preserve acoustic fidelity, support sequence modeling, and maintain cross-track structure.
Terminology
Abstract
Multi-track music generation requires tokens that preserve acoustic fidelity, support sequence modeling, and maintain cross-track structure. Reconstruction-oriented codecs retain acoustic detail but are difficult to model, while semantic tokenizers may sacrifice fidelity or cross-track alignment. We present DuoTok, a source-aware dual-track music tokenizer for vocal-accompaniment generation based on staged disentanglement. DuoTok first learns a semantic audio representation through self-supervised pretraining, then shapes source-aware structure using feature replacement noise and multi-task supervision: spectral reconstruction, music source separation regularization, and an ASR head for lyric alignment. It freezes the encoder and learns hard-routed codebooks for vocals and accompaniment, while a diffusion decoder restores fine acoustic detail from discrete tokens. Across public benchmarks, DuoTok achieves a favorable predictability-fidelity trade-off at ultra-low bitrate. Under held-constant dual-track language modeling, it improves both unconditional vocal-accompaniment modeling and vocal-conditioned accompaniment prediction. Controlled diagnostics show larger predictability costs under cross-track corruption and greater gains from longer temporal context, supporting stronger use of cross-track and temporal structure rather than merely easier local prediction. DuoTok also maintains competitive reconstruction quality and preserves control-relevant musical attributes in its discrete space. These results suggest that tokenizer design is a core modeling problem for multi-track music generation, beyond compression alone.
Sources
- MusicLM: Generating Music From Text
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Optimized Self-supervised Training with BEST-RQ for Speech Recognition
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- DISCO-10M: A Large-Scale Music Dataset
- FMA: A Dataset For Music Analysis
- Music Source Separation in the Waveform Domain
- LeVo: High-Quality Song Generation with Multi-Preference Alignment
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- High Fidelity Neural Audio Compression
- SongCreator: Lyrics-based Universal Song Generation
- Moshi: a speech-text foundation model for real-time dialogue
- SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
- Stable Audio Open
- Music Source Separation with Band-Split RoPE Transformer
- Scalable Diffusion Models with Transformers
- Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
- WASIL: In-the-Wild Arabic Spoken Interactions with LLMs
- Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment