SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

arXiv:2610.11361 · cs.SD, cs.CV · Submitted 2026-10-08 · Read on arXiv

cs.SD, cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Project page: https://sepgen.github.io

Terminology

• Benchmark — 53× in this paper · explained in 2 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• Cross-Stem Attention Guidance — 6× in this paper
• Ablation — 5× in this paper · explained in 1 episode
• Diffusion Model — 5× in this paper · explained in 8 episodes
• Diffusion Models — 5× in this paper · explained in 6 episodes
• Embeddings — 4× in this paper · explained in 2 episodes
• Alignment — 3× in this paper · explained in 1 episode
• Conference on Computer Vision and Pattern Recognition — 3× in this paper
• Flow Matching — 3× in this paper · explained in 7 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Fidelity — 2× in this paper · explained in 2 episodes
• International Conference on Computer Vision — 2× in this paper
• Two-stage training — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• LoRA Adapters — 1× in this paper · explained in 3 episodes
• Personalization — 1× in this paper · explained in 2 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Rectified Flow — 1× in this paper · explained in 3 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• LibriMix: An Open-Source Dataset for Generalizable Speech Separation
• CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• LTX-2: Efficient Joint Audio-Visual Foundation Model
• LoRA: Low-Rank Adaptation of Large Language Models
• In-Context LoRA for Diffusion Transformers
• DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
• Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
• Separate What You Describe: Language-Queried Audio Source Separation
• Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
• Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
• Movie Gen: A Cast of Media Foundation Models
• HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
• SAM Audio: Segment Anything in Audio
• Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation
• Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
• A Reference-free Metric for Language-Queried Audio Source Separation using Contrastive Language-Audio Pretraining
• Multi-Source Music Generation with Latent Diffusion
• Cinematic Audio Source Separation Using Visual Cues

Related papers

• Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
• Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
• AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
• SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
• WASIL: In-the-Wild Arabic Spoken Interactions with LLMs
• Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment