Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy

arXiv:2609.34381 · cs.CV, cs.MM, cs.SD, eess.AS · Submitted 2026-09-28 · Read on arXiv

cs.CV, cs.MM, cs.SD, eess.AS

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/researchmm/MM-Diffusion

Terminology

• Alignment — 66× in this paper · explained in 1 episode
• Diffusion Model — 27× in this paper · explained in 8 episodes
• Diffusion Models — 23× in this paper · explained in 6 episodes
• Transformer — 20× in this paper · explained in 2 episodes
• Conference on Computer Vision and Pattern Recognition — 19× in this paper
• Fidelity — 16× in this paper · explained in 2 episodes
• Adaptation — 15× in this paper · explained in 1 episode
• Benchmark — 14× in this paper · explained in 2 episodes
• Consistency — 12× in this paper · explained in 3 episodes
• Flow Matching — 12× in this paper · explained in 8 episodes
• Generative Model — 7× in this paper · explained in 2 episodes
• Foundation Model — 6× in this paper · explained in 3 episodes
• Personalization — 5× in this paper · explained in 2 episodes
• CVF International Conference on Computer Vision — 4× in this paper
• Foundation Models — 4× in this paper · explained in 7 episodes
• Rectified Flow — 4× in this paper · explained in 3 episodes
• Distillation — 3× in this paper · explained in 2 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• Latent Diffusion Model — 3× in this paper · explained in 2 episodes
• Latent Diffusion Models — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Adam — 2× in this paper · explained in 2 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• Generative Modeling — 2× in this paper · explained in 3 episodes
• Generative Models — 2× in this paper · explained in 2 episodes
• Latent Space — 2× in this paper · explained in 7 episodes
• Semantic Alignment — 2× in this paper · explained in 3 episodes
• Actor — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Frozen model — 1× in this paper · explained in 3 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Guided attention — 1× in this paper · explained in 2 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Latent Representations — 1× in this paper · explained in 3 episodes
• LoRA Adaptation — 1× in this paper · explained in 2 episodes
• Nash Equilibrium — 1× in this paper · explained in 2 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Optical Flow — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes

Sources

• UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing
• DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing
• JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
• Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning
• TrajectoryMover: Generative Movement of Object Trajectories in Videos
• MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions
• Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models
• Do Joint Audio-Video Generation Models Understand Physics?
• High Fidelity Neural Audio Compression
• $^R$FLAV: Rolling Flow matching for infinite Audio Video generation
• EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers
• Object-AVEdit: An Object-level Audio-Visual Editing Model
• VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editing
• Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
• ALIVE: Animate Your World with Lifelike Audio-Video Generation
• LTX-2: Efficient Joint Audio-Visual Foundation Model
• Classifier-Free Diffusion Guidance
• VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet
• A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation
• Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models