TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models

arXiv:2609.01277 · cs.CV, cs.AI, cs.MM · Submitted 2026-09-01 · Read on arXiv

cs.CV, cs.AI, cs.MM

Submitted: 2026-09-01

Updated: 2026-09-01

License: http://creativecommons.org/licenses/by/4.0/

The gist: Although pretrained joint audio-visual diffusion models offer rich control over what to generate, they provide no explicit control over when an utterance should occur.

Terminology

Abstract

Although pretrained joint audio-visual diffusion models offer rich control over what to generate, they provide no explicit control over when an utterance should occur. To address this, we study inference-time speech scheduling, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose TimeSteer, a training-free framework that localizes each utterance's source span through Source Span Localization and transfers the associated audio-visual latent content from the source interval to the specified target interval through Region-Aware Latent Remapping. We further introduce SpeechShift, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.

Sources

Related papers