UniVVT: A Unified End-to-End Framework for High-Fidelity Video Virtual Try-on

The paper introduces UniVVT, a unified end-to-end framework for Video Virtual Try-On (VVT), which synthesizes a video of a person wearing a target garment while preserving identity, motion, and scene dynamics. The authors identify that dominant approaches cast VVT as mask-conditioned video inpainting and rely on separate modules for human parsing, pose estimation, and garment warping. They argue this multi-stage design complicates deployment and, more critically, allows errors in explicit geometric priors to propagate irreversibly into the generated video.

The paper describes three coupled limitations of existing multi-stage pipelines: "First, errors from explicit geometric cues propagate to outputs and cause source residuals, incomplete garment replacement and boundary artifacts under occlusions or large motions. Second, many dedicated sub-models increase computational overhead and deployment complexity. Third, disjoint stage-wise optimization stops the generator from compensating upstream errors."

The authors propose "UniVVT, a unified end-to-end framework that instead formulates VVT as semantically conditioned video generation. Our key insight is that what to transfer and where and how to transfer it can be encoded implicitly by multimodal semantics rather than prescribed through masks, poses, or warping fields."

UniVVT comprises three main components:

1. Scene-Task Perceiver: A Multimodal Large Language Model (MLLM) serves as a Scene-Task Perceiver, jointly encoding the source video, target garment, and task instruction into compact task-aware latent tokens. The authors instantiate it with the pretrained Qwen3-VL-2B-Instruct and use its hidden states as multimodal features rather than decoding text. They prepend N learnable task queries Qtask to the multimodal sequence that aggregate task-relevant information across the instruction, video, and garment tokens. The resulting representation jointly captures garment attributes, scene context, body–garment correspondence, and temporal editing intent in a dense semantic representation.

2. Semantic Bridge: A lightweight trainable Semantic Bridge Pθ that projects ztask into generator-compatible conditioning tokens. Implemented as a compact multi-layer perceptron, it serves as a semantic adapter rather than learning new generative priors, translating high-level cues into conditioning signals.

3. Video Generation Module: Built on a pretrained latent video diffusion model with a Diffusion Transformer (DiT) backbone. The source video and garment are encoded via a pretrained causal video VAE, and a frozen CLIP image encoder extracts a complementary garment feature. Each DiT block injects both conditions through cross-attention, fusing the task condition (conveying what, where, and how to edit) with the CLIP feature (reinforcing garment appearance).

The authors devise a three-stage progressive training strategy to bridge representation gaps:

• Stage 1: Semantic alignment: We first freeze the video generator Gϕ, optimize the semantic bridge Pθ, and adapt the MLLM πmllm using Low-Rank Adaptation (LoRA). This establishes a reliable semantic interface for stable and effective subsequent joint training.

• Stage 2: Joint end-to-end adaptation: We then jointly optimize the VVT objective by applying LoRA to both πmllm and Gϕ while fully training Pθ at base resolution 512×384.

• Stage 3: Flexible-resolution refinement: We freeze πmllm and Pθ and continue fine-tuning the LoRA parameters of Gϕ from the Stage 2 checkpoint, using randomly sampled resolutions from 256p to 1024p.

Regarding training data, since aligned triplets are hard to collect, the authors train separate DensePose-conditioned inpainters for images and videos to synthesize only source inputs and construct ⟨source, garment, target⟩ triplets. They emphasize that All garment references and ground-truth targets come from original datasets, meaning UniVVT is supervised purely by real images and videos. DensePose is limited to offline data construction and unused during inference.

Training uses a mixed image–video corpus consisting of VITON-HD (11,647 pairs at 768 × 1024), DressCode (48,392 pairs at 768 × 1024), and ViViD (7,759 pairs at 624 × 832). Evaluation covers ViViD-S with the video-level protocol under both paired and unpaired settings and standard image-level protocols for DressCode and VITON-HD.

Key quantitative results:

• On ViViD-S video try-on, UniVVT delivers the best overall performance on most metrics: it achieves the lowest VFIDI and VFIDR under both paired and unpaired settings (paired VFIDI: 8.3623; unpaired VFIDI: 12.3640), and obtains the best LPIPS and a competitive SSIM under paired evaluation.

• On DressCode and VITON-HD image benchmarks, UniVVT consistently achieves the best or second-best performance across FID, KID, SSIM, and LPIPS.

• Latency analysis shows explicit preprocessing grows nearly linearly from 36.36 s to 111.39 s as video length increases from 30 to 90 frames, whereas UniVVT increases only from 2.06 s to 2.92 s, delivering a 17.7–38.1× speedup and removing 108.47 s of preprocessing overhead for 90 frames.

Qualitative results: The paper reports sharper garment textures and stronger temporal consistency compared to baselines like ViViD, CatV2TON, and MagicTryOn, and demonstrates success on in-the-wild TikTok dance videos that are out-of-distribution (OOD) relative to our training data.

Interpretation via attention visualization: Cross-attention visualization shows that attention concentrates on the upper body and remains high along the arms across frames, while responses on the face and background stay comparatively weak, indicating that ztask jointly encodes what to transfer, where to edit, and how to maintain appearance over time, acting as an implicit semantic guide that enables precise try-on without mask supervision.

• Scene-task perceiver necessity: Replacing STP with a text-only encoder degrades all metrics under both paired and unpaired settings (paired VFIDI increases from 8.3623 to 11.7308). Without STP, the model loses holistic task understanding and fails to localize valid edit regions — for example, a short skirt is incorrectly placed on the upper body.

• Progressive training: Stage 1 alone leaves a large residual gap to real data; Stage 2 yields the main gains in distributional realism and garment fidelity; Stage 3 further improving SSIM and LPIPS at 256 × 192 and 832 × 624.

• Complementarity of conditions: Removing ztask degrades all metrics substantially, while removing fclip produces consistently smaller, yet non-negligible, drops, confirming the task-aware pathway as the principal source of semantic control.

• MLLM adaptation: LoRA adaptation of the MLLM improves every paired and unpaired metric versus a frozen variant.

• Semantic bridge: Removing the bridge degrades all metrics (unpaired VFIDI increases from 12.3640 to 14.0105), and a heavy 12-layer Transformer bridge surprisingly performs slightly worse than the MLP, justifying the lightweight design.

The paper concludes that UniVVT "delivers strong garment fidelity and temporal consistency across diverse garments and motion without sacrificing identity or scene context, establishing end-to-end joint optimization with implicit semantic guidance as a practical alternative to multi-stage pipelines."

The authors acknowledge three limitations: (1) Fast body motion, abrupt viewpoint changes, and severe occlusion can induce local texture drift, discontinuous garment patterns, or short-term flicker; (2) ambiguity in cross-category transfer, since data primarily support garment replacement within the same semantic category; and (3) inference efficiency, as iterative diffusion denoising remains the dominant runtime bottleneck, making the model not yet suitable for real-time interactive applications.

Based on the paper, here are concrete improvements to AI systems and what the enhanced system can do:

1. Replace multi-stage, error-prone pipelines with a single end-to-end semantic generation model. Instead of relying on explicit masks, pose estimates, or warping fields, encode the person, garment, and edit instruction into task-aware latent tokens via a multimodal LLM (like Qwen3-VL), and feed those tokens directly into a diffusion transformer via a lightweight semantic bridge.

• This removes irreversible error propagation from upstream geometric predictors, cuts preprocessing latency by 17.7–38.1× compared to explicit pipelines (e.g., 2.92 s vs. 111.39 s for 90 frames), and simplifies deployment to one unified model.

2. Use multimodal hidden states, not text decoding, to capture spatial-temporal editing intent. Treat the MLLM as a scene-task perceiver that aggregates video, garment, and instruction information into compact latent tokens. Prepend learnable task queries to the multimodal sequence so the representation implicitly encodes what to transfer, where to edit, and how to maintain consistency over time—without mask supervision.

• The system learns to localize edit regions correctly (e.g., placing a skirt only on the lower body) and to sustain attention on the upper body and arms across frames while ignoring face/background, enabling precise try-on under occlusion and large motion.

3. Inject complementary garment appearance features alongside semantic task tokens. Fuse a frozen CLIP image embedding of the target garment with the task-conditioning tokens in every diffusion transformer block through cross-attention.

• This dual-conditioning improves garment fidelity and texture sharpness, with the task tokens providing the main semantic control and CLIP reinforcing fine appearance details. The system produces sharper logos, patterns, and fabric textures than single-condition baselines.

4. Adopt a three-stage progressive training strategy to align representations across heterogeneous modules.

• Stage 1: Freeze the video generator, train only the semantic bridge, and adapt the MLLM with LoRA to establish a stable semantic interface.

• Stage 2: Jointly optimize the MLLM LoRA, bridge, and generator LoRA for end-to-end objective alignment.

• Stage 3: Freeze the MLLM and bridge, fine-tune only the generator LoRA across resolutions from 256p to 1024p for flexible-resolution generation.

• This yields large gains in distributional realism and garment fidelity while avoiding catastrophic forgetting of the pretrained diffusion prior, and improves quality at both low and high resolutions.

5. Synthetically construct training triplets from real data without fake supervision. Train separate DensePose-conditioned inpainters to synthesize only source inputs, then build ⟨source, garment, target⟩ triplets where the garment reference and ground-truth target come from real datasets. Remove DensePose entirely at inference.

• This creates abundant, real-supervised training data for video try-on without needing expensive aligned video triplets, while keeping inference fully mask-free, pose-free, and warping-free.

6. Use a lightweight MLP as the semantic bridge instead of a heavy transformer. Project MLLM task tokens into generator-compatible conditioning with a compact multi-layer perceptron.

• This reduces parameter overhead and actually improves performance relative to a 12-layer transformer bridge, enabling efficient end-to-end training and faster inference without sacrificing semantic fidelity.

• Perform video virtual try-on in a single forward pass: Given a source video of a person and a target garment image (plus a text instruction), the system generates a high-fidelity video of that person wearing the garment, preserving identity, motion, and scene dynamics—without needing human parsing, pose estimation, or garment warping at inference time.

• Operate in real-world, out-of-distribution scenarios: It succeeds on in-the-wild TikTok dance videos with large motion and dynamic backgrounds, where previous multi-stage baselines fail due to error accumulation.

• Handle long videos efficiently: The system maintains near-constant preprocessing cost (2–3 s) as video length grows, while traditional pipelines degrade linearly (from 36 s to 111 s), enabling practical deployment on longer clips.

• Maintain temporal consistency and garment fidelity: It produces videos with minimal flicker, correct garment placement, sharp textures, and stable identity/background across frames, outperforming baselines on VFID, LPIPS, SSIM, FID, and KID benchmarks.

• Adapt across both image and video try-on tasks: The same unified framework achieves top-tier results on standard image benchmarks (VITON-HD, DressCode) and video benchmarks (ViViD-S) under both paired and unpaired evaluation protocols.

• Provide interpretable, implicit semantic control: Attention visualizations reveal that the system learns to focus on relevant body regions (e.g., arms and torso) while ignoring irrelevant areas, acting as an implicit mask-free guide for precise editing.

• Remain flexible across resolutions: With the third training stage, the model generates high-quality results at resolutions from 256×192 to 832×624, allowing adaptation to different display or bandwidth requirements.

• Run with substantially lower deployment complexity and cost: By eliminating dedicated sub-models and explicit preprocessing, the system reduces both computational overhead and engineering complexity, making video try-on more accessible for interactive and edge applications—although further work is needed to accelerate the diffusion denoising itself for true real-time use.