Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models

This paper introduces Autoregressive Video Inverse problem Solver (AVIS) and its highly accelerated variant, AVIS Flash, which leverage autoregressive video diffusion models to restore videos in a streaming manner. This approach addresses the critical limitations of existing non-autoregressive solvers—namely high initial latency and low throughput caused by multiple VAE passes—by initializing the reverse diffusion process with a measurement-consistent estimate. The work demonstrates that AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS, while AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU, paving the way toward real-time deployment of diffusion-based video inverse problem solvers.

AVIS Framework and Core Mechanism

The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. This is achieved by drawing inspiration from CCDF [27], which demonstrates that initializing the reverse diffusion process from a coarse estimate significantly reduces the required sampling steps. Unlike CCDF, AVIS does not rely on an auxiliary pretrained restorer; instead, it obtains an initial restoration by directly minimizing the measurement consistency objective. AVIS enforces measurement consistency for every video chunk during the streaming process.

AVIS Flash Acceleration Strategy

To push efficiency further, the authors introduce AVIS Flash. Unlike AVIS, which applies iterative measurement guidance to every chunk, AVIS Flash enforces measurement consistency solely on the first video chunk. The authors observe that this is sufficient to restore the entire sequence because subsequent chunks are generated through autoregressive propagation from this corrected prefix. This process completely bypass[es] explicit measurement guidance, which eliminates iterative VAE passes for subsequent chunks, thereby accelerating restoration by 5× while preserving zero-shot performance.

Measurement Consistency Enforcement

Measurement consistency is enforced during the reverse diffusion process of the n-th noisy video chunk at each denoising step to guide the sampling trajectory toward the posterior. The conventional approach involves computing gradients of a measurement consistency term, but AVIS employs Decomposed Diffusion Sampling (DDS) [31] using conjugate gradient (CG) to efficiently enforce consistency. This is done by:

1. Obtaining the clean latent estimate from the noisy state using Eq. (6).

2. Decoding this estimate into pixel space: xˆn 0t = D(zˆn 0t).

3. Solving a proximal optimization problem via CG to find an updated pixel-space estimate x˜n 0t, minimizing the objective: x˜n 0t:= arg min x γ 2∥y − A(x)∥ squared + 1/2∥x − xˆn 0t∥ squared.

4. Re-encoding the updated pixel-space estimate back into the latent space: z˜n 0t = E(x˜n 0t).

Experimental Validation and Results

The methods were evaluated on 100 high-resolution videos across five restoration tasks, including SuperResolution, Random Inpainting, Gaussian Deblur, Temporal Average, and Spatio-Temporal Average. The results show that AVIS achieves the best restoration performance overall. Specifically:

- Computational Efficiency:

AVIS reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS compared to the leading non-autoregressive solver LVTINO (0.71 FPS). AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU, making it over 8× faster than VISION-XL and LVTINO in some tasks.

- Restoration Performance:

AVIS consistently attains the best or near-best PSNR, SSIM, LPIPS, and FVD across most tasks. AVIS Flash maintains competitive performance with baselines while reducing initial latency by over 100 seconds and boosting throughput by over 12× and 8×.

Ablation Study Insights

The ablation studies on AVIS Flash provide further insight into the framework's components:

- Autoregressive Propagation:

Removing the KV cache for subsequent chunks degrades all metrics, highlighting the benefit of autoregressive propagation, which provides performance gains complementary to the initialization. The authors show that autoregressive propagation alone preserves context but gradually drifts from the desired restoration, a drift mitigated by their initialization.

- Start Time (t0):

A smaller start time t0 consistently improves most metrics, with t0 = 0.1 yielding the best overall performance.

Here are specific improvements to AI systems based on the proposed Autoregressive Video Inverse problem Solver (AVIS) and AVIS Flash framework:

The proposed framework, AVIS/AVIS Flash, offers significant advancements in the deployment of zero-shot video inverse problem solvers by addressing the critical bottlenecks of high initial latency and low throughput. The improvements can be applied across several domains: Real-time Video Restoration, High-Throughput Generative AI Inference, and Novel View Synthesis.

Here are the specific improvements and capabilities:

1. Real-Time, Low-Latency Video Restoration

The primary improvement is the transition from holistic restoration (high latency) to a streaming (autoregressive) approach.

The improved system can perform video restoration in near real-time by:

Achieving initial latency of only 4 seconds for high-resolution super-resolution tasks (compared to 167s for VISION-XL), and achieving throughput up to 5.91 FPS (AVIS Flash) or 10.2 FPS on an NVIDIA H100 GPU.

2. High-Throughput Video Inpainting

The AVIS Flash variant drastically increases the speed of inpainting by eliminating iterative measurement consistency passes for subsequent chunks, making it suitable for high-demand applications like live video editing or real-time content moderation.

The improved system can perform:

Video inpainting with a throughput of 5.91 FPS (AVIS Flash), significantly faster than existing methods, while maintaining competitive perceptual quality metrics (e.g., LPIPS).

3. Robust and Scalable Long Video Reconstruction

The framework includes a mechanism for periodic re-injection of measurement consistency guidance every 7 chunks, which prevents the temporal drift inherent in purely autoregressive propagation over long sequences.

The improved system can handle:

Long video restoration (e.g., one-minute videos consisting of 960 frames at 16 FPS) while successfully retaining ground truth context and preventing noticeable temporal drift in later frames.

4. Inference-Time Novel View Synthesis

AVIS Flash can be adapted to serve as an inference operator for novel view synthesis, effectively acting as a high-quality inpainting mechanism to fill occluded regions during dynamic camera movement.

The improved system can perform:

Inference-time novel view synthesis by using AVIS Flash as an inpainting operator to plausibly synthesize disoccluded content under target camera trajectories.

5. Improved Generative Priors via Autoregressive Modeling

By leveraging autoregressive diffusion models, the system establishes a strong temporal prior that is conditioned on previously generated frames, leading to superior temporal coherence and fidelity compared to non-autoregressive solvers.

The improved AI model can:

Generate perceptually realistic video sequences with superior Subject Consistency (Sub. C.) and Background Consistency (Bg. C.) metrics by utilizing the sequential conditioning of AR diffusion models.

6. Enhanced Initialization for Zero-Shot Restoration

The use of a measurement-consistent initial estimate, derived via a multi-step optimization (CG), ensures that the reverse diffusion process starts from a better point than random noise, enhancing fidelity in zero-shot tasks.

The improved system can:

Solve complex video inverse problems in a zero-shot manner with superior restoration quality by initializing the reverse diffusion process with a measurement-consistent estimate derived from pixel-space optimization.

Diffusion models provide powerful priors for zero-shot video inverse problems, but their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes to enforce measurement consistency in pixel space. To overcome these limitations, we propose Autoregressive Video Inverse problem Solver (AVIS). The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. Compared to leading non-autoregressive solvers, AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. We further introduce a highly accelerated variant, dubbed AVIS Flash, that enforces measurement consistency solely on the first chunk. AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance and achieving a favorable efficiency-performance trade-off, paving the way toward real-time deployment.