Time-Correlated Video Bridge Matching

arXiv:2510.12453 · cs.LG · Submitted 2026-08-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Time-Correlated Video Bridge Matching".

Jane: The paper was written by Viacheslav Vasilev, Arseny Ivanov, Nikita Gushchin, Maria Kovaleva and Alexander Korotin from Kandinsky Lab and Applied AI Institute and HSE University and AXXX.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Alright, welcome back to the show. Today we're digging into a new arXiv paper called "Time-Correlated Video Bridge Matching." Jane, what caught your eye first?

Jane: Tom, honestly, the title alone tells you they're trying to fix something that's been bugging me for a while. Video generation models are great at making individual frames look good, but they often forget that frames in a video are supposed to talk to each other. This paper says, hey, let's build that connection right into the math.

Tom: Exactly. And the authors are from Kandinsky Lab, Applied AI Institute, HSE University — a solid crew out of Moscow. They're not just slapping a new layer on an old model; they're rethinking the whole bridge matching framework.

Jane: Right. So for our listeners, bridge matching is like building a bridge between two pictures — say, a blurry image and a sharp one — and then walking across it step by step. But if you're making a video, you've got ten frames, and each one needs its own bridge. Most methods just build ten separate bridges and hope they line up.

Tom: And they don't, right? You get flicker, you get jumps, you get a digit that morphs into a blob halfway through. This paper says, what if we build one bridge that connects all the frames at once, with springs between them so they stay in sync?

Jane: That's the "time-correlated" part. They're saying the frames aren't independent — frame five should be close to frame four and frame six. So they build that assumption into the prior process itself, not as an afterthought.

Tom: And the results on MovingMNIST — that's the dataset with moving digits — they beat plain Bridge Matching on almost every metric. Frame interpolation, image-to-video, super-resolution. The videos just look more consistent.

Jane: I love that they didn't just tune an existing model. They changed the underlying stochastic process. That's a fundamental improvement, not a patch.

Tom: And that's what we're going to unpack today. How they did it, what it means for real-world video tools, and whether it scales beyond toy datasets.

Jane: Let's get into it.

Summary: Tom: So, Jane, let's recap where we are. The paper "Time-Correlated Video Bridge Matching" takes the bridge matching idea and makes it video-aware. But what's the actual problem they're solving in plain terms?

Jane: Think of it like this. You have a video of a ball rolling across a table. You want to generate the middle frames between two known frames. A standard diffusion model treats each frame as a separate puzzle piece and solves each one alone. But a ball in frame five should be somewhere between where it was in frame four and where it'll be in frame six. Ignore that, and you get a ball that teleports or wobbles.

Tom: So they're saying, let's not ignore it. They define a prior process where each frame is pulled toward its neighbors. Mathematically, they use a tridiagonal matrix — that's a fancy way of saying each frame only talks to the ones right next to it. Frame five gets pulled toward four and six, but not toward frame one.

Jane: And that's a really smart choice, because it's cheap to compute. You don't need to model long-range dependencies if you just want smooth motion. And they prove that with this prior, the bridge distribution — that's the path between the start and end frames — is still Gaussian. So you can sample from it in closed form.

Tom: That's huge. It means they don't have to approximate anything during training. They can write down the exact formula for the mean and covariance of the bridge, and just train a network to predict the clean frames.

Jane: And they show it works on three tasks. Frame interpolation, where you fill in missing frames. Image-to-video, where you animate a single still image. And video super-resolution, where you blow up low-res frames to high-res.

Tom: The numbers are pretty convincing. On frame interpolation, they get an SSIM of zero point eight one three versus zero point seven nine four for plain Bridge Matching. That's a measure of structural similarity — higher is better. And on video super-resolution, they jump from zero point nine five four to zero point nine seven zero. That's a big leap.

Jane: But the FVD scores — that's a video quality metric — are a bit mixed on super-resolution. They're slightly worse than plain Bridge Matching there. So it's not a universal win.

Tom: Right, but the paper acknowledges that. And honestly, the visual examples in the appendix look cleaner. The digits don't smear as much.

Jane: So the core idea is solid. Now the question is, how do you actually build this thing? What does the training look like?

Improvements: Tom: So, Jane, we've covered the problem and the basic idea. But what does "Time-Correlated Video Bridge Matching" actually change in practice? What's the improvement over just running Bridge Matching on video data?

Jane: The key improvement is that they're not treating the video as a bag of independent frames anymore. In standard Bridge Matching, you'd sample each frame from a Gaussian bridge independently. Here, they define a joint bridge over the whole sequence, with a covariance matrix that encodes the frame-to-frame relationships.

Tom: And that covariance matrix comes from that tridiagonal matrix we talked about. It's like putting springs between adjacent frames. During the denoising process, each frame is pulled toward its neighbors, so they can't drift apart.

Jane: Exactly. And they show that the score function — the thing the network learns — can be reparameterized. Instead of predicting the score directly, the network just predicts the clean video frames. That's a much simpler learning task, and it's the same trick used in modern diffusion models.

Tom: So the training loss is just mean squared error between predicted frames and ground truth frames. That's it. No fancy adversarial loss, no perceptual loss. Just a simple regression.

Jane: And at inference time, they iterate backward through time, each step sampling from the correlated bridge using the predicted clean frames. It's the same structure as standard bridge matching, but with the temporal correlation baked in.

Tom: One thing I really appreciate is that they also handle task-specific constraints. For frame interpolation, the first and last frames are fixed, so they add a bias term that pulls the generated frames toward those boundaries. For image-to-video, only the first frame is fixed. For super-resolution, no boundaries at all.

Jane: That flexibility is important. It means you don't need a different architecture for each task. You just change the bias vector and the matrix, and the same training loop works.

Tom: And they also ran ablation studies on hyperparameters — the noise scale and the strength of the correlation. The results are a bit noisy, but they found that moderate correlation and moderate noise work best.

Jane: They even tried making the correlation time-dependent — stronger at the end of the generation — but that didn't help. Constant correlation was just as good and simpler.

Tom: So the improvement is real, but it's not magic. It's a principled change to the prior that pays off in temporal consistency.

Jane: And that's exactly what video generation needs. Let's bring in Lu and Meng to talk about what this means beyond the toy dataset.

Lu: I'm excited about this because the framework is general. You could replace the tridiagonal matrix with something that models longer-range dependencies, like a full attention matrix. That could capture complex motions like a camera pan or a person turning around.

Meng: But the computational cost would go up. The paper shows that with a tridiagonal matrix, the overhead is tiny — just a few matrix multiplications per step. If you go full attention, you're looking at quadratic cost in the number of frames. For a one hundred-frame video, that's ten thousand entries. It might still be feasible, but it's not free.

Lu: True, but for short clips — say ten to twenty frames — it's totally doable. And you could use a learned matrix instead of a fixed one. Train it to capture the actual motion patterns in your dataset.

Meng: That's a natural next step. And I'd like to see how this handles real-world video, not just MovingMNIST. The paper admits that's a limitation. But the math is sound, so I'm optimistic.

Jane: Great points. Let's wrap this up.

Conclusion: Tom: Alright, we're at the end of our discussion on "Time-Correlated Video Bridge Matching." Jane, what's the one thing you want listeners to remember?

Jane: That temporal consistency isn't something you bolt on after the fact. This paper shows you can build it into the very structure of the generative process. By defining a prior that correlates frames, they get smoother, more coherent videos without changing the architecture.

Tom: And the results back it up. Better SSIM on interpolation, better FVD on image-to-video, and visually cleaner super-resolution. Not every metric wins, but the trend is clear.

Jane: The limitations are honest too. It's tested on MovingMNIST, a simple dataset. Real-world video with complex motion and camera movement is still an open question.

Lu: But the framework is ready for that. Swap the matrix, add more frames, use a learned correlation — the math holds.

Meng: And the computational overhead is small enough that it could be integrated into existing pipelines without a major rewrite. That's a practical win.

Tom: So we're saying goodbye to this paper, but not to the idea. Time-correlated priors could become a standard tool in video generation.

Jane: Absolutely. Thanks for listening, and we'll see you on the next one.

Viacheslav Vasilev, Arseny Ivanov, Nikita Gushchin, Maria Kovaleva, Alexander Korotin

Kandinsky Lab · Applied AI Institute · HSE University · AXXX

cs.LG

Submitted: 2026-08-16

Updated: 2026-08-18

Code: https://github.com/black-forest-labs/flux

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: “While Bridge Matching models address this by finding the translation between data distributions, their application to time-correlated data sequences remains unexplored.

Key concepts

Time-Correlated Video Bridge Matching
This method improves video generation by defining a joint process that connects all frames in a sequence. It ensures that each frame is mathematically pulled toward its neighbors, making the generated video smoother and more temporally consistent than methods treating frames independently.
Bridge Matching
This framework models the path or distribution between two images (like blurry to sharp). In video, it involves defining a 'bridge' that guides the generation process across multiple sequential frames to maintain structural integrity.
Tridiagonal Matrix
This is a specific matrix used in the paper's prior process. It mathematically ensures that each frame only interacts with, or is 'pulled' toward, the frames immediately before and after it. This keeps computation cheap while modeling local motion.

Terminology

Summary

Summary

The paper introduces Time-Correlated Video Bridge Matching (TCVBM), a novel framework that extends Bridge Matching (BM) to time-correlated data sequences in the video domain. The authors state: “While Bridge Matching models address this by finding the translation between data distributions, their application to time-correlated data sequences remains unexplored. This is a critical limitation for video generation and manipulation tasks, where maintaining temporal coherence is particularly important. To address this gap, we propose Time-Correlated Video Bridge Matching (TCVBM), a framework that extends BM to time-correlated data sequences in the video domain. TCVBM explicitly models inter-sequence dependencies within the diffusion bridge, directly incorporating temporal correlations into the sampling process.”

The core contribution is the explicit modeling of temporal correlations between video frames within the prior diffusion process. The authors note: “Unlike prior work, TCVBM explicitly takes into account inter-sequence dependencies, ensuring faithful translation between source and target distributions.” They compare TCVBM to classical methods based on bridge matching and diffusion models (DDPM, DDIM, and standard Bridge Matching) on three video-related tasks: frame interpolation, image-to-video generation, and video super-resolution. They report: “TCVBM achieves superior performance across multiple quantitative metrics, demonstrating enhanced generation quality and reconstruction fidelity.”

Methodology. The authors define a time-correlated prior process for sequences of length N, represented as X = (x1, …, xN), where each xn ∈ RD. They assume feature dimensions evolve independently but share the same temporal dynamics. The prior process is defined by the SDE: dXt = (AXt + b)dt + √ε dWt, where A ∈ RNˣN is a symmetric, invertible matrix encoding temporal correlations, b ∈ RNˣD is a drift correction term, and Wt is a matrix of independent Wiener processes. They derive closed-form expressions for the marginal distribution (Proposition 1) and the bridge distribution (Proposition 2) of this correlated process. The marginal is Gaussian: q(XtX0) = N(Xt μt0(X0), Σt0), with μt0(X0) = e(At)X0 + (e(At) − I)A−1b and Σt0 = ε(e(2At) − I)A−1/2. The bridge distribution is also Gaussian: q(XtX0, Xt′) = N(Xt μt0,t′, Σt0,t′), with explicit formulas for the mean and covariance.

The training objective minimizes the squared error between the predicted score function v φ(Xt, t) and the score of the prior process, averaged over bridge samples. The authors prove (Proposition 3) that learning the score function reduces to learning a predictor for the clean data X0 via a regression loss: min φ E[‖X̂0 φ(Xt, t) − X0‖2]. The training procedure is summarized in Algorithm 1, and inference (Algorithm 2) iteratively refines the estimate of X0 by sampling from the posterior using the learned predictor.

Choice of prior process. The authors define the matrix A with a tridiagonal structure (with −2 on the diagonal and 1 on the off-diagonals) to encourage smooth transitions between consecutive frames. The per-element prior is: dxtn = (xtn−1 − xtn) + (xtn+1 − xtn)dt + √ε dWt, which “naturally encourages a linear relationship between the elements of the sequence.” They provide task-specific choices for the vector b: for frame interpolation, b = (x0, 0, …, 0, xN+1)T enforces boundary conditions from fixed endpoints; for image-to-video generation, b = (x0, 0, …, 0)T with one fixed left point; for video super-resolution, b = 0 (no fixed endpoints).

Experiments. The authors evaluate on the MovingMNIST dataset (10,000 video sequences, 20 frames each, 64×64 resolution). They train three models per method for each task. For frame interpolation, the network inputs 10 frames with fixed first and last frames and generates 8 middle frames. For image-to-video generation, the task is to generate 9 frames from the first frame (initialized with copies of the first frame, which performed best). For video super-resolution, the network inputs 10 low-resolution frames (16×16) and outputs 10 frames at 64×64 resolution. They use a small U-Net with approximately 8.7 million parameters, trained for 150,000 iterations with batch size 128, EMA rate 0.999, AdamW optimizer, and 1000 reverse steps.

Quantitative results. The authors report metrics FVD, LPIPS, PSNR, and SSIM on 500 validation videos. For frame interpolation (Table 1): TCVBM achieves FVD 30.542 (vs. 34.315 for BM, 32.404 for DDPM, 33.609 for DDIM), LPIPS 0.077, PSNR 17.280, SSIM 0.813 — best or second-best on all metrics. For image-to-video generation (Table 2): TCVBM achieves FVD 44.96 (vs. 49.32 for BM, 75.86 for DDPM, 77.72 for DDIM), LPIPS 0.258, PSNR 10.75, SSIM 0.591 — best FVD and LPIPS. For video super-resolution (Table 3): TCVBM achieves FVD 59.491 (vs. 53.516 for BM, 607.880 for DDPM, 334.700 for DDIM), LPIPS 0.020, PSNR 22.670, SSIM 0.970 — best PSNR and SSIM, with LPIPS tied with BM. The authors also provide standard deviations across three seeds in Appendix C, noting that TCVBM is best or second-best on most metrics.

Additional analyses. The authors explore initialization strategies (Gaussian noise vs. linear interpolation for frame interpolation; static video vs. noise for image-to-video; low-res vs. low-res+noise for super-resolution), hyperparameter search for ε and α (Tables 10–12), computational cost analysis (the overhead of the bridge update is comparable to a few convolutional layers), and a dynamic correlation approach where αt increases as t → 0 (Appendix G), which showed no advantage over constant α.

Discussion. The authors note the potential impact: “The proposed Time-Correlated Video Bridge Matching (TCVBM) framework is designed for generative modeling and manipulation with sequential video data. Unlike traditional diffusion and bridge matching methods, which often ignore the intrinsic temporal structure of data, TCVBM explicitly models inter-sequence correlations. This principle are applicable not only to video but also to other types of sequences, such as audio signals or time series, where temporal consistency is important.” They acknowledge limitations: “the predefined tridiagonal prior may be insufficient for complex, long-range, or non-linear temporal dependencies in real-world data; future work should explore more sophisticated interpolants, such as those modeling correlations in feature space. Second, due to the computational cost, empirical validation is currently limited to MovingMNIST dataset.”

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems:

Improvement: Replace standard bridge matching or diffusion priors with the TCVBM framework that uses a tridiagonal matrix A to enforce inter-frame correlations during the generative process.

Implementation:

  • Add a tridiagonal coupling matrix A (size N×N) to the prior SDE: dXt = (AXt + b)dt + √ϵ dWt

  • Use the closed-form bridge distribution from Proposition 2 for sampling

  • Set A with-2 on diagonal, 1 on off-diagonals to enforce smooth transitions between consecutive frames

What the improved system can do: Generate video sequences where adjacent frames maintain temporal coherence, avoiding the flickering and discontinuity artifacts common in standard diffusion-based video generation.

The improved AI system can:

  1. Generate temporally coherent videos from images or low-resolution inputs

  2. Interpolate frames with smooth, realistic motion between keyframes

  3. Maintain visual consistency across long video sequences

  4. Adapt to different video tasks through configurable boundary conditions

  5. Train more stably with the reparameterized objective

  6. Operate efficiently in production environments with minimal overhead

These improvements directly address the paper's core contribution: explicitly modeling inter-sequence dependencies in the diffusion bridge, which standard methods ignore.

Abstract

Diffusion models excel in noise-to-data generation tasks, providing a mapping from a Gaussian distribution to a more complex data distribution. However they struggle to model translations between complex distributions, limiting their effectiveness in data-to-data tasks. While Bridge Matching models address this by finding the translation between data distributions, their application to time-correlated data sequences remains unexplored. This is a critical limitation for video generation and manipulation tasks, where maintaining temporal coherence is particularly important. To address this gap, we propose Time-Correlated Video Bridge Matching (TCVBM), a framework that extends BM to time-correlated data sequences in the video domain. TCVBM explicitly models inter-sequence dependencies within the diffusion bridge, directly incorporating temporal correlations into the sampling process. We compare our approach to classical methods based on bridge matching and diffusion models for three video-related tasks: frame interpolation, image-to-video generation, and video super-resolution. TCVBM achieves superior performance across multiple quantitative metrics, demonstrating enhanced generation quality and reconstruction fidelity.

Sources

Related papers