VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval".

Jane: The gist Finding compact evidence encoded once and reused as new queries arrive allows accurate video search to operate on compact evidence,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we've touched on the name and what it promises, but let’s look at who wrote this thing. The paper is called VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval, and the authors are Shahaf Wagner, Gabriele Serussi, Dan Ben Ami, Tomer Galanti, and Chaim Baskin.

Jane: Yeah, it’s good to see a team from different places contributing to this work. It tells you they’re bringing in expertise from various directions when building these kinds of systems.

Lu: From what I can gather about their work, they are addressing the problem that finding the right video often requires distinguishing similar scenes in which different events occur, which is something joint matching helps with but processing rich video representations for each query is costly.

Meng: That cost is huge when you think about scaling up retrieval systems. So VEDJE’s approach seems to be a direct response to that cost by focusing on compressing features within sampled frames while keeping their representations separate in a reusable cache.

Lalam: They are using feature-change prediction as an auxiliary training signal, which they say improves retrieval from those compressed tokens even without needing an inference time predictor at query time.

The paper's summary: Tom: Let's unpack the core idea of the VEDJE paper. They’re compressing features inside sampled frames and keeping their representations separate across different sampled times to create this cache, which is what they call Z(v) equals

Z1;...;ZT: . What does that mean for how we store video evidence?

Jane: It means they are managing to keep evidence for different objects, actions, and moments in a video while keeping the total storage budget small enough for scale. They achieve this compression through that specific feature-change supervision they use during training.

Lu: The feature-change supervision is a clever trick because it allows them to train the compressed representation using future delta prediction, which helps improve retrieval from those compact tokens without needing an inference time predictor at query time.

Meng: That sounds like a way to get better performance on tight storage budgets without adding extra computation when someone is actually searching for something, which is exactly what engineers care about in deployment.

Lalam: And they show that this setup, with the lightweight joint encoder reading the cache alongside the query text and first-stage score, improves R@one in both retrieval directions across benchmarks like MSR-VTT and MSVD.

The paper's improvements: Tom: So let’s look at what they actually suggest as specific improvements in their methodology. They mention a few things that really stand out when you look at how they organized the cache and trained the system.

Jane: First, they focus on building compact frame-indexed caches, which means they compress information inside the sampled frames while keeping those frame groups separate. This gives the joint scorer access to evidence from across the whole sequence under a fixed storage budget.

Lu: They also have this feature-change supervision for compression, which is training them to predict differences between frozen features of sampled frames and then predicting those differences helps improve retrieval from the compressed representation.

Meng: I like that they show the benefit of retaining separate frame groups over pooling in their controlled comparisons, because it shows that organizing the cache differently actually boosts R@one by two points in both directions when storage stays fixed.

Lalam: They also used hardware-aware precision controls, like FP8 or FP4 quantization, and found that this reinforces the R@one reading and leaves text-to-video recall within zero point two points from the BF16 reference.

Conclusion: Tom: We’re wrapping up with the conclusion now for VEDJE. So what’s the big picture here for video search right now? What does this paper really mean for someone who only listens to our show?

Jane: The main point is establishing a practical path to joint video search where you compute compact evidence once and reuse it for later queries. This cache is query-blind, meaning it serves every subsequent query about a specific video, so selectively reading or expanding the cache at query time becomes a natural next step.

Lu: They also noted that the first and second stages aren't trained jointly yet, so closing that loop is another direction for future work. And they did mention that these evaluations span different video lengths but didn't establish performance on multilingual video retrieval.

Meng: For practical deployment, it shows a configuration like VideoPrism where shrinking the per-video cache fourfold to twelve kilobytes still keeps text-to-video recall within zero point two points, which is really impressive for resource constraints.

Lalam: And I think the final point is about how this system provides a flexible query pipeline by integrating a residual prior where the score is calculated as sθ(q, v) equals hψ(˜c(q, v)), which combines that joint representation with an embedded first-stage score to produce a final ranking.

Tom: So VEDJE shows how you can get high accuracy on video retrieval by making the evidence compact and reusing it intelligently for many queries without needing massive online computation. That’s the picture we have today on VEDJE.