CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

arXiv:2609.03675 · cs.CV · Submitted 2026-09-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding".

Tom: Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, to recap what we’ve heard, the main point of this paper, "CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding," is that they tackle the challenge of video understanding under tight latency constraints by proposing a new framework.

Jane: Essentially, the thesis is that existing methods often focus on pruning tokens after the vision encoder has already run its course, which doesn't really help much because you’ve already paid for that visual computation.

Lu: CoFiE proposes a coarse-to-fine evidence selection framework that separates evidence selection into two parts: a query-agnostic filtering stage before the vision encoder and a fine query-specific refinement stage during LLM prefill.

Meng: That separation is key because it means they first identify visually distinctive frames generally, and then they narrow that set down based on what the user is asking about specifically during the prefill phase.

Lalam: The goal of this framework is to keep a high-recall set of visually distinctive candidates initially, and then rank those specific candidates into evidence frames that are actually important for the user's question.

Tom: That sounds like a very logical two-step approach, Jane. So, what does this mean in practical terms for how we handle continuous video streams?

Jane: It means they introduce two lightweight modules: the coarse Novelty-Guided Frame Filtering and the fine Query-Specific Evidence Refinement. These modules are designed to work together to select the best visual evidence efficiently.

Lu: Specifically, the first stage uses a module called Novelty-Guided Frame Filtering, which assigns a histogram-based novelty score to each frame to keep visually distinctive frames while discarding redundant ones before they hit the vision encoder.

Meng: So that means we’re trying to get rid of visual noise and redundancy right at the input stage, which is smart for reducing the computational load on the vision component.

Lalam: And then in the second stage, they use Query-Specific Evidence Refinement during LLM prefill to rank those candidates using text-to-visual attention scores available at middle-to-late decoder layers.

Tom: So, Jane, if I'm understanding this correctly, CoFiE is about making intelligent decisions about which visual data to keep before the heavy lifting of encoding happens?

Jane: That’s exactly right, Tom. It’s a framework that decouples the evidence selection process into two distinct stages to ensure both broad visual context retention and query-specific relevance during answer generation.

Lu: This approach seems very creative because it tackles the non-uniform nature of evidence in streaming videos by using novelty scores to find distinctive frames initially, and then refining those based on semantic query needs later.

Meng: I'm focusing on the efficiency aspect here; if this framework can cut down on the visual tokens being processed before encoding, that has direct implications for reducing latency across the board.

Lalam: And when we look at the results from StreamingBench and OvO-Bench, CoFiE shows that it improves over prior best methods by up to three point one five percent in accuracy on those benchmarks.

Conclusion: Tom: So, wrapping up this discussion on "CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding," the authors are Jing Jiang, Yiran Ling, Ruonan Li, Dimitrios Stamoulis, and Jie Liu.

Jane: The main implication we should grasp is that this framework shows a way to manage the trade-off between getting good accuracy from video understanding and maintaining fast response times in streaming scenarios.

Lu: What I think is that this work suggests we can move towards AI systems that are more responsive in real-time, capable of handling the continuous flow of video data without slowing down significantly.

Meng: From an engineering perspective, this means we might see more practical applications where processing live streams—like monitoring complex visual environments—becomes feasible without massive hardware investment.

Lalam: I think the impact could be that multimodal AI becomes much better at understanding nuanced, time-sensitive visual information in a continuous stream, which is crucial for applications like real-time safety monitoring.

Tom: It really boils down to having a method that intelligently decides what visual evidence to prioritize at every step of processing. CoFiE provides that intelligent decision-making structure for streaming video AI.

Jane: By focusing on this dual selection strategy, the authors give us a concrete way to improve the accuracy-efficiency trade-off in practical, high-demand scenarios.

Lu: It opens up new avenues for research into how to dynamically adapt evidence selection based on real-time stream characteristics and user queries.

Meng: I’m just glad we’ve seen this work because it moves us closer to building more robust AI that can operate reliably in challenging, high-throughput video situations.

Lalam: It really reinforces the idea that sophisticated selection mechanisms are what will enable the next generation of video understanding to be truly practical for real-world, continuous use.

Harbin Institute of Technology · State Key Laboratory of Smart Farm Technologies and Systems

cs.CV

Submitted: 2026-09-03

Updated: 2026-10-01

Comments: Accepted at EMNLP 2026 main conference

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints.

Key concepts

Novelty-Guided Frame Filtering (NGFF): This initial stage filters video frames by assigning a novelty score based on histogram changes between consecutive frames. Frames with high novelty, indicating visually distinctive content, are retained as high-recall candidates, while redundant frames are discarded. Temporal safeguards ensure the retention of key start/end frames.
Query-Specific Evidence Refinement (QER): The second stage ranks the filtered candidate frames based on their relevance to the user's question semantics. QER uses text-to-visual attention scores available in LLM decoder layers to compute a compatibility score between each frame and the query tokens, retaining only the most relevant evidence frames.
Coarse-to-Fine Evidence Selection: This is a two-stage process designed to balance accuracy and efficiency. The first stage quickly prunes large sets of visually uninteresting frames (coarse filtering), and the second stage precisely selects the most relevant frames for answering the specific user question (fine refinement).
Total Drop Ratio ($D_{total}$): This metric measures overall compression effectiveness, calculated as 1 minus the product of frame dropping ratios in both stages. It quantifies how much input data is removed to achieve a specific accuracy level, showing the combined impact of coarse filtering and fine refinement.

Terminology

Accuracy — 43× in this paper · explained in 1 episode
Benchmark — 21× in this paper · explained in 2 episodes
Novelty-Guided Frame Filtering — 10× in this paper
Pruning — 10× in this paper · explained in 4 episodes
Query-Specific Evidence Refinement — 9× in this paper
Ablation — 8× in this paper · explained in 1 episode
Conference on Computer Vision and Pattern Recognition — 6× in this paper
Decode — 4× in this paper · explained in 2 episodes
Redundancy — 4× in this paper · explained in 2 episodes
Ablation Studies — 3× in this paper · explained in 1 episode
Optical Flow — 3× in this paper · explained in 2 episodes
Aggregation — 2× in this paper · explained in 1 episode
Benchmarking — 2× in this paper · explained in 4 episodes
CVF International Conference on Computer Vision — 2× in this paper
Fine-Tuning — 2× in this paper · explained in 5 episodes
Hyperparameters — 2× in this paper · explained in 2 episodes
Vision-language models — 2× in this paper · explained in 2 episodes
Causal Reasoning — 1× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Feature Extraction — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalization — 1× in this paper · explained in 13 episodes
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes
Vision Transformer (ViT) — 1× in this paper · explained in 2 episodes

Summary

Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. The gist: CoFiE establishes a new state-of-the-art accuracy–efficiency trade-off with open-source VLLMs across multiple video understanding benchmarks, reaching 78.86% on StreamingBench and 68.72% on OvO-Bench, improving over prior best methods by up to 3.15%.

How it works

CoFiE is a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into two stages: a coarse query-agnostic filtering stage before the vision encoder and a fine query-specific refinement stage during LLM prefill. The key intuition is to first keep a high-recall set of visually distinctive candidate frames, and then rank these candidates into evidence frames that are important for the user question.

In the coarse stage, the Novelty-Guided Frame Filtering (NGFF) module assigns each frame a histogram-based novelty score to retain visually distinctive frames as high-recall candidates while discarding redundant ones. This is achieved using a heuristic novelty score based on grayscale histogram change, comparing the distance between an intermediate frame's histogram and both the immediately previous sampled frame and the most recent candidate frame at index r(t). Temporal safeguards are also introduced: always retaining the first and last sampled frames, adding uniformly spaced frames if fewer than Mmin candidates remain, or retaining a frame if its novelty exceeds a threshold or the distance from the previous candidate becomes too large.
In the fine stage, the Query-Specific Evidence Refinement (QER) module ranks these retained candidates by their relevance to question semantics using text-to-visual attention at LLM prefill. QER leverages attention scores available at middle-to-late decoder layers to compute compatibility between the user query and video frames without introducing additional computational overhead. The process involves computing a frame-level relevance score, where each visual token's score is determined by taking the maximum textual association across all query tokens, then averaging over the frame’s visual tokens. Finally, QER retains top-scoring candidate frames based on a keep ratio ρ: k = max(1,round(ρSc)), Se = TopKf∈Sc (wf, k).

Experimental Evaluation and Results

CoFiE was evaluated across both streaming video understanding benchmarks (StreamingBench and OvO-Bench) and offline long-video understanding benchmarks. The total drop ratio is defined as Dtotal = 1 − (1 − DNGFF)(1 − DQER). The results demonstrate that CoFiE outperforms prior best methods by up to 3.15%. Specifically, on StreamingBench, CoFiE achieves 78.86% accuracy with a 50% frame dropping ratio, improving over the strongest prior open-source baseline StreamForest by 1.32%. On OvO-Bench, CoFiE reaches an average score of 68.72% under the same drop setting, surpassing Qwen3-VL-8B by 3.69%. Furthermore, even under aggressive evidence-frame filtering up to 80%, CoFiE outperforms open-source frontier multimodal models while improving end-to-end inference latency by up to 2.54×.

Ablation and Sensitivity Analysis

The ablation studies confirm the complementary nature of the two stages. NGFF alone improves the unpruned baseline by 0.54% at a 20% drop, but its accuracy decreases with increasing drop ratio, suggesting that a query-agnostic signal becomes insufficient once the compression budget discards sparse answer evidence. QER consistently shifts this trade-off by refining NGFF candidates with query-dependent attention; adding QER to NGFF20% raises the average score while increasing the total drop ratio to 50%. The default filtering signal, Histogram Difference, achieves the highest accuracy among all adaptive signals under a 20% drop ratio, yielding an 8.2% end-to-end speedup while retaining 96.9% of the baseline accuracy.

Latency and Efficiency Gains

CoFiE is designed to reduce redundant visual computation early while preserving query-relevant evidence needed for accurate responses. The stage-wise latency breakdown on OvO-Bench shows that CoFiE achieves a 2.54× speedup when dropping 80% of the input frames, compared to the full-frame baseline. This gain is mainly attributed to reducing computation before visual encoding, as frame-level dropping is more effective for end-to-end acceleration because it removes redundant visual computation before the vision encoder, while token-level dropping only shortens the downstream multimodal context after the dominant front-end cost has already been paid. CoFiE achieves its best accuracy of 68.

Improvements for AI systems

As a fastidious researcher, I have meticulously analyzed the CoFiE framework presented in this paper. The core innovation lies in decoupling evidence selection into a coarse, query-agnostic stage (Novelty-Guided Frame Filtering - NGFF) and a fine, query-specific refinement stage (Query-Specific Evidence Refinement - QER).

Here are the specific improvements I propose for AI systems and what these improved systems can achieve:

)

A new inference pipeline architecture that implements the CoFiE framework:
A novel pre-encoder filtering module (NGFF) based on histogram-based novelty scoring:
A lightweight, attention-based evidence refinement module (QER) during LLM prefill:
An optimized hyperparameter allocation strategy for NGFF and QER based on the desired accuracy/efficiency trade-off:
Specific capabilities of the improved system:
Improved latency for streaming video understanding by up to 2.54× (compared to baseline VLLMs) while maintaining high accuracy (up to 3.15% improvement over prior state-of-the-art).
Real-time processing of long, continuous video streams under strict latency constraints, capable of answering user questions immediately as the video progresses.
Robust performance across multiple demanding benchmarks (StreamingBench, OvO-Bench), maintaining high accuracy even under aggressive frame dropping (up to 80%) and preserving near-baseline performance when using a 50% drop ratio.
Superior handling of non-uniform evidence in streaming data, ensuring that rare but critical events are not discarded prematurely by uniform compression rules.
Enhanced long-video reasoning capabilities, demonstrated by outperforming prior online video VLLMs on benchmarks like MLVU and LongVideoBench by up to 10.86%.