A Skill-augmented Agentic Framework and Benchmark for Multi-Video Understanding

This paper introduces MVX-Bench, a new benchmark designed to rigorously evaluate Multimodal Large Language Models (MLLMs) on complex multi-video understanding tasks that go beyond simple event-level comparison. It addresses the limitations of existing methods by proposing SAMA, a Skill-Augmented Agentic Framework, which enables iterative and structured reasoning across multiple video streams through the integration of visual tools, task-specific skills, and a conflict-aware verification mechanism.

MVXBench Benchmark

The authors introduce MVXBench, a MultiVideo CrossDimension Benchmark, which reformulates 11 classical computer vision tasks into a unified multi-video question-answering framework. This benchmark spans lowlevel perception, mid-level cross-video matching, and high-level reasoning. The benchmark contains 1,442 questions over 4,255 videos from diverse real-world datasets, covering indoor household scenes as well as outdoor environments. Key features include introducing tasks such as identity-level matching, finegrained discrimination, and structurally complex reasoning scenarios such as counterfactual inference and defeasible entailment.

SAMA Framework Overview

SAMA is proposed as a framework to overcome the structural challenges of direct multi-video concatenation, which suffer from a mismatch between single-video training and multi-video inference and information loss caused by aggressive frame compression. SAMA is organized into two main layers: an Agent Core and an external Verification Layer. The Agent Core consists of:

1. A text-only LLM Planner that performs task analysis and decision-making.

2. A multimodal toolkit of Tools for extracting evidence from videos (e.g., Video Reader, Scene Graph, Object Tracker).

3. Reusable Skills that provide structured workflows and domain guidance to support tool use.

Conflict Detection and Adaptive Reasoning

The Verification Layer is crucial for enabling iterative and structured reasoning. It includes a Conflict Detection module that identifies inconsistencies among tool outputs (e.g., when the Video Reader describes a traffic light as red while the Scene Graph detects green). When conflicts are detected, the system triggers an Adaptive Reading module which guides the Planner to invoke additional tools for verification until the conflict is resolved. This process ensures that verified evidence is aggregated before further reasoning, maintaining accuracy through a loop that continues until sufficient, validated evidence is collected.

Task-Specific Skills and Tool Orchestration

To guide the LLM planner in complex coordination, SAMA incorporates task-specific skills such as the Multi-video Comparison skill. This skill enforces evidence-based decision-making by instructing the planner to prioritize tools based on the question type—for instance, prioritizing Visual Similarity for appearance questions or relying on Video Reader for event alignment in similarity tasks. The framework also includes skills like the Temporal Grounding Skill, which defines a protocol to propose candidate timestamps and mandates verification with the Video Reader before using them in reasoning.

Experimental Results

Experimental results on MVXBench demonstrate that SAMA "outperforms all evaluated baselines, including open-source model families such as InternVL3.5 (Wang et al., 2025b), Qwen2.5-VL (Bai et al., 2025), and Gemma-3 (Kamath et al., 2025), as well as proprietary models such as GPT. The study shows that the agentic framework substantially improves a vision model's performance, with SAMA achieving 52.7% overall accuracy," surpassing GPT-4o (47.7%) and other open-source end-to-end models. Ablation studies confirm the effectiveness of these components: removing skills reduces accuracy by 8.3 percentage points, and removing conflict detection leads to a performance decline, proving that resolving cross-tool inconsistencies is critical for reliable multi-video reasoning.

Contributions Summary

The main contributions are threefold: 1) Constructing MVXBench, a benchmark with 11 canonical computer vision tasks adapted to multivideo scenarios. 2) Presenting SAMA, the first agentic framework explicitly tailored to multi-video scenarios, which integrates diverse visual tools and a conflict resolution mechanism for structured reasoning. 3) Demonstrating that SAMA effectively outperforms strong open-source baselines and GPT on MVXBench, with ablation studies validating the importance of skill design and conflict detection.

Limitations

The authors note a limitation: Our benchmark focuses on multiple-choice question-answering settings, which may not fully reflect open-ended generation scenarios. Exploring free-form multi-video reasoning is identified as a direction for future work. The paper concludes by detailing the agent's execution process, including strict rules for thinking, tool calling, and iterative refinement to ensure accuracy.

Here are the specific improvements that can be made to AI systems based on the provided scientific paper (SAMA framework and MVX-Bench), along with what these improved systems can achieve:

The primary improvement lies in moving from single-video, direct inference models to sophisticated, iterative, and cross-modal agentic reasoning systems.

Here are the specific improvements:

1. Implementation of Skill-Augmented Agentic Frameworks (SAMA):

2. Integration of Diverse Visual Tools: Utilizing a rich toolkit including Perception (Scene Graph, Object Tracker), Detection (Scene Graph), and Other tools (Visual Similarity, Subtitle Retriever/Extractor).

3. Structured Procedural Guidance via Task-Specific Skills: Embedding explicit how-to logic into the LLM planner to dictate the correct sequence and invocation of tools based on the question type (e.g., prioritizing Visual Similarity for style questions vs. using Video Reader for event alignment).

4. Conflict-Aware Verification Mechanism: Introducing a dedicated layer that automatically detects inconsistencies between heterogeneous tool outputs (e.g., Scene Graph counting vs. Video Reader description) and triggers an Adaptive Re-reading loop to iteratively refine evidence until contradictions are resolved.

5. Structured Memory for Evidence Aggregation: Maintaining a memory dictionary to store validated evidence per video, using concise distilled representations for long sequences, ensuring that crucial information is not lost as the context window grows.

6. Multi-Step Reasoning and Iterative Planning (Agent Core): Employing a text-only LLM Planner that performs task analysis, selects appropriate tools/skills, reasons over intermediate evidence, and initiates new reasoning cycles when current evidence is insufficient—essentially simulating structured human problem-solving.

The resulting improved AI system (SAMA) can perform the following specific capabilities:

1. Identity-Level Matching: Accurately determine if two individuals appearing in different videos are the same person, even when clothing, viewpoint, and background vary (e.g., Re-Identification task).

2. Fine-Grained Discrimination (Forensic Detection): Distinguish between authentic and manipulated video content by cross-referencing visual artifacts with other tools (e.g., detecting inconsistencies between object counts reported by the Scene Graph and the Video Reader).

3. Complex Structural Reasoning: Execute high-level reasoning tasks such as counterfactual inference (predicting outcomes under hypothetical changes, like What if they had held on to the handlebar?) and defeasible entailment (dynamically revising a hypothesis based on new visual evidence).

4. Quantitative Cross-Video Comparison: Perform precise numerical comparisons across videos, such as accurately counting specific objects and determining relationships between those counts (e.g., Which video has more labeled objects than the reference?).

5. Style and Aesthetic Analysis: Identify and compare abstract visual properties like art style, color palettes, line quality, or animation fluidity across different video sources without relying solely on content matching (e.g., Which candidate clip comes from the same show based solely on visual style?).

6. Robustness to Information Loss: Maintain high accuracy even when dealing with compressed or multi-modal inputs by using structured tools and verification loops instead of simple concatenation, mitigating the training–inference mismatch inherent in direct multi-video processing.

Multimodal Large Language Models have achieved strong performance in single-video understanding, yet their ability to reason across multiple videos remains limited. Existing approaches typically concatenate multiple videos into a single input and perform direct inference, which introduces training-inference mismatch, information loss from frame compression, and a lack of explicit cross-video coordination. Meanwhile, current multi-video benchmarks primarily emphasize event-level comparison, leaving identity-level matching, fine-grained discrimination, and structured multi-step reasoning underexplored. To address these gaps, we introduce MVX-Bench, a Multi-Video Cross-Dimension Benchmark that reformulates 11 classical computer vision tasks into a unified multi-video question-answering framework, comprising 1,442 questions over 4,255 videos from diverse real-world datasets. We further propose SAMA, a Skill-Augmented Agentic Framework for Multi-Video Understanding, which integrates visual tools, task-specific skills, and a conflict-aware verification mechanism to enable iterative and structured reasoning. Experimental results show that SAMA outperforms strong open-source baselines and GPT on MVX-Bench, and ablations validate the effectiveness of skill design and conflict resolution.