SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding

Multimodal Large Language Models (MLLMs) have shown rapid progress in understanding single videos, but their ability to reason across multiple independent video streams remains poorly understood. Existing multi-video benchmarks rely heavily on human annotation of real-world footage, which limits the precision of ground truth and makes it difficult to diagnose model failures. This paper introduces SYNCR, a controlled synthetic benchmark designed for cross-video reasoning with programmatically verified grounding, addressing this gap by isolating core reasoning skills through simulation.

The Problem and Motivation

The rapid evolution of MLLMs has led to substantial progress in single-video understanding, but human perception and many real-world applications rarely involve a single isolated perspective. Intelligent systems must synthesize fragmented spatiotemporal information from multiple independent sources, motivating a shift toward cross-video reasoning: the ability to align, compare, and integrate information across disjointed video streams. Current benchmarks like MVUEval and CVBench are valuable for broad semantic understanding but rely on human annotation which limits the precision of variables such as exact 3D distances, sub-second temporal offsets, camera-aligned object trajectories, making it hard to attribute model errors.

The SYNCR Framework and Diagnostic Pillars

SYNCR is a controlled simulation-based framework built using three complementary engines: Habitat [21], Kubric [8], and CLEVRER [31]. It generates distinct multi-video subsets by independently leveraging these engines to construct isolated tasks with controlled temporal offsets, camera viewpoints, object identities, physical trajectories, semantic instances, and topological routes. This design decomposes cross-video understanding into four diagnostic pillars:

1. Temporal Alignment: For synchronizing and sequencing disjointed events.

2. Spatial Tracking: For maintaining object permanence and resolving cross-view geometry.

3. Comparative Reasoning: For comparing kinematic or structural differences across videos.

4. Holistic Synthesis: For aggregating fragmented observations into a global scene-level representation.

Dataset Generation with Synthetic Grounding

The dataset comprises 8,163 question-answer pairs grounded in 9,650 unique videos, ensuring that the ground truth is precise, consistent, and available across temporal, spatial, physical, and topological variables. This grounding is achieved via a three-step generation pipeline: (1) simulator state extraction to gather precise, frame-aware variables; (2) synchronized video rendering and sequence cropping; and (3) programmatic QA formulation. Specific task implementations include:

** Habitat:**

1. Object Re-identification: Ground truth is extracted programmatically from frame-wise object ID matrices to track exact timestamps of a target instance's appearance in the reference video.

2. Object Counting: Ground-truth counts are derived by extracting the exact number of unique semantic instance IDs for a queried object category across all generated trajectories, with constraints like requiring the target instance to occupy at least 5% of the frame pixels.

3. Route Planning: Ground truth is extracted from NavMesh connectivity logs, requiring a minimum path length of three nodes to infer navigable connectivity.

** Kubric:**

1. Multi-Angle Synchronization: Requires inferring temporal offsets between three asynchronous videos by sampling crops and asking the model to determine starting timestamps relative to a reference video.

2. Spatial Measurement: Requires identifying the object with the minimum simulator-derived 3D Euclidean distance to the target object at the event frame.

** CLEVRER:**

1. Sequential Ordering: Involves reconstructing a continuous physical event from shuffled clips by splitting simulations into segments and asking for the correct chronological order of these video segments.

2. Kinematic Comparison: Evaluates whether a model can compare continuous motion across videos by identifying the object with the highest peak velocity.

Experimental Results and Conclusions

The zero-shot evaluation reveals a substantial gap between current models and humans, with the best model achieving only 52.5% average accuracy compared to an 89.5% human baseline. Models perform relatively well on temporal ordering but struggle with precise physical and spatial reasoning, particularly in Kinematic Comparison, where the best open-weight result is only 26.0%. While parameter scaling and reasoning-specialized post-training improve temporal alignment capabilities, they do not reliably address fine-grained physical tracking or global spatial synthesis. Exploratory sim-to-real correlation analysis suggests that several SYNCR tasks track model trends on real-world benchmarks, while others expose reasoning capabilities underrepresented by existing evaluations, positioning SYNCR as a controlled testbed for diagnosing cross-video reasoning failures.

Limitations

The framework has limitations, including the inherent constraints of the simulation engines regarding visual diversity and photorealism, reliance on strict exact-match Question Answering (QA), exclusion of audio streams, and a sample size bounded by recent architectural emergence.

Based on the SYNCR paper, here are specific improvements that can be made to existing Multimodal Large Language Models (MLLMs) and a description of what those improved systems could achieve:

1. Improve MLLM capabilities in cross-video reasoning by introducing a structured synthetic benchmark framework called SYNCR.

2. Develop MLLMs capable of performing four distinct diagnostic pillars:

3. Temporal Alignment (synchronizing and sequencing disjointed events).

4. Spatial Tracking (maintaining object permanence and resolving cross-view geometry).

5. Comparative Reasoning (comparing kinematic, structural, or numerical differences across videos).

6. Holistic Synthesis (aggregating fragmented observations into a global scene-level representation).

7. Implement reasoning-specialized post-training techniques (e.g., the Thinking checkpoint approach) specifically targeting temporal alignment to improve chronological ordering accuracy without sacrificing other reasoning skills like spatial tracking or kinematic comparison.

8. Enhance model architectures and training objectives to resolve persistent bottlenecks in fine-grained physical and global spatial reasoning, particularly in Kinematic Comparison and Holistic Synthesis, which currently show near-chance performance across various scales.

9. Integrate simulation engines (Habitat for semantic navigation/topological synthesis, Kubric for multi-camera spatial/temporal reasoning, and CLEVRER for kinematic/collision reasoning) into a unified data generation pipeline to create high-precision ground truth grounded in programmatic physics and geometry, eliminating human annotation noise.

10. Develop robust cross-video grounding mechanisms that allow models to reliably track object identity across drastically different viewpoints (Spatial Tracking) and accurately estimate 3D relative distances between objects in dynamic scenes (Spatial Measurement).

11. The improved AI system could achieve the following:

12. Accurately synchronize events captured from multiple, unaligned surveillance feeds or autonomous vehicle cameras, allowing for precise event reconstruction across time offsets.

13. Maintain persistent object tracking and identity across continuous motion when an object moves from a single camera view to another (e.g., tracking a vehicle as it passes behind a building).

14. Perform complex comparative analysis of physical interactions between objects in different videos, such as identifying which object exhibits the highest peak velocity during an interaction or quantifying the difference in collision counts between scenes.

15. Synthesize fragmented visual evidence from multiple camera angles into a single, coherent global map or scene description, enabling tasks like autonomous navigation path planning based on inferred topological connectivity.

16. Achieve a performance level that approaches human expertise (89.5% baseline) on complex cross-video reasoning tasks by specifically addressing the current scaling plateaus and architectural limitations exposed by the SYNCR benchmark.

Multimodal Large Language Models (MLLMs) have made rapid progress in single-video understanding, yet their ability to reason across multiple independent video streams remains poorly understood. Existing multi-video benchmarks rely largely on human-annotated real-world footage, limiting the precision of spatial, temporal, and physical ground truth and making it difficult to diagnose model failures. We introduce SYNCR, a controlled synthetic benchmark for cross-video reasoning with programmatically verified grounding. Built using Habitat, Kubric, and CLEVRER simulator engines, SYNCR contains 4,000 multi-video question-answer pairs grounded in 4,827 unique videos. It evaluates MLLMs across eight tasks spanning four diagnostic pillars: Temporal Alignment, Spatial Tracking, Comparative Reasoning, and Holistic Synthesis. Our zero-shot evaluation of leading open- and closed-weight MLLMs reveals a substantial gap between current models and humans: the best model achieves only 64.5% average accuracy, compared to an 89.5% human baseline. Models perform relatively well on temporal ordering but struggle with precise physical and spatial reasoning, with the best model reaching only 29.8% accuracy on Kinematic Comparison. We further find that parameter scaling and reasoning-specialized post-training improve temporal alignment capabilities, but do not reliably address fine-grained physical tracking or global spatial synthesis. Finally, a sim-to-real correlation analysis suggests that SYNCR tracks model-level trends on a real-world multi-video benchmark.