Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos

Given multiple video inputs from freely moving cameras, this work proposes a two-stage optimization framework that decouples camera tracking and dense depth refinement to achieve consistent dynamic scene reconstruction and accurate camera pose estimation.

Problem Definition

The objective is to estimate per-frame camera states, specifically the per-frame camera states, which include the camera pose (represented by Tt i in SE(3)) and the corresponding depth map (Dt i in R H x W). The input consists of a set of time-synchronized monocular video sequences, where I t i is the image captured by the i-th camera at timestamp t, and K i are its intrinsic parameters. The core challenge addressed is reconstructing dense, dynamic scenes from multiple free cameras, overcoming issues like Scale ambiguity, Limited overlap, and handling Dynamic content.

Two-Stage Optimization Framework

The proposed method employs a two-stage optimization framework to tackle the complex problem:

1. The first stage focuses on robust camera tracking by extending single-camera SLAM to multiple cameras through the construction of a spatio-temporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling joint bundle adjustment with consistent scale. To handle initialization under limited overlap, a wide-baseline initialization strategy using a feed-forward reconstruction model is introduced to provide a unified scale anchor.

2. The second stage refines the scene by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow, which leverages the coarse camera poses obtained in the first stage to refine per-pixel depth and improve scene consistency.

Spatio-temporal Multi-Camera Tracking

The tracking process is governed by a spatio-temporal connection graph, denoted as Omega, which consists of three parts:

1. Temporal connection (Omega temp): This follows common practice in one camera setting, maintaining a temporal window to hold the latest keyframes for each camera.

2. Spatial connection (Omega spat): This is established by evaluating overlap across different camera frames; a spatial connection is made if most of the projected pixels (more than 75%) are within the boundary of the image.

3. Spatio-temporal connection (Omega st): This exploits historical information by connecting an active keyframe with those inactive frames from other cameras if there is enough overlap, while a connection balance strategy manages memory by removing the oldest edges if the maximum number of connections is exceeded.

Wide-Baseline Initialization

To overcome the unreliability of conventional overlap-based initialization in multi-camera settings, the method adopts a robust feed-forward scene reconstruction model VGGT [38] for initialization. This model is used to select initial frames and obtain initial camera pose estimates along with per-frame depth predictions (DVGGT). To ensure metric consistency, the system aligns these predictions by estimating a global scale 's' and offset 'o' through minimizing a loss function, ensuring that the initial state provides a coarse but globally consistent geometric initialization across multiple cameras.

Multiple-View Scene Consistency Refinement

The refinement stage optimizes per-frame scale and per-pixel depth corrections to achieve dense consistency. This involves:

1. Constructing an augmented connection graph Omega refine that includes the tracking graph and additional dense temporal connections within each camera sequence, connecting frames at offsets like +2, +4, +8.

2. Minimizing the weighted reprojection error loss (L reproj), which combines a flow reprojection term (L flow) and a disparity consistency term (L disp). The L flow term penalizes misalignment between optical flow correspondences and geometric reprojections, while the L disp term penalizes inconsistencies between forward-projected and estimated depths.

3. This optimization is performed in two phases: Phase 1 fixes camera poses to initial estimates to optimize per-frame scale parameters (s i, β i); Phase 2 refines both depth and camera poses iteratively by minimizing the overall loss L, augmented with pose regularization terms (Lprior + Lsmooth) to enforce temporal smoothness along each camera trajectory.

Real-World Evaluation

The method is evaluated on the self-collected MultiCamRobolab dataset, which includes 24 RGB-D sequences from Microsoft Azure Kinect cameras with ground-truth poses from a Qualisys motion capture system. Performance metrics include Absolute Translation Error (ATE), Relative Translation Error (RTE), and Relative Rotation Error (RRE) for camera trajectories, and Absolute Relative Depth (Abs.Rel) and Delta accuracy for depth quality, alongside scene consistency measured by the median Euclidean distance (Md) between corresponding 3D points in the predicted and ground-truth reconstructions.

Based on this research paper, here are specific improvements that can be made to existing AI systems, categorized by the capabilities they enable:

) Multi-Camera Dynamic Scene Reconstruction System (The Core Improvement)

This system moves beyond single-camera SLAM or rigid multi-camera setups to accurately reconstruct a shared dynamic event captured by multiple free-moving cameras.

1. It will estimate the precise 6DOF camera poses and dense per-frame depth maps simultaneously for all cameras at any given time step.

2. It will maintain metric consistency across all cameras throughout the entire video sequence, overcoming scale ambiguity inherent in monocular depth estimation using a wide-baseline initialization strategy anchored by a feed-forward model (VGGT).

3. It will handle limited or intermittent view overlap by constructing a sophisticated spatio-temporal connection graph that explicitly models both intra-camera temporal continuity and inter-camera spatial overlap, allowing it to robustly track scenes even in challenging, non-overlapping scenarios (e.g., RoboDognon-overlap scenes).

4. It will achieve superior performance in dynamic environments compared to state-of-the-art feed-forward models by decoupling camera pose estimation from depth refinement, leading to higher fidelity reconstructions and better scalability for long video sequences.

) Enhanced Scene Understanding for Robotics and AR

This system enables richer contextual understanding of dynamic scenes for robotics and augmented reality applications.

1. It will provide a dense 3D representation of moving objects in the scene, allowing robots to perform accurate collision avoidance, manipulation planning, or interactive tasks within complex dynamic environments.

2. It will enable robust tracking of dynamic entities (like pedestrians or robots) across multiple viewpoints simultaneously, improving object recognition and trajectory prediction accuracy in real-time AR applications.

) Robust Training Data Generation Infrastructure

This research contributes a new benchmark for training and evaluating future 3D reconstruction models.

1. The introduction of the MultiCamRobolab dataset provides a standardized, real-world ground truth (via motion capture) for multi-camera pose estimation and depth accuracy, allowing researchers to quantitatively compare novel algorithms against established baselines in complex dynamic settings.

) Memory-Efficient and Scalable Reconstruction Pipelines

This system addresses the practical constraints of deploying complex reconstruction models on resource-limited hardware.

1. The proposed two-stage optimization framework (initial coarse tracking followed by dense refinement) is designed to be memory efficient, allowing for the processing of long video sequences that would cause memory overflow in purely feed-forward or attention-based models, making it viable for real-time or near real-time deployment.

We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras -- a setting that arises naturally when multiple observers capture a shared event. Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability. We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement. In the first stage, we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. To ensure robustness under limited overlap, we introduce a wide-baseline initialization strategy using feed-forward reconstruction models. In the second stage, we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow. Additionally, we introduce MultiCamRobolab, a new real-world dataset with ground-truth poses from a motion capture system. Finally, we demonstrate that our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world benchmarks, while requiring less memory.