Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events

arXiv:2603.14528 · cs.CV, cs.RO · Submitted 2026-03-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events".

Jane: Interp3R introduces a novel framework that extends existing pointmap-based models to estimate depth and camera poses in continuous time by leveraging asynchronous event data.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's look at the title and authors again—"Interp3R: Continuous-time three dee Geometry Estimation with Frames and Events." The folks behind this work are Shuang Guo, Filbert Febryanto, Lei Sun, and Guillermo Gallego. It immediately tells us the core focus is continuous time estimation using events alongside traditional frames.

Jane: That title really hammers home the main contribution: extending pointmap models to estimate geometry at any arbitrary time instant rather than just discrete frame times. It suggests they’ve found a way to bridge that temporal gap effectively.

Lu: The authors are clearly building on existing work, as they show compatibility with different pointmap-based methods like MonST3R and Align3R, which is significant because it means this technique isn't isolated; it's an enhancement for several existing architectures.

Meng: So, if I understand correctly, the paper points out that current frame-based methods struggle when there's fast motion or large frame intervals because they can't provide continuous representations of scene geometry during those blind times.

Lalam: It’s about achieving a level of temporal understanding that existing methods simply don't have, which could drastically improve how our AI interprets complex, fast-moving real-world scenarios.

The paper's summary: Tom: Moving into the summary of "Interp3R: Continuous-time three dee Geometry Estimation with Frames and Events," it’s clear they propose using asynchronous event data to interpolate the pointmaps generated by frame-based models, creating these temporally continuous geometric representations.

Jane: Essentially, they start with what a standard model predicts at two frames and then use both forward and backward event data to fill in the missing information at any time between those two captures. It’s about temporal continuity through interpolation.

Lu: The methodology involves taking those co-captured events and processing them through an event encoder to create spatiotemporal grids, which are then fused with the predicted pointmaps using zero convolution within the Interp3R decoders. That's a specific mechanism for combining the different data streams.

Meng: That sounds like a lot of complex fusion happening in that decoder stage; I wonder how computationally intensive that interpolation step is when you’re dealing with high-resolution pointmaps and dense event streams simultaneously.

Lalam: From an AI culture viewpoint, this continuous representation means our systems gain a much richer understanding of scene dynamics, moving beyond static snapshots to modeling the actual flow of things in space and time.

The paper's improvements: Tom: Now let's discuss the specific improvements they highlight in "Interp3R: Continuous-time three dee Geometry Estimation with Frames and Events." They focus on solving the temporal consistency issue by jointly recovering geometry through a novel coarse-to-fine alignment of interpolated and frame-based pointmaps.

Jane: The key is that they don't just interpolate blindly; they use this two-stage process—a coarse alignment to get initial poses, followed by a fine stage where the interpolated pointmaps are aligned with those source pointmaps to solve for the geometry at the target time tau.

Lu: They also incorporated explicit time encoding into their prediction heads, which conditions the model on that specific interpolation time tau within the interval between frames, ensuring more consistent performance regardless of where you pick your target instant.

Meng: The training objective is defined by a loss function that sums up interpolation errors from both temporal directions and includes confidence terms, specifically using a weighted Euclidean distance for three dee errors. That’s quite detailed mathematical conditioning for the learning process.

Lalam: Training exclusively on synthetic data is interesting; it shows they can build this foundation robustly before moving to real-world complexity, which is a smart way to validate this continuous time estimation capability.

Conclusion: Tom: So, wrapping up "Interp3R: Continuous-time three dee Geometry Estimation with Frames and Events," the authors successfully demonstrate that they can extend pointmap models to estimate depth and camera poses at arbitrary time instants by leveraging asynchronous event data for temporal interpolation.

Jane: The main implication is that we can move toward truly continuous geometric representations, which should greatly benefit applications requiring precise tracking over long durations or in scenarios with very fast motion between captured frames.

Lu: It confirms their compatibility with other architectures like MonST3R and Align3R, showing this framework is adaptable and can be plugged into existing pipelines to enhance them for continuous time.

Meng: The paper flags a limitation: the interpolation quality might degrade if the source pointmaps are unreliable because of motion blur or poor illumination in the input frames, which they suggest could be fixed by using event data for deblurring or enhancement.

Lalam: And another challenge is that if there's flickering or rapidly changing light sources in the scene, the events might not represent true scene motion accurately, potentially causing artifacts during interpolation.

Tom: So while they show a really solid framework with strong performance on datasets like Sintel and TUM, we definitely have to keep an eye on those potential issues with unreliable source data and flickering lights when deploying this technology.

Jane: It’s exciting work that pushes the boundaries of what we can get from visual foundation models by allowing them to model time in a much more fluid way. We'll be looking for updates on how this continues to evolve in the coming months.

Shuang Guo, Filbert Febryanto, Lei Sun, Guillermo Gallego

TU Berlin · Robotics Institute Germany

cs.CV, cs.RO

Submitted: 2026-03-15

Updated: 2026-09-29

Comments: 22 pages, 16 figures, 5 tables

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 86/100

The gist: Interp3R introduces a novel framework that extends existing pointmap-based models to estimate depth and camera poses in continuous time by leveraging asynchronous event data.

Key concepts

Pointmap-based Models: These are existing methods that predict 3D geometry (pointmaps) from 2D images. They work well for static scenes but only provide a snapshot of the scene's structure at specific moments in time, which is not continuous.
Asynchronous Event Data: These are data points generated by event cameras that only record changes in brightness. Unlike standard video frames, they capture motion at high temporal resolution and can be used to estimate movement between two captured moments.
Pointmap Interpolation: This is the core technique where the model estimates the 3D geometry (pointmaps) at any arbitrary time between two input frames. It uses event data from both before and after a target time to create a temporally continuous geometric representation of the scene.

Terminology

Alignment — 23× in this paper · explained in 1 episode
Generalization — 10× in this paper · explained in 13 episodes
Accuracy — 9× in this paper · explained in 1 episode
Consistency — 6× in this paper · explained in 3 episodes
Foundation Model — 6× in this paper · explained in 3 episodes
Foundation Models — 6× in this paper · explained in 7 episodes
Benchmark — 5× in this paper · explained in 2 episodes
Optical Flow — 5× in this paper · explained in 2 episodes
Ablation — 4× in this paper · explained in 1 episode
Ablation Studies — 4× in this paper · explained in 1 episode
Diffusion Model — 4× in this paper · explained in 8 episodes
Loss Function — 4× in this paper · explained in 2 episodes
Diffusion Models — 3× in this paper · explained in 6 episodes
Robustness — 3× in this paper · explained in 5 episodes
Temporal consistency — 3× in this paper · explained in 2 episodes
Transformer — 3× in this paper · explained in 2 episodes
Core — 2× in this paper · explained in 2 episodes
Adam — 1× in this paper · explained in 2 episodes
AdamW Optimizer — 1× in this paper · explained in 1 episode
Aggregation — 1× in this paper · explained in 1 episode
Downstream Tasks — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Point Clouds — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Unified Framework — 1× in this paper · explained in 5 episodes

Summary

Interp3R introduces a novel framework that extends existing pointmap-based models to estimate depth and camera poses in continuous time by leveraging asynchronous event data. This method addresses the limitation of current frame-based models, which only recover geometry at discrete capture instants, by enabling temporally continuous geometric representations through the interpolation of pointmaps between frames.

The gist

Interp3R leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous geometric representations.

Methodology and Architecture

Interp3R begins with a frame-based model (e.g., DUSt3R) that predicts pairwise pointmaps from input images, denoted as the source pointmaps (X00, X01).
The core of the method involves interpolating these pointmaps at an arbitrary target time instant τ within the interval between frames, utilizing both forward and backward event data (E0→τ and E1→τ).
The co-captured events are processed by an event encoder (EE) to create spatiotemporal grids (V0→τ and V1→τ), which are then fused with the predicted pointmaps using zero convolution in the Interp3R decoders (D0→τ, D1→τ).
Explicit time encoding is incorporated into the prediction heads to condition the model on the target interpolation time τ ∈ (0, 1), ensuring more constant performance across different target times.

Training Objective

The training objective is defined by a loss function that combines interpolation errors from both temporal directions and confidence terms:

)&Linterp = Xv∈[0,1] (Cv→τ / z) Xv→τ − 1/z¯ X¯v→τ - α log Cv→τ (3)

where the loss is the sum of interpolation errors (3D weighted Euclidean distance) from both temporal directions. The model is trained exclusively on synthetic data, and the Interp3R component is trained while other components (like Align3R) are frozen initially, gradually injecting information about source pointmaps and motion to guarantee training stability.

Coarse-to-Fine Global Alignment

After obtaining pointmaps for all frame and interpolation timestamps, a coarse-to-fine global alignment is performed to solve for the expected camera intrinsics, poses, and depth at all those timestamps:

The coarse alignment aligns the source pointmaps (from Align3R) to calculate the initial variables Θ0 and Θ1.
In the fine stage, the interpolated pointmaps produced by Interp3R are aligned with these source pointmaps to solve for variables at the target time, Θτ.
This process is repeated by symmetrizing consecutive image pairs and event sets to obtain new pairs (I1, I0) with (E1→τ, E0→τ), allowing the recovery of depth and pose at t = τ in the fine-alignment stage.

Contributions and Generalization

The paper highlights several key contributions:

** It is the first method that extends pointmap-based models to continuous time 3D geometry estimation.**

** It demonstrates compatibility with different pointmap-based methods (e.g., MonST3R and Align3R), enhancing them to the continuous-time capability.**

** The model is trained exclusively on a synthetic dataset but demonstrates strong generalization across a wide range of synthetic and real-world datasets.**

Extensive experiments show that Interp3R outperforms baselines that follow a two-stage pipeline of 2D video frame interpolation followed by 3D geometry estimation. The method consistently achieves the best performance on PointOdyssey, Sintel, and TUM across all skip settings in depth estimation and camera pose evaluation.

Limitations

The paper notes two primary limitations:

The interpolation quality may degrade if the source pointmaps are unreliable due to motion blur or challenging illumination conditions in the input frames. This could be mitigated by leveraging event data for event-guided deblurring or image enhancement.
If the scene contains flickering or rapidly changing light sources, the resulting events may not correspond to true scene motion, potentially introducing artifacts and affecting interpolation results. This limitation could be addressed by detecting and removing such nonmotion events or injecting additional knowledge.

Compatibility with Other Models

The study confirms that Interp3R is compatible with other pointmap models by showing that replacing Align3R with MonST3R (resulting in MonST3R + Interp3R) yields comparable depth accuracy while often outperforming the original model in terms of ATE and RTE. This demonstrates its plug-n-play manner capability.

Improvements for AI systems

Based on the provided research paper, Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events, here are specific improvements that can be made to existing AI systems, along with what those improved systems can achieve:

The proposed Interp3R framework fundamentally enhances 3D visual foundation models by extending their capability from discrete frame estimation to continuous-time geometric reconstruction. The key improvements are detailed below:

Improve the temporal consistency of 3D geometry and camera pose estimation in dynamic scenes across large temporal gaps.
Enable robust, temporally continuous scene evolution modeling by integrating sparse event data into frame-based pointmap models for intermediate time instants.
Achieve superior accuracy in depth and camera pose recovery, particularly under conditions of fast motion or large frame intervals (high temporal skips).

The improved AI system based on Interp3R can perform the following specific tasks:

Improve the fidelity of 3D reconstruction in long-duration video sequences (e.g., surveillance footage, drone footage) by accurately estimating scene geometry and camera poses at any arbitrary time instant, not just at captured frame times.
Enable seamless tracking and scene understanding during periods where conventional frame rates are insufficient (i.e., blind time between frames), resulting in a temporally continuous geometric representation of the environment.
Accurately predict depth maps and camera poses for intermediate time steps (interpolation) between two input frames by leveraging asynchronous event cameras, leading to more precise reconstructions than methods that rely solely on frame-based interpolation or traditional 2D video interpolation.
Maintain high accuracy in dynamic environments characterized by rapid motion or extreme lighting conditions, as the system uses event data to provide complementary motion cues that stabilize geometric estimation when RGB frames are degraded.
Perform robust camera pose estimation (including rotation and translation) even when the frame-to-frame temporal gap is large, as demonstrated by maintaining low Absolute Trajectory Error (ATE) and Relative Rotation Error (RRE) across various skip values.

Abstract

In recent years, 3D visual foundation models, pioneered by pointmap-based approaches such as DUSt3R, have attracted a lot of interest, achieving impressive accuracy and strong generalization across diverse scenes. However, these methods are inherently limited to recovering scene geometry only at the discrete time instants when images are captured, leaving the scene evolution during the blind time between consecutive frames largely unexplored. We introduce Interp3R, to the best of our knowledge, the first method that enhances pointmap-based models to estimate depth and camera poses at arbitrary time instants. It leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous geometric representations. Depth and camera poses are then jointly recovered by aligning the interpolated pointmaps together with those predicted by the underlying frame-based models into a consistent spatial framework. We train Interp3R exclusively on a synthetic dataset, yet demonstrate strong generalization across six datasets, both synthetic and real. Compared with the best two-stage baseline, Interp3R reduces absolute relative depth error by 15%-32% on DSEC and absolute trajectory error by up to 51% on EDS.