Event-based Continuous Color Video Decompression from Single Frames

This paper presents ContinuityCam, a novel approach to generate continuous video from a single static RGB image and an event camera stream, addressing limitations in high-speed motion capture by leveraging event cameras' ability to encode compressed change information at high temporal resolution. This method is significant because it combines continuous long-range motion modeling with a neural synthesis model to reconstruct temporally continuous videos without requiring subsequent sharp frames, thereby increasing robustness to sudden motions and minimizing prediction latency.

The gist: Our approach features two ways that encode long-term continuous videos, using a neural synthesis module that factorizes the continuous spatiotemporal feature into three feature planes, and a continuous trajectory field module that parameterizes dense pixel trajectories with motion priors. <ref:2312.00113#pg6>

How it works

The core of the method involves two main branches: a neural synthesis module and a continuous trajectory field module. The synthesis module encodes event-based spatiotemporal features, while the motion estimation module computes time-continuous nonlinear trajectories parameterized by learned priors. These two branches merge into a multiscale feature fusion network that flexibly generates color images at any desired timestamp <ref:2312.00113#pg7>.

The continuous trajectory field module models long-range motions using a continuous-time function to represent the event trajectory field <ref:2312.00113#pg5>. This is achieved by replacing discrete-time motion basis functions with a set of learned basis functions, allowing the network to map quasi-continuous motion information of events to continuous trajectories, where the trajectory corresponding to a seed point ui is given by xi(t) = PNb k=1 αk(ui)g θk(t), (2) <ref:2312.00113#pg5>. The selection of a motion basis offers possibilities including the Discrete Cosine Transform [33, 63], Fourier basis [32], and polynomial basis, with the network employing a learned multi-layer perceptron (MLP) to model the motion basis and optimize it during training <ref:2312.00113#pg6>.

Event Features Encoding for Neural Synthesis

Reconstructing continuous videos requires a compact feature space, which is addressed by using an Event-based Tri-Planes Feature Encoding. This involves assuming that characteristics are encoded on three orthogonal planes, x-y, x-t, y-t <ref:2312.00113#pg7>. These three multichannel images contain the feature information for the continuous video field on discretized grids; specifically, the resolution of the time dimension in gxt and gyt is sufficiently large for fine-grained temporal information <ref:2312.00113#pg7>.

The synthesis operation is performed once to reconstruct a short video clip, and the lightweight decoder can predict in parallel with little computational cost <ref:2312.00113#pg7>. The decoded image value is formally written as ˆIτ (x, y)=ϕ gxy(πxy(q)), gxt(πxt(q)), gyt(πyt(q)) where q = (x, y, τ)⊤ <ref:2312.00113#pg7>.

Latent-Frame Flow Refinement

The continuous flow field captured in the trajectory field module is refined using a novel latent frame model that takes advantage of the iterative matching power of frame-based flow networks. This involves obtaining an intermediate latent frame ˆIt through the neural event integration module <ref:2312.00113#pg7>. This latent frame is then used for computing the latent-frame flow via iterative flow refinement using RAFT [60] <ref:2312.00113#pg7>. The correlation volume in (4) resembles a matching process without a motion model, allowing matching at any two arbitrary times <ref:2312.00113#pg7>.

Multi-scale Feature Fusion

The three main outputs from the intermediate networks are fused via a multiscale feature fusion network <ref:2312.00113#pg5>. The input to the fusion model is G.= C TMt (Ψ0), TMt (I0), TM˜t (I0), TM˜t (Ψ0), Iˆt) <ref:2312.00113#pg6>. The final image prediction is passed through the multi-level merging network fm, implemented by a series of convolution layers with small receptive field and non-linear activation, to produce the final image prediction It.= fm(G) <ref:2312.00113#pg6>. This process uses Softmax Splatting [42] to warp images and features at each pyramid scale, with learned multiscale splatting weights predicted along with motion parameters <ref:2312.00113#pg6>.

Training

The training strategy involves a sequential approach where the trajectory field branch and synthesis branch are trained independently first, and then jointly optimize a shared feature fusion network <ref:2312.00113#pg8>. For the K-plane synthesis network, it is trained for 20,000 iterations with a learning rate of 10−4 with an Adam optimizer <ref:2312.00113#pg8>. The motion network is trained for 15,000 iterations with a learning rate of 10−4 with an Adam optimizer <ref:2312.00113#pg8>. In the final training, the entire network is trained 100,000 steps with the same learning rate and optimizer configuration as the motion network <ref:2312.00113#pg8>.

Evaluation

The method is evaluated on BS-ERGB [62] and E2D2, reporting Peak-Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) [75], and Structural Similarity (SSIM) [64] <ref:2312.00113#pg5>. On the E2D2 dataset, ContinuityCam outperforms baselines by 3.61 dB in PSNR and by 33% decrease in LPIPS <ref:2312.00113#pg5>. In downstream tasks, the decompressed method increases AprilTag detection by 20% and produces sharper Gaussian Splatting models <ref:2312.00113#pg5>.

Conclusion

ContinuityCam is a novel method for event-based continuous color video decompression that uses a single static RGB image and following events, enhancing robustness to sudden lighting changes, minimizing prediction latency, and reducing bandwidth requirements <ref:2312.00113#pg7>. The approach demonstrates state-of-the-art performance in event-based video color video decompression on standard and challenging datasets like E2D2. It also showcases practical applications in 3D reconstruction and camera fiducial tag detection, even under challenging lighting and motion conditions.

REFERENCES

[1] Ryad Benosman, Sio-Hoi Ieng, Charles Clercq, Chiara Bartolozzi, and Mandyam Srinivasan. Asynchronous frameless event-based optical flow. Neural Netw., 27:32–37, 2012.<ref:2312.00113#pg9>

[2] Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi. Event-based visual flow. IEEE Trans. Neural Netw. Learn. Syst., 25(2):407–417, 2014.<ref:2312.00113#pg9>

[3] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240x180 130dB 3µs latency global shutter spatiotemporal vision sensor. IEEE J. SolidState Circuits, 49(10):2333–2341, 2014.<ref:2312.00113#pg9>

[4] Christian Brandli, Lorenz Muller, and Tobi Delbruck. Realtime, high-speed video decompression using a frame- and event-based DAVIS sensor. In IEEE Int. Symp. Circuits Syst.(ISCAS), pages 686–689, 2014.<ref:2312.00113#pg9>

[5] Tobias Brosch, Stephan Tschechne, and Heiko Neumann. On event-based optical flow detection. Front. Neurosci., 9(137), 2015.<ref:2312.00113#pg9>

[6] Kenneth Chaney, Fernando Cladera Ojeda, Ziyun Wang, Anthony Bisulco, M. Ani Hsieh, Christopher Korpela, Vijay Kumar, Camillo Jose Taylor, and Kostas Daniilidis. M3ED: Multi-robot, multi-sensor, multi-environment event dataset. In IEEE Conf. Comput. Vis. Pattern Recog.

1. Bold Trajectory Field Modeling: The system can model continuous long-range motions up to 1 second by replacing discrete-time motion modeling with a continuous-time function representation of event trajectories, as described in equation (2): xi(t) = PNb k=1 αk(ui)g θk(t), (2).

2. Bold Tri-Plane Feature Encoding: The system reduces computational burden by encoding continuous video features onto three orthogonal planes, x-y, x-t, y-t, allowing the decoder to synthesize images using equation (3): ˆIτ (x, y)=ϕ gxy(πxy(q)), gxt(πxt(q)), gyt(πyt(q)).

3. Bold Latent Frame Flow Refinement: The system achieves spatially consistent warping by utilizing a novel latent frame model that takes advantage of the iterative matching power of frame-based flow networks to refine noisy event-based flow fields, as detailed in Section 3.3.

4. Bold Multi-Scale Feature Fusion: The final high-quality color image prediction is achieved by fusing the outputs from the motion module, synthesis module, and latent frame refinement through a multiscale feature fusion network (Sec. 3.4) that uses Softmax Splatting [42] to warp images and features at each pyramid scale.

5. Bold Robustness to Frame Degradation: The system demonstrates enhanced accuracy in reconstruction even when frames are degraded, as shown by the claim that Our method is able to reconstruct in these scenarios due to the removal of the dependency on the second frame, addressing issues where interpolation methods fail due to sharp frames followed by blurry frames.

6. Bold Downstream Task Enhancement: The decompressed video can be used for practical applications such as 3D reconstruction and fiducial tag detection, achieving specific gains like an increase in AprilTag detection by 20% and produces sharper Gaussian Splatting models, as noted in the evaluation results.

We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image and an event camera stream. Conventional cameras struggle with high-speed motion capture due to bandwidth and dynamic range limitations. Event cameras are ideal sensors to solve this problem because they encode compressed change information at high temporal resolution. In this work, we tackle the problem of event-based continuous color video decompression, pairing single static color frames and event data to reconstruct temporally continuous videos. Our approach combines continuous long-range motion modeling with a neural synthesis model, enabling frame prediction at arbitrary times within the events. Our method only requires an initial image, thus increasing the robustness to sudden motions, light changes, minimizing the prediction latency, and decreasing bandwidth usage. We also introduce a novel single-lens beamsplitter setup that acquires aligned images and events, and a novel and challenging Event Extreme Decompression Dataset (E2D2) that tests the method in various lighting and motion profiles. We thoroughly evaluate our method by benchmarking color frame reconstruction, outperforming the baseline methods by 3.61 dB in PSNR and by 33% decrease in LPIPS, as well as showing superior results on two downstream tasks.