MambaVF: State Space Model for Efficient Video Fusion

Video fusion is a fundamental technique in various video processing tasks, but existing methods heavily rely on optical flow estimation and feature warping, resulting in severe computational overhead and limited scalability. This paper presents MambaVF, an efficient video fusion framework based on state space models (SSMs) that performs temporal modeling without explicit motion estimation.

The gist

MambaVF proposes a novel and efficient SSM-driven video fusion framework that eliminates the need for optical flow and feature warping by leveraging a hidden state transition mechanism to implicitly capture temporal dynamics, achieving state-of-the-art performance while reducing parameters by 92.25% and computational FLOPs by 88.79%.

How it works

MambaVF reformulates video fusion as a sequential state update process to capture long-range temporal dependencies with linear complexity, significantly reducing computation and memory costs compared to flow-based approaches. The framework consists of three main stages: first, input videos are processed by a patch embedding layer to map raw pixels into a latent embedding space; second, a dual-stream Tri-Axis Mamba Encoder extracts deep spatio-temporal features for each source independently through VSS blocks; and third, the features from both streams are concatenated and fed into a Mamba Decoder to reconstruct the final fused frames.

Key Architectural Components

The core innovation lies in replacing explicit motion estimation with an implicit temporal modeling mechanism. The model utilizes a dual-stream architecture to preserve the distinct characteristics of each source, beginning with a Tubelet Embedding layer to convert videos into sequences of spatiotemporal tokens, which are then passed through VSS blocks. To enhance content perception, MambaVF introduces an innovative spatio-temporal bidirectional (STB) scanning mechanism. This mechanism performs a comprehensive traversal across spatial coordinates, temporal frames, and feature channels, utilizing an 8-way scanning strategy based on different priority axes: Spatial-Priority Scanning (traversing across width W then height H), Temporal-Priority Scanning (treating the temporal dimension T as the primary axis), and Symmetrical Bidirectional Flipping for each path.

Fusion and Reconstruction

After deep feature extraction, the features from both streams, F1 and F2, are merged via channel-wise concatenation to form Ffused. This fused representation is then processed by the Mamba Video Decoder. The decoder utilizes a sequence of VSS blocks to characterize complex cross-modal dependencies within the latent space, followed by 2D Residual Blocks (He et al., 2016) to restore fine-grained textures and project the high-dimensional embeddings back to the pixel domain. Notably, since the decoder is optimized for reconstruction, it employs a 4-way spatial scanning strategy (i.e., imagedomain scanning) rather than the 8-way spatio-temporal traversal, concentrating state-space modeling on intra-frame structural refinement.

Performance and Efficiency

MambaVF was rigorously evaluated across four diverse benchmarks: multi-exposure fusion (MEF), multi-focus fusion (MFF), infrared-visible fusion (IVF), and medical video fusion (MVF). The model achieves performance comparable to state-of-the-art methods, demonstrating superior ability in recovering details in extreme conditions. In terms of efficiency, MambaVF requires only 7.75% of the parameters and 11.21% of the FLOPs compared to the leading flow-based baseline UniVF, achieving a 2.1× speedup. Ablation studies confirmed that the 8-way STB scanning achieves an optimal equilibrium, as a standard 2D spatial scan fails to model interframe dependencies, and a 1D temporal scan neglects fine-grained spatial textures. The framework is optimized with three VSS blocks and an embedding dimension of 32 for optimal trade-off between performance and latency.

Conclusion

MambaVF provides a new, highly efficient paradigm for video fusion by eliminating the need for optical flow and feature warping. Its proposed Spatio-Temporal Bidirectional (STB) scanning mechanism enables holistic understanding of video content across dimensions with linear complexity, establishing it as a promising baseline for deployment on resource-constrained platforms. The framework achieves state-of-the-art performance in multi-exposure, multi-focus, infrared-visible, and medical video fusion tasks while offering significant reductions in computational footprint.

References

Argaw, D. M., Liu, X., Chung, J. S., Liu, M.-Y., and Reda, F. Mambavideo for discrete video tokenization with channelsplit quantization. arXiv preprint arXiv:2507.04559, 2025.

Bai, H., Zhang, J., Zhao, Z., Wu, Y., Deng, L., Cui, Y., Feng, T., and Xu, S. Task-driven image fusion with learnable fusion loss.

Here are the specific improvements that MambaVF enables for AI systems, based on its core innovations:

1. Replacement of Optical Flow Estimation and Feature Warping: The system eliminates the need for computationally expensive, error-prone modules like optical flow estimation and explicit feature warping (which traditionally account for over 78% of runtime in methods like UniVF).

2. Achieving Linear Complexity for Temporal Modeling: By leveraging the State Space Model (SSM) architecture, MambaVF captures long-range temporal dependencies with linear complexity relative to sequence length, whereas Transformer-based approaches suffer from quadratic complexity.

3. High Efficiency and Real-Time Deployment: The system achieves significant computational savings (reducing FLOPs by 88.79% compared to UniVF) and a speedup of 2.1×, making it suitable for deployment on resource-constrained edge devices (smartphones, drones, wearable robots).

4. Robust Multi-Source Fusion: The framework can effectively fuse heterogeneous video sources (e.g., infrared-visible and visible light) or different acquisition conditions (multi-exposure and multi-focus) without relying on flow accuracy for alignment.

5. High Fidelity in Complex Scenarios: The introduction of the Spatio-Temporal Bidirectional (STB) scanning mechanism allows the model to holistically perceive video content across spatial coordinates, temporal frames, and feature channels, ensuring that salient textures and local structures are preserved during fusion.

The improved AI systems powered by MambaVF can perform the following specific tasks:

1. Organizing medical data for clinical diagnosis: By fusing MRI, CT, and PET scans without explicit motion estimation errors or heavy registration modules (as seen in the Medical Video Fusion task), the system can provide a unified, high-fidelity view of complex tissue structures.

2. Autonomous Navigation and Perception: In autonomous driving or robotic perception systems, MambaVF can fuse infrared-visible data with visible light data to create robust environmental representations that are invariant to illumination variations and weather degradation, leading to more reliable object detection and scene understanding.

3. Real-Time Video Processing on Edge Devices: The high efficiency allows for the implementation of complex video fusion algorithms directly on low-power hardware, enabling real-time video enhancement or surveillance applications where traditional methods are too slow or power-intensive.

4. High-Fidelity Multi-Modal Reconstruction: The system can generate synthesized fused videos for tasks like multi-exposure synthesis (recovering details in extremely dark/saturated regions) and multi-focus integration (maintaining sharp edges across complex depth-of-field changes).