From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning
cs.MM, cs.AI, cs.CL, cs.CV, cs.LG
Submitted: 2026-09-16
Updated: 2026-09-16
Comments: TMLR
Journal ref: Transactions on Machine Learning Research, 2026
Code: https://github.com/pwang322/Efficient-Multimodal-Learning-Survey
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: The rapid expansion of multimodal models has surfaced formidable bottlenecks in computation, memory, and deployment, catalyzing the rise of Efficient Multimodal Learning (EML) as a pivotal research
Terminology
Abstract
The rapid expansion of multimodal models has surfaced formidable bottlenecks in computation, memory, and deployment, catalyzing the rise of Efficient Multimodal Learning (EML) as a pivotal research frontier. Despite intensive progress, a cohesive understanding of what, how, and where efficiency is manifested across the learning stack remains fragmented. This survey systematizes the EML landscape by introducing the first structured, model-to-system taxonomy. We distill insights from over 300 seminal works into three hierarchical levels--model, algorithm, and system--addressing architectural parsimony, execution refinement, and hardware-aware orchestration, respectively. Moving beyond a purely categorical review, we offer a methodological synthesis of the vertical synergies between these layers, elucidating how cross-layer co-design contributes to the fundamental "Efficiency-Utility-Privacy" trade-off. Through an integrative case study of Multimodal Large Language Models (MLLMs), we trace the field's evolutionary trajectory from initial structural adjustments to modern full-stack resource orchestration. Furthermore, we provide a holistic discussion and application-specific optimization blueprints for diverse domains and posit a paradigm shift toward self-regulating intelligence, where efficiency is an intrinsic, emergent property of the model's fundamental design rather than a post-hoc constraint. Finally, we present open challenges and future directions that will define the trajectory of EML research. This survey establishes a structured framework for multimodal systems that are not only high-performing and generalizable but natively efficient and ready for ubiquitous deployment. A continuously updated version is available at https://github.com/pwang322/Efficient-Multimodal-Learning-Survey.
Sources
- Qwen Technical Report
- CAPEEN: Image Captioning with Early Exits and Knowledge Distillation
- Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast
- Longformer: The Long-Document Transformer
- Towards Audio Token Compression in Large Audio Language Models
- LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
- Token Merging: Your ViT But Faster
- PuMer: Pruning and Merging Tokens for Efficient Vision Language Models
- COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition
- Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos
- LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
- Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- High Fidelity Neural Audio Compression
- Moshi: a speech-text foundation model for real-time dialogue
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- CROME: Cross-Modal Adapters for Efficient Multimodal LLM
- Depth-Adaptive Transformer
- Not All Layers of LLMs Are Necessary During Inference
Related papers
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
- Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection
- A Rate-Distortion-Classification Approach for Lossy Image Compression