Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

arXiv:2608.05000 · cs.CV, cs.LG, cs.MM · Submitted 2026-08-05 · Read on arXiv

Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis

cs.CV, cs.LG, cs.MM

Submitted: 2026-08-05

Comments: Project page: https://junlinhan.github.io/projects/physics_of_mm_pretrain/

Project page: https://junlinhan.github.io/projects/physics_of_mm_pretrain

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 33× in this paper · explained in 1 episode
• Fine-Tuning — 32× in this paper · explained in 5 episodes
• Alignment — 24× in this paper · explained in 1 episode
• Transformer — 19× in this paper · explained in 2 episodes
• Benchmark — 17× in this paper · explained in 2 episodes
• Perplexity — 15× in this paper · explained in 3 episodes
• Foundation Model — 13× in this paper · explained in 3 episodes
• Decoupling — 10× in this paper · explained in 2 episodes
• Foundation Models — 10× in this paper · explained in 7 episodes
• Tokenization — 10× in this paper · explained in 2 episodes
• Core — 7× in this paper · explained in 2 episodes
• Zero-Shot Transfer — 7× in this paper · explained in 4 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Generative Modeling — 5× in this paper · explained in 3 episodes
• Embeddings — 4× in this paper · explained in 2 episodes
• Generalization — 4× in this paper · explained in 13 episodes
• Latent Space — 4× in this paper · explained in 7 episodes
• World Model — 4× in this paper · explained in 7 episodes
• Attention Mechanism — 3× in this paper · explained in 3 episodes
• Catastrophic Forgetting — 3× in this paper · explained in 5 episodes
• Fidelity — 3× in this paper · explained in 2 episodes
• Flow Matching — 3× in this paper · explained in 8 episodes
• Multimodal Learning — 3× in this paper · explained in 2 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• Diffusion Model — 2× in this paper · explained in 8 episodes
• Generative Model — 2× in this paper · explained in 2 episodes
• Generative Models — 2× in this paper · explained in 2 episodes
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Multimodal Reasoning — 2× in this paper · explained in 5 episodes
• Rectified Flow — 2× in this paper · explained in 3 episodes
• Representation Learning — 2× in this paper · explained in 2 episodes
• Scaling Laws — 2× in this paper · explained in 4 episodes
• Semantic Understanding — 2× in this paper · explained in 2 episodes
• Alignment Scores — 1× in this paper · explained in 1 episode
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Grouped Query Attention — 1× in this paper
• Late Fusion — 1× in this paper · explained in 2 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Latent Representations — 1× in this paper · explained in 3 episodes
• Multimodal Integration — 1× in this paper · explained in 2 episodes
• Nash Equilibrium — 1× in this paper · explained in 2 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Transferability — 1× in this paper · explained in 4 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• CM3: A Causal Masked Multimodal Model of the Internet
• Mirage Probes: How Vision Models Fake Visual Understanding
• Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
• BLIP3o-NEXT: Next Frontier of Native Image Generation
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
• IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
• Emerging Properties in Unified Multimodal Pretraining
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
• Lance: Unified Multimodal Modeling by Multi-Task Synergy
• Planting a SEED of Vision in Large Language Model
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
• Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
• TV2TV: A Unified Framework for Interleaved Language and Video Generation
• Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models