CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
cs.CL
Submitted: 2026-01-29
Updated: 2026-09-22
Code: https://github.com/Z1zs/Causal-Embed
License: http://creativecommons.org/licenses/by/4.0/
The gist: Although Multimodal Large Language Models (MLLMs) have shown remarkable potential in Visual Document Retrieval (VDR) through generating high-quality multi-vector embeddings, the substantial storage
Terminology
Abstract
Although Multimodal Large Language Models (MLLMs) have shown remarkable potential in Visual Document Retrieval (VDR) through generating high-quality multi-vector embeddings, the substantial storage overhead caused by representing a page with thousands of visual tokens limits their practicality in real-world applications. To address this challenge, we propose an auto-regressive generation approach, CausalEmbed, for constructing multi-vector embeddings. By incorporating iterative margin loss during contrastive training, CausalEmbed encourages the embedding models to learn compact and well-structured representations. Our method enables efficient VDR tasks using only dozens of visual tokens, achieving a 30-155x reduction in token count while maintaining highly competitive performance across various backbones and benchmarks. Theoretical analysis and empirical results demonstrate the unique advantages of auto-regressive embedding generation in terms of training efficiency and scalability at test time. As a result, CausalEmbed introduces a flexible test-time scaling strategy for multi-vector VDR representations and sheds light on the generative paradigm within multimodal document retrieval. Our code is available at https://github.com/Z1zs/Causal-Embed.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends
- PaliGemma: A versatile 3B VLM for transfer
- ReinPool: Reinforcement Learning Pooling Multi-Vector Embeddings for Retrieval System
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Think Then Embed: Generative Context Improves Multimodal Embedding
- ColPali: Efficient Document Retrieval with Vision Language Models
- Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
- The Llama 3 Herd of Models
- Training Large Language Models to Reason in a Continuous Latent Space
- RzenEmbed: Towards Comprehensive Multimodal Retrieval
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
- M3DR: Towards Universal Multilingual Multimodal Document Retrieval
- LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
- LLaVA-OneVision: Easy Visual Task Transfer
- Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering