Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings
cs.AI, cs.CL
Submitted: 2026-09-14
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Universal multimodal embedding (UME) learns unified representations across modalities, enabling a single model to support diverse retrieval tasks.
Terminology
Abstract
Universal multimodal embedding (UME) learns unified representations across modalities, enabling a single model to support diverse retrieval tasks. Recent methods use Chain-of-Thought (CoT) reasoning to better interpret multimodal inputs before generating embeddings for complex retrieval tasks and further optimize this reasoning process through GRPO with retrieval-based rewards. However, two limitations hinder corpus-scale deployment. GRPO assigns all CoT tokens the same advantage, without identifying input-supported claims or evidence that distinguishes the positive from negatives. Moreover, generating a complete CoT before each embedding introduces substantial latency, even when a partial trace already provides sufficient retrieval evidence. To address these limitations, we propose Reason What Matters (ReWAM), a retrieval-grounded reasoning framework that uses retrieval feedback to guide both credit assignment and reasoning computation. Specifically, we introduce Retrieval-aware Self-Distillation (RASD), which constructs privileged guidance from input-supported evidence that distinguishes the positive item from retrieved hard negatives. An on-policy self-teacher uses this guidance to refine trajectory-level feedback into token-specific supervision for retrieval-relevant reasoning. We further develop Retrieval-adaptive Inference (RAI), which uses a retrieval confidence head to estimate the remaining retrieval utility of a partial CoT. It stops unproductive traces early and accelerates useful continuations with speculative decoding. Extensive experiments on MMEB-V2 and MRMR demonstrate that ReWAM achieves state-of-the-art retrieval performance while delivering up to 5x the inference throughput of competitive explicit-CoT UME methods. These results bridge the gap between retrieval quality and inference efficiency, making reasoning-enhanced UME practical for large-scale deployment.
Sources
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- DFlash: Block Diffusion for Flash Speculative Decoding
- UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
- DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
- Better & Faster Large Language Models via Multi-token Prediction
- PLUME: Latent Reasoning Based Universal Multimodal Embedding
- Reinforcement Learning via Self-Distillation
- Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings
- E5-V: Universal Embeddings with Multimodal Large Language Models
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
- Self-Distilled Agentic Reinforcement Learning
- VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- A Survey of On-Policy Distillation for Large Language Models
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
- Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
- LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck
- Self-Distilled RLVR
- OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection