Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation
cs.CL, cs.CV, cs.SD
Submitted: 2026-08-10
Updated: 2026-08-10
Comments: 14 pages, 3 figures, 6 tables. Models: https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview and https://huggingface.co/EximiusLabs/fusion-embedding-2-ember. Code: https://github.com/Eximius-Labs/fusion-embedding
Code: https://github.com/Eximius-Labs/fusion-embedding
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs.
Terminology
Abstract
Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient methods silently changes those outputs; LoRA-style adaptation rewrites the text path whether or not the weights are merged, invalidating every stored embedding. We propose modality-gated deep adapters: bottleneck adapters attached to every decoder layer of a frozen multimodal embedding LLM, grouped into per-modality packs that execute only while their own modality is being encoded. The result is a modality added with zero change to existing outputs: inputs no pack claims traverse the base model's own computation graph, bit-for-bit unchanged, and co-loaded packs compose with an exact-zero isolation matrix. Both properties are stated as propositions, hold after arbitrary training rather than only at initialization, require no task labels or routing metadata at inference, and are verified by exact-equality tests on the released checkpoints. On one frozen 2B base, the audio pack (injected as connector tokens) improves audio-to-text R@10 by +3.4 to +5.4 points over an identically trained control, positive at every seed and reproduced at eleven times the data; the thermal pack, reusing the base's own frozen vision path, clears its pre-registered acceptance gate roughly sevenfold at every seed and lifts thermal-to-text R@10 from 0.224 to 0.785. An encoder swap locates the missing capacity: an external audio encoder that outranks Whisper-family encoders in CLAP-style comparisons loses by 16 R@10 points inside the frozen LLM, so the capacity belongs in the layers, exactly where the gated adapters place it. We release the audio model, the thermal pack, and the training, evaluation and invariance suites: models at huggingface.co/EximiusLabs, code on GitHub.
Sources
- Flamingo: a Visual Language Model for Few-Shot Learning
- ReZero is All You Need: Fast Convergence at Large Depth
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
- VGGSound: A Large-scale Audio-Visual Dataset
- Net2Net: Accelerating Learning via Knowledge Transfer
- GLAP: General contrastive audio-text pretraining across domains and languages
- Scaling up masked audio encoder learning for general audio classification
- Clotho: An Audio Captioning Dataset
- ImageBind: One Embedding Space To Bind Them All
- Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
- jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
- Parameter-Efficient Transfer Learning for NLP
- LoRA: Low-Rank Adaptation of Large Language Models
- LLVIP: A Visible-infrared Paired Dataset for Low-light Vision
- Matryoshka Representation Learning
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding
- WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
- VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering