jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
summary
The gist
The paper introduces GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to constructing multimodal embedding models.
In short
The episode discusses the "jina-embeddings-v5-omni" paper, which introduces the GELATO method for creating multimodal embeddings. By using frozen vision and audio encoders connected to a text model via small projectors, researchers preserve existing text knowledge and achieve high efficiency, allowing for interleaved text, image, and audio processing.
Key concepts
- Geometry-preserving
- This refers to keeping the existing "map" of text knowledge intact while adding new senses like vision or audio. It ensures that adding new information does not corrupt or move the established relationships within the original text model, maintaining the foundation's stability.
- Locked Aligned Towers
- A method where specialized vision and audio encoders are kept frozen during training. Instead of retraining the whole model, researchers only train the connection, or "projector," between these encoders and the text model to align them without destroying existing knowledge.
- Matryoshka feature
- A technique that allows embeddings to be shrunk while keeping the most important information in the first few dimensions. This provides a robust way to handle data compression, making it easier to manage and process large amounts of data.
Terminology used across episodes
This episode discusses
- jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers · Paper Radio
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- Qwen3-VL Technical Report
- e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
- Enhancing Vision Foundation Models via Multimodal Continual Pre-Training
- Qwen2.5-Omni Technical Report
- MAEB: Massive Audio Embedding Benchmark
- MMTEB: Massive Multilingual Text Embedding Benchmark
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- E5-V: Universal Embeddings with Multimodal Large Language Models
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
- jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images
- Jina CLIP: Your CLIP Model Is Also Your Text Retriever
- A resource- and computationally-efficient protocol for multipartite entanglement distribution in Bell-pair networks
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
- Nomic Embed Vision: Expanding the Latent Space
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Multilingual E5 Text Embeddings: A Technical Report
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
The paper
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers · Read on arXiv
Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, Han Xiao
Jina by Elastic
In this work, we introduce GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a language model, which in turn generates embeddings for all varieties of input. We present the result: the jina-embeddings-v5-omni suite, a pair of models that encode text, image, audio, and video input into a single semantic embedding space. GELATO extends the two Jina Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Training is therefore much more efficient than full-parameter retraining. Additionally, the language model remains effectively unaltered, producing exactly the same embeddings for text inputs as the Jina Embeddings v5 Text models. Our evaluations show that GELATO produces results that are competitive with the state-of-the-art, yielding nearly equal performance to larger multimodal embedding models.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Next we'll be talking about the paper "jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers".
Jane: The paper was written by Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua et al. from Jina by Elastic.
Tom: Stay tuned as we take you through the paper and discuss its implications.
Title: Tom: We're starting our show with a look at a new paper called "jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers."
Jane: That title is definitely a mouthful for our listeners.
Tom: It is, but it actually describes exactly what the researchers at Jina eye are doing.
Jane: Can you explain what they mean by "geometry-preserving" in plain English?
Tom: Imagine you have a perfect map of a city where every street is in the right place.
Jane: So they want to add new landmarks without moving the existing streets?
Tom: That's a great way to put it.
Jane: And the researchers, like Florian Hönicke and Michael Günther, are trying to keep that text "map" perfect.
Lu: This approach could allow us to layer new senses onto our digital world without destroying the foundation we've already built.
Meng: I'm curious if "locked towers" means they're avoiding the hard work of actually teaching the model new things.
Lu: It's more about being smart with what we've already learned, Meng.
Lalam: It creates a unified way for machines to experience the rhythm of speech and the color of images simultaneously.
Jane: The "locked" part refers to the fact that they keep the vision and audio encoders frozen.
Tom: They aren't changing the internal workings of those specialized encoders at all.
Jane: They only train the connection between those encoders and the text model.
Tom: That's where the "alignment" part of the title comes in.
Lu: It's a brilliant way to respect the specialized knowledge already inside those pre-trained models.
Meng: I see the appeal, but does keeping them "locked" limit how well they can learn to talk to the text model?
Lu: It actually prevents the new information from corrupting the existing text knowledge.
Jane: It's like building a new bridge to an existing city instead of tearing the city down to build a new road.
Tom: That stability is why they focused so much on preserving the geometry.
Lalam: This allows the digital world to expand its senses without losing its original voice.
Jane: It sounds like they are prioritizing compatibility over everything else.
Tom: Let's see how they actually pull off that feat in the summary.
Summary: Tom: We've touched on the concept, but now we need to talk about how the GELATO method actually works.
Jane: The paper explains that they use these small "projectors" to bridge the gap between modalities.
Tom: They aren't just using any encoders, either.
Jane: They're using Qwen3 point 5 for vision and Qwen2 point 5-Omni for audio.
Tom: These are already very smart, language-aligned encoders.
Jane: So the projector just has to translate their output into the text model's hidden space?
Tom: Precisely, and they've made it incredibly efficient.
Meng: How much of the model are we actually talking about when we say "efficient"?
Tom: They only trained about zero point three five percent of the total weights.
Meng: That's an incredibly small fraction of the parameters.
Lu: It's a very surgical approach to machine learning.
Jane: They also use these special "delimiter tokens" to tell the model where an image or an audio clip starts and ends.
Tom: It's like putting brackets around a quote in a sentence.
Lu: It allows the model to process a sequence that might have text, then an image, then more text.
Meng: Does this mean they can handle video too?
Tom: They do, by treating a video as a sequence of sampled frames.
Jane: It's a very modular way of thinking about data.
Lalam: It's how we move toward a truly omni-modal understanding of human culture.
Tom: But does this efficient approach actually deliver high-quality results?
Improvements: Tom: The results in "jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers" are quite striking.
Jane: I was particularly struck by the ViDoRe scores for document retrieval.
Tom: The nano version actually hit seventy-nine point two five.
Jane: And it did that with only zero point three one billion active parameters.
Meng: That's much more efficient than the LCO-Omni-7B model, which uses nearly nine billion parameters.
Lu: But we have to be honest about the video performance.
Tom: You're right, the video scores are definitely the weak point here.
Jane: They also looked at the "modality gap" using UMAP plots.
Tom: Those plots show how the different types of data live in the same space.
Jane: In some models, text and images live in completely separate clusters.
Tom: But in the Jina models, they're all interleaved and mixed together.
Lu: That's the "geometry-preserving" part in action.
Meng: I noticed they also did an ablation on whether to unfreeze the encoders.
Tom: Yeah, they found that unfreezing the vision encoder actually made things worse.
Jane: It destabilized the whole system because the projector wasn't ready yet.
Meng: That makes sense; you don't want to change the foundation while you're still building the walls.
Lu: They did find that a two-stage process for audio might be a way to improve things later.
Jane: And we can't forget the Matryoshka feature, which lets you shrink the embeddings.
Tom: It keeps the important information in the first few dimensions.
Jane: It's a very robust way to handle data compression.
Tom: It seems like they've found a very stable way to scale up.
Conclusion: Tom: We're wrapping up our look at "jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers."
Jane: It's been a fascinating look at how to expand eye's senses efficiently.
Tom: They've shown that you don't need to retrain the whole world to add new modalities.
Lu: This modularity is going to change how we approach multimodal research.
Meng: It definitely makes the engineering side of things a lot more manageable.
Lalam: It brings us closer to a digital intelligence that perceives the world as a single, cohesive experience.
Tom: Thanks for joining us, we'll see you at the next paper.
More episodes
- 2610.10857-Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning
- 2610.10768-Strategic Investment Decision Making for Value Creation in Energy Transition: A Reinforcement Learning Approach
- 2610.10858-RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design
- 2610.10613-Temporal transformer CAN encoder with federated lightweight heads for anomaly detection
- 2610.10616-When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
- 2610.10655-Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning
- 2610.11031-Language Modeling is Monotone Compression
- 2610.01253-Context-Aware Error Mitigation Orchestration for Hybrid Quantum Reinforcement Learning on NISQ Systems
- 2604.24201-CMGL: Confidence-guided Multi-omics Graph Learning for Cancer Subtype Classification
- 2609.34069-Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization