Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval

This paper presents the first direct comparison of natively multimodal embedding models against frontier LLMs as zero-shot direct visual rankers for text-to-image retrieval. The study evaluates four systems: Gemini Embedding 2 (GE2), Amazon Nova 2, GPT-4.1, and Claude Sonnet 4.6, using hard-negative candidate sets constructed from Flickr30k.

The authors sampled a pool of 5,000 images from Flickr30k, each paired with five independent human-written captions. They applied stratified caption sampling, partitioning images into five equally-sized groups of 1,000, each assigned exclusively to one caption index (0 to 4). For each of the 5,000 pool images, they constructed a candidate set of 25 images: the ground-truth image and the 24 hardest negatives. Hard negatives were selected by embedding all representative captions with sentence-transformers/all-mpnet-base-v2, a general-purpose text encoder intentionally decoupled from all evaluated retrieval systems. They computed a 5,000×5,000 pairwise cosine similarity matrix and retained the 24 most similar captions to the query. From the full pool, they randomly sampled 1,000 captions as queries.

The authors note: "We construct hard negatives in caption embedding space rather than image embedding space for two reasons. First, grounding distractor difficulty in caption-space similarity increases ambiguity, making images hard to distinguish given the caption, which is precisely the retrieval difficulty we wish to evaluate. Second, since human perception of image similarity is naturally expressed through language, using caption similarity to construct candidate sets is a well-motivated design choice."

They report three metrics over 1,000 queries: Recall@1 (R@1), Recall@3 (R@3), and Mean Reciprocal Rank (MRR), with 95% bootstrap confidence intervals. For pairwise significance testing of R@1, they applied McNemar's test; for MRR, they applied the Wilcoxon signed-rank test, using a significance threshold of α = 0.05 with Bonferroni correction.

The results show that GE2 achieves the highest score across all three metrics, followed closely by Claude Sonnet 4.6 and GPT-4.1, with Amazon Nova 2 trailing by 13 percentage points. Specifically, GE2 achieved R@1 of 0.804, Claude Sonnet 4.6 achieved 0.796, GPT-4.1 achieved 0.789, and Amazon Nova 2 achieved 0.673.

The primary finding is stated as: "The absence of a statistically significant gap between the embedding paradigm (Gemini Embedding 2) and the LLM reasoning paradigm (GPT-4.1, Claude Sonnet 4.6) is the primary finding of this study (given the dataset and candidate set). Pairwise significance testing revealed that Gemini Embedding 2, GPT-4.1, and Claude Sonnet 4.6 are statistically indistinguishable from one another on both R@1 (McNemar, all p > 0.25) and MRR (Wilcoxon, all p > 0.09), while the performance of Amazon Nova 2 is significantly different from every other system (p < 0.0001 on both tests)."

The authors attribute Amazon Nova 2's lower performance to the use of GENERIC INDEX as the embedding purpose, which is designed for general vector database creation rather than optimized text-image matching. They note that the Amazon Bedrock API offers a dedicated IMAGE RETRIEVAL purpose that may yield stronger performance, leaving this to future work.

The paper highlights a critical practical distinction regarding precomputation. Table 2 shows that once embeddings are precomputed, online ranking latency for 1,000 queries and 25 candidates drops to under 2 seconds for both Gemini Embedding 2 (0.66 seconds) and Amazon Nova 2 (1.16 seconds), compared to 6,100 seconds and 9,415 seconds for GPT-4.1 and Claude Sonnet 4.6, respectively. The authors state: "This speedup is four orders of magnitude faster. For applications requiring sub-second retrieval over a large but fixed set of text and/or images, embedding models are the better choice (especially GE2 because of high recall scores)."

They also note: One critical distinction is that embedding model costs are incurred once, when creating the embeddings, and amortized across all subsequent queries. On the other hand, LLM costs scale linearly with every query. However, they acknowledge that LLM-based rankers may be preferable when the image gallery is frequently updated or small, rendering precomputation impractical or its amortized benefit negligible.

The paper includes qualitative examples showing that GE2 retrieved the correct image despite it being dark, while Nova 2 and GPT-4.1 selected the same image as their top-ranked image. Additional analysis reveals cases where both LLMs correct, both embedding models wrong illustrate the advantage of cross-candidate joint reasoning: the LLM can compare all 25 images simultaneously and apply semantic or relational understanding that independent embedding scoring cannot replicate. Conversely, cases where both embedding models correct, both LLMs wrong suggest that dense cross-modal embedding captures visual correspondence that joint visual reasoning consistently misses.

The authors acknowledge several limitations: caption-image correspondence ambiguity (a caption from one image may plausibly describe another), hard negatives defined in caption space may not be equally hard in image embedding space, different inference conditions between paradigms (LLMs see all candidates simultaneously while embedding models score independently), proprietary model access through paid APIs, and the 25-image candidate set measuring ranking rather than full-gallery retrieval. They also note that changes to LLM prompts may affect ranking results.

The authors conclude: "Gemini Embedding 2, GPT-4.1, and Claude Sonnet 4.6 achieve statistically indistinguishable retrieval accuracy on all three metrics - Recall@1, Recall@3, and MRR. The practical implication favors multimodal embedding models for low-latency applications. They suggest Extending this comparison to additional datasets and to video and audio modalities are natural directions for future work."

Improvements to AI Systems:

1. Hybrid Retrieval-Ranking Architecture: Build a two-stage system where a multimodal embedding model (e.g., GE2) precomputes and retrieves a shortlist of 25–50 candidates in under a second, then a frontier LLM (e.g., GPT-4.1 or Claude Sonnet 4.6) performs joint cross-candidate reasoning on that shortlist to re-rank the final results. This combines the sub-2-second latency of embeddings with the LLM’s ability to resolve ambiguity via simultaneous comparison, addressing the paper’s finding that LLMs excel when joint reasoning is needed.

2. Adaptive Paradigm Selection Based on Gallery Dynamics: Create a controller that monitors image gallery update frequency and query volume. If the gallery is static and large (>10k images), default to embedding-only retrieval (GE2) for cost/latency amortization. If the gallery is small (<1k images) or frequently updated, switch to LLM-based ranking to avoid precomputation overhead, using the paper’s latency/cost tradeoff data to trigger the switch automatically.

3. Purpose-Specific Embedding Optimization: For embedding-based systems, implement a dynamic embedding-purpose selector that matches the API’s embedding mode (e.g., Amazon’s IMAGE RETRIEVAL vs. GENERIC INDEX) to the downstream task. The paper shows GENERIC INDEX caused a 13-point R@1 drop for Nova 2; an improved system should auto-detect task type (text-to-image retrieval vs. general vector search) and set the embedding purpose accordingly, potentially recovering significant accuracy.

4. Caption-Space Hard Negative Generation for Training: Use the paper’s methodology to improve training data for retrieval models. Generate hard negatives by embedding captions with a decoupled text encoder (e.g., all-mpnet-base-v2) and selecting the top-24 most similar captions per query. Train embedding models on these caption-space hard negatives to increase discrimination in ambiguous cases, directly addressing the paper’s finding that caption similarity is a well-motivated proxy for human perceptual similarity.

5. Confidence-Based Fallback to LLM Reasoning: Implement a confidence threshold on the top-1 score from an embedding model. When the margin between the top-1 and top-2 candidate scores is below a learned threshold (indicating ambiguity), automatically invoke an LLM ranker on that small candidate set. This reduces LLM cost by only using it for the 20% of queries where embeddings are uncertain, while preserving the LLM’s joint-reasoning advantage shown in the paper’s qualitative “both LLMs correct” cases.

6. Multimodal Embedding with Explicit Cross-Candidate Awareness: Modify embedding training to incorporate a contrastive loss that explicitly penalizes cases where a caption’s top-1 image is incorrect but a hard negative (from caption-space similarity) is ranked higher. This directly targets the paper’s finding that embeddings fail when visual correspondence is ambiguous, potentially closing the gap with LLMs on joint-reasoning tasks without adding latency.

What the Improved AI System Can Do:

• Retrieve images from a 5,000+ gallery in under 2 seconds with accuracy statistically indistinguishable from GPT-4.1/Claude Sonnet 4.6 (R@1 ≈ 0.80), while also resolving ambiguous cases via optional LLM re-ranking.

• Automatically adapt its retrieval strategy based on gallery size and update frequency, minimizing cost (e.g., 4 orders of magnitude lower latency for static galleries) without sacrificing accuracy.

• Achieve higher retrieval accuracy on hard-negative benchmarks by training on caption-space distractors, and avoid the 13-point performance drop seen with misconfigured embedding purposes.

• Provide a tunable latency-accuracy tradeoff: pure embedding mode for speed, hybrid mode for maximum accuracy, with the system deciding when to escalate to LLM reasoning based on confidence margins.

Multimodal retrieval and classification across different types of media, spanning text, images,video and audio, has traditionally relied on dual-encoder models that align visual and textual representations through contrastive learning. The March 2026 release of Gemini Embedding 2, Google's first natively multimodal embedding model to map text, images, video, audio, and documents into a single shared space, raises competition among multimodal retrieval systems. Simultaneously, frontier Large language models (LLMs) have also demonstrated strong visual understanding, raising the question of whether they can serve as effective zero-shot rankers. Our study provides the first direct comparison of native multimodal embeddings against LLM-based visual ranking on Flickr30k. We observe that GPT-4.1 and Claude Sonnet 4.6 perform on par with Gemini Embedding 2. Additionally, once embeddings are precomputed, multimodal embeddings are better suited for low-latency applications.