ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
cs.CL
Submitted: 2026-06-30
Updated: 2026-08-31
Comments: EMNLP 2026 Main
Code: https://github.com/Andrian0s/any-lang-embed-eval
License: http://creativecommons.org/licenses/by/4.0/
The gist: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge.
Terminology
Abstract
Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval
Sources
- Granite Embedding Models
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- Training Sparse Mixture Of Experts Text Embedding Models
- NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- EmbeddingGemma: Powerful and Lightweight Text Representations
- Multilingual E5 Text Embeddings: A Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering