M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608.03803 · cs.CL, cs.LG · Submitted 2026-08-04 · Read on arXiv

Tomáš Burkert, Angelika Peljak-Łapińska, David Zelený

cs.CL, cs.LG

Submitted: 2026-08-04

Comments: 45 pages (97 incl. appendices), 6 figures

Code: https://github.com/tatsu-lab/alpaca_eval

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 102× in this paper · explained in 2 episodes
• Accuracy — 24× in this paper · explained in 1 episode
• Low-Resource Languages — 21× in this paper · explained in 4 episodes
• LLM judge — 18× in this paper · explained in 2 episodes
• Actor — 9× in this paper · explained in 3 episodes
• Pragmatics — 9× in this paper · explained in 2 episodes
• Calibration — 7× in this paper · explained in 7 episodes
• Meeting of the Association for Computational Linguistics — 6× in this paper
• AUC — 5× in this paper · explained in 2 episodes
• Core — 4× in this paper · explained in 2 episodes
• Failure Modes — 4× in this paper · explained in 3 episodes
• Selection Bias — 3× in this paper · explained in 4 episodes
• Human-in-the-Loop — 2× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Thinking mode — 2× in this paper · explained in 2 episodes
• Transactions of the Association for Computational Linguistics — 2× in this paper
• Adaptive Reasoning — 1× in this paper · explained in 2 episodes
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Data Augmentation — 1× in this paper · explained in 7 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Empirical Methods in Natural Language Processing — 1× in this paper
• Ethnologue Digital Language Support — 1× in this paper
• F1 Score — 1× in this paper · explained in 3 episodes
• International Conference on Learning Representations — 1× in this paper
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Systematic Bias — 1× in this paper · explained in 3 episodes
• Tokenization — 1× in this paper · explained in 2 episodes

Sources

• Non-Determinism of "Deterministic" LLM Settings
• Systematic Inequalities in Language Technology Performance across the World's Languages
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• BLEU might be Guilty but References are not Innocent
• Annotation Artifacts in Natural Language Inference Data
• Prompting is not a substitute for probability measurements in large language models
• MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
• Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
• On Length Divergence Bias in Textual Matching Models
• The State and Fate of Linguistic Diversity and Inclusion in the NLP World
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Proving Test Set Contamination in Black Box Language Models
• LLM Evaluators Recognize and Favor Their Own Generations
• Language Model Tokenizers Introduce Unfairness Between Languages
• COMET: A Neural Framework for MT Evaluation
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• BLEURT: Learning Robust Metrics for Text Generation
• Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering