Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

arXiv:2505.18673 · cs.CL · Submitted 2025-05-24 · Read on arXiv

cs.CL

Submitted: 2025-05-24

Updated: 2025-05-24

Comments: ACL 2025. Code available at https://github.com/xzx34/Cross-Lingual-Pitfalls

Journal ref: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8254-8284, 2025

DOI: 10.18653/v1/2025.acl-long.404

Code: https://github.com/xzx34/Cross-Lingual-Pitfalls

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 38× in this paper · explained in 1 episode
• Fine-Tuning — 20× in this paper · explained in 5 episodes
• Consistency — 9× in this paper · explained in 3 episodes
• Adam — 7× in this paper · explained in 2 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Direct Perturbation — 3× in this paper
• Embeddings — 3× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Natural Language Processing — 3× in this paper
• Adaptation — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 4 episodes
• Cross-lingual Transfer — 2× in this paper · explained in 2 episodes
• Data Augmentation — 2× in this paper · explained in 7 episodes
• International Conference on Learning Representations — 2× in this paper
• No Perturbation — 2× in this paper
• Redundancy — 2× in this paper · explained in 2 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Adversarial Training — 1× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Chain-of-Thought (CoT) Prompting — 1× in this paper · explained in 3 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Continual Learning — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Information Asymmetry — 1× in this paper · explained in 2 episodes
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• Knowledge graphs — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Low-Resource Languages — 1× in this paper · explained in 4 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Relative Affinity Score — 1× in this paper
• Representation Learning — 1× in this paper · explained in 2 episodes
• Semantic Similarity — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Unsupervised Cross-lingual Representation Learning at Scale
• Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
• Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
• Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models
• Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
• Understanding Cross-Lingual Alignment -- A Survey
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• 1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?
• On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspective
• MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark
• Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
• Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasets
• Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?
• AlignBench: Benchmarking Chinese Alignment of Large Language Models
• GPT-4o System Card
• OpenAI o1 System Card
• AgentBench: Evaluating LLMs as Agents

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering