MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

arXiv:2601.21225 · cs.CL, cs.AI · Submitted 2026-01-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation".

Jane: Large language models have made substantial progress in mathematical reasoning, but benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're diving into "MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation." The authors are a big team from various institutions, which is always promising when you're looking at cross-lingual research.

Jane: They’ve essentially created a dataset that gives five different versions of every math problem by changing names, digits, and adding irrelevant context across nine languages. It sounds like they’re trying to make the testing process much tougher than just using one example per language.

Lu: The title suggests a practical method for making these evaluations more stable and representative when dealing with multiple languages. It points toward moving beyond English-centric benchmarks which we know are often biased in difficulty and what's currently being studied.

Meng: I see the core idea is robustness; they want to see if a model can solve the problem even when the surface details—like names or numbers—are slightly altered, which is crucial for real-world deployment.

Lalam: It suggests that for math reasoning, we need to test stability across different linguistic contexts simultaneously, not just testing accuracy in isolation.

The paper's summary: Tom: So the paper explains that while models have improved at math reasoning generally, the benchmarks haven't kept up with how complex and varied multilingual tasks are becoming. They address this by introducing MGSM-Pro, which builds on GSM-Symbolic to provide five instantiations per question across nine languages.

Jane: Essentially, they created a system where they take an English template and use an LLM to translate it into multiple languages, then they add human verification to make sure those new versions are actually solvable and varied enough for testing.

Lu: The structure of the dataset is key here; they categorize languages into high-resource and low-resource groups, which allows them to specifically track where performance drops happen most severely.

Meng: They organized the variations into two sets, the Symbolic Series and the Irrelevant Context Series, which suggests they are testing both how models handle simple surface changes and how they manage noise from extra text.

Lalam: This summary really highlights that robustness isn't just about one language; it’s about understanding how a model holds up when things get messy or when the linguistic environment shifts dramatically.

The paper's improvements: Tom: What the paper is proposing is a set of specific changes to how we test models, primarily suggesting that evaluation should always involve at least five instances of the same problem with different digits, and they’re recommending that this "Avg-five setting" be our default.

Jane: They found something interesting: low-resource languages experience a sharp drop in performance when accuracy is averaged over those five instances, which is much more pronounced than what we see in high-resource languages.

Lu: The paper explicitly states that model robustness doesn't necessarily transfer from high-resource settings to low-resource ones, which explains why their findings on LRLs show a huge drop in performance and sometimes loses more than a twenty percent drop.

Meng: They also found that proprietary models like Gemini three point zero Pro show more robustness to digit variations compared to Gemini three point five Flash, while open models like GPT-OSS 120B and DeepSeek v3 show stronger robustness overall against these kinds of degradations.

Lalam: This tells us that we can’t just look at a model's size or its general performance level; we need to check its stability specifically within the context of different languages to truly assess its capability.

Conclusion: Tom: So, to wrap up, the main takeaway from "MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation" is that math reasoning evaluation needs a more rigorous approach by testing problems with at least five variations, and we should default to using that average performance metric.

Jane: It seems the implication is that we need to stop relying on single examples and start demanding stability across different linguistic contexts, especially when evaluating models for languages with fewer training examples.

Lu: The authors really push for releasing this dataset because they believe it encourages a more robust evaluation process overall, moving us closer to a fairer assessment of multilingual capabilities.

Meng: Practically speaking, if we adopt their suggestion to test five instances by modifying digits, it gives us a concrete way to check if the model is just memorizing patterns or actually grasping the underlying arithmetic rules.

Lalam: This work on MGSM-Pro is really encouraging because it suggests that improving linguistic comprehension and logical reasoning in low-resource languages is a critical area for development if we want models to perform reliably everywhere.

Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya, David Ifeoluwa Adelani

McGill University · Mila-Quebec AI Institute · University of Toronto · Hanyang University, Rep. of Korea · Instituto Politécnico Nacional, Mexico · University of Ibadan, Nigeria · McPherson University, Nigeria

cs.CL, cs.AI

Submitted: 2026-01-29

Updated: 2026-09-30

Importance score: 88/100

The gist: Large language models have made substantial progress in mathematical reasoning, but benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency.

Key concepts

MGSM-Pro Dataset: This dataset provides five different versions of a math problem for each question. Variations involve changing names, digits, or adding irrelevant context across nine languages to create a realistic test of how well models handle multilingual math reasoning problems.
Low-Resource Languages (LRLs): These are languages with fewer available training data for AI models. The study found that LRLs experience a much sharper performance drop when accuracy is averaged over five examples compared to high-resource languages, indicating they are more fragile during evaluation.
Symbolic Series (SYM) and Irrelevant Context Series (IC): These are the two main ways the dataset creates test variations. The Symbolic Series changes elements like names or numbers, while the Irrelevant Context Series adds a distracting sentence to increase difficulty, testing both basic manipulation and contextual understanding.
Model Robustness: This refers to how stable a model's performance is when small changes are made to a math problem. The study found that robustness differs by language and training recipe, suggesting that size alone doesn't guarantee stability across different linguistic contexts.

Terminology

Robustness — 34× in this paper · explained in 5 episodes
Accuracy — 26× in this paper · explained in 1 episode
Benchmark — 25× in this paper · explained in 2 episodes
Low-Resource Languages — 9× in this paper · explained in 4 episodes
Verification — 6× in this paper · explained in 2 episodes
Irrelevant Context Series — 3× in this paper
Meeting of the Association for Computational Linguistics — 2× in this paper
Natural Sciences and Engineering Research Council — 2× in this paper
Resource Constraints — 2× in this paper · explained in 3 episodes
Alignment — 1× in this paper · explained in 1 episode
Core — 1× in this paper · explained in 2 episodes
Cross-lingual Transfer — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
Verifier — 1× in this paper · explained in 2 episodes

Summary

Large language models have made substantial progress in mathematical reasoning, but benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. This paper introduces MGSM-Pro, an extension of the MGSM dataset that provides five instantiations per question by varying names, digits, and irrelevant context across nine languages to obtain a more robust and realistic assessment of math reasoning.

Key Findings on Robustness Across Languages

The evaluation reveals that low-resource languages (LRLs) experience a sharp performance drop when accuracy is averaged over five instances instead of a single example, unlike high-resource languages (HRLs). Specifically, LRLs suffer the most in terms of huge drop in performance and in some cases losing more than −20.0 drop in performance. Furthermore, model robustness in HRL setting do not necessarily translate to LRL. The study found that proprietary models like Gemini 3.0 Pro are more robust to digit, whereas Gemini 3.5 Flash is more robust to this degradation, while open models such as GPT-OSS 120B and DeepSeek v3 show stronger robustness.

MGSM-Pro Dataset Construction Methodology

The paper introduces MGSM-Pro in two steps: (1) template construction in English that allows easy replacement of names and digits and (2) dataset construction that translates the template to multiple languages (with an LLM), followed by human verification—this helps to generate different instantiations of same question. The dataset covers nine languages with various resource levels, including HRLs (English, Chinese, French, Japanese) and LRLs (Swahili, Amharic, Igbo, Yoruba, Twi). The variations are organized into two series: the Symbolic Series (SYM), which includes SYM N (replacing names), SYM% (changing numerical data), and SYM N% (varying both names and numbers simultaneously); and the Irrelevant Context Series (IC), which introduces a distinct layer of difficulty by inserting an irrelevant sentence, denoted as IC N, IC%, and IC N%.

Experimental Setup and Evaluation Settings

The models were benchmarked in a zero-shot setting across six variations within the SYM and IC series for each language. To ensure robustness, every variation is evaluated five times using different values and we report the mean performance across these iterations. The evaluation prompt instructs the model to Explain your reasoning step by step in clear English to solve the problem and end with a final numerical answer without units. The results are presented comparing original data (DO) against variations like IC N, IC N%, SYM N%, and SYM N%.

Analysis of Model Performance and Reliability

The results indicate that Simply changing the names of person or items (i.e. SYM N setting) does not necessarily hurt performance. However, Numerical variation leads to huge drop in performance while names variation leads to small drop in performance. The study also investigates the importance of arithmetic competence versus language understanding, finding that strong linguistic understanding is just as important as arithmetic capabilities, noting that initial linguistic errors frequently propagate into logical reasoning failures. Finally, concerning leaderboard rankings, the paper notes that Our results in Table 3 show that these rankings are unstable, and repeating experiments 10 times gave similar results as the Avg-5, suggesting evaluation should use the Avg-5 setting as the default for a more robust estimation.

Recommendations for Future Evaluation

Based on findings across nine typologically diverse languages, the authors recommend that math reasoning evaluation should be performed on a minimum of five instances of the same problem by modifying digits. They suggest that both open LLMs and proprietary LLMs are expected to be robust to these small changes, encouraging the release of MGSM-Pro to encourage a more robust evaluation. The study concludes that model robustness differ per language, and LRL suffer more, suggesting that model robustness is not a direct result of model size but rather other factors like training recipe.

Limitations

The study acknowledges limitations, including the dataset covering only nine languages due to resource constraints and the significant human labor required for verification. It also notes that the evaluation covers only 18 models because of limited compute budget, and it remains to be seen how other model families might perform. The authors suggest expanding MGSM-Pro to other languages like Tamil would provide a more complete picture of multilingual mathematical robustness.

Acknowledgment

This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada and in part by the AI2050 program at Schmidt Sciences, Mila’s computing resources (mila.quebec), and Digital Alliance of Canada. This work was also partially supported by Azure sponsorship credits granted by Microsoft’s AI for Good Research Lab.

References

(A full list of references is provided in the paper.

Improvements for AI systems

Based on the scientific paper MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation, here are specific, actionable improvements for AI systems and what those improved systems can achieve:

Replace single-instance evaluation with a five-instance average across various instantiations (names, digits, irrelevant context) for all mathematical reasoning tasks.
Implement a robust evaluation pipeline that systematically generates at least five distinct problem variations (using the MGSM-Pro methodology: Symbolic and Irrelevant Context series) for every question before testing the model.
Develop language-specific robustness thresholds; specifically, prioritize testing low-resource languages (LRLs) with a minimum of five instances to accurately gauge performance degradation, as LRLs suffer much larger drops than High-Resource Languages (HRLs).
For proprietary models, favor more robust architectures (e.g., Gemini 3.0 Pro over Gemini 2.5 Flash) when testing robustness against numerical and contextual variations to ensure reliable performance assessment.
Design model evaluation prompts that explicitly instruct the LLM to reason in English first (as it shows better reasoning performance) but then report the final answer in the target native language, allowing for a comparative analysis of linguistic versus arithmetic competence.
In model fine-tuning or pre-training, focus on improving linguistic comprehension and logical reasoning within low-resource languages, as these factors are shown to be critical failure points for models in LRLs when facing noisy contexts.
Develop a method to decouple model performance from simple parameter size; investigate training recipes that promote robustness against input noise (digit/context variation) rather than relying solely on larger model counts.

This improved AI system can achieve the following:

Improve the reliability and accuracy of mathematical reasoning benchmarks by moving beyond single-answer reliance to a statistically robust measure that reflects real-world variability (i.e., an Avg-5 or Avg-10 setting).
Provide high-fidelity, linguistically relevant performance metrics for models across a global set of languages, ensuring that the evaluation is not biased toward English proficiency alone.
Accurately identify which models and which language families are most vulnerable to robustness degradation when faced with common real-world noise (like context switching or numerical typos).
Determine whether a model's failure on a complex math problem stems from a lack of arithmetic competence, poor language understanding, or failure to handle noisy input—allowing researchers to target specific training improvements.
Create more realistic and challenging evaluation datasets that mimic the variability encountered by students or users solving problems in diverse linguistic environments.