MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

summary

The gist

The gist The study introduces MasalBench, a comprehensive benchmark designed to probe the contextual and cross-cultural understanding of Persian proverbs by LLMs.

In short

MasalBench is a benchmark created to test how well multilingual Large Language Models (LLMs) understand Persian proverbs in context and across cultures. The study found that LLMs excel at understanding Persian proverbs within their language but struggle significantly when asked to find equivalent English proverbs, highlighting limitations in cultural knowledge.

Key concepts

MasalBench: A comprehensive benchmark designed to assess the contextual and cross-cultural understanding of Persian proverbs by LLMs. It involves two main tasks: understanding proverbs within a dialogue context and identifying their equivalent meaning in English.
Contextual Understanding: This task tests an LLM's ability to correctly interpret a Persian proverb within a given conversation. The benchmark uses multiple-choice questions where distractors are designed to be literal traps or plausible but incorrect meanings, measuring how well the model grasps the intended meaning in that specific situation.
Cross-Cultural Understanding: This task evaluates an LLM's ability to identify the English equivalent of a Persian proverb. The benchmark uses binary choices where one option is the correct equivalent, and distractors are crafted to sound similar in wording or tone, testing the model's grasp of cultural analogy.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs · Read on arXiv

Ghazal Kalhor, Behnam Bahrak

School of Electrical and Computer Engineering, University of Tehran · Tehran Institute for Advanced Studies, Khatam University

In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at https://github.com/kalhorghazal/MasalBench.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs".

Jane: The gist The study introduces MasalBench, a comprehensive benchmark designed to probe the contextual and cross-cultural understanding of Persian proverbs by LLMs.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: : So we’re wrapping up this look at MasalBench, which is this new benchmark for testing how well large language models understand Persian proverbs in context and across cultures.

Jane: : Right, so what’s the main thing we should know about the paper itself? What is its final message?

Tom: : The authors are focused on showing that while models can handle Persian proverbs within a local conversation, they struggle when asked to map those concepts onto an equivalent proverb in English.

Lu: : They created this benchmark to test if these models can manage that kind of cross-cultural abstraction, moving past simple memorization of definitions <ref:2601.22050#pg3>.

Meng: : From a deployment angle, it seems like we need better methods for teaching cultural analogy rather than just more data in the same language.

Lalam: : The paper shows that they perform well at Persian proverbs when they are in context within a Persian conversation, achieving accuracies above zero point nine zero <ref:2601.22050#pg1>.

Jane: : But when we test them on finding the equivalent English proverb, the performance drops to about zero point seven nine accuracy for the best model <ref:2601.22050#pg1>.

Tom: : That difference shows a real weakness in their cross-cultural reasoning capabilities when it comes to these specific cultural knowledge domains <ref:2601.22050#pg3>.

Lu: : It really shows that there’s a gap between understanding the language structure itself and applying that knowledge across different cultural frameworks.

Meng: : So for us, it means focusing on how we train models to reason across cultures, not just how well they predict the next word in Persian <ref:2601.22050#pg3>.

Jane: : It suggests that current models are good at local context but need significant work on deep, abstract cross-cultural reasoning for real utility.

Tom: : Exactly, and MasalBench gives us a concrete framework for assessing that specific kind of cultural understanding in other low-resource languages we might be looking at <ref:2601.22050#pg1>.

Conclusion: Tom: So we’re wrapping up our look at MasalBench, which is this new benchmark for testing how well large language models understand Persian proverbs in context and across cultures.

Jane: Right, so what’s the main thing we should know about the paper itself?

Tom: The authors are looking at a specific problem because we already know these models struggle with cultural knowledge and analogy when they move between languages.

Lu: They created this benchmark to see if the models can handle that kind of cross-cultural abstraction, not just memorizing definitions.

Meng: I’m curious about what it actually means for deployment, though I’m skeptical about their testing environment.

Lalam: Basically, the results show that the models are pretty good at Persian proverbs when they are in context within a Persian conversation, hitting above zero point nine zero accuracy in those tasks.

Jane: But what happens when you ask them to translate that same idea into an equivalent English proverb?

Tom: That’s where things get interesting, because the performance drops significantly there; the best model only hit about zero point seven nine accuracy for that task.

Lu: It really shows a gap between how well they understand the language structure itself and their ability to apply that knowledge across different cultural frameworks.

Meng: So what does this tell us about where we need to focus our efforts when we build these next generation models?

Jane: It suggests that current models are strong at understanding the local context but weak when it comes to deep, abstract cross-cultural reasoning.

Tom: Exactly, and MasalBench gives us a framework for assessing that specific kind of cross-cultural understanding in other low-resource languages.

Lalam: It’s a tool for figuring out if an AI truly gets the cultural layer of something beyond just pattern matching.

Jane: And it opens up a lot of avenues for research into building models with better cultural awareness, which is what we need to talk about next.

More episodes

← Home