To What Extent Do Large Language Models Understand Bangla Idioms?

arXiv:2609.03410 · cs.CL · Submitted 2026-09-03 · Read on arXiv

cs.CL

Submitted: 2026-09-03

Updated: 2026-09-03

License: http://creativecommons.org/licenses/by-sa/4.0/

The gist: Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low-resource languages.

Terminology

Benchmark — 16× in this paper · explained in 2 episodes
Benchmarking — 5× in this paper · explained in 3 episodes
Embeddings — 5× in this paper · explained in 2 episodes
Low-Resource Languages — 5× in this paper · explained in 4 episodes
Annotator Agreement — 4× in this paper · explained in 1 episode
Accuracy — 3× in this paper · explained in 1 episode
European Language Resources Association — 3× in this paper
Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
Semantic Similarity — 3× in this paper · explained in 2 episodes
Alignment — 2× in this paper · explained in 1 episode
Core — 2× in this paper · explained in 2 episodes
Cosine Similarity — 2× in this paper · explained in 3 episodes
Open Educational Resources — 2× in this paper
Semantic Understanding — 2× in this paper · explained in 2 episodes
Adaptation — 1× in this paper · explained in 1 episode
Cross-lingual Transfer — 1× in this paper · explained in 2 episodes
Hallucination — 1× in this paper · explained in 3 episodes
In-Context Learning — 1× in this paper · explained in 4 episodes
National Curriculum and Text Book — 1× in this paper
Positional Bias — 1× in this paper · explained in 2 episodes
Transformer — 1× in this paper · explained in 2 episodes

Abstract

Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low-resource languages. In this paper, we present the first large-scale benchmark dataset of Bangla idioms, complemented by a synthetic multiple-choice question (MCQ) dataset for idiom meaning identification. We conduct a comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies. Our results reveal substantial variability in model performance, with no single LLM consistently outperforming others across all tasks. Notably, Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification. We believe that our datasets and analyses will provide valuable resources to guide future research in improving LLM comprehension of idiomatic expressions, particularly in Bangla and other low-resource languages.

To What Extent Do Large Language Models Understand Bangla Idioms?

Terminology

Abstract

Sources

Related papers