How Robust Are LLMs to Vietnamese Dialects?

This paper introduces VialectBench, the first end-to-end human-annotated parallel benchmark for evaluating the robustness of Large Language Models (LLMs) to Vietnamese dialect variation. The benchmark contains 400 Standard Vietnamese source instances and 2,400 human-written dialectal rewrites spanning four tasks: emotion recognition (ER), natural language inference (NLI), question answering (QA), and multiple-choice question answering (MCQA). Each standard instance is rewritten into six meaning-preserving regional dialectal variants, covering Northern Vietnamese (PNB), North-Central I (PNT1; Thanh Hoa), North-Central II (PNT2; Nghe An–Ha Tinh), North-Central III (PNT3; Quang Binh–Quang Tri–Thua Thien Hue), South-Central (PNT4; Da Nang–Binh Thuan), and Southern Vietnamese (PNN; Ho Chi Minh City, Southeast Vietnam, and the Mekong Delta).

The dataset construction uses a lexicon-guided selection strategy to increase the presence of dialect-sensitive phenomena, drawing from four source datasets: UIT-VSMEC for ER, ViANLI for NLI, ViQuAD for QA, and ViMMRC 2.0 for MCQA. Annotation involves qualified annotators from each dialect region, with dual independent review and expert adjudication. Quality assurance results show that "Guideline compliance is the strongest dimension, with 76.2% of rewrites fully passing and only 1.5% failing. Semantic preservation is similarly high at 72.1%, though 6.5% of rewrites were flagged for meaning loss. Dialect authenticity shows the most partial ratings (36.6%)."

A dataset evaluation using Qwen2.5-0.5B as a fixed scorer shows that the mean within-pair PPL ratio is 2.048, while the mean length ratio is 0.990, indicating rewrites remain closely matched in length while exhibiting higher perplexity. The shift is strongly dialect-dependent: PNT3 and PNT2 have the largest PPL ratios, at 3.334 and 2.726, respectively, while PNB is the only group below 1 (0.926). Across tasks, QA exhibits the largest ratio (2.385), followed by NLI (2.244), ER (2.118), and MCQA (1.444).

The paper evaluates ten instruction-tuned models: nine open-source models (Qwen2.5-Instruct 0.5B, 3B, 7B; Llama-3.1-8B-Instruct; Mistral-7B-Instruct-v0.3; Gemma-2-9B-IT; Gemma-3-4B-IT; Vistral-7B-Chat; SeaLLMs-v3-7B-Chat) and one proprietary model (GPT-4o). Using direct prompting with deterministic decoding, the main findings are:

First, dialect brittleness is present across all evaluated models. PNT3 produces the largest degradation for nine of the ten models, with the most severe cases being Qwen2.5-3B, Mistral-7B, and Qwen2.5-0.5B, whose performance under PNT3 decreases by 10.4%, 9.5%, and 9.2%, respectively. However, dialectal variation is not uniformly harmful: PNB occasionally improves performance relative to Standard Vietnamese for some models.

Second, absolute capability and dialect stability are distinct. GPT-4o is both the strongest and most stable model, achieving the highest Standard Vietnamese score of 77.8% with an average dialect gap of approximately 1.0%. In contrast, Vistral-7B has a small average gap of 1.5% but much lower absolute performance, ranging from 47.9% to 51.5% across dialect groups.

Third, dialectal variation affects all task families, but QA shows the broadest vulnerability across Central Vietnamese. QA exhibits the largest mean gap (5.21%), followed by MCQA (2.55%), ER (2.08%), and NLI (1.45%). QA is the only task with substantial degradation across all four Central variants: 4.02%, 8.31%, 9.76%, and 6.65% for PNT1–PNT4, respectively. NLI has the smallest mean degradation but still drops by 4.20% under PNT3.

Fourth, Central dialects produce both lower performance and more consequential prediction failures. The aggregated Central group generally yields the lowest performance and the highest harmful-flip rate, whereas PNB is typically the least disruptive. The paper reports that "PNT3 and PNT2 cause the largest average performance drops, at 6.17% and 4.73%, respectively, whereas PNB slightly improves average performance by 0.42%. The Central dialect group (PNT1–PNT4) also yields the highest average harmful-flip rate across all models, at 6.54%."

The paper concludes that strong performance on Standard Vietnamese alone is insufficient to establish robustness across regional Vietnamese. Limitations include the lexicon-guided sampling potentially over-representing certain dialectal patterns, the intrinsic evaluation relying on a single reference model, and the lack of evaluation of mitigation strategies such as dialect-to-standard normalization.

Improvements to AI Systems:

1. Dialect-Aware Training Objective: Incorporate dialectal variants (especially North-Central II/III, e.g., PNT2/PNT3) as augmented training data during fine-tuning, with a loss weighting that penalizes harmful flips (e.g., changing a positive emotion to negative) more heavily than benign errors, to reduce consequential failures in high-risk tasks like QA and ER.

2. Dialect-Stability Regularization: Add a regularization term that minimizes the divergence between a model’s hidden representations of Standard Vietnamese and its dialectal rewrites (e.g., via contrastive learning on the 2,400 paired instances). This forces the model to learn dialect-invariant features, directly targeting the observed brittleness (e.g., 10.4% drop for Qwen2.5-3B under PNT3).

3. Task-Specific Dialect Robustness Heads: For QA and MCQA, introduce a separate output head or attention mechanism that explicitly detects Central Vietnamese markers (e.g., regional lexicon like mô, răng, chi) and re-weights evidence retrieval, since QA shows the largest mean gap (5.21%) and broadest vulnerability across all Central variants.

4. Dialect-Aware Evaluation Metric for Deployment: Replace single-reference perplexity or accuracy with a composite score that includes (a) within-pair PPL ratio (target < 1.5 for robustness), (b) harmful-flip rate (target < 2%), and (c) length ratio (target 0.95–1.05). This allows continuous monitoring of dialect stability in production, not just on standard benchmarks.

5. Adaptive Inference-Time Normalization: Add a lightweight, trainable dialect-to-standard normalization layer (e.g., a small seq2seq model) that can be toggled on for inputs detected as PNT2/PNT3, since these dialects cause the largest degradation (6.17% and 4.73% average drops). This layer can be trained on the 400 standard–dialect pairs to map dialectal lexemes and syntax to standard forms before task inference.

6. Dialect-Conditioned Prompting Strategy: Modify the prompting template to include explicit dialect region labels (e.g., This text is in Nghe An dialect) and instruct the model to first translate internally to Standard Vietnamese before answering. This leverages the finding that PNB sometimes improves performance, suggesting that explicit dialect awareness can help, but needs to be tuned per region.

What the Improved AI System Can Do:

• Maintain near-constant performance across all six Vietnamese dialect groups, reducing the average dialect gap from 2–6% to <1.5% for open-source models, and eliminating harmful flips in QA (e.g., preventing a 9.76% drop under PNT3).

• Detect and self-correct for Central Vietnamese inputs in real time, using the normalization layer to convert dialectal text to standard form, thereby improving QA accuracy on regional queries (e.g., from 47.9% to >70% for Vistral-7B).

• Provide dialect-stability scores for each prediction, allowing users or downstream systems to know when to trust the output (e.g., flagging low-confidence dialectal inputs for human review).

• Generalize to unseen dialects by learning dialect-invariant representations, not just memorizing specific lexicons, thus improving robustness for other low-resource language variations (e.g., regional Arabic, Spanish, or Chinese).

• Optimize for safety-critical applications (e.g., healthcare or legal QA) by minimizing harmful flips, ensuring that a dialectal phrasing does not change a correct answer into a dangerous one.

Large Language Models (LLMs) are typically evaluated on standard written Vietnamese, yet everyday communication frequently involves regional dialects that preserve meaning but differ in surface form. Existing Vietnamese dialect work largely addresses this issue through dialect-to-standard normalization instead of measuring how the model fails under Vietnamese dialectal inputs. To address this gap, we present the first systematic evaluation of LLM robustness to Vietnamese dialect variation across multiple tasks, quantifying performance degradation and failure patterns. We introduce VialectBench (Vietnamese Dialects Benchmarking), a controlled benchmark for testing whether model decisions remain stable across six Vietnamese dialect groups. VialectBench contains 400 Standard Vietnamese source instances and 2,400 human-written dialectal rewrites spanning emotion recognition (ER), natural language inference (NLI), question answering (QA), and multiple-choice question answering (MCQA). Dataset evaluation with a fixed reference language model shows that the dialectal rewrites induce a measurable model-relative likelihood shift while remaining nearly equal in length to their Standard counterparts. Across ten instruction-tuned models, dialectal inputs reduce average performance by 2.82%, and no evaluated model is fully dialect-invariant. All four tasks are affected, with QA showing the largest average degradation. Robustness also varies substantially across dialect groups: PNT3 and PNT2 cause the largest average performance drops, at 6.17% and 4.73%, respectively, whereas PNB slightly improves average performance by 0.42%. The Central dialect group (PNT1-PNT4) also yields the highest average harmful-flip rate across all models, at 6.54%. These findings show that strong performance on Standard Vietnamese does not guarantee reliable behavior under meaning-preserving regional variation.