BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian

BAVGROUND is a benchmark introduced to evaluate Bavarian regional cultural grounding and dialect competence in large language models (LLMs) across English, German, and Bavarian. It contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. The benchmark combines general-knowledge questions with grounded questions derived from regional journalism, anthropological monographs, and specialist historical sources, targeting culturally specific knowledge unlikely to appear frequently in standard LLM pretraining corpora.

The paper evaluates fifteen 7B–10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. The strongest open-weight group consists of EuroLLM-9B-Instruct, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, all clustered around 69% accuracy. The open-weight mean is 53.0%, indicating that BAVGROUND remains difficult for 7B to 10B open-weight models under standard multiple-choice letter scoring. Averaged across models, German is highest at 57.6%, followed by English at 55.7%, while Bavarian drops to 45.9%. The GEN–GRD gap shows average accuracy drops from 63.1% on GEN to 46.7% on GRD, a 16.4-point decrease. The closed-model reference gpt-5.4-mini reaches 89.6% under first-token answer-letter log-probability scoring, 20.2 points above the strongest open-weight model, with performance more stable across languages.

The paper further shows that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Across all fifteen models, accuracy rises from 53.0% under letter to 59.3% under option text avg. This shift is model-specific: the strongest letter models remain stable or decrease slightly under option text avg, whereas lower-ranked or strongly label-skewed models gain substantially. GENBA-10B provides a clear case study of this protocol sensitivity, increasing from 41.9% with letter scoring to 58.9% with option text avg and 61.5% with semantic matching of generated answers.

Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. Across 85 checkpoints, letter accuracy remains low and unstable, rising from 11.2% to a peak of 19.3% before ending at 12.1%. By contrast, option text avg improves from 25.4% to 50.8%, and semantic embed generated answer peaks at 50.3% around checkpoint 14.5k. The gains are uneven across languages and domains: Bavarian remains below English and German at the final checkpoint, and Language remains the weakest domain despite stronger improvements in Politics, Arts & Identity, Living Traditions & Customs, and Culinary. BAVGROUND supports localized, protocol-aware evaluation of cultural representation in LLMs.

Improvements to AI systems:

1. Add dialect-aware cultural grounding layers – Train or fine-tune models on regional dialect corpora (e.g., Bavarian) alongside standard languages, with explicit cross-lingual alignment for culturally specific terms, idioms, and historical references. This reduces the observed 11.7-point accuracy drop from German to Bavarian.

2. Implement protocol-robust answer selection – Replace single-letter scoring with option-text likelihood averaging during inference. This raises average accuracy from 53.0% to 59.3% across models, and for label-skewed models like GENBA-10B, from 41.9% to 58.9%. The improved system should compute log-probabilities over full option texts, not just answer letters, to avoid positional bias.

3. Add source-grounded retrieval augmentation – For questions marked as GRD (source-grounded), the system should retrieve from regional journalism, historical archives, or specialist literature before answering. This directly targets the 16.4-point GEN–GRD gap, improving performance on localized factual knowledge that is underrepresented in pretraining.

4. Develop dialect-specific continued pretraining with domain-balanced curricula – Use checkpoint analysis (e.g., GENBA-10B) to guide training: allocate more updates to weak domains like Language (dialect grammar/lexicon) while maintaining gains in Politics, Arts, and Culinary. The improved system should monitor per-domain option-text likelihood during training, not just letter accuracy, to avoid false plateaus.

5. Add semantic matching for open-ended generation – When the model generates free-text answers, use semantic embedding similarity (e.g., cosine distance to correct option) instead of exact string matching. This lifts performance for dialect-heavy outputs, as seen with GENBA-10B rising to 61.5% accuracy under semantic matching.

6. Build a multilingual cultural knowledge adapter – A lightweight, plug-in module that maps culturally specific concepts (e.g., Bavarian traditions, historical figures) across English, German, and Bavarian. This reduces cross-lingual performance variance and helps smaller 7B–10B models approach the closed-model reference (89.6%) more closely.

What the improved AI system can do:

• Answer multiple-choice questions about regional culture with >20% higher accuracy on dialect and source-grounded items, especially for low-resource dialects.

• Maintain stable performance across evaluation protocols (letter, shuffled, option-text, semantic), avoiding ranking flips and score inflation.

• Generate culturally appropriate, dialect-aware responses in Bavarian and similar regional languages, with improved handling of idioms, local history, and customs.

• Self-monitor its own training progress using protocol-aware metrics, preventing overfitting to answer-letter biases and ensuring balanced gains across cultural domains.

• Retrieve and cite regional sources (journalism, monographs) for grounded answers, reducing hallucination on niche cultural facts.

• Serve as a robust evaluation tool for cultural representation in LLMs, enabling developers to detect and correct dialectal and regional blind spots before deployment.

Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. BavGround contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. We evaluate fifteen 7B-10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. We further show that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. BavGround supports localized, protocol-aware evaluation of cultural representation in LLMs.