When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

Large language models (LLMs) perform strongly on academic benchmarks but show substantial weaknesses when tested on everyday, culturally grounded knowledge. This research introduces TRIVIAROOMQA, a multilingual benchmark designed to evaluate LLMs across common and niche topics in a quiz-style setting, revealing significant knowledge gaps that existing academic benchmarks fail to capture.

The gist

Models are strong on knowledge-intensive topics such as history, geography, and mathematics, but substantially weaker on everyday popular-culture topics such as celebrities, music, movies, and news.

Benchmark Introduction and Structure

The paper introduces TRIVIAROOMQA to evaluate everyday knowledge by extending research beyond academic benchmarks like MMLU or GSM8K. The benchmark is divided into two parts: a parallel multilingual subset and a larger French-only subset. The multilingual part consists of 3,300 parallel questions spanning 110 topics and covering geographical, historical, sensory, pop-culture, and everyday-life knowledge in six European languages (English, French, Italian, Spanish, German, Dutch). Additionally, there is a larger French-only evaluation set of 5,340 additional questions covering 178 additional topics. Each topic in the dataset includes 30 questions about the same domain with three varying levels of difficulty. Topics are grouped into broader categories and annotated with time period and geographic region metadata to allow for fine-grained analysis across multiple dimensions.

Model Evaluation Methodology

The evaluation is framed as multiple-choice question answering, where models are given the question, and answer candidates are scored using log-likelihood-based evaluation via the Language Model Evaluation Harness library (lm-eval). Thirty open-weight LLMs from European, Asian, and North American providers were evaluated across a range of model sizes from 7B to 70B parameters. The analysis separates variation into three sources: the language of the question, the type of knowledge being tested, and the metadata associated with each question.

Key Findings on Knowledge Gaps

The analysis revealed several specific performance patterns:

1. Models perform strongly on topics typically covered in widely available encyclopedic and educational sources, such as scientific and historical knowledge.

2. They struggle with long-tail and everyday culture-specific topics, exemplified by the finding that only 4 models out of 30 correctly answered a specific popular-culture question while all models correctly answered a different encyclopedic fact.

3. Model performance drops across all difficulty levels once a topic is outside the model’s knowledge boundary.

4. LLMs perform better on questions related to older time periods, and some models do not answer consistently when the same question is presented in different languages, suggesting access to factual knowledge is not always language-independent.

Analysis Across Dimensions

The paper examined performance across several dimensions using the TRIVIAROOMQA-FRENCH subset:

(Language)

Performance varies substantially across languages; OLMo-3-7B obtained the lowest overall performance, while highly multilingual models like Salamandra7B and Apertus-8B/70B show strong performance on all languages. Cross-lingual consistency remains uneven, with some models showing noticeable variation across languages.

(Category)

widely documented encyclopedic categories, such as history and geography, obtain the highest average scores, while everyday and popular-culture categories, such as music, movies, people, and news, are more difficult. For example, music, movies, people, and news remained below 50% accuracy for 18 of the 30 models.

(Difficulty Level)

Model performance decreases with the annotated difficulty level (average accuracy is 0.60 on beginner questions, 0.52 on confirmed questions, and 0.48 on expert questions). Crucially, humans show a gradual decrease in performance from beginner to expert questions, while models, once they are unfamiliar with a topic, remain close to the random baseline across all difficulty levels.

(Time Period)

historical and timeless questions obtain the highest scores, while performance decreases for more recent periods, especially for questions from the 2000-2020s. This trend is aligned with lower performance in popular culture and news topics.

(Continent)

For models below 20B parameters, average performance by model origin broadly aligns with the question region, but this trend becomes less clear for larger models. North America-related questions obtain high scores across most models, consistent with prior work reporting stronger model performance on North American cultural knowledge.

Conclusion and Future Directions

The study concludes that while larger models generally improve absolute accuracy, they do not eliminate the main performance gaps between encyclopedic and everyday cultural categories or cross-lingual inconsistencies. The benchmark reveals that "what models find difficult does not always align with what humans find difficult.

Here are specific improvements for AI systems based on the findings in TRIVIAROOMQA:

1. Meticulous knowledge gap identification: The benchmark TRIVIAROOMQA reveals a critical weakness where LLMs excel at encyclopedic/academic knowledge (history, geography, mathematics) but fail significantly on everyday cultural topics (movies, music, celebrities).

2. Targeted Fine-tuning for Cultural Knowledge: AI systems should be fine-tuned specifically on long-tail and popular culture data to bridge the gap between academic recall and everyday knowledge.

3. Cross-lingual Robustness Enhancement: Since model performance varies across languages even for the same questions, systems need training or prompting strategies that ensure language independence for factual retrieval, moving beyond simple multilingual pre-training.

4. Difficulty-Aware Knowledge Retrieval: AI agents should be designed to adjust their retrieval depth based on perceived topic difficulty. When encountering a topic outside their known knowledge boundary, they should signal uncertainty rather than hallucinating a potentially incorrect answer, mimicking the human degradation pattern observed in TRIVIAROOMQA.

5. Temporal and Geographic Context Awareness: Systems must incorporate explicit metadata (time period and geographic region) into their reasoning process when answering factual queries. This prevents reliance on outdated or contextually irrelevant information, especially for recent news or region-specific facts (e.g., French cheeses).

6. Search Augmentation Integration: For complex, low-frequency facts that are not in the model's static parameters (as suggested by the search experiment), AI systems should be integrated with reliable retrieval augmented generation (RAG) or external search tools to dynamically update and verify information beyond their training cutoff.

7. Adaptive Performance Scaling: System architectures should incorporate mechanisms that allow for dynamic scaling of knowledge access based on the required accuracy level, rather than relying solely on a fixed parameter size (e.g., 7B vs 70B), which showed diminishing returns on specific domains like pop culture.