CHARM: Character Hallucination for Multicultural Role Play Benchmark

arXiv:2609.01352 · cs.CL, cs.AI · Submitted 2026-09-01 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-09-01

Updated: 2026-09-01

Comments: 16 pages, 1 figure. Accepted to Findings of EMNLP 2026

Code: https://github.com/sunkyoung19/CHARM

License: http://creativecommons.org/licenses/by/4.0/

The gist: Role-playing large language models (LLMs) are expected to adopt a character's style while also respecting that character's knowledge boundaries.

Terminology

Abstract

Role-playing large language models (LLMs) are expected to adopt a character's style while also respecting that character's knowledge boundaries. Prior evaluations detect character hallucination but rarely distinguish whether errors arise from failure to recognize a boundary or from failure to comply despite recognition. We introduce CHARM, a multicultural benchmark of 40 real and fictional characters drawn from five cultural-linguistic regions, and validated by native reviewers. It probes two boundary types, Temporal (historical vs. modern) and Cross-Universe (entities outside a character's narrative or historical universe), using abstention-enabled multiple-choice questions. We propose a two-stage evaluation that separates Boundary-Awareness (explicit recognition that a query is out of scope) from Boundary-Compliance (abstention when answering concrete questions). Evaluations across six LLMs show that hallucination is driven predominantly by compliance failures. Models frequently acknowledge that a query lies outside the character's knowledge yet still provide factual, out-of-character answers. By re-posing the same questions to the target character, we confirm that a large fraction of these cases are verified parametric overrides; the model stores the relevant fact but fails to suppress it. We also observe systematic cultural variation in these failures, consistent with imbalances in how characters from different regions are represented in model knowledge.

Sources

Related papers