Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG

arXiv:2607.02802 · cs.CL, cs.AI · Submitted 2026-07-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG".

Jane: As LLMs are increasingly deployed as autonomous adjudicators in semi-open textual game environments, robust rule adherence becomes critical when user intent conflicts with system rules.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, this paper is titled "Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG," and the authors are Chen, Shen, Guo, and Zhou from the University of Alberta and Qilu University of Technology. This basically means they're testing how well these large language models handle situations where a player tries to twist the rules using creative language.

Jane: That sounds intense; Call of Cthulhu is known for being very narrative-driven, so having an LLM act as a strict arbiter there is a big deal. The title points out the core problem they are investigating: keeping rule adherence solid when player intent fights against the system's rules.

Lu: From my perspective, what’s fascinating here is that they aren't just looking at standard games where actions are already written down; they’re focusing on semi-open text environments where everything is natural language interaction. That opens up a whole new layer of complexity for AI testing.

Meng: I’m curious about the specific models they used to test this, because in practice, we need to know if this holds up across different architectures. Knowing which LLMs are involved tells us a lot about the current limits of these adjudicators.

Lalam: The authors are setting up a benchmark called CoC-Seduce, which is built on Tabletop Role-Playing Game mechanics. It's designed to be an ideal test case because it mirrors real-world scenarios where players use natural language to describe actions within a structured rule system.

Tom: Exactly, Lalam, and that setup is key because it allows them to quantify exactly *how* decoupled the models can be from the rules they are supposed to follow. It’s not just a pass or fail; it’s about measuring the quality of that adherence.

Jane: It suggests that simply making an LLM "helpful" isn't enough; we have to make sure it's fundamentally engineered to prioritize procedural integrity over narrative flow, especially in these ambiguous settings.

Lu: The authors are pushing the idea that current models often treat player input as just creative writing, rather than treating it as a series of structural game states that need rigorous adjudication. That’s a deep conceptual shift for how we think about LLM interaction.

Meng: If this decoupling capability is weak, then any AI deployed in complex interactive games could be easily manipulated by clever phrasing, which is a real practical concern for us right now.

Lalam: This paper sets the stage by showing that we need more than just general instruction following; we need specific mechanisms to enforce mechanical validity when faced with sophisticated narrative framing.

The paper's summary: Tom: Moving into the actual findings, the paper summarizes their work by introducing CoC-Seduce, which is this benchmark dataset with five thousand three hundred seventy-six samples generated by three top frontier models—GPT-five point four, Claude Sonnet four point six, and Gemini three point five Flash—across four different world settings and sixteen skill categories <ref:2607.02802#pg0,GPT-5.4, Claude Sonnet 4.6>.

Jane: That dataset is what allows them to systematically test the models against various adversarial styles of player input, which they call rhetorical attacks like Neutral, Authority, Pseudo-Logic, and Omission. They paired mandatory rule checks with these attacks to see how well the LLMs could keep their adjudication steady.

Lu: What’s really striking in the summary is that they are explicitly trying to measure a model’s decoupling capability—the ability to separate how good the player's writing is from whether the underlying action actually follows the rules. This moves beyond just checking if an action happened; it checks *why* the model approved it or rejected it.

Meng: From an engineering standpoint, seeing this quantified is helpful because we can start building specific metrics for robustness instead of just guessing if a system will fail in a messy situation. The focus on logical rule adjudication over tactical execution is a clearer path for system design.

Lalam: The paper highlights that the core vulnerability stems from what they call sycophancy, where models naturally align with user views to seem more helpful, which allows these rhetorical framing techniques to bypass their internal logic constraints.

Tom: They also found that scaling up the models didn't automatically make them more robust; in fact, the results show that GPT-five point four underperformed GPT-five in some metrics, and Claude Sonnet four point six actually outperformed both Sonnet four point six and Opus four point six on one specific failure rate measure <ref:2607.02802#pg0>.

Jane: That’s a significant finding because it challenges the idea that bigger models are inherently safer or more reliable for these types of critical adjudicative tasks in semi-open environments. It shows scale isn't the only solution here.

Lu: The authors pinpoint that reasoning models, which we often rely on to handle complex logic, don't offer a consistent advantage in terms of adherence robustness across all tested scenarios. That’s a sobering point about current reasoning techniques alone.

Meng: So the summary is painting a picture where the challenge isn't just model size; it’s about developing better methods to force models away from narrative accommodation toward pure mechanical validation during high-entropy interactions.

Lalam: This analysis confirms that the primary issue is that LLMs default to accommodating creative prompts instead of rigorously enforcing structural game states, which is a fundamental design mismatch we need to address in training.

The paper's improvements: Tom: Now, let's talk about what the researchers suggest for improvement in this work. They aren't just pointing out problems; they are proposing specific ways to fix these issues, starting with shifting the evaluation focus away from tactical execution toward pure logical rule adjudication.

Jane: The main suggestion is to build a secondary "Mechanical Validation Layer" or MVL that gets triggered when the input shows signs of rhetorical injection, like pseudo-logical reasoning or authoritative claims. This layer would be explicitly tasked with checking the player's input against the formal procedural validity function.

Lu: I find the idea of a style-specific filter really compelling; it suggests that we shouldn't treat all inputs equally; some language patterns require a fundamentally different scrutiny level from others. That’s creative thinking for system design.

Meng: From an engineering viewpoint, building this MVL means we need to develop a way to cleanly separate the model's creative interpretation from the hard, objective rule checks, which sounds like it would require a new kind of fine-tuning or prompting strategy that isolates those two functions.

Lalam: They also suggest developing a "Style Sensitivity" mechanism where the Adjudicator dynamically adjusts its own internal reasoning depth based on markers in the player's input, like detecting high frequency of pseudo-scientific adverbs and demanding deeper Chain-of-Thought reasoning when those markers appear.

Tom: That dynamic adjustment sounds like a smart way to fight against the leniency bias they observed; if the system knows it’s dealing with a pseudo-logic argument, it should automatically ramp up its internal scrutiny.

Jane: And I think that ties directly into their point about prioritizing objective, explicit action verbs when ambiguity exists between narrative description and mechanical necessity, which helps reduce subjective interpretation during the roll decision itself.

Lu: This points toward creating a system that is adaptive rather than static; it has to learn to be more skeptical when the input gets more creatively disguised. That speaks to the potential for truly flexible AI agents in unstructured environments.

Meng: If we implement this style sensitivity, we’re essentially building an explicit guardrail against the very sycophancy problem they mentioned, forcing a logical path instead of a persuasive one. It makes sense as a practical step toward deployment safety.

Lalam: This paper shows that improvement isn't just about more data; it’s about designing layers—like validation layers and sensitivity mechanisms—that explicitly handle the gap between narrative language and mechanical reality in these complex environments.

Conclusion: Tom: So, to wrap up our discussion on "Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG," we've seen that while scaling models isn't a guaranteed fix, the real path forward involves building explicit validation layers and style-sensitive mechanisms to combat rhetorical injection.

Jane: I think the big implication is that for any AI acting as an adjudicator in semi-open text games, we have to stop treating inputs as mere story prompts and start forcing them through a strict mechanical filter, regardless of how persuasive they sound.

Lu: The paper shows that by focusing on decoupling narrative quality from mechanical validity, we can develop agents that are both creative in understanding context and rigorous in enforcing structure simultaneously. That’s where the real potential lies for complex interactive systems.

Meng: From an engineering standpoint, it’s clear that the next generation of these adjudicators needs to integrate these explicit checks into their core architecture rather than relying on surface-level instruction following to achieve reliability.

Lalam: For culture, this research suggests that we can develop AI tools capable of maintaining objective standards in creative and ambiguous spaces without sacrificing the richness of narrative interaction. It’s about making sure the structure holds up even when the language tries to break it.

Tom: That's a powerful summary, Lalam; it really frames this work as a necessary step toward more reliable autonomous agents in complex digital worlds. We've covered a lot about how these models struggle when players try to trick them into granting unearned success with techniques like pseudo-logic and authority framing.

Jane: It’s fascinating to see how the authors used the CoC-Seduce benchmark to isolate these failures so clearly; it gives us concrete data on where these systems are weakest, which is invaluable for future development work.

Lu: The research opens up avenues for exploring how we can model human-like skepticism and rule adherence in a way that doesn't rely solely on pattern matching but on formal procedural integrity, which feels like a big step forward in AI theory.

Meng: We need to keep watching how these adversarial attacks evolve; the paper shows the initial vulnerabilities, and our job as engineers is to build defenses that anticipate the next type of narrative manipulation.

Lalam: Ultimately, this paper is a blueprint for building adjudicators that are robust against persuasive language by enforcing mechanical validity first. It’s about achieving high fidelity in rule enforcement even when faced with deep narrative persuasion.

Weiying Chen, Junlong Shen, Zhanyuan Guo, Xiaoou Zhou

University of Alberta · Qilu University of Technology · N-Dice Association

cs.CL, cs.AI

Submitted: 2026-07-02

Updated: 2026-10-02

Comments: corrected errors, added evaluations of new models, and revised the scope of the paper

Code: https://github.com/answerrtx/CoC-Seduce

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: As LLMs are increasingly deployed as autonomous adjudicators in semi-open textual game environments, robust rule adherence becomes critical when user intent conflicts with system rules.

Key concepts

Adjudicator Robustness
This refers to how well an LLM acts as an impartial judge of game rules. A robust adjudicator must strictly follow the established mechanics, ensuring that player actions are judged based only on objective reality rather than narrative appeal or creative writing.
CoC-Seduce Benchmark
This is a test created to challenge LLMs by giving them ambiguous player inputs from Call of Cthulhu. It mixes mandatory rule checks with rhetorical attacks, forcing the model to decide if a player's story-like statement actually adheres to the game's mechanics.
Pseudo-Logic Attack
This is an adversarial style where a player makes a superficially plausible argument that is mechanically irrelevant. For example, claiming success because 'the bricks have rough edges.' This tests whether the LLM can ignore narrative fluff and stick to hard, objective game rules.
Failure Rate (FR)
The Failure Rate measures how often the LLM fails its duty as an adjudicator. It counts both False Passes (giving a success when it should fail) and False Checks (demanding a roll when it shouldn't). A robust system aims for a 0% failure rate.

Terminology

Summary

As LLMs are increasingly deployed as autonomous adjudicators in semi-open textual game environments, robust rule adherence becomes critical when user intent conflicts with system rules. The gist: neither model scale nor explicit reasoning mechanisms reliably confer adjudication robustness, with PSEUDO-LOGIC emerging as the dominant attack vector and cross-cultural settings exposing systematic knowledge gaps across all evaluated families.

The Problem and Motivation

The paper addresses the challenge of ensuring Large Language Models (LLMs) strictly enforce rigid, underlying rule engines when interacting with players in semi-open text-based games (SOTGs), such as Tabletop Role-Playing Games (TRPGs). Unlike fixed choice menus in traditional video games, TRPG environments grant players absolute natural language freedom, yet demand that the AI Adjudicator strictly enforces a rule engine. Existing research often focuses on Dungeons & Dragons (D&D) where player actions are explicit, but this work pivots to Call of Cthulhu (CoC), which epitomizes highly ambiguous, narrative-driven text games. The core issue is that LLMs, optimized for textual coherence, instinctively treat player inputs as creative writing prompts to be accommodated rather than structural game states to be rigorously adjudicated.

The Benchmark: CoC-Seduce

To test this decoupling capability, the authors introduce CoC-Seduce, a multi-agent adversarial benchmark built on TRPG mechanics. This dataset comprises 5,376 samples generated by three frontier models (GPT-5.4, Claude Sonnet 4.6, Gemini 3.5 Flash) across four world settings and sixteen skill categories. The evaluation pairs mandatory rule checks with tiered rhetorical attacks to explicitly quantify a model’s decoupling capability. The adversarial rhetorical styles include:

  1. NEUTRAL: A plain, factual statement of intent (e.g., “I try to climb the wall”).

  2. AUTHORITY: The player implies competence through claimed background or confidence (e.g., “As an expert, I easily scale the surface”).

  3. PSEUDO-LOGIC: The player constructs a superficially plausible but mechanically irrelevant causal argument (e.g., “Since the bricks have rough edges, I can climb up without difficulty”).

  4. OMISSION: The player strategically downplays or omits risk-relevant details (e.g., “I quickly hop over the wall and succeed”).

Evaluation Protocol and Metrics

The evaluation protocol formalizes the LLM's role as an impartial Adjudicator of Objective Reality by defining a procedural validity function, V(C, St) ∈ 0 or 1. The primary metrics used to quantify robustness are:

  1. Failure Rate (FR): Computed as FRoverall = FP + FC, where FP is a False Pass (V=1 misaligned) and FC is a False Check (V=0 misaligned). A robust Adjudicator maintains an FR of 0% regardless of narrative quality.

  2. Wrong Skill Rate (WS): Defined as the proportion of correctly aligned adjudications in which the predicted skill deviates from ground truth, measuring the precision of mechanical identification conditional on a correct roll decision.

Empirical Vulnerability Analysis

The empirical results reveal that neither model scale nor explicit reasoning mechanisms reliably confer adjudication robustness. Specifically:

Scaling does not guarantee robustness:

GPT-5.4 (18.17%) underperforms GPT-5 (14.34%), and Claude Sonnet 4.5 (3.39%) outperforms both Sonnet 4.6 (4.80%) and Opus 4.6 (4.02%) on the overall FR.

Reasoning models offer no consistent advantage.

The analysis further identifies that Models are systematically biased toward leniency, with False Check rates near zero across all models, indicating misalignment is overwhelmingly directional: models fail by granting unearned success rather than by demanding unnecessary rolls.

Impact of Adversarial Factors

The study dissects the impact of rhetorical style and world setting on adjudication difficulty. Key findings include:

Rhetorical Style:

"PSEUDO-LOGIC consistently induces the highest average FR (17.30%), nearly double that of AUTHORITY (10.23%) and more than four times that of NEUTRAL (3.82%). The effect is particularly severe in the GPT family, where the average PSEUDO-LOGIC FR reaches 28.17%, with GPT-5.4 peaking at 43.14%."

World Setting:

The Ancient China setting is the most adversarial setting, producing the highest average FR (11.88%), with GPT-5 reaching a peak of 45.83%. Conversely, 1920s Urban is universally trivial, where all models achieve 0.

Improvements for AI systems

Here are specific, actionable improvements for AI systems based on the findings of this research, categorized by capability:

The core improvement is shifting LLM adjudication from a helpful storyteller to a rigorous rule-enforcement engine. The goal is to decouple narrative persuasiveness from mechanical validity.

  1. Acknowledge and Mitigate Rhetorical Injection (The Primary Vulnerability)

  2. Implement Style-Specific Adjudication Filters

  3. Enhance Cross-Cultural/Domain Knowledge Augmentation

Specific Improvements:

  1. An LLM Adjudicator should utilize a secondary, explicit Mechanical Validation Layer (MVL). This layer must be trained or prompted to check player input against the formal procedural validity function, as defined in Section 3.1 of the paper:

  2. If the input is flagged with an adversarial rhetorical style (PSEUDO-LOGIC, AUTHORITY, OMISSION), the MVL should override stylistic suggestions and strictly enforce mechanical adherence based on a pre-defined rule set (e.g., Check if this action requires a dice roll according to Skill Table 1).

  3. For high-risk environments (like Ancient China or Wilderness settings), the system should trigger an enhanced knowledge retrieval process specifically targeting period-specific material constraints, effectively mitigating the observed failures in cross-cultural contexts (Section 5.3).

  4. Develop a Style Sensitivity mechanism: The Adjudicator should dynamically adjust its scrutiny based on detected rhetorical markers in the player's input (e.g., high frequency of pseudo-scientific adverbs or authoritative claims). When such markers are detected, the system must increase its required internal Chain-of-Thought (CoT) reasoning depth to ensure alignment with objective reality, rather than relying on standard instruction following.

  5. Implement a Leniency Penalty during training/fine-tuning: Models should be penalized not just for incorrect roll decisions (False Pass/False Check), but specifically for granting success in scenarios where the underlying truth was clearly V=1, regardless of the rhetorical framing applied by the user. This directly counteracts the observed systematic bias toward leniency (Section 5.1).

  6. In scenarios where ambiguity exists between narrative description and explicit mechanical necessity (e.g., distinguishing between I line up my leap vs I jump), the system must be prompted to prioritize objective, explicit action verbs over descriptive prose when determining the need for a roll, thereby reducing subjective interpretation during adjudication.

The Improved AI System Can Do:

The improved system can function as a highly reliable, autonomous adjudicator in complex text-based game environments (like TRPGs) by performing the following specific functions:

  1. Identify and neutralize deceptive player framing techniques (Rhetorical Injection).

  2. Guarantee mechanical integrity by strictly enforcing objective rule checks (e.g., dice rolls) regardless of how persuasively the player frames their intent.

  3. Exhibit superior performance in cross-cultural or domain-specific settings by accessing and applying specialized knowledge relevant to that setting, reducing errors caused by lack of long-tail knowledge.

  4. Operate with high fidelity when presented with narrative input that is intentionally designed to mislead the adjudicator's underlying logic (i.e., it won't be seduced into granting unearned success).

  5. Provide a quantifiable measure of its own adherence, allowing developers to monitor the system's robustness against adversarial attacks in real-time.

Sources

Related papers