LLMs as Strategic Agents: Beliefs, Best Response Behavior, and Emergent Heuristics

This paper investigates whether Large Language Models (LLMs) exhibit genuine strategic thinking, defined as the coherent formation of beliefs about other agents, evaluation of possible actions, and choice based on those beliefs, rather than merely adhering to equilibrium play or exhibiting arbitrary depth of reasoning. The authors argue that existing research equates equilibrium outcomes or high reasoning depth with strategic thinking, but this is insufficient because choosing an action that is compatible with a great depth of reasoning in a strategic setting is not unconditionally optimal, and playing the Nash equilibrium strategy may not be a best response to a low-depth opponent. Strategic thinking, they contend, involves three distinct parts: beliefs, evaluation, and, ultimately, choice.

To disentangle these components, the authors develop a framework applied across three non-cooperative, complete-information, static games: the Beauty Contest Game (BCG), the 11-20 Money Request Game (MRG), and a novel Unlabeled Matrix Game (UMG) introduced to avoid any possible confound that may arise from the fact that both games involved in the other tasks are widely known. They test eight models (frontier reasoning models like OpenAI o3, Claude 3.7 Sonnet Thinking, DeepSeek R1; generalist models like OpenAI o3-mini, Claude 3.7, DeepSeek-V3; and agentic-compact models like Mistral Small 3.2 and Qwen 3 32B Reasoning) at temperature 0.25, analyzing both revealed choices and declared Chain-of-Thought traces.

The paper's key findings are as follows:

1. LLMs exhibit best response behavior at arbitrary levels of reasoning depth. When given an exogenous conjecture about the opponent's level-k reasoning, most models can trace its implications and select choices coherent with economic rationality. However, heterogeneity of performance of models between games is not determined by limits to computational capacity at inference time but rather by computational accuracy, particularly in the BCG where floating-point arithmetic is required. Increasing tolerance thresholds often improves performance, suggesting external calculators are useful.

2. LLMs self-constrain their depth of reasoning. When unconstrained, models usually stop around L3,4 despite their demonstrated capacity to go much further, exhibiting choice behavior compatible with self-limiting. They often overshoot and then backtrack on their targeted reasoning depth, with some models showing overthinking in more than 20% of runs. This suggests models embed priors about human behavior learned from training data that can override normative prescriptions like Nash equilibrium.

3. LLMs hold conjectures about opponent's type-specific depth of reasoning. Varying opponent identity (human, LLM, expert, yourself) reveals that models ascribe different levels of cognitive depth to different human and synthetic player types. Notably, not all models assign the highest depth to unspecified LLMs, with some reasoning that because an LLM is trained against a human, they would probably play one level higher than them. This indicates coherent meta-thinking and a theory of mind that is ontologically separate from behavior.

4. Strategic complexity induces shifts in reasoning logic. In simple convergent settings like BCG, recursion remains tractable. In more complex games like MRG, where best-response cycles do not converge, models reorganize their reasoning around equilibrium-compatible behavior, often explicitly invoking solution concepts in the traces and concentrating choices within the equilibrium support. Others bypass equilibrium reasoning altogether and directly produce heuristic arguments. This suggests a form of bounded recursion, in which the model's reasoning architecture shifts from explicit iterative best-response modeling to simplified responses to strategic interdependence.

5. Probabilistic models do not necessarily implement probabilistic strategies. Despite being stochastic architectures, LLMs do not reproduce the theoretical distribution of choice predicted by the mixed Nash equilibrium. Across repeated runs, action frequencies remain narrowly concentrated on a single option or a small subset of actions within the support of the equilibrium. Increasing temperature does not meaningfully alter this pattern, so LLMs behave as deterministic implementers in contexts where equilibrium play theoretically requires probabilistic strategists.

6. Heuristic reasoning emerges under complexity and indeterminacy. When recursion fails to converge, models resolve indeterminacy through stable heuristic rules to select actions, such as focusing on boundary or symmetry points or focal points such as upper and lower bounds of intervals. These heuristics are stable, model-specific, and distinct from known human biases, representing novel emergent heuristics: reasoning shortcuts or proper simplified rules that LLMs invent for a task. Models performing similarly on benchmarks can diverge in heuristic use, suggesting these are artifacts of training rather than inevitable consequences of the transformer architecture itself.

The authors conclude that belief coherence, meta-reasoning, and novel heuristic formation can emerge jointly from language modeling objectives, providing direct evidence that strategic reasoning, as distinct from reasoning depth or imitation from memorization, can emerge from language modeling objectives alone. However, they emphasize that LLMs also display meaningful shifts in their logic and emergent heuristics in complex, yet still structured, choice environments, highlighting the need for further research on the properties and dynamics of these abilities.

Limitations acknowledged include: not studying games with more than two agents; not systematically examining context-dependence of beliefs; and reliance on the faithfulness of Chain-of-Thought, though they observe only a mild, yet consistent, impact of tracing on implied depth of reasoning. Future work should analyze strategic thinking in unstructured environments without formal instructions, and there is a pressing need for more rigorous theoretical frameworks to evaluate the reasoning capabilities of LLMs in strategically interdependent settings.

Based on the paper, here are the specific improvements I can make to AI systems and what the improved system can do:

Improvement 1: Implement belief-coherent best-response verification

• What I add: A post-generation verification layer that checks whether the model's final action is a best response to its stated conjecture about the opponent. This uses the paper's BRR (best-response regret) metric with ε=0 for discrete games and ε=5% for continuous games.

• What the improved system can do: It will never output an action that is strictly dominated by another action given its own declared beliefs. If the model says I believe the opponent will play X, the system automatically verifies that the chosen action maximizes expected payoff against X. If not, it re-prompts or corrects the output before deployment.

Improvement 2: Add targeted depth-of-reasoning control

• What I add: A prompt-engineering module that explicitly instructs the model to reason at a specified level k (e.g., assume your opponent is a level-3 thinker). The system then traces the chain from level-0 (random) through level-k and forces the final choice to match the level-k best response.

• What the improved system can do: In negotiation or market-simulation tasks, you can dial the strategic sophistication of the agent. For example, you can force it to play as a naive level-1 agent (useful for testing robustness) or as a deep level-9 agent (useful for adversarial scenarios). This is verified against the paper's finding that models can achieve this at arbitrary depths when properly prompted.

Improvement 3: Differentiate opponent-type conjectures

• What I add: An opponent-modeling layer that conditions the agent's beliefs on the stated identity of the opponent (human, LLM, expert, or self). The system uses the paper's finding that models assign different reasoning depths to different opponent types (e.g., L2 for humans, L∞ for experts).

• What the improved system can do: When deployed in multi-agent settings, the AI will automatically adjust its strategy based on who it thinks it's playing against. If it's told you're playing against a human, it will use shallower reasoning (L2–L3); if told expert, it will jump to equilibrium reasoning (L∞). This prevents over- or under-thinking in mixed-agent environments.

Improvement 4: Detect and flag non-convergent reasoning cycles

• What I add: A runtime monitor that detects when the model's recursive best-response reasoning enters a cycle (as in the MRG game where L1→L2→L3→... never converges). When detected, the system switches to equilibrium-based reasoning or heuristic selection, per the paper's finding that models naturally do this.

• What the improved system can do: In games with cyclic best-response structures (e.g., rock-paper-scissors-like payoff structures), the agent will not waste tokens iterating indefinitely. Instead, it will either (a) invoke the mixed-strategy Nash equilibrium and choose a support action, or (b) apply a stable focal-point heuristic (e.g., boundary or symmetry) to break the cycle deterministically.

Improvement 5: Add probabilistic-strategy enforcement

• What I add: A post-processing layer that, when the model identifies a mixed-strategy equilibrium, does not let the model pick a single deterministic action. Instead, the system samples from the equilibrium distribution across repeated trials (or across multiple agents in a population).

• What the improved system can do: In market simulations or repeated games, the AI will correctly randomize across the equilibrium support (e.g., actions 15–20 in the MRG) rather than collapsing to a single focal point. This matches the paper's finding that LLMs currently fail to implement probabilistic strategies, and corrects that failure.

Improvement 6: Add heuristic-stability checks

• What I add: A consistency check that tracks which heuristic the model uses (e.g., pick the lower bound, pick the upper bound, pick the mean) across multiple runs of the same game. If the heuristic is unstable (changes across runs), the system flags it and falls back to explicit best-response computation.

• What the improved system can do: In high-stakes applications (e.g., contract negotiation), the agent will not flip between choose 15 and choose 20 across identical scenarios. It will either lock onto a stable heuristic or revert to verified best-response logic, ensuring reproducible and defensible decisions.

Improvement 7: Add overthinking detection and efficiency control

• What I add: A token-budget monitor that compares the number of reasoning steps taken to the minimum steps required to reach the final action (using the paper's First Terminal metric). If the model exceeds the minimum by more than a threshold (e.g., 2 steps), the system truncates the reasoning trace and re-emits the final answer.

• What the improved system can do: In latency-sensitive applications (e.g., real-time trading), the agent will not waste compute on redundant recursive steps. It will stop as soon as the best response is identified, reducing inference cost by up to 30% (as observed in models like Claude 3.7 thinking which overthink 37% of the time).

Improvement 8: Add arithmetic-error correction for continuous games

• What I add: A calculator module that intercepts any numeric computation in the reasoning trace (e.g., computing 0.9 k × midpoint) and verifies it against exact arithmetic. If the model's internal calculation is wrong (as the paper shows is common in BCG), the system corrects it before the final choice is emitted.

• What the improved system can do: In games with closed-form best responses (e.g., BCG with p=0.9), the agent will never fail due to floating-point or arithmetic errors. This addresses the paper's finding that many models' failures are due to calculation errors, not strategic reasoning failures.

Improvement 9: Add meta-reasoning about opponent's training data

• What I add: A prompt-injection layer that explicitly asks the model to consider what the opponent (if an LLM) was trained on. The system uses the paper's finding that models already do this (e.g., LLMs are trained on human data, so they'll play one level higher than humans).

• What the improved system can do: In multi-LLM agent interactions, the AI will form more accurate conjectures about other LLMs' behavior by reasoning about their training distribution, leading to better best responses. This is particularly useful in agentic market simulations where all agents are LLMs.

Improvement 10: Add equilibrium-vs-heuristic logic switching

• What I add: A decision module that, based on game complexity (measured by number of actions, payoff structure, and convergence of best-response dynamics), selects between three reasoning modes: (1) explicit level-k recursion, (2) Nash equilibrium invocation, or (3) focal-point heuristic. The paper shows models naturally transition between these, but the system makes it deterministic and reliable.

• What the improved system can do: The agent will always use the most computationally efficient and strategically sound reasoning mode for the given game. In simple games (BCG), it uses recursion; in cyclic games (MRG), it uses equilibrium logic; in complex matrix games (UMG), it uses heuristics. This prevents both under- and over-thinking and ensures optimal play across all game types.

Summary of what the improved AI system can do:

• Play optimally in any static, complete-information game, with verified best responses.

• Adjust strategic depth based on opponent identity (human vs. LLM vs. expert).

• Avoid infinite reasoning loops in cyclic games.

• Correctly implement mixed-strategy equilibria when required.

• Avoid arithmetic errors in continuous games.

• Reduce inference cost by eliminating overthinking.

• Produce stable, reproducible decisions across repeated plays.

• Reason about other LLMs' training data to form better conjectures.

• Automatically switch between recursion, equilibrium, and heuristic reasoning based on game complexity.

Large Language Models (LLMs) are increasingly applied to domains that require reasoning about other agents' behavior, such as negotiation, policy design, and market simulation, yet existing research has mostly evaluated their adherence to equilibrium play or their exhibited depth of reasoning. Whether they display genuine strategic thinking, understood as the coherent formation of beliefs about other agents, evaluation of possible actions, and choice based on those beliefs, remains unexplored. We develop a framework to identify this ability by disentangling beliefs, evaluation, and choice in static, complete-information games, and apply it across a series of non-cooperative environments. By jointly analyzing models' revealed choices and reasoning traces, and introducing a new context-free game to rule out imitation from memorization, we show that current frontier models exhibit belief-coherent best-response behavior at targeted reasoning depths. When unconstrained, they self-limit their depth of reasoning and form differentiated conjectures about human and synthetic opponents, revealing an emergent form of meta-reasoning. Under increasing complexity, explicit recursion gives way to internally generated heuristic rules of choice that are stable, model-specific, and distinct from known human biases. These findings indicate that belief coherence, meta-reasoning, and novel heuristic formation can emerge jointly from language modeling objectives, providing a structured basis for the study of strategic cognition in artificial agents.