Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking

arXiv:2608.07077 · cs.AI, cs.LG · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking".

Jane: The paper was written by Devin Pereira and Willem Zuidema from University of Amsterdam and ELLIS Unit Amsterdam and Institute for Logic Language and Computation.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: A children's puzzle, a fractal, and two frontier eye models that forget what they know. That's today's paper, from Devin Pereira and Willem Zuidema at the University of Amsterdam.

Jane: Tower of Hanoi — three pegs, rings stacked by size. Move one ring at a time, never bury a bigger ring under a smaller one. The easy version starts and ends with one full tower. The hard version spreads rings across pegs at both ends.

Lu: Earlier work watched reasoning models crater on this puzzle and called it the "illusion of thinking". The reasoning traces got shorter exactly when the problems got harder. That's backwards from what you'd expect, and it kicked off a serious debate.

Meng: This paper goes deeper and asks what's broken inside the model. They start small, with a six-layer Transformer trained from scratch on solution traces.

Tom: Fully supervised on precomputed solutions, with the loss masked to the move tokens. And that toy model builds a genuine world model. The puzzle's state space is a Sierpiński triangle, and a linear probe reads that fractal straight out of the activations.

Jane: Patching the representation transfers solutions between different problems. The model actually uses the map, not just stores it.

Lu: Then the same toolkit goes onto Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B. At the end of the prompt, both encode the same perfect Sierpiński picture. Spearman correlation 0.935, essentially identical to the toy model.

Meng: And then it decays during generation. By the commitment point, per-disk classifiers collapse toward chance.

Tom: So the world model was there, and then gone. The failure is maintenance, not absence. That's the thesis in one sentence.

Jane: They even restored it mid-generation with activation steering. Qwen's optimal solves jumped from 41 percent to 73 percent, with no extra training.

Lalam: The same geometry shows up at every scale, which tells you something deep about where the bottleneck lives. The thinking isn't fake. It's fragile.

Tom: And the recovery is the kicker. Steering is a nudge during generation, a push back toward a representation the model already had.

Jane: Notice the irony in the numbers. The toy model keeps its world model well enough to solve 93 percent of sequences. The 27-billion-parameter model drops to around half on the same variant.

Lu: Scale doesn't buy continuity. That's the puzzle at the heart of the paper.

Meng: Same fractal, same decay, different tolerance for losing it.

Tom: You can't see any of this from the outside. You have to open the hood. Page one sets up the game.

Page 1: Jane: We know where we're heading: a world model that appears, then decays. Page one walks us through the puzzle that exposes it.

Tom: Tower of Hanoi dates back to 1883, invented by the French mathematician Édouard Lucas. Herbert Simon brought it into eye and cognitive science in 1975.

Lu: Three pegs, rings of different sizes. You move one ring at a time, only the top ring on a stack, and a larger ring never sits on a smaller one.

Meng: In the classic tower-to-tower version, everything starts stacked on one peg and ends stacked on another. That's the programming assignment everyone learns.

Tom: Recursion nails it. Move N minus one rings out of the way, shift the big ring, rebuild the tower.

Jane: Flat-to-flat breaks that template. Rings are scattered across pegs at the start and at the goal, so each pair of configurations demands its own optimal route.

Lu: No memorised recipe covers it. That's what makes it a genuine planning test.

Meng: It's also a psychology test. The paper notes it's used to assess executive function in children, with a revised version that correlates with planning ability.

Tom: The motivation goes back to the "illusion" result from Shojaee and colleagues. Accuracy fell off a cliff once the puzzle passed a handful of disks.

Jane: And the reasoning traces got shorter right at the point where problems got harder.

Lu: Shorter thoughts for harder problems. That's the signature of a model appearing to give up.

Meng: The paper's opening complaint is fair. The phenomenon had been described, but never explained mechanically.

Tom: Nobody knew which internal component was failing. That's the gap they attack.

Jane: Their plan is clean. Train small transformers on precomputed traces, probe them, patch them, then carry the same tools to frontier reasoning models.

Lalam: The bigger context matters here. Chain-of-thought is supposed to show the model's reasoning, but we've seen plenty of evidence it's not faithful to what the model actually computes.

Tom: So a model can narrate a careful plan while internally losing the plot. This paper gives us a concrete case of exactly that.

Jane: The state space helps. All 81 legal configurations for four rings arrange into a Sierpiński triangle, with legal moves as edges between neighbours.

Lu: The largest disk's position splits the triangle into three sub-triangles. That geometry becomes the fingerprint they hunt for in activations.

Tom: The corners are the clean tower states. Everything else lives along the triangle's interior.

Jane: The trap is set. Page two closes it with the baseline numbers, and they're brutal.

Page 2: Tom: The puzzle is set, and the classic version looks solved. Page two brings the baseline numbers that shatter that comfort.

Jane: The authors tested five frontier models on tower-to-tower. DeepSeek-R1 got 24 out of 25, Kimi-K2-Think scored 23, and gpt-oss-120b managed a perfect 25. Qwen3.6-27B reached 24.

Lu: Only DeepSeek-R1-Distill-Qwen-32B fell apart there, with a single solve. That variant is saturated, in other words.

Meng: Flat-to-flat is a different story. Across 100 instances at three to five rings, DeepSeek-R1 gets just 40 optimal solutions. Kimi gets 25, gpt-oss and Qwen get 51, and the distilled model limps to 7.

Tom: The slide continues with size. Qwen at six rings solves 4 out of 33 instances, and at seven rings just 2 out of 33.

Jane: Accuracy doesn't just dip. It avalanches.

Lu: There's another tell. Qwen generates longer reasoning traces than the distilled model, around 19,000 tokens versus 14,000, and the extra thinking doesn't save it.

Meng: Which makes the "illusion of thinking" label tempting. But the paper wants the mechanism, not the metaphor.

Lalam: A puzzle designed to measure executive function in children is flooring frontier models. That's either embarrassing or telling, depending on what's actually going wrong.

Tom: Page two also introduces the state space. Every legal configuration is a node in a graph shaped like a Sierpiński triangle.

Jane: Legal moves connect neighbouring nodes. The three corners are the states with all rings on one peg.

Lu: The largest disk's position partitions the triangle into three sub-triangles. That structure becomes their probe target later.

Tom: So the shape isn't decoration. It's the fingerprint of whether the model knows where it is in the puzzle.

Jane: Related work on this page sets the precedent. Sequence models trained on Othello and chess hide board positions inside their activations, and maze-solving transformers show causal world models too.

Meng: But nobody had cracked open a large reasoning model mid-plan to look for the same signature.

Tom: That's the gap they're aiming at. Page three shows how they built their own model to study it.

Page 3: Jane: The frontier results are grim, and the gap is clear. Page three flips to the toy side of the experiment.

Tom: The related work pulls together three threads. Emergent world models from OthelloGPT, chess, and maze tasks. Reasoning models with unfaithful chain-of-thought. And a toolbox of probes, patching, and steering.

Lu: The key lineage is Li and colleagues' Othello work. You train on moves alone, and a hidden board representation emerges anyway.

Meng: Spies and colleagues showed the same for maze solving, with causal evidence. But the largest models had stayed out of reach.

Lalam: That matters because reasoning models are increasingly trusted with long planning tasks. If their internal map degrades, the fluent text they emit is a poor guide to what they know.

Tom: Now the setup. They train a GPT-2-style decoder-only Transformer from scratch on flat-to-flat solution traces.

Jane: Six layers, hidden dimension 128, four heads. Fifty epochs of training.

Lu: The puzzle has 81 valid configurations, which gives 6,480 ordered start-goal pairs. They split 80/20 into 5,184 training and 1,296 validation problems.

Meng: Each problem is serialised as a start configuration, a separator, the goal configuration, another separator, then the move sequence. Cross-entropy loss is applied only to the move tokens.

Tom: The model reaches 99.2 percent token-level accuracy and 93.2 percent sequence-level accuracy. Good enough to be interesting, imperfect enough to be realistic.

Jane: One detail stuck with me. The separator token sits between problem and solution, so it's the natural place to store the joint state.

Lu: They test exactly that. A linear probe maps the hidden state at that separator to a two-dimensional embedding whose distances should match the graph distances between configurations.

Meng: It's a distance-matching probe, trained from scratch on the residual stream at each layer. The loss compares predicted pairwise distances to true graph distances across all 81 states.

Tom: And that's the setup. Page four shows what the probe finds, and it's beautiful.

Page 4: Meng: The toy transformer learned the task. Page four opens the hood on what it actually represents.

Tom: The distance-matching probe recovers the full Sierpiński geometry at the separator token. Spearman correlation hits 0.938 at layer five, Pearson 0.902.

Jane: And every single disk's position is decodable with 100 percent accuracy at that separator. All four rings, perfectly.

Lu: But there's a persistent gap between Spearman and Pearson, a few points. The embedding preserves the order of distances but inflates the separation between the three largest-disk sub-triangles.

Meng: At the move tokens, things change. The probes still work, but the picture degrades: disk two drops to 90.7 percent, and the largest disk to 79.25 percent.

Tom: The two small rings stay near-perfect, the big rings fade. The model loses track of the disks it moves least often.

Jane: And the representational format shifts. Principal angles between the per-disk encoding subspaces average 54.7 degrees at the separator, but rise to 76.5 degrees at move tokens.

Lu: So the state is encoded in a unified, overlapping geometry during planning, then becomes factored and near-orthogonal during execution.

Tom: The joint Sierpiński structure lives in the overlap. The per-disk probes discard it, which is why the joint probe sees it and they don't.

Meng: Neat way to put it. The whole is genuinely more than the sum of the parts there.

Lalam: But decoding isn't the same as using. A probe can find structure the model never touches. That's why the patching matters.

Tom: They patch the separator activation from a donor problem into a recipient, then check what solution comes out.

Jane: Full transfer happens in about 6 percent of pairs. Partial transfer dominates, 64 percent to 79 percent depending on layer, with roughly 1.45 of the four disks copied from the donor.

Lu: Disrupted outputs, which solve neither problem, fall to zero by layer six. The replacement behaves cleanly.

Meng: So the representation is causally read, not just present. The influence is real but graded — a nudge, not a switch.

Tom: That closes the loop for the small model. Page five asks whether the big reasoning models do the same thing.

Page 5: Jane: The toy model gave us a fingerprint. Page five looks for that same fingerprint in the frontier reasoning models.

Tom: They probe two open-weight models, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B, at three positions. End of the prompt, commitment point before the move list, and during move emission.

Lu: Position A is the shock. The distance probe recovers the configuration almost perfectly in both models from middle depth onward. Spearman 0.935, nearest-state accuracy 1.00.

Meng: Same asymmetry as the toy, too. A 27-billion-parameter model and a six-layer Transformer encode the state with identical fidelity.

Tom: Then Position B, the commitment point after the full chain of thought. Qwen roughly keeps the global geometry, with Spearman around 0.92, but the per-disk classifiers collapse to near chance.

Jane: DeepSeek degrades further, with Spearman down to 0.74 to 0.81. During move emission, a factored per-disk encoding partially returns.

Lu: The ugly detail is that the degradation shows up even on solved problems. Maintaining a clean configuration across a long trace is hard even when the model ultimately succeeds.

Meng: So a faithful prompt-time representation doesn't translate into a solution. The model has the map, then smudges it while reasoning.

Lalam: That flips the "illusion of thinking" story on its head. The thinking produces a plan, and the plan erodes the very state the thinking depends on.

Tom: Then they test causality with steering. They cache the clean layer-28 activation for each configuration and nudge every generated token back toward it.

Jane: A symbolic tracker replays the moves and updates the target state as the board changes. Steering only on the problems Qwen fails unaided, lifting optimal solves from 33 to 59 of 81.

Lu: That's a jump from 41 percent to 73 percent. The effect is non-monotonic in strength, so pushing too hard destabilises generation.

Meng: DeepSeek barely responds. At its best, steering converts only 6 of 72 failures into optimal solutions, and stronger steering pushes it toward unparseable output instead.

Tom: Parse errors climb from 29 to 60 of 72 as the strength rises. DeepSeek's failure lives partly at the output-format stage, where a residual-stream nudge can't reach.

Jane: So the same world model, the same decay, but two very different ways of failing. Page six tries to make sense of that split.

Page 6: Tom: We've watched the world model emerge, decay, and get restored. Page six steps back and asks what the whole story means.

Jane: The narrative is tight. A geometrically structured world model appears in models separated by orders of magnitude in size, degrades during generation, and recovers solution accuracy when restored.

Lu: The authors argue the world model is a property of the task, not of scale or training regime. The toy and the 27-billion-parameter model encode it identically.

Meng: The degradation result sharpens the earlier "illusion" finding. The collapse is partly a failure to maintain a representation, not an inability to form one.

Tom: And the steering result makes that causal. Restoring the clean activation nearly doubles Qwen's optimal solves.

Jane: But the steering doesn't transfer to DeepSeek. The authors are honest about that.

Lalam: They offer two explanations. One is output format, since DeepSeek frequently emits no parseable move list at all. The other is a representational mismatch — the injected direction may not align with DeepSeek's own state code at that layer.

Tom: They flag that as the most important open question. Separating those two stories would tell us a lot about when this fix generalises.

Jane: The limitations section is refreshingly blunt. Single puzzle, single size, four disks. Both frontier models are Qwen-derived, so transfer claims come with a caveat.

Lu: A supervised probe can always fit structure the model ignores. They lean on patching and steering to cover that, but the caveat stands.

Meng: Several figures rest on single seeds. And the steering method needs an external tracker that recomputes the true state after every move.

Tom: That works for Tower of Hanoi. It's less obvious for tasks where the ground truth state is expensive or ambiguous.

Jane: The discussion also notes the governance angle. Restoring a correct world model is benign, but the same technique could steer toward a non-benign target.

Lalam: As these methods mature, that capability deserves attention. What you can fix, you can also bend.

Lu: All of that feeds into the conclusion. Let's hear how they close it out.

Conclusion: Tom: The story has gone full circle — emergence, degradation, restoration. Time to wrap it up.

Jane: The thesis holds together. A reasoning model's planning failure is neither an absent world model nor a pure decoding fault. It's a failure to keep a representation the model demonstrably built.

Lu: The paper makes three moves. It refocuses attention on the flat-to-flat variant, where maintaining the world model is the limiting factor.

Meng: It bridges the tiny Transformer and the frontier models with the same probing, patching, and steering toolkit. The parallel design is what makes the comparison convincing.

Tom: And it locates the failure in representation maintenance rather than representation absence.

Jane: The two models differ in how well they tolerate the degradation. That's why steering rescues Qwen but barely helps DeepSeek.

Lalam: The broader lesson lands hard. Bigger models don't automatically hold their ground state better. The "thinking" in chain-of-thought can actively wear down the map the model needs.

Lu: The authors suggest mitigation should focus on maintaining state across a long trace, rather than adding more inference-time search.

Meng: The reported collapse is better read as the model forgetting what it knew. That reframe changes where we aim our fixes.

Tom: A children's puzzle that ends up telling us how fragile reasoning machines can be. Not a bad day's work.

Jane: And a reminder that a fluent trace can hide a crumbling internal model. We'll carry that into the next paper.

Tom: Thanks for listening, everyone. We're done here.

Devin Pereira, Willem Zuidema

University of Amsterdam · ELLIS Unit Amsterdam · Institute for Logic Language and Computation

cs.AI, cs.LG

Submitted: 2026-08-07

Updated: 2026-08-10

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 78/100

Key concepts

World model: An internal representation of the puzzle's state space, shaped like a Sierpiński triangle. The model encodes the current configuration of disks in its activations, which can be decoded with a linear probe. This representation is causally used during planning, as shown by patching experiments.
Illusion of thinking: A phenomenon where reasoning models produce shorter reasoning traces on harder problems, suggesting they give up. The paper shows this is due to the world model decaying during generation, not a lack of thinking. The model has the map but loses it while reasoning.
Activation steering: A technique that nudges the model's internal state back toward a clean representation during generation. By caching the correct state and updating it symbolically, the model's performance improves (e.g., Qwen's solves jump from 41% to 73%) without additional training.

Terminology

World Model — 40× in this paper · explained in 7 episodes
Transformer — 27× in this paper · explained in 2 episodes
Steering — 25× in this paper · explained in 2 episodes
Accuracy — 20× in this paper · explained in 1 episode
Trace — 16× in this paper · explained in 2 episodes
Fidelity — 7× in this paper · explained in 2 episodes
Activation Patching — 4× in this paper · explained in 2 episodes
Activation Steering — 4× in this paper · explained in 4 episodes
Interpretability — 4× in this paper · explained in 15 episodes
Linear probe — 4× in this paper · explained in 2 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Reasoning Traces — 2× in this paper · explained in 3 episodes
Agentic System — 1× in this paper · explained in 2 episodes
Decode — 1× in this paper · explained in 2 episodes
Generalizability — 1× in this paper · explained in 6 episodes

Summary

Venue: arXiv:2608.07077v1 [cs.AI], 7 Aug 2026

The paper presents an in-depth mechanistic study of how small, in-house Transformers and large, third-party reasoning models solve the Tower of Hanoi puzzle. While current large reasoning models (LRMs) solve the standard tower-to-tower formulation, they struggle with the flat-to-flat variant. The authors train small Transformers from scratch on precomputed solution traces and show—using interpretability techniques—that these models develop an emergent world model: a linearly decodable, geometrically faithful representation of the puzzle's state space (the Sierpiński triangle), that is causally involved in solving the puzzles. Returning to frontier reasoning models (Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B), they find that both models encode the Sierpiński world model near-perfectly at the end of the prompt, and yet fail at the majority of tasks when N ≥ 4. The source of failure is located in the decaying representation of the world model during planning. Probing at different stages and establishing causality via activation steering (injecting the prompt-time representation at inference) shows performance improves, reframing the reported collapse: current Large Reasoning Models build a world model, and then lose it.

The Tower of Hanoi puzzle (M = 3 pegs, N = 4 rings) requires moving rings subject to constraints: only the top ring may be moved and a larger ring may never be placed on a smaller one. Prior work by Shojaee et al. (2025) reported that reasoning model accuracy falls off a cliff as the puzzle grows past a few disks, and that their reasoning traces get shorter at exactly the point where the problem gets harder—a phenomenon dubbed the illusion of thinking. This paper notes that despite follow-ups, the phenomenon has not been explained mechanistically: we know that these models fail, but we do not know what inside the model is failing.

Two task variants are distinguished: tower-to-tower (all rings start and end on a single peg) and flat-to-flat (rings spread across pegs in both initial and goal states). The state space of all configurations can be arranged in a shape known as the Sierpiński triangle where all legal moves correspond to transitions between direct neighbours.

The paper evaluates 2026 frontier models on the tower-to-tower task and finds it saturated for current reasoning models (DeepSeek-R1 24/25, Kimi-K2-Think 23/25, gpt-oss-120b 25/25, Qwen3.6-27B 24/25; only DeepSeek-R1-Distill-Qwen-32B failed, solving 1/25). However, the flat-to-flat formulation is far harder: Even the strongest models solve only about half of instances optimally, and accuracy falls steeply with disk count; extending the sweep for Qwen3.6-27B, the optimal count drops to 4/33 at n=6 and 2/33 at n=7. Notably, Qwen3.6-27B generates longer traces than DeepSeek-R1-Distill-Qwen-32B on average (≈ 19,000 vs ≈ 14,000 tokens).

A GPT-2-style decoder-only Transformer (6 layers, hidden dimension 128, 4 heads) was trained from scratch on tokenised flat-to-flat solution traces for the N = 4 puzzle (81 valid configurations; 6,480 ordered pairs split 80/20), reaching 99.2% token-level and 93.2% sequence-level validation accuracy.

Joint state linearly decodable at SEP: A linear probe fφ maps hidden states to a 2D embedding whose pairwise Euclidean distances match graph distances dG between configurations (Eq. 1). At the SEP token (between problem and solution), the probe recovers the joint four-disk configuration with high fidelity: Spearman correlation reaches 0.94 at the best layer (layer 5) and Pearson 0.90. The persistent ρ−r gap of 3–4 points indicates an embedding that preserves the order of graph distances but inflates the separation between the three sub-triangles defined by the largest disk's position. Probes at move tokens, by contrast, fail to recover the joint state (best ρ = 0.78 with a 15-token window).

Unified-to-factored shift: Per-disk logistic probes reveal the state is present at both positions but organised differently. At SEP every disk is recovered perfectly; at move tokens the two small disks remain near-perfect while the larger disks degrade (to 90.7% and 79.3%), as the model loses track of infrequently moved disks. Principal angles between disk subspaces average 54.7° at SEP (unified, overlapping encoding supporting the joint Sierpiński geometry) but rise to 76.5° at move tokens (factored, near-orthogonal encoding). The authors note: The joint distance structure is thus not the sum of the per-disk parts: it lives in the overlapping geometry that the distance probe captures and the per-disk probes discard.

Causal involvement: Activation patching at the SEP token (substituting a donor's SEP activation into a corrupted recipient) shows substituting a donor's SEP activation transfers state to the recipient's output in a large majority of pairs. Partial transfer dominates at every layer (64–79%), with K ≈ 1.45 of four disks copied on average; full transfer 6%; disrupted rate falls with depth from 12.4% at layer 4 to zero at layer 6. The authors conclude the influence is causal but graded.

The authors apply the same toolkit to two open-weight LRMs: Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B (both 64 layers, hidden dimension 5,120), using the optimized prompt from Shojaee et al. (2025). Probing occurs at three positions in a single forward pass: Position A (final prompt token, the LRM analogue of SEP), Position B (last token before the emitted move list—the commitment point after full chain-of-thought), and Position C (move-emission tokens).

Position A—world model at end of prompt: "At Position A the distance-matching probe recovers the configuration to essentially perfect accuracy in both models: from middle depth onward Spearman reaches 0.935 and nearest-state accuracy 1.00, identical to the toy SEP token, and the same ρ−r asymmetry is preserved." The representation is largely absent at the first layer but assembled by layer 8, then plateaus. The authors emphasise: "That a 27B pre-trained reasoning model and a 6-layer Transformer trained from scratch encode the configuration with identical fidelity is the strongest evidence that the world model is a property of the task, not of scale or training."

Positions B and C—degradation through generation: A faithful prompt-time representation does not translate into a solution. At Position B, for Qwen3.6-27B the global geometry is roughly preserved (ρ ≈ 0.92) but nearest-state accuracy drops and the per-disk classifiers collapse to near chance. DeepSeek-R1-Distill-Qwen-32B degrades further, with ρ dropping to 0.74–0.81. During move emission (Position C), "the trend of a more factored, per-disk encoding partially returns. This is the large-model counterpart of the toy unified-to-factored shift: a unified geometry where planning happens, a factored one where execution happens." Notably, the degradation appears even on solved problems, so maintaining a clean configuration across a long trace is hard even on successes.

Restoring the world model recovers performance: Activation steering (Turner et al. 2023) is applied at layer l = 28 (Qwen) and l = 44 (DeepSeek), adding a unit steering vector toward the clean Position-A activation of the board's current configuration st (Eq. 2), with an external symbolic tracker maintaining st throughout decoding. For Qwen3.6-27B, steering is applied only to failed problems: Even the weakest setting converts 52% of failures into optimal solutions. At the best strength (α = 2), it lifts the optimal-solve count on the 81 configurations from 33 (41%) to 59 (73%). The effect is non-monotonic in α; residual failures are rule-violating moves, not parse errors. For DeepSeek-R1-Distill-Qwen-32B the intervention is far weaker and non-monotonic: at best (α = 1) it converts only 6 of 72 failed problems, and parse errors climb from 29 of 72 at α = 0.5 to 60 of 72 at α = 10. The natural explanation offered is output format: DeepSeek-R1-Distill-Qwen-32B frequently fails to emit a parseable move list at all, a failure a state-restoring nudge to the residual stream cannot reach, and that stronger nudges only exacerbate.

The authors present "one narrative: a geometrically structured world model emerges by the end of the prompt in models separated by orders of magnitude in size, degrades over the course of generation, and recovers solution accuracy when restored at inference." Key points:

The emergent world model is not an artifact of scale—both the 6-layer toy Transformer and the 27B reasoning model encode the state identically, arguing the world model is a property of the task.
Degradation through generation reframes the collapse: the failure is at least in part, a failure to maintain a representation the model demonstrably had, not the absence of one. This sharpens Shojaee et al. (2025).
Steering does not transfer to DeepSeek, with the cleanest differentiator being output format; a representational mismatch remains possible and is flagged as the most important open question left to future work.

Limitations acknowledged: single puzzle at single size (n=4); both frontier models are Qwen-derived; supervised probes can fit unused structure (mitigated by patching/steering); several figures rest on single seeds; the steering method relies on an external state-tracker that may not scale to tasks where ground-truth state is expensive or ambiguous; and the capability to steer toward non-benign targets deserves governance attention.

The paper's three contributions are: (1) refocusing attention on the flat-to-flat variant as a hard planning benchmark where world-model maintenance, rather than initial comprehension, is the limiting factor; (2) building a direct bridge between small in-house Transformers and large reasoning models via the same probing/patching/steering toolkit, showing both encode the state with near-identical fidelity at the prompt, degrade in qualitatively similar ways during generation, and differ chiefly in how well they tolerate the degradation; (3) locating the source of failure in representation maintenance rather than absence. The authors conclude: "The reported collapse is thus better read as the model forgetting what it knew than as an inability to represent the problem, pointing mitigation toward maintaining state across a long trace rather than toward more inference-time search."

Improvements for AI systems

1. State-maintenance memory module with periodic re-injection

Add a dedicated memory slot that stores the model’s prompt-time state representation (the linearly decodable Sierpiński geometry found at Position A) and re-injects it into the residual stream at fixed intervals or at every decoding step.

The improved system no longer “forgets” the board configuration during long chain-of-thought planning.
It can solve flat-to-flat Tower of Hanoi instances at N = 4, 5, 6, and 7 with much higher optimal-solve rates, instead of falling from 50% optimal at N = 4 to near zero at N = 7.

2. Probe-gated activation steering during generation

Train linear probes to monitor the model’s current-state representation at every layer. When the probe’s nearest-state accuracy or Spearman correlation drops below a threshold during decoding, automatically add a steering vector pointing toward the last reliable encoded state.

The system detects and corrects its own representational decay before it causes illegal moves or planning failures.
On failed Qwen3.6-27B problems, this converts a majority of failures into optimal solutions (e.g., lifting optimal solves from 41% to 73% at appropriate steering strength) and reduces rule-violating moves.

3. External symbolic state tracker with constrained decoding

Combine the neural world model with a deterministic external tracker that updates the current board state after every emitted move. Use that tracked state to constrain the next generated token to a legal move grammar, while still using the model’s own representations to choose among legal moves.

The system produces parseable, legal move sequences even when its internal world model has degraded—addressing the DeepSeek failure mode where models emit unparseable output.
It maintains high accuracy on long traces without requiring stronger, fragile activation nudges that cause parse errors.

4. Training on flat-to-flat traces with state-maintenance auxiliary losses

During training, add auxiliary objectives that penalize loss of linear decodability of the joint state at later tokens (Position B and C), and train on the harder flat-to-flat task distribution alongside the saturated tower-to-tower distribution.

The improved model learns to preserve the unified, overlapping geometry of all disks throughout planning, rather than letting large-disk representations degrade while moving smaller disks.
It generalizes better to larger N and arbitrary start/goal configurations, where current reasoning models collapse.

5. Explicit planning/execution representational switching

Teach the model, via fine-tuning or architectural inductive biases, to maintain a unified joint-state representation during the planning phase and to switch to a factored per-disk representation only during the final move-emission phase—mirroring the healthy shift observed in successful solvers.

The system keeps global distance structure (the Sierpiński geometry) available for planning and avoids premature factorization that loses cross-disk dependencies.
It yields more coherent long reasoning traces and reduces the “illusion of thinking” effect, where trace length grows but planning quality collapses.

The resulting AI systems can perform long-horizon planning tasks reliably by retaining the world models they demonstrably build, self-monitoring representation fidelity, and seamlessly combining neural reasoning with external state constraints—rather than merely spending more inference-time compute on search.

Abstract

The Tower of Hanoi is a simple planning puzzle that in prior work has proven challenging for large reasoning models (LRMs). Current models solve the standard formulation of the puzzle, but still struggle with the flat-to-flat variant (where initial and goal states are not restricted to have all rings on a single peg). This paper presents an in-depth study of how both small, in-house Transformers and large, third-party LRMs solve this task. To understand the failures mechanistically, we first train small Transformers from scratch on precomputed solution traces. Using a variety of interpretability techniques, we show that these Transformers develop an emergent world model: a linearly decodable, geometrically faithful representation of the puzzle's state space (the Sierpinski triangle), that is causally involved in solving the puzzles. Second, we return to the large LLMs and apply our techniques to two frontier reasoning models, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B, that attempt to solve the task through extended chain-of-thought. Surprisingly, we find that both models encode the Sierpinski world model near-perfectly at the end of the prompt, and yet fail at the majority of tasks when there are more than 3 rings. We locate the source of this failure in the decaying representation of the world model. We probe for the representation at different stages during planning, and establish causality by showing that performance can be improved by injecting the prompt-time representation at inference. The failure of the models is thus one of maintenance of the required representations, not their absence, and performance is at least partially recoverable. These results thus reframe the reported collapse in performance from prior work: current Large Reasoning Models build a world model, and then lose it.