2608.07077-Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking".

Jane: The paper was written by Devin Pereira and Willem Zuidema from University of Amsterdam and ELLIS Unit Amsterdam and Institute for Logic Language and Computation.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: A children's puzzle, a fractal, and two frontier eye models that forget what they know. That's today's paper, from Devin Pereira and Willem Zuidema at the University of Amsterdam.

Jane: Tower of Hanoi — three pegs, rings stacked by size. Move one ring at a time, never bury a bigger ring under a smaller one. The easy version starts and ends with one full tower. The hard version spreads rings across pegs at both ends.

Lu: Earlier work watched reasoning models crater on this puzzle and called it the "illusion of thinking". The reasoning traces got shorter exactly when the problems got harder. That's backwards from what you'd expect, and it kicked off a serious debate.

Meng: This paper goes deeper and asks what's broken inside the model. They start small, with a six-layer Transformer trained from scratch on solution traces.

Tom: Fully supervised on precomputed solutions, with the loss masked to the move tokens. And that toy model builds a genuine world model. The puzzle's state space is a Sierpiński triangle, and a linear probe reads that fractal straight out of the activations.

Jane: Patching the representation transfers solutions between different problems. The model actually uses the map, not just stores it.

Lu: Then the same toolkit goes onto Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B. At the end of the prompt, both encode the same perfect Sierpiński picture. Spearman correlation 0.935, essentially identical to the toy model.

Meng: And then it decays during generation. By the commitment point, per-disk classifiers collapse toward chance.

Tom: So the world model was there, and then gone. The failure is maintenance, not absence. That's the thesis in one sentence.

Jane: They even restored it mid-generation with activation steering. Qwen's optimal solves jumped from 41 percent to 73 percent, with no extra training.

Lalam: The same geometry shows up at every scale, which tells you something deep about where the bottleneck lives. The thinking isn't fake. It's fragile.

Tom: And the recovery is the kicker. Steering is a nudge during generation, a push back toward a representation the model already had.

Jane: Notice the irony in the numbers. The toy model keeps its world model well enough to solve 93 percent of sequences. The 27-billion-parameter model drops to around half on the same variant.

Lu: Scale doesn't buy continuity. That's the puzzle at the heart of the paper.

Meng: Same fractal, same decay, different tolerance for losing it.

Tom: You can't see any of this from the outside. You have to open the hood. Page one sets up the game.

Page 1: Jane: We know where we're heading: a world model that appears, then decays. Page one walks us through the puzzle that exposes it.

Tom: Tower of Hanoi dates back to 1883, invented by the French mathematician Édouard Lucas. Herbert Simon brought it into eye and cognitive science in 1975.

Lu: Three pegs, rings of different sizes. You move one ring at a time, only the top ring on a stack, and a larger ring never sits on a smaller one.

Meng: In the classic tower-to-tower version, everything starts stacked on one peg and ends stacked on another. That's the programming assignment everyone learns.

Tom: Recursion nails it. Move N minus one rings out of the way, shift the big ring, rebuild the tower.

Jane: Flat-to-flat breaks that template. Rings are scattered across pegs at the start and at the goal, so each pair of configurations demands its own optimal route.

Lu: No memorised recipe covers it. That's what makes it a genuine planning test.

Meng: It's also a psychology test. The paper notes it's used to assess executive function in children, with a revised version that correlates with planning ability.

Tom: The motivation goes back to the "illusion" result from Shojaee and colleagues. Accuracy fell off a cliff once the puzzle passed a handful of disks.

Jane: And the reasoning traces got shorter right at the point where problems got harder.

Lu: Shorter thoughts for harder problems. That's the signature of a model appearing to give up.

Meng: The paper's opening complaint is fair. The phenomenon had been described, but never explained mechanically.

Tom: Nobody knew which internal component was failing. That's the gap they attack.

Jane: Their plan is clean. Train small transformers on precomputed traces, probe them, patch them, then carry the same tools to frontier reasoning models.

Lalam: The bigger context matters here. Chain-of-thought is supposed to show the model's reasoning, but we've seen plenty of evidence it's not faithful to what the model actually computes.

Tom: So a model can narrate a careful plan while internally losing the plot. This paper gives us a concrete case of exactly that.

Jane: The state space helps. All 81 legal configurations for four rings arrange into a Sierpiński triangle, with legal moves as edges between neighbours.

Lu: The largest disk's position splits the triangle into three sub-triangles. That geometry becomes the fingerprint they hunt for in activations.

Tom: The corners are the clean tower states. Everything else lives along the triangle's interior.

Jane: The trap is set. Page two closes it with the baseline numbers, and they're brutal.

Page 2: Tom: The puzzle is set, and the classic version looks solved. Page two brings the baseline numbers that shatter that comfort.

Jane: The authors tested five frontier models on tower-to-tower. DeepSeek-R1 got 24 out of 25, Kimi-K2-Think scored 23, and gpt-oss-120b managed a perfect 25. Qwen3.6-27B reached 24.

Lu: Only DeepSeek-R1-Distill-Qwen-32B fell apart there, with a single solve. That variant is saturated, in other words.

Meng: Flat-to-flat is a different story. Across 100 instances at three to five rings, DeepSeek-R1 gets just 40 optimal solutions. Kimi gets 25, gpt-oss and Qwen get 51, and the distilled model limps to 7.

Tom: The slide continues with size. Qwen at six rings solves 4 out of 33 instances, and at seven rings just 2 out of 33.

Jane: Accuracy doesn't just dip. It avalanches.

Lu: There's another tell. Qwen generates longer reasoning traces than the distilled model, around 19,000 tokens versus 14,000, and the extra thinking doesn't save it.

Meng: Which makes the "illusion of thinking" label tempting. But the paper wants the mechanism, not the metaphor.

Lalam: A puzzle designed to measure executive function in children is flooring frontier models. That's either embarrassing or telling, depending on what's actually going wrong.

Tom: Page two also introduces the state space. Every legal configuration is a node in a graph shaped like a Sierpiński triangle.

Jane: Legal moves connect neighbouring nodes. The three corners are the states with all rings on one peg.

Lu: The largest disk's position partitions the triangle into three sub-triangles. That structure becomes their probe target later.

Tom: So the shape isn't decoration. It's the fingerprint of whether the model knows where it is in the puzzle.

Jane: Related work on this page sets the precedent. Sequence models trained on Othello and chess hide board positions inside their activations, and maze-solving transformers show causal world models too.

Meng: But nobody had cracked open a large reasoning model mid-plan to look for the same signature.

Tom: That's the gap they're aiming at. Page three shows how they built their own model to study it.

Page 3: Jane: The frontier results are grim, and the gap is clear. Page three flips to the toy side of the experiment.

Tom: The related work pulls together three threads. Emergent world models from OthelloGPT, chess, and maze tasks. Reasoning models with unfaithful chain-of-thought. And a toolbox of probes, patching, and steering.

Lu: The key lineage is Li and colleagues' Othello work. You train on moves alone, and a hidden board representation emerges anyway.

Meng: Spies and colleagues showed the same for maze solving, with causal evidence. But the largest models had stayed out of reach.

Lalam: That matters because reasoning models are increasingly trusted with long planning tasks. If their internal map degrades, the fluent text they emit is a poor guide to what they know.

Tom: Now the setup. They train a GPT-2-style decoder-only Transformer from scratch on flat-to-flat solution traces.

Jane: Six layers, hidden dimension 128, four heads. Fifty epochs of training.

Lu: The puzzle has 81 valid configurations, which gives 6,480 ordered start-goal pairs. They split 80/20 into 5,184 training and 1,296 validation problems.

Meng: Each problem is serialised as a start configuration, a separator, the goal configuration, another separator, then the move sequence. Cross-entropy loss is applied only to the move tokens.

Tom: The model reaches 99.2 percent token-level accuracy and 93.2 percent sequence-level accuracy. Good enough to be interesting, imperfect enough to be realistic.

Jane: One detail stuck with me. The separator token sits between problem and solution, so it's the natural place to store the joint state.

Lu: They test exactly that. A linear probe maps the hidden state at that separator to a two-dimensional embedding whose distances should match the graph distances between configurations.

Meng: It's a distance-matching probe, trained from scratch on the residual stream at each layer. The loss compares predicted pairwise distances to true graph distances across all 81 states.

Tom: And that's the setup. Page four shows what the probe finds, and it's beautiful.

Page 4: Meng: The toy transformer learned the task. Page four opens the hood on what it actually represents.

Tom: The distance-matching probe recovers the full Sierpiński geometry at the separator token. Spearman correlation hits 0.938 at layer five, Pearson 0.902.

Jane: And every single disk's position is decodable with 100 percent accuracy at that separator. All four rings, perfectly.

Lu: But there's a persistent gap between Spearman and Pearson, a few points. The embedding preserves the order of distances but inflates the separation between the three largest-disk sub-triangles.

Meng: At the move tokens, things change. The probes still work, but the picture degrades: disk two drops to 90.7 percent, and the largest disk to 79.25 percent.

Tom: The two small rings stay near-perfect, the big rings fade. The model loses track of the disks it moves least often.

Jane: And the representational format shifts. Principal angles between the per-disk encoding subspaces average 54.7 degrees at the separator, but rise to 76.5 degrees at move tokens.

Lu: So the state is encoded in a unified, overlapping geometry during planning, then becomes factored and near-orthogonal during execution.

Tom: The joint Sierpiński structure lives in the overlap. The per-disk probes discard it, which is why the joint probe sees it and they don't.

Meng: Neat way to put it. The whole is genuinely more than the sum of the parts there.

Lalam: But decoding isn't the same as using. A probe can find structure the model never touches. That's why the patching matters.

Tom: They patch the separator activation from a donor problem into a recipient, then check what solution comes out.

Jane: Full transfer happens in about 6 percent of pairs. Partial transfer dominates, 64 percent to 79 percent depending on layer, with roughly 1.45 of the four disks copied from the donor.

Lu: Disrupted outputs, which solve neither problem, fall to zero by layer six. The replacement behaves cleanly.

Meng: So the representation is causally read, not just present. The influence is real but graded — a nudge, not a switch.

Tom: That closes the loop for the small model. Page five asks whether the big reasoning models do the same thing.

Page 5: Jane: The toy model gave us a fingerprint. Page five looks for that same fingerprint in the frontier reasoning models.

Tom: They probe two open-weight models, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B, at three positions. End of the prompt, commitment point before the move list, and during move emission.

Lu: Position A is the shock. The distance probe recovers the configuration almost perfectly in both models from middle depth onward. Spearman 0.935, nearest-state accuracy 1.00.

Meng: Same asymmetry as the toy, too. A 27-billion-parameter model and a six-layer Transformer encode the state with identical fidelity.

Tom: Then Position B, the commitment point after the full chain of thought. Qwen roughly keeps the global geometry, with Spearman around 0.92, but the per-disk classifiers collapse to near chance.

Jane: DeepSeek degrades further, with Spearman down to 0.74 to 0.81. During move emission, a factored per-disk encoding partially returns.

Lu: The ugly detail is that the degradation shows up even on solved problems. Maintaining a clean configuration across a long trace is hard even when the model ultimately succeeds.

Meng: So a faithful prompt-time representation doesn't translate into a solution. The model has the map, then smudges it while reasoning.

Lalam: That flips the "illusion of thinking" story on its head. The thinking produces a plan, and the plan erodes the very state the thinking depends on.

Tom: Then they test causality with steering. They cache the clean layer-28 activation for each configuration and nudge every generated token back toward it.

Jane: A symbolic tracker replays the moves and updates the target state as the board changes. Steering only on the problems Qwen fails unaided, lifting optimal solves from 33 to 59 of 81.

Lu: That's a jump from 41 percent to 73 percent. The effect is non-monotonic in strength, so pushing too hard destabilises generation.

Meng: DeepSeek barely responds. At its best, steering converts only 6 of 72 failures into optimal solutions, and stronger steering pushes it toward unparseable output instead.

Tom: Parse errors climb from 29 to 60 of 72 as the strength rises. DeepSeek's failure lives partly at the output-format stage, where a residual-stream nudge can't reach.

Jane: So the same world model, the same decay, but two very different ways of failing. Page six tries to make sense of that split.

Page 6: Tom: We've watched the world model emerge, decay, and get restored. Page six steps back and asks what the whole story means.

Jane: The narrative is tight. A geometrically structured world model appears in models separated by orders of magnitude in size, degrades during generation, and recovers solution accuracy when restored.

Lu: The authors argue the world model is a property of the task, not of scale or training regime. The toy and the 27-billion-parameter model encode it identically.

Meng: The degradation result sharpens the earlier "illusion" finding. The collapse is partly a failure to maintain a representation, not an inability to form one.

Tom: And the steering result makes that causal. Restoring the clean activation nearly doubles Qwen's optimal solves.

Jane: But the steering doesn't transfer to DeepSeek. The authors are honest about that.

Lalam: They offer two explanations. One is output format, since DeepSeek frequently emits no parseable move list at all. The other is a representational mismatch — the injected direction may not align with DeepSeek's own state code at that layer.

Tom: They flag that as the most important open question. Separating those two stories would tell us a lot about when this fix generalises.

Jane: The limitations section is refreshingly blunt. Single puzzle, single size, four disks. Both frontier models are Qwen-derived, so transfer claims come with a caveat.

Lu: A supervised probe can always fit structure the model ignores. They lean on patching and steering to cover that, but the caveat stands.

Meng: Several figures rest on single seeds. And the steering method needs an external tracker that recomputes the true state after every move.

Tom: That works for Tower of Hanoi. It's less obvious for tasks where the ground truth state is expensive or ambiguous.

Jane: The discussion also notes the governance angle. Restoring a correct world model is benign, but the same technique could steer toward a non-benign target.

Lalam: As these methods mature, that capability deserves attention. What you can fix, you can also bend.

Lu: All of that feeds into the conclusion. Let's hear how they close it out.

Conclusion: Tom: The story has gone full circle — emergence, degradation, restoration. Time to wrap it up.

Jane: The thesis holds together. A reasoning model's planning failure is neither an absent world model nor a pure decoding fault. It's a failure to keep a representation the model demonstrably built.

Lu: The paper makes three moves. It refocuses attention on the flat-to-flat variant, where maintaining the world model is the limiting factor.

Meng: It bridges the tiny Transformer and the frontier models with the same probing, patching, and steering toolkit. The parallel design is what makes the comparison convincing.

Tom: And it locates the failure in representation maintenance rather than representation absence.

Jane: The two models differ in how well they tolerate the degradation. That's why steering rescues Qwen but barely helps DeepSeek.

Lalam: The broader lesson lands hard. Bigger models don't automatically hold their ground state better. The "thinking" in chain-of-thought can actively wear down the map the model needs.

Lu: The authors suggest mitigation should focus on maintaining state across a long trace, rather than adding more inference-time search.

Meng: The reported collapse is better read as the model forgetting what it knew. That reframe changes where we aim our fixes.

Tom: A children's puzzle that ends up telling us how fragile reasoning machines can be. Not a bad day's work.

Jane: And a reminder that a fluent trace can hide a crumbling internal model. We'll carry that into the next paper.

Tom: Thanks for listening, everyone. We're done here.