The Shape of Wisdom: How Language Models Converge on an Answer

summary

Video file (mp4)

The gist

The gist: An answer is better understood as a trajectory than as a single endpoint, and correctness and stability are different properties that can be separated by tracking state, motion, and

In short

This study tracked how language model answers move across their internal layers during factual question answering. Researchers found that answers follow specific trajectories characterized by state, motion, and boundary distance. Correctness and stability are distinct properties; the best performance comes from trajectories that not only reach the correct answer but also maintain it throughout the network.

Key concepts

Answer Centroid
This is a central point in a model's internal state space where questions leading to the same answer converge. It represents a learned 'ideal' state for that specific answer, showing how different questions about related topics lead to similar internal representations.
State, Motion, and Boundary Distance
These three metrics describe the movement of an answer's preference through the model layers. State shows which option is currently favored; motion indicates if the next layer pushes that preference up or down; and boundary distance measures how sensitive the preferred answer is to small changes.
Trajectory Regimes
Models exhibit four main types of movement: stable-correct, stable-wrong, unstable-correct, and unstable-wrong. The most surprising finding is that 'unstable-correct' trajectories are the largest group, meaning models often successfully reach the right answer even if their internal path is erratic.

Terminology used across episodes

This episode discusses

The paper

The Shape of Wisdom: How Language Models Converge on an Answer · Read on arXiv

Shailesh Rana

When a language model answers a factual question, where does its internal state go? We asked four open models (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B and Llama-3.2-1B) 2,256 factual questions whose answers recur, such as the country a person is a citizen of. At every layer, we recorded the state from which the answer is produced. Questions that receive the same answer converge on a shared region of state space, which we summarise by its centre, the answer centroid. The answer becomes readable within a single layer. In every model, the share of questions whose state lies nearest its own answer's centroid jumps at one layer, between 47% and 71% of the way through the network. The centroids follow the answer the model gives rather than the question: the centroid of a country learned from questions about people's citizenship also identifies that country in questions about towns and films. The model also uses the centroids. After the jump, moving the state of a wrongly answered question to the right answer's centroid makes the model give the right answer in 83%-97% of cases late in the network, against 2%-4% for a random move of the same size. Before the jump, moves of the same size almost never work. Near misses, in which the model answers wrongly but picks the right option in multiple choice, can be seen from the onset layer, where the right answer's centroid is already among the nearest to the state. This signal is about as informative as the right answer's rank in the model's final output, and more informative than the model's confidence in its reply.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "The Shape of Wisdom".

Jane: Here's the gist of the paper. The gist: An answer is better understood as a trajectory than as a single endpoint, and correctness and stability are different properties that can be separated by tracking state, motion, and boundary distance.

Tom: Answers are trajectories Language models do not simply choose an answer at the output layer; in a 9,000-trajectory MMLU study across Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instructv0.3, the score of the answer moves across depth in structured ways.

Jane: We describe each trajectory with three quantities: the current answer margin, the next-layer change in that margin, and the distance from a decision flip.

Tom: The main empirical picture is that correctness and stability are different; the largest group is unstable-correct, not stable-correct.

Paper summary: Tom: So Jane, so we're looking at this paper called "The Shape of Wisdom: How Language Models Converge on an Answer." Basically, it claims that language models don't just pick an answer at the very end; instead, the score of that answer moves through different layers in a structured way.

Jane: That sounds like they're arguing for trajectories rather than a single endpoint. What does that actually mean for how we think about these models?

Tom: They describe each trajectory with three things: the current answer margin, the next-layer change in that margin, and how far away it is from a decision flip. The main finding is that correctness and stability are actually different properties when you trace this movement.

Jane: So if we have correct answers, they aren't always stable? That's an interesting distinction.

Lu: The paper shows that the largest group of trajectories, in terms of raw count across all models, is unstable-correct rather than stable-correct >

Tom: Let's look at the specifics from "The Shape of Wisdom: How Language Models Converge on an Answer." They did a study with nine thousand trajectories across three models: Qwen2 point 5-7B-Instruct, Llama-three point one-8B-Instruct, and Mistral-7B-Instructv0 point 3 > <ref:2606.01202#pg1,Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B>

Jane: And they found that the answer score moves across depth in structured ways instead of just being picked at the output layer. They call this movement state, motion, and boundary distance >

Meng: So they're trying to map out exactly how a model gets from uncertainty to a final choice through its hidden layers. That makes sense for debugging when things go wrong at scale >

Lalam: The paper says that the centroids follow the answer the model gives rather than the question itself, which is pretty cool because it means that if a country's name appears in citizenship questions, that country's identity shows up in questions about towns and films too >

Tom: They also showed that after a certain layer jump, if you move a wrongly answered state to the right answer’s centroid, it makes the model give the right answer in about eighty-three percent to ninety-seven percent of cases late in the network >

Jane: That suggests that we can use these centroids as markers. If we see where a trajectory is headed toward, it tells us something about its final outcome later on.

Lu: It's a practical way to see which answers are settled, which remain fragile, and which measured sources actually move them >

Conclusion: Tom: So what's the big picture here with "The Shape of Wisdom: How Language Models Converge on an Answer"? The authors are showing that final accuracy is a mix of two things: whether the answer is correct and whether the model has actually settled on it >

Jane: It means we need to look at how the correct option pulls ahead earlier in the network and then keep it ahead through later layers. The paper points out that attention gives us clear small layer-by-layer pushes toward stable-correct answers >

Meng: From a deployment standpoint, this suggests that if we can identify which components—like attention or MLP scalars—are pushing the margin, we might be able to guide the model's path instead of just hoping for a good final output >

Lalam: It’s about getting it right and then staying settled. The authors say that the best behavior isn't just getting the answer right, but getting it right and then staying settled >

Lu: I think this work is valuable because it gives us a concrete way to see which answers are fragile, which are stable, and which parts of the network are most important for forming an answer >

More episodes

← Home