Small Foundation Models of Human Cognition and Behaviour

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Small Foundation Models of Human Cognition and Behaviour".

Jane: The paper was written by Nick Oh and Fernand Gobet from socius labs and London School of Economics and Political Science.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everyone. Today we’re digging into a paper that’s got a wonderfully ambitious title: “Small Foundation Models of Human Cognition and Behaviour.” And Jane, I have to say, the word “small” in that title is doing a lot of heavy lifting.

Jane: It really is, Tom. When we think of foundation models, we usually picture these massive, hundred-billion-parameter beasts. But this team from Socius Labs and the LSE, Nick Oh and Fernand Gobet, they’re asking a pretty cheeky question: do you actually need all that bulk to simulate how a person thinks?

Tom: And their answer, at least for a lot of tasks, is a resounding no. They took the big Centaur model, which is a seventy-billion-parameter model fine-tuned on human behavioral data, and they tried to shrink it down. They trained fourteen smaller models, some as small as one hundred thirty-five million parameters.

Jane: Right, that’s a factor of over five hundred smaller than the original. And the wild part is, when they tested these models on the same psychological experiments they were trained on, the small ones basically matched the giant one. It’s like discovering you can drive a compact car at the same speed as a Formula one car, as long as you’re just going around the block.

Tom: That’s a great analogy, because the difference shows up when you leave the block. On new, unseen experiments, the bigger models still have a clear advantage. So the size isn’t useless, it’s just not for the job everyone thought it was for.

Jane: Exactly. The paper is essentially saying that scale is a tool for generalization, not for fitting the training data. And that reframes how we should think about building these cognitive proxies in the first place.

Tom: So we’re not just talking about a smaller model, we’re talking about a different philosophy of what these models are for. And that’s a perfect segue into what they actually found when they started poking at what these models learn.

Jane: And that’s where it gets really interesting, because it’s one thing to predict a choice, it’s another thing to use the information the way a human would. Let’s get into that.

Summary: Tom: So, Jane, we’ve established that smaller models can match the big guy on familiar tasks. But the paper’s summary goes deeper. It’s not just about size; it’s about *what* the models are actually paying attention to.

Jane: Right. There’s been a lot of skepticism about these Centaur-style models. Critics said they might just be exploiting statistical shortcuts in the data, like patterns in the choice history, without really understanding the task. So the authors ran a clever set of experiments to test that.

Tom: And this is my favorite part. They systematically stripped away different parts of the prompt. They removed the task instructions, they masked the content of the stimuli, they even removed the feedback. And they found that if you take away the actual content—the shapes, the points, the feedback—the model’s performance collapses.

Jane: It drops below chance, actually. That’s a huge finding. It means the model isn’t just memorizing sequences of button presses. It’s using the information about what the participant saw and what happened as a result. The choice history alone is not enough.

Tom: Exactly. They call it the “content-masked” condition, and it destroys about seventy-five percent of the learned information. That’s the smoking gun that these models are processing task structure, not just statistical regularities.

Jane: And they even tested something called exchangeability. They took a task where the trials are independent, like picking the odd-one-out from a set of pictures, and they shuffled the order of past trials. A model that’s just learning sequences would be confused, but these models didn’t care. They knew the order didn’t matter.

Tom: But then they did the same shuffle on a task where the order *does* matter, like an adaptive test where the next question depends on your last answer. And the models were very sensitive to that shuffle. So they’re not just ignoring order; they’re using it only when the task demands it.

Jane: That’s task-adaptive information use. It’s a really strong piece of evidence that these models are learning something about the *logic* of the experiments, not just surface patterns. It makes them much more credible as scientific tools.

Tom: So we have these small, efficient models that seem to understand the tasks they’re trained on. The obvious next question is, what can we actually *do* with them? What’s the practical improvement here?

Improvements: Tom: So we’ve got these small, verified models. Jane, what’s the big improvement they offer over the current state of the art?

Jane: I think the biggest one is accessibility. The original Centaur model requires a serious amount of computing power to run. These small models, some of them can run on a single GPU, or maybe even on a laptop. That means a single lab, or even a single researcher, can now use a cognitive proxy without needing a supercomputer.

Tom: That’s a huge democratization of the tool. But they also frame it as something more specific: a “noise ceiling estimator.” Can you break that down for our listeners?

Jane: Sure. Think of it as a measuring stick for how predictable human behavior is in a given experiment. If you have a theory about how people make decisions, you can compare its predictions to what this model predicts. The model represents the best possible prediction you can make from the data.

Tom: So if your theory is worse than the model, it means your theory is missing something. There’s a gap.

Jane: Exactly. And that gap is a map. It tells you *where* your theory is failing, which experiments are more structured than your theory accounts for. That’s the “scientific regret minimization” idea from earlier work. This paper provides a cheap, accessible way to generate that map.

Meng: If I can jump in here, from a practical standpoint, the fact that they’ve released all the code and the model weights is huge. We can actually start testing these things in our own pipelines. The training times they report, even for the larger small models, are like a day or two on a single A100. That’s nothing compared to training a 70B model.

Tom: So it’s not just a theoretical improvement, it’s an engineering one. It lowers the barrier to entry so much that it changes what kind of projects are feasible.

Jane: And that’s the point. It moves these models from being a curiosity that only a few big labs can play with, to a standard tool in the cognitive scientist’s kit. But, of course, they’re careful to point out the limits. These models are only as good as the paradigms they’ve seen.

Tom: Right, they’re not a theory of cognition. They don’t explain *how* a choice is made, they just predict it very well. That’s a crucial distinction, and it’s a great note to end on for this segment. Let’s wrap up our thoughts.

Conclusion: Tom: Well, we’ve had a fascinating conversation about “Small Foundation Models of Human Cognition and Behaviour.” Let’s try to tie it all together.

Jane: I think the core message is that we don’t need to build bigger and bigger models to simulate human behavior. We need to build the right-sized model for the job, and then we need to verify that it’s using the information the way a human would. This paper does both.

Tom: It shows that a zero point six-billion-parameter model can match a seventy-billion-parameter model on familiar tasks, and that these models are genuinely processing task structure, not just memorizing sequences. That’s a powerful combination.

Jane: And it gives us a practical tool. A way to estimate the noise ceiling of an experiment and find where our theories fall short. It’s not a replacement for theories of cognition, but it’s a fantastic guide for where to build them.

Tom: It’s a bit like having a really accurate map of the territory before you start exploring. You know which mountains are actually there and which ones are just mirages.

Jane: That’s a nice way to put it. So we’re saying goodbye to this paper, but we’re taking that map with us. It’s a genuinely useful contribution to cognitive science, and we’re excited to see what people do with it.

Tom: Absolutely. A huge thank you to Nick Oh and Fernand Gobet for their work, and to our listeners for joining us. We’ll be back soon to break down the next big idea. Until then, keep thinking.