On Benchmarking Human-Like Intelligence in Machines

arXiv:2502.20502 · cs.AI · Submitted 2026-08-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "On Benchmarking Human-like Intelligence in Machines".

Jane: The paper was written by Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu et al. from Harvard University and University of Cambridge and Massachusetts Institute of Technology and New York University and Johns Hopkins University and Princeton University and Stanford University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everybody. Today we’re digging into a paper that’s got a title that really makes you stop and think: “On Benchmarking Human-like Intelligence in Machines.” Jane, what’s your first reaction to that title?

Jane: Honestly, Tom, it feels like a bit of a challenge. Like, we keep hearing that AI is getting more human-like every week, but this paper is basically saying, “Hold on, are we even measuring that the right way?” And I love that they’re asking that question.

Tom: Right? And the authors are a who’s who of cognitive science and AI. You’ve got Joshua Tenenbaum and Thomas Griffiths on there, along with a bunch of folks from MIT, Harvard, Princeton. This is serious brainpower.

Jane: Yeah, and that’s what makes it exciting. They’re not just armchair philosophers. They went out and actually tested some of the most popular AI benchmarks with real human participants to see if the labels even make sense.

Lu: That’s the part that got me hooked. They took nine benchmarks—things like moral reasoning, social IQ, theory of mind—and they had over a hundred people rate the same questions. And what they found is that the “correct” answers in these benchmarks often don’t match what real people think.

Tom: Lu, you’re jumping ahead, but I love it. So the title is really a promise. It’s saying, “If you want to claim your AI is human-like, you need to actually compare it to humans, not just to a dataset label.”

Jane: And that’s the big implication, right? We’re building these systems to work with us, to understand us. If the benchmarks we use to test them are flawed, we’re basically flying blind.

Meng: From an engineering standpoint, that’s terrifying. I mean, we tune models to score well on these benchmarks. If the benchmarks are measuring the wrong thing, we’re just optimizing for the wrong target.

Tom: Exactly, Meng. And that’s what we’re going to unpack today. We’re going to look at what they found when they put these benchmarks under the microscope, and then we’ll get into their recommendations for how to fix it.

Jane: And trust me, some of those findings are wild. We’re talking about tasks where human agreement with the benchmark label was below chance. That’s not a small problem.

Tom: So stick around. We’re about to find out why your favorite AI might be acing tests that humans would fail.

Summary: Jane: So, Tom, we’ve set the stage. Now let’s get into the meat of “On Benchmarking Human-like Intelligence in Machines.” What did they actually do?

Tom: They ran a human evaluation study on nine AI benchmarks. They took tasks like irony detection, dark humor, moral permissibility, social support, and they had one hundred seventeen people rate them using sliders instead of multiple choice.

Lu: And that’s a key detail. Instead of forcing people to pick one answer, they let them express graded agreement. So if you think something is *mostly* supportive but not totally, you can put the slider at seventy instead of being forced to say yes or no.

Jane: And they also asked people how confident they were in each answer. So you get two separate measures: what you believe, and how sure you are about that belief.

Tom: Right. And the results are pretty damning. On average, human agreement with the benchmark labels was only about seventy percent. But here’s the kicker: on the Social Support task, humans agreed with the label less than half the time. That’s worse than flipping a coin.

Meng: Wait, so the benchmark label is basically wrong more often than not? How does that even happen?

Jane: That’s the thing, Meng. These labels were probably created by one or two annotators, or maybe by the benchmark designers themselves. They assumed there was one right answer, but when you ask a hundred people, you get a whole distribution of opinions.

Lu: And it’s not just noise. They show examples where people split into two camps, both very confident, but with opposite answers. Like, one person is sure the statement is supportive, another is sure it’s not, and they’re both certain they’re right.

Tom: That’s what they call structured disagreement. It’s not that people are confused; it’s that the task is genuinely ambiguous, and the benchmark just picked one answer and called it the truth.

Jane: And that’s a huge problem if you’re trying to build a model that behaves like a human. If you train it to match that single label, you’re training it to ignore the natural variation that exists in human judgment.

Meng: So what does that mean for me? If I’m deploying a model that was trained on these benchmarks, am I just shipping something that’s fundamentally misaligned with what people actually think?

Tom: That’s exactly the concern. And it’s not just about accuracy. It’s about whether the model’s *patterns* of judgment—its uncertainty, its errors—look anything like a human’s.

Jane: And that’s the core of their argument. They’re saying we need to stop measuring against a single “correct” answer and start measuring against the full distribution of human responses.

Lu: Which brings us to their recommendations. And that’s where the real actionable stuff is.

Tom: Exactly. So we’ve seen the problem. Now let’s talk about how they think we should fix it.

Improvements: Jane: Alright, so we’ve seen that the current benchmarks are shaky. What does “On Benchmarking Human-like Intelligence in Machines” actually propose we do differently?

Tom: They give us five concrete recommendations, and the first one is almost obvious: actually collect human data. Don’t just make up labels. If you want to claim your AI is human-like, you need real human responses as the gold standard.

Lu: And not just a handful. They’re saying you need robust sample sizes. Because if you ask five people, you might get a very different distribution than if you ask a hundred.

Meng: So that’s recommendation one. What’s next?

Jane: Recommendation two is about not collapsing all those human responses into a single label. Instead of taking the majority vote, you should keep the full distribution. So if sixty percent of people say “yes” and forty percent say “no,” that’s the target, not just “yes.”

Tom: And that’s a big shift. It means evaluating a model on how well it matches that distribution, not just whether it picks the most popular answer.

Lu: Right. And that connects to recommendation three: measure gradedness and uncertainty. Not just what people think, but how strongly they think it, and how confident they are.

Jane: They actually show that people often give intermediate slider values with high confidence. So it’s not that they’re unsure; it’s that the true answer is genuinely somewhere in the middle.

Meng: Okay, so we’re collecting richer data. But what about the tasks themselves? Are they even the right tasks?

Tom: That’s recommendation four. They say we need to ground our benchmarks in cognitive theory. Don’t just grab a classic psychology test like the Sally-Anne test and assume it measures everything about theory of mind.

Lu: Exactly. That test is designed for kids at a specific developmental stage. It’s not a comprehensive measure of adult social reasoning. So using it to claim an AI has theory of mind is a stretch.

Jane: And recommendation five is about making the tasks richer and more naturalistic. Real-world problems are messy. They involve ambiguity, incomplete information, and multiple cognitive processes at once.

Tom: So instead of simple multiple-choice questions, they want tasks that require integrating reasoning, perception, social understanding, maybe even planning. And they want to include stimuli that are deliberately ambiguous, because that’s where human cognition really shows its character.

Meng: So, if I’m building a benchmark, I need to collect lots of human data, keep the full distribution, measure graded confidence, ground it in theory, and make the tasks complex. That’s a lot of work.

Jane: It is, but it’s the only way to actually know if we’re building machines that think like us, rather than machines that are just really good at guessing what a benchmark designer thought was the right answer.

Tom: And that’s the hook for our final segment. Because if we do all this, what does it actually change in the real world?

Conclusion: Jane: So, Tom, we’ve covered the problems and the fixes. Let’s wrap up what “On Benchmarking Human-like Intelligence in Machines” really means for the future.

Tom: For me, the big takeaway is that we need to stop treating AI benchmarks like standardized tests with one right answer. They should be more like opinion polls, capturing the range of what people actually think.

Lu: And that has huge implications. If we build models that match human distributions, they’ll be better at collaborating with us. They’ll understand when we’re uncertain, when we’re joking, when we’re being sarcastic.

Meng: From a practical side, it means we need to invest in collecting high-quality human data. That’s expensive and time-consuming, but the paper argues it’s necessary if we want trustworthy AI.

Jane: And it’s not just about making AI more useful. It’s about understanding human intelligence better. By trying to match human behavior, we learn more about what that behavior actually is.

Tom: That’s the beautiful part. This paper is as much a call to action for cognitive science as it is for AI. It’s saying, “Let’s use these tools to understand ourselves.”

Lu: And the authors acknowledge it’s not easy. They talk about the challenges of scaling human data collection, and they admit that not every AI needs to be human-like. If you’re predicting protein structures, you don’t want human error.

Meng: Right, but for the tasks that matter for human interaction—moral judgment, social reasoning, humor—this is the way forward.

Jane: So we’re saying goodbye to this paper, but we’re taking its message with us. Next time someone claims their AI is human-like, we’re going to ask, “Compared to what?”

Tom: And with that, we’ll close out this discussion. Thanks for joining us on the channel. We’ll be back soon with another paper to pick apart.

Jane: Until then, keep questioning the benchmarks.

Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum

Harvard University · University of Cambridge · Massachusetts Institute of Technology · New York University · Johns Hopkins University · Princeton University · Stanford University

cs.AI

Submitted: 2026-08-12

Comments: 22 pages

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 65/100

The gist: This paper argues that current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities, and proposes recommendations for future benchmarks based on cognitive

Key concepts

Human Evaluation Study
The researchers tested nine popular AI benchmarks by having over 100 people rate the same questions. Instead of picking one answer, participants used sliders to express graded agreement and also indicated how confident they were in their response.
Structured Disagreement
This occurs when a task is genuinely ambiguous, and different people hold opposite views with high confidence. The benchmark label simply picks one answer as 'truth,' ignoring this natural variation in human judgment.
Gradedness and Uncertainty
This involves measuring not just what people think, but how strongly they believe it. People often give intermediate slider values with high confidence, indicating the true answer lies somewhere in the middle.

Terminology

Summary

This paper argues that current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities, and proposes recommendations for future benchmarks based on cognitive science best practices.

The authors define human-likeness at the behavioral level as "the capacity of an AI system to produce judgments and behaviors that are qualitatively similar to those of humans across a range of cognitive tasks, capturing not only which response the system gives, but also its patterns of errors, uncertainties, and the distributional variability observed across human populations. They distinguish this from human-level intelligence, which refers to achieving quantitative performance parity with humans."

The paper identifies three key shortcomings in current evaluation: "the too-frequent absence of human validation in dataset labeling, inadequate representation of human variability in collected human data, and over-reliance on simplified tasks that lack ecological validity and fail to reflect the complexity of real-world scenarios."

To support these claims, the authors conducted a human evaluation study on 9 AI benchmarks (7 from BIG-Bench, plus BigToM and ToMBench), recruiting 117 participants from Prolific and collecting 2,685 annotated trials. They used slider-based graded judgments (0–100) instead of multiple choice, plus separate confidence ratings. Results showed human accuracy, measured as the percentage of trials where a participant’s response agreed with the benchmark’s ground-truth label, varies substantially across tasks, with an average of 69.7% across all nine benchmarks. Notably, Social support (45%, chance = 50%) falls below chance, and several tasks showed substantial disagreement.

The paper proposes five concrete recommendations:

  1. Measure 'human-like AI' against actual humans, and collect robust, replicable sample sizes of human data — the ground-truth labels for measuring whether AI is human-like should be response data collected from humans themselves. The authors note that many benchmarks claim to measure human-like performance without any human data, and that even tasks with objective labels can reveal structured disagreement (e.g., The Dress illusion).

  2. Evaluate models of human populations against population-level distributions of human judgments — model evaluations should explicitly collect, analyze, and use population-level distributions of human responses as the 'gold' soft labels for evaluating model performance. They criticize majority voting for effectively discarding valuable information about the range and distribution of human judgments.

  3. Evaluate model gradedness and uncertainty against gradedness in individual human judgments — graded beliefs and uncertainties are an essential part of human cognition. The authors found that a large fraction of human ratings fall in intermediate ranges, demonstrating that participants hold graded beliefs that are poorly captured by discrete labels, and that participants often report high confidence in their intermediate judgments, indicating that the gradedness reflects genuine degrees of belief or feature intensity rather than epistemic confusion.

  4. Situate tasks with respect to meta-reviews of existing cognitive theory — benchmarks should begin with "an explicit specification of the construct that defines what it is (and what it is not), decomposes it into subcomponents, and identifies the observable behavioral signatures that should arise under controlled manipulations. They critique the use of the Sally-Anne test for Theory of Mind, noting that ToM embodies a wide range of subcomponents beyond those assessed by the Sally-Anne test" and that such tests risk data contamination.

  5. Design Cognitively Rich Tasks that Elicit Naturalistic Behavior — benchmarks should incorporate integration of cognitive capacities, systematic ablation, and structured ambiguity, moving beyond simple, unambiguous question-answer formats.

The paper acknowledges limitations, including scalability of human data collection and the open question of whether AI should replicate human cognitive biases. The authors state: "AI systems are growing increasingly powerful; we need more robust and reliable evaluation not only if we want to build more human-compatible AI thought partners that we understand but also if we want to deeply understand human intelligence."

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems, and what the improved system can do:

1. Replace single ground-truth labels with human-derived distributional targets

  • Improvement: For any benchmark claiming to measure human-like cognition (social reasoning, moral judgment, humor, theory of mind, etc.), I will no longer train or evaluate against a single “correct” answer. Instead, I will collect human response distributions (e.g., from 100+ diverse annotators per stimulus) and use those as soft labels.

  • What the improved system can do: It will learn to produce a distribution of plausible responses rather than a single confident answer. For example, on a moral permissibility question, it will output a graded probability across options (e.g., 60% permissible, 40% not) that matches the human population’s spread, rather than forcing a binary choice.

2. Model and report inter-annotator disagreement explicitly

  • Improvement: I will add a disagreement-aware evaluation metric (e.g., Wasserstein distance or KL divergence between model output distribution and human response distribution) alongside accuracy. I will also flag stimuli where human annotators disagree strongly (e.g., bimodal distributions) and treat those as separate evaluation cases.

  • What the improved system can do: It will be able to identify when a task is genuinely ambiguous and respond with calibrated uncertainty. For instance, on a dark humor detection task where humans split 50/50, the system will not pretend to be certain but will output a near-50% probability, matching human behavior.

3. Capture graded judgments, not just binary choices

  • Improvement: I will redesign output layers and training losses to support continuous or multi-point scales (0–100) for each answer option, rather than forcing a single categorical choice. I will use the paper’s two-measure approach: a graded belief slider plus a separate confidence slider.

  • What the improved system can do: It will produce nuanced responses like “I rate this statement as 72% supportive, and I am 85% confident in that rating.” This is critical for tasks like social support detection, where humans rarely give absolute 0/100 judgments.

4. Evaluate against population-level distributions, not just the majority vote

  • Improvement: I will stop collapsing human annotations to a single majority label. Instead, I will keep the full distribution and evaluate the model’s ability to match that distribution, including subpopulation patterns (e.g., how a utilitarian vs. deontological responder would answer).

  • What the improved system can do: It will be able to condition on a persona or demographic profile (e.g., “respond as a conservative-leaning individual”) and produce a response distribution consistent with that subgroup, rather than always defaulting to the majority view.

5. Incorporate human uncertainty into training objectives

  • Improvement: I will add an auxiliary loss that penalizes the model when its confidence is misaligned with human confidence. For example, if humans are uncertain (low confidence) on a stimulus, the model should also output high entropy, even if it could guess correctly.

  • What the improved system can do: It will avoid overconfident predictions on inherently ambiguous inputs. For instance, on a Theory of Mind false-belief task where humans are only 70% confident, the model will output a probability distribution with similar entropy, rather than a near-100% confident answer.

6. Design tasks with structured ambiguity and systematic ablations

  • Improvement: I will create evaluation suites that include stimuli deliberately designed to elicit structured disagreement (e.g., ambiguous social scenarios, moral dilemmas with no clear answer). I will also include ablated versions (e.g., removing context, adding misleading information) to test how the model’s uncertainty changes.

  • What the improved system can do: It will demonstrate human-like sensitivity to context. For example, if I remove a crucial piece of background information from a social reasoning task, the model’s output distribution should widen (more uncertainty), just as humans’ would.

7. Use meta-theory-driven task design, not just borrowed psychological tests

  • Improvement: I will stop using single-item tests like the Sally-Anne false-belief task as a proxy for full Theory of Mind. Instead, I will build benchmarks that decompose the construct into subcomponents (belief tracking, intention inference, desire reasoning, knowledge attribution) and test each with multiple controlled scenarios.

  • What the improved system can do: It will be able to report which specific subcomponent of Theory of Mind it handles well or poorly (e.g., “strong at belief tracking, weak at intention inference”), rather than giving a single pass/fail score.

8. Report calibration against human confidence distributions

  • Improvement: I will add a calibration metric that compares the model’s confidence distribution to the human confidence distribution across all stimuli, not just accuracy.

  • What the improved system can do: It will be able to tell a user, “On this set of moral questions, my confidence is well-calibrated to human confidence (within 5% on average), but on humor detection, I am systematically overconfident.” This enables targeted improvements.

9. Handle human error patterns as signal, not noise

  • Improvement: I will train models to reproduce human error patterns (e.g., systematic biases, content effects on reasoning) rather than only optimizing for correctness.

  • What the improved system can do: It will make the same kinds of mistakes humans make on reasoning tasks (e.g., being influenced by irrelevant content), which is essential for human-AI collaboration and for simulating realistic human agents in social science research.

10. Provide distributional outputs in real-world deployment

  • Improvement: For deployed systems (e.g., chatbots, decision-support tools), I will change the output interface to offer a range of plausible answers with associated probabilities, rather than a single definitive answer, especially for subjective tasks.

  • What the improved system can do: In a medical triage chatbot, for a symptom that humans are 60% likely to interpret as mild and 40% as serious, the system will present both possibilities with those probabilities, rather than forcing a single recommendation. This improves trust and safety.

These improvements directly address the paper’s three core criticisms: (1) lack of human-validated labels, (2) inadequate representation of human variability and uncertainty, and (3) reliance on ecologically invalid simplified tasks. The resulting AI systems will be more genuinely human-like in their judgments, more useful for human-AI collaboration, and more scientifically valid as models of human cognition.

Abstract

Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.

Sources

Related papers