DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models".

Jane: The paper was written by Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie et al. from Shanghai Jiao Tong University and SII and SPIRAL Lab and Generative AI Research Lab (GAIR) and Shanghai Chest Hospital and Beijing Anzhen Hospital, Capital Medical University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: Welcome back to the show, everyone. Today we’re diving into a paper that’s been making waves in the medical AI world: "DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models." Jane, this one’s from a team at Shanghai Jiao Tong University, and honestly, the title alone tells you they’re serious about testing these models where it actually matters.

Jane: Absolutely, Tom. And I love the name "DiagnosisArena" because it really captures what they’ve built — a proving ground, a place where AI models have to step into the ring and face real clinical cases. It’s not a multiple-choice test from a textbook. It’s like putting a medical student in front of a patient and saying, "Okay, what’s going on here?"

Tom: Right, and that’s the big shift. We’ve seen models ace medical exams, but this paper argues those exams have become too easy. They’re saturated. So the team went and pulled over a thousand real case reports from top journals like NEJM and The Lancet, and they structured them into a format that mimics actual clinical encounters.

Jane: And that’s the key, Tom. They didn’t just grab random cases. They built a whole pipeline to make sure these cases are genuinely hard — they filtered out anything a model could solve with simple recall. They even had doctors and AI experts double-check each case to make sure there was enough information to reach a diagnosis, but not so much that it becomes obvious.

Lu: If I can jump in here — I’m Lu, by the way. What excites me most is that they’re not just testing knowledge. They’re testing reasoning. The cases require you to connect symptoms, physical exam findings, and test results into a coherent diagnosis. That’s the kind of cognitive work we hope AI can eventually support in clinics.

Meng: And as an engineer, I’m curious about the scale. They ended up with one thousand one hundred thirteen cases across twenty-eight specialties. That’s a solid benchmark. But what really caught my eye is the performance gap — even the best model, o3, only got about fifty-one percent accuracy. That’s barely better than a coin flip.

Tom: Yeah, that number is sobering. And it’s not like they used weak models. They tested o1, DeepSeek-R1, GPT-4o, Claude, all the big names. And the results show that when you take away the multiple-choice crutch, these models struggle. Jane, what do you think that says about where we are with AI in medicine?

Jane: I think it says we’re still in the early innings. The models are great at recalling facts, but they’re not yet great at the messy, integrative thinking that diagnosis requires. And that’s exactly why a benchmark like this is so valuable — it gives us a clear, honest picture of what’s missing.

Lu: And that’s the exciting part. Once you can measure the gap, you can start closing it. This paper gives the research community a target to aim for.

Meng: Yeah, but I also wonder — is the gap because the models lack medical knowledge, or because they lack a way to structure their thinking? That’s the question I’d love to dig into next.

Jane: Great question, Meng. And that’s exactly where we’re heading in the next segment — we’re going to look at how they actually built this benchmark and what makes it so much harder than what came before.

Summary of the Paper: Tom: Welcome back. We’re still on "DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models." Jane, let’s get into the meat of it — how did they actually build this thing?

Jane: So the process was really meticulous, Tom. They started with over four thousand case reports from ten top-tier journals. Then they used a mix of rule-based filtering and AI models to restructure each case into four clean sections: case information, physical exam, diagnostic tests, and the final diagnosis. That last part is the ground truth they use to score the models.

Lu: And here’s the clever part — they didn’t just take all those cases. They ran an iterative filtering process. They used models like GPT-4o and DeepSeek-V3 to try each case, and if any of them could solve it easily, they threw it out. That way, they kept only the cases that genuinely require deep reasoning.

Meng: That’s a smart way to avoid the "too easy" problem. But they also did something even more interesting — they had AI experts and board-certified physicians review each remaining case. If the diagnosis was ambiguous or the case didn’t have enough information, it got cut. So what’s left is a set of cases where a correct answer is actually possible, but only if you reason well.

Tom: And that’s why the benchmark is so tough. It’s not just about having the right knowledge — it’s about applying it in a messy, real-world context. The paper shows that even the best reasoning models, like o1 and DeepSeek-R1, score in the seventeen percent to thirty-one percent range. That’s a huge drop from the ninety percent plus we see on traditional medical benchmarks.

Jane: And that drop is the whole point, right? It shows that the old benchmarks were measuring something different — mostly recall. DiagnosisArena measures something harder: the ability to synthesize information and make a judgment call under uncertainty.

Lu: Exactly, Jane. And I think that’s why this paper is so important. It’s not just a new dataset. It’s a new way of thinking about evaluation. We’re moving from "can the model remember the answer" to "can the model reason its way to the answer."

Meng: But I want to push back a little. They also created a multiple-choice version of the same cases, and the models did much better on that — o1 jumped to almost sixty-two percent. Doesn’t that suggest the models actually know the answers, but just can’t express them in an open-ended format?

Jane: That’s a really good point, Meng. And the paper addresses that directly. They argue that multiple-choice questions give the model a huge hint — it just has to pick the best option, not generate a diagnosis from scratch. That’s like giving a student a list of possible answers and saying "one of these is right." It’s a fundamentally easier task.

Tom: So the open-ended format is the real test. And the fact that models struggle with it tells us something important about their limitations. They can recognize the right answer when they see it, but they can’t always produce it on their own.

Lu: And that’s the gap we need to close. The next question is — can we actually improve these models, or is there a fundamental limit? That’s what we’re going to talk about next.

Improvements Suggested: Tom: Back on the air with "DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models." We’ve talked about the benchmark itself, but now let’s get into what the paper suggests we do about these results. Jane, what’s the path forward?

Jane: Well, Tom, the paper doesn’t propose a specific new model, but it does point to a clear direction. The biggest takeaway is that reasoning ability matters more than raw knowledge. The models that did best, like o3 and o1, are specifically trained to think step-by-step. That’s why they outperformed models like GPT-4o, which is powerful but not designed for deep reasoning.

Lu: And I think that’s the key insight. The paper shows that when you give a model time to "think" — to generate a long chain of reasoning before answering — it does much better. That’s the whole idea behind inference-time scaling. It’s not about making the model bigger; it’s about letting it think longer.

Meng: But that comes at a cost. More thinking means more compute, which means more money and more latency. In a real clinical setting, you can’t always wait five minutes for a diagnosis. So there’s a trade-off between accuracy and practicality.

Tom: That’s a real concern, Meng. But the paper also suggests another angle — that multiple-choice formats are a crutch. If we want models to be useful in real clinics, we need to train them to generate diagnoses from scratch, not just pick from a list.

Jane: And that’s where the benchmark can help. By giving researchers a hard, realistic test, DiagnosisArena can drive the development of better reasoning strategies. Maybe it’s better training data, maybe it’s new prompting techniques, maybe it’s models that can ask for more tests when they’re uncertain.

Lu: I’d love to see that last one. Imagine a model that says, "I’m not sure, but if we ran this specific test, it would help me narrow it down." That’s how real doctors work. That’s the kind of behavior we should be encouraging.

Meng: And that would require a different kind of benchmark too — one that rewards asking the right questions, not just giving the right answer. But that’s a bigger project.

Tom: Definitely. But for now, DiagnosisArena gives us a solid foundation. It’s a hard test, and it shows us where we stand. And the fact that even the best models are struggling means there’s a lot of room for improvement.

Jane: And that’s exciting, Tom. It means the field is still wide open. The next breakthrough could come from anywhere — a new training method, a new model architecture, or a new way of thinking about clinical reasoning.

Lu: I think the biggest impact will be on how we design AI systems for healthcare. Instead of just building bigger models, we’ll need to build systems that can reason, ask questions, and collaborate with doctors. That’s a much richer vision.

Meng: And it’s a vision that’s grounded in reality. This paper gives us the tools to measure progress toward that vision. That’s what I appreciate most.

Tom: Well said. And that brings us to our final segment, where we’ll wrap up and look at the bigger picture.

Conclusion: Tom: And we’re back for the final stretch. We’ve been talking about "DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models," and I have to say, this paper has given us a lot to think about.

Jane: It really has, Tom. The core message is simple but powerful: current AI models are not ready for real-world clinical diagnosis. They can ace exams, but they struggle when faced with the messy, complex cases that doctors deal with every day. And that’s a crucial reality check.

Lu: And it’s a constructive one, too. By building a benchmark that’s actually hard, the authors have given the research community a clear target. We now know exactly what we need to improve — not just knowledge, but reasoning.

Meng: I’d add that the paper also highlights the importance of evaluation design. The fact that models do so much better on multiple-choice questions shows that the format of the test can dramatically change the results. That’s a lesson that goes beyond medicine.

Tom: That’s a great point, Meng. And it makes me think about the broader implications. If we’re going to use AI in high-stakes fields like healthcare, we need to be honest about its limitations. Benchmarks like this help us do that.

Jane: And they also help us track progress. A year from now, we can run the same models on DiagnosisArena and see if they’ve gotten better. That’s how we know we’re moving in the right direction.

Lu: I’m optimistic. The gap between fifty-one percent and one hundred percent is large, but it’s not insurmountable. With better reasoning models, better training data, and better evaluation methods, we can close it. And when we do, AI could become a truly valuable partner for doctors.

Meng: And that’s the dream, right? Not replacing doctors, but supporting them — helping them see things they might have missed, and giving them more time to focus on their patients.

Tom: Well said, Meng. And on that note, we’re going to wrap up our discussion of "DiagnosisArena." It’s a paper that’s both humbling and inspiring. Humbling because it shows how far we still have to go, and inspiring because it gives us a clear path forward.

Jane: Thanks for listening, everyone. We’ll be back next time with another paper from the arXiv. Until then, keep thinking, keep questioning, and keep pushing the boundaries of what’s possible.

Tom: Take care, and see you on the next episode.