Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Let Large Language Models Judge Each Other".

Jane: The gist The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper today, "Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering." It sounds like a pretty big deal in how we test these models on medical stuff. Jane Yeah, it’s about taking those large language models and making them do more than just spit out an answer; they start acting like both the solver and the judge.

Lu: It's really about moving beyond just one model giving an answer, you have multiple agents working together to check each other's work on medical questions <ref:2606.15419#pg2>.

Meng: So instead of just picking the majority vote on the final answers, this method focuses on evaluating the reasoning process itself to pick the best one >

Tom: Exactly. It’s about making sure that when we get a medical answer from these models, we can trust *why* they got it, because in medicine, factual or logical errors can have serious consequences >

Jane: Right. This paper is showing how this multi-agent approach helps boost the accuracy of models on tasks like MedQA.

The paper's summary: Tom: So what’s the actual setup here? Essentially, they designed a system where several LLM agents independently create their own chain of thought reasoning and propose candidate answers for a single medical question >

Jane: And then those agents don't just stop there; they switch roles and start reviewing each other’s reasoning for things like factual correctness or logical soundness >

Lu: It’s inspired by the human peer-review process in academic papers, where you don't just look at the final paper but you check if the arguments are solid <ref:2606.15419#pg3>.

Meng: The key difference they point out is that instead of relying on majority voting over the answers, this method evaluates and selects the reasoning process itself >

Tom: Which means they’re not just picking the most popular answer; they are selecting the most credible chain of thought available among all their peers >

Jane: They use a specific six-point rubric to score each other’s reasoning quality, meaning judges don't look at the ground truth labels, but on their own understanding of medical knowledge >

Lu: And they aggregate these scores, selecting the top peer-reviewed reasoning chain to generate the final response >

The paper's improvements: Tom: They show that this method consistently beats both using a single model for chain-of-thought and even traditional majority voting on final answers across three different datasets like HeadQA, MedQA(USMLE), and PubMedQA >

Jane: And the results are pretty strong. The best combination they tested achieved an average accuracy of zero point eight two zero across those datasets, which is better than the strongest single model at zero point seven seven seven >

Meng: What’s really interesting is how they measure reasoning quality separately; high-performing models like GPT-oss-20B and Phi-four scored much higher on that peer review, while weaker models like DeepSeekLLM-7B were rated lower >

Lu: They found a strong positive correlation of zero point nine one between the mean peer review score of each model and its own single model accuracy, which shows that models that get high reasoning scores tend to be more accurate on their own > <ref:2606.15419#pg1>

Tom: So it suggests that the focus on reasoning quality really matters, not just getting the final answer right in isolation >

Jane: And they also noted that this method scales well; performance actually gets better as you introduce more participating models into the system >

Conclusion: Tom: So to wrap up, this paper proposes a multi-agent peer-reviewed reasoning method for medical question answering, and it consistently improved accuracy over both chain-of-thought reasoning and majority voting across multiple benchmarks >

Jane: It also enhanced interpretability by selecting the reasoning chains with higher logical quality, giving us those traceable paths through the decision process >

Lu: The main idea is that LLMs can serve as both solvers and reviewers, which helps produce answers that are more accurate and more trustworthy for downstream use >

Meng: From an engineering side, it’s interesting because it shows how leveraging different models’ strengths creates a stronger ensemble method without falling into simple majority bias >

Lalam: I think this whole process of having the AI judge the AI helps us build systems that are more robust and less likely to just get stuck on surface-level patterns >

Tom: It’s definitely something worth paying attention to if you want to improve how we use these models in clinical or research settings >

Jane: And for anyone listening, this paper shows that focusing on the quality of the reasoning path is a much better way to get reliable results than just relying on a simple majority vote >