Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology

Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology

Summary

This paper introduces Social Chain of Thought (SCoT), a multi-round pipeline for medical differential diagnosis that structures multi-agent interaction as a deliberative framework for collaborative LLM reasoning. The authors motivate the work by noting that medical diagnostic reasoning is a high-impact use case for LLMs, with OpenAI (2026) reporting that more than 5% of ChatGPT messages globally are healthcare-related. They argue that transparency is a serious design concern, especially for complex cases where differential diagnosis often requires integrating multiple forms of specialist reasoning.

The core idea is that SCoT may be viewed as a form of scaling, where multiple agents act as additional reasoning passes for each other, much like the think-again mechanic of traditional Chain of Thought methods. Beyond simply stacking the deck, multi-agent systems come with a built-in solution to the endogeneity problem of single-model self-evaluation. As Huang et al. (2024) explain, LLM self-evaluation propagates correlation bias inherent to the source of generation. Endogeneity is thus not a problem that can be solved simply by scaling a model or giving a longer time to think; it must be approached with intentional heterogeneity. SCoT is architecturally designed to mitigate this.

SCoT is heterogeneous inference that produces measurable error decorrelation through persona conditioning, offering multiple, specialized perspectives for diagnostic reasoning. The authors acknowledge that with the same model backend, endogeneity remains an issue, but find the value of incorporating a social dimension into the reasoning process is a more transparent inference. SCoT structures reasoning variance through persona conditioning to simulate cross-model approaches proposed in multi-agent literature.

The method extends the Chain of Thought (CoT) dual-inference deployed by Zhou et al. (2025) for medical differential diagnoses, using an LLM prompted with a complex symptomatology as a backend for generating a panel of five medical specialists relevant to the case at hand. Using the physician-curated benchmark Open-XDDx, the authors deploy SCoT across a range of model families where these dynamically generated, persona-conditioned agents deliberate over the course of a seven-round pipeline to arrive at a final differential diagnosis.

The SCoT pipeline consists of seven rounds:

• Round 1: Specialized Ranking – Each agent assesses the case from the perspective of their specialty and provides a self-assessment of their relevance.

• Round 2: Symptom Management – Models the triage process, reflecting a realistic healthcare environment where a patient would be treated for obvious or life-threatening conditions prior to seeing a specialist.

• Round 3: Team Independent Differentials – Each agent delivers their independent assessment as a list of most likely diagnoses from the perspective of their specialty.

• Round 4: Master List – Compiles the independent differentials into an unranked and deduplicated list.

• Round 5: Refinement – Three sub-rounds of debate where agents state Support, Challenge, or Neutral positions; challenge other agents' diagnoses; and defend or update their positions.

• Round 6: Voting – Each agent ranks each diagnosis; scores are aggregated and weighted by diagnostic insight, breadth, and actionability before Borda counting synthesizes a final differential.

• Round 7: Can't Miss – A final safety check for critical or life-threatening diagnoses.

The authors test several backend models: Qwen-2.5 (1.5B, 3B, 32B), Gemma-4-MoE-26B, Gemma-4-31B, GPT-5-4 nano, Claude Haiku 4.5, and Claude Sonnet 4.6. Temperature settings are 0.3 for conservative agents and 0.7 for innovative agents, with each team composed of two innovative and three conservative specialists.

Key Results:

1. RQ1-H1 (Minimum viable configuration): Across models, SCoT improves recall by four to twelve percentage points on every backend with the exception of Qwen-2.5-1.5B. The 1.5B model degrades by 3.7 percentage points relative to its single-agent baseline. The viability threshold falls between 1.5B and 3B parameters: Qwen-2.5-1.5B degrades under SCoT, while Qwen-2.5-3B gains 11.8 recall points over its single-agent baseline.

2. RQ1-H2 (Deliberative headroom): The hypothesis that performance gain is greater for smaller models is supported. Models with lower-performing single-agent baselines have more recoverable diagnostic capacity than models already closer to the SCoT performance ceiling. For example, GPT-5-4 Nano gains +12.0 recall points, Haiku 4.5 gains +12.3, while Gemma-4 dense gains only +4.6 and Sonnet 4.6 gains +4.4.

3. RQ2-H3 (Multi-agent structure versus multi-round prompting): A single agent in the SCoT pipeline degrades recall by 5.44 percentage points relative to its baseline, but improves precision by 10.74 percentage points. This suggests the pipeline functions as a precision-positive filter for a single agent, but as a recall-positive broadening mechanism for multiple agents. Best-of-n tests show that repeated sampling does not reproduce SCoT's recall advantage. With 35 samples, repeated inference achieved a recall of 0.537, precision of 0.403, and F1 of 0.454, while SCoT N=5 achieved recall of 0.607, precision of 0.509, and F1 of 0.531.

4. RQ2-H4 (Difficulty-conditioned consensus formation): SCoT has its most significant impact in cases where the baseline performs worst, boosting aggregate F1 score by 15.1 percentage points (0.212 – 0.363), for a relative lift of 71.4%. In the easiest quartile, SCoT shows an 8.3 percentage point decrease in precision. The share of true-positive diagnoses entering through late refinement grows with case difficulty: 7 percent in the easiest quartile, 19 percent in the hardest, and 36 percent on the subset where the baseline returns zero. Across baseline-failure cases (where single-agent baselines recovered zero ground-truth diagnoses), SCoT recovered an average of 16.3% of ground-truth terms and recovered at least one ground-truth diagnosis in 53.9% of cases, rescuing 48 of 89 complete failures.

The authors conclude that SCoT demonstrates that performance in medical diagnostic reasoning can be increased through a social architecture. They argue that SCoT performance gain comes from social scaling that broadens the space of active inference to come to a conclusion through deliberation. This is a complementary approach to iterative scaling that bears consequence on diagnostic systems with public-facing application. The paper notes limitations including the exploratory nature of frontier model comparisons, the specific format of the 570-case dataset, and compute requirements for running viable LLMs.

Improvements to AI Systems:

1. Implement multi-agent deliberation with persona-conditioned heterogeneity – Instead of relying on a single model’s self-evaluation, deploy 5 dynamically generated specialist agents (e.g., cardiologist, neurologist) with distinct temperature settings (0.3 conservative, 0.7 innovative) to reduce correlation bias and endogeneity in reasoning.

2. Add a structured 7-round pipeline for complex decision-making – Integrate rounds for specialized ranking, symptom triage, independent differentials, master list compilation, debate with Support/Challenge/Neutral positions, weighted Borda voting, and a final “Can’t Miss” safety check for critical outcomes.

3. Enable difficulty-conditioned escalation – Automatically trigger multi-agent deliberation only for hard cases (e.g., when baseline confidence is low or zero), since SCoT provides a 71.4% relative F1 improvement on the hardest quartile but degrades precision by 8.3 points on easy cases.

4. Use multi-agent voting with weighted scoring – Replace simple majority voting with Borda counting weighted by diagnostic insight, breadth, and actionability to synthesize final outputs, improving recall by 4–12 points across model families.

5. Add a late-stage refinement mechanism – Incorporate iterative debate rounds where agents can challenge, defend, or update positions, allowing 19–36% of true-positive diagnoses to enter only during refinement for hard cases, rescuing 53.9% of complete baseline failures.

6. Implement a “Can’t Miss” safety layer – Add a final round dedicated to checking for life-threatening or critical diagnoses, ensuring that even if the primary differential is wrong, high-stakes conditions are flagged.

What the Improved AI System Can Do:

• Achieve higher diagnostic recall (up to +12.3 points) on complex medical cases without requiring a larger base model, by using social scaling across multiple specialist personas.

• Recover ground-truth diagnoses in 53.9% of cases where a single model returns zero correct answers, effectively rescuing complete failures.

• Maintain high precision on easy cases by conditionally activating multi-agent deliberation only when difficulty warrants it, avoiding unnecessary precision loss.

• Provide transparent, auditable reasoning through structured debate rounds, where each agent’s support/challenge/neutral positions and final votes are traceable, improving trust in public-facing diagnostic tools.

• Scale reasoning capability across model sizes – even a 3B-parameter model gains 11.8 recall points, making high-quality diagnostic reasoning accessible on smaller, cost-efficient backends.

Medical diagnostic reasoning is a high-impact use case for LLMs that carries significant implications for the health and wellbeing of users. When OpenAI (2026) reports that more than 5% of ChatGPT messages globally are healthcare-related, the transparency of these systems becomes a serious design concern. This is especially true for complex cases, where differential diagnosis often requires integrating multiple forms of specialist reasoning. Existing work has proposed multi-agent approaches to medical diagnosis, but it remains unclear when such systems are needed, why they help, and where they outperform monolithic inference. We introduce Social Chain of Thought (SCoT),a multi-round pipeline for medical differential diagnosis that structures multi-agent interaction as a deliberative framework for collabora. tive LLM reasoning. Evaluating SCoT against single-agent baselines, one-agent pipeline ablations, and best-of-n scaling, we show that its recall advantage is not reproduced by monolithic inference alone. SCoT is most successful in the hardest diagnostic cases, where multiple rounds of specialist conversation help recover ground-truth diagnoses and converge on a higher-recall differential.