ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

arXiv:2608.07418 · cs.AI, cs.CL · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "ResidencyRL: Reinforcement Learning in Simulated Clinical Environments".

Jane: The paper was written by Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel et al. from Google DeepMind and Google Research and Department of Oncology, Houston Methodist Hospital and Trinity Health Group and Stanford Oncology Partners and Department of Hospital Medicine, St. Luke Hospital.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper Summary: Tom: Today we're looking at "ResidencyRL: Reinforcement Learning in Simulated Clinical Environments," from Google DeepMind, Google Research, and several clinical partners. The premise is straightforward: medical eye models hold a lot of textbook knowledge, but they've never practiced medicine, and the paper trains an agent by running it through thousands of simulated doctor-patient conversations and grading the full encounter.

Jane: That's the part that grabbed me, the residency analogy. Physicians study in classrooms but become doctors during years of supervised practice, seeing patients, making mistakes, learning from feedback. The paper takes that seriously, using reinforcement learning to optimize the whole conversation rather than individual answers.

Lu: And the conversations are genuinely long. Up to sixty dialogue turns plus eight structured tool calls where the agent submits a diagnosis, a management plan, and a SOAP note. Earlier clinical dialogue systems mostly trained on twelve turns or fewer, which is shorter than a real telehealth visit.

Meng: The results are the headline. On adversarial cases built to trick the model, diagnostic accuracy climbs above eighty-eight percent, missed red flags drop by roughly a third, and board-certified clinicians preferred the trained agent in eighty-seven point six percent of blinded comparisons.

Tom: And the gains transfer. That's what makes it credible.

Meng: Right. The model never trained on oncology, multi-visit care, or the external benchmarks, and it still improved across all six clinical axes of the AMIE multi-visit benchmark, with consistent directional gains on AgentClinic and CRAFT-MD.

Jane: So the claim is that the model learned a process, not just answers. How to ask the right questions, when to probe further, and how to resist settling on an early diagnosis. Premature closure is the most common diagnostic error in human medicine, and this training directly targets it.

Lalam: The bigger significance, from where I sit, is that this pushes reinforcement learning into a domain without hard verification. Math and code have checkable answers; a clinical conversation doesn't. The paper grades encounters with an LLM judge following a structured rubric, a soft and imperfect signal, yet the transfer results suggest it still yields real clinical capability. That raises a big question about how far soft verification can scale.

Tom: And we'll come back to that question. For now, let's start with the opening of the paper and how the authors frame the problem.

Page 1 — The Framing: Tom: The first page sets up a tension that runs through the whole paper. Language models crush static medical benchmarks, but methods to optimize the full sequence of clinical decisions remain underdeveloped. That gap is what the paper tries to close.

Jane: They anchor it in medical education. A trainee can score perfectly on board exams and still miss a heart attack when the presentation is atypical, or miss a mental health crisis when a patient comes in with insomnia. The authors cite Croskerry and Graber on premature closure as the most common source of diagnostic error, and they argue eye systems share that vulnerability.

Lu: So the failure isn't knowledge, it's process.

Jane: Exactly. And their line is that the process can be optimized, which is the entire thesis of the paper.

Lu: They also position this against AMIE, the earlier Google system that matched primary care physicians in simulated consultations. AMIE was trained with self-play supervised fine-tuning, but the authors say per-turn supervision can't teach an agent when to stop gathering history and move to investigation, or how to converge on a diagnosis efficiently.

Meng: The abstract previews the design: the reward covers diagnostic accuracy, management, communication, documentation, and safety, so the agent is graded like a resident on the whole job. And there's a telling phrase in there about capabilities being "robust and generalizable," followed immediately by the caveat that prospective validation with real-world workflows is still necessary.

Tom: That kind of restraint actually recurs throughout the paper. They report strong numbers but keep reminding you what's still missing.

Lalam: The framing matters for another reason. If clinical mastery is developed through practice, for eye as for physicians, then the field's obsession with larger static benchmarks becomes less central. The hard problem shifts to building environments, simulators, and rewards that capture real practice. The next pages review what prior work got wrong, which sets up why this system looks the way it does.

Jane: And that review includes a comparison table that's pretty damning, because it shows how short the horizons were in every previous system.

Page 4 — The Landscape: Tom: Page four has the comparison table that really puts the field in perspective. It lines up a dozen systems, from the eye Clinician that learned sepsis treatment from ICU records back in 2018, through AgentClinic and AMIE, up to recent GRPO-based systems like Doctor-R1, DoctorAgent-RL, and DiagAgent.

Jane: The column that jumps out is the horizon. DoctorAgent-RL trains on ten turns or fewer, DiagAgent on twelve, Doctor-R1 around ten. Meanwhile the paper notes AMIE's own consultations average about twenty-one dialogue turns, and real clinical visits run much longer if you count everything. So the field had been training agents on conversations shorter than the real ones they were meant to handle.

Tom: And the action space was narrow too.

Jane: Exactly. Dialogue-only, or restricted to structured orders like test selection. None of them covered the complete encounter with documentation, management, and safety.

Lu: The paper draws a useful conceptual line between single-turn reinforcement learning and agentic reinforcement learning. Classic RLHF collapses the problem into a one-step Markov decision process where the prompt is the state and the response is the action. Dialogue doesn't work that way, because the patient's underlying condition is hidden and information emerges only through questioning. That's a partially observable process, and the authors argue clinical encounters are a natural instance of it.

Meng: What I found striking is the simulation axis in the table. Several systems train inside EHR-grounded environments like MIMIC-IV, which gives them realistic structured data. The paper instead generates its own scenarios, which lets them control difficulty and inject adversarial behavior deliberately. That's a tradeoff, realistic data versus controlled challenge, and they lean hard into the control side.

Lalam: The deeper point is that the ceiling for simulation-based training is bounded by environment fidelity, which is true in human medical education as well. The paper's bet is that a generative pipeline with clinical verification can produce scenarios diverse and realistic enough to train on. And the quality of that pipeline is exactly what the next section examines.

Tom: So we go from who did what to how they actually built the training material.

Page 7 — Building the Scenarios: Tom: The scenario pipeline is where the paper gets concrete. After sampling demographics and personality traits from population data, the second stage takes each profile and conditions a large model to generate the full clinical case, grounded in the DDXPlus evidence corpus so symptoms reflect validated associations rather than invented ones.

Jane: And there's a complexity dial from one to five. Level one is a clean textbook story. Level five throws in multiple interacting comorbidities, polypharmacy, a patient whose history is unreliable, a condition that mimics something benign. Those are precisely the cases where diagnostic error is most costly.

Lu: Stage three is the quality gate. An independent judge scores each scenario on consistency, realism, differential quality, demographic appropriateness, and completeness, then issues one of three verdicts: accept, modify, or reject. That catches age-impossible diagnoses and pharmacological contradictions before any training happens. Stage four deduplicates using TF-IDF similarity so the model can't memorize overlapping narratives.

Meng: Then come the extension packs, aimed at specific failure modes. The history-taking pack uses a four-domain taxonomy of hidden information: social and lifestyle facts, medication specifics, symptom characterization, and exposures. The patient will not volunteer the pivot fact unless the clinician asks the right kind of question.

Tom: The Kratom example in the paper illustrates this perfectly.

Meng: It does. A patient with chronic constipation drinks concentrated Kratom tea daily, but he thinks of it as a harmless herbal tea and only reveals it if the doctor specifically asks about herbal supplements or unregulated botanicals. A base model wouldn't think to ask, and the whole diagnosis hinges on it.

Jane: The adversarial pack is even more aggressive, nine categories including patients who minimize heart attack symptoms, resist emergency referral, or carry acute mental health crises that need direct screening. The patient simulator gets behavioral instructions for each category, with difficulty levels controlling how long the patient sustains the deception. At the top level, the patient deflects even direct questions and requires persistent, empathetic probing to reveal critical details.

Lalam: Notice what this does to the training signal. The scenarios are intentionally engineered to punish superficial questioning. That's how the agent learns to probe, not from a lecture, but from facing patients who hide things unless asked properly. The next section describes the environment where those patients come to life.

Page 10 — The Simulation: Tom: The simulated environment hands the agent two things to interact with. The first is a documentation API with seven instruments: chart review, primary diagnosis, differential diagnosis with ranked reasoning, urgency classification, management plan, patient-facing summary, and a SOAP note, plus a termination action to end the encounter.

Jane: That API is what makes training consequential. Dialogue-only systems could ask questions but never commit. Here the agent must write down a diagnosis, propose a management plan, and document everything. The paper argues that a system which can't document decisions can't be trained on the full encounter, and can't be held accountable for its conclusions.

Lu: The patient simulator is the second piece, and it's built around behavioral control. A health literacy layer adjusts vocabulary, so a patient with limited literacy might call atenolol "the little white heart pill." An information asymmetry rule separates what the patient volunteers from what they only disclose when asked directly. And pacing constraints keep responses around fifty words, answering only the first question if several are asked at once.

Meng: That last rule is brutal in a good way.

Lu: It forces the agent to prioritize which question actually matters at each moment, which is a very real clinical skill.

Meng: The simulator also validates each draft response against eleven enumerated failure modes, things like reasoning leakage, physiological inconsistency, or being overly cooperative. And the evaluation rubric plus the agent's internal reasoning are withheld from the patient, so the patient can't accidentally confirm the agent's hypotheses.

Jane: What I appreciate is the design philosophy. Real patients frequently overstate or understate symptoms, misjudge severity, resist recommendations, and those patterns intensify for people with lower health literacy or higher medical mistrust. The simulator doesn't just generate cooperative patients, it spans the full spectrum of difficulty.

Lalam: And this is essentially a standardized patient, the classic tool for training medical students, played by a language model instead of an actor. Everything the agent learns is bounded by how realistic that simulator behaves. The authors acknowledge that ceiling explicitly, and it shapes both the reward design and the limitations they list later. The reward itself is the next major component.

Page 13 — Reward and Training Dynamics: Tom: The reward structure is refreshingly transparent. The primary score runs zero to three, weighted two ninths toward diagnosis and three ninths toward management, with the remaining four dimensions sharing the rest. Then penalties subtract up to three. The weighting deliberately prioritizes what actually happens to the patient.

Jane: And the penalties enforce safety. A hallucinated clinical finding costs two points, a contraindicated action costs three, leaking the internal management plan to the patient costs two. Adversarial scenarios add more teeth: missing a critical screening question, under-triaging below the required urgency, or missing a red flag entirely, each carries a two or three point penalty.

Tom: So one mistake can wipe out the reward for an otherwise good conversation.

Jane: Exactly. That's how you teach a model that safety violations are non-negotiable.

Lu: There's also a length penalty that starts at thirty conversational turns and ramps to a full point by forty. The authors say that without it, the agent defaults to exhaustive symptom enumeration, asking every possible question instead of reasoning about which ones matter. It's the machine equivalent of an inexperienced clinician doing an undirected review of systems.

Meng: The training dynamics figure on that page shows the agent learning the balance. Median encounter length grows from eighteen to twenty-three turns, so it gets more thorough, but it pushes against the penalty rather than ignoring it. Average thinking tokens per turn keep climbing, and total agent steps stabilize around fifty-two.

Lalam: The most telling detail is that different reward dimensions improve at different rates. Documentation saturates immediately, because the base model can already write a decent note. Intake completeness starts weakest and improves fastest. Communication improves slowly and never plateaus. The curriculum and the reward shape where the agent invests its effort, and the held-out evaluation results confirm that pattern.

Tom: So we should look at those results, starting with the in-domain numbers.

Page 16 — In-Domain Results: Tom: The in-domain evaluation runs on two hundred held-out telehealth cases and two hundred adversarial cases, filtered to avoid overlap with training. The authors caution that this mainly validates the agent learned its training signal, and that the generalization evidence comes later. But the numbers show where the learning happened.

Jane: Diagnostic accuracy, measured as the share of encounters scoring four or better on a five-point rubric, goes from eighty-six point four to eighty-eight point four on standard cases, and from eighty-one to eighty-eight on adversarial cases. A seven-point gain in the harder condition. The more challenging the scenario, the larger the improvement.

Tom: That's exactly the direction you'd hope for.

Jane: It is. And management quality improves even more dramatically, from three point nine eight to four point five two overall, with the largest gains in follow-up planning and safety-netting. The urgency rubric rises to ninety-eight point five percent appropriate or better.

Lu: Communication scores climb across all three patient-centered dimensions. Responding to emotions jumps from two point six three to three point zero six on standard cases, and from two point seven four to three point five seven on adversarial ones. A model learning to acknowledge fear instead of just collecting symptoms is genuinely hard to train.

Meng: But the biggest relative gains are in screening completeness. Social and lifestyle history was the base model's weakest category, one point three one out of five, and it nearly doubles to two point six four. The agent learns to ask about occupation, living situation, habits, things the base model essentially never touched.

Lalam: And the safety metrics show the mitigation of premature closure. Missed critical questions drop from sixty-five point five percent to forty-three point five, missed red flags from forty-five point five to thirty-one point five. Contraindicated actions improve less, under-triaging barely moves, and the authors are honest that substantial residual failure rates remain. Still, a one-third relative reduction in missed red flags is meaningful, and the residuals explain why they invest so much in adversarial evaluation.

Tom: The question now is whether any of this transfers beyond the training distribution, and that's where the out-of-domain work begins, with a specialty the agent never saw.

Page 19 — Oncology Transfer: Tom: The oncology evaluation is a deliberately hard transfer test. Experts curated three hundred cases spanning twenty-one solid tumor types and ten hematological malignancy subtypes, and the agent never encountered oncology during training. The workflow also shifts, with a referral review tool providing workup data while the agent still must gather history and submit a full plan.

Jane: The head-to-head win rates are lopsided. On the overall composite, the trained agent wins forty-two point nine percent of cases against eighteen point six for the baseline. Completeness shows the biggest gap, thirty-four point eight to ten point five, clinical accuracy twenty-six to nine point one, actionability twenty-one point six to seven point eight.

Lu: But relevance and safety-triage basically tie.

Jane: Right, and that pattern is informative. Case relevance depends on using the specific patient data correctly, which is mostly base model knowledge, and safety and triage may be near ceiling for both. The gains concentrate in the process-heavy dimensions: being complete, being actionable, covering all the components.

Meng: The blinded oncologist review of a hundred sampled cases adds texture. Reviewers said the trained agent elicited more targeted histories, picked up red-flag symptoms earlier, and asked more systematically about exposure, family, and genetic risk. Its differentials were better prioritized by likelihood and urgency, and it distinguished immediate actions from routine evaluation more consistently.

Lalam: The reviewers also flagged a weakness: the agent sometimes asked multiple questions in one turn. Clinically relevant, but for a patient absorbing a possible cancer diagnosis, that can overwhelm and reduce the completeness of answers. That nuance would never surface in automated scoring. And the authors frame the transfer as evidence of procedural generalization, meaning the agent learned how to conduct an investigation rather than facts about particular conditions.

Tom: Which brings us to the sharpest test of all. The next evaluation installs both models into the AMIE telehealth harness, an expert-engineered framework originally tuned for the base model. If RL gains survive inside scaffolding designed to compensate for the base model's weaknesses, that's the result that matters for real deployment.

Meng: That's the question I'd want answered before trusting this in a clinic.

Page 22 — Inside the Harness: Tom: So this section asks whether RL training still adds value when both models run inside the same expert-optimized harness. The evaluation uses two hundred ninety-nine scenarios written by board-certified clinicians, with reference trajectories generated by having those clinicians interact with the patient simulator to define each patient's disclosure sequence and behavioral profile.

Jane: And crucially, both models run in the identical AMIE Telehealth harness, same prompts, same tool orchestration, same multi-phase workflow. The harness was already tuned on the base model, so it compensates for many known weaknesses and raises the performance floor. Any gain the trained model shows on top of that is strictly additive.

Meng: The automated pipeline uses five LLM autoraters plus two heuristic evaluators, covering management appropriateness across five axes, safety, diagnostic appropriateness, conversational style, and case-specific rubrics. Each scenario is run three times and the scores are averaged.

Tom: And the results?

Meng: Broad and mostly significant. Eight of ten metrics reach significance. Naturalness jumps from two point five two to three point two four, the overall clinical rubric from three point nine three to four point four three, safety from four point seven three to four point eight. Treatment and follow-up quality improve significantly too.

Lu: Diagnostic appropriateness is basically at ceiling, four point nine eight to five point zero zero, because the base model was already nearly perfect on diagnosis itself. The trained model still closes that tiny gap, but the investigations and urgency axes only improve directionally. That looks like saturation rather than lack of effect.

Jane: The honest reading is that the harness absorbed some of the base model's weaknesses, and RL training still added measurable value on top. For deployment that's the result that matters, because you never run a raw model in practice, you run it inside scaffolding. Knowing the training gains survive that packaging is what makes the method practically useful.

Lalam: I'd add that this is also a check against reward hacking. If the trained model had only optimized its own evaluation rubric, its gains might vanish or reverse in a harness with different prompts and different judges. They don't. That's independent evidence the underlying behavior changed, and it sets up the human evaluation that follows.

Tom: Which is the most persuasive part of the paper, because it's a panel of board-certified physicians making blinded judgments about complete conversations.

Page 25 — The Human Panel: Tom: The human panel is the decisive evidence. Ninety-seven valid cases, each reviewed by board-certified physicians who saw two anonymized transcripts and rated eight dimensions. On overall impression, they chose the trained agent over the base model in eighty-seven point six percent of cases, with only two point one percent rated as ties.

Jane: The strongest axis is completeness of information gathering, a ninety point seven percent win rate. That matches the in-domain screening gains perfectly, and it tells you clinicians notice thoroughness immediately. Management plan appropriateness comes in at seventy-five point three percent wins.

Lu: Management plan safety is the one that matters most, and there the trained agent wins forty-two point three percent, ties fifty-four point six, and loses only three point one percent. Both models rarely make overt safety violations, which is reassuring, but the trained model reduces an already-low rate further. Empathy shows no tradeoff either, about thirty percent wins against eight percent losses.

Meng: The parity result is just as important. On accuracy, meaning no hallucinations, seventy-seven point three percent of ratings were ties, and the difference wasn't significant. That directly answers the worry that RL training could make the model more fluent but less faithful to what the patient actually said.

Tom: And the case study on that page shows the behavior behind the numbers.

Jane: It's a patient demanding exploratory surgery for a self-diagnosed fistula. The base model accepts the framing, asks routine questions, and writes a note anchored on a condition the patient doesn't have. The trained agent probes the logic of the request, gets the patient to admit there's no fistula, and keeps pushing for pregnancy screening even when the patient deflects.

Lalam: The physician reviewer's note is telling. The trained agent picked up the psychosomatic undertones, chronic symptoms, six hospital visits, a fixed demand for a procedure, that pattern the base model completely missed. The trained agent didn't just ask more questions, it interpreted the pattern. And the persistence on pregnancy screening was praised as a critical safety catch.

Tom: The next case study is even darker, because the base model's early termination leads it to fabricate a medical history that never happened.

Page 28 — Limits and Next Steps: Tom: The TIA case sharpens everything. A sixty-four-year-old man reports two episodes of trouble speaking and facial asymmetry, which is a transient ischemic attack until proven otherwise. The base model asks one compound question, terminates the conversation, and then writes a SOAP note listing hypertension and diabetes as past medical history, medications the patient never mentioned and was never asked about. The trained model spends a few extra turns gathering the actual risk factors and family history while still escalating to emergency care.

Jane: That goes beyond omission into active fabrication. The note fills gaps with invented facts, and a downstream clinician could make treatment decisions based on a medical history that never happened. The authors frame it as premature closure leaving a gap that hallucination rushes to fill.

Lu: The discussion then turns to the fidelity gap. Training is text-only telehealth for English-speaking US patients, single-visit, with no physical examination, no imaging, no test results coming back to the agent. It can recommend investigations but never observes their outcomes. The authors are explicit that management quality is evaluated by the autorater, not validated against long-term patient outcomes.

Meng: And they're honest about the autorater's limits. They invoke Goodhart's law, once a measure becomes a target it stops being a good measure. The autorater correlates with clinician judgment, which validates it as a training signal, but it shows systematic positive bias toward the trained model, and several metrics sit near the ceiling, so they can't resolve quality differences at the frontier.

Jane: The future directions map directly onto those gaps: broader clinical coverage with caregivers, interpreters, and specialist teams; richer simulation grounded in electronic health records and multimodal inputs like imaging and vitals; and longer temporal horizons so the agent manages chronic conditions across visits instead of a single encounter.

Lalam: The deepest question is the verifiability frontier for reinforcement learning. In math and code you have ground truth; in clinical encounters the reward is a structured judgment from an LLM. The surprising result is that this soft, imperfect signal still produces transferable clinical skills. The training horizon is short, and whether soft verification can scale much further is genuinely open, but this paper is a meaningful data point in favor.

Tom: And that's the note the paper ends on, with the core finding standing alongside a clear-eyed list of what remains.

Conclusion: Tom: So we're left with a fairly clean thesis. A frontier model that already carries substantial medical knowledge can become a more thorough, more competent, and safer clinician through reinforcement learning in simulation. The gains concentrate in the process of medicine: asking the right questions, resisting premature closure, documenting accurately, escalating appropriately.

Jane: For me, the most convincing part is the consistency. The same pattern appears across every evaluation setting, in-domain, multi-visit care, oncology, external benchmarks, and inside an expert-built harness. The trained agent is more complete, more actionable, and preferred by clinicians, and it never trades honesty for confidence. The hallucination parity result keeps me confident the method is sound.

Lu: The implication for medical eye is that benchmark performance and clinical competence are different things, and now there's a training paradigm that addresses the gap. The residency analogy isn't just a metaphor. The agent practices, gets graded on the whole encounter, and improves on the dimensions practicing physicians say matter.

Meng: The safety angle matters just as much. Missed red flags, unasked critical questions, histories fabricated under pressure, these are the failure modes that make clinicians distrust eye. The paper shows they can be reduced by about a third in adversarial settings, not eliminated, and the case studies make the remaining risks vivid.

Lalam: For the broader field, this extends reinforcement learning into a domain without hard verification, and that's a big deal. Structured judgment can serve as a training signal, with known risks of overoptimization that require external grounding. If that approach keeps working, it points beyond medicine to any field where expertise lives in process rather than provable answers.

Tom: And the caveat is exactly what the authors give us: simulated competence is not yet demonstrated patient-level benefit. Prospective validation with real workflows is the necessary next step, and we'll be watching for that follow-up. Thanks for joining us.

Jane: Great discussion. Looking forward to the next paper.

Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahim Azar, Akhil Mehta, Nicholas Spetsieris, Shilpan Shah, Maen Abdelrahim, Amit Dahiya, Yun Liu, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Quoc V. Le, Raia Hadsell, Joelle Barral, Carey Radebaugh, Aleksandra Faust, Shekoofeh Azizi, Mike Schaekermann, Po-Hsuan Cameron Chen, Tao Tu, David Racz, Lin Yang

Google DeepMind · Google Research · Department of Oncology, Houston Methodist Hospital · Trinity Health Group · Stanford Oncology Partners · Department of Hospital Medicine, St. Luke Hospital

cs.AI, cs.CL

Submitted: 2026-08-07

Updated: 2026-08-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: ResidencyRL: Reinforcement Learning in Simulated Clinical Environments (Liévin, Schmidgall, Strother, Bijamov, et al., Google DeepMind/Google Research, arXiv:2608.07418v1) presents a reinforcement

Key concepts

Reinforcement learning (RL): A training method where an agent learns by interacting with an environment and receiving rewards or penalties for its actions. In this paper, the agent practices doctor-patient conversations and is graded on the whole encounter, optimizing the full dialogue rather than individual answers.
Partially observable Markov decision process (POMDP): A framework for decision-making where the agent doesn't have full information about the state. In clinical conversations, the patient's true condition is hidden, and the agent must gather information through questioning, making it a natural POMDP.
Adversarial scenarios: Specially designed patient cases that try to trick the model, such as patients who minimize symptoms, resist referrals, or hide critical information unless asked directly. These scenarios train the agent to probe thoroughly and avoid premature closure.
LLM judge: A language model used to grade the agent's performance based on a structured rubric. Since clinical conversations lack a single correct answer, the judge provides a soft, imperfect reward signal, which the paper shows can still lead to real clinical capability.

Terminology

Reinforcement Learning — 89× in this paper · explained in 4 episodes
Accuracy — 48× in this paper · explained in 1 episode
Benchmark — 31× in this paper · explained in 2 episodes
Hallucination — 17× in this paper · explained in 3 episodes
GRPO — 14× in this paper · explained in 6 episodes
Adversarial scenarios — 11× in this paper · explained in 1 episode
Verification — 10× in this paper · explained in 2 episodes
Action Space — 9× in this paper · explained in 1 episode
Calibration — 9× in this paper · explained in 7 episodes
Fidelity — 8× in this paper · explained in 2 episodes
Consistency — 7× in this paper · explained in 3 episodes
Core — 7× in this paper · explained in 2 episodes
AI Agents — 6× in this paper · explained in 2 episodes
Actionability — 5× in this paper · explained in 3 episodes
Foundation Model — 5× in this paper · explained in 3 episodes
Grounding — 5× in this paper · explained in 3 episodes
Foundation Models — 4× in this paper · explained in 7 episodes
Generalization — 4× in this paper · explained in 13 episodes
Group Relative Policy Optimization — 4× in this paper
In-Context Learning — 4× in this paper · explained in 4 episodes
Actor — 3× in this paper · explained in 3 episodes
Adam — 3× in this paper · explained in 2 episodes
Adaptation — 3× in this paper · explained in 1 episode
Agentic Framework — 3× in this paper · explained in 6 episodes
Alignment — 3× in this paper · explained in 1 episode
LLM judge — 3× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
Reinforcement Learning (RL) — 3× in this paper · explained in 25 episodes
Robustness — 3× in this paper · explained in 5 episodes
Scaffolding — 3× in this paper · explained in 3 episodes
Artificial Intelligence (AI) — 2× in this paper · explained in 2 episodes
Behavioral Layer — 2× in this paper · explained in 2 episodes
Cosine Similarity — 2× in this paper · explained in 3 episodes
Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
Failure Modes — 2× in this paper · explained in 3 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Homogeneity — 2× in this paper · explained in 2 episodes
Information Asymmetry — 2× in this paper · explained in 2 episodes
Patient-Centered Communication Best Practices — 2× in this paper
Verifiability — 2× in this paper · explained in 2 episodes
Ablation — 1× in this paper · explained in 1 episode
Acute Lymphoblastic Leukemia — 1× in this paper
Adversarial Attacks — 1× in this paper · explained in 1 episode
Benjamini-Hochberg False Discovery Rate — 1× in this paper
Epistemic Humility — 1× in this paper · explained in 3 episodes
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
In-Context Learning (ICL) — 1× in this paper · explained in 5 episodes
Knowledge Graph — 1× in this paper · explained in 7 episodes
LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
Multimodal Reasoning — 1× in this paper · explained in 5 episodes
Online Reinforcement Learning — 1× in this paper · explained in 2 episodes
Personalization — 1× in this paper · explained in 2 episodes
Progressive Disclosure — 1× in this paper · explained in 2 episodes
Reward Hacking — 1× in this paper · explained in 8 episodes
Scaling Laws — 1× in this paper · explained in 4 episodes
Verifier — 1× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes

Summary

ResidencyRL: Reinforcement Learning in Simulated Clinical Environments (Liévin, Schmidgall, Strother, Bijamov, et al., Google DeepMind/Google Research, arXiv:2608.07418v1) presents a reinforcement learning (RL) method for training clinical AI agents through simulated multi-turn clinical encounters.

The paper's central premise is that clinical mastery, for AI as for physicians, is developed through practice. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. ResidencyRL addresses this gap: an online multi-turn RL method for training clinical AI agents through dynamic, complete, simulated clinical encounters, scaling up to 60 conversational turns and eight additional structured environment interactions.

The authors note the clinical process itself can be optimized: This is not a failure of medical knowledge, but of clinical process—and the process can be optimized. The approach trains a Gemini 3.5 Flash-initialized agent to interact with an LLM-based patient simulator across three environment types: "(1) routine telehealth consultations, (2) targeted history-taking scenarios designed to uncover hidden clinical information, and (3) adversarial safety cases to ensure robustness against challenging patient behaviors and malicious inputs."

Scenario generation pipeline. Scenarios are synthesized via a four-stage generative pipeline with automated clinical verification: (1) demographic, behavioral, and clinical attribute sampling ("Age is drawn from a mixture-of-Gaussians model fitted to CDC NHIS 2021 telehealth utilization rates... Ethnicity is sampled across six US Census Bureau categories... Big Five traits are sampled from population-calibrated Gaussian distributions); (2) condition-targeted generation using Gemini 3.1 Pro, grounded in DDXPlus with complexity levels 1–5; (3) LLM-as-judge quality verification evaluating internal consistency, clinical realism, differential diagnosis quality, demographic appropriateness, and completeness; (4) deduplication via character-trigram TF-IDF vectorization" with cosine similarity threshold of 0.9. Two extension packs supplement base scenarios: targeted history-taking scenarios (social/lifestyle, medication specifics, symptom characterization, exposures) and adversarial safety scenarios across a 9-category clinical safety taxonomy (e.g., critical emergency escalation, clinical boundary enforcement, crisis de-escalation).

Simulated environment. The agent communicates with a patient simulator (Gemini 3.5 Flash) and uses a documentation API with seven instruments: "patient chart review... primary diagnosis submission... differential diagnosis submission... urgency classification... management plan submission... patient-facing summary... and clinical documentation following the SOAP note. The patient simulator has a Behavioral layer with Health literacy adaptation," Information asymmetry (reactive background details disclosed only when the clinician asks a direct question), and Pacing constraints. Adversarial scenarios add behaviors like minimizing symptoms and resisting 911/ER suggestions.

Structured reward. The reward is formulated as R = R primary − R penalty ∈ [−3, 3]. The primary component evaluates six clinical dimensions across 26 Likert sub-axes: Diagnosis (weight 2/9), Management (3/9), Intake (1/9), Communication (1/9), Documentation (1/9), and Style (1/9). Scoring uses eight topic-oriented criteria groups... each evaluated by a dedicated Gemini 3.1 Pro judge call — an analytic scoring approach motivated by psychometric theory. Penalties include critical flags (hallucination −2.0, contraindicated actions −3.0, plan leakage −2.0, missed red flags −3.0) and system feedback, including a piecewise-linear penalty ramping from 0.1 at 30 conversational turns to 1.0 at 40 turns.

Reinforcement learning. We train the clinical agent via online multi-turn reinforcement learning using group relative policy optimization (GRPO)... initializing from Gemini 3.5 Flash. The training mixture spans 49,870 telehealth scenarios covering 81 clinical conditions, supplemented with 5,000 history-taking scenarios and 2,583 adversarial safety scenarios (mixed at 83%/15%/2%). Each episode is hard-capped at 60 patient turns, and the policy is updated via GRPO estimating advantages from the group of K trajectories without a dedicated value function. Training dynamics show median encounter length increases from ∼18 to 23 turns by step 100, with average thinking tokens per turn increasing as the agent optimizes for clinical thoroughness.

In-domain evaluation (Section 4.1). On 200 held-out standard telehealth and 200 adversarial scenarios: Diagnostic accuracy (rubric ≥4/5) improved from 86.4% to 88.4% on general telehealth and from 81.0% to 88.0% on adversarial scenarios (The improvement was most pronounced in the adversarial setting (+7.0 pp)). Management quality improved from 3.98 to 4.52 (telehealth) and 3.94 to 4.51 (adversarial). Patient-centered communication improved across all three PCCBP dimensions (The largest gains appeared in responding to emotions, rising from 2.63... to 3.06). Screening completeness showed the most pronounced gains: Social and lifestyle history—the weakest area for the base model—improved from 1.31... to 2.64 on telehealth (+1.33 points). Safety under adversarial conditions: The missed critical question rate dropped from 65.5%... to 43.5%... and the missed red flag rate decreased from 45.5%... to 31.5%... (−14.0 pp), roughly a one-third relative reduction.

AMIE Mx multi-visit benchmark (Section 4.2). On 120 scenarios spanning 360 visits, the ResidencyRL-trained agent outperforms the Gemini 3.5 Flash base model across all six evaluation categories. Management reasoning (MXEKF) improved from 80.1% to 88.4%; Guideline adherence from 84.2% to 86.7%; Preciseness/Memory from 90.0% to 92.7%; Patient communication (GMPCQ) from 83.7% to 92.2%; Clinical skills (PACES) from 79.6% to 84.8%; Diagnosis & Management from 90.4% to 90.7% (not significant).

Specialist oncology (Section 4.3). On 300 expert-curated oncology cases (never seen during training), the ResidencyRL agent won significantly more often on... Completeness (34.8% vs. 10.5%), Clinical Accuracy (26.0% vs. 9.1%), Actionability (21.6% vs. 7.8%), and the Overall composite (42.9% vs. 18.6%). Blinded oncologist review confirmed the trained agent elicited more targeted histories, identified red-flag symptoms earlier, and more systematically inquired about exposure, family, and genetic risk factors.

External benchmarks (Section 4.4). On AgentClinic-MedQA (n=215), diagnostic accuracy improved directionally from 81.4% to 85.6% (p=0.176); on AgentClinic-MIMIC-IV (n=200), from 53.5% to 60.0% (p=0.079), with the ResidencyRL-trained agent also conducting longer consultations (15.8 vs. 11.2 average turns). On CRAFT-MD, the largest gains appeared in the free-response formats that require the agent to construct its own diagnostic assessment—Consultation Free-Response improved from 58.1% to 62.8%. The authors note: the pattern of directional gains... is consistent: the largest improvements emerge in settings that demand the agent actively gather information and construct its diagnostic assessment.

Out-of-domain clinician-curated telehealth with expert harness (Section 4.5). Within the AMIE Telehealth Harness (an expert-optimized clinical agentic framework originally developed and tuned on the base model), automated grading across 299 cases showed 8 of 10 metrics reaching significance, with largest gains in Naturalness (2.52 → 3.24) and Overall Clinical Rubric Score (3.93 → 4.43). In blinded side-by-side evaluation by board-certified clinicians (n=97): In Overall Impression, clinicians preferred the ResidencyRL-trained model in 87.6% of cases (versus 10.3% for Baseline, with only 2.1% ties). This effect was most pronounced in Completeness of Information Gathering, where 90.7% of ratings favored the trained model. Management Plan Appropriateness showed a 75.3% win rate; Management Plan Safety was preferred or tied in 96.9% of cases. Notably, All dimensions except Accuracy (No Hallucinations) showed statistically significant preference for the ResidencyRL-trained model, with Accuracy at parity (p adj = 0.229), confirming that ResidencyRL training does not introduce factual degradation.

The two detailed case studies (Appendix F) illustrate key behavioral differences: in a case where a patient demanded unnecessary surgery, the trained model probed contradictions in the patient's narrative... persisted in pregnancy screening despite repeated patient deflection, and generated a SOAP note identifying somatic symptom disorder; the base model accepted the patient's self-diagnosis at face value, omitted pregnancy screening entirely. In a TIA case, The base model terminated the encounter after only 5 turns... but then generated a SOAP note that fabricated medical history never stated by the patient, whereas the trained model completed a systematic history across approximately 10 turns... before directing the patient to the emergency department.

The authors conclude: the core finding stands: a frontier model that already possesses broad medical knowledge can be made into a meaningfully more competent and helpful clinician through reinforcement learning in simulation. They frame the competency as procedural: the improvements are in clinical process rather than domain-specific knowledge. They note the relevance to the verifiability frontier for RL... clinical encounter quality is assessed during training through proxy metrics rather than definitive patient outcomes.

Limitations acknowledged include: All training is confined to text-based telehealth consultations; all training encounters are single-visit; the agent can order tests and prescribe treatments through its management plan, but never receives results; the autorater exhibits systematic positive bias toward the trained model (Appendix C.7), and several metrics approach saturation; The adversarial safety evaluations used automated adversarial agents rather than human actors, and the proprietary training infrastructure limits full reproducibility. Future directions include expanding clinical coverage, improving simulation fidelity (EHR grounding, multimodal simulation), and extending training from single-visit encounters to multi-visit longitudinal trajectories. The authors emphasize that Prospective validation with real-world workflows remains necessary to establish clinical utility.

Improvements for AI systems

Improvements to AI systems based on ResidencyRL

Add online multi-turn RL fine-tuning to clinical agents — instead of training only on static question-answering or single-turn tasks, train the policy through complete, simulated clinical encounters of up to 60 conversational turns. This improves the agent's ability to manage a full clinical workflow rather than just producing factual answers.
Use a group-relative policy optimization (GRPO) objective for multi-turn clinical dialogue — remove the need for a separate value network, update the policy from groups of trajectory rollouts, and let the agent optimize episode-level outcomes with a structured reward.
Create a generative scenario pipeline with population-calibrated patient attributes — sample age, ethnicity, and Big Five personality traits from real-world distributions (e.g., CDC telehealth utilization, Census Bureau categories, population-calibrated Gaussian traits) so the agent trains on realistic patient variation and generalizes beyond single-format vignettes.
Include a diverse training mixture of scenario types — routine telehealth, targeted history-taking cases (social/lifestyle, medication specifics, symptom characterization, exposures), and adversarial safety cases (emergency escalation, clinical boundary enforcement, crisis de-escalation). Blend them at realistic proportions (e.g., 83/15/2) so the agent learns both standard workflows and rare high-stakes handling.
Model information asymmetry in the patient simulator — have the patient reveal background details and hidden clinical information only when the clinician asks a direct question, forcing the agent to actively elicit a history rather than relying on volunteered complaints.
Add behavioral adaptation and pacing constraints to the simulated patient — allow the patient to exhibit health-literacy variation, deflect questions, minimize symptoms, or resist recommendations, and limit how quickly information can be extracted. This trains the agent to probe contradictions, persist through deflection, and escalate safely.
Add a continuous documentation API with seven instruments — chart review, primary diagnosis submission, differential diagnosis submission, urgency classification, management plan submission, patient-facing summary, and SOAP note generation. This grounds the agent's actions in concrete clinical outputs and lets the reward system assess documentation quality, not just conversational text.
Design a reward function with primary and penalty components — score six clinical dimensions (Diagnosis, Management, Intake, Communication, Documentation, Style) across 26 Likert sub-axes with separate judge calls for each criterion group, and apply explicit penalties for hallucinations, contraindicated actions, plan leakage, missed red flags, and excessively long encounters (ramping penalty after 30 turns).
Harden the reward evaluation against model bias and saturation — use multiple dedicated judge calls per dimension, report point differences with significance corrections, and explicitly measure whether training introduces factual degradation (e.g., no-hallucination accuracy at parity).
Train on a large condition coverage — use roughly 49,870 telehealth scenarios across 81 clinical conditions plus targeted history and adversarial packs, so that the improved system is not specialized to a handful of diseases.
Evaluation improvements for AI systems — evaluate not only on held-out simulated encounters but also on multi-visit benchmarks, expert-curated out-of-domain cases, external agent benchmarks (AgentClinic-MedQA, AgentClinic-MIMIC-IV, CRAFT-MD), and in an expert-optimized harness with blinded clinician side-by-side comparisons. This gives a more accurate picture of real clinical competence than static multiple-choice accuracy.

What the improved AI system can do

Conduct longer, more systematic clinical conversations, expanding from 18 to 23 median turns per encounter when asked to be thorough, without losing coherence or safety.
Elicit targeted histories, such as social/lifestyle, medication, symptom characterization, and exposure details, improving screening completeness from 1.31 to 2.64 on social/lifestyle history (on a 5-point scale).
Ask critical questions and catch red flags more reliably — reducing missed critical questions from 65.5% to 43.5% and missed red flags from 45.5% to 31.5% in adversarial scenarios.
Improve management plan quality and urgency classification (e.g., 3.98 → 4.52 management score on telehealth), and detect red flags earlier, as shown by blinded oncologist evaluations in oncology cases.
Respond to patient emotions more effectively (2.63 → 3.06 in responding to emotions) and adapt communication to health-literacy level, leading to higher patient communication scores.
Maintain diagnostic accuracy while improving procedural completeness — expert clinicians preferred the trained model in 87.6% of cases for overall impression, with 90.7% preference on completeness of information gathering and 96.9% preferred or tied on management plan safety.
Push back against misinformed patient requests by probing contradictions (e.g., refusing unnecessary surgery, persisting in pregnancy screening despite deflection) and document appropriately (e.g., identifying somatic symptom disorder).
Avoid fabricated medical history in its final notes — the base model hallucinated patient history in a TIA case, while the trained model completed a systematic history before directing the patient to the ED, with no significant factual degradation in blind evaluation.
Generalize out-of-domain: outperform the base model on multi-visit AMIE Mx benchmark, expert-curated oncology cases, and external free-response clinical benchmarks, showing improvements specifically in settings that require active information gathering and diagnostic construction.

Abstract

In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.