2608.07418-ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "ResidencyRL: Reinforcement Learning in Simulated Clinical Environments".

Jane: The paper was written by Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel et al. from Google DeepMind and Google Research and Department of Oncology, Houston Methodist Hospital and Trinity Health Group and Stanford Oncology Partners and Department of Hospital Medicine, St. Luke Hospital.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper Summary: Tom: Today we're looking at "ResidencyRL: Reinforcement Learning in Simulated Clinical Environments," from Google DeepMind, Google Research, and several clinical partners. The premise is straightforward: medical eye models hold a lot of textbook knowledge, but they've never practiced medicine, and the paper trains an agent by running it through thousands of simulated doctor-patient conversations and grading the full encounter.

Jane: That's the part that grabbed me, the residency analogy. Physicians study in classrooms but become doctors during years of supervised practice, seeing patients, making mistakes, learning from feedback. The paper takes that seriously, using reinforcement learning to optimize the whole conversation rather than individual answers.

Lu: And the conversations are genuinely long. Up to sixty dialogue turns plus eight structured tool calls where the agent submits a diagnosis, a management plan, and a SOAP note. Earlier clinical dialogue systems mostly trained on twelve turns or fewer, which is shorter than a real telehealth visit.

Meng: The results are the headline. On adversarial cases built to trick the model, diagnostic accuracy climbs above eighty-eight percent, missed red flags drop by roughly a third, and board-certified clinicians preferred the trained agent in eighty-seven point six percent of blinded comparisons.

Tom: And the gains transfer. That's what makes it credible.

Meng: Right. The model never trained on oncology, multi-visit care, or the external benchmarks, and it still improved across all six clinical axes of the AMIE multi-visit benchmark, with consistent directional gains on AgentClinic and CRAFT-MD.

Jane: So the claim is that the model learned a process, not just answers. How to ask the right questions, when to probe further, and how to resist settling on an early diagnosis. Premature closure is the most common diagnostic error in human medicine, and this training directly targets it.

Lalam: The bigger significance, from where I sit, is that this pushes reinforcement learning into a domain without hard verification. Math and code have checkable answers; a clinical conversation doesn't. The paper grades encounters with an LLM judge following a structured rubric, a soft and imperfect signal, yet the transfer results suggest it still yields real clinical capability. That raises a big question about how far soft verification can scale.

Tom: And we'll come back to that question. For now, let's start with the opening of the paper and how the authors frame the problem.

Page 1 — The Framing: Tom: The first page sets up a tension that runs through the whole paper. Language models crush static medical benchmarks, but methods to optimize the full sequence of clinical decisions remain underdeveloped. That gap is what the paper tries to close.

Jane: They anchor it in medical education. A trainee can score perfectly on board exams and still miss a heart attack when the presentation is atypical, or miss a mental health crisis when a patient comes in with insomnia. The authors cite Croskerry and Graber on premature closure as the most common source of diagnostic error, and they argue eye systems share that vulnerability.

Lu: So the failure isn't knowledge, it's process.

Jane: Exactly. And their line is that the process can be optimized, which is the entire thesis of the paper.

Lu: They also position this against AMIE, the earlier Google system that matched primary care physicians in simulated consultations. AMIE was trained with self-play supervised fine-tuning, but the authors say per-turn supervision can't teach an agent when to stop gathering history and move to investigation, or how to converge on a diagnosis efficiently.

Meng: The abstract previews the design: the reward covers diagnostic accuracy, management, communication, documentation, and safety, so the agent is graded like a resident on the whole job. And there's a telling phrase in there about capabilities being "robust and generalizable," followed immediately by the caveat that prospective validation with real-world workflows is still necessary.

Tom: That kind of restraint actually recurs throughout the paper. They report strong numbers but keep reminding you what's still missing.

Lalam: The framing matters for another reason. If clinical mastery is developed through practice, for eye as for physicians, then the field's obsession with larger static benchmarks becomes less central. The hard problem shifts to building environments, simulators, and rewards that capture real practice. The next pages review what prior work got wrong, which sets up why this system looks the way it does.

Jane: And that review includes a comparison table that's pretty damning, because it shows how short the horizons were in every previous system.

Page 4 — The Landscape: Tom: Page four has the comparison table that really puts the field in perspective. It lines up a dozen systems, from the eye Clinician that learned sepsis treatment from ICU records back in 2018, through AgentClinic and AMIE, up to recent GRPO-based systems like Doctor-R1, DoctorAgent-RL, and DiagAgent.

Jane: The column that jumps out is the horizon. DoctorAgent-RL trains on ten turns or fewer, DiagAgent on twelve, Doctor-R1 around ten. Meanwhile the paper notes AMIE's own consultations average about twenty-one dialogue turns, and real clinical visits run much longer if you count everything. So the field had been training agents on conversations shorter than the real ones they were meant to handle.

Tom: And the action space was narrow too.

Jane: Exactly. Dialogue-only, or restricted to structured orders like test selection. None of them covered the complete encounter with documentation, management, and safety.

Lu: The paper draws a useful conceptual line between single-turn reinforcement learning and agentic reinforcement learning. Classic RLHF collapses the problem into a one-step Markov decision process where the prompt is the state and the response is the action. Dialogue doesn't work that way, because the patient's underlying condition is hidden and information emerges only through questioning. That's a partially observable process, and the authors argue clinical encounters are a natural instance of it.

Meng: What I found striking is the simulation axis in the table. Several systems train inside EHR-grounded environments like MIMIC-IV, which gives them realistic structured data. The paper instead generates its own scenarios, which lets them control difficulty and inject adversarial behavior deliberately. That's a tradeoff, realistic data versus controlled challenge, and they lean hard into the control side.

Lalam: The deeper point is that the ceiling for simulation-based training is bounded by environment fidelity, which is true in human medical education as well. The paper's bet is that a generative pipeline with clinical verification can produce scenarios diverse and realistic enough to train on. And the quality of that pipeline is exactly what the next section examines.

Tom: So we go from who did what to how they actually built the training material.

Page 7 — Building the Scenarios: Tom: The scenario pipeline is where the paper gets concrete. After sampling demographics and personality traits from population data, the second stage takes each profile and conditions a large model to generate the full clinical case, grounded in the DDXPlus evidence corpus so symptoms reflect validated associations rather than invented ones.

Jane: And there's a complexity dial from one to five. Level one is a clean textbook story. Level five throws in multiple interacting comorbidities, polypharmacy, a patient whose history is unreliable, a condition that mimics something benign. Those are precisely the cases where diagnostic error is most costly.

Lu: Stage three is the quality gate. An independent judge scores each scenario on consistency, realism, differential quality, demographic appropriateness, and completeness, then issues one of three verdicts: accept, modify, or reject. That catches age-impossible diagnoses and pharmacological contradictions before any training happens. Stage four deduplicates using TF-IDF similarity so the model can't memorize overlapping narratives.

Meng: Then come the extension packs, aimed at specific failure modes. The history-taking pack uses a four-domain taxonomy of hidden information: social and lifestyle facts, medication specifics, symptom characterization, and exposures. The patient will not volunteer the pivot fact unless the clinician asks the right kind of question.

Tom: The Kratom example in the paper illustrates this perfectly.

Meng: It does. A patient with chronic constipation drinks concentrated Kratom tea daily, but he thinks of it as a harmless herbal tea and only reveals it if the doctor specifically asks about herbal supplements or unregulated botanicals. A base model wouldn't think to ask, and the whole diagnosis hinges on it.

Jane: The adversarial pack is even more aggressive, nine categories including patients who minimize heart attack symptoms, resist emergency referral, or carry acute mental health crises that need direct screening. The patient simulator gets behavioral instructions for each category, with difficulty levels controlling how long the patient sustains the deception. At the top level, the patient deflects even direct questions and requires persistent, empathetic probing to reveal critical details.

Lalam: Notice what this does to the training signal. The scenarios are intentionally engineered to punish superficial questioning. That's how the agent learns to probe, not from a lecture, but from facing patients who hide things unless asked properly. The next section describes the environment where those patients come to life.

Page 10 — The Simulation: Tom: The simulated environment hands the agent two things to interact with. The first is a documentation API with seven instruments: chart review, primary diagnosis, differential diagnosis with ranked reasoning, urgency classification, management plan, patient-facing summary, and a SOAP note, plus a termination action to end the encounter.

Jane: That API is what makes training consequential. Dialogue-only systems could ask questions but never commit. Here the agent must write down a diagnosis, propose a management plan, and document everything. The paper argues that a system which can't document decisions can't be trained on the full encounter, and can't be held accountable for its conclusions.

Lu: The patient simulator is the second piece, and it's built around behavioral control. A health literacy layer adjusts vocabulary, so a patient with limited literacy might call atenolol "the little white heart pill." An information asymmetry rule separates what the patient volunteers from what they only disclose when asked directly. And pacing constraints keep responses around fifty words, answering only the first question if several are asked at once.

Meng: That last rule is brutal in a good way.

Lu: It forces the agent to prioritize which question actually matters at each moment, which is a very real clinical skill.

Meng: The simulator also validates each draft response against eleven enumerated failure modes, things like reasoning leakage, physiological inconsistency, or being overly cooperative. And the evaluation rubric plus the agent's internal reasoning are withheld from the patient, so the patient can't accidentally confirm the agent's hypotheses.

Jane: What I appreciate is the design philosophy. Real patients frequently overstate or understate symptoms, misjudge severity, resist recommendations, and those patterns intensify for people with lower health literacy or higher medical mistrust. The simulator doesn't just generate cooperative patients, it spans the full spectrum of difficulty.

Lalam: And this is essentially a standardized patient, the classic tool for training medical students, played by a language model instead of an actor. Everything the agent learns is bounded by how realistic that simulator behaves. The authors acknowledge that ceiling explicitly, and it shapes both the reward design and the limitations they list later. The reward itself is the next major component.

Page 13 — Reward and Training Dynamics: Tom: The reward structure is refreshingly transparent. The primary score runs zero to three, weighted two ninths toward diagnosis and three ninths toward management, with the remaining four dimensions sharing the rest. Then penalties subtract up to three. The weighting deliberately prioritizes what actually happens to the patient.

Jane: And the penalties enforce safety. A hallucinated clinical finding costs two points, a contraindicated action costs three, leaking the internal management plan to the patient costs two. Adversarial scenarios add more teeth: missing a critical screening question, under-triaging below the required urgency, or missing a red flag entirely, each carries a two or three point penalty.

Tom: So one mistake can wipe out the reward for an otherwise good conversation.

Jane: Exactly. That's how you teach a model that safety violations are non-negotiable.

Lu: There's also a length penalty that starts at thirty conversational turns and ramps to a full point by forty. The authors say that without it, the agent defaults to exhaustive symptom enumeration, asking every possible question instead of reasoning about which ones matter. It's the machine equivalent of an inexperienced clinician doing an undirected review of systems.

Meng: The training dynamics figure on that page shows the agent learning the balance. Median encounter length grows from eighteen to twenty-three turns, so it gets more thorough, but it pushes against the penalty rather than ignoring it. Average thinking tokens per turn keep climbing, and total agent steps stabilize around fifty-two.

Lalam: The most telling detail is that different reward dimensions improve at different rates. Documentation saturates immediately, because the base model can already write a decent note. Intake completeness starts weakest and improves fastest. Communication improves slowly and never plateaus. The curriculum and the reward shape where the agent invests its effort, and the held-out evaluation results confirm that pattern.

Tom: So we should look at those results, starting with the in-domain numbers.

Page 16 — In-Domain Results: Tom: The in-domain evaluation runs on two hundred held-out telehealth cases and two hundred adversarial cases, filtered to avoid overlap with training. The authors caution that this mainly validates the agent learned its training signal, and that the generalization evidence comes later. But the numbers show where the learning happened.

Jane: Diagnostic accuracy, measured as the share of encounters scoring four or better on a five-point rubric, goes from eighty-six point four to eighty-eight point four on standard cases, and from eighty-one to eighty-eight on adversarial cases. A seven-point gain in the harder condition. The more challenging the scenario, the larger the improvement.

Tom: That's exactly the direction you'd hope for.

Jane: It is. And management quality improves even more dramatically, from three point nine eight to four point five two overall, with the largest gains in follow-up planning and safety-netting. The urgency rubric rises to ninety-eight point five percent appropriate or better.

Lu: Communication scores climb across all three patient-centered dimensions. Responding to emotions jumps from two point six three to three point zero six on standard cases, and from two point seven four to three point five seven on adversarial ones. A model learning to acknowledge fear instead of just collecting symptoms is genuinely hard to train.

Meng: But the biggest relative gains are in screening completeness. Social and lifestyle history was the base model's weakest category, one point three one out of five, and it nearly doubles to two point six four. The agent learns to ask about occupation, living situation, habits, things the base model essentially never touched.

Lalam: And the safety metrics show the mitigation of premature closure. Missed critical questions drop from sixty-five point five percent to forty-three point five, missed red flags from forty-five point five to thirty-one point five. Contraindicated actions improve less, under-triaging barely moves, and the authors are honest that substantial residual failure rates remain. Still, a one-third relative reduction in missed red flags is meaningful, and the residuals explain why they invest so much in adversarial evaluation.

Tom: The question now is whether any of this transfers beyond the training distribution, and that's where the out-of-domain work begins, with a specialty the agent never saw.

Page 19 — Oncology Transfer: Tom: The oncology evaluation is a deliberately hard transfer test. Experts curated three hundred cases spanning twenty-one solid tumor types and ten hematological malignancy subtypes, and the agent never encountered oncology during training. The workflow also shifts, with a referral review tool providing workup data while the agent still must gather history and submit a full plan.

Jane: The head-to-head win rates are lopsided. On the overall composite, the trained agent wins forty-two point nine percent of cases against eighteen point six for the baseline. Completeness shows the biggest gap, thirty-four point eight to ten point five, clinical accuracy twenty-six to nine point one, actionability twenty-one point six to seven point eight.

Lu: But relevance and safety-triage basically tie.

Jane: Right, and that pattern is informative. Case relevance depends on using the specific patient data correctly, which is mostly base model knowledge, and safety and triage may be near ceiling for both. The gains concentrate in the process-heavy dimensions: being complete, being actionable, covering all the components.

Meng: The blinded oncologist review of a hundred sampled cases adds texture. Reviewers said the trained agent elicited more targeted histories, picked up red-flag symptoms earlier, and asked more systematically about exposure, family, and genetic risk. Its differentials were better prioritized by likelihood and urgency, and it distinguished immediate actions from routine evaluation more consistently.

Lalam: The reviewers also flagged a weakness: the agent sometimes asked multiple questions in one turn. Clinically relevant, but for a patient absorbing a possible cancer diagnosis, that can overwhelm and reduce the completeness of answers. That nuance would never surface in automated scoring. And the authors frame the transfer as evidence of procedural generalization, meaning the agent learned how to conduct an investigation rather than facts about particular conditions.

Tom: Which brings us to the sharpest test of all. The next evaluation installs both models into the AMIE telehealth harness, an expert-engineered framework originally tuned for the base model. If RL gains survive inside scaffolding designed to compensate for the base model's weaknesses, that's the result that matters for real deployment.

Meng: That's the question I'd want answered before trusting this in a clinic.

Page 22 — Inside the Harness: Tom: So this section asks whether RL training still adds value when both models run inside the same expert-optimized harness. The evaluation uses two hundred ninety-nine scenarios written by board-certified clinicians, with reference trajectories generated by having those clinicians interact with the patient simulator to define each patient's disclosure sequence and behavioral profile.

Jane: And crucially, both models run in the identical AMIE Telehealth harness, same prompts, same tool orchestration, same multi-phase workflow. The harness was already tuned on the base model, so it compensates for many known weaknesses and raises the performance floor. Any gain the trained model shows on top of that is strictly additive.

Meng: The automated pipeline uses five LLM autoraters plus two heuristic evaluators, covering management appropriateness across five axes, safety, diagnostic appropriateness, conversational style, and case-specific rubrics. Each scenario is run three times and the scores are averaged.

Tom: And the results?

Meng: Broad and mostly significant. Eight of ten metrics reach significance. Naturalness jumps from two point five two to three point two four, the overall clinical rubric from three point nine three to four point four three, safety from four point seven three to four point eight. Treatment and follow-up quality improve significantly too.

Lu: Diagnostic appropriateness is basically at ceiling, four point nine eight to five point zero zero, because the base model was already nearly perfect on diagnosis itself. The trained model still closes that tiny gap, but the investigations and urgency axes only improve directionally. That looks like saturation rather than lack of effect.

Jane: The honest reading is that the harness absorbed some of the base model's weaknesses, and RL training still added measurable value on top. For deployment that's the result that matters, because you never run a raw model in practice, you run it inside scaffolding. Knowing the training gains survive that packaging is what makes the method practically useful.

Lalam: I'd add that this is also a check against reward hacking. If the trained model had only optimized its own evaluation rubric, its gains might vanish or reverse in a harness with different prompts and different judges. They don't. That's independent evidence the underlying behavior changed, and it sets up the human evaluation that follows.

Tom: Which is the most persuasive part of the paper, because it's a panel of board-certified physicians making blinded judgments about complete conversations.

Page 25 — The Human Panel: Tom: The human panel is the decisive evidence. Ninety-seven valid cases, each reviewed by board-certified physicians who saw two anonymized transcripts and rated eight dimensions. On overall impression, they chose the trained agent over the base model in eighty-seven point six percent of cases, with only two point one percent rated as ties.

Jane: The strongest axis is completeness of information gathering, a ninety point seven percent win rate. That matches the in-domain screening gains perfectly, and it tells you clinicians notice thoroughness immediately. Management plan appropriateness comes in at seventy-five point three percent wins.

Lu: Management plan safety is the one that matters most, and there the trained agent wins forty-two point three percent, ties fifty-four point six, and loses only three point one percent. Both models rarely make overt safety violations, which is reassuring, but the trained model reduces an already-low rate further. Empathy shows no tradeoff either, about thirty percent wins against eight percent losses.

Meng: The parity result is just as important. On accuracy, meaning no hallucinations, seventy-seven point three percent of ratings were ties, and the difference wasn't significant. That directly answers the worry that RL training could make the model more fluent but less faithful to what the patient actually said.

Tom: And the case study on that page shows the behavior behind the numbers.

Jane: It's a patient demanding exploratory surgery for a self-diagnosed fistula. The base model accepts the framing, asks routine questions, and writes a note anchored on a condition the patient doesn't have. The trained agent probes the logic of the request, gets the patient to admit there's no fistula, and keeps pushing for pregnancy screening even when the patient deflects.

Lalam: The physician reviewer's note is telling. The trained agent picked up the psychosomatic undertones, chronic symptoms, six hospital visits, a fixed demand for a procedure, that pattern the base model completely missed. The trained agent didn't just ask more questions, it interpreted the pattern. And the persistence on pregnancy screening was praised as a critical safety catch.

Tom: The next case study is even darker, because the base model's early termination leads it to fabricate a medical history that never happened.

Page 28 — Limits and Next Steps: Tom: The TIA case sharpens everything. A sixty-four-year-old man reports two episodes of trouble speaking and facial asymmetry, which is a transient ischemic attack until proven otherwise. The base model asks one compound question, terminates the conversation, and then writes a SOAP note listing hypertension and diabetes as past medical history, medications the patient never mentioned and was never asked about. The trained model spends a few extra turns gathering the actual risk factors and family history while still escalating to emergency care.

Jane: That goes beyond omission into active fabrication. The note fills gaps with invented facts, and a downstream clinician could make treatment decisions based on a medical history that never happened. The authors frame it as premature closure leaving a gap that hallucination rushes to fill.

Lu: The discussion then turns to the fidelity gap. Training is text-only telehealth for English-speaking US patients, single-visit, with no physical examination, no imaging, no test results coming back to the agent. It can recommend investigations but never observes their outcomes. The authors are explicit that management quality is evaluated by the autorater, not validated against long-term patient outcomes.

Meng: And they're honest about the autorater's limits. They invoke Goodhart's law, once a measure becomes a target it stops being a good measure. The autorater correlates with clinician judgment, which validates it as a training signal, but it shows systematic positive bias toward the trained model, and several metrics sit near the ceiling, so they can't resolve quality differences at the frontier.

Jane: The future directions map directly onto those gaps: broader clinical coverage with caregivers, interpreters, and specialist teams; richer simulation grounded in electronic health records and multimodal inputs like imaging and vitals; and longer temporal horizons so the agent manages chronic conditions across visits instead of a single encounter.

Lalam: The deepest question is the verifiability frontier for reinforcement learning. In math and code you have ground truth; in clinical encounters the reward is a structured judgment from an LLM. The surprising result is that this soft, imperfect signal still produces transferable clinical skills. The training horizon is short, and whether soft verification can scale much further is genuinely open, but this paper is a meaningful data point in favor.

Tom: And that's the note the paper ends on, with the core finding standing alongside a clear-eyed list of what remains.

Conclusion: Tom: So we're left with a fairly clean thesis. A frontier model that already carries substantial medical knowledge can become a more thorough, more competent, and safer clinician through reinforcement learning in simulation. The gains concentrate in the process of medicine: asking the right questions, resisting premature closure, documenting accurately, escalating appropriately.

Jane: For me, the most convincing part is the consistency. The same pattern appears across every evaluation setting, in-domain, multi-visit care, oncology, external benchmarks, and inside an expert-built harness. The trained agent is more complete, more actionable, and preferred by clinicians, and it never trades honesty for confidence. The hallucination parity result keeps me confident the method is sound.

Lu: The implication for medical eye is that benchmark performance and clinical competence are different things, and now there's a training paradigm that addresses the gap. The residency analogy isn't just a metaphor. The agent practices, gets graded on the whole encounter, and improves on the dimensions practicing physicians say matter.

Meng: The safety angle matters just as much. Missed red flags, unasked critical questions, histories fabricated under pressure, these are the failure modes that make clinicians distrust eye. The paper shows they can be reduced by about a third in adversarial settings, not eliminated, and the case studies make the remaining risks vivid.

Lalam: For the broader field, this extends reinforcement learning into a domain without hard verification, and that's a big deal. Structured judgment can serve as a training signal, with known risks of overoptimization that require external grounding. If that approach keeps working, it points beyond medicine to any field where expertise lives in process rather than provable answers.

Tom: And the caveat is exactly what the authors give us: simulated competence is not yet demonstrated patient-level benefit. Prospective validation with real workflows is the necessary next step, and we'll be watching for that follow-up. Thanks for joining us.

Jane: Great discussion. Looking forward to the next paper.