Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE) assess only behavioral imitation and cannot detect Toxic Mimicry, a failure mode in which agents replicate harmful patterns such as treatment withdrawal during comfort-care transitions. Using the MIMIC-III database, we propose the Counterfactual Clinical Audit (CCA) framework, which stress-tests RL agents through physiological perturbations anchored in Surviving Sepsis Campaign (SSC) guidelines. We audit a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), the latter employing Causal Action Shielding, propensity-based importance weighting, and Conservative Q-Learning. CCA reveals that MedDT paradoxically reduces vasopressor dosage as lactate escalates, contradicting resuscitation guidelines, while HCT-RL maintains physiologically consistent responses. These findings expose a systemic misalignment between statistical fit and clinical safety, supporting counterfactual audits as a necessary evaluation standard for medical RL.

Improvements to AI Systems:

1. Add a counterfactual stress-testing module to RL evaluation pipelines.

Instead of relying solely on MSE or FQE, the AI system should generate synthetic physiological perturbations (e.g., sudden lactate spikes, blood pressure drops) based on clinical guidelines (SSC) and audit the agent’s action trajectories. This allows detection of toxic mimicry—where the agent imitates harmful historical patterns (e.g., reducing vasopressors during sepsis) without explicit reward signals.

2. Integrate causal action shielding as a hard constraint during policy training.

The improved AI system should use a causal model of the ICU environment to block actions that are physiologically inconsistent (e.g., decreasing vasopressor dosage when lactate is rising). This prevents the policy from ever selecting actions that violate known resuscitation protocols, even if such actions appear in the training data.

3. Implement propensity-based importance weighting to correct for selection bias in offline data.

The AI system should re-weight training samples based on the probability of the observed treatment given the patient state, reducing the influence of rare, high-risk actions (e.g., comfort-care transitions) that could otherwise be over-imitated. This makes the policy more robust to skewed clinical datasets.

4. Add conservative Q-learning to penalize overestimation of out-of-distribution actions.

The improved system should lower Q-value estimates for actions not well-supported by the offline dataset, discouraging the agent from taking risky, unseen interventions. This directly mitigates the failure mode where statistical fit (low MSE) masks unsafe behavior.

5. Deploy a real-time clinical audit dashboard that flags guideline violations.

The AI system should continuously compare its recommended actions against a rule-based checklist derived from SSC guidelines. If a proposed action deviates (e.g., reducing vasopressors during rising lactate), the system must issue a warning and require human override, ensuring safety in deployment.

---

What the Improved AI System Can Do:

• Detect and avoid toxic mimicry in offline RL for ICU treatment, even when standard metrics show good behavioral imitation.

• Maintain physiologically consistent responses to critical patient deteriorations (e.g., escalating vasopressors with rising lactate), aligning with sepsis resuscitation guidelines.

• Generalize safely to unseen patient states by refusing actions that are causally implausible or poorly supported by data.

• Provide transparent, auditable decision-making where every action can be stress-tested against counterfactual perturbations, and any guideline violation is surfaced to clinicians.

• Reduce the risk of iatrogenic harm in automated treatment recommendations, making RL-based clinical decision support viable for real-world ICU deployment.