The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing

arXiv:2606.25108 · cs.AI, cs.HC · Submitted 2026-08-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing".

Jane: The paper was written by Eileanor LaRocco, Sarah Tan, Adarsh Subbaswamy, Anne Andrews, Andrew Taylor et al. from University of Virginia and Cornell University and University of Maryland, Baltimore and National Institute of Standards and Technology.

Tom: Stay tuned as we take you through the paper and discuss its implications.

The Summary of Risk: Tom: We've established how the policy landscape is pushing AI toward autonomy, but now we want to look at what the research found regarding risk. The paper suggests that clinicians aren't okay with blind autonomy, and that their comfort level changes drastically depending on whether the AI is uncertain for a valid clinical reason or because it just hasn't been trained on enough examples.

Jane: The data shows that this distinction—between aleatoric uncertainty, which reflects real clinical ambiguity, and epistemic uncertainty, which is model ignorance—is very important to the clinicians. They view these scenarios as fundamentally different things from one another.

Lu: That's a huge insight; they don't treat the lack of data in training and real-world complexity as interchangeable when deciding how to handle the AI’s output.

Meng: This means that if we are building any kind of interface for this system, we must know the root cause of every single uncertainty signal, not just whether there is *any* level of uncertainty present.

Lalam: The idea that "aleatoric" versus "epistemic" should fundamentally change how a system escalates a case is important; it’s not just about a generic confidence score failing.

Tom: And that ties into the second major finding: when the uncertainty is aleatoric, they prefer seeing a summary of competing options, but when it's epistemic, their preference shifts strongly toward abstention.

Jane: That’s a powerful behavioral observation; they trust the AI with real options in complex scenarios, but they completely distrust it when the AI simply hasn't encountered enough examples to know what to do.

Lu: This is a perfect example of how human intuition overrides technical capability—when the model is just ignorant, they want a hard stop and a human decision.

Meng: I see this as demanding that we build two distinct pathways into the system, not just one general "low confidence" warning that covers both types of failure.

Lalam: This structure of trust suggests that if we can respect the model’s limits, even when those limits are due to ignorance rather than complexity, we might be able to build a culture where people accept the technology responsibly.

Tom: That brings us into our next segment, where the paper details its specific recommendations for architectural solutions.

The Recommended Improvements: Tom: We’ve seen that clinicians don't trust blind autonomy, so the paper proposes three minimum requirements to ensure safe deployment of autonomous AI systems. These are the technical features needed to bridge the gap between current policy and real-world safety concerns.

Jane: The first requirement is using calibrated, action-gated confidence thresholds, which is not just a low confidence score—it' essentially means forcing the AI to pause and wait for a human when its internal score says it isn't sure enough to act autonomously.

Lu: That’s a critical safety mechanism; an uncalibrated model can confidently make a mistake, but we need that mechanism to actively stop the action before the human even sees it.

Meng: That provides a necessary guard rail that is preventing catastrophic failures when the system from an engineering viewpoint lacks confidence in its output.

Lalam: The implication of this is that "autonomy" isn't about doing things independently; it’s about having an independent, verifiable check on *when* you do things.

Tom: And the second requirement builds on that is differentiated uncertainty communication—separating epistemic ignorance from aleatoric ambiguity.

Jane: It’s not enough to know the AI was uncertain; we have to know *why* it was uncertain in order to make a proper decision about how to handle the case.

Lu: This distinction is vital because it tells the clinician whether they are solving a problem that is inherently difficult, or a problem that simply hasn't been taught yet by the model.

Meng: From an engineering standpoint, this requires developing robust scoring rules and monitoring systems for each specific type of uncertainty during deployment.

Lalam: By building in these specific communication methods, we move away from just letting the system fail and instead guide the entire decision-making process toward a more responsible outcome for patients.

Tom: Finally, the third requirement is inferential transparency—showing us the reasoning for every single step of making a decision.

Jane: This isn't just having general model documentation; it's about providing per-prediction transparency so that when we look back at an adverse outcome, we know exactly how and why the AI reached that specific conclusion.

Lu: This provides the ultimate audit trail, allowing us to trace every input and every decision back to its source for accountability.

Meng: Without this traceability, you cannot even assign liability fairly when a mistake is made, so this requirement is essential for legal compliance as well as safety.

Lalam: The goal of ensuring transparency isn't just for the sake of accountability; it encourages the clinician to trust the entire process in a way that supports their professional judgment.

Conclusion: Tom: We've seen these three requirements—calibrated gating, differentiated uncertainty, and inferential transparency—and they lead to a major conclusion in the paper’s findings regarding autonomy. They suggest that if we want safety, we have to accept a much smaller degree of autonomy than is currently envisioned by lawmakers.

Jane: The authors argue that this set of minimal conditions collapses much of what we conventionally mean by 'autonomous' in AI; it becomes more like a heavily supervised decision-support tool rather than a fully self-governing agent.

Lu: I think the biggest takeaway is that the paper argues for redefining what "agent" means, which fundamentally changes how we view our relationship with this technology.

Meng: That’s the practical implication, Tom; the system isn't making a final judgment on its own, it’s providing highly vetted suggestions that requires human review.

Lalam: This shift in definition is critical because it impacts our cultural relationship with healthcare—it moves us from replacing the clinician to actively augmenting them.

Tom: And we saw that when things go wrong, the clinicians tend to assign more responsibility for an adverse outcome to the organizational bodies like health systems and regulatory agencies than to individual clinicians or patients.

Jane: This is a huge finding for liability alignment; it confirms that if you build in transparency, you can align responsibility with the actors who control system design and deployment.

Lu: It’s a way of saying that accountability must follow the technical capacity, which is currently held by the designers and deployers of these complex AI systems.

Meng: From an operational standpoint, this means our focus should be on how we structure corporate governance around these high-risk AI tools to ensure that transparency is maintained.

Lalam: The ultimate vision here for a healthier society is one where the technology isn't just an efficiency tool, but a framework that upholds professional standards and legal accountability.

Tom: It sounds like the paper The Clinician’s Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing is providing us with a much-needed roadmap for how to implement this balance.

Jane: We've seen the findings, we've understood the technical requirements, and we are ready to look at what comes next.

Lu: I think understanding that the Veto is necessary really allows us to see how policy must catch up with this structural constraint on our AI systems.

Meng: I agree; if legislative bodies don’t recognize these minimum safety requirements, they are setting up a system that will eventually fail in the real world.

Lalam: To ensure the future is one where technology serves human needs safely, we must respect these boundaries defined in The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing.

Conclusion: Tom: So, we’ve seen that The Clinician’s Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing isn't just a technical paper; it's a roadmap for achieving safe AI in medicine.

Jane: Exactly, Tom. The core the message is that the path to trustworthy autonomous prescribing requires specific architectural safeguards—calibrated action gating, understanding why uncertainty comes from both real ambiguity and model ignorance, and ensuring per-prediction transparency.

Lu: I’m thrilled by the implications of this work; it moves us away from viewing AI as a magic "black box" that can just decide things on its own.

Meng: It forces a practical reality: we can't just push an agent to act; we have to build in hard stops and specific communication paths that allow us to manage risk when the system fails.

Lalam: The way this paper frames the relationship between technology and human judgment suggests a more respectful, shared responsibility in healthcare culture.

Tom: And it's not just about safety either, Jane; we found that liability shifts significantly toward organizations rather than individuals when we build in these accountability features.

Jane: That's a crucial point for the policy discussion, as it aligns the person who is responsible for the data with the people who are responsible for making decisions about that data.

Lu: I think this suggests a future where our AI systems aren't just tools, but partners whose limits we deeply understand and respect.

Meng: We need to ensure that when we build these systems, they won't just be compliant with the law, but robust enough to handle real-world clinical complexity.

Lalam: This framework shows that true technological progress isn't about replacing human expertise, but elevating it through a well-designed partnership.

Tom: It’s definitely a conversation that The Clinician’s Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing demands.

Jane: It provides us with the necessary vocabulary to discuss both the technical hurdles and the ethical responsibility of moving forward with these systems.

Lu: I can't wait to see how this framework influences future deployments across various specialties.

Meng: I am looking forward to seeing how our industry adapts these constraints into actual, scalable software architecture.

Lalam: It’s a powerful reminder that when technology serves human needs safely, it helps us build a more just and transparent healthcare system.

Eileanor LaRocco, Sarah Tan, Adarsh Subbaswamy, Anne Andrews, Andrew Taylor, Cree Gaskin, Chirag Agarwal, Eileanor LaRocco's affiliation is University of Virginia. Sarah Tan's affiliation is Cornell University. Adarsh Subbaswamy's affiliation is University of Maryland, Baltimore. Anne Andrews and Andrew Taylor are affiliated with the University of Virginia. Cree Gaskin is affiliated with the National Institute of Standards and Technology. Chirag Agarwal is also affiliated with the University of Virginia.

University of Virginia · Cornell University · University of Maryland, Baltimore · National Institute of Standards and Technology

cs.AI, cs.HC

Submitted: 2026-08-24

Updated: 2026-08-25

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 95/100

The gist: This paper investigates the technical and ethical requirements for transitioning AI from advisory roles to autonomous agents capable of prescribing medications.

Key concepts

Aleatoric Uncertainty
This type of uncertainty reflects genuine clinical ambiguity. It arises from the inherent complexity of a real-world medical situation, rather than because the AI lacks sufficient data in its training set.
Epistemic Uncertainty
This refers to model ignorance. It occurs when an AI has not been trained on enough examples to know how to handle a specific scenario, leading clinicians to distrust the system and prefer human intervention.
Inferential Transparency
This requires providing a complete audit trail by showing the reasoning for every single step the AI takes when making a decision. This allows users to trace inputs and decisions back to their source for accountability.

Terminology

Summary

This paper investigates the technical and ethical requirements for transitioning AI from advisory roles to autonomous agents capable of prescribing medications. As legislative efforts like H.R. 238 and state-level pilots move toward authorizing AI as legally recognized prescribers, there is a critical need to establish the architectural conditions necessary for safe deployment. The authors argue that without specific safeguards regarding uncertainty and transparency, autonomous prescribing risks creating an accountability gap where clinicians bear liability for decisions they cannot meaningfully control or verify.

The Core Argument

The authors posit that the minimum architectural conditions required for safe autonomous prescribing are stringent enough to collapse much of what ‘autonomy’ conventionally means. Rather than functioning as fully independent agents, a safe system would operate more as a heavily supervised decision-support tool. To achieve this, the paper proposes three essential technical requirements:

  1. Calibrated, action-gated confidence thresholds that enforce abstention on low-confidence decisions.

  2. Differentiated uncertainty communication that separates epistemic uncertainty (model ignorance) from aleatoric uncertainty (irreducible clinical ambiguity).

  3. Inferential transparency, defined as per-prediction transparency at the moment of decision, to ensure every autonomous action is auditable and allows for proper liability allocation.

Empirical Findings on Safety and Escalation

Through a survey of 136 U.S. prescribing clinicians, the researchers tested these hypotheses and found significant resistance to unsupervised autonomy. Clinicians expressed very low support for autonomous prescribing across most drug categories, with opposition exceeding 83% for new prescriptions and high-risk refills. However, the study found that:

A majority will indicate they would not permit autonomous prescribing without such a mechanism

Specifically, 67.4% of respondents stated they would not permit AI agents to prescribe drugs without a calibrated confidence-based escalation mechanism. Furthermore, clinicians rated the patient safety of calibrated systems as substantially safer than uncalibrated ones.

Uncertainty and Automation Bias

The research highlights how the source of uncertainty should dictate the interface provided to a clinician during an escalated review. The authors argue that displaying a preferred option when a model lacks validated data increases the risk of automation bias. The survey results confirmed that clinicians require different information depending on why the AI is uncertain:

In aleatoric cases, a competing-options summary is appropriate... In epistemic cases, autonomous action should halt entirely and no AI recommendation should be displayed.

The data showed that while 71.0% of clinicians preferred a summary of competing options for aleatoric uncertainty (Scenario A), the preference shifted significantly toward abstention (presenting clinical data only) when faced with epistemic uncertainty (Scenario B).

Liability and Accountability

A central concern addressed is the many-hands problem, where responsibility is distributed across various actors. The study found that clinicians assign significantly more liability to organizational entities—such as AI companies, regulatory bodies, and health systems—than to individuals when an AI acts autonomously. However, a critical shift occurs when transparency is provided: clinicians are more willing to accept personal responsibility only when the agent communicates its confidence and enables them to make a substantive judgment under acknowledged uncertainty. This suggests that for autonomous prescribing to be ethically defensible, regulation must align liability with the institutional actors who control system design and deployment.of.

Improvements for AI systems

To improve autonomous AI prescribing systems based on the findings of this paper, I would implement the following three architectural upgrades. These improvements transition the system from a black-box agent to a supervised agentic system.

  1. Implement Calibrated, Action-Gated Confidence Thresholds

The system must move beyond simple probability outputs to mathematically calibrated confidence scores.

The improved AI system will:

  • Calculate a real-time confidence score for every proposed prescription.

  • Compare this score against a pre-set Action Gate threshold (determined by clinical oversight boards).

  • If confidence falls below the threshold, the system will automatically trigger a hard stop, preventing the prescription from being issued and forcing an immediate escalation to a human clinician.

  • Ensure that high-confidence errors (overconfidence) are minimized through calibration techniques, preventing the agent from acting on hallucinated certainty.

  1. Implement Differentiated Uncertainty Communication (Aleatoric vs. Epistemic)

The system must distinguish between clinical ambiguity and model ignorance to prevent automation bias during clinician review.

The improved AI system will:

  • Identify the source of uncertainty:

  • In cases of Aleatoric Uncertainty (irreducible clinical ambiguity, e.g., two equally valid drugs exist), the system will present the clinician with a Competing-Options Summary that lists the alternative treatments alongside their relative confidence values.

  • In cases of Epistemic Uncertainty (model ignorance, e.g., the patient has a rare profile not found in training data), the system will suppress all recommendations and present only the Raw Clinical Data. This forces the clinician to perform a primary analysis rather than reflexively agreeing with a flawed AI suggestion.

  1. Implement Per-Prediction Inferential Transparency

The system must provide just-in-time reasoning records for every individual decision to enable auditability and proper liability allocation.

The improved AI system will:

  • Generate a real-time, traceable reasoning provenance for every prescription, detailing the specific data points and logic paths used to reach the conclusion.

  • Expose this reasoning to the clinician at the exact moment of escalation, allowing them to make a substantive judgment rather than a rubber-stamp approval.

  • Maintain a granular, auditable log of both the AI’s confidence levels and its reasoning, which allows post-hoc forensic analysis to determine if a failure was due to a model architectural error (AI company liability) or a clinician's incorrect concurrence (clinician liability).

Abstract

Autonomous AI systems are transitioning from advisory roles to autonomous ones for medication prescriptions. Recent U.S. bill H.R. 238 and Utah's prescription-renewal pilot program both authorize AI to prescribe medications in an agentic capacity. While many regulatory guidelines suggest aggregate model performance metrics at the point of clearance, they do not require i) calibrated per-prediction confidence for action-gated thresholds, ii) differentiated communication between uncertainty arising from model ignorance (epistemic) from genuine clinical ambiguity (aleatoric), and iii) inferential transparency at the moment of decision enabling liability allocation. Here, we argue these three architectural features are minimum conditions for safe autonomous prescribing, and validate them with a survey of 136 U.S. prescribing clinicians. Our results suggest prescribing clinicians i) would not permit autonomous prescribing without a confidence-based escalation mechanism, ii) preferred a competing-options summary for aleatoric uncertainty but preferred abstention for epistemic uncertainty, and iii) were only willing to accept liability when inferential transparency enabled them to make a decision under acknowledged uncertainty. These findings indicate that our recommended architectural features would encourage higher rates of clinician adoption of autonomous AI prescribing, largely through collapsing much of what "autonomy" conventionally means.

Sources

Related papers