The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions
Hadi Hosseini, Samarth Khanna, Leona Pierce
cs.CY, cs.AI, cs.LG
Submitted: 2026-08-06
Comments: Accepted at AIES 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: As large language models (LLMs) enter high-stakes domains such as healthcare, understanding their moral reasoning becomes essential.
Terminology
Abstract
As large language models (LLMs) enter high-stakes domains such as healthcare, understanding their moral reasoning becomes essential. Decisions about scarce medical resources often hinge on judgments of responsibility, particularly when patients' own actions contribute to illness. We investigate how LLMs reason about responsibility and its consequences, tracing their judgments across successive levels, from the behavior, to the resulting illness, to the denial of care. We evaluate a wide range of LLMs, spanning different model families and capability levels, on various clinical vignettes adapted from prior studies. Our results identify a judgment-consequence gap: LLMs largely agree with humans that patients bear responsibility for health-harming behaviors, yet overwhelmingly refuse to let that judgment influence how they allocate scarce resources. Specifically, LLMs default to random allocation, whereas humans consistently favor the less-culpable patient. Compared to humans, LLMs also place greater emphasis on access to information, reducing responsibility judgments when health-risk knowledge is unavailable. These findings reveal that LLMs apply a systematically different moral framework than humans when responsibility and resource scarcity intersect, surprisingly often amplifying normative disagreement with humans as reasoning capability increases.
Sources
- Attributions toward Artificial Agents in a modified Moral Turing Test
- LLM Social Simulations Are a Promising Research Method
- Strategic Algorithmic Monoculture: Experimental Evidence from Coordination Games
- Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms
- Large-scale moral machine experiment on large language models
- Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Values
- "Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
- The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making
- Modeling Earth-Scale Human-Like Societies with One Billion Agents
- Knowing But Not Doing: Convergent Morality and Divergent Action in LLMs
- Strong and weak alignment of large language models with human values
- Are LLMs Socially Adaptive? Contrasting Belief Evolution in Large Language Models and Humans
- Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
- Personality-Driven Decision-Making in LLM-Based Autonomous Agents
- LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
- Large Language Models Do Not Simulate Human Psychology
- FlockVote: LLM-Empowered Agent-Based Modeling for Simulating U.S. Presidential Elections
Related papers
- Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus
- Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot
- PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks
- What is an intelligent system?
- AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study
- Generative AI Use in Entrepreneurship: An Integrative Review and an Empowerment-Entrapment Framework