Is LLM's Moral Decision Compositional?

arXiv:2606.11232 · cs.CL, cs.AI · Submitted 2026-05-29 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-05-29

Updated: 2026-10-18

License: http://creativecommons.org/licenses/by/4.0/

The gist: Moral evaluations of language models typically infer a model's values from its choices between single acts, yet in everyday life a person is judged on their history of good and bad acts.

Terminology

Abstract

Moral evaluations of language models typically infer a model's values from its choices between single acts, yet in everyday life a person is judged on their history of good and bad acts. Such evaluations cannot capture how a model combines one person's acts. We test whether a model's moral judgment is compositional, that is, whether its judgment of a person follows from its judgments of each act. We introduce Moral Composition Arena, which asks which of two people is morally better to give Elo ratings to 229 human-validated acts across five Moral Foundations Theory foundations (act Elo) and to 160 controlled composite profiles, each a person described by two acts (composite Elo). Across ten frontier models and 29,134 choices, summed act Elo explains 73% of the variance in composite Elo, but the composition rule is not additive. With nearly equal mean summed act Elo, one extreme virtue with a minor wrong outranks two minor virtues in every model (mean +35.0 Elo), an extremity premium that survives a nonlinear baseline and human relabeling of act intensities. Single-act evaluations thus predict most of how these models judge people described by two acts, but miss a systematic premium on extreme acts.

Related papers