ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation

arXiv:2605.29791 · cs.CL · Submitted 2026-05-28 · Read on arXiv

cs.CL

Submitted: 2026-05-28

Updated: 2026-09-09

Comments: Accepted to Findings of EMNLP 2026. Camera-ready version

Code: https://github.com/Selina233/ActTraitBench

License: http://creativecommons.org/licenses/by/4.0/

The gist: While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap (G

Terminology

Abstract

While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap (G KD). Existing benchmarks struggle to measure this discrepancy due to limited construct validity, multidimensional entanglement, and distributional biases in LLM-based evaluation. To address these issues, we propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs. Grounded in empirical human data, ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms and applies Distributional Calibration via Quantile Mapping to reduce distributional mismatch between LLM-judge scores and human responses. Experiments on 14 mainstream LLMs reveal substantial knowledge-decision gaps and show that assigned personas are reflected more consistently in self-reports than in behavioral decisions for most evaluated models. To mitigate this gap, we further introduce the Chain of Cognitive Alignment (CoCA), an inference-time intervention that reduces G KD for 12 of the 13 models with paired results. Code and resources are available at https://github.com/Selina233/ActTraitBench.

Sources

Related papers