Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs

arXiv:2503.20182 · cs.CL, cs.AI · Submitted 2026-08-17 · Read on arXiv

Huanhuan Ma, Haisong Gong, Xiaoyuan Yi, Xing Xie, Philip S. Yu, Dongkuan Xu

cs.CL, cs.AI

Submitted: 2026-08-17

Updated: 2026-08-19

Code: https://github.com/dependentsign/CSI

Terminology

• Consistency — 21× in this paper · explained in 3 episodes
• Implicit Association Test — 6× in this paper
• Core — 5× in this paper · explained in 2 episodes
• Core Sentiment Inventory — 5× in this paper
• Ablation — 4× in this paper · explained in 1 episode
• Alignment — 4× in this paper · explained in 1 episode
• Ablation Studies — 3× in this paper · explained in 1 episode
• Instruction Tuning — 3× in this paper · explained in 2 episodes
• Downstream Tasks — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Big Five Inventory — 1× in this paper
• Fidelity — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Inducing anxiety in large language models can induce bias
• COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
• Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches
• Machine Psychology
• Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases
• GPT-4 Technical Report
• LongForm: Effective Instruction Tuning with Reverse Instructions
• Instruction Tuning with GPT-4
• Evaluating Large Language Models in Theory of Mind Tasks
• The opportunities and risks of large language models in mental health
• Personality Traits in Large Language Models
• Systematic Biases in LLM Simulations of Debates
• Evaluating General-Purpose AI with Psychometrics
• From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering