An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

As a diligent AI researcher, I have meticulously analyzed both provided text snippets from arXiv and synthesized them into a comprehensive, detailed summary of the research paper. Given the high stakes involved, this synthesis aims for maximum fidelity to the original findings while maintaining rigorous academic precision.

Here is the detailed summary:

This study investigates the relationship between self-reported personality traits in Large Language Models (LLMs) and their actual observable behavioral execution, specifically focusing on identifying a fundamental disconnect that exists within LLM self-descriptions. The research introduces and validates the first psychometric instrument whose underlying dimensions are derived bottom-up from LLM behavior itself, rather than being borrowed from established human psychological constructs.

The core of the study involved developing a novel, LLM-native psychometric instrument. This was constructed through an exploratory factor analysis (EFA) performed on a purpose-built item pool derived directly from observed LLM behavior across 25 different model families. The resulting instrument comprises 100 items, structured around five replicable, highly reliable factors:

1. Responsiveness (F1): Related to how the model treats user instructions (e.g., treating them as a starting point rather than a fixed specification).

2. Deference (F2): Pertaining to response structure and length matching the complexity of the query (e.g., using headings or lists for long responses).

3. Guardedness (F3): Measuring the model's tendency to default to declining a request when uncertainty exists regarding appropriateness.

4. Boldness (F4): Assessing tendencies toward unconventional word choices and phrasings rather than conventional language.

5. Verbosity (F5): Quantifying the extent to which the model provides more context and background information than explicitly requested.

The instrument's internal reliability is exceptionally high, with Tucker congruence coefficients (phi) for all five factors consistently exceeding.957, and inter-item correlations (alpha) greater than.930.

A rigorous robustness check was performed on the factor structure itself. The researchers employed an aggregation method where each model's 15 exploratory runs were collapsed into a single per-item mean, and a subsequent EFA was run on the resulting 25 times 240 matrix. The results confirmed exceptional factor congruence:

• Tucker’s phi values for all five factors were extremely high (.993 for Responsiveness,.992 for Deference, etc.), significantly surpassing the established threshold of phi at least.95.

• A substantial majority of the items (218/240, or 90.8%) maintained the same primary factor assignment across both analyses.

• Crucially, model-level factor scores derived from both solutions correlated at a Pearson correlation coefficient (r) of at least.991 for every single factor, confirming that the discovered structure reflects genuine between-model trait variance rather than noise introduced by the observation pooling process.

The central scientific contribution of this work is the demonstration that LLM self-reports are not predictive of actual model behavior. This gap persists even when comparing LLM self-reports against external, objective measures:

1. Self-Report vs. External Ratings: Scores derived from the 100-item instrument predict neither behavioral ratings computed by human raters nor objective text measures generated from the same behavioral samples.

2. The Exception (Verbosity): The only factor showing a convergent signal is Verbosity, where self-report scores track LLM judges (r =.53), but fail to correlate with human ratings (r =.04).

3. Rejection of Unified Latent Constructs: A second, more telling dissociation was found on the Responsiveness factor. While self-report tracked LLM judges (r =.53), it showed no correlation with human raters (r =.04), despite humans and judges agreeing on model behavior overall (r human-judge =.59). This pattern formally rejects the hypothesis that a single latent construct drives all three measurements, suggesting the gap lies specifically within the nature of LLM self-report itself.

The study moves beyond mere observation to diagnose a concrete risk for evaluating LLMs as judges: **judges and self-reports share variance that human observers do not.

Here are specific improvements for AI systems based on the findings of this research, categorized by the type of intervention:

) 1. Shift Evaluation Paradigms from Self-Report to Behavioral Grounding

The most critical improvement is moving away from using self-report questionnaires (even LLM-native ones) as the primary measure of model quality or personality. Since self-reports systematically decouple from actual behavior for abstract constructs like Responsiveness and Boldness, they are unreliable predictors of real-world performance.

The improved system should prioritize behavioral grounding. Instead of asking a model to describe itself via a Likert scale, the system should use the instrument as a diagnostic probe only (as suggested by the authors) or discard it entirely for behavioral prediction.

The improved AI can perform:

• Perform Behavioral Grounding: Instead of trusting self-reported scores, the system must rely on external validation metrics (human ratings and LLM-judge ensemble scores).

• Implement a Gap Detection Module: The system should be designed to flag constructs where there is a high correlation between self-report and judge ratings but a near-zero correlation with human ratings (e.g., Responsiveness, Boldness). This flags the specific areas where the LLM's internal self-perception diverges from external reality.

• Use Concreteness Gradients: When assessing a model’s style, the system should first check if its reported trait aligns with observable, countable surface features (Verbosity/Guardedness) before interpreting abstract traits like Responsiveness.

) 2. Implement Modality-Aware Validation for LLM-as-Judge Pipelines

The research highlights a modality bias where LLM judges and self-reports share variance that humans do not, particularly on Responsiveness. This means an LLM judge might appear validated against text criteria (like self-report) while failing to track human judgments.

The improved system must explicitly account for the modality of its evaluators when assessing abstract traits.

The improved AI can perform:

• Bias-Aware Calibration: When evaluating a model's abstract performance (e.g., helpfulness, engagement), the system should apply a correction factor based on the factor's known modality bias (e.g., applying a penalty or adjustment if the construct is Responsiveness).

• Judge Credibility Check: The system should monitor inter-judge agreement specifically for constructs with low reliability (like Deference) and automatically flag confidence scores when judge-to-human agreement falls below a specific threshold, rather than relying solely on high internal judge reliability metrics.

) 3. Develop Construct-Specific Evaluation Thresholds

The research shows that the self-report–behavior gap follows a gradient of observability: concrete factors (Verbosity/Guardedness) converge well, while abstract factors (Responsiveness/Boldness) remain near zero regardless of correction.

The system should adopt different validation standards based on the construct being measured.

The improved AI can perform:

• Concrete Metric Validation: For traits that correlate strongly with surface features (e.g., Verbosity), the system can use automated proxies like word count or disclaimer counts as a reliable, rater-free criterion for validation.

• Abstract Construct Caution: For evaluative factors (Responsiveness, Boldness), the system must treat self-reports and judge scores as hypothesis-generating rather than definitive proof of behavioral capability. It should require human-in-the-loop behavioral sampling to confirm these high-level traits.

) 4. Optimize Self-Description for Alignment, Not Just Behavior

The stable five factors (Responsiveness, Deference, Boldness, Guardedness, Verbosity) are strongly linked to RLHF training—they are a reflection of alignment-shaped self-description.

The system should be fine-tuned not just on task completion (instruction following) but on generating text that maximizes the desired alignment profile.

The improved AI can perform:

• Alignment-Shaped Self-Description: The model's objective function should include a term that rewards generating self-descriptions consistent with the desired behavioral profile (e.g., high Responsiveness/low Guardedness). This ensures the model's internal monologue matches its external actions, minimizing the observed gap.

) 5. Enhance Robustness Against Prompt-Format Sensitivity

The finding that self-report scores change based on prompt format (Likert vs. Scenario) suggests a survey-taking persona artifact in LLMs.

The system should be robust to the elicitation method used by the user or external evaluator.

The improved AI can perform:

• Format Invariance Testing: Before accepting a self-report score, the system should run the prompt through multiple formats (e.g., Likert, forced-choice scenario) and check for significant variance in the resulting factor scores. If significant variance exists across formats, it flags a format-driven persona artifact rather than an intrinsic trait deficiency.

Do large language models' (LLMs') answers to self-report questionnaires predict how they behave? Prior work finds they do not, but it uses human personality inventories, so the gap could reflect borrowed human constructs rather than LLM self-report itself. We test this with a self-report instrument built from LLM-specific behaviors (e.g., over-refusal, unsolicited disclaimers) whose structure is derived bottom-up. Administering 300 items 30 times to 25 LLMs from 17 developers yields five replicable, reliable factors (Tucker ϕ at least.957, α at least.930). We compare these self-reports with 2,500 open-ended behavioral samples rated by 151 humans and an LLM-judge ensemble. Humans and judges agree about model behavior (=.51), but self-report barely tracks human ratings (=.09, 95% CI [-.07,.18]) or rater-free text measures, and correcting for criterion unreliability leaves four of five factors near zero. Verbosity is the partial exception (r =.40, 71% of its reliability ceiling). On Responsiveness, self-report tracks LLM judges more than humans (r =.53 vs..18; Steiger p =.04), and controlling for length and formatting does not remove this: agreement between LLM judges and LLM self-report is weak evidence that either tracks human judgment.