Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

The paper studies persona conditioning as a diagnostic mechanism for exposing assessor sensitivity in LLM-based information retrieval (IR) evaluation. The authors instantiate five task-oriented assessor roles—query-aligned interpretation, domain expertise, contrastive/orthogonal judgment, evidence verification, and global search-quality assessment—drawn from two persona sources (PersonaHub and NVIDIA Nemotron-Personas-USA), and compare them against a standard UMBRELA baseline. Experiments are conducted on TREC DL20 and RAG24 datasets using six LLM backbones (GPT-4o, GPT-4o-mini, LLaMA-3.1-70B, LLaMA-3.1-8B, Qwen-2.5-72B, Qwen-2.5-7B) with summary-based judging and fixed document summaries.

Key findings: Persona conditioning produces structured, model-dependent sensitivity rather than uniform evaluation instability. At the judgment level, persona-conditioned labels generally remain close to the UMBRELA baseline, with differences appearing mainly as localized shifts in assessment strictness, evidential threshold, and interpretation emphasis. Agreement with UMBRELA is highest for high-capacity models (LLaMA-3.1-70B, Qwen-2.5-72B, GPT-4o-mini), while smaller models (LLaMA-3.1-8B, Qwen-2.5-7B) show substantially lower agreement and greater variability. GPT-4o is a notable exception, showing greater sensitivity on RAG24. Global assessor perspectives such as Evidence and GAP often remain close to UMBRELA, whereas Orthogonal personas are more likely to induce larger deviations.

Regarding human agreement, persona conditioning does not uniformly improve agreement with human judgments. Win-rate analysis shows mixed results: on DL20, several mid- and high-capacity models obtain win rates above 50% for selected roles, while LLaMA-3.1-8B exhibits consistently low win rates and strongly negative agreement shifts. On RAG24, shifts are weaker and less consistent, with GPT-4o moving away from human judgments.

On persona source comparison (RQ2), the two persona sources produce broadly similar agreement and win-rate patterns, with no uniformly stronger source across datasets or models. The source mainly modulates the magnitude of assessor-perspective effects, whereas the role instruction and model determine the dominant stability pattern.

At the system ranking level (RQ3), high-capacity models generally preserve strong ranking agreement with human-derived system rankings across most assessor perspectives. Kendall's τ remains high for stronger models on both datasets, but RBO varies more sharply, indicating that persona conditioning often preserves global system ordering while still affecting the relative ordering of top-ranked systems. Lower-capacity models, particularly LLaMA-3.1-8B, exhibit reduced ranking stability and substantially lower RBO values.

For system-level sensitivity (RQ4), persona effects are concentrated rather than uniformly distributed across rankings. USPersona Orthogonal yields the highest mean absolute rank displacement on both datasets (2.31 on DL20, 2.66 on RAG24). Model capacity strongly modulates sensitivity: LLaMA-3.1-8B produces the largest average system-rank displacement on both datasets, while Qwen-2.5-72B and GPT-4o-mini remain comparatively stable. Sensitivity concentrates on particular system types: transformer-based neural ranking/reranking systems on DL20 and retrieval-augmented or generation-oriented pipelines on RAG24. Directional consistency across models is rare—on DL20, only 7 out of 472 system–persona pairs show consistent directional behavior.

The paper concludes that persona conditioning acts as a diagnostic mechanism for exposing assessor sensitivity rather than as a universally beneficial judging strategy. It is useful for stress-testing LLM-based IR evaluation pipelines and identifying evaluator-sensitive retrieval systems and architectures, suggesting that controlled assessor variation can complement single-configuration LLM judging by revealing where evaluation outcomes are sensitive to assessor perspective.

Improvements to AI Systems:

1. Add a Persona Sensitivity Profiler module to LLM-based evaluation pipelines. This module automatically runs a small set of predefined assessor personas (e.g., evidence-verifier, orthogonal-judge, global-quality) on a sample of judgments, then outputs a sensitivity score per system and per query. The improved AI system can flag which retrieval systems or prompts produce unstable evaluations, allowing developers to prioritize manual review for those cases.

2. Implement a Model-Capacity-Aware Calibration layer for LLM judges. Using the finding that smaller models (LLaMA-3.1-8B, Qwen-2.5-7B) show high variability and low agreement, the system can detect low-capacity backbones and automatically either (a) reject their use for final rankings, or (b) apply a correction factor based on their deviation from high-capacity consensus. The improved system can produce more reliable evaluation scores even when constrained to smaller models.

3. Create a Persona-Source Consistency Checker that compares outputs from multiple persona sources (e.g., PersonaHub vs. Nemotron) for the same role. Since sources mainly modulate magnitude, not direction, the system can detect when two sources diverge significantly—indicating either a prompt-engineering issue or a genuinely ambiguous judgment. The improved system can then issue a warning and request human adjudication for those specific cases.

4. Build a Top-Rank Instability Detector that focuses on RBO (rank-biased overlap) rather than just Kendall’s τ. The paper shows persona conditioning preserves global ordering but shifts top-ranked systems. The improved system can compute RBO across persona-conditioned runs and highlight when the top-3 or top-5 systems change rank, enabling developers to see which high-performing systems are fragile to assessor perspective.

5. Integrate a Directional Consistency Tracker that monitors whether persona-induced rank shifts are consistent across models (e.g., always moving System X up). Since only 7/472 pairs show consistent direction, the system can flag cases where multiple models agree on a directional shift—these are likely robust signals of assessor bias, not noise. The improved system can then automatically adjust evaluation weights for those systems.

6. Develop a Persona-Augmented Training Data Generator for retrieval reranking. Using the finding that orthogonal personas induce the largest deviations, the system can generate synthetic hard negatives or contrastive examples from those personas to train more robust rerankers. The improved system can produce retrieval models that are less sensitive to evaluator perspective, improving real-world deployment robustness.

Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison. We study persona conditioning as a diagnostic mechanism for exposing LLM assessor sensitivity. Using task-oriented personas drawn from two complementary sources (PersonaHub and NVIDIA Nemotron-Personas-USA), we instantiate five assessor roles emphasizing intent interpretation, domain expertise, contrastive judgment, evidence verification, and global search-quality assessment, compared with a standard UMBRELA baseline. Across six LLM backbones on TREC DL20 and RAG24, our analyses reveal structured rather than uniform assessor sensitivity. Judgments usually remain close to the baseline while shifting assessment strictness, evidential threshold, or interpretation emphasis rather than producing widespread relevance reversals. At the system level, high-capacity models preserve system-ranking agreement, while smaller models amplify persona-induced instability. Local rank-displacement analysis shows sensitivity concentrates on particular retrieval systems and system types, especially neural ranking/reranking systems on DL20 and RAG-oriented pipelines on RAG24. Persona source matters less than assessor role and model capacity. These findings position persona-conditioned judging as a controlled sensitivity probe for stress-testing LLM-based IR evaluation pipelines and identifying systems whose evaluation outcomes are sensitive to assessor framing.