Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. This work introduces PERSPECTIVEDRIVEN INFERENCE, a method that treats the distribution of annotations across groups as the quantity of interest and estimates it using a small human annotation budget through an adaptive sampling strategy.

The gist

This framework formalizes multi-perspective corpus inference as estimation of a vector estimand (one entry per demographic group) rather than a single scalar, and develops an adaptive sampling strategy that concentrates human annotation effort on groups where LLM annotations are predicted to be less reliable, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines while maintaining coverage.

Background and Motivation

Annotators’ background shows considerable heterogeneity in preferences on subjective tasks, and collapsing annotator disagreement into a majority vote discards variation that matters for understanding how people use and perceive language. While methods like persona prompting exist to simulate group perspectives, they do not always improve LLM performance; they can exaggerate group differences or misrepresent minority viewpoints. The goal of this research is not to train LLMs to make more accurate predictions, but to leverage their annotations to conduct robust downstream statistical inference that faithfully retains annotator disagreement.

Framework and Problem Setup

The problem is set on a corpus of texts where each instance is annotated by multiple annotators, each from a particular demographic group. The objective is to estimate a vector of group-specific quantities, denoted as estimate Θ∗ = (Θ∗g1,..., Θ∗gk), where each θ∗gk equals the expected annotation value within demographic group gk. This estimation is performed using LLM annotations (Hˆi) as a cheap but potentially inaccurate proxy for human judgments, defined by the LLM error as (Hˆi − Hi)2.

Perspective-Driven Inference Mechanism

The core of PERSPECTIVEDRIVEN INFERENCE involves an adaptive loop:

1. Collect initial human annotations via burn-in uniform sampling.

2. Enter an adaptive loop where the human annotation probability πi is set proportional to the predicted LLM squared error from annotator demographics, specifically: πi ∝ err di(di), normalized so that Pn i=1 πi = nhuman.

3. The function err di is approximated by training a black-box classifier (XGBoost) on feature–label pairs of (dj,(Hˆj−Hj)2) to predict the squared LLM error from the annotator’s demographic profile.

Evaluation and Key Findings

The framework was evaluated on politeness and offensiveness rating tasks using GPT-5.2 across eight different LLMs and three prompting strategies (zero-shot, few-shot, persona). The key findings include:

1. PDI maintains coverage above 90% across all three age groups for both politeness and offensiveness, unlike LLM-only methods which often fall below 90%.

2. PDI achieves the lowest delta for Age 50+ in both tasks (e.g., 11.23% delta for politeness), outperforming PPI (13.63%) and all three LLM-only variants, demonstrating that error-driven upsampling is useful.

3. The adaptive allocation confirms the utility of the error signal: PDI allocates a 33% increase in annotations to Age 50+ compared to uniform sampling (PPI).

4. Persona prompting is found to be unreliable, sometimes shifting predictions systematically in the wrong direction, and often fails precisely for the groups it is meant to represent.

Limitations and Ethical Considerations

The researchers note several limitations:

1. The stratification by gender, age, and education are independent but correlate in practice; modeling intersectional subgroups would require larger budgets.

2. The error predictor relies only on demographic features, potentially leaving performance on the table if richer textual signals were included.

3. Adaptive upsampling concentrates effort where LLMs perform worst, which often means marginalized groups, raising concerns about the labor borne by these annotators and requiring explicit compensation tradeoffs for participants.

4. The framework is a proof of concept, and its results should be interpreted in light of how demographic attributes were collected, as they are coarse proxies for perspective.

Conclusion

PERSPECTIVEDRIVEN INFERENCE demonstrates that correcting LLM annotation error can be important when the target estimand is a distribution over perspectives rather than a single ground truth. It shows that PDI can be viewed as a targeted subgroup-improvement method (rather than a universally superior alternative to uniform PPI) by directing human effort where LLM performance disparities are large and groups are skewed, while maintaining validity and coverage. This approach is task-agnostic and applicable to settings like RLHF or LLM-as-judge evaluation.

Here are specific, high-impact improvements to AI systems based on the PERSPECTIVEDRIVEN INFERENCE (PDI) framework described in the paper:

1. Improve Subjective Task Annotation Reliability for Demographic Subgroups

2. System Capability: Targeted Annotation Budget Allocation

3. The improved system can dynamically allocate a limited human annotation budget across different demographic groups based on the predicted reliability of existing Large Language Model (LLM) annotations for that specific group.

4. Mechanism: Perspective-Driven Inference (PDI)

5. The system utilizes an adaptive sampling strategy. It trains a black-box error predictor (e.g., using XGBoost) on annotator demographic features to estimate the squared LLM error, specifically targeting groups where the LLM proxy is least accurate.

6. System Capability: Precision Improvement for Hard-to-Model Groups

7. The system concentrates expensive human annotation effort on underrepresented or poorly modeled perspectives (e.g., older age strata like Age 50+ in politeness ratings, as shown in the results), leading to significantly improved accuracy (lower Delta) for those groups compared to uniform sampling baselines.

8. System Capability: Valid Downstream Inference

9. The system produces group-level estimates of the quantity of interest (e.g., mean politeness rating per demographic group, rather than a single aggregate), allowing for statistically valid inference even when no single ground truth exists.

10. System Capability: Robust Confidence Intervals

11. The system provides provably valid confidence intervals for these group-level estimates using Inverse Probability Weighting (IPW) correction and bootstrap methods, ensuring the reported uncertainty reflects the true distribution of perspectives.

12. System Capability: Task Agnosticism

13. The framework is task-agnostic and can be applied to any subjective annotation task where annotator identity shapes labels (e.g., politeness, offensiveness, theory of mind modeling), provided a demographic axis for stratification can be defined.

14. System Capability: Detecting LLM Bias vs. Sampling Variance

15. The framework explicitly distinguishes between error driven by poor model performance (which PDI corrects) and variance caused by insufficient human data, allowing researchers to diagnose the root cause of annotation inaccuracy.

16. System Capability: Enhanced Model Selection Guidance

17. By analyzing the performance across multiple LLM variants (zero-shot vs. few-shot vs. persona prompting), the system can guide developers toward prompt engineering strategies (e.g., favoring few-shot examples over persona conditioning for certain tasks) that yield better initial proxies.

18. System Capability: Identifying Systemic Disagreement

19. The framework, when combined with disagreement analysis tools (like Fightin' Words or LLooM), can identify high-level thematic disagreements between demographic groups, revealing exactly what aspects of text cause annotator divergence across perspectives.

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.