Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics

The paper "Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics investigates the apparent gap in the acceptance of human research" within the AI Safety & Ethics (AISE) field. The authors note that while AISE's mission is to anticipate and prevent harm to humans, empirical human-centered research methodologies are often sidelined in favor of technical methods, such as benchmarks and AI simulations. To examine this, the researchers conducted an "expert survey (n = 93) and expert interviews (n = 17) with AI Safety & Ethics (AISE) researchers from Technical, Sociotechnical, Governance, and Normative backgrounds."

RQ1: Epistemic Value and Disciplinary Divides

The study explores the epistemic fit of human research by examining how experts perceive its ability to generate legitimate evidence. The findings indicate that "although there is a consensus that human research is valuable for generating evidence for AISE, its adoption and acceptance are constrained by perceived validity issues, tangible resource barriers, epistemic and personal preferences in methods, and infrastructural constraints."

A key finding is the aspiration-practice gap, where across all disciplines and dimensions, gap scores were consistently positive, indicating that human research is valued beyond what is currently practiced. Specifically, researchers aspire to conduct field studies more than lab studies, which signals a collective interest in ecological validity over experimental control.

The researchers also identified a Commensurability Gap, noting that disciplines differ in how they value human versus non-human methods. While experts generally agree that more evidence of AI’s impact on humans is needed, they hold varied opinions on which human methods generate valid evidence. This is driven by epistemic incompatibility and low familiarity with human subject methodologies. Notably, Technical researchers tend to value human research less and collaborate across disciplines less, suggesting an epistemic tension towards human methods. The study found that human methods are valued less when the scenarios have low risk and imminence, and by the Technical discipline in comparison to Sociotechnical.

RQ2: Barriers to Human Research

The paper identifies three levels of constraints that impede human research:

• Resource Limitations: The most direct constraints involve time, funding, and participant access. Lack of time was ranked most impactful, followed by funding, then participant access.

• Infrastructural Limitations: These include organizational arrangements, mentorship structures, and professional incentives. A significant barrier is mentorship, with interviews capturing accounts of advisors actively discouraging human research. Furthermore, career and publication pressures... can be acutely constraining, as the field often privileges formal, quantitative, and technical contributions.

• Sector-Level Constraints: In non-profits and industry, the absence of formal ethics review infrastructure impedes engagement with human subjects. Additionally, there are concerns regarding the influence of unregulated, commercially incentivized actors and the rigour and neutrality of industry-produced research.

Discussion and Recommendations

The authors propose several strategies to bridge these gaps:

• Epistemic Implications: They advocate for epistemological pluralism, suggesting that different ‘ways of knowing’ can be integrated to gain a fuller description of a field. They also call for Methodological Literacy for Researchers, particularly among Technical researchers, to improve their understanding of the established methodological literature of adjacent fields. Crucially, they warn against "human-washing, defined as the superficial inclusion of human subjects that creates the appearance of validity without the necessary methodological rigour."

• Resource and Infrastructure Implications: To overcome resource challenges, funding bodies... should explicitly account for the higher cost structures of human research. To address mentorship issues, departments and graduate programs should formalize pathways for interdisciplinary mentorship, such as co-supervision arrangements and funded rotations.

• Cross-Sector Collaboration: The authors suggest a model of integrated academic-industry partnerships and propose that non-profit organizations... act as neutral intermediaries, insulating research directives from commercial influence while preserving access to industry resources.

1. Sociotechnical Alignment Frameworks

• The Improvement: Transitioning from purely technical Reinforcement Learning from Human Feedback (RLHF) to Deep Sociotechnical Alignment that incorporates qualitative human research data.

• What the improved system can do: Instead of optimizing for surface-level preferences (e.g., politeness or helpfulness that may be human-washing), the AI will be trained on reward models derived from ethnographic studies, expert interviews, and longitudinal human-impact data. This allows the system to navigate complex social norms, recognize subtle power dynamics, and avoid harms that are invisible to quantitative, lab-based benchmarks.

2. Ecological Validity Benchmarking (EVB)

• The Improvement: Replacing static, lab-style technical benchmarks with Dynamic Field-Testing Modules that prioritize ecological validity.

• What the improved system can do: The system will be evaluated against unconstrained interaction scenarios that simulate real-world deployment. Rather than just solving math problems or coding tasks, the AI will be tested on its ability to maintain safety and ethical boundaries during unpredictable, high-stakes human interactions in diverse cultural and social contexts, mirroring the field studies researchers aspire to conduct.

3. Multidisciplinary Red-Teaming Protocols

• The Improvement: Integrating Epistemic Pluralism into the AI safety testing lifecycle by mandating combined technical and sociotechnical red-teaming.

• What the improved system can do: The system will undergo a dual-layer stress test: a technical layer (identifying prompt injections, model weights vulnerabilities, etc.) and a sociotechnical layer (identifying how those technical vulnerabilities could be exploited to cause psychological manipulation, social polarization, or systemic bias). This bridges the Commensurability Gap by forcing technical safety tools to account for human-centric risks.

4. Human-Impact Monitoring & Governance Dashboards

• The Improvement: Developing Sociotechnical Risk Indicators that serve as neutral, intermediary metrics for AI governance.

• What the improved system can do: The AI system will include a monitoring layer that tracks emergent social harms by aggregating anonymized, qualitative feedback from diverse user demographics. This provides developers and regulators with a Sociotechnical Risk Score that goes beyond error rates to show how the AI is actually impacting human behavior, social structures, and psychological well-being in real-time.

Safety risks of AI are becoming increasingly evident in human interactions with AI technologies. The prominent approaches to evaluating these risks favor technical methods, such as model benchmarks and LLM simulations, often sidelining empirical research with human subjects. To examine this apparent gap in the acceptance of human research, we conduct an expert survey (n=93) and expert interviews (n=17) with AI Safety & Ethics (AISE) researchers from Technical, Sociotechnical, Governance, and Normative backgrounds. Our findings suggest that although there is a consensus that human research is valuable for generating evidence for AISE, its adoption and acceptance are constrained by perceived validity issues, tangible resource barriers, epistemic and personal preferences in methods, and infrastructural constraints from the broader research community. In particular, Technical researchers tend to value human research less and collaborate across disciplines less, suggesting an epistemic tension towards human methods. We propose recommendations for establishing the epistemic fit of human research within AISE and bridging the prohibitive limitations that researchers face, while avoiding performative 'human-washing'.