Towards interactive evaluations for interaction harms in human-AI systems

arXiv:2405.10632 · cs.CY, cs.AI, cs.HC · Submitted 2024-05-17 · Read on arXiv

cs.CY, cs.AI, cs.HC

Submitted: 2024-05-17

Updated: 2026-09-14

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 7× in this paper · explained in 2 episodes
• Generative AI — 7× in this paper · explained in 2 episodes
• Generative Model — 4× in this paper · explained in 2 episodes
• Trace — 4× in this paper · explained in 2 episodes
• Accountability — 3× in this paper · explained in 2 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Generative Models — 3× in this paper · explained in 2 episodes
• Personalization — 2× in this paper · explained in 2 episodes
• Regret — 2× in this paper · explained in 3 episodes
• Actor — 1× in this paper · explained in 3 episodes
• Agent Architectures — 1× in this paper · explained in 1 episode
• Agentic AI — 1× in this paper · explained in 1 episode
• Agentic AI systems — 1× in this paper · explained in 1 episode
• Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Dynamic Adaptation — 1× in this paper · explained in 2 episodes

Sources

• The illusion of artificial inclusion
• Should agentic conversational AI change how we think about ethics? Characterising an interactional ethics centred on respect
• LLM Social Simulations Are a Promising Research Method
• Designing a Dashboard for Transparency and Control of Conversational AI
• Evaluating Language Models for Mathematics through Interactions
• On Measures of Biases and Harms in NLP
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• More than Marketing? On the Information Value of AI Benchmarks for Practitioners
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
• Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models
• Characterizing and modeling harms from interactions with design patterns in AI interfaces
• Implicit Personalization in Language Models: A Systematic Study
• Evaluating Human-Language Model Interaction
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Rethinking Model Evaluation as Narrowing the Socio-Technical Gap
• Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
• The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• Red Teaming Language Models with Language Models

Related papers

• Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus
• Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot
• PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks
• What is an intelligent system?
• AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study
• Generative AI Use in Entrepreneurship: An Integrative Review and an Empowerment-Entrapment Framework