AgentFairBench: Do LLM Agents Discriminate When They Act?
cs.AI
Submitted: 2026-06-15
Updated: 2026-09-05
Comments: 28 pages, 2 figures, 5 tables. Code, data, and released traces: https://github.com/rohithreddybc/AgentFairBench
Code: https://github.com/rohithreddybc/AgentFairBench
Project page: https://rohithreddybc.github.io/AgentFairBench
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- BBQ: A Hand-Built Bias Benchmark for Question Answering
- StereoSet: Measuring stereotypical bias in pretrained language models
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- AgentBench: Evaluating LLMs as Agents
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Mind2Web: Towards a Generalist Agent for the Web
- Language (Technology) is Power: A Critical Survey of "Bias" in NLP
- Bias and Fairness in Large Language Models: A Survey
- Evaluating and Mitigating Discrimination in Language Model Decisions
- What's in a Name? Auditing Large Language Models for Race and Gender Bias
- MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
- FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering
- mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Support
- EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage
- Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System
- Equality of Opportunity in Supervised Learning
- Counterfactual Fairness
- Inherent Trade-Offs in the Fair Determination of Risk Scores
- Fair prediction with disparate impact: A study of bias in recidivism prediction instruments
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection