BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks

Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics, but there remains no systematic understanding of how well they perform across diverse behavioral science tasks, contexts, and populations. BehaviorBench introduces a comprehensive benchmark that evaluates foundation models along four core capabilities: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application.

BehaviorBench Overview

The paper introduces BehaviorBench as a comprehensive benchmark designed to systematically evaluate the performance of foundation models on behavioral science tasks. This benchmark is motivated by the observation that human behavior is jointly shaped by multiple factors: the behavioral context, the subject’s traits, and underlying motivations, formally represented as a conditional probability p(y x, c; K). To address limitations in existing benchmarks—which often focus narrowly on survey response prediction or treat subjects as independent data points—BehaviorBench evaluates models at both the individual and distributional levels. This dual evaluation captures not only persubject accuracy but also population-level alignment, which is deemed an essential requirement for behavioral validity.

Core Capabilities and Tasks

BehaviorBench comprises 12 distinct tasks spanning four core capabilities:

1. Behavior prediction and simulation: This includes tasks like Single-round game behavior simulation (Game Behav. Sim.) and Multi-round game behavior prediction (Multi-Round Pred.) using experimental data from MobLab economic games, as well as survey response prediction tasks like Survey response prediction given demographics (Demo. To Resp.).

2. Strategic decision-making: This involves tasks like Strategic game play, where the model must make decisions in interactive play against human players, such as the Beauty Contest game.

3. Subject-trait inference: This capability focuses on inverse inference over subject traits (x), including tasks such as Personality score prediction given demographics (Demo. To Pers.) and Age prediction given personality scores (Pers. To Demo.).

4. Behavioral knowledge application: This evaluates the ability to apply behavioral science knowledge (K) to research problems, covering tasks like Scientific workflow prediction and solving complex problems using multiple-choice questions from the International Economics Olympiad (IEO).

Evaluation Metrics

The benchmark employs distinct metrics for individual and distributional evaluation. For individual-level evaluation, metrics include Mean absolute error (MAE) for numeric quantities, Accuracy for categorical predictions, and Win rate for strategic decision-making. For distribution-level evaluation, the Wasserstein distance (W) is used to compare predicted behavior distributions against observed human distributions because it captures both the shape and the mean of two distributions.

Be.FM-1.5 Development

Motivated by these goals, the authors developed Be.FM-1.5, extending the original Be.FM family of behavioral foundation models specifically designed for behavioral science tasks. This model extends training by fine-tuning open-source LLMs on a substantially broader set of behavioral tasks, with explicit coverage of diverse capabilities, contexts, and populations. The results reveal that while general-purpose proprietary LLMs excel at individual-level prediction and knowledge-intensive tasks, behavioral foundation models achieve stronger distributional alignment on average.

Key Findings

The evaluation reveals an uneven strength across tasks: frontier proprietary LLMs are strongest on individual-level prediction and knowledge-intensive tasks, whereas behavioral foundation models are generally better at behavior simulation in economic games and distributional-level behavioral alignment. Notably, Be.FM-1.5 leads on distributional metrics while remaining competitive on individual-level metrics, suggesting that proper behavioral adaptation can close the gap. The results highlight the importance of distributional evaluation and establish BehaviorBench as a foundation for developing and assessing behaviorally aligned AI systems.

Model Comparison

The study benchmarks three types of models: open-source general-purpose LLMs, proprietary LLMs, and behavioral foundation models. Findings indicate that Be.FM-1.5 is fine-tuned on data held out from BehaviorBench, yet it leads on distributional metrics while remaining highly competitive on individual-level metrics, demonstrating the potential for simultaneous achievement of both objectives through behavioral adaptation. The work also addresses limitations by exploring prompting strategies to improve simulation and contextual reasoning, and analyzes model regression patterns, such as the drop in performance for Be.FM-1.5-4B on IEO tasks due to a loss of reasoning capability in smaller models.

Conclusion

BehaviorBench establishes a higher bar for behavioral foundation models by covering four capability categories and evaluating performance at both individual and population levels. The overall conclusion is that while frontier proprietary LLMs are strong in knowledge-intensive reasoning, behavioral foundation models fine-tuned on behavior-related data tend to perform more strongly on distributional alignment, emphasizing the importance of distributional evaluation for assessing whether AI systems reflect human population heterogeneity.

Here are specific improvements for AI systems based on the BehaviorBench framework and Be.FM-1.5 model, along with what those improved systems can achieve:

) Behavioral Foundation Models (Be.FM) for Next-Generation Human-Centric AI

The core improvement involves moving beyond general-purpose foundation models to behaviorally aligned foundation models by fine-tuning them specifically on behavioral data and evaluating them rigorously against a comprehensive benchmark.

1. Acknowledge and Address the Distributional Alignment Gap:

2. Develop Be.FM-1.5: A specialized behavioral foundation model fine-tuned on diverse behavioral data (experimental records, survey responses, literature) to achieve superior distributional alignment across populations compared to general models like GPT or Claude Opus 4.6.

3. Implement Multi-Level Evaluation: Integrate a two-tiered evaluation system for any new foundation model:

4. Behavioral Benchmarking: Utilize the BehaviorBench benchmark (covering prediction, decision-making, subject-trait inference, and knowledge application) to test performance at both individual and distributional levels simultaneously.

5. Enforce Distributional Metrics: Prioritize models that demonstrate strong distributional alignment (using Wasserstein distance 'W') over those that only show high individual accuracy. This ensures the AI system reflects the diversity of the human population it is intended to serve, rather than overfitting to a majority or average behavior.

6. Enable Population-Level Inference: Improve AI systems' ability to infer latent subject traits (e.g., personality scores) from observed behaviors, and conversely, infer population distributions of these traits from observed behaviors, allowing for better personalization and policy design.

7. Enhance Strategic Decision-Making: Equip AI agents with the capability to make strategic decisions in interactive environments (like economic games) against human players, moving beyond playing against other LLMs to simulating interaction with actual human decision-makers.

8. Integrate Contextual Reasoning: Develop AI systems that can infer underlying contextual factors (incentives, social norms, framing) from observed behaviors and generate plausible experimental interventions to test hypotheses about those factors.

9. Develop Adaptive Prompting Strategies: Implement dynamic prompting techniques—such as grounding prompts in rich subject context or modeling the prompt itself as a distribution—to allow models to adapt their behavior simulation across different population subgroups or contexts effectively.

) Capabilities of the Improved AI System

The resulting improved AI systems will be significantly more robust and scientifically valid for applications in behavioral science, policymaking, and personalized intervention: