HumanStudy-Bench: Towards AI Agent Design for Participant Simulation

📊 See Leaderboard & Results | 📖 Read the Paper

LLMs are increasingly used to simulate human participants in social science research, but existing evaluations conflate base model capabilities with agent design choices, making it unclear whether results reflect the model or the configuration.

HumanStudy-Bench treats participant simulation as an agent design problem and provides a standardized testbed — combining an Execution Engine that reconstructs full experimental protocols from published studies and a Benchmark with standardized evaluation metrics — for replaying human-subject experiments end-to-end with alignment evaluation at the level of scientific inference.

git clone https://github.com/<your-github-id>/HumanStudy-Bench.git
cd HumanStudy-Bench
git checkout -b contrib-<yourgithubid>-013

Add a new directory under studies/ with the required folders:

studies/<yourgithubid>_013/
  ├── index.json
  ├── source/
  ├── scripts/
  └── README.md

See the docs below for what goes inside each folder and the exact schemas:

bash scripts/verify_study.sh <yourgithubid>_013

git add studies/<yourgithubid>_013/
git commit -m "Add study: <Your Study Title>"
git push origin contrib-<yourgithubid>-013

Open a PR on GitHub targeting the dev branch. Maintainers assign final study_XXX numbering by merge order. CI runs validation automatically; confirmation is by human review.

You can also submit a study via web upload at hs-bench.clawder.ai/contribute.

The 12 foundational studies (cognition, strategic interaction, social psychology) serve as reference examples. Browse them on the website or locally under studies/.

If you use HumanStudy-Bench, please cite:

@misc{liu2026humanstudybenchaiagentdesign,
      title={HumanStudy-Bench: Towards AI Agent Design for Participant Simulation},
      author={Xuan Liu and Haoyang Shang and Zizhang Liu and Xinyan Liu and Yunze Xiao and Yiwen Tu and Haojian Jin},
      year={2026},
      eprint={2602.00685},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.00685},
}

Hugging Face: Benchmark and resources are available on the Hugging Face Hub — fuyyckwhy/HS-Bench-results.

To test and reproduce the exact benchmark and results reported in our paper, check out the arXiv-benchmark-version:

git checkout v1.0.0

MIT License. See LICENSE for details.