Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires
cs.CL
Submitted: 2025-10-29
Updated: 2026-08-28
Comments: Accepted to the 5th Workshop on NLP for Positive Impact 2026 @EMNLP 2026, Budapest, Hungary
Project page: https://ai-mh.github.io/SQPsych.html
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health.
Terminology
Abstract
Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced previous work to rely mainly on generic information. We present a comprehensive corpus of synthetic therapist-client conversations generated through LLMs. We construct our generation pipeline, SQPsych (Structured Questionnaire-based Psychotherapy), which uses real structured client profiles and psychological questionnaires without leaking any sensitive data. We fine-tune various open-weight LLMs on our generated corpus, SQPsychConv, and test them through both automatic benchmarks and human evaluation with trained psychotherapists. We find that standard benchmarks do not adequately capture the strengths of our dataset, but expert judgment shows that SQPsych makes LLMs significantly better at therapist roleplaying. Experts also consistently prefer therapy sessions generated by our models compared to other mental-health-oriented LLMs. We release our code, fine-tuned models SQPsychLLM, and corpora at https://ai-mh.github.io/SQPsych.html.
Sources
- CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering
- ChatCounselor: A Large Language Models for Mental Health Support
- Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals
- The Llama 3 Herd of Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering