Automated Survey Collection with LLM-based Conversational Agents

summary

Video file (mp4)

The gist

The gist The study introduces an end-to-end survey collection framework driven by conversational Large Language Models (LLMs) to overcome the limitations of traditional, costly, and labor-intensive

In short

The study developed an end-to-end framework using conversational Large Language Models (LLMs) to automate phone surveys for biomedical research, addressing the high cost and labor intensity of traditional methods. An LLM agent calls participants, and a second LLM analyzes the resulting transcripts to accurately extract survey responses with 98% accuracy, demonstrating feasibility for scalable AI-powered data collection.

Key concepts

Conversational Phone Agent
This is an AI system powered by an LLM designed to conduct phone surveys. It calls participants and interacts with them conversationally, simulating a human interviewer. Its goal is to efficiently administer the survey while managing the interaction in real-time.
GPT-4o Analysis LLM
This second, powerful LLM is responsible for analyzing the text transcripts generated during the survey calls. It takes the raw conversation data and deduces or extracts specific answers to each survey question, achieving high accuracy even when the initial transcript has errors.
End-to-End Framework
This refers to a complete system pipeline that handles every step of survey collection, from designing the survey and recruiting participants to the AI agent calling them, analyzing the resulting audio transcripts using an LLM, and finally storing the organized results in a database.
Word Error Rate (WER)
WER is a metric used to measure how accurately an AI agent transcribes spoken audio into written text. It calculates errors based on words inserted, deleted, or substituted relative to the correct reference transcript. A lower WER indicates a more accurate transcription.

Terminology used across episodes

This episode discusses

The paper

Automated Survey Collection with LLM-based Conversational Agents · Read on arXiv

Kurmanbek Kaiyrbekov, Nicholas J Dobbins, Sean D Mooney

National Human Genome Research Institute · Johns Hopkins University

Objective: Traditional phone-based surveys are among the most accessible and widely used methods to collect biomedical and healthcare data, however, they are often costly, labor intensive, and difficult to scale effectively. To overcome these limitations, we propose an end-to-end survey collection framework driven by conversational Large Language Models (LLMs). Materials and Methods: Our framework consists of a researcher responsible for designing the survey and recruiting participants, a conversational phone agent powered by an LLM that calls participants and administers the survey, a second LLM (GPT-4o) that analyzes the conversation transcripts generated during the surveys, and a database for storing and organizing the results. To test our framework, we recruited 8 participants consisting of 5 native and 3 non-native english speakers and administered 40 surveys. We evaluated the correctness of LLM-generated conversation transcripts, accuracy of survey responses inferred by GPT-4o and overall participant experience. Results: Survey responses were successfully extracted by GPT-4o from conversation transcripts with an average accuracy of 98% despite transcripts exhibiting an average per-line word error rate of 7.7%. While participants noted occasional errors made by the conversational LLM agent, they reported that the agent effectively conveyed the purpose of the survey, demonstrated good comprehension, and maintained an engaging interaction. Conclusions: Our study highlights the potential of LLM agents in conducting and analyzing phone surveys for healthcare applications. By reducing the workload on human interviewers and offering a scalable solution, this approach paves the way for real-world, end-to-end AI-powered phone survey collection systems.

DOI: 10.1093/jamiaopen/ooaf103

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Automated Survey Collection with LLM-based Conversational Agents".

Tom: The gist The study introduces an end-to-end survey collection framework driven by conversational Large Language Models (LLMs) to overcome the limitations of traditional, costly,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Let’s look at who wrote this stuff. The paper is called "Automated Survey Collection with LLM-based Conversational Agents," written by Kurmanbek Kaiyrbekov, Nicholas J Dobbins, and Sean D Mooney.

Jane: That’s a big team of folks from institutions like the National Human Genome Research Institute and Johns Hopkins University.

Lu: The authors are clearly deep into the intersection of cyberinfrastructure and AI platforms, which is where this research sits.

Meng: It shows they’re connecting the dots between how these models work in theory and what that actually means for practical data collection tasks.

Lalam: This approach suggests that we can use these LLMs to automate complex, high-volume data gathering tasks that currently rely heavily on human effort.

The paper's summary: Tom: The core of the paper is this framework they built for collecting biomedical and healthcare data using conversational Large Language Models to replace old phone surveys.

Jane: They tested it by using an existing survey format, the Covid Impact Survey, and recruited eight participants—five native English speakers and three non-native ones—to run forty surveys in total.

Lu: The researchers focused on testing three main things: first, how accurately the LLM phone agent could transcribe the spoken conversation into text.

Meng: Second, they looked at how well a second LLM, GPT-4o, could take those transcripts and deduce the correct answers to each survey question <ref:2504.02891#pg2,a second LLM, GPT-4o>.

Lalam: And third is checking the overall experience of the participant while they were going through this AI-driven collection process.

The paper's improvements: Tom: The paper details how they evaluated these components, measuring transcription accuracy using a Word Error Rate or WER formula, which is S plus D plus I over N.

Jane: And then they looked at GPT-4o’s performance in guessing the survey answers by repeating the inference process five times and picking the most common answer <ref:2504.02891#pg2>.

Lu: The results showed that even though transcripts had an average WER of seven point seven percent, GPT-4o achieved a response deduction accuracy of ninety-eight percent across those eight participants <ref:2504.02891#pg2>.

Meng: That’s a strong number for how well the system handles errors in the spoken dialogue, suggesting that transcription errors don't necessarily ruin the final answer deduction significantly.

Lalam: The study also included participant feedback, using a questionnaire adapted from the Chatbot Usability Questionnaire to gauge personality and error handling.

Conclusion: Tom: So what’s the big picture here? The paper shows that using AI agents for health-related phone surveys is actually highly effective and feasible in the real world.

Jane: It really points toward a way to reduce human interviewer bias because the AI acts as a more objective surveyor, which is important in research.

Lu: This framework has huge potential for scaling up data collection at a much lower cost than traditional methods, potentially reaching more diverse groups of participants.

Meng: The cost estimate they give is really interesting; they said the total costs to conduct the experiments were only about seventy-five cents per survey. That’s a massive efficiency gain for research teams.

Lalam: Overall, this study on "Automated Survey Collection with LLM-based Conversational Agents" shows that this AI-powered approach can significantly simplify operations and widen participation opportunities in biomedical research.

More episodes

← Home