背景:沟通是医疗专业人员的核心能力,对患者安全至关重要。虽然医学课程强调沟通训练,传统格式,例如真实或模拟的患者互动,可以表现出心理压力,并且在重复中受到限制。最近出现的大型语言模型(LLM),如生成式预训练变压器(GPT),提供了克服这些限制的机会。
目的:本研究的目的是探索GPT驱动的聊天机器人实践历史获取的可行性,沟通的核心能力之一。
方法:我们开发了一个交互式聊天机器人界面,使用GPT-3.5和一个特定的提示,包括聊天机器人优化的疾病脚本和行为组件。遵循混合方法方法,我们邀请医学生自愿练习历史。为了确定GPT是否作为模拟患者提供合适的答案,使用定量和定性方法记录和分析对话.我们分析了问题和答案与提供的脚本一致的程度,以及医学合理性的答案。最后,学生填写了Chatbot可用性问卷(CUQ)。
结果:共有28名学生与我们的聊天机器人一起练习(平均年龄23.4,SD2.9岁)。我们总共记录了826个问答对(QAP),每次对话的QAP中位数为27.5,与历史记录有关的QAP中位数为94.7%(n=782)。当脚本明确涵盖问题时(n=502,60.3%),GPT提供的答案主要基于明确的脚本信息(n=471,94.4%).对于脚本未涵盖的问题(n=195,23.4%),GPT答案使用了56.4%(n=110)的虚构信息。关于合理性,860个QAP中有842个(97.9%)被评为合理。在14个(2.1%)令人难以置信的答案中,GPT提供了被评为社会理想的答案,离开角色身份,忽略脚本信息,不合逻辑的推理,和计算错误。尽管有这些结果,CUQ显示出整体积极的用户体验(77/100分)。
结论:我们的数据显示,如GPT,可以提供模拟的患者体验,并产生良好的用户体验和大多数合理的答案。我们的分析显示,GPT提供的答案使用明确的脚本信息或基于可用信息,可以理解为归纳推理。虽然罕见,基于GPT的聊天机器人在某些情况下提供了不合理的信息,主要趋势是社会期望的,而不是医学上合理的信息。
BACKGROUND: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions.
OBJECTIVE: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication.
METHODS: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ).
RESULTS: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points).
CONCLUSIONS: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.