OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students\' performance in history taking with a simulated patient.
METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients\' responses and provide immediate feedback on the comprehensiveness of the students\' history taking. Students\' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
RESULTS: Most of the study\'s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4\'s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model\'s assessments were overly specific or diverged from human judgement.
CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.
目的:在我们的研究中,我们旨在评估生成预训练变压器(GPT)4模型的有效性,以对医学生在模拟患者的历史表现提供结构化反馈.
方法:我们进行了一项前瞻性研究,涉及医学生使用GPT驱动的聊天机器人进行历史学习。为此,我们设计了一个聊天机器人来模拟病人的反应,并提供对学生的全面性的即时反馈。分析了学生与聊天机器人的互动,并将聊天机器人的反馈与人类评估者的反馈进行了比较。我们测量了评估者间的可靠性,并进行了描述性分析以评估反馈的质量。
结果:研究的大多数参与者都在医学院三年级。我们的分析中总共包括了来自106个对话的1894个问答对。在超过99%的病例中,GPT-4的角色扮演和反应在医学上是合理的。GPT-4与人类评估者之间的评估者间可靠性显示出“几乎完美”的一致性(Cohenκ=0.832)。在45个反馈类别中的8个中,检测到的一致性较低(κ<0.6)突出了模型评估过于具体或与人类判断不同的主题。
结论:GPT模型在医学生提供的关于历史记录对话的结构化反馈方面是有效的。尽管我们揭示了某些反馈类别的反馈特异性的一些限制,与人类评估者的总体高度一致表明,LLM可以成为医学教育的宝贵工具。我们的发现,因此,倡导在医疗培训中仔细整合人工智能驱动的反馈机制,并在这种情况下使用LLM时突出重要方面。