PDF(Pubmed)
Abstract：
BACKGROUND: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.
OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students\' performance in history taking with a simulated patient.
METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients\' responses and provide immediate feedback on the comprehensiveness of the students\' history taking. Students\' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
RESULTS: Most of the study\'s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4\'s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model\'s assessments were overly specific or diverged from human judgement.
CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.
OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students\' performance in history taking with a simulated patient.
METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients\' responses and provide immediate feedback on the comprehensiveness of the students\' history taking. Students\' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
RESULTS: Most of the study\'s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4\'s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model\'s assessments were overly specific or diverged from human judgement.
CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.
摘要：
背景：虽然病史是诊断疾病的基础，由于资源限制，教学和提供技能反馈可能具有挑战性。因此，虚拟模拟患者和基于网络的聊天机器人已经成为教育工具，随着人工智能(AI)的最新进展，如大型语言模型(LLM)，增强了它们的真实性和提供反馈的潜力。
目的：在我们的研究中，我们旨在评估生成预训练变压器(GPT)4模型的有效性,以对医学生在模拟患者的历史表现提供结构化反馈.
方法：我们进行了一项前瞻性研究，涉及医学生使用GPT驱动的聊天机器人进行历史学习。为此,我们设计了一个聊天机器人来模拟病人的反应，并提供对学生的全面性的即时反馈。分析了学生与聊天机器人的互动，并将聊天机器人的反馈与人类评估者的反馈进行了比较。我们测量了评估者间的可靠性，并进行了描述性分析以评估反馈的质量。
结果：研究的大多数参与者都在医学院三年级。我们的分析中总共包括了来自106个对话的1894个问答对。在超过99%的病例中，GPT-4的角色扮演和反应在医学上是合理的。GPT-4与人类评估者之间的评估者间可靠性显示出“几乎完美”的一致性（Cohenκ=0.832）。在45个反馈类别中的8个中，检测到的一致性较低（κ<0.6）突出了模型评估过于具体或与人类判断不同的主题。
结论：GPT模型在医学生提供的关于历史记录对话的结构化反馈方面是有效的。尽管我们揭示了某些反馈类别的反馈特异性的一些限制，与人类评估者的总体高度一致表明，LLM可以成为医学教育的宝贵工具。我们的发现,因此，倡导在医疗培训中仔细整合人工智能驱动的反馈机制，并在这种情况下使用LLM时突出重要方面。
目的：在我们的研究中，我们旨在评估生成预训练变压器(GPT)4模型的有效性,以对医学生在模拟患者的历史表现提供结构化反馈.
方法：我们进行了一项前瞻性研究，涉及医学生使用GPT驱动的聊天机器人进行历史学习。为此,我们设计了一个聊天机器人来模拟病人的反应，并提供对学生的全面性的即时反馈。分析了学生与聊天机器人的互动，并将聊天机器人的反馈与人类评估者的反馈进行了比较。我们测量了评估者间的可靠性，并进行了描述性分析以评估反馈的质量。
结果：研究的大多数参与者都在医学院三年级。我们的分析中总共包括了来自106个对话的1894个问答对。在超过99%的病例中，GPT-4的角色扮演和反应在医学上是合理的。GPT-4与人类评估者之间的评估者间可靠性显示出“几乎完美”的一致性（Cohenκ=0.832）。在45个反馈类别中的8个中，检测到的一致性较低（κ<0.6）突出了模型评估过于具体或与人类判断不同的主题。
结论：GPT模型在医学生提供的关于历史记录对话的结构化反馈方面是有效的。尽管我们揭示了某些反馈类别的反馈特异性的一些限制，与人类评估者的总体高度一致表明，LLM可以成为医学教育的宝贵工具。我们的发现,因此，倡导在医疗培训中仔细整合人工智能驱动的反馈机制，并在这种情况下使用LLM时突出重要方面。
