背景:ChatGPT的快速发展引起了极大的兴趣,并在公共和学术领域引起了广泛的讨论,特别是在医学教育的背景下。
目的:本研究旨在通过与三年级医学生的对比分析,评估ChatGPT在肺病学检查中的表现。
方法:在这项横断面研究中,我们对2个不同的组进行了比较分析.第一组包括244名三年级医学生,他们以前参加过我们机构2020年的肺科检查,这是用法语进行的。第二组将ChatGPT-3.5分为两组对话:没有语境化(V1)和语境化(V2)。在V1和V2中,ChatGPT收到了对学生的相同问题集。
结果:V1在放射学方面表现出非凡的熟练程度,微生物学,和胸外科,超过了这些领域的大多数医学生。然而,它面临着病理学的挑战,药理学,和临床肺炎。相比之下,V2在各种问题类别中始终提供更准确的回答,无论专业化。与医学生相比,ChatGPT在多项选择题中表现欠佳。V2擅长回答结构化的开放式问题。两次ChatGPT谈话,特别是V2,在解决低难度和中等难度的问题方面优于学生。有趣的是,学生在面对极具挑战性的问题时表现出更高的熟练程度。V1未能通过考试。相反,V2成功取得考试成功,表现优于139名(62.1%)医学生。
结论:虽然ChatGPT可以访问基于Web的全面数据集,它的表现与普通医学生的表现非常相似。结果受问题格式的影响,项目复杂性,和上下文细微差别。该模型在需要信息综合的医疗环境中面临挑战,先进的分析能力,和临床判断,以及在非英语语言评估中以及面对主流互联网来源之外的数据时。
BACKGROUND: The rapid evolution of ChatGPT has generated substantial interest and led to extensive discussions in both public and academic domains, particularly in the context of medical education.
OBJECTIVE: This study aimed to evaluate ChatGPT\'s performance in a pulmonology examination through a comparative analysis with that of third-year medical students.
METHODS: In this cross-sectional study, we conducted a comparative analysis with 2 distinct groups. The first group comprised 244 third-year medical students who had previously taken our institution\'s 2020 pulmonology examination, which was conducted in French. The second group involved ChatGPT-3.5 in 2 separate sets of conversations: without contextualization (V1) and with contextualization (V2). In both V1 and V2, ChatGPT received the same set of questions administered to the students.
RESULTS: V1 demonstrated exceptional proficiency in radiology, microbiology, and thoracic surgery, surpassing the majority of medical students in these domains. However, it faced challenges in pathology, pharmacology, and clinical pneumology. In contrast, V2 consistently delivered more accurate responses across various question categories, regardless of the specialization. ChatGPT exhibited suboptimal performance in multiple choice questions compared to medical students. V2 excelled in responding to structured open-ended questions. Both ChatGPT conversations, particularly V2, outperformed students in addressing questions of low and intermediate difficulty. Interestingly, students showcased enhanced proficiency when confronted with highly challenging questions. V1 fell short of passing the examination. Conversely, V2 successfully achieved examination success, outperforming 139 (62.1%) medical students.
CONCLUSIONS: While ChatGPT has access to a comprehensive web-based data set, its performance closely mirrors that of an average medical student. Outcomes are influenced by question format, item complexity, and contextual nuances. The model faces challenges in medical contexts requiring information synthesis, advanced analytical
aptitude, and clinical judgment, as well as in non-English language assessments and when confronted with data outside mainstream internet sources.