OBJECTIVE: This study aimed to quantitatively and qualitatively compare the performance of GPT-3.5, GPT-4, and Family Medicine residents in a multiple-choice medical knowledge test appropriate for the level of a Family Medicine resident.
METHODS: An official University of Toronto Department of Family and Community Medicine Progress Test consisting of multiple-choice questions was inputted into GPT-3.5 and GPT-4. The artificial intelligence chatbot\'s responses were manually reviewed to determine the selected answer, response length, response time, provision of a rationale for the outputted response, and the root cause of all incorrect responses (classified into arithmetic, logical, and information errors). The performance of the artificial intelligence chatbots were compared against a cohort of Family Medicine residents who concurrently attempted the test.
RESULTS: GPT-4 performed significantly better compared to GPT-3.5 (difference 25.0%, 95% CI 16.3%-32.8%; McNemar test: P<.001); it correctly answered 89/108 (82.4%) questions, while GPT-3.5 answered 62/108 (57.4%) questions correctly. Further, GPT-4 scored higher across all 11 categories of Family Medicine knowledge. In 86.1% (n=93) of the responses, GPT-4 provided a rationale for why other multiple-choice options were not chosen compared to the 16.7% (n=18) achieved by GPT-3.5. Qualitatively, for both GPT-3.5 and GPT-4 responses, logical errors were the most common, while arithmetic errors were the least common. The average performance of Family Medicine residents was 56.9% (95% CI 56.2%-57.6%). The performance of GPT-3.5 was similar to that of the average Family Medicine resident (P=.16), while the performance of GPT-4 exceeded that of the top-performing Family Medicine resident (P<.001).
CONCLUSIONS: GPT-4 significantly outperforms both GPT-3.5 and Family Medicine residents on a multiple-choice medical knowledge test designed for Family Medicine residents. GPT-4 provides a logical rationale for its response choice, ruling out other answer choices efficiently and with concise justification. Its high degree of accuracy and advanced reasoning capabilities facilitate its potential applications in medical education, including the creation of exam questions and scenarios as well as serving as a resource for medical knowledge or information on community services.
目的:本研究旨在定量和定性地比较GPT-3.5,GPT-4和家庭医学居民在适合家庭医学居民水平的多项选择医学知识测试中的表现。
方法:由多项选择题组成的多伦多大学官方家庭和社区医学系进步测试被输入GPT-3.5和GPT-4。人工智能聊天机器人的回答被手动审查,以确定选择的答案,响应长度,响应时间,提供输出响应的理由,以及所有不正确响应的根本原因(分类为算术,合乎逻辑,和信息错误)。将人工智能聊天机器人的性能与同时尝试测试的一群家庭医学居民进行了比较。
结果:GPT-4的表现明显优于GPT-3.5(差异25.0%,95%CI16.3%-32.8%;McNemar测试:P<.001);它正确回答了89/108(82.4%)问题,GPT-3.5正确回答了62/108(57.4%)的问题。Further,GPT-4在所有11个家庭医学知识类别中得分较高。在86.1%(n=93)的回答中,与GPT-3.5实现的16.7%(n=18)相比,GPT-4提供了为什么没有选择其他多项选择选项的理由。定性,对于GPT-3.5和GPT-4响应,逻辑错误是最常见的,而算术错误是最不常见的。家庭医学居民的平均表现为56.9%(95%CI为56.2%-57.6%)。GPT-3.5的表现与普通家庭医学居民的表现相似(P=0.16),而GPT-4的性能超过了表现最好的家庭医学住院医师(P<.001)。
结论:GPT-4在为家庭医学居民设计的多项选择医学知识测试中显著优于GPT-3.5和家庭医学居民。GPT-4为其响应选择提供了逻辑原理,有效地排除其他答案选择,并有简洁的理由。其高度的准确性和先进的推理能力促进了其在医学教育中的潜在应用,包括创建考试问题和方案,以及作为医疗知识或社区服务信息的资源。