PDF(Pubmed)
Abstract：
BACKGROUND: Large language model (LLM)-based chatbots are evolving at an unprecedented pace with the release of ChatGPT, specifically GPT-3.5, and its successor, GPT-4. Their capabilities in general-purpose tasks and language generation have advanced to the point of performing excellently on various educational examination benchmarks, including medical knowledge tests. Comparing the performance of these 2 LLM models to that of Family Medicine residents on a multiple-choice medical knowledge test can provide insights into their potential as medical education tools.
OBJECTIVE: This study aimed to quantitatively and qualitatively compare the performance of GPT-3.5, GPT-4, and Family Medicine residents in a multiple-choice medical knowledge test appropriate for the level of a Family Medicine resident.
METHODS: An official University of Toronto Department of Family and Community Medicine Progress Test consisting of multiple-choice questions was inputted into GPT-3.5 and GPT-4. The artificial intelligence chatbot\'s responses were manually reviewed to determine the selected answer, response length, response time, provision of a rationale for the outputted response, and the root cause of all incorrect responses (classified into arithmetic, logical, and information errors). The performance of the artificial intelligence chatbots were compared against a cohort of Family Medicine residents who concurrently attempted the test.
RESULTS: GPT-4 performed significantly better compared to GPT-3.5 (difference 25.0%, 95% CI 16.3%-32.8%; McNemar test: P<.001); it correctly answered 89/108 (82.4%) questions, while GPT-3.5 answered 62/108 (57.4%) questions correctly. Further, GPT-4 scored higher across all 11 categories of Family Medicine knowledge. In 86.1% (n=93) of the responses, GPT-4 provided a rationale for why other multiple-choice options were not chosen compared to the 16.7% (n=18) achieved by GPT-3.5. Qualitatively, for both GPT-3.5 and GPT-4 responses, logical errors were the most common, while arithmetic errors were the least common. The average performance of Family Medicine residents was 56.9% (95% CI 56.2%-57.6%). The performance of GPT-3.5 was similar to that of the average Family Medicine resident (P=.16), while the performance of GPT-4 exceeded that of the top-performing Family Medicine resident (P<.001).
CONCLUSIONS: GPT-4 significantly outperforms both GPT-3.5 and Family Medicine residents on a multiple-choice medical knowledge test designed for Family Medicine residents. GPT-4 provides a logical rationale for its response choice, ruling out other answer choices efficiently and with concise justification. Its high degree of accuracy and advanced reasoning capabilities facilitate its potential applications in medical education, including the creation of exam questions and scenarios as well as serving as a resource for medical knowledge or information on community services.
OBJECTIVE: This study aimed to quantitatively and qualitatively compare the performance of GPT-3.5, GPT-4, and Family Medicine residents in a multiple-choice medical knowledge test appropriate for the level of a Family Medicine resident.
METHODS: An official University of Toronto Department of Family and Community Medicine Progress Test consisting of multiple-choice questions was inputted into GPT-3.5 and GPT-4. The artificial intelligence chatbot\'s responses were manually reviewed to determine the selected answer, response length, response time, provision of a rationale for the outputted response, and the root cause of all incorrect responses (classified into arithmetic, logical, and information errors). The performance of the artificial intelligence chatbots were compared against a cohort of Family Medicine residents who concurrently attempted the test.
RESULTS: GPT-4 performed significantly better compared to GPT-3.5 (difference 25.0%, 95% CI 16.3%-32.8%; McNemar test: P<.001); it correctly answered 89/108 (82.4%) questions, while GPT-3.5 answered 62/108 (57.4%) questions correctly. Further, GPT-4 scored higher across all 11 categories of Family Medicine knowledge. In 86.1% (n=93) of the responses, GPT-4 provided a rationale for why other multiple-choice options were not chosen compared to the 16.7% (n=18) achieved by GPT-3.5. Qualitatively, for both GPT-3.5 and GPT-4 responses, logical errors were the most common, while arithmetic errors were the least common. The average performance of Family Medicine residents was 56.9% (95% CI 56.2%-57.6%). The performance of GPT-3.5 was similar to that of the average Family Medicine resident (P=.16), while the performance of GPT-4 exceeded that of the top-performing Family Medicine resident (P<.001).
CONCLUSIONS: GPT-4 significantly outperforms both GPT-3.5 and Family Medicine residents on a multiple-choice medical knowledge test designed for Family Medicine residents. GPT-4 provides a logical rationale for its response choice, ruling out other answer choices efficiently and with concise justification. Its high degree of accuracy and advanced reasoning capabilities facilitate its potential applications in medical education, including the creation of exam questions and scenarios as well as serving as a resource for medical knowledge or information on community services.
摘要：
背景：随着ChatGPT的发布，基于大型语言模型（LLM）的聊天机器人正以前所未有的速度发展，特别是GPT-3.5及其后继者，GPT-4.他们在通用任务和语言生成方面的能力已经发展到在各种教育考试基准上表现出色的地步，包括医学知识测试。将这2个LLM模型的性能与家庭医学居民在多项选择医学知识测试中的性能进行比较，可以洞悉他们作为医学教育工具的潜力。
目的：本研究旨在定量和定性地比较GPT-3.5，GPT-4和家庭医学居民在适合家庭医学居民水平的多项选择医学知识测试中的表现。
方法：由多项选择题组成的多伦多大学官方家庭和社区医学系进步测试被输入GPT-3.5和GPT-4。人工智能聊天机器人的回答被手动审查，以确定选择的答案，响应长度,响应时间,提供输出响应的理由，以及所有不正确响应的根本原因(分类为算术，合乎逻辑,和信息错误)。将人工智能聊天机器人的性能与同时尝试测试的一群家庭医学居民进行了比较。
结果：GPT-4的表现明显优于GPT-3.5（差异25.0％，95％CI16.3％-32.8％；McNemar测试：P<.001）；它正确回答了89/108（82.4％）问题，GPT-3.5正确回答了62/108（57.4%）的问题。Further,GPT-4在所有11个家庭医学知识类别中得分较高。在86.1%(n=93)的回答中，与GPT-3.5实现的16.7%(n=18)相比,GPT-4提供了为什么没有选择其他多项选择选项的理由。定性,对于GPT-3.5和GPT-4响应，逻辑错误是最常见的，而算术错误是最不常见的。家庭医学居民的平均表现为56.9％（95％CI为56.2％-57.6％）。GPT-3.5的表现与普通家庭医学居民的表现相似(P=0.16)，而GPT-4的性能超过了表现最好的家庭医学住院医师（P<.001）。
结论：GPT-4在为家庭医学居民设计的多项选择医学知识测试中显著优于GPT-3.5和家庭医学居民。GPT-4为其响应选择提供了逻辑原理，有效地排除其他答案选择，并有简洁的理由。其高度的准确性和先进的推理能力促进了其在医学教育中的潜在应用，包括创建考试问题和方案，以及作为医疗知识或社区服务信息的资源。
目的：本研究旨在定量和定性地比较GPT-3.5，GPT-4和家庭医学居民在适合家庭医学居民水平的多项选择医学知识测试中的表现。
方法：由多项选择题组成的多伦多大学官方家庭和社区医学系进步测试被输入GPT-3.5和GPT-4。人工智能聊天机器人的回答被手动审查，以确定选择的答案，响应长度,响应时间,提供输出响应的理由，以及所有不正确响应的根本原因(分类为算术，合乎逻辑,和信息错误)。将人工智能聊天机器人的性能与同时尝试测试的一群家庭医学居民进行了比较。
结果：GPT-4的表现明显优于GPT-3.5（差异25.0％，95％CI16.3％-32.8％；McNemar测试：P<.001）；它正确回答了89/108（82.4％）问题，GPT-3.5正确回答了62/108（57.4%）的问题。Further,GPT-4在所有11个家庭医学知识类别中得分较高。在86.1%(n=93)的回答中，与GPT-3.5实现的16.7%(n=18)相比,GPT-4提供了为什么没有选择其他多项选择选项的理由。定性,对于GPT-3.5和GPT-4响应，逻辑错误是最常见的，而算术错误是最不常见的。家庭医学居民的平均表现为56.9％（95％CI为56.2％-57.6％）。GPT-3.5的表现与普通家庭医学居民的表现相似(P=0.16)，而GPT-4的性能超过了表现最好的家庭医学住院医师（P<.001）。
结论：GPT-4在为家庭医学居民设计的多项选择医学知识测试中显著优于GPT-3.5和家庭医学居民。GPT-4为其响应选择提供了逻辑原理，有效地排除其他答案选择，并有简洁的理由。其高度的准确性和先进的推理能力促进了其在医学教育中的潜在应用，包括创建考试问题和方案，以及作为医疗知识或社区服务信息的资源。
