UNASSIGNED: The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE).
UNASSIGNED: The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt\'s designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model\'s accuracy and consistency.
UNASSIGNED: GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response.
UNASSIGNED: GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model\'s reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.
■本研究的目的是评估ChatGPT的临床表现,重点关注其在中国国家医师资格考试(CNMLE)中的准确性和可靠性。
■CNMLE2022问题集,由500个单答案多选题组成,被重新分类为15个医学亚专科。从2023年4月24日至5月15日,每个问题在OpenAI平台上用中文进行了8到12次测试。考虑了三个关键因素:GPT-3.5和4.0版本,针对医疗亚专科定制的系统角色的提示指定,为了连贯性而重复。通过准确度阈值被建立为60%。采用χ2检验和κ值评估模型的准确性和一致性。
■GPT-4.0达到了72.7%的通过精度,显著高于GPT-3.5(54%;P<.001)。GPT-4.0重复反应的变异性低于GPT-3.5(9%vs19.5%;P<.001)。然而,两个模型都显示出相对较好的响应一致性,κ值分别为0.778和0.610。系统角色在数值上提高了GPT-4.0(0.3%-3.7%)和GPT-3.5(1.3%-4.5%)的准确性,并将变异性降低了1.7%和1.8%,分别(P>0.05)。在亚组分析中,ChatGPT在不同题型之间取得了相当的准确率(P>.05)。GPT-4.0在15个亚专业中的14个超过了准确性阈值,而GPT-3.5在第一次反应的15人中有7人这样做。
■GPT-4.0通过了CNMLE,并在准确性等关键领域优于GPT-3.5,一致性,和医学专科专业知识。添加系统角色不会显着增强模型的可靠性和答案的连贯性。GPT-4.0在医学教育和临床实践中显示出有希望的潜力,值得进一步研究。