背景:大型语言模型(LLM)已经改变了医学的各个领域,协助复杂的任务和临床决策,与OpenAI的GPT-4,GPT-3.5,谷歌的Bard,和人类的克劳德其中最广泛使用。虽然GPT-4在一些研究中表现出优异的性能,这些模型之间的综合比较仍然有限。认识到国家医学考试委员会(NBME)考试在评估医学生的临床知识方面的重要性,这项研究旨在比较流行的LLM在NBME临床科目考试样本问题上的准确性。
方法:本研究中使用的问题是从NBME官方网站获得的多项选择题,并且可以公开获得。NBME医学科目考试的问题,儿科,妇产科,临床神经病学,门诊护理,家庭医学,精神病学,和手术用于查询每个LLM。来自GPT-4,GPT-3.5,克劳德,和吟游诗人在2023年10月被收集。将每个LLM的响应与NBME提供的答案进行比较,并检查准确性。使用单向方差分析(ANOVA)进行统计学分析。
结果:每个LLM总共查询了163个问题。GPT-4得分为163/163(100%),GPT-3.5得分为134/163(82.2%),巴德得分为123/163(75.5%),克劳德得分为138/163(84.7%)。GPT-4的总性能在统计学上优于GPT-3.5,克劳德,和巴德17.8%,15.3%,和24.5%,分别。GPT-3.5的总性能,克劳德,和巴德没有显著差异。GPT-4在特定受试者中的表现明显优于Bard,包括医学,儿科,家庭医学,和门诊护理,和GPT-3.5在门诊护理和家庭医学方面。在所有LLM中,手术检查平均分最高(18.25/20),而家庭医学考试的平均分最低(3.75/5)。结论:GPT-4在NBME临床科目考试样本问题上的卓越表现突显了其在医学教育和实践中的潜力。虽然LLM表现出承诺,在它们的应用中辨别是至关重要的,考虑到偶尔的不准确。随着技术的进步,必须定期重新评估和改进,以保持其在医学中的可靠性和相关性。
BACKGROUND: Large language models (LLMs) have transformed various domains in medicine, aiding in complex tasks and clinical decision-making, with OpenAI\'s GPT-4, GPT-3.5, Google\'s Bard, and Anthropic\'s
Claude among the most widely used. While GPT-4 has demonstrated superior performance in some studies, comprehensive comparisons among these models remain limited. Recognizing the significance of the National Board of Medical Examiners (NBME) exams in assessing the clinical knowledge of medical students, this study aims to compare the accuracy of popular LLMs on NBME clinical subject exam sample questions.
METHODS: The questions used in this study were multiple-choice questions obtained from the official NBME website and are publicly available. Questions from the NBME subject exams in medicine, pediatrics, obstetrics and gynecology, clinical neurology, ambulatory care, family medicine, psychiatry, and surgery were used to query each LLM. The responses from GPT-4, GPT-3.5,
Claude, and Bard were collected in October 2023. The response by each LLM was compared to the answer provided by the NBME and checked for accuracy. Statistical analysis was performed using one-way analysis of variance (ANOVA).
RESULTS: A total of 163 questions were queried by each LLM. GPT-4 scored 163/163 (100%), GPT-3.5 scored 134/163 (82.2%), Bard scored 123/163 (75.5%), and
Claude scored 138/163 (84.7%). The total performance of GPT-4 was statistically superior to that of GPT-3.5,
Claude, and Bard by 17.8%, 15.3%, and 24.5%, respectively. The total performance of GPT-3.5,
Claude, and Bard was not significantly different. GPT-4 significantly outperformed Bard in specific subjects, including medicine, pediatrics, family medicine, and ambulatory care, and GPT-3.5 in ambulatory care and family medicine. Across all LLMs, the surgery exam had the highest average score (18.25/20), while the family medicine exam had the lowest average score (3.75/5). Conclusion: GPT-4\'s superior performance on NBME clinical subject exam sample questions underscores its potential in medical education and practice. While LLMs exhibit promise, discernment in their application is crucial, considering occasional inaccuracies. As technological advancements continue, regular reassessments and refinements are imperative to maintain their reliability and relevance in medicine.