背景:GPT-4的可靠性,GPT-4是专门研究临床推理和医学知识的最先进的扩展语言模型,在非英语语言中仍未得到验证。
目的:本研究旨在通过使用普通医学培训考试(GM-ITE)比较日本居民和GPT-4之间的基本临床能力。
方法:我们使用OpenAI提供的GPT-4模型和2020年,2021年和2022年的GM-ITE考试试题进行了比较分析。这项分析的重点是评估与GPT-4相比,即将结束居住第二年的个人的表现。鉴于GPT-4目前的能力,我们的研究只包括单项选择题,不包括那些涉及音频的,视频,或图像数据。评估包括4个类别:一般理论(专业和医学面试),症状学和临床推理,体检和临床程序,具体的疾病。此外,我们将问题分为7个专业领域和3个难度级别,这是根据居民的正确反应率确定的。
结果:在检查了137个日语GM-ITE问题后,GPT-4得分显著高于居民平均得分(居民:55.8%,GPT-4:70.1%;P<.001)。就具体学科而言,GPT-4在“特定疾病”中得分高23.5分,“妇产科高出30.9分,“和26.1点在内科。“相比之下,GPT-4在“医学面试和专业性”方面的得分,\"\"一般做法,“和”精神病学“低于居民,尽管这种差异没有统计学意义.在根据问题难度分析分数后,GPT-4分数对于容易的问题(P=.007)低17.2分,但对于正常和困难的问题高25.4分和24.4分,分别(P<.001)。在同比比较中,在2020年(P=0.01)和2022年(P=0.003)考试中,GPT-4得分分别高出21.7和21.5分,分别,但在2021年的考试中仅高出3.5分(没有显着差异)。
结论:在日语中,GPT-4在GM-ITE测试中的表现也优于普通医疗居民,最初是为他们设计的。具体来说,GPT-4在居民正确应答率低的难题和要求对疾病有更全面了解的难题上表现出得分较高的趋势。然而,GPT-4在居民可以轻易回答的问题上得分相对较低,比如测试对病人的态度和专业精神,以及那些需要理解上下文和沟通的人。这些发现凸显了人工智能在医学教育和实践中的优势和局限性。
BACKGROUND: The reliability of GPT-4, a state-of-the-art expansive language model specializing in clinical reasoning and medical knowledge, remains largely unverified across non-English languages.
OBJECTIVE: This study aims to compare fundamental clinical competencies between Japanese residents and GPT-4 by using the General Medicine In-Training Examination (GM-ITE).
METHODS: We used the GPT-4 model provided by OpenAI and the GM-ITE examination questions for the years 2020, 2021, and 2022 to conduct a comparative analysis. This analysis focused on evaluating the performance of individuals who were concluding their second year of residency in comparison to that of GPT-4. Given the current abilities of GPT-4, our study included only single-choice exam questions, excluding those involving audio, video, or image data. The assessment included 4 categories: general theory (professionalism and medical interviewing), symptomatology and clinical reasoning, physical examinations and clinical procedures, and specific diseases. Additionally, we categorized the questions into 7 specialty fields and 3 levels of difficulty, which were determined based on residents\' correct response rates.
RESULTS: Upon examination of 137 GM-ITE questions in Japanese, GPT-4 scores were significantly higher than the mean scores of residents (residents: 55.8%, GPT-4: 70.1%; P<.001). In terms of specific disciplines, GPT-4 scored 23.5 points higher in the \"specific diseases,\" 30.9 points higher in \"obstetrics and gynecology,\" and 26.1 points higher in \"internal medicine.\" In contrast, GPT-4 scores in \"medical interviewing and professionalism,\" \"general practice,\" and \"psychiatry\" were lower than those of the residents, although this discrepancy was not statistically significant. Upon analyzing scores based on question difficulty, GPT-4 scores were 17.2 points lower for easy problems (P=.007) but were 25.4 and 24.4 points higher for normal and difficult problems, respectively (P<.001). In year-on-year comparisons, GPT-4 scores were 21.7 and 21.5 points higher in the 2020 (P=.01) and 2022 (P=.003) examinations, respectively, but only 3.5 points higher in the 2021 examinations (no significant difference).
CONCLUSIONS: In the Japanese language, GPT-4 also outperformed the average medical residents in the GM-ITE test, originally designed for them. Specifically, GPT-4 demonstrated a tendency to score higher on difficult questions with low resident correct response rates and those demanding a more comprehensive understanding of diseases. However, GPT-4 scored comparatively lower on questions that residents could readily answer, such as those testing attitudes toward patients and professionalism, as well as those necessitating an understanding of context and communication. These findings highlight the strengths and limitations of artificial intelligence applications in medical education and practice.