背景:ChatGPT,基于大规模语言模型的人工智能(AI),引起了人们对医疗保健领域的兴趣。尽管如此,人工智能在文本理解和生成方面的能力受到特定语言可用训练数据的质量和数量的限制,不同语言的AI性能需要进一步调查。虽然人工智能在医学上拥有巨大的潜力,必须应对挑战,例如制定临床护理标准;促进医学教育和实践中的文化转变;以及管理包括数据隐私在内的道德问题,同意,和偏见。
目的:该研究旨在评估ChatGPT在处理中国临床医学研究生考试中的表现,评估其临床推理能力,调查汉语的潜在局限性,并探索其作为中国医疗专业人员的宝贵工具的潜力。
方法:使用中国研究生临床医学考试问题数据集来评估ChatGPT(3.5版)的中文医学知识的有效性,其中有165个医学问题的数据集,分为三类:(1)常见问题(n=90)评估基本医学知识,(2)病例分析问题(n=45),侧重于通过患者病例评估进行临床决策,(3)多项选择题(n=30),要求选择多个正确答案。首先,我们评估了ChatGPT是否可以达到政府机构定义的严格的截止分数,这需要在前20%的候选人中表现。此外,在我们对ChatGPT在原始和编码医学问题上的表现进行评估时,使用了3个主要指标:准确性,一致性(验证答案),和洞察力的频率。
结果:我们的评估显示,ChatGPT在中文原始问题的300分中得分为153.5分,这表示设定的最低分数,以确保通过的候选人至少比入学配额多20%。然而,ChatGPT回答开放式医疗问题的准确性较低,只有31.5%的总精度。常见问题的准确性,多选题,案例分析问题占42%,37%,17%,分别。ChatGPT在所有问题中实现了90%的一致性。在正确的回答中,一致性是100%,显著超过不正确的反应(n=57,50%;P<.001)。ChatGPT为所有问题的80%(n=132)提供了创新见解,每个准确响应平均有2.95个见解。
结论:尽管ChatGPT超过了中国临床医学研究生考试的及格门槛,其在回答开放式医学问题方面的表现欠佳.尽管如此,ChatGPT表现出高度的内部一致性,并且能够在中文中产生多种见解。未来的研究应该调查医疗保健背景下ChatGPT表现中基于语言的差异。
BACKGROUND: ChatGPT, an artificial intelligence (AI) based on large-scale language models, has sparked interest in the field of health care. Nonetheless, the capabilities of AI in text comprehension and generation are constrained by the quality and volume of available training data for a specific language, and the performance of AI across different languages requires further investigation. While AI harbors substantial potential in medicine, it is imperative to tackle challenges such as the formulation of clinical care standards; facilitating cultural transitions in medical education and practice; and managing ethical issues including data privacy, consent, and bias.
OBJECTIVE: The study aimed to evaluate ChatGPT\'s performance in processing Chinese Postgraduate Examination for Clinical Medicine questions, assess its clinical reasoning ability, investigate potential limitations with the Chinese language, and explore its potential as a valuable tool for medical professionals in the Chinese context.
METHODS: A data set of Chinese Postgraduate Examination for Clinical Medicine questions was used to assess the effectiveness of ChatGPT\'s (version 3.5) medical knowledge in the Chinese language, which has a data set of 165 medical questions that were divided into three categories: (1) common questions (n=90) assessing basic medical knowledge, (2) case analysis questions (n=45) focusing on clinical decision-making through patient case evaluations, and (3) multichoice questions (n=30) requiring the selection of multiple correct answers. First of all, we assessed whether ChatGPT could meet the stringent cutoff score defined by the government agency, which requires a performance within the top 20% of candidates. Additionally, in our evaluation of ChatGPT\'s performance on both original and encoded medical questions, 3 primary indicators were used: accuracy, concordance (which validates the answer), and the frequency of insights.
RESULTS: Our evaluation revealed that ChatGPT scored 153.5 out of 300 for original questions in Chinese, which signifies the minimum score set to ensure that at least 20% more candidates pass than the enrollment quota. However, ChatGPT had low accuracy in answering open-ended medical questions, with only 31.5% total accuracy. The accuracy for common questions, multichoice questions, and case analysis questions was 42%, 37%, and 17%, respectively. ChatGPT achieved a 90% concordance across all questions. Among correct responses, the concordance was 100%, significantly exceeding that of incorrect responses (n=57, 50%; P<.001). ChatGPT provided innovative insights for 80% (n=132) of all questions, with an average of 2.95 insights per accurate response.
CONCLUSIONS: Although ChatGPT surpassed the passing threshold for the Chinese Postgraduate Examination for Clinical Medicine, its performance in answering open-ended medical questions was suboptimal. Nonetheless, ChatGPT exhibited high internal concordance and the ability to generate multiple insights in the Chinese language. Future research should investigate the language-based discrepancies in ChatGPT\'s performance within the health care context.