背景:ChatGPT(OpenAI)在临床诊断中显示出巨大的潜力,并可能成为临床实践中出色的辅助工具。本研究通过比较GPT-3.5和GPT-4.0在模型迭代中的性能来调查和评估ChatGPT的诊断能力。
目的:本研究旨在评估GPT-3.5和GPT-4.0对结肠癌的精确诊断能力及其作为外科医生辅助诊断工具的潜力,并比较GTP-3.5和GPT-4.0的诊断准确率。我们精确评估主要和次要诊断的准确性,并根据7个类别分析GPT-3.5和GPT-4.0中误诊的原因:患者病史,症状,身体体征,实验室检查,影像学检查,病理检查,和术中发现。
方法:我们从中华医学会出版社数据库检索了316例肠癌病例报告,其中286例数据清理后被认为是有效的。将案例从普通话翻译成英语,然后使用简单的方法输入GPT-3.5和GPT-4.0,直接提示引发原发性和继发性诊断。我们进行了一项比较研究,以评估GPT-4.0和GPT-3.5的诊断准确性。普外科的三位资深外科医生,专攻结直肠手术,在中国人民解放军(中国人民解放军)总医院评估诊断信息。主要和次要诊断的准确性根据预定标准进行评分。此外,我们根据7个类别分析和比较了两种模型中误诊的原因:患者病史,症状,身体体征,实验室检查,影像学检查,病理检查,和术中发现。
结果:在286例中,GPT-4.0和GPT-3.5对初级诊断均表现出较高的诊断准确性。但GPT-4.0的准确率显著高于GPT-3.5(平均值0.972,SD0.137vs平均值0.855,SD0.335;t285=5.753;P<.001)。对于二级诊断,GPT-4.0的准确率也显著高于GPT-3.5(平均值0.908,SD0.159vs平均值0.617,SD0.349;t285=-7.727;P<.001).GPT-3.5在处理患者病史方面显示出局限性,症状表现,实验室测试,和成像数据。尽管GPT-4.0比GPT-3.5有所改善,但在识别症状和实验室测试数据方面仍然存在局限性。对于主要和次要诊断,与年龄相关的准确性没有显着差异,性别,或GPT-4.0和GPT-3.5之间的系统组。
结论:这项研究表明,ChatGPT,特别是GPT-4.0,具有显著的诊断潜力,与GPT-4.0表现出更高的精度比GPT-3.5。然而,GPT-4.0仍然有局限性,特别是在识别患者症状和实验室数据方面,这表明需要在现实世界的临床环境中进行更多的研究,以增强其诊断能力。
BACKGROUND: ChatGPT (OpenAI) has shown great potential in clinical diagnosis and could become an excellent auxiliary tool in clinical practice. This study investigates and evaluates ChatGPT in diagnostic capabilities by comparing the performance of GPT-3.5 and GPT-4.0 across model iterations.
OBJECTIVE: This study aims to evaluate the precise diagnostic ability of GPT-3.5 and GPT-4.0 for colon cancer and its potential as an auxiliary diagnostic tool for surgeons and compare the diagnostic accuracy rates between GTP-3.5 and GPT-4.0. We precisely assess the accuracy of primary and secondary diagnoses and analyze the causes of misdiagnoses in GPT-3.5 and GPT-4.0 according to 7 categories: patient histories, symptoms, physical signs, laboratory examinations, imaging examinations, pathological examinations, and intraoperative findings.
METHODS: We retrieved 316 case reports for intestinal cancer from the Chinese Medical Association Publishing House database, of which 286 cases were deemed valid after data cleansing. The cases were translated from Mandarin to English and then input into GPT-3.5 and GPT-4.0 using a simple, direct prompt to elicit primary and secondary diagnoses. We conducted a comparative study to evaluate the diagnostic accuracy of GPT-4.0 and GPT-3.5. Three senior surgeons from the General Surgery Department, specializing in Colorectal Surgery, assessed the diagnostic information at the Chinese PLA (People\'s Liberation Army) General Hospital. The accuracy of primary and secondary diagnoses was scored based on predefined criteria. Additionally, we analyzed and compared the causes of misdiagnoses in both models according to 7 categories: patient histories, symptoms, physical signs, laboratory examinations, imaging examinations, pathological examinations, and intraoperative findings.
RESULTS: Out of 286 cases, GPT-4.0 and GPT-3.5 both demonstrated high diagnostic accuracy for primary diagnoses, but the accuracy rates of GPT-4.0 were significantly higher than GPT-3.5 (mean 0.972, SD 0.137 vs mean 0.855, SD 0.335; t285=5.753; P<.001). For secondary diagnoses, the accuracy rates of GPT-4.0 were also significantly higher than GPT-3.5 (mean 0.908, SD 0.159 vs mean 0.617, SD 0.349; t285=-7.727; P<.001). GPT-3.5 showed limitations in processing patient history, symptom presentation, laboratory tests, and imaging data. While GPT-4.0 improved upon GPT-3.5, it still has limitations in identifying symptoms and laboratory test data. For both primary and secondary diagnoses, there was no significant difference in accuracy related to age, gender, or system group between GPT-4.0 and GPT-3.5.
CONCLUSIONS: This study demonstrates that ChatGPT, particularly GPT-4.0, possesses significant diagnostic potential, with GPT-4.0 exhibiting higher accuracy than GPT-3.5. However, GPT-4.0 still has limitations, particularly in recognizing patient symptoms and laboratory data, indicating a need for more research in real-world clinical settings to enhance its diagnostic capabilities.