背景:基于人工智能(AI)的工具可以重塑医疗保健实践。这包括ChatGPT,它被认为是最受欢迎的基于AI的会话模型之一。然而,不同版本的ChatGPT的性能需要在不同的环境下进一步评估,以评估其在各种医疗保健相关任务中的可靠性和可信度.因此,本研究旨在评估免费提供的ChatGPT-3.5和付费版本ChatGPT-4在10种不同诊断性临床微生物学病例中的表现.
方法:当前的研究遵循度量(模型,评价,时间/透明度,范围/随机化,个别因素,伯爵,提示/语言的特异性)检查表,用于医疗保健中基于AI的研究的设计和报告的标准化。2023年12月3日测试的模型包括ChatGPT-3.5和ChatGPT-4,对ChatGPT生成的内容的评估基于CLEAR工具(完整性,缺乏虚假信息,证据支持,适当性,和相关性)以5点Likert量表评估,CLEAR评分范围为1-5。ChatGPT输出由两名评估者独立评估,评估者之间的协议基于科恩的κ统计量。在对约旦观察到的常见病例进行内部讨论之后,由三位具有不同专业知识水平的微生物学家用英语创建了十个诊断临床微生物学实验室病例方案。主题范围包括细菌学,真菌学,寄生虫学,和病毒学病例。根据CLEAR工具定制了具体提示,并在提示每个案例场景后选择了新的会话。
结果:ChatGPT-3.5和ChatGPT-4的五个CLEAR项目的Cohenκ值分别为0.351-0.737和0.294-0.701,表明对分析的一致性和适用性。根据平均清晰分数,ChatGPT-4的表现优于ChatGPT-3.5(平均值:2.64±1.06vs.3.21±1.05,P=.012,t检验)。每个模型的性能根据CLEAR项目而变化,“相关性”项目的性能最低(ChatGPT-3.5为2.15±0.71,ChatGPT-4为2.65±1.16)。仅在ChatGPT-4中观察到在评估每个CLEAR项目的性能时存在统计学上的显着差异,在“完整性”中表现最佳,“缺乏虚假信息”,和“证据支持”(P=0.043)。在抗微生物药敏试验(AST)查询中观察到两种模型的最低性能水平,而在细菌和真菌学鉴定中观察到最高性能水平。
结论:在不同诊断临床微生物学病例中对ChatGPT表现的评估表明,ChatGPT-4优于ChatGPT-3.5。ChatGPT的表现表现出明显的变异性,具体取决于所评估的主题。两种ChatGPT模型的主要缺点是倾向于生成不相关的内容而缺乏所需的重点。尽管ChatGPT在这些诊断微生物学病例中的总体表现最好被描述为“高于平均水平”,仍有很大的改进潜力,考虑到在少数情况下确定的局限性和不令人满意的结果。
BACKGROUND: Artificial intelligence (AI)-based tools can reshape healthcare practice. This includes ChatGPT which is considered among the most popular AI-based conversational models. Nevertheless, the performance of different versions of ChatGPT needs further evaluation in different settings to assess its reliability and credibility in various healthcare-related tasks. Therefore, the current study aimed to assess the performance of the freely available ChatGPT-3.5 and the paid version ChatGPT-4 in 10 different diagnostic clinical microbiology case scenarios.
METHODS: The current study followed the METRICS (Model, Evaluation, Timing/Transparency, Range/Randomization, Individual factors, Count, Specificity of the prompts/language) checklist for standardization of the design and reporting of AI-based studies in healthcare. The models tested on December 3, 2023 included ChatGPT-3.5 and ChatGPT-4 and the evaluation of the ChatGPT-generated content was based on the CLEAR tool (Completeness, Lack of false information, Evidence support, Appropriateness, and Relevance) assessed on a 5-point Likert scale with a range of the CLEAR scores of 1-5. ChatGPT output was evaluated by two raters independently and the inter-rater agreement was based on the Cohen\'s κ statistic. Ten diagnostic clinical microbiology laboratory case scenarios were created in the English language by three microbiologists at diverse levels of expertise following an internal discussion of common cases observed in Jordan. The range of topics included bacteriology, mycology, parasitology, and virology cases. Specific prompts were tailored based on the CLEAR tool and a new session was selected following prompting each case scenario.
RESULTS: The Cohen\'s κ values for the five CLEAR items were 0.351-0.737 for ChatGPT-3.5 and 0.294-0.701 for ChatGPT-4 indicating fair to good agreement and suitability for analysis. Based on the average CLEAR scores, ChatGPT-4 outperformed ChatGPT-3.5 (mean: 2.64±1.06 vs. 3.21±1.05, P=.012, t-test). The performance of each model varied based on the CLEAR items, with the lowest performance for the \"Relevance\" item (2.15±0.71 for ChatGPT-3.5 and 2.65±1.16 for ChatGPT-4). A statistically significant difference upon assessing the performance per each CLEAR item was only seen in ChatGPT-4 with the best performance in \"Completeness\", \"Lack of false information\", and \"Evidence support\" (P=0.043). The lowest level of performance for both models was observed with antimicrobial susceptibility testing (AST) queries while the highest level of performance was seen in bacterial and mycologic identification.
CONCLUSIONS: Assessment of ChatGPT performance across different diagnostic clinical microbiology case scenarios showed that ChatGPT-4 outperformed ChatGPT-3.5. The performance of ChatGPT demonstrated noticeable variability depending on the specific topic evaluated. A primary shortcoming of both ChatGPT models was the tendency to generate irrelevant content lacking the needed focus. Although the overall ChatGPT performance in these diagnostic microbiology case scenarios might be described as \"above average\" at best, there remains a significant potential for improvement, considering the identified limitations and unsatisfactory results in a few cases.