背景:屈光手术研究旨在根据患者对各种类型手术的适用性对患者进行最佳预分类。最近的进步导致了人工智能驱动的算法的发展,包括机器学习方法,评估风险并增强工作流程。像ChatGPT-4(OpenAILP)这样的大型语言模型(LLM)已经成为潜在的通用人工智能工具,可以跨各个学科提供帮助。可能包括屈光手术决策。然而,他们根据真实世界参数对屈光手术患者进行预分类的实际能力仍未得到探索.
目的:这项探索性研究旨在验证ChatGPT-4根据常用临床参数对屈光手术患者进行预分类的能力。目的是评估ChatGPT-4在对批次输入进行分类时的性能是否与屈光外科医生的性能相当。比较了一组简单的二进制类别(是否适合激光屈光手术的患者)以及一组更详细的类别。
方法:对来自屈光诊所的100名连续患者的数据进行匿名化和分析。参数包括年龄,性别,明显的折射,视敏度,以及Scheimpflug成像的各种角膜测量和指数。本研究使用科恩κ系数将ChatGPT-4的表现与临床医生的分类进行了比较,卡方检验,一个混淆矩阵,准确度,精度,召回,F1分数,曲线下的接收机工作特征面积。
结果:在ChatGPT-4和临床医生的分类之间发现了具有统计学意义的非巧合,其中6个类别的Cohenκ系数为0.399(95%CI0.256-0.537),二元分类为0.610(95%CI0.372-0.792)。该模型显示了时间不稳定性和响应变异性,however.对6个类别的卡方检验表明2个评分者分布之间存在关联(χ²5=94.7,P<.001)。这里,准确率为0.68,准确率为0.75,召回率为0.68,F1评分为0.70.对于两个类别,准确率为0.88,精确度为0.88,召回率为0.88,F1评分为0.88,曲线下面积为0.79.
结论:这项研究表明,ChatGPT-4在屈光手术中具有作为分类前工具的潜力,与临床医生分类显示出有希望的一致性。然而,其主要局限性包括,其中,只依赖一个人类评估者,小样本量,ChatGPT(OpenAILP)输出在迭代之间的不稳定性和可变性以及底层模型的不透明度。结果鼓励进一步探索ChatGPT-4等LLM在医疗保健中的应用,特别是在需要了解大量临床数据的决策过程中。未来的研究应该集中在定义模型的准确性与提示和插图标准化,检测混杂因素,并与其他版本的ChatGPT-4和其他LLM进行比较,为更大规模的验证和现实世界的实现铺平道路。
BACKGROUND: Refractive surgery research aims to optimally precategorize patients by their suitability for various types of surgery. Recent advances have led to the development of artificial intelligence-powered algorithms, including machine learning approaches, to assess risks and enhance workflow. Large language models (LLMs) like ChatGPT-4 (OpenAI LP) have emerged as potential general artificial intelligence tools that can assist across various disciplines, possibly including refractive surgery decision-making. However, their actual capabilities in precategorizing refractive surgery patients based on real-world parameters remain unexplored.
OBJECTIVE: This exploratory
study aimed to validate ChatGPT-4\'s capabilities in precategorizing refractive surgery patients based on commonly used clinical parameters. The goal was to assess whether ChatGPT-4\'s performance when categorizing batch inputs is comparable to those made by a refractive surgeon. A simple binary set of categories (patient suitable for laser refractive surgery or not) as well as a more detailed set were compared.
METHODS: Data from 100 consecutive patients from a refractive clinic were anonymized and analyzed. Parameters included age, sex, manifest refraction, visual acuity, and various corneal measurements and indices from Scheimpflug imaging. This
study compared ChatGPT-4\'s performance with a clinician\'s categorizations using Cohen κ coefficient, a chi-square test, a confusion matrix, accuracy, precision, recall, F1-score, and receiver operating characteristic area under the curve.
RESULTS: A statistically significant noncoincidental accordance was found between ChatGPT-4 and the clinician\'s categorizations with a Cohen κ coefficient of 0.399 for 6 categories (95% CI 0.256-0.537) and 0.610 for binary categorization (95% CI 0.372-0.792). The model showed temporal instability and response variability, however. The chi-square test on 6 categories indicated an association between the 2 raters\' distributions (χ²5=94.7, P<.001). Here, the accuracy was 0.68, precision 0.75, recall 0.68, and F1-score 0.70. For 2 categories, the accuracy was 0.88, precision 0.88, recall 0.88, F1-score 0.88, and area under the curve 0.79.
CONCLUSIONS: This
study revealed that ChatGPT-4 exhibits potential as a precategorization tool in refractive surgery, showing promising agreement with clinician categorizations. However, its main limitations include, among others, dependency on solely one human rater, small sample size, the instability and variability of ChatGPT\'s (OpenAI LP) output between iterations and nontransparency of the underlying models. The results encourage further exploration into the application of LLMs like ChatGPT-4 in health care, particularly in decision-making processes that require understanding vast clinical data. Future research should focus on defining the model\'s accuracy with prompt and vignette standardization, detecting confounding factors, and comparing to other versions of ChatGPT-4 and other LLMs to pave the way for larger-scale validation and real-world implementation.