背景:开源人工智能模型(OSAIMs)越来越多地应用于各个领域,包括IT和医学,为诊断和治疗干预提供有希望的解决方案。为了响应对AI用于临床诊断的日益增长的兴趣,我们评估了几个OSAIM-比如ChatGPT4,微软Copilot,双子座,PopAi,你聊天,克劳德,以及专门的PMC-LLaMA13B评估其分类脊柱侧凸严重程度的能力,并根据APX射线照片的放射学描述推荐治疗方法。方法:我们的研究采用两阶段方法,其中单曲线脊柱侧凸的描述在由两名独立的神经外科医生进行评估后通过AI模型进行分析.统计分析涉及正态的Shapiro-Wilk检验,使用中位数和四分位数范围描述的非正态分布。使用Fleiss\'kappa评估评估者间的可靠性,和性能指标,比如准确性,灵敏度,特异性,和F1得分,用于评估人工智能系统的分类准确性。结果:分析表明,尽管一些人工智能系统,比如ChatGPT4副驾驶,还有PopAi,准确反映了推荐的Cobb角范围,用于疾病的严重程度和治疗,其他人,比如双子座和克劳德,需要进一步校准。特别是,PMC-LLaMA13B扩大了中度脊柱侧凸的分类范围,潜在影响临床决策和延迟干预措施。结论:这些发现强调了需要不断改进AI模型以增强其临床适用性。
Background: Open-source artificial intelligence models (OSAIMs) are increasingly being applied in various fields, including IT and medicine, offering promising solutions for diagnostic and therapeutic interventions. In response to the growing interest in AI for clinical diagnostics, we evaluated several OSAIMs-such as ChatGPT 4, Microsoft Copilot, Gemini, PopAi, You Chat, Claude, and the specialized PMC-LLaMA 13B-assessing their abilities to classify scoliosis severity and recommend treatments based on radiological descriptions from AP radiographs. Methods: Our study employed a two-stage methodology, where descriptions of single-curve scoliosis were analyzed by AI models following their evaluation by two independent neurosurgeons. Statistical analysis involved the Shapiro-Wilk test for normality, with non-normal distributions described using medians and interquartile ranges. Inter-rater reliability was assessed using Fleiss\' kappa, and performance metrics, like accuracy, sensitivity, specificity, and F1 scores, were used to evaluate the AI systems\' classification accuracy. Results: The analysis indicated that although some AI systems, like ChatGPT 4, Copilot, and PopAi, accurately reflected the recommended Cobb angle ranges for disease severity and treatment, others, such as Gemini and Claude, required further calibration. Particularly, PMC-LLaMA 13B expanded the classification range for moderate scoliosis, potentially influencing clinical decisions and delaying interventions. Conclusions: These findings highlight the need for the continuous refinement of AI models to enhance their clinical applicability.