■我们找到了LLM,传统的机器学习,深度学习在认知衰退识别方面有不同的错误特征,和LLM的合奏,机器学习,和深度学习实现了最先进的表现。
■早期发现老年人认知能力下降有助于临床试验登记和及时的医疗干预。本研究旨在应用,评估,并比较先进的自然语言处理技术,以识别临床笔记中认知能力下降的迹象。
■这项研究,在弥撒将军布莱根(MGB)进行,波士顿,MA,包括2019年诊断为轻度认知障碍(MCI)前4年≥50岁患者的临床记录。关于认知衰退的注释部分是手动标记的。使用与认知功能相关的关键字过滤的4,949个注释部分的随机样本用于传统AI模型开发,和200个随机子集用于LLM和提示开发;另一个1996年没有关键字过滤的笔记部分的随机样本用于测试。大型语言模型(LLM)的提示模板,亚马逊网络服务上的Llama2和微软Azure上的GPT-4,采用多种提示方法开发,以选择最佳的基于LLM的方法。使用XGBoost和基于分层注意力的深度神经网络模型进行基线比较。然后使用多数票构建了三个模型的集合。
■GPT-4表现出优于Llama2的准确性和效率。集成模型优于单个模型,达到90.3%的精度,召回94.2%,F1评分为92.2%。值得注意的是,与表现最好的单一模型相比,集成模型的精度显着提高(从70%-79%到90%以上)。误差分析显示,至少有一个模型错误地预测了63个样本;然而,只有2例(3.2%)是所有模型的相互错误,表明它们之间存在不同的误差曲线。
■我们的研究结果表明,LLM和传统模型表现出不同的误差曲线。在EHR数据上的LLM和本地训练的机器学习模型的集合被发现是互补的,提高性能和提高诊断的准确性。
UNASSIGNED: Large language models (LLMs) have shown promising performance in various healthcare domains, but their effectiveness in identifying specific clinical conditions in real medical records is less explored. This study evaluates LLMs for detecting signs of cognitive decline in real electronic health record (EHR) clinical notes, comparing their error profiles with traditional models. The insights gained will inform strategies for performance enhancement.
UNASSIGNED: This study, conducted at Mass General Brigham in Boston, MA, analyzed clinical notes from the four years prior to a 2019 diagnosis of mild cognitive impairment in patients aged 50 and older. We used a randomly annotated sample of 4,949 note sections, filtered with keywords related to cognitive functions, for model development. For testing, a random annotated sample of 1,996 note sections without keyword filtering was utilized. We developed prompts for two LLMs, Llama 2 and GPT-4, on HIPAA-compliant cloud-computing platforms using multiple approaches (e.g., both hard and soft prompting and error analysis-based instructions) to select the optimal LLM-based method. Baseline models included a hierarchical attention-based neural network and XGBoost. Subsequently, we constructed an ensemble of the three models using a majority vote approach.
UNASSIGNED: GPT-4 demonstrated superior accuracy and efficiency compared to Llama 2, but did not outperform traditional models. The ensemble model outperformed the individual models, achieving a precision of 90.3%, a recall of 94.2%, and an F1-score of 92.2%. Notably, the ensemble model showed a significant improvement in precision, increasing from a range of 70%-79% to above 90%, compared to the best-performing single model. Error analysis revealed that 63 samples were incorrectly predicted by at least one model; however, only 2 cases (3.2%) were mutual errors across all models, indicating diverse error profiles among them.
UNASSIGNED: LLMs and traditional machine learning models trained using local EHR data exhibited diverse error profiles. The ensemble of these models was found to be complementary, enhancing diagnostic performance. Future research should investigate integrating LLMs with smaller, localized models and incorporating medical data and domain knowledge to enhance performance on specific tasks.