背景：脓毒症是一种异质性综合征，纳入更同质的患者对于提高临床试验的效率至关重要。人工智能(AI)促进了同质子组的识别，但是如何在将AI应用于临床决策时估计模型输出的不确定性仍然未知。
目的：我们旨在设计一个基于AI的模型，用于有目的的患者登记，确保纳入试验的脓毒症患者在拟议治疗可能影响患者结局时仍将持续患病.我们还期望该模型可以提供可解释的因素，并以定制的置信水平估计模型输出的不确定性。
方法：在这项回顾性研究中，从BethIsraelDeaconess医疗中心招募9135例脓毒症患者，这些患者需要在脓毒症发病后24小时内进行血管加压药治疗。这个队列用于模型开发，和50重复的10倍交叉验证用于内部验证。总的来说,来自eICU协作研究数据库的3743例脓毒症患者被用作外部验证队列。所有纳入的脓毒症患者根据疾病进展轨迹进行分层：快速死亡，recovery,持续的病。总共选择了148个变量来预测3个轨迹。使用了具有3种不同设置的四种机器学习算法。我们使用共形预测(CP)估计模型输出的不确定性。采用Shapley加性解释方法对模型进行了解释。
结果：在两个验证队列中，多类梯度增强机被确定为性能最佳的模型，具有良好的辨别和校准性能。快速死亡的受试者工作特征曲线下的平均面积为SD为0.906（0.018），0.843(0.008)用于恢复，内部验证队列中的持续性疾病为0.807（0.010）。在外部验证队列中，快速死亡的受试者工作特征曲线下平均面积（SD）为0.878（0.003），0.764（0.008）的回收率，持续性疾病为0.696（0.007）。去甲肾上腺素的最大当量,总尿量,急性生理学评分III,平均收缩压,氧饱和度的变异系数贡献最大。与没有CP的模型相比，在内部和外部验证队列中，以混合置信度方法使用具有CP的模型将总体预测误差降低了27.6%(n=62)和30.7%(n=412)，分别,以及能够识别更多潜在的持续性疾病患者。
结论：实施我们的模型有可能减少异质性，并在脓毒症临床试验中招募更多同质患者。使用CP来估计模型输出的不确定性可以更全面地了解模型的可靠性，并有助于根据预测结果做出明智的决策。
BACKGROUND: Sepsis is a heterogeneous syndrome, and enrollment of more homogeneous patients is essential to improve the efficiency of clinical trials. Artificial intelligence (AI) has facilitated the identification of homogeneous subgroups, but how to estimate the uncertainty of the model outputs when applying AI to clinical decision-making remains unknown.
OBJECTIVE: We aimed to design an AI-based model for purposeful patient enrollment, ensuring that a patient with sepsis recruited into a trial would still be persistently ill by the time the proposed therapy could impact patient outcome. We also expected that the model could provide interpretable factors and estimate the uncertainty of the model outputs at a customized confidence level.
METHODS: In this retrospective study, 9135 patients with sepsis requiring vasopressor treatment within 24 hours after sepsis onset were enrolled from Beth Israel Deaconess Medical Center. This cohort was used for model development, and 10-fold cross-validation with 50 repeats was used for internal validation. In total, 3743 patients with sepsis from the eICU Collaborative Research Database were used as the external validation cohort. All included patients with sepsis were stratified based on disease progression trajectories: rapid death, recovery, and persistent ill. A total of 148 variables were selected for predicting the 3 trajectories. Four machine learning algorithms with 3 different setups were used. We estimated the uncertainty of the model outputs using conformal prediction (CP). The Shapley Additive Explanations method was used to explain the model.
RESULTS: The multiclass gradient boosting machine was identified as the best-performing model with good discrimination and calibration performance in both validation cohorts. The mean area under the receiver operating characteristic curve with SD was 0.906 (0.018) for rapid death, 0.843 (0.008) for recovery, and 0.807 (0.010) for persistent ill in the internal validation cohort. In the external validation cohort, the mean area under the receiver operating characteristic curve (SD) was 0.878 (0.003) for rapid death, 0.764 (0.008) for recovery, and 0.696 (0.007) for persistent ill. The maximum norepinephrine equivalence, total urine output, Acute Physiology Score III, mean systolic blood pressure, and the coefficient of variation of oxygen saturation contributed the most. Compared to the model without CP, using the model with CP at a mixed confidence approach reduced overall prediction errors by 27.6% (n=62) and 30.7% (n=412) in the internal and external validation cohorts, respectively, as well as enabled the identification of more potentially persistent ill patients.
CONCLUSIONS: The implementation of our model has the potential to reduce heterogeneity and enroll more homogeneous patients in sepsis clinical trials. The use of CP for estimating the uncertainty of the model outputs allows for a more comprehensive understanding of the model\'s reliability and assists in making informed decisions based on the predicted outcomes.
PDF(Pubmed)