PDF(Pubmed)
Abstract：
BACKGROUND: Robust and accurate prediction of severity for patients with COVID-19 is crucial for patient triaging decisions. Many proposed models were prone to either high bias risk or low-to-moderate discrimination. Some also suffered from a lack of clinical interpretability and were developed based on early pandemic period data. Hence, there has been a compelling need for advancements in prediction models for better clinical applicability.
OBJECTIVE: The primary objective of this study was to develop and validate a machine learning-based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers.
METHODS: We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods.
RESULTS: Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910).
CONCLUSIONS: RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.
OBJECTIVE: The primary objective of this study was to develop and validate a machine learning-based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers.
METHODS: We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods.
RESULTS: Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910).
CONCLUSIONS: RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.
摘要：
背景：对COVID-19患者的严重程度进行稳健而准确的预测对于患者的分诊决策至关重要。许多提出的模型倾向于高偏差风险或低至中度歧视。有些还缺乏临床可解释性，并且是根据早期大流行时期的数据开发的。因此，为了更好的临床适用性,迫切需要改进预测模型.
目的：本研究的主要目的是开发和验证一种基于机器学习的健壮和可解释的早期分诊支持（RIETS）系统，该系统可预测严重程度进展（涉及以下任何事件：重症监护病房入院，在医院死亡,需要机械通风，或需要体外膜氧合）根据常规可用的临床和实验室生物标志物在住院后15天内。
方法：我们纳入了2020年1月至2022年8月收集的来自韩国19家医院的5945例COVID-19住院患者的数据。对于模型开发和外部验证，根据医院类型(普通和三级治疗)和地理位置(大城市和非大城市),通过分层随机整群抽样将整个数据集分为2个独立队列.机器学习模型通过开发队列的交叉验证技术进行了训练和内部验证。在外部验证队列上使用自举采样技术对它们进行了外部验证。主要根据受试者工作特征曲线下面积（AUROC）选择性能最佳的模型，并使用偏差风险评估来评估其稳健性。对于模型的可解释性，我们使用Shapley和患者聚类方法。
结果：我们的最终模型，RIETS,是基于11个临床和实验室生物标志物的深度神经网络开发的，这些生物标志物在住院的第一天内很容易获得。严重程度的预测特征包括乳酸脱氢酶，年龄,绝对淋巴细胞计数，呼吸困难,呼吸频率,糖尿病,c反应蛋白,中性粒细胞绝对计数,血小板计数,白细胞计数,和外周血氧饱和度。RIETS表现出优异的辨别力(AUROC=0.937;95%CI0.935-0.938)和高校准(积分校准指数=0.041),在风险评估工具中满足低偏差风险的所有标准，并提供了模型参数和患者聚类的详细解释。此外,RIETS对Omicron病例的可持续预测显示出跨变异期的可运输性潜力（AUROC=0.903，95％CI0.897-0.910）。
结论：开发并验证了RIETS，可通过及时预测COVID-19住院患者的严重程度来协助早期分类。其高性能和低偏差风险确保相当可靠的预测。在模型开发和验证中使用全国多中心队列意味着可泛化性。使用常规收集的特征可以实现广泛的适应性。模型参数和患者的解释可以促进临床适用性。一起,我们预计,当纳入常规临床实践时,RIETS将促进患者分诊工作流程和有效的资源分配.
目的：本研究的主要目的是开发和验证一种基于机器学习的健壮和可解释的早期分诊支持（RIETS）系统，该系统可预测严重程度进展（涉及以下任何事件：重症监护病房入院，在医院死亡,需要机械通风，或需要体外膜氧合）根据常规可用的临床和实验室生物标志物在住院后15天内。
方法：我们纳入了2020年1月至2022年8月收集的来自韩国19家医院的5945例COVID-19住院患者的数据。对于模型开发和外部验证，根据医院类型(普通和三级治疗)和地理位置(大城市和非大城市),通过分层随机整群抽样将整个数据集分为2个独立队列.机器学习模型通过开发队列的交叉验证技术进行了训练和内部验证。在外部验证队列上使用自举采样技术对它们进行了外部验证。主要根据受试者工作特征曲线下面积（AUROC）选择性能最佳的模型，并使用偏差风险评估来评估其稳健性。对于模型的可解释性，我们使用Shapley和患者聚类方法。
结果：我们的最终模型，RIETS,是基于11个临床和实验室生物标志物的深度神经网络开发的，这些生物标志物在住院的第一天内很容易获得。严重程度的预测特征包括乳酸脱氢酶，年龄,绝对淋巴细胞计数，呼吸困难,呼吸频率,糖尿病,c反应蛋白,中性粒细胞绝对计数,血小板计数,白细胞计数,和外周血氧饱和度。RIETS表现出优异的辨别力(AUROC=0.937;95%CI0.935-0.938)和高校准(积分校准指数=0.041),在风险评估工具中满足低偏差风险的所有标准，并提供了模型参数和患者聚类的详细解释。此外,RIETS对Omicron病例的可持续预测显示出跨变异期的可运输性潜力（AUROC=0.903，95％CI0.897-0.910）。
结论：开发并验证了RIETS，可通过及时预测COVID-19住院患者的严重程度来协助早期分类。其高性能和低偏差风险确保相当可靠的预测。在模型开发和验证中使用全国多中心队列意味着可泛化性。使用常规收集的特征可以实现广泛的适应性。模型参数和患者的解释可以促进临床适用性。一起,我们预计,当纳入常规临床实践时,RIETS将促进患者分诊工作流程和有效的资源分配.
