背景:机器学习技术开始在各种医疗保健数据集中使用,以识别可能从干预中受益的体弱者。然而,与传统回归相比,关于机器学习技术性能的证据好坏参半。还不清楚哪些方法和数据库因素与性能相关。
目的:本研究旨在比较各种机器学习分类器在不同场景下识别体弱老年人的死亡率预测准确性。
方法:我们使用2012年1月1日至2016年12月31日在新西兰使用interRAI-HomeCare工具评估的老年人(65岁及以上)收集的去识别数据。总共使用138个InterRAI评估项目来预测6个月和12个月的死亡率。使用3个机器学习分类器(随机森林[RF],极端梯度增强[XGBoost],和多层感知器[MLP])和正则化逻辑回归。我们进行了一项模拟研究,比较了机器学习模型与逻辑回归和内部RAI家庭护理脆弱量表的性能,并检查了样本量的影响,功能的数量,和列车测试分流比。
结果:共有95,042名老年人(平均年龄82.66岁,IQR77.92-88.76;n=37,462,39.42%男性)接受家庭护理。曲线下平均面积(AUC)和6个月死亡率预测的敏感性表明,机器学习分类器的表现并不优于正则逻辑回归。就AUC而言,正则化逻辑回归的性能优于XGBoost,MLP,当特征数量≤80且样本量≤16,000时,和RF;当特征数量≥40且样本量≥4000时,MLP在灵敏度方面优于正则逻辑回归。相反,在所有情况下,RF和XGBoost均表现出比正则逻辑回归更高的特异性。
结论:研究表明,当使用不同的指标进行评估时,机器学习模型在预测性能方面表现出显著差异。正则逻辑回归是一种有效的模型,用于识别体弱的老年人接受家庭护理,如AUC所示,特别是当特征的数量和样本大小不太大时。相反,MLP显示出优越的灵敏度,而当特征数量和样本量大时,RF表现出优异的特异性。
BACKGROUND: Machine learning techniques are starting to be used in various health care data sets to identify frail persons who may benefit from interventions. However, evidence about the performance of machine learning techniques compared to conventional regression is mixed. It is also unclear what methodological and database factors are associated with performance.
OBJECTIVE: This study aimed to compare the mortality prediction accuracy of various machine learning classifiers for identifying frail older adults in different scenarios.
METHODS: We used deidentified data collected from older adults (65 years of age and older) assessed with interRAI-Home Care instrument in New Zealand between January 1, 2012, and December 31, 2016. A total of 138 interRAI assessment items were used to predict 6-month and 12-month mortality, using 3 machine learning classifiers (random forest [RF], extreme gradient boosting [XGBoost], and multilayer perceptron [MLP]) and regularized logistic regression. We conducted a simulation study comparing the performance of machine learning models with logistic regression and interRAI Home Care Frailty Scale and examined the effects of sample sizes, the number of
features, and train-test split ratios.
RESULTS: A total of 95,042 older adults (median age 82.66 years, IQR 77.92-88.76; n=37,462, 39.42% male) receiving home care were analyzed. The average area under the curve (AUC) and sensitivities of 6-month mortality prediction showed that machine learning classifiers did not outperform regularized logistic regressions. In terms of AUC, regularized logistic regression had better performance than XGBoost, MLP, and RF when the number of
features was ≤80 and the sample size ≤16,000; MLP outperformed regularized logistic regression in terms of sensitivities when the number of
features was ≥40 and the sample size ≥4000. Conversely, RF and XGBoost demonstrated higher specificities than regularized logistic regression in all scenarios.
CONCLUSIONS: The study revealed that machine learning models exhibited significant variation in prediction performance when evaluated using different metrics. Regularized logistic regression was an effective model for identifying frail older adults receiving home care, as indicated by the AUC, particularly when the number of
features and sample sizes were not excessively large. Conversely, MLP displayed superior sensitivity, while RF exhibited superior specificity when the number of
features and sample sizes were large.