有监督的机器学习(ML)在医疗保健文献中得到了体现,研究结果经常使用准确性等指标报告。灵敏度,特异性,召回,或F1得分。尽管每个指标都提供了不同的性能视角,它们仍然是整个样本的总体衡量标准,忽略每个病例或患者的独特性。直觉上,我们知道不是所有的情况都是平等的,但是目前的评估方法没有考虑案例难度。
更多基于案例的,全面的方法是必要的,以评估监督ML的结果,并构成本研究的理由。这项研究旨在证明如何使用项目反应理论(IRT)根据每个案例的分类难度对数据进行分层,独立于兴趣的结果度量(例如,准确性)。这种分层允许ML分类器的评估采用分布而不是单个标量值的形式。
两个大的,公共重症监护病房数据集,重症监护III和电子重症监护病房的医疗信息集市,被用来展示这种预测死亡率的方法。对于每个数据集,绘制了一个平衡样本(分别为n=8078和n=21,940)和一个不平衡样本(分别为n=12,117和n=32,910)。使用2参数逻辑模型为每种情况提供分数。演示中使用了几种ML算法,根据与健康相关的特征对病例进行分类:逻辑回归,线性判别分析,K-最近的邻居,决策树,天真的贝叶斯,和神经网络。广义线性混合模型分析用于评估案例难度层的影响,ML算法,以及它们之间在预测准确性方面的相互作用。
结果显示,对病例困难地层有显著影响(P<.001),ML算法,以及它们在预测准确性方面的相互作用,并说明所有分类器在更容易分类的情况下表现更好,并且总体上神经网络表现最好。重要的相互作用表明,落在最艰苦的阶层中的病例应该通过逻辑回归来处理,线性判别分析,决策树,或神经网络,但不是通过朴素贝叶斯或K近邻。已经报道了用于ML分类的常规度量用于方法比较。
此演示表明,使用IRT是理解提供给ML算法的数据的可行方法,独立于结果衡量标准,并强调分类器如何很好地区分不同难度的情况。该方法解释了哪些特征指示健康状态以及为什么。它使最终用户能够定制适合患者的难度级别的分类器,以进行个性化医疗。
Supervised machine learning (ML) is being featured in the health care literature with study results frequently reported using metrics such as accuracy, sensitivity, specificity, recall, or F1 score. Although each metric provides a different perspective on the performance, they remain to be overall measures for the whole sample, discounting the uniqueness of each
case or patient. Intuitively, we know that all cases are not equal, but the present evaluative approaches do not take
case difficulty into account.
A more
case-based, comprehensive approach is warranted to assess supervised ML outcomes and forms the rationale for this study. This study aims to demonstrate how the item response theory (IRT) can be used to stratify the data based on how difficult each case is to classify, independent of the outcome measure of interest (eg, accuracy). This stratification allows the evaluation of ML classifiers to take the form of a distribution rather than a single scalar value.
Two large, public intensive care unit data sets, Medical Information Mart for Intensive Care III and electronic intensive care unit, were used to showcase this method in predicting mortality. For each data set, a balanced sample (n=8078 and n=21,940, respectively) and an imbalanced sample (n=12,117 and n=32,910, respectively) were drawn. A 2-parameter logistic model was used to provide scores for each
case. Several ML algorithms were used in the demonstration to classify cases based on their health-related features: logistic regression, linear discriminant analysis, K-nearest neighbors, decision tree, naive Bayes, and a neural network. Generalized linear mixed model analyses were used to assess the effects of
case difficulty strata, ML algorithm, and the interaction between them in predicting accuracy.
The results showed significant effects (P<.001) for case difficulty strata, ML algorithm, and their interaction in predicting accuracy and illustrated that all classifiers performed better with easier-to-classify cases and that overall the neural network performed best. Significant interactions suggest that cases that fall in the most arduous strata should be handled by logistic regression, linear discriminant analysis, decision tree, or neural network but not by naive Bayes or K-nearest neighbors. Conventional metrics for ML classification have been reported for methodological comparison.
This demonstration shows that using the IRT is a viable method for understanding the data that are provided to ML algorithms, independent of outcome measures, and highlights how well classifiers differentiate cases of varying difficulty. This method explains which features are indicative of healthy states and why. It enables end users to tailor the classifier that is appropriate to the difficulty level of the patient for personalized medicine.