PDF(Pubmed)
Abstract：
OBJECTIVE: To investigate and implement semiautomated screening for meta-analyses (MA) in urology under consideration of class imbalance.
METHODS: Machine learning algorithms were trained on data from three MA with detailed information of the screening process. Different methods to account for class imbalance (Sampling (up- and downsampling, weighting and cost-sensitive learning), thresholding) were implemented in different machine learning (ML) algorithms (Random Forest, Logistic Regression with Elastic Net Regularization, Support Vector Machines). Models were optimized for sensitivity. Besides metrics such as specificity, receiver operating curves, total missed studies, and work saved over sampling were calculated.
RESULTS: During training, models trained after downsampling achieved the best results consistently among all algorithms. Computing time ranged between 251 and 5834 s. However, when evaluated on the final test data set, the weighting approach performed best. In addition, thresholding helped to improve results as compared to the standard of 0.5. However, due to heterogeneity of results no clear recommendation can be made for a universal sample size. Misses of relevant studies were 0 for the optimized models except for one review.
CONCLUSIONS: It will be necessary to design a holistic methodology that implements the presented methods in a practical manner, but also takes into account other algorithms and the most sophisticated methods for text preprocessing. In addition, the different methods of a cost-sensitive learning approach can be the subject of further investigations.
METHODS: Machine learning algorithms were trained on data from three MA with detailed information of the screening process. Different methods to account for class imbalance (Sampling (up- and downsampling, weighting and cost-sensitive learning), thresholding) were implemented in different machine learning (ML) algorithms (Random Forest, Logistic Regression with Elastic Net Regularization, Support Vector Machines). Models were optimized for sensitivity. Besides metrics such as specificity, receiver operating curves, total missed studies, and work saved over sampling were calculated.
RESULTS: During training, models trained after downsampling achieved the best results consistently among all algorithms. Computing time ranged between 251 and 5834 s. However, when evaluated on the final test data set, the weighting approach performed best. In addition, thresholding helped to improve results as compared to the standard of 0.5. However, due to heterogeneity of results no clear recommendation can be made for a universal sample size. Misses of relevant studies were 0 for the optimized models except for one review.
CONCLUSIONS: It will be necessary to design a holistic methodology that implements the presented methods in a practical manner, but also takes into account other algorithms and the most sophisticated methods for text preprocessing. In addition, the different methods of a cost-sensitive learning approach can be the subject of further investigations.
摘要：
目的：在考虑类失衡的情况下，研究并实施泌尿外科荟萃分析（MA）的半自动筛查。
方法：对来自三个MA的数据以及筛选过程的详细信息进行了机器学习算法的训练。解释类不平衡的不同方法(采样(向上和向下采样，加权和成本敏感学习)，阈值处理)在不同的机器学习(ML)算法中实现(随机森林，基于弹性网络正则化的Logistic回归，支持向量机)。模型的灵敏度进行了优化。除了特异性等指标外，接收器工作曲线，总错过的研究，并计算了采样节省的工作量。
结果：在培训期间，在降采样后训练的模型在所有算法中始终获得最佳结果。计算时间介于251和5834s之间。当在最终测试数据集上进行评估时，加权方法表现最好。此外,与0.5的标准相比，阈值有助于改善结果。然而,由于结果的异质性,我们无法对通用样本量提出明确的建议.除一篇综述外，优化模型的相关研究缺失为0。
结论：有必要设计一种整体方法，以实用的方式实现所提出的方法，还考虑了其他算法和最复杂的文本预处理方法。此外,成本敏感学习方法的不同方法可以成为进一步研究的主题。
方法：对来自三个MA的数据以及筛选过程的详细信息进行了机器学习算法的训练。解释类不平衡的不同方法(采样(向上和向下采样，加权和成本敏感学习)，阈值处理)在不同的机器学习(ML)算法中实现(随机森林，基于弹性网络正则化的Logistic回归，支持向量机)。模型的灵敏度进行了优化。除了特异性等指标外，接收器工作曲线，总错过的研究，并计算了采样节省的工作量。
结果：在培训期间，在降采样后训练的模型在所有算法中始终获得最佳结果。计算时间介于251和5834s之间。当在最终测试数据集上进行评估时，加权方法表现最好。此外,与0.5的标准相比，阈值有助于改善结果。然而,由于结果的异质性,我们无法对通用样本量提出明确的建议.除一篇综述外，优化模型的相关研究缺失为0。
结论：有必要设计一种整体方法，以实用的方式实现所提出的方法，还考虑了其他算法和最复杂的文本预处理方法。此外,成本敏感学习方法的不同方法可以成为进一步研究的主题。
