目的:改善报告和可比性,并减少牙科计算机视觉研究中的偏差,我们的目标是为这一领域制定一套核心成果衡量标准(COMS)。COMS是在世卫组织/国际电联/知识产权组织全球健康人工智能倡议(世卫组织/国际电联/知识产权组织AI4H)的基础上达成共识的。
方法:我们首先评估了诊断准确性研究的现有指导文件,并与该领域专家进行了访谈。结果结果测量列表被映射到计算机视觉建模任务,临床领域和报告水平。由此产生的系统化侧重于提供相关的成果措施,同时保留元研究和技术复制的细节,显示针对(1)不同临床领域和任务的报告级别的建议,(2)结果测量。COMS是使用2阶段的e-Delphi同意的,来自不同IADR小组的26名参与者,世卫组织/国际电联/WIPOAI4H,ADEA和AAOMFR。
结果:我们为不同的计算机视觉任务分配了一致的报告级别。我们一致认为,人类专家评估和诊断准确性考虑是达到临床意义评估水平的唯一可行方法。研究至少应报告八个核心成果指标:混淆矩阵,准确度,灵敏度,特异性,精度,F-1得分,接收器工作特性曲线下的面积,和精确召回曲线下的面积。
结论:牙科研究人员应致力于报告计算机视觉研究沿着概述的COMS。审稿人和编辑在评估研究时可能会考虑定义的COMS,建议作者在不使用COMS时证明其合理性。
结论:比较和综合牙科计算机视觉研究受到报告结果测量的多样性的阻碍。遵守定义的COMS有望增加研究之间的可比性,启用合成,减少选择性报告。
OBJECTIVE: To improve reporting and comparability as well as to reduce bias in dental computer vision studies, we aimed to develop a Core Outcome Measures Set (COMS) for this field. The COMS was derived consensus based as part of the WHO/ITU/WIPO Global Initiative AI for Health (WHO/ITU/WIPO AI4H).
METHODS: We first assessed existing guidance documents of diagnostic accuracy studies and conducted interviews with experts in the field. The resulting list of outcome measures was mapped against computer vision modeling tasks, clinical fields and reporting levels. The resulting systematization focused on providing relevant outcome measures whilst retaining details for meta-research and technical replication, displaying recommendations towards (1) levels of reporting for different clinical fields and tasks, and (2) outcome measures. The COMS was consented using a 2-staged e-Delphi, with 26 participants from various IADR groups, the WHO/ITU/WIPO AI4H, ADEA and AAOMFR.
RESULTS: We assigned agreed levels of reporting to different computer vision tasks. We agreed that human expert assessment and diagnostic accuracy considerations are the only feasible method to achieve clinically meaningful evaluation levels. Studies should at least report on eight core outcome measures: confusion matrix, accuracy, sensitivity, specificity, precision, F-1 score, area-under-the-receiver-operating-characteristic-curve, and area-under-the-precision-recall-curve.
CONCLUSIONS: Dental researchers should aim to report computer vision studies along the outlined COMS. Reviewers and editors may consider the defined COMS when assessing studies, and authors are recommended to justify when not employing the COMS.
CONCLUSIONS: Comparing and synthesizing dental computer vision studies is hampered by the variety of reported outcome measures. Adherence to the defined COMS is expected to increase comparability across studies, enable synthesis, and reduce selective reporting.