背景:人工智能(AI)医疗设备具有改变现有临床工作流程并最终改善患者预后的潜力。人工智能医疗设备已经显示出用于诊断等一系列临床任务的潜力。预测,和治疗决策,如药物剂量。有,然而,迫切需要确保这些技术对所有人口都是安全的。最近的文献表明,需要进行严格的性能误差分析,以识别诸如伪相关性的算法编码等问题(例如,受保护的特征)或可能导致患者伤害的特定故障模式。评估人工智能医疗设备的研究报告指南要求提及性能错误分析;然而,仍然缺乏对临床研究中应如何分析性能错误的理解,以及作者应该旨在发现和报告的危害。
目的:本系统评价将评估研究AI医疗设备作为临床干预措施的随机对照试验(RCT)中AI错误和不良事件(AE)的频率和严重程度。审查还将探讨如何分析绩效错误,包括分析是否包括对子组级结果的调查。
方法:本系统综述将确定和选择评估AI医疗设备的RCT。搜索策略将部署在MEDLINE(Ovid)中,Embase(Ovid),科克伦中部,和临床试验登记处,以确定相关论文。书目数据库中确定的RCT将与临床试验注册中心交叉引用。感兴趣的主要结果是AI错误的频率和严重程度,病人的伤害,并报告AE。RCT的质量评估将基于Cochrane偏差风险工具(RoB2)的第2版。数据分析将包括比较研究小组之间的错误率和患者伤害,在适当情况下,将对对照组和干预组的患者伤害率进行荟萃分析.
结果:该项目于2023年2月在PROSPERO上注册。初步搜索已经完成,搜索策略是与信息专家和方法学家协商设计的。标题和摘要筛选于2023年9月开始。全文筛选正在进行中,数据收集和分析于2024年4月开始。
结论:对人工智能医疗器械的评估显示出了有希望的结果;然而,研究报告是可变的。检测,分析,以及报告性能错误和患者危害对于可靠地评估RCT中AI医疗设备的安全性至关重要。范围搜索表明,危害的报告是可变的,通常没有提到AE。这项系统评价的结果将确定AI表现错误和患者危害的频率和严重程度,并深入了解如何分析错误以考虑整体和小组表现。
背景:PROSPEROCRD42023387747;https://www.crd.约克。AC.uk/prospro/display_record.php?RecordID=387747。
■PRR1-10.2196/51614。
BACKGROUND: Artificial intelligence (AI) medical devices have the potential to transform existing clinical workflows and ultimately improve patient outcomes. AI medical devices have shown potential for a range of clinical tasks such as diagnostics, prognostics, and therapeutic decision-making such as drug dosing. There is, however, an urgent need to ensure that these technologies remain safe for all populations. Recent literature demonstrates the need for rigorous performance error analysis to identify issues such as algorithmic encoding of spurious correlations (eg, protected characteristics) or specific failure modes that may lead to patient harm. Guidelines for reporting on studies that evaluate AI medical devices require the mention of performance error analysis; however, there is still a lack of understanding around how performance errors should be analyzed in clinical studies, and what harms authors should aim to detect and report.
OBJECTIVE: This systematic review will assess the frequency and severity of AI errors and adverse events (AEs) in randomized controlled trials (RCTs) investigating AI medical devices as interventions in clinical settings. The review will also explore how performance errors are analyzed including whether the analysis includes the investigation of subgroup-level outcomes.
METHODS: This systematic review will identify and select RCTs assessing AI medical devices. Search strategies will be deployed in MEDLINE (Ovid), Embase (Ovid), Cochrane CENTRAL, and clinical trial registries to identify relevant papers. RCTs identified in bibliographic databases will be cross-referenced with clinical trial registries. The primary outcomes of interest are the frequency and severity of AI errors, patient harms, and reported AEs. Quality assessment of RCTs will be based on version 2 of the Cochrane risk-of-bias tool (RoB2). Data analysis will include a comparison of error rates and patient harms between study arms, and a meta-analysis of the rates of patient harm in control versus intervention arms will be conducted if appropriate.
RESULTS: The project was registered on PROSPERO in February 2023. Preliminary searches have been completed and the search strategy has been designed in consultation with an information specialist and methodologist. Title and abstract screening started in September 2023. Full-text screening is ongoing and data collection and analysis began in April 2024.
CONCLUSIONS: Evaluations of AI medical devices have shown promising results; however, reporting of studies has been variable. Detection, analysis, and reporting of performance errors and patient harms is vital to robustly assess the safety of AI medical devices in RCTs. Scoping searches have illustrated that the reporting of harms is variable, often with no mention of AEs. The findings of this systematic review will identify the frequency and severity of AI performance errors and patient harms and generate insights into how errors should be analyzed to account for both overall and subgroup performance.
BACKGROUND: PROSPERO CRD42023387747; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=387747.
UNASSIGNED: PRR1-10.2196/51614.