Abstract：
To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR).
Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.
From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).
A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers.
From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69).
A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
摘要：
目的：评估一种使用自动化和众包的方法，以在实时系统评价（LSR）中识别和分类类风湿关节炎（RA）的随机对照试验（RCT）。
方法：首先通过机器学习和CochraneCrowd筛选RA中RCT的数据库搜索记录，以排除非RCT，然后由受训审稿人使用人口，干预,比较和结果（PICO）注释器平台，以评估合格性并将试验分类为适当的审查。专家使用自定义在线工具解决了分歧。我们评估了效率收益，灵敏度,审稿人之间的准确性和评分者之间的一致性(kappa分数)。
结果：来自42,452条记录，机器学习和Cochrane人群排除了28,777（68％），实习审稿人排除了4,529人(11%)，专家排除了7200人(17%)。符合我们LSR条件的1,946条记录代表220条RCT，并纳入148/149(99.3%)已知的来自先前审查的符合条件的试验.虽然被排除在我们的LSR之外，6,420条记录被归类为RA中的其他RCT，以告知未来的审查。在RCT领域，学员的假阴性率最高(12%)，尽管其中只有1.1％是主要记录。两名审稿人的Kappa评分范围从中等到实质一致（0.40到0.69）。
结论：一种结合机器学习的筛选方法，众包,受训人员的参与大大减轻了专家评审人员的筛查负担，并且高度敏感。
方法：首先通过机器学习和CochraneCrowd筛选RA中RCT的数据库搜索记录，以排除非RCT，然后由受训审稿人使用人口，干预,比较和结果（PICO）注释器平台，以评估合格性并将试验分类为适当的审查。专家使用自定义在线工具解决了分歧。我们评估了效率收益，灵敏度,审稿人之间的准确性和评分者之间的一致性(kappa分数)。
结果：来自42,452条记录，机器学习和Cochrane人群排除了28,777（68％），实习审稿人排除了4,529人(11%)，专家排除了7200人(17%)。符合我们LSR条件的1,946条记录代表220条RCT，并纳入148/149(99.3%)已知的来自先前审查的符合条件的试验.虽然被排除在我们的LSR之外，6,420条记录被归类为RA中的其他RCT，以告知未来的审查。在RCT领域，学员的假阴性率最高(12%)，尽管其中只有1.1％是主要记录。两名审稿人的Kappa评分范围从中等到实质一致（0.40到0.69）。
结论：一种结合机器学习的筛选方法，众包,受训人员的参与大大减轻了专家评审人员的筛查负担，并且高度敏感。
