PDF(Pubmed)
Abstract：
BACKGROUND: The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.
METHODS: In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.
RESULTS: From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.
CONCLUSIONS: Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.
METHODS: In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.
RESULTS: From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.
CONCLUSIONS: Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.
摘要：
背景：对基于证据的医疗决策的高质量系统文献综述（SRs）的需求正在增长。SR成本很高，需要高技能审稿人的稀缺资源。已经提出了自动化技术来节省工作量并加快SR工作流程。我们旨在全面概述PubMed索引的SR自动化研究，专注于这些技术在现实世界实践中的适用性。
方法：2022年11月，我们提取，合并,并在SR自动化上运行了对SR的集成PubMed搜索。全文包括英文同行评审文章，如果他们报告了对SR自动化方法(SSAM)的研究，或自动SR(ASR)。书目分析和知识发现研究被排除在外。记录筛选由单个审阅者进行，全文论文的选择一式两份。我们总结了出版物的细节，自动审查阶段，自动化目标，应用工具，数据源,方法,结果，和谷歌学者对SR自动化研究的引用。
结果：根据标题和摘要筛选的5321条记录，我们收录了123篇全文，其中SSAM108个，ASR15个。自动化用于搜索(19/123，15.4%)，记录筛查（89/123，72.4%），全文选择（6/123，4.9%），数据提取(13/123，10.6%)，偏见风险评估(9/123，7.3%)，证据综合(2/123，1.6%)，证据质量评估(2/123，1.6%)，和报告（2/123，1.6%）。11项（8.9％）研究将多个SR阶段自动化。自动记录筛选的性能在SR主题中差异很大。在已发布的ASR中，我们找到了自动搜索的例子，记录筛选,全文选择，和数据提取。在某些ASR中，自动化完全补充了手动审核，以提高灵敏度，而不是节省工作量。在ASR中，自动化详细信息的报告通常是不完整的。
结论：正在为所有SR阶段开发自动化技术，但现实世界的采用率有限。大多数SR自动化工具以单个SR阶段为目标，在整个SR过程中节省了适度的时间，并且在研究中具有不同的灵敏度和特异性。因此,SR自动化的实际好处仍然不确定。标准化术语，reporting,和研究报告的指标可以增强SR自动化技术在现实世界实践中的采用。
方法：2022年11月，我们提取，合并,并在SR自动化上运行了对SR的集成PubMed搜索。全文包括英文同行评审文章，如果他们报告了对SR自动化方法(SSAM)的研究，或自动SR(ASR)。书目分析和知识发现研究被排除在外。记录筛选由单个审阅者进行，全文论文的选择一式两份。我们总结了出版物的细节，自动审查阶段，自动化目标，应用工具，数据源,方法,结果，和谷歌学者对SR自动化研究的引用。
结果：根据标题和摘要筛选的5321条记录，我们收录了123篇全文，其中SSAM108个，ASR15个。自动化用于搜索(19/123，15.4%)，记录筛查（89/123，72.4%），全文选择（6/123，4.9%），数据提取(13/123，10.6%)，偏见风险评估(9/123，7.3%)，证据综合(2/123，1.6%)，证据质量评估(2/123，1.6%)，和报告（2/123，1.6%）。11项（8.9％）研究将多个SR阶段自动化。自动记录筛选的性能在SR主题中差异很大。在已发布的ASR中，我们找到了自动搜索的例子，记录筛选,全文选择，和数据提取。在某些ASR中，自动化完全补充了手动审核，以提高灵敏度，而不是节省工作量。在ASR中，自动化详细信息的报告通常是不完整的。
结论：正在为所有SR阶段开发自动化技术，但现实世界的采用率有限。大多数SR自动化工具以单个SR阶段为目标，在整个SR过程中节省了适度的时间，并且在研究中具有不同的灵敏度和特异性。因此,SR自动化的实际好处仍然不确定。标准化术语，reporting,和研究报告的指标可以增强SR自动化技术在现实世界实践中的采用。
