Background  Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods  Researchers manually annotated 70 items of information (\'entities\') in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the \'FLAIR\' framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results  The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator\'s score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms ( e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions  While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper 3).

BACKGROUND: Objectives of this study were to analyze the impact of including librarians and information specialist as methodological peer-reviewers. We sought to determine if and how librarians\' comments differed from subject peer-reviewers\'; whether there were differences in the implementation of their recommendations; how this impacted editorial decision-making; and the perceived utility of librarian peer-review by librarians and authors.
METHODS: We used a mixed method approach, conducting a qualitative analysis of reviewer reports, author replies and editors\' decisions of submissions to the International Journal of Health Governance. Our content analysis categorized 16 thematic areas, so that methodological and subject peer-reviewers\' comments, decisions and rejection rates could be compared. Categories were based on the standard areas covered in peer-review (e.g., title, originality, etc.) as well as additional in-depth categories relating to the methodology (e.g., search strategy, reporting guidelines, etc.). We developed and used criteria to judge reviewers\' perspectives and code their comments. We conducted two online multiple-choice surveys which were qualitatively analyzed: one of methodological peer-reviewers\' perceptions of peer-reviewing, the other of published authors\' views on the suggested revisions.
RESULTS: Methodological peer-reviewers assessed 13 literature reviews submitted between September 2020 and March 2023. 55 reviewer reports were collected: 25 from methodological peer-reviewers, 30 from subject peer-reviewers (mean: 4.2 reviews per manuscript). Methodological peer-reviewers made more comments on methodologies, with authors more likely to implement their changes (52 of 65 changes, vs. 51 of 82 by subject peer-reviewers); they were also more likely to reject submissions (seven vs. four times, respectively). Where there were differences in recommendations to editors, journal editors were more likely to follow methodological peer-reviewers (nine vs. three times, respectively). The survey of published authors (87.5% response rate) revealed four of seven found comments on methodologies helpful. Librarians\' survey responses (66.5% response rate) revealed those who conducted peer-reviews felt they improved quality of publications.
CONCLUSIONS: Librarians can enhance evidence synthesis publications by ensuring methodologies have been conducted and reported appropriately. Their recommendations helped authors revise submissions and facilitated editorial decision-making. Further research could determine if sharing reviews with subject peer-reviewers and journal editors could benefit them in better understanding of evidence synthesis methodologies.

In 2019 we invented the 2-week systematic review (2weekSR) methodology, to complete full, Preferred Reporting Items for Systematic Reviews and Meta-Analyses-compliant systematic reviews in approximately 2 weeks. Since then we have continued to develop and adapt the 2weekSR methodology for completing larger and more complex systematic reviews, including less experienced or inexperienced team members.
For ten 2weekSRs, we collected data on (1) systematic review characteristics; (2) systematic review teams; and (3) time to completion and publication. We have also continued to develop new tools and integrate them into the 2weekSR processes.
The 10 2weekSRs addressed intervention, prevalence, and utilization questions, and included a mix of randomized and observational studies. Reviews involved screening from 458 to 5,471 references, and included between 5 and 81 studies. The median team size was 6. Most reviews (7/10) included team members with limited systematic review experience; three included team members with no prior experience. Reviews required a median of 11 workdays (range: 5-20) and 17 calendar days (range: 5-84) to complete; time from journal submission to publication ranged from 99 to 260 days.
The 2weekSR methodology scales with review size and complexity, offering a considerable time-saving over traditionally conducted systematic reviews without relying on methodological shortcuts associated with \"rapid reviews.\"

Various case descriptions and scoring systems have been used to define neonatal calf diarrhea (NCD) and diverse diarrhea-related outcomes are reported, which limits direct comparison between studies. Therefore, the objective of this scoping review was to characterize the case definitions used for NCD and describe diarrhea-related outcomes to inform future efforts towards standardization. A literature search identified articles using 3 databases (Medline, CAB Direct, Agricola), along with Google and Google Scholar. This returned 16,854 unique articles, which were then screened for eligibility by two independent reviewers, resulting in 555 being selected for data extraction. Among articles, the study populations included mostly dairy-breed calves (88%; n = 486) while the remainder evaluated beef, crossbred, or dual purpose beef/dairy calves (10%; n = 53), or did not report breed (3%; n = 16). Studies used between 1 and 8 metrics to define NCD, with 933 unique metrics extracted in total. The most common metric was fecal consistency alone (30%; n = 281), or with at least 1 other metric (26%; n = 241). To define diarrhea, fecal consistency was either described qualitatively (e.g., \"profuse liquid feces\"), or semi-quantitatively, for example using a scoring system that frequently included 4 levels (n = 208). Some NCD case definitions included fecal color, volume, or odor (10%; n = 98), physical exam parameters (8%; n = 79), the duration of abnormal feces (7%; n = 67), the presence of abnormal contents (e.g., blood, 7%; n = 61), farm treatment records (6%; n = 54), fecal dry matter (1%; n = 12), or another metric (4%; n = 41). One or more references were cited for the NCD case definition by 49% of studies (n = 273/555), with the most common references being Larson et al. (1977) (n = 85), and McGuirk (2008) (n = 59). In the 555 included articles, 979 unique diarrhea-related outcomes were found, most commonly a binary categorization of calves having or not having diarrhea (49%; n = 483). Other articles reported statistical outcomes calculated from fecal scores (16%; n = 159), multiple diarrhea severities (10%; n = 95), or the age calves first developed NCD (8%; n = 76). This review characterized substantial heterogeneity among NCD case definitions and diarrhea-related outcomes, which limits interpretation and comparison of studies. Future work is required to develop and validate reporting standards for NCD to optimize knowledge synthesis and support rigorous and ethical calf health research.

OBJECTIVE: Randomized controlled trials (RCTs) are the preferred source of evidence for the relative effect of healthcare interventions summarized in knowledge syntheses. Nonrandomized studies of interventions (NRSI) may provide replacement, sequential, or complementary evidence to RCTs. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach can provide different options for properly using RCTs and NRSI integrated in health syntheses. In this article, we discuss different implications on the certainty of evidence when authors consider the use of NRSI and RCTs in systematic reviews using GRADE. Although this is a GRADE-related article, it is not an official GRADE guidance or concept article.
METHODS: We present case studies used during GRADE working group meetings for discussion of the effects of using NRSI and RCTs on GRADE domains and on the certainty of evidence. Several concepts were discussed through iterative feedback with experts in GRADE methods and Cochrane authors. We compared suggested solutions for possible scenarios that can be met in evidence syntheses informing decisions and future guidance.
RESULTS: Different scenarios for the use of RCTs and NRSI in evidence syntheses are presented, focusing on how different GRADE ratings between RCTs and NRSI affect the overall assessment of the evidence and possible health recommendations.
CONCLUSIONS: Considering differences and similarities grounded in the GRADE approach between NRSI and RCTs may help complement one another and maximize the value of knowledge syntheses and health recommendations.

Evidence syntheses that engage librarians as co-authors produce higher-quality results than those that do not. Trained as teachers, researchers, and information managers, librarians possess expert knowledge on research methodologies and information retrieval approaches that are critical for evidence synthesis. Researchers are under increasing pressure to produce evidence syntheses to inform practice and policymaking. Many fields outside of health science and medicine, however, do not have established guidelines, processes, or methodologies. This article describes how librarians led the creation of an interdisciplinary toolkit for researchers new to evidence synthesis. The implementation of the tools, including a protocol, supported eight evidence syntheses focused on effective agricultural interventions published in a special collection in Nature Research in October 2020. This article is a step-by-step overview of the tools and process. We advocate that librarian collaboration in evidence synthesis must become the norm, not the exception. Evidence synthesis project leads without access to a qualified librarian may use this toolkit as a point of entry for production of transparent, reproducible reviews.

Decision-makers increasingly request rapid answers to clinical or public health questions. To save time, personnel, and financial resources, rapid reviews streamline the methodological steps of the systematic review process. We aimed to explore the validity of a rapid review approach that combines a substantially abbreviated literature search with a single-reviewer screening of abstracts and full texts using three case studies.
We used a convenience sample of three ongoing Cochrane reviews as reference standards. Two reviews addressed oncological topics and one addressed a public health topic. For each of the three topics, three reviewers screened the literature independently. Our primary outcome was the change in conclusions between the rapid reviews and the respective Cochrane reviews. In case the rapid approach missed studies, we recalculated the meta-analyses for the main outcomes and asked Cochrane review authors if the new body of evidence would change their original conclusion compared with the reference standards. Additionally, we assessed the sensitivity of the rapid review approach compared with the results of the original Cochrane reviews.
For the two oncological topics (case studies 1 and 2), the three rapid reviews each yielded the same conclusions as the Cochrane reviews. However, the authors would have had less certainty about their conclusion in case study 2. For case study 3, the public health topic, only one of the three rapid reviews led to the same conclusion as the Cochrane review. The other two rapid reviews provided insufficient information for the authors to draw conclusions. Using the rapid review approach, the sensitivity was 100% (3 of 3) for case study 1. For case study 2, the three rapid reviews identified 40% (4 of 10), 50% (5 of 10), and 60% (6 of 10) of the included studies, respectively; for case study 3, the respective numbers were 38% (8 of 21), 43% (9 of 21), and 48% (10 of 21).
Within the limitations of these case studies, a rapid review approach that combines abbreviated literature searches with single-reviewer screening may be feasible for focused clinical questions. For complex public health topics, sensitivity seems to be insufficient.

Background: The intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR) has been developed to facilitate detailed assessments of intervention complexity in systematic reviews. Worked examples of the tool\'s application are needed to promote its use and refinement. The aim of this case study was to apply the iCAT_SR to a subset of 20 studies included in a Cochrane review of interventions aimed at improving appropriate polypharmacy in older people. Methods: Interventions were assessed independently by two authors using the six core iCAT_SR dimensions: (1) \'Target organisational levels/categories\'; (2) \'Target behaviour/actions\'; (3) \'Active intervention components\'; (4) \'Degree of tailoring\'; (5) \'Level of skill required by intervention deliverers\'; (6) \'Level of skill required by intervention recipients\'. Attempts were made to apply four optional dimensions: \'Interaction between intervention components\'; \'Context/setting\'; \'Recipient/provider factors\'; \'Nature of causal pathway\'. Inter-rater reliability was assessed using Cohen\'s Kappa coefficient. Disagreements were resolved by consensus discussion. The findings are presented narratively. Results: Assessments involving the core iCAT_SR dimensions showed limited consistency in intervention complexity across included studies, even when categorised according to clinical setting. Interventions were delivered across various organisational levels and categories (i.e. healthcare professionals and patients) and typically comprised multiple components. Intermediate skill levels were required by those delivering and receiving the interventions across all studies. A lack of detail in study reports precluded application of the iCAT_SR\'s optional dimensions. The inter-rater reliability was substantial (Cohen\'s Kappa = 0.75) Conclusions: This study describes the application of the iCAT_SR to studies included in a Cochrane systematic review. Future intervention studies need to ensure more detailed reporting of interventions, context and the causal pathways underlying intervention effects to allow a more holistic understanding of intervention complexity and facilitate replication in other settings. The experience gained has helped to refine the original guidance document relating to the application of iCAT_SR.

Systematic evidence mapping offers a robust and transparent methodology for facilitating evidence-based approaches to decision-making in chemicals policy and wider environmental health (EH). Interest in the methodology is growing; however, its application in EH is still novel. To facilitate the production of effective systematic evidence maps for EH use cases, we survey the successful application of evidence mapping in other fields where the methodology is more established. Focusing on issues of \"data storage technology,\" \"data integrity,\" \"data accessibility,\" and \"transparency,\" we characterize current evidence mapping practice and critically review its potential value for EH contexts. We note that rigid, flat data tables and schema-first approaches dominate current mapping methods and highlight how this practice is ill-suited to the highly connected, heterogeneous, and complex nature of EH data. We propose this challenge is overcome by storing and structuring data as \"knowledge graphs.\" Knowledge graphs offer a flexible, schemaless, and scalable model for systematically mapping the EH literature. Associated technologies, such as ontologies, are well-suited to the long-term goals of systematic mapping methodology in promoting resource-efficient access to the wider EH evidence base. Several graph storage implementations are readily available, with a variety of proven use cases in other fields. Thus, developing and adapting systematic evidence mapping for EH should utilize these graph-based resources to ensure the production of scalable, interoperable, and robust maps to aid decision-making processes in chemicals policy and wider EH.

In systematic reviews, meta-analyses are routinely applied to summarize the results of the relevant studies for a specific research question. If one can assume that in all studies the same true effect is estimated, the application of a meta-analysis with common effect (commonly referred to as fixed-effect meta-analysis) is adequate. If between-study heterogeneity is expected to be present, the method of choice is a meta-analysis with random effects. The widely used DerSimonian and Laird method for meta-analyses with random effects has been criticized due to its unfavorable statistical properties, especially in the case of very few studies. A working group of the Cochrane Collaboration recommended the use of the Knapp-Hartung method for meta-analyses with random effects. However, as heterogeneity cannot be reliably estimated if only very few studies are available, the Knapp-Hartung method, while correctly accounting for the corresponding uncertainty, has very low power. Our aim is to summarize possible methods to perform meaningful evidence syntheses in the situation with only very few (ie, 2-4) studies. Some general recommendations are provided on which method should be used when. Our recommendations are based on the existing literature on methods for meta-analysis with very few studies and consensus of the authors. The recommendations are illustrated by 2 examples coming from dossier assessments of the Institute for Quality and Efficiency in Health Care.