OBJECTIVE: In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels.
METHODS: We used a mixed-methods design. Two independent clinicians\' manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue.
METHODS: Inter-rater reliability was assessed using Cohen\'s Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers.
RESULTS: Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between - 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively.
CONCLUSIONS: While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

BACKGROUND: Development of guidelines for public health, health system, and health policy interventions demands complex systems thinking to understand direct and indirect effects of interventions within dynamic systems. The WHO-INTEGRATE framework, an evidence-to-decision framework rooted in the norms and values of the World Health Organization (WHO), provides a structured method to assess complexities in guidelines systematically, such as the balance of an intervention\'s health benefits and harms and their human rights and socio-cultural acceptability. This paper provides a worked example of the application of the WHO-INTEGRATE framework in developing the WHO guidelines on parenting interventions to prevent child maltreatment, and shares reflective insights regarding the value added, challenges encountered, and lessons learnt.
METHODS: The methodological approach comprised describing the intended step-by-step application of the WHO-INTEGRATE framework and gaining reflective insights from introspective sessions within the core team guiding the development of the WHO guidelines on parenting interventions and a methodological workshop.
RESULTS: The WHO-INTEGRATE framework was used throughout the guideline development process. It facilitated reflective deliberation across a broad range of decision criteria and system-level aspects in the following steps: (1) scoping the guideline and defining stakeholder engagement, (2) prioritising WHO-INTEGRATE sub-criteria and guideline outcomes, (3) using research evidence to inform WHO-INTEGRATE criteria, and (4) developing and presenting recommendations informed by WHO-INTEGRATE criteria. Despite the value added, challenges, such as substantial time investment required, broad scope of prioritised sub-criteria, integration across diverse criteria, and sources of evidence and translation of insights into concise formats, were encountered.
CONCLUSIONS: Application of the WHO-INTEGRATE framework was crucial in the integration of effectiveness evidence with insights into implementation and broader implications of parenting interventions, extending beyond health benefits and harms considerations and fostering a whole-of-society-perspective. The evidence reviews for prioritised WHO-INTEGRATE sub-criteria were instrumental in guiding guideline development group discussions, informing recommendations and clarifying uncertainties. This experience offers important lessons for future guideline panels and guideline methodologists using the WHO-INTEGRATE framework.

BACKGROUND: Guidelines depend on effect estimates, usually derived from randomised controlled trials, to inform their decisions. Qualitative research evidence may improve decisions made but where in the process and the methods to do this have not been so clearly established. We sought to describe and appraise how qualitative research has been used to inform World Heath Organization guidance since 2020.
METHODS: We conducted a document analysis of WHO guidelines from 2020 to 2022. We purposely sampled guidelines on the topics of maternal and newborn health (MANH) and infectious diseases, as most of the qualitative synthesis to date has been conducted on these topics, likely representing the \'best case\' scenario. We searched the in-built repository feature of the WHO website and used standardised search terms to identify qualitative reporting. Using deductive frameworks, we described how qualitative evidence was used to inform guidelines and appraised the standards of this use.
RESULTS: Of the 29 guidelines, over half used qualitative research to help guide decisions (18/29). A total of 8 of these used qualitative research to inform the guideline scope, all 18 to inform recommendations, and 1 to inform implementation considerations. All guidelines drew on qualitative evidence syntheses (QES), and five further supplemented this with primary qualitative research. Qualitative findings reported in guidelines were typically descriptive, identifying people\'s perception of the benefits and harms of interventions or logistical barriers and facilitators to programme success. No guideline provided transparent reporting of how qualitative research was interpreted and weighed used alongside other evidence when informing decisions, and only one guideline reported the inclusion of qualitative methods experts on the panel. Only a few guidelines contextualised their recommendations by indicating which populations and settings qualitative findings could be applied.
CONCLUSIONS: Qualitative research frequently informed WHO guideline decisions particularly in the field of MANH. However, the process often lacked transparency. We identified unmet potential in informing implementation considerations and contextualisation of the recommendations. Use in these areas needs further methods development.

BACKGROUND: The success of abdominal cancer surgery depends not only on the surgery itself but is influenced by the overall perioperative management. Given the multitude of perioperative measures and the ever-increasing number of studies on perioperative management, it is difficult to keep track and provide evidence-based perioperative management. The planned guideline on perioperative management will review the existing evidence and derive treatment recommendations.
METHODS: The processing of the evidence is carried out by 6 working groups according to an 8-step scheme: after drafting the guideline questions in PICO format (1), a systematic literature search is carried out (2), and the records found are screened by two independent reviewers from the coordination team. Subsequently, the full texts of the potentially relevant articles are made available to the working groups for full text screening (3). All articles to be included are reviewed for methodological quality (4) before summary of findings tables are generated (5). In line with the GRADE approach, confidence in the evidence is assessed (6) before a recommendation is derived from the evidence, using a modified GRADE Evidence to Decision Framework (7). Finally, all recommendations are compiled and agreed within the guideline group (8).
CONCLUSIONS: Guidelines serve as foundation for therapy decisions in everyday clinical practice and should therefore be based on up-to-date research results. However, while primary studies and systematic reviews are critically reviewed for their methodological quality, the process of guideline development is often not comprehensible. A protocol with predefined methodology should therefore create transparency and strengthen confidence in the recommendations.
BACKGROUND: The guideline is registered in the AWMF (Association of the Scientific Medical Societies) Guideline Register (088-010OL).

BACKGROUND: Several guideline organizations produce patient versions of clinical practice guidelines (PVGs) which translate recommendations into simple language. A former study of our working group revealed that few guideline organizations publish their methods used to develop PVGs. Clear definitions of PVGs do not prevail and their purposes often remain unclear. We aimed to explore experts\' perspectives on developing, disseminating and implementing PVGs to discuss and incorporate these experiences when consenting on methodological guidance and further improving PVGs.
METHODS: We conducted 17 semi-structured telephone interviews with international experts working with PVGs from September 2021 through January 2022. We conducted the interviews in English or German, they were recorded and transcribed verbatim. We utilized Mayring\'s qualitative content analysis with MAXQDA software to analyze the data.
RESULTS: In two interviews two participants were interviewed at the same time. This resulted in a total of 19 participants from 16 different organizations and eight different countries participated. Most were female (16/19) and their experience in working with PVGs ranged from 1 to 20 years. All follow methodological standards when developing PVGs, but the extent of these standards and their public accessibility differs. Aims and target groups of PVGs vary between organizations. Facilitators for developing PVGs are working with a multidisciplinary team, financial resources, consultation processes and a high-quality underlying CPG. Facilitators for disseminating and implementing PVGs are using various strategies. Barriers, on the other hand, are the lack of these factors. All participants mentioned patient involvement as a key aspect in PVG development.
CONCLUSIONS: The steps in the PVG development process are largely similar across the countries. Focus is placed on the involvement of patients in the development process, although the extent of participation varies. The experts collectively attribute great importance to PVGs overall, but in order to constantly adapt to medical progress and changing conditions, the focus in the future may be more on formats like living guidelines. Although there are different views on the mandatory development of PVGs, there is a consistent call for more transparency regarding the methodology used for PVGs.

UNASSIGNED: Despite mounting evidence that the inclusion of race and ethnicity in clinical prediction models may contribute to health disparities, existing critical appraisal tools do not directly address such equity considerations.
UNASSIGNED: This study developed a critical appraisal tool extension to assess algorithmic bias in clinical prediction models.
UNASSIGNED: A modified e-Delphi approach was utilized to develop and obtain expert consensus on a set of racial and ethnic equity-based signaling questions for appraisal of risk of bias in clinical prediction models. Through a series of virtual meetings, initial pilot application, and an online survey, individuals with expertise in clinical prediction model development, systematic review methodology, and health equity developed and refined this tool.
UNASSIGNED: Consensus was reached for ten equity-based signaling questions, which led to the development of the Critical Appraisal for Racial and Ethnic Equity in Clinical Prediction Models (CARE-CPM) extension. This extension is intended for use along with existing critical appraisal tools for clinical prediction models.
UNASSIGNED: CARE-CPM provides a valuable risk-of-bias assessment tool extension for clinical prediction models to identify potential algorithmic bias and health equity concerns. Further research is needed to test usability, interrater reliability, and application to decision-makers.

暂无摘要。

OBJECTIVE: To understand developers\' perception of patient (versions of) guidelines (PVGs), and identify challenges during the PVG development, with the aim to inform methodological guidance for future PVG development.
METHODS: We used a descriptive qualitative design. Semi-structured interviews were conducted virtually from December 2021 to April 2022, with a purposive sampling of 12 PVG developers from nine teams in China. Conventional and directed content analysis was used for data analysis.
RESULTS: The interviews identified PVG developers\' understanding of PVGs, their current practice experience, and the challenges of developing PVGs. Participants believed PVGs were a type of health education material for patients; therefore, it should be based on patient needs and be understandable and accessible. Participants suggested that PVGs could be translated/adapted from one or several clinical practice guidelines (CPG), or developed de novo (i.e., the creation of an entirely new PVG with its own set of research questions that are independent of existing CPGs). Participants perceived those existing methodological guidelines for PVG development might not provide clear instructions for PVGs developed from multiple CPGs and from de novo development. Challenges to PVG development include (1) a lack of standardized and native guidance on developing PVGs; (2) a lack of standardized guidance on patient engagement; (3) other challenges: no publicly known and trusted platform that could disseminate PVGs; concerns about the conflicting interests with health professionals.
CONCLUSIONS: Our study suggests clarifying the concept of PVG is the primary task to develop PVGs and carry out related research. There is a need to make PVG developers realize the roles of PVGs, especially in helping decision-making, to maximize the effect of PVG. It is necessary to develop native consensus-based guidance considering developers\' perspectives regarding PVGs.

While reporting of individual conflicts of interest is formalised, it is unclear to what extent the funding of clinical practice guidelines (CPGs) is formally reported. The aim of this study is to explore the accuracy and comprehensiveness of reporting on funding in German CPGs.
We searched for CPGs in the registry of the Association of the Scientific Medical Societies in Germany in July 2020. Information on guideline funding was categorised by two reviewers independently and discrepancies were clarified by discussion with a third reviewer. Accuracy and comprehensiveness of reporting on funding was assessed using the German Instrument for Methodological Guideline Appraisal (DELBI).
We included 507 CPGs published between 2015 and 2020 in the main analysis. 23/507 (4.5%) of the CPGs achieved the highest DELBI score by including information on funding sources, expenses and the amount of funding provided, as well as a statement on the independence of the guideline authors from the funding institution(s). CPGs with more rigorous methodological requirements (systematic review of the literature and/or structured consensus-building) received higher DELBI scores.
German CPGs do not communicate their funding transparently. Transparency of CPG funding could be achieved by making it mandatory to publish information for all guidelines. For that purpose, a standardised form and guidance should be developed.

Established in 2015, the Multi-Stakeholder Engagement (MuSE) Consortium is an international network of over 120 individuals interested in stakeholder engagement in research and guidelines. The MuSE group is developing guidance for stakeholder engagement in the development of health and healthcare guideline development. The development of this guidance has included multiple meetings with stakeholders, including patients, payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, the public, and purchasers of health services and has identified a number of key issues. These include: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. In this paper, we discuss these issues and our plan to develop guidance to facilitate stakeholder engagement in all stages of the development of health and healthcare guideline development.
A group of international researchers, patient partners, and other stakeholders are working together to create a checklist for when and how to involve stakeholders in health guideline development. Health guidelines include clinical practice guidelines, which your healthcare provider uses to determine treatments for health conditions. While working on this checklist, the team identified key issues to work on, including: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. This paper describes each issue and how the team plans to produce guidance papers to address them.