Abstract：
OBJECTIVE: The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach is a systematic method for assessing the certainty of evidence (CoE) and strength of recommendations in health care. We aimed to verify the effects of an online-based GRADE course on multirater consistency in the evaluation of the CoE in systematic reviews (SRs) analysis.
METHODS: Sixty-five Brazilian methodologists and researchers participated in an online course over 8 weeks. Asynchronous lessons and weekly synchronous meetings addressed the GRADE system in the context of CoE assessment. We asked participants to evaluate the CoE of random SRs (two before and another two after the course). Analyzes focused on the multirater agreement with a standard response, in the interrater agreement, and before-after changes in the proportion of participants that rated down the domains.
RESULTS: 48 individuals completed the course. Participants presented improvements in the raters\' assessment of the CoE using the GRADE approach after the course. The multirater consistency of indirectness, imprecision, and the overall CoE increased after the course, as well as the agreement between raters and the standard response. Furthermore, interrater reliability increased for risk of bias, inconsistency, indirectness, publication bias, and overall CoE, indicating progress in between-raters consistency. After the course, approximately 78% of individuals rated down the overall CoE to a low/very low degree, and participants presented more explanations for the judgment of each domain.
CONCLUSIONS: An online GRADE course improved the consistency and agreement of the CoE assessment by Brazilian researchers. Online training courses have the potential to improve skills in guideline methodology development.
METHODS: Sixty-five Brazilian methodologists and researchers participated in an online course over 8 weeks. Asynchronous lessons and weekly synchronous meetings addressed the GRADE system in the context of CoE assessment. We asked participants to evaluate the CoE of random SRs (two before and another two after the course). Analyzes focused on the multirater agreement with a standard response, in the interrater agreement, and before-after changes in the proportion of participants that rated down the domains.
RESULTS: 48 individuals completed the course. Participants presented improvements in the raters\' assessment of the CoE using the GRADE approach after the course. The multirater consistency of indirectness, imprecision, and the overall CoE increased after the course, as well as the agreement between raters and the standard response. Furthermore, interrater reliability increased for risk of bias, inconsistency, indirectness, publication bias, and overall CoE, indicating progress in between-raters consistency. After the course, approximately 78% of individuals rated down the overall CoE to a low/very low degree, and participants presented more explanations for the judgment of each domain.
CONCLUSIONS: An online GRADE course improved the consistency and agreement of the CoE assessment by Brazilian researchers. Online training courses have the potential to improve skills in guideline methodology development.
摘要：
目标：等级(建议的等级，评估,发展,和评估）方法是一种评估医疗保健中证据确定性（CoE）和建议强度的系统方法。我们旨在验证基于在线的GRADE课程在系统评价（SRs）分析中对CoE评估的多评分者一致性的影响。
方法：65名巴西方法学家和研究人员参加了为期8周的在线课程。异步课程和每周同步会议在CoE评估的背景下讨论了GRADE系统。我们要求参与者评估随机SR的CoE（课程前两个，课程后两个）。分析集中在具有标准响应的多评分者协议上，在评分者之间的协议中，以及对域进行评级的参与者比例的前后变化。
结果：48个人完成了课程。参与者在课程结束后使用GRADE方法对评估者进行了评估。间接性的多评估者一致性，不精确,课程结束后，整体CoE增加了，以及评估者之间的协议和标准回应。此外，评估者间的可靠性因偏差风险而增加，不一致,间接性,出版偏见，和整个CoE，表明评估者之间一致性的进展。课程结束后，大约78%的人将整体CoE评级降低到较低/非常低的程度，参与者对每个领域的判断提出了更多的解释。
结论：在线GRADE课程提高了巴西研究人员对CoE评估的一致性和一致性。在线培训课程有可能提高指南方法开发的技能。
方法：65名巴西方法学家和研究人员参加了为期8周的在线课程。异步课程和每周同步会议在CoE评估的背景下讨论了GRADE系统。我们要求参与者评估随机SR的CoE（课程前两个，课程后两个）。分析集中在具有标准响应的多评分者协议上，在评分者之间的协议中，以及对域进行评级的参与者比例的前后变化。
结果：48个人完成了课程。参与者在课程结束后使用GRADE方法对评估者进行了评估。间接性的多评估者一致性，不精确,课程结束后，整体CoE增加了，以及评估者之间的协议和标准回应。此外，评估者间的可靠性因偏差风险而增加，不一致,间接性,出版偏见，和整个CoE，表明评估者之间一致性的进展。课程结束后，大约78%的人将整体CoE评级降低到较低/非常低的程度，参与者对每个领域的判断提出了更多的解释。
结论：在线GRADE课程提高了巴西研究人员对CoE评估的一致性和一致性。在线培训课程有可能提高指南方法开发的技能。
