BACKGROUND: Genome-wide association studies have enabled Mendelian randomization analyses to be performed at an industrial scale. Two-sample summary data Mendelian randomization analyses can be performed using publicly available data by anyone who has access to the internet. While this has led to many insightful papers, it has also fuelled an explosion of poor-quality Mendelian randomization publications, which threatens to undermine the credibility of the whole approach.
RESULTS: We detail five pitfalls in conducting a reliable Mendelian randomization investigation: (1) inappropriate research question, (2) inappropriate choice of variants as instruments, (3) insufficient interrogation of findings, (4) inappropriate interpretation of findings, and (5) lack of engagement with previous work. We have provided a brief checklist of key points to consider when performing a Mendelian randomization investigation; this does not replace previous guidance, but highlights critical analysis choices. Journal editors should be able to identify many low-quality submissions and reject papers without requiring peer review. Peer reviewers should focus initially on key indicators of validity; if a paper does not satisfy these, then the paper may be meaningless even if it is technically flawless.
CONCLUSIONS: Performing an informative Mendelian randomization investigation requires critical thought and collaboration between different specialties and fields of research.

While undisputedly important, and part of any systematic review (SR) by definition, evaluation of the risk of bias within the included studies is one of the most time-consuming parts of performing an SR. In this paper, we describe a case study comprising an extensive analysis of risk of bias (RoB) and reporting quality (RQ) assessment from a previously published review (CRD42021236047). It included both animal and human studies, and the included studies compared baseline diseased subjects with controls, assessed the effects of investigational treatments, or both. We compared RoB and RQ between the different types of included primary studies. We also assessed the \"informative value\" of each of the separate elements for meta-researchers, based on the notion that variation in reporting may be more interesting for the meta-researcher than consistently high/low or reported/non-reported scores. In general, reporting of experimental details was low. This resulted in frequent unclear risk-of-bias scores. We observed this both for animal and for human studies and both for disease-control comparisons and investigations of experimental treatments. Plots and explorative chi-square tests showed that reporting was slightly better for human studies of investigational treatments than for the other study types. With the evidence reported as is, risk-of-bias assessments for systematic reviews have low informative value other than repeatedly showing that reporting of experimental details needs to improve in all kinds of in vivo research. Particularly for reviews that do not directly inform treatment decisions, it could be efficient to perform a thorough but partial assessment of the quality of the included studies, either of a random subset of the included publications or of a subset of relatively informative elements, comprising, e.g. ethics evaluation, conflicts of interest statements, study limitations, baseline characteristics, and the unit of analysis. This publication suggests several potential procedures.

BACKGROUND: The Foods with Function Claim was introduced in Japan in April 2015 to make more products available that are labeled with health functions. A product\'s functionality of function claims must be explained by the scientific evidence presented in clinical trials (CTs) or systematic reviews, but the quality of recent CTs is unclear. The purpose of this study was to evaluate the risk of bias (RoB) using \"a revised tool to assess risk (RoB 2)\" published in 2018 for notifications based on all recent CTs published on the Consumer Affairs Agency website.
METHODS: A total of 38 submitted papers based on CTs that were published on the Consumer Affairs Agency website during the period from 1 January 2023 to 30 June 2024 were eligible. The RoB 2 tool provides a framework for considering the risk of bias in the findings of any type of randomized trial. This tool with five domains was used to evaluate the quality of research methods.
RESULTS: Eligible CTs were assessed as \"low risk\" (11%, n = 4), \"medium risk\" (13%, n = 5), and \"high risk\" (76%, n = 29). A number of highly biased papers were published. Bias occurred in all five domains, especially \"bias in selection of the reported result (Domain 5)\", which was the most serious (\"high risk\"; 75%). For elements correlated with RoB, there was no significant difference (p = 0.785) in the RoB 2 score between for-profit and academic research in the author\'s affiliated organization. There was no significant difference (p = 0.498) in the RoB score between the published year categories of 2000-2019 and 2020-2024, and no significant difference (p = 0.643) in the RoB score between English and Japanese language publications.
CONCLUSIONS: Overall, the quality of the latest CTs submitted after 2023 was very low, occurring in all five domains, and was most serious for \"bias in selection of the reported result (Domain 5)\".

暂无摘要。

Existing systems for automating the assessment of risk-of-bias (RoB) in medical studies are supervised approaches that require substantial training data to work well. However, recent revisions to RoB guidelines have resulted in a scarcity of available training data. In this study, we investigate the effectiveness of generative large language models (LLMs) for assessing RoB. Their application requires little or no training data and, if successful, could serve as a valuable tool to assist human experts during the construction of systematic reviews. Following Cochrane\'s latest guidelines (RoB2) designed for human reviewers, we prepare instructions that are fed as input to LLMs, which then infer the risk associated with a trial publication. We distinguish between two modelling tasks: directly predicting RoB2 from text; and employing decomposition, in which a RoB2 decision is made after the LLM responds to a series of signalling questions. We curate new testing data sets and evaluate the performance of four general- and medical-domain LLMs. The results fall short of expectations, with LLMs seldom surpassing trivial baselines. On the direct RoB2 prediction test set (n = 5993), LLMs perform akin to the baselines (F1: 0.1-0.2). In the decomposition task setup (n = 28,150), similar F1 scores are observed. Our additional comparative evaluation on RoB1 data also reveals results substantially below those of a supervised system. This testifies to the difficulty of solving this task based on (complex) instructions alone. Using LLMs as an assisting technology for assessing RoB2 thus currently seems beyond their reach.

OBJECTIVE: An important consideration when combining RCTs and NRSIs is how to address their potential biases in the pooled estimates. This study aimed to propose a Bayesian bias-adjusted random effects model for the synthesis of evidence from RCTs and NRSIs.
METHODS: We present a Bayesian bias-adjusted random effects model based on power prior method, which combines the likelihood contribution of the NRSIs, raised to the power parameter of alpha, with the likelihood of the RCT data, modeled with an additive bias. The method was illustrated using a meta-analysis on the association between low-dose methotrexate exposure and melanoma. We also combined RCTs and NRSIs using the naïve data synthesis.
RESULTS: The results including only RCTs has a posterior median and 95% credible interval (CrI) of 1.18 (0.31-4.04), the posterior probability of any harm (> 1.0) and a meaningful association (> 1.15) were 0.61 and 0.52, respectively. The posterior median and 95% CrI based on the naïve data synthesis resulted in 1.17 (0.96-1.47), and the posterior probability of any harm and a meaningful association were 0.96 and 0.60, respectively. For the Bayesian bias-adjusted analysis, the median OR was 1.16 (95% CrI: 0.83-1.71), and the posterior probabilities of any and a meaningful clinical association were 0.88 and 0.53, respectively.
CONCLUSIONS: The results indicated that integrating NRSIs into meta-analysis could increase the certainty of the body of evidence. However, directly combining RCTs and NRSIs in the same meta-analysis without distinction may lead to misleading conclusions.

BACKGROUND: Although aggregate data (AD) from randomised clinical trials (RCTs) are used in the majority of network meta-analyses (NMAs), other study designs (e.g., cohort studies and other non-randomised studies, NRS) can be informative about relative treatment effects. The individual participant data (IPD) of the study, when available, are preferred to AD for adjusting for important participant characteristics and to better handle heterogeneity and inconsistency in the network.
RESULTS: We developed the R package crossnma to perform cross-format (IPD and AD) and cross-design (RCT and NRS) NMA and network meta-regression (NMR). The models are implemented as Bayesian three-level hierarchical models using Just Another Gibbs Sampler (JAGS) software within the R environment. The R package crossnma includes functions to automatically create the JAGS model, reformat the data (based on user input), assess convergence and summarize the results. We demonstrate the workflow within crossnma by using a network of six trials comparing four treatments.
CONCLUSIONS: The R package crossnma enables the user to perform NMA and NMR with different data types in a Bayesian framework and facilitates the inclusion of all types of evidence recognising differences in risk of bias.

OBJECTIVE: The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, first published in 2009, has been widely endorsed and compliance is high in systematic reviews (SRs) of intervention studies. SRs of prevalence studies are increasing in frequency, but their characteristics and reporting quality have not been examined in large studies. Our objectives were to describe the characteristics of SRs of prevalence studies in adults, evaluate the completeness of reporting, and explore study-level characteristics associated with the completeness of reporting.
METHODS: We did a metaresearch study. We searched 5 databases from January 2010 to December 2020 to identify SRs of prevalence studies in adult populations. We used the PRISMA 2009 checklist to assess completeness of reporting and recorded additional characteristics. We conducted a descriptive analysis of review characteristics and linear regression to assess the relationship between compliance with PRISMA and publication characteristics.
RESULTS: We included 1172 SRs of prevalence studies. The number of reviews increased from 25 in 2010 to 273 in 2020. The median PRISMA score for SRs without meta-analysis was 17.5 of a maximum of 23, and for SRs with meta-analysis, 22 of a maximum of 25. Completeness of reporting, particularly for key items in the methods section, was suboptimal. SRs that included a meta-analysis or reported using a reporting or conduct guideline were the factors most strongly associated with increased compliance with PRISMA 2009.
CONCLUSIONS: Reporting of SRs of prevalence was adequate for many PRISMA items. Nonetheless, this study highlights aspects for which special attention is needed. Development of a specific tool to assess the risk of bias in prevalence studies and an extension to the PRISMA statement could improve the conduct and reporting of SRs of prevalence studies.

An assessment of the validity of studies is an essential component of most evidence syntheses (systematic reviews) to understand the risk of bias (ROB) and applicability of the evidence. A formal validity assessment requires a structured and comprehensive approach, which can be implemented using an assessment tool, specifically developed for this purpose. Many different tools are available, marking it difficult for researchers to choose the best tool for their evidence synthesis. We have established the LATITUDES Network to assist researchers in identifying the most appropriate tool to use in their evidence synthesis and to support researchers using these tools. The LATITUDES website (www.latitudes-network.org) includes a searchable library of validity assessment tools designed for use in evidence syntheses, bringing tools together in one place and providing researchers with clear information on suitable tools, categorized by study design. The website also provides links to training on the process of validity assessment and a list of tools currently under development. To be included in the LATITUDES library, tools must meet the following criteria: be designed for use in evidence syntheses; assess multidimensional aspects of validity of individual studies or reviews; and be developed for use by the wider research community rather than for a single research group. We highlight \'key\' tools, those that are considered to be the most robust and reliable tools based on prespecified criteria agreed in conjunction with our advisory board, an international group of experts in the area of evidence synthesis and ROB tools.

EFSA requested its Scientific Committee to prepare a guidance document on appraising and integrating evidence from epidemiological studies for use in EFSA\'s scientific assessments. The guidance document provides an introduction to epidemiological studies and illustrates the typical biases, which may be present in different epidemiological study designs. It then describes key epidemiological concepts relevant for evidence appraisal. This includes brief explanations for measures of association, exposure assessment, statistical inference, systematic error and effect modification. The guidance then describes the concept of external validity and the principles of appraising epidemiological studies. The customisation of the study appraisal process is explained including tailoring of tools for assessing the risk of bias (RoB). Several examples of appraising experimental and observational studies using a RoB tool are annexed to the document to illustrate the application of the approach. The latter part of this guidance focuses on different steps of evidence integration, first within and then across different streams of evidence. With respect to risk characterisation, the guidance considers how evidence from human epidemiological studies can be used in dose-response modelling with several different options being presented. Finally, the guidance addresses the application of uncertainty factors in risk characterisation when using evidence from human epidemiological studies.