PDF(Pubmed)
Abstract：
Accurate predictive modeling of pandemics is essential for optimally distributing biomedical resources and setting policy. Dozens of case prediction models have been proposed but their accuracy over time and by model type remains unclear. In this study, we systematically analyze all US CDC COVID-19 forecasting models, by first categorizing them and then calculating their mean absolute percent error, both wave-wise and on the complete timeline. We compare their estimates to government-reported case numbers, one another, as well as two baseline models wherein case counts remain static or follow a simple linear trend. The comparison reveals that around two-thirds of models fail to outperform a simple static case baseline and one-third fail to outperform a simple linear trend forecast. A wave-by-wave comparison of models revealed that no overall modeling approach was superior to others, including ensemble models and errors in modeling have increased over time during the pandemic. This study raises concerns about hosting these models on official public platforms of health organizations including the US CDC which risks giving them an official imprimatur and when utilized to formulate policy. By offering a universal evaluation method for pandemic forecasting models, we expect this study to serve as the starting point for the development of more accurate models.
摘要：
流行病的准确预测模型对于优化分配生物医学资源和制定政策至关重要。已经提出了数十种病例预测模型，但它们随时间和模型类型的准确性仍不清楚。在这项研究中,我们系统分析了美国疾控中心所有的COVID-19预测模型，首先对它们进行分类，然后计算它们的平均绝对百分比误差，波浪式和完整的时间表。我们将他们的估计与政府报告的病例数进行比较，彼此,以及两个基线模型，其中病例计数保持静态或遵循简单的线性趋势。比较显示，大约三分之二的模型无法超过简单的静态案例基线，三分之一的模型无法超过简单的线性趋势预测。模型的逐波比较表明，没有任何整体建模方法优于其他建模方法，包括集成模型和建模中的错误在大流行期间随着时间的推移而增加。这项研究引起了人们对在包括美国疾病预防控制中心在内的卫生组织的官方公共平台上托管这些模型的担忧，这些模型可能会给它们一个官方的认可，并用于制定政策。通过为大流行预测模型提供通用的评估方法，我们希望这项研究能够成为开发更准确模型的起点。
