Abstract：
Predictions under interventions are estimates of what a person\'s risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.
摘要：
干预措施下的预测是对一个人的风险的估计，如果他们遵循特定的治疗策略，考虑到他们的个性。这样的预测可以为医疗决策提供重要的输入。然而,评估介入预测的预测性能是具有挑战性的。使用观察数据时，评估预测性能的标准方法不适用，因为干预下的预测涉及在与验证数据集中个体子集观察到的条件不同的条件下获得结果的预测。这项工作描述了在干预措施下评估预测的反事实表现的方法。这意味着如果所有个体都遵循进行预测的治疗策略，我们的目标是评估预测与验证数据的匹配程度。我们专注于使用纵向观测数据进行反事实绩效评估，以及在涉及随着时间的推移维持特定治疗方案的治疗策略下。我们引入了一种使用人工审查和逆概率加权的估计方法，该方法涉及创建一个模拟治疗策略的验证数据集，在该策略下进行预测。我们扩展了校准措施，歧视（c指数和累积/动态AUCt）和总体预测误差（Brier评分），以评估反事实表现。这些方法是使用模拟研究进行评估的，包括方法应检测性能不佳的场景。在肝移植的背景下应用我们的方法表明，我们的程序可以量化支持器官分配关键决策的预测性能。
