PDF(Pubmed)
Abstract：
UNASSIGNED: To automate scientific claim verification using PubMed abstracts.
UNASSIGNED: We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER\'s performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021.
UNASSIGNED: In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively.
UNASSIGNED: CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.
UNASSIGNED: We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER\'s performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021.
UNASSIGNED: In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively.
UNASSIGNED: CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.
摘要：
■使用PubMed摘要自动进行科学索赔验证。
■我们开发了CliVER，一个端到端的科学索赔验证系统，利用检索增强技术自动检索相关的临床试验摘要，提取相关的句子，并使用PICO框架来支持或反驳科学主张。我们还创建了三个最先进的深度学习模型的集合，以对支持的基本原理进行分类，反驳,中立。然后我们建造了Covert，一个新的COVID验证数据集,包括15项PICO编码的药物声明,以及96份人工选择和标记的临床试验摘要,这些摘要支持或驳斥了每项声明.我们使用CoVERT和SciFact（一个公共科学声明验证数据集）来评估CliVER在预测标签方面的表现。最后,我们将CliVER与临床医生进行了比较，验证了来自6个疾病领域的19项索赔，使用189.648PubMed摘要提取2010年1月至2021年10月。
■在CoVERT上评估标签预测精度时，CliVER的F1得分为0.92，突显了检索增强模型的功效。集成模型在F1评分中从3％到11％的绝对增加优于每个单独的最新模型。此外，与四名临床医生相比，CliVER的摘要检索精度为79.0%，67.4%的句子选择，标签预测为63.2%，分别。
■CliVER展示了其使用检索增强策略自动进行科学声明验证的早期潜力，以利用PubMed中丰富的临床试验摘要。未来的研究有必要进一步测试其临床实用性。
■我们开发了CliVER，一个端到端的科学索赔验证系统，利用检索增强技术自动检索相关的临床试验摘要，提取相关的句子，并使用PICO框架来支持或反驳科学主张。我们还创建了三个最先进的深度学习模型的集合，以对支持的基本原理进行分类，反驳,中立。然后我们建造了Covert，一个新的COVID验证数据集,包括15项PICO编码的药物声明,以及96份人工选择和标记的临床试验摘要,这些摘要支持或驳斥了每项声明.我们使用CoVERT和SciFact（一个公共科学声明验证数据集）来评估CliVER在预测标签方面的表现。最后,我们将CliVER与临床医生进行了比较，验证了来自6个疾病领域的19项索赔，使用189.648PubMed摘要提取2010年1月至2021年10月。
■在CoVERT上评估标签预测精度时，CliVER的F1得分为0.92，突显了检索增强模型的功效。集成模型在F1评分中从3％到11％的绝对增加优于每个单独的最新模型。此外，与四名临床医生相比，CliVER的摘要检索精度为79.0%，67.4%的句子选择，标签预测为63.2%，分别。
■CliVER展示了其使用检索增强策略自动进行科学声明验证的早期潜力，以利用PubMed中丰富的临床试验摘要。未来的研究有必要进一步测试其临床实用性。
