这项研究评估了自然语言处理(NLP)工具的准确性和可移植性,该工具用于从两个大型医疗保健系统的临床笔记中提取流感的临床发现。评估了NLP对下游流感病例检测进行疾病监测的支持程度。
我们独立开发了两个NLP解析器,一个在犹他州的IntermountainHealthcare(IH),另一个在匹兹堡大学医学中心(UPMC),使用急诊科(ED)遇到流感的当地临床记录。我们测量了存在和不存在指示流感的70个临床发现的NLP解析器性能。然后,我们从NLP处理的报告中开发了贝叶斯网络模型,并测试了它们区分(1)流感病例的能力,(2)非流感流感样疾病(NI-ILI),和(3)\'其他\'诊断。
关于IntermountainHealthcare的报告,IHNLP解析器的召回率和精确度分别为0.71和0.75,和UPMCNLP解析器,0.67和0.79。匹兹堡大学医学中心报告,UPMCNLP解析器的召回率和准确率分别为0.73和0.80,和IHNLP解析器,0.53和0.80。在IntermountainHealthcare病例中,通过AUROC对流感与非流感进行测量的贝叶斯病例检测性能为0.93(使用IHNLP解析器)和0.93(使用UPMCNLP解析器)。匹兹堡大学医学中心病例的病例检测为0.95(使用UPMCNLP解析器)和0.83(使用IHNLP解析器)。对于流感与NI-ILI在IntermountainHealthcare病例中的表现分别为0.70(使用IHNLP解析器)和0.76(使用UPMCNLP解析器)。关于皮斯堡大学医学中心的案例,0.76(使用UPMCNLP解析器)和0.65(使用IHNLP解析器)。
在所有情况下(使用IH病例的流感与NI-ILI),尽管非本地解析器的性能合理,但本地解析器在支持案例检测方面更有效。
This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza
case-detection for disease surveillance.
We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed
reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) \'other\' diagnosis.
On Intermountain Healthcare
reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center
reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser).
Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser).
In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.