PDF(Sci-hub)
Abstract：
Free-text descriptions in electronic health records (EHRs) can be of interest for clinical research and care optimization. However, free text cannot be readily interpreted by a computer and, therefore, has limited value. Natural Language Processing (NLP) algorithms can make free text machine-interpretable by attaching ontology concepts to it. However, implementations of NLP algorithms are not evaluated consistently. Therefore, the objective of this study was to review the current methods used for developing and evaluating NLP algorithms that map clinical text fragments onto ontology concepts. To standardize the evaluation of algorithms and reduce heterogeneity between studies, we propose a list of recommendations.
Two reviewers examined publications indexed by Scopus, IEEE, MEDLINE, EMBASE, the ACM Digital Library, and the ACL Anthology. Publications reporting on NLP for mapping clinical text from EHRs to ontology concepts were included. Year, country, setting, objective, evaluation and validation methods, NLP algorithms, terminology systems, dataset size and language, performance measures, reference standard, generalizability, operational use, and source code availability were extracted. The studies\' objectives were categorized by way of induction. These results were used to define recommendations.
Two thousand three hundred fifty five unique studies were identified. Two hundred fifty six studies reported on the development of NLP algorithms for mapping free text to ontology concepts. Seventy-seven described development and evaluation. Twenty-two studies did not perform a validation on unseen data and 68 studies did not perform external validation. Of 23 studies that claimed that their algorithm was generalizable, 5 tested this by external validation. A list of sixteen recommendations regarding the usage of NLP systems and algorithms, usage of data, evaluation and validation, presentation of results, and generalizability of results was developed.
We found many heterogeneous approaches to the reporting on the development and evaluation of NLP algorithms that map clinical text to ontology concepts. Over one-fourth of the identified publications did not perform an evaluation. In addition, over one-fourth of the included studies did not perform a validation, and 88% did not perform external validation. We believe that our recommendations, alongside an existing reporting standard, will increase the reproducibility and reusability of future studies and NLP algorithms in medicine.
Two reviewers examined publications indexed by Scopus, IEEE, MEDLINE, EMBASE, the ACM Digital Library, and the ACL Anthology. Publications reporting on NLP for mapping clinical text from EHRs to ontology concepts were included. Year, country, setting, objective, evaluation and validation methods, NLP algorithms, terminology systems, dataset size and language, performance measures, reference standard, generalizability, operational use, and source code availability were extracted. The studies\' objectives were categorized by way of induction. These results were used to define recommendations.
Two thousand three hundred fifty five unique studies were identified. Two hundred fifty six studies reported on the development of NLP algorithms for mapping free text to ontology concepts. Seventy-seven described development and evaluation. Twenty-two studies did not perform a validation on unseen data and 68 studies did not perform external validation. Of 23 studies that claimed that their algorithm was generalizable, 5 tested this by external validation. A list of sixteen recommendations regarding the usage of NLP systems and algorithms, usage of data, evaluation and validation, presentation of results, and generalizability of results was developed.
We found many heterogeneous approaches to the reporting on the development and evaluation of NLP algorithms that map clinical text to ontology concepts. Over one-fourth of the identified publications did not perform an evaluation. In addition, over one-fourth of the included studies did not perform a validation, and 88% did not perform external validation. We believe that our recommendations, alongside an existing reporting standard, will increase the reproducibility and reusability of future studies and NLP algorithms in medicine.
摘要：
电子健康记录（EHR）中的自由文本描述可能对临床研究和护理优化感兴趣。然而,自由文本不容易被计算机解释，因此,价值有限。自然语言处理（NLP）算法可以通过向其附加本体概念来使自由文本机器可解释。然而,NLP算法的实现并没有得到一致的评估。因此,本研究的目的是回顾当前用于开发和评估将临床文本片段映射到本体概念的NLP算法的方法.为了规范算法的评估，减少研究之间的异质性，我们提出了一份建议清单。
两位审稿人检查了Scopus索引的出版物，IEEE,MEDLINE,EMBASE,ACM数字图书馆,和ACL选集。包括有关NLP的出版物，用于将临床文本从EHR映射到本体论概念。Year,国家,设置,目标,评估和验证方法，NLP算法，术语系统，数据集大小和语言，绩效指标，参考标准,概括性，操作使用,并提取了源代码可用性。这些研究的目标是通过归纳的方式进行分类的。这些结果用于定义建议。
确定了两千三百五十五个独特的研究。256项研究报告了将自由文本映射到本体概念的NLP算法的开发。77项描述了发展和评价。22项研究未对未知数据进行验证，68项研究未进行外部验证。在23项声称他们的算法是可推广的研究中，5通过外部验证对此进行了测试。关于使用NLP系统和算法的16项建议列表，数据的使用，评估和验证，结果的介绍，并开发了结果的普适性。
我们发现了许多异构方法来报告NLP算法的开发和评估，这些算法将临床文本映射到本体概念。超过四分之一的已确定出版物没有进行评估。此外,超过四分之一的纳入研究没有进行验证，88%未进行外部验证。我们认为，我们的建议，除了现有的报告标准之外，将增加未来研究和NLP算法在医学中的可重复性和可重用性。
两位审稿人检查了Scopus索引的出版物，IEEE,MEDLINE,EMBASE,ACM数字图书馆,和ACL选集。包括有关NLP的出版物，用于将临床文本从EHR映射到本体论概念。Year,国家,设置,目标,评估和验证方法，NLP算法，术语系统，数据集大小和语言，绩效指标，参考标准,概括性，操作使用,并提取了源代码可用性。这些研究的目标是通过归纳的方式进行分类的。这些结果用于定义建议。
确定了两千三百五十五个独特的研究。256项研究报告了将自由文本映射到本体概念的NLP算法的开发。77项描述了发展和评价。22项研究未对未知数据进行验证，68项研究未进行外部验证。在23项声称他们的算法是可推广的研究中，5通过外部验证对此进行了测试。关于使用NLP系统和算法的16项建议列表，数据的使用，评估和验证，结果的介绍，并开发了结果的普适性。
我们发现了许多异构方法来报告NLP算法的开发和评估，这些算法将临床文本映射到本体概念。超过四分之一的已确定出版物没有进行评估。此外,超过四分之一的纳入研究没有进行验证，88%未进行外部验证。我们认为，我们的建议，除了现有的报告标准之外，将增加未来研究和NLP算法在医学中的可重复性和可重用性。
