Two reviewers examined publications indexed by Scopus, IEEE, MEDLINE, EMBASE, the ACM Digital Library, and the ACL Anthology. Publications reporting on NLP for mapping clinical text from EHRs to ontology concepts were included. Year, country, setting, objective, evaluation and validation methods, NLP algorithms, terminology systems, dataset size and language, performance measures, reference standard, generalizability, operational use, and source code availability were extracted. The studies\' objectives were categorized by way of induction. These results were used to define recommendations.
Two thousand three hundred fifty five unique studies were identified. Two hundred fifty six studies reported on the development of NLP algorithms for mapping free text to ontology concepts. Seventy-seven described development and evaluation. Twenty-two studies did not perform a validation on unseen data and 68 studies did not perform external validation. Of 23 studies that claimed that their algorithm was generalizable, 5 tested this by external validation. A list of sixteen recommendations regarding the usage of NLP systems and algorithms, usage of data, evaluation and validation, presentation of results, and generalizability of results was developed.
We found many heterogeneous approaches to the reporting on the development and evaluation of NLP algorithms that map clinical text to ontology concepts. Over one-fourth of the identified publications did not perform an evaluation. In addition, over one-fourth of the included studies did not perform a validation, and 88% did not perform external validation. We believe that our recommendations, alongside an existing reporting standard, will increase the reproducibility and reusability of future studies and NLP algorithms in medicine.
两位审稿人检查了Scopus索引的出版物,IEEE,MEDLINE,EMBASE,ACM数字图书馆,和ACL选集。包括有关NLP的出版物,用于将临床文本从EHR映射到本体论概念。Year,国家,设置,目标,评估和验证方法,NLP算法,术语系统,数据集大小和语言,绩效指标,参考标准,概括性,操作使用,并提取了源代码可用性。这些研究的目标是通过归纳的方式进行分类的。这些结果用于定义建议。
确定了两千三百五十五个独特的研究。256项研究报告了将自由文本映射到本体概念的NLP算法的开发。77项描述了发展和评价。22项研究未对未知数据进行验证,68项研究未进行外部验证。在23项声称他们的算法是可推广的研究中,5通过外部验证对此进行了测试。关于使用NLP系统和算法的16项建议列表,数据的使用,评估和验证,结果的介绍,并开发了结果的普适性。
我们发现了许多异构方法来报告NLP算法的开发和评估,这些算法将临床文本映射到本体概念。超过四分之一的已确定出版物没有进行评估。此外,超过四分之一的纳入研究没有进行验证,88%未进行外部验证。我们认为,我们的建议,除了现有的报告标准之外,将增加未来研究和NLP算法在医学中的可重复性和可重用性。