量化性能中的权衡,再现性,以及开发临床相关词嵌入的几种策略的资源需求。
我们在PubmedCentral(PMC)开放获取子集中对所有全文手稿进行了单独的嵌入训练,其中的病例报告,英文维基百科语料库,重症监护医疗信息集市(MIMIC)III数据集,以及宾夕法尼亚大学卫生系统(UPHS)电子健康记录中的所有注释。我们在六个临床相关任务中测试了嵌入,包括死亡率预测和去识别,并使用缩放的Brier得分(SBS)和成功取消识别的笔记比例来评估性能,分别。
来自UPHS的嵌入注意到最好的预测死亡率(SBS0.30,95%CI0.15至0.45),而维基百科嵌入表现最差(SBS0.12,95%CI-0.05至0.28)。维基百科嵌入最一致(78%的笔记)和完整的PMC语料库嵌入最不一致(48%)去识别笔记。在所有六项任务中,完整的PMC语料库展示了最一致的性能,维基百科语料库最少。语料库大小从4900万个令牌(PMC病例报告)到100亿(UPHS)不等。
在大多数任务中,在已发布的病例报告上训练的嵌入以及在其他语料库上训练的嵌入最少。临床语料库的表现始终优于非临床语料库。没有一个语料库在所有任务中都产生了严格占主导地位的嵌入集,因此最佳的训练语料库取决于预期的用途。
在已发布的病例报告上训练的嵌入在大多数临床任务上的表现与在较大语料库上训练的嵌入相当。开放获取语料库允许对临床相关的培训,有效,和可重复的嵌入。
Quantify tradeoffs in performance, reproducibility, and resource demands across several strategies for developing clinically relevant word embeddings.
We trained separate embeddings on all full-text manuscripts in the
Pubmed Central (PMC) Open Access subset,
case reports therein, the English Wikipedia corpus, the Medical Information Mart for Intensive Care (MIMIC) III dataset, and all notes in the University of Pennsylvania Health System (UPHS) electronic health record. We tested embeddings in six clinically relevant tasks including mortality prediction and de-identification, and assessed performance using the scaled Brier score (SBS) and the proportion of notes successfully de-identified, respectively.
Embeddings from UPHS notes best predicted mortality (SBS 0.30, 95% CI 0.15 to 0.45) while Wikipedia embeddings performed worst (SBS 0.12, 95% CI -0.05 to 0.28). Wikipedia embeddings most consistently (78% of notes) and the full PMC corpus embeddings least consistently (48%) de-identified notes. Across all six tasks, the full PMC corpus demonstrated the most consistent performance, and the Wikipedia corpus the least. Corpus size ranged from 49 million tokens (PMC
case reports) to 10 billion (UPHS).
Embeddings trained on published
case reports performed as least as well as embeddings trained on other corpora in most tasks, and clinical corpora consistently outperformed non-clinical corpora. No single corpus produced a strictly dominant set of embeddings across all tasks and so the optimal training corpus depends on intended use.
Embeddings trained on published
case reports performed comparably on most clinical tasks to embeddings trained on larger corpora. Open access corpora allow training of clinically relevant, effective, and reproducible embeddings.