目的:临床记录包含患者病史的非结构化表示,包括医疗问题和处方药之间的关系。探讨癌症药物与相关症状负担之间的关系,我们提取结构化的,来自肿瘤学笔记临床叙述的医学问题和药物信息的语义表示。
方法:我们提出了癌症事件和关系(CACER)的临床概念注释,一种新颖的语料库,具有超过48.000个医学问题和药物事件的细粒度注释,以及10.000个药物-问题和问题-问题关系。利用CACER,我们开发和评估基于变压器的信息提取模型,如变压器的双向编码器表示(BERT),微调语言网络文本到文本转换转换器(Flan-T5),大型语言模型元AI(Llama3),以及使用微调和上下文学习(ICL)的生成预训练变压器4(GPT-4)。
结果:在事件提取中,经过微调的BERT和Llama3型号在88.2-88.0F1达到了最高性能,与88.4F1的注释者间协议(IAA)相当。在关系提取中,微调的BERT,Flan-T5和Llama3在61.8-65.3F1达到了最高性能。带有ICL的GPT-4在两个任务中都实现了最差的性能。
结论:在ICL中,微调模型的性能明显优于GPT-4,强调带注释的训练数据和模型优化的重要性。此外,BERT模型的表现与Llama3相似。为了我们的任务,大型语言模型与较小的BERT模型相比没有性能优势。
结论:我们介绍CACER,一种新颖的语料库,具有针对医学问题的细粒度注释,毒品,以及它们在肿瘤学临床叙事中的关系。最先进的变压器模型在多个提取任务中实现了与IAA相当的性能。
OBJECTIVE: Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes.
METHODS: We present Clinical concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48 000 medical problems and drug events and 10 000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction models such as Bidirectional Encoder Representations from Transformers (BERT), Fine-tuned Language Net Text-To-Text Transfer Transformer (Flan-T5), Large Language Model Meta AI (Llama3), and Generative Pre-trained Transformers-4 (GPT-4) using fine-tuning and in-context learning (ICL).
RESULTS: In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks.
CONCLUSIONS: The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, large language models offer no performance advantage over the smaller BERT models.
CONCLUSIONS: We introduce CACER, a novel corpus with fine-grained annotations for medical problems, drugs, and their relationships in clinical narratives of oncology notes. State-of-the-art transformer models achieved performance comparable to IAA for several extraction tasks.