■电子健康记录文本来源,如药物标记(sigs)包含有价值的信息,这些信息并不总是以结构化形式提供。通常通过手动注释处理,这个重复和耗时的任务可以使用大型语言模型(LLM)完全自动化。虽然大多数sigs包括简单的指令,有些包括复杂的模式。
■我们旨在将GPT-3.5和GPT-4的性能与较小的微调型号进行比较(ClinicalBERT,Bluebert)在提取2种具有频繁复杂sigs的免疫调节药物的平均日剂量:羟氯喹,和泼尼松。
■使用手动注释的sigs作为黄金标准,我们比较了这些模型在702份羟氯喹和22.104份泼尼松处方中的表现.
■GPT-4在任何级别的上下文学习中都大大优于此任务的所有其他模型。有100个上下文示例,该模型正确地注释了94%的羟氯喹和95%的泼尼松sigs在1个有效数字以内。由另外2个手动注释器对注释器模型分歧进行的错误分析表明,绝大多数分歧都是模型错误。许多模型错误与模棱两可的sig有关,在这些sig上也经常存在注释者的分歧。
■与最少的手动注释配对,GPT-4在复杂药物治疗的语言回归方面取得了出色的表现,并且大大优于GPT-3.5,ClinicalBERT,还有Bluebert.然而,达到最大性能所需的上下文示例数量与GPT-3.5相似.
■LLM显示出巨大的潜力,可以以无代码的方式从sigs中快速提取结构化数据,用于临床和研究应用。
UNASSIGNED: Electronic health record textual sources such as medication signeturs (sigs) contain valuable information that is not always available in structured form. Commonly processed through manual annotation, this repetitive and time-consuming task could be fully automated using large language models (LLMs). While most sigs include simple instructions, some include complex patterns.
UNASSIGNED: We aimed to compare the performance of GPT-3.5 and GPT-4 with smaller fine-tuned models (ClinicalBERT, BlueBERT) in extracting the average daily dose of 2 immunomodulating medications with frequent complex sigs: hydroxychloroquine, and prednisone.
UNASSIGNED: Using manually annotated sigs as the gold standard, we compared the performance of these models in 702 hydroxychloroquine and 22 104 prednisone prescriptions.
UNASSIGNED: GPT-4 vastly outperformed all other models for this task at any level of in-context learning. With 100 in-context examples, the model correctly annotates 94% of hydroxychloroquine and 95% of prednisone sigs to within 1 significant digit. Error analysis conducted by 2 additional manual annotators on annotator-model disagreements suggests that the vast majority of disagreements are model errors. Many model errors relate to ambiguous sigs on which there was also frequent annotator disagreement.
UNASSIGNED: Paired with minimal manual annotation, GPT-4 achieved excellent performance for language regression of complex medication sigs and vastly outperforms GPT-3.5, ClinicalBERT, and BlueBERT. However, the number of in-context examples needed to reach maximum performance was similar to GPT-3.5.
UNASSIGNED: LLMs show great potential to rapidly extract structured data from sigs in no-code fashion for clinical and research applications.