背景:像ChatGPT这样的大型语言模型(LLM)已经变得越来越普遍。在医学上,许多潜在的领域出现了LLM可能提供附加值的地方。我们的研究重点是使用开源LLM替代品,如Llama3,Gemma,米斯特拉尔,和Mixtral从德国临床文献中提取医学参数。由于观察到非英语任务的研究差距,我们专注于德语。
目的:评估开源LLM从德国临床文本中提取医学参数的有效性,特别关注心脏MRI报告中的心血管功能指标。
方法:我们提取了14项心血管功能指标,包括左心室和右心室射血分数(LV-EF和RV-EF),来自497份不同配方的心脏磁共振成像(MRI)报告。我们的系统分析涉及评估拉玛3号,杰玛,米斯特拉尔,和混合模型在正确注释和命名实体识别(NER)准确性方面。
结果:分析证实了在不同架构中具有高达95.4%的正确注释和99.8%的NER准确性的强大性能,尽管这些模型没有针对数据提取和德语进行明确的微调。
结论:结果强烈建议使用开源LLM从临床文本中提取医学参数,包括德语,由于它们的高精度和有效性,即使没有具体的微调。
BACKGROUND: Large Language Models (LLMs) like ChatGPT have become increasingly prevalent. In medicine, many potential areas arise where LLMs may offer added value. Our research focuses on the use of open-source LLM alternatives like Llama 3, Gemma,
Mistral, and Mixtral to extract medical parameters from German clinical texts. We concentrate on German due to an observed gap in research for non-English tasks.
OBJECTIVE: To evaluate the effectiveness of open-source LLMs in extracting medical parameters from German clinical texts, specially focusing on cardiovascular function indicators from cardiac MRI reports.
METHODS: We extracted 14 cardiovascular function indicators, including left and right ventricular ejection fraction (LV-EF and RV-EF), from 497 variously formulated cardiac magnetic resonance imaging (MRI) reports. Our systematic analysis involved assessing the performance of Llama 3, Gemma,
Mistral, and Mixtral models in terms of right annotation and named entity recognition (NER) accuracy.
RESULTS: The analysis confirms strong performance with up to 95.4% right annotation and 99.8% NER accuracy across different architectures, despite the fact that these models were not explicitly fine-tuned for data extraction and the German language.
CONCLUSIONS: The results strongly recommend using open-source LLMs for extracting medical parameters from clinical texts, including those in German, due to their high accuracy and effectiveness even without specific fine-tuning.