背景:营养不良是老年护理机构(RACF)中普遍存在的问题,导致不良健康结果。从电子健康记录(EHR)的大量数据中有效提取关键临床信息的能力可以提高对问题严重程度的理解并制定有效的干预措施。这项研究旨在测试零射提示工程应用于生成人工智能(AI)模型的有效性,并结合检索增强生成(RAG)。用于在EHR中汇总结构化和非结构化数据并提取重要营养不良信息的自动化任务。
方法:我们使用了带零射提示的Llama213B模型。该数据集包括40个澳大利亚RACF中与营养不良管理相关的非结构化和结构化EHR。我们首先只对模型进行零射学习,然后将其与RAG相结合以完成两项任务:生成有关客户营养状况的结构化摘要,并提取有关营养不良风险因素的关键信息。我们在第一个任务中使用了25个音符,在第二个任务中使用了1,399个音符。我们根据黄金标准数据集手动评估了每个任务的模型输出。
结果:评估结果表明,应用于生成AI模型的零射学习在总结和提取有关RACF客户营养状况的信息方面非常有效。生成的摘要提供了原始数据的简洁和准确的表示,总体准确率为93.25%。RAG的加入改进了总结过程,导致6%的增长,达到99.25%的精度。该模型还证明了其提取风险因素的能力,准确率为90%。然而,添加RAG并没有进一步提高这项任务的准确性.总的来说,当信息在注释中明确说明时,该模型显示出稳健的性能;然而,它可能会遇到幻觉限制,特别是当细节没有明确提供时。
结论:这项研究证明了将零射学习应用于生成AI模型以自动生成EHR数据的结构化摘要并提取关键临床信息的高性能和局限性。RAG方法的加入提高了模型性能并减轻了幻觉问题。
BACKGROUND: Malnutrition is a prevalent issue in aged care facilities (RACFs), leading to adverse health outcomes. The ability to efficiently extract key clinical information from a large volume of data in electronic health records (EHR) can improve understanding about the extent of the problem and developing effective interventions. This research aimed to test the efficacy of zero-shot prompt engineering applied to generative artificial intelligence (AI) models on their own and in combination with retrieval augmented generation (RAG), for the automating tasks of summarizing both structured and unstructured data in EHR and extracting important malnutrition information.
METHODS: We utilized Llama 2 13B model with zero-shot prompting. The dataset comprises unstructured and structured EHRs related to malnutrition management in 40 Australian RACFs. We employed zero-shot learning to the model alone first, then combined it with RAG to accomplish two tasks: generate structured summaries about the nutritional status of a client and extract key information about malnutrition risk factors. We utilized 25 notes in the first task and 1,399 in the second task. We evaluated the model\'s output of each task manually against a gold standard dataset.
RESULTS: The evaluation outcomes indicated that zero-shot learning applied to generative AI model is highly effective in summarizing and extracting information about nutritional status of RACFs\' clients. The generated summaries provided concise and accurate representation of the original data with an overall accuracy of 93.25%. The addition of RAG improved the summarization process, leading to a 6% increase and achieving an accuracy of 99.25%. The model also proved its capability in extracting risk factors with an accuracy of 90%. However, adding RAG did not further improve accuracy in this task. Overall, the model has shown a robust performance when information was explicitly stated in the notes; however, it could encounter hallucination limitations, particularly when details were not explicitly provided.
CONCLUSIONS: This study demonstrates the high performance and limitations of applying zero-shot learning to generative AI models to automatic generation of structured summarization of EHRs data and extracting key clinical information. The inclusion of the RAG approach improved the model performance and mitigated the hallucination problem.