Abstract：
OBJECTIVE: The study developed framework that leverages an open-source Large Language Model (LLM) to enable clinicians to ask plain-language questions about a patient\'s entire echocardiogram report history. This approach is intended to streamline the extraction of clinical insights from multiple echocardiogram reports, particularly in patients with complex cardiac diseases, thereby enhancing both patient care and research efficiency.
METHODS: Data from over 10 years were collected, comprising echocardiogram reports from patients with more than 10 echocardiograms on file at the Mount Sinai Health System. These reports were converted into a single document per patient for analysis, broken down into snippets and relevant snippets were retrieved using text similarity measures. The LLaMA-2 70B model was employed for analyzing the text using a specially crafted prompt. The model\'s performance was evaluated against ground-truth answers created by faculty cardiologists.
RESULTS: The study analyzed 432 reports from 37 patients for a total of 100 question-answer pairs. The LLM correctly answered 90% questions, with accuracies of 83% for temporality, 93% for severity assessment, 84% for intervention identification, and 100% for diagnosis retrieval. Errors mainly stemmed from the LLM\'s inherent limitations, such as misinterpreting numbers or hallucinations.
CONCLUSIONS: The study demonstrates the feasibility and effectiveness of using a local, open-source LLM for querying and interpreting echocardiogram report data. This approach offers a significant improvement over traditional keyword-based searches, enabling more contextually relevant and semantically accurate responses; in turn showing promise in enhancing clinical decision-making and research by facilitating more efficient access to complex patient data.
METHODS: Data from over 10 years were collected, comprising echocardiogram reports from patients with more than 10 echocardiograms on file at the Mount Sinai Health System. These reports were converted into a single document per patient for analysis, broken down into snippets and relevant snippets were retrieved using text similarity measures. The LLaMA-2 70B model was employed for analyzing the text using a specially crafted prompt. The model\'s performance was evaluated against ground-truth answers created by faculty cardiologists.
RESULTS: The study analyzed 432 reports from 37 patients for a total of 100 question-answer pairs. The LLM correctly answered 90% questions, with accuracies of 83% for temporality, 93% for severity assessment, 84% for intervention identification, and 100% for diagnosis retrieval. Errors mainly stemmed from the LLM\'s inherent limitations, such as misinterpreting numbers or hallucinations.
CONCLUSIONS: The study demonstrates the feasibility and effectiveness of using a local, open-source LLM for querying and interpreting echocardiogram report data. This approach offers a significant improvement over traditional keyword-based searches, enabling more contextually relevant and semantically accurate responses; in turn showing promise in enhancing clinical decision-making and research by facilitating more efficient access to complex patient data.
摘要：
目的：该研究开发了框架，该框架利用开源的大型语言模型（LLM），使临床医生能够对患者的整个超声心动图报告历史提出简单的问题。这种方法旨在简化从多个超声心动图报告中提取临床见解的过程。特别是在患有复杂心脏病的患者中，从而提高患者护理和研究效率。
方法：收集了超过10年的数据，包括在西奈山卫生系统存档的超过10个超声心动图的患者的超声心动图报告。这些报告被转换成每个患者的单一文件进行分析，分解为片段，并使用文本相似性度量检索相关片段。LLaMA-270B模型用于使用特制提示分析文本。该模型的性能是根据心脏病学家创建的地面实况答案进行评估的。
结果：该研究分析了37例患者的432份报告，共100份问答对。LLM正确回答了90%的问题，时间性的准确率为83%，93%用于严重程度评估，84%用于干预识别，100%用于诊断检索。错误主要源于LLM的固有限制，比如误解数字或幻觉。
结论：该研究证明了使用本地，用于查询和解释超声心动图报告数据的开源LLM。这种方法比传统的基于关键字的搜索有了显著的改进，实现更多上下文相关和语义上准确的反应;反过来，通过促进更有效地访问复杂的患者数据，在加强临床决策和研究方面显示出希望。
方法：收集了超过10年的数据，包括在西奈山卫生系统存档的超过10个超声心动图的患者的超声心动图报告。这些报告被转换成每个患者的单一文件进行分析，分解为片段，并使用文本相似性度量检索相关片段。LLaMA-270B模型用于使用特制提示分析文本。该模型的性能是根据心脏病学家创建的地面实况答案进行评估的。
结果：该研究分析了37例患者的432份报告，共100份问答对。LLM正确回答了90%的问题，时间性的准确率为83%，93%用于严重程度评估，84%用于干预识别，100%用于诊断检索。错误主要源于LLM的固有限制，比如误解数字或幻觉。
结论：该研究证明了使用本地，用于查询和解释超声心动图报告数据的开源LLM。这种方法比传统的基于关键字的搜索有了显著的改进，实现更多上下文相关和语义上准确的反应;反过来，通过促进更有效地访问复杂的患者数据，在加强临床决策和研究方面显示出希望。
