■大型语言模型(LLM)有可能提高从电子病历中的非结构化临床笔记中提取信息的效率。
■要评估LLM的实用性和可靠性,ChatGPT-4(OpenAI),分析临床叙述并确定在微移动相关事故中受伤的患者的头盔使用状况。
■这项公开可用的横断面研究,取消了美国消费品安全委员会国家电子伤害监测系统的2019年至2022年数据,美国96家医院的全国代表性分层概率样本。电动自行车的未加权估计,自行车,气垫板,以及导致急诊部门就诊的电动踏板车相关伤害。从2023年11月至2024年4月进行统计分析。
■使用以下方法从临床叙述中提取患者头盔状态(佩戴与未佩戴与未知):(1)使用研究人员生成的文本字符串进行文本字符串搜索,以及(2)通过提示系统使用低,中介-,和高细节提示。使用Cohenκ检验统计量分析了所有3个提示中2种方法之间的一致性水平。计算Fleissκ以测量5个新聊天会话和天的高细节提示的测试-重测可靠性。性能统计是通过将高细节提示的结果与研究人员阅读临床注释生成的头盔状态分类进行比较来计算的(即,标准标准审查)。
■在54569种临床笔记中,中等(Cohenκ=0.74[95%CI,0.73-0.75)和弱(Cohenκ=0.53[95%CI,0.52-0.54])一致在文本字符串搜索方法和LLM之间找到了低和中等细节提示,分别。高细节提示几乎完全一致(κ=1.00[95%CI,1.00-1.00]),但需要最大的时间才能完成。LLM并没有在新的疗程和天数内完美地复制其分析(Fleissκ=0.91,在5项试验中;P<.001)。LLM经常产生幻觉,并且在复制其幻觉方面是一致的。与标准标准(n=400;κ=0.98[95%CI,0.96-1.00])相比,它也显示出很高的有效性。
■这项研究的结果表明,尽管使用LLM从临床笔记中提取信息可以提高效率,与文本字符串搜索方法相比,可靠性不足,幻觉,和不一致的性能严重阻碍了当前可用的LLM的潜力。
UNASSIGNED: Large language models (LLMs) have potential to increase the efficiency of information extraction from unstructured clinical notes in electronic medical records.
UNASSIGNED: To assess the utility and reliability of an LLM, ChatGPT-4 (OpenAI), to analyze clinical narratives and identify helmet use status of patients injured in micromobility-related accidents.
UNASSIGNED: This cross-sectional study used publicly available, deidentified 2019 to 2022 data from the US Consumer Product Safety Commission\'s National Electronic Injury Surveillance System, a nationally representative stratified probability sample of 96 hospitals in the US. Unweighted estimates of e-bike, bicycle, hoverboard, and powered scooter-related injuries that resulted in an emergency department visit were used. Statistical analysis was performed from November 2023 to April 2024.
UNASSIGNED: Patient helmet status (wearing vs not wearing vs unknown) was extracted from clinical narratives using (1) a text string search using researcher-generated text strings and (2) the LLM by prompting the system with low-, intermediate-, and high-detail prompts. The level of agreement between the 2 approaches across all 3 prompts was analyzed using Cohen κ test statistics. Fleiss κ was calculated to measure the test-retest reliability of the high-detail prompt across 5 new chat sessions and days. Performance statistics were calculated by comparing results from the high-detail prompt to classifications of helmet status generated by researchers reading the clinical notes (ie, a criterion standard review).
UNASSIGNED: Among 54 569 clinical notes, moderate (Cohen κ = 0.74 [95% CI, 0.73-0.75) and weak (Cohen κ = 0.53 [95% CI, 0.52-0.54]) agreement were found between the text string-search approach and the LLM for the low- and intermediate-detail prompts, respectively. The high-detail prompt had almost perfect agreement (κ = 1.00 [95% CI, 1.00-1.00]) but required the greatest amount of time to complete. The LLM did not perfectly replicate its analyses across new sessions and days (Fleiss κ = 0.91 across 5 trials; P < .001). The LLM often hallucinated and was consistent in replicating its hallucinations. It also showed high validity compared with the criterion standard (n = 400; κ = 0.98 [95% CI, 0.96-1.00]).
UNASSIGNED: This study\'s findings suggest that although there are efficiency gains for using the LLM to extract information from clinical notes, the inadequate reliability compared with a text string-search approach, hallucinations, and inconsistent performance significantly hinder the potential of the currently available LLM.