Abstract：
OBJECTIVE: The integration of these preventive guidelines with Electronic Health Records (EHRs) systems, coupled with the generation of personalized preventive care recommendations, holds significant potential for improving healthcare outcomes. Our study investigates the feasibility of using Large Language Models (LLMs) to automate the assessment criteria and risk factors from the guidelines for future analysis against medical records in EHR.
METHODS: We annotated the criteria, risk factors, and preventive medical services described in the adult guidelines published by United States Preventive Services Taskforce and evaluated 3 state-of-the-art LLMs on extracting information in these categories from the guidelines automatically.
RESULTS: We included 24 guidelines in this study. The LLMs can automate the extraction of all criteria, risk factors, and medical services from 9 guidelines. All 3 LLMs perform well on extracting information regarding the demographic criteria or risk factors. Some LLMs perform better on extracting the social determinants of health, family history, and preventive counseling services than the others.
CONCLUSIONS: While LLMs demonstrate the capability to handle lengthy preventive care guidelines, several challenges persist, including constraints related to the maximum length of input tokens and the tendency to generate content rather than adhering strictly to the original input. Moreover, the utilization of LLMs in real-world clinical settings necessitates careful ethical consideration. It is imperative that healthcare professionals meticulously validate the extracted information to mitigate biases, ensure completeness, and maintain accuracy.
CONCLUSIONS: We developed a data structure to store the annotated preventive guidelines and make it publicly available. Employing state-of-the-art LLMs to extract preventive care criteria, risk factors, and preventive care services paves the way for the future integration of these guidelines into the EHR.
METHODS: We annotated the criteria, risk factors, and preventive medical services described in the adult guidelines published by United States Preventive Services Taskforce and evaluated 3 state-of-the-art LLMs on extracting information in these categories from the guidelines automatically.
RESULTS: We included 24 guidelines in this study. The LLMs can automate the extraction of all criteria, risk factors, and medical services from 9 guidelines. All 3 LLMs perform well on extracting information regarding the demographic criteria or risk factors. Some LLMs perform better on extracting the social determinants of health, family history, and preventive counseling services than the others.
CONCLUSIONS: While LLMs demonstrate the capability to handle lengthy preventive care guidelines, several challenges persist, including constraints related to the maximum length of input tokens and the tendency to generate content rather than adhering strictly to the original input. Moreover, the utilization of LLMs in real-world clinical settings necessitates careful ethical consideration. It is imperative that healthcare professionals meticulously validate the extracted information to mitigate biases, ensure completeness, and maintain accuracy.
CONCLUSIONS: We developed a data structure to store the annotated preventive guidelines and make it publicly available. Employing state-of-the-art LLMs to extract preventive care criteria, risk factors, and preventive care services paves the way for the future integration of these guidelines into the EHR.
摘要：
目的：将这些预防指南与电子健康记录（EHRs）系统集成，加上个性化预防护理建议的产生，具有改善医疗保健结果的巨大潜力。我们的研究调查了使用大型语言模型（LLM）自动评估标准和风险因素的可行性，该指南用于未来对EHR医疗记录的分析。
方法：我们注释了标准，危险因素，和美国预防服务工作组发布的成人指南中描述的预防性医疗服务，并评估了3种最新的LLM自动从指南中提取这些类别的信息。
结果：我们在本研究中纳入了24条指南。LLM可以自动提取所有标准，危险因素，和9个指南的医疗服务。所有3个LLM在提取有关人口统计学标准或风险因素的信息方面表现良好。一些LLM在提取健康的社会决定因素方面表现更好，家族史,和预防性咨询服务比其他服务。
结论：虽然LLM证明了处理冗长的预防性护理指南的能力，几个挑战依然存在，包括与输入令牌的最大长度和生成内容而不是严格遵守原始输入的趋势相关的约束。此外，在现实世界的临床环境中使用LLM需要仔细的伦理考虑。医疗保健专业人员必须仔细验证提取的信息，以减轻偏见，确保完整性，保持准确性。
结论：我们开发了一种数据结构来存储注释的预防指南，并使其公开可用。采用最先进的LLM来提取预防性护理标准，危险因素，预防性护理服务为将来将这些指南纳入EHR铺平了道路。
方法：我们注释了标准，危险因素，和美国预防服务工作组发布的成人指南中描述的预防性医疗服务，并评估了3种最新的LLM自动从指南中提取这些类别的信息。
结果：我们在本研究中纳入了24条指南。LLM可以自动提取所有标准，危险因素，和9个指南的医疗服务。所有3个LLM在提取有关人口统计学标准或风险因素的信息方面表现良好。一些LLM在提取健康的社会决定因素方面表现更好，家族史,和预防性咨询服务比其他服务。
结论：虽然LLM证明了处理冗长的预防性护理指南的能力，几个挑战依然存在，包括与输入令牌的最大长度和生成内容而不是严格遵守原始输入的趋势相关的约束。此外，在现实世界的临床环境中使用LLM需要仔细的伦理考虑。医疗保健专业人员必须仔细验证提取的信息，以减轻偏见，确保完整性，保持准确性。
结论：我们开发了一种数据结构来存储注释的预防指南，并使其公开可用。采用最先进的LLM来提取预防性护理标准，危险因素，预防性护理服务为将来将这些指南纳入EHR铺平了道路。
