背景:尽管在实施方面存在不确定性,人工智能驱动的生成语言模型(GLM)在医学领域具有巨大的潜力。GLM的部署可以提高患者对临床文本的理解,并改善低健康素养。
目的:本研究的目的是评估ChatGPT-3.5和GPT-4的潜力,以适应患者特定输入教育水平的医疗信息的复杂性,这是至关重要的,如果它是作为解决低健康素养的工具。
方法:设计了与2种常见慢性疾病-II型糖尿病和高血压-相关的输入模板。针对假设的患者教育水平调整每个临床小插图,以评估输出个性化。要评估GLM(GPT-3.5和GPT-4)在定制输出编写方面的成功,使用Flesch阅读缓解评分(FKRE)和Flesch-Kincaid等级(FKGL)对转换前后输出的可读性进行量化.
结果:使用GPT-3.5和GPT-4在2个临床小插曲中产生反应(n=80)。对于GPT-3.5,FKRE平均值为57.75(SD4.75),51.28(标准差5.14),32.28(标准差4.52),六年级为28.31(SD5.22),8年级,高中,和单身汉,分别;FKGL平均得分为9.08(SD0.90),10.27(标准差1.06),13.4(标准差0.80),和13.74(标准差1.18)。GPT-3.5仅与学士学位的预设教育水平保持一致。相反,GPT-4的FKRE平均得分为74.54(SD2.6),71.25(标准差4.96),47.61(标准差6.13),和13.71(标准差5.77),FKGL平均得分为6.3(SD0.73),6.7(标准差1.11),11.09(标准差1.26),和17.03(标准差1.11),分别为相同的教育水平。GPT-4符合除6级FKRE平均值外的所有组的目标可读性。两种GLM的产出均具有统计学上的显着差异(P<.001;8年级P<.001;高中P<.001;学士P=.003;FKGL:6年级P=.001;8年级P<.001;高中P<.001;学士P<.001)。
结论:GLM可以根据输入指定的教育来改变医学文本输出的结构和可读性。然而,GLM将输入教育指定分类为3个广泛的输出可读性等级:容易(6年级和8年级),中等(高中),和困难(学士学位)。这是第一个结果表明GLM在输出文本简化方面的成功存在更广泛的界限。未来的研究必须确定GLM如何可靠地将医学文本个性化到预定的教育水平,以便对医疗保健素养产生更广泛的影响。
BACKGROUND: Although uncertainties exist regarding implementation, artificial intelligence-driven
generative language models (GLMs) have enormous potential in medicine. Deployment of GLMs could improve patient comprehension of clinical texts and improve low health literacy.
OBJECTIVE: The goal of this study is to evaluate the potential of ChatGPT-3.5 and GPT-4 to tailor the complexity of medical information to patient-specific input education level, which is crucial if it is to serve as a tool in addressing low health literacy.
METHODS: Input templates related to 2 prevalent chronic diseases-type II diabetes and hypertension-were designed. Each clinical vignette was adjusted for hypothetical patient education levels to evaluate output personalization. To assess the success of a GLM (GPT-3.5 and GPT-4) in tailoring output writing, the readability of pre- and posttransformation outputs were quantified using the Flesch reading ease score (FKRE) and the Flesch-Kincaid grade level (FKGL).
RESULTS: Responses (n=80) were generated using GPT-3.5 and GPT-4 across 2 clinical vignettes. For GPT-3.5, FKRE means were 57.75 (SD 4.75), 51.28 (SD 5.14), 32.28 (SD 4.52), and 28.31 (SD 5.22) for 6th grade, 8th grade, high school, and bachelor\'s, respectively; FKGL mean scores were 9.08 (SD 0.90), 10.27 (SD 1.06), 13.4 (SD 0.80), and 13.74 (SD 1.18). GPT-3.5 only aligned with the prespecified education levels at the bachelor\'s degree. Conversely, GPT-4\'s FKRE mean scores were 74.54 (SD 2.6), 71.25 (SD 4.96), 47.61 (SD 6.13), and 13.71 (SD 5.77), with FKGL mean scores of 6.3 (SD 0.73), 6.7 (SD 1.11), 11.09 (SD 1.26), and 17.03 (SD 1.11) for the same respective education levels. GPT-4 met the target readability for all groups except the 6th-grade FKRE average. Both GLMs produced outputs with statistically significant differences (P<.001; 8th grade P<.001; high school P<.001; bachelors P=.003; FKGL: 6th grade P=.001; 8th grade P<.001; high school P<.001; bachelors P<.001) between mean FKRE and FKGL across input education levels.
CONCLUSIONS: GLMs can change the structure and readability of medical text outputs according to input-specified education. However, GLMs categorize input education designation into 3 broad tiers of output readability: easy (6th and 8th grade), medium (high school), and difficult (bachelor\'s degree). This is the first result to suggest that there are broader boundaries in the success of GLMs in output text simplification. Future research must establish how GLMs can reliably personalize medical texts to prespecified education levels to enable a broader impact on health care literacy.