背景:患者投诉是全球医疗机构面临的长期挑战,需要医护人员大量的时间和精力。尽管做出了这些努力,患者的不满情绪仍然很高。最近关于使用大型语言模型(LLM)的研究,例如OpenAI在医疗保健领域开发的GPT模型,已经显示出巨大的前景。与医生相比,能够提供更详细和同情的反应。LLM可能用于响应患者投诉,以提高患者满意度和投诉响应时间。
目的:本研究旨在评估LLM在解决三级医疗机构收到的患者投诉方面的表现,以提高患者满意度为目标。
方法:从患者关系科获得匿名患者投诉电子邮件和相关回复。ChatGPT-4.0(OpenAI,向Inc)提供了相同的投诉电子邮件,并负责生成回复。投诉和各自的答复被上传到基于网络的问卷上。受访者被要求在10点李克特量表上对4个项目的两个回答进行评分:适当性,完整性,同理心,和满意度。参与者还被要求在每个方案结束时选择一个首选的回答。
结果:共有188名受访者,其中115人(61.2%)是医护人员。大多数受访者,包括卫生保健和非卫生保健工作者,ChatGPT的首选答复(n=164,87.2%到n=183,97.3%)。与人类反应(适当性5,IQR3-7;同理心4,IQR3-6;质量5,IQR3-6;满意度5,IQR3-6;P<.001)相比,在所有4个评估项目中,GPT-4.0反应的中位数均为8分(IQR7-9),平均字数高于人类反应(比238个单词)。回归分析表明,在所有4个项目中,较高的单词计数是较高得分的统计学显着预测因子,每增加1个单词,得分增加0.015-0.019(所有P<.001)。然而,关于作者身份的亚组分析,这仅适用于由患者关系科工作人员撰写的回复,而不适用于由ChatGPT产生的回复,无论回复长度如何,这些回复均获得持续的高分.
结论:这项研究提供了支持LLM在解决患者投诉方面的有效性的重要证据。ChatGPT在反应适当性方面表现出优越性,同理心,质量,与人类对患者投诉的实际反应相比,以及总体满意度。可以进行未来的研究来衡量人工智能生成的响应在节省时间方面可以带来的改善程度,成本效益,患者满意度,和减轻医疗保健系统的压力。
BACKGROUND: Patient complaints are a perennial challenge faced by health care institutions globally, requiring extensive time and effort from health care workers. Despite these efforts, patient dissatisfaction remains high. Recent studies on the use of large language models (LLMs) such as the GPT models developed by OpenAI in the health care sector have shown great promise, with the ability to provide more detailed and empathetic responses as compared to physicians. LLMs could potentially be used in responding to patient complaints to improve patient satisfaction and complaint response time.
OBJECTIVE: This study aims to evaluate the performance of LLMs in addressing patient complaints received by a tertiary health care institution, with the goal of enhancing patient satisfaction.
METHODS: Anonymized patient complaint emails and associated responses from the patient relations department were obtained. ChatGPT-4.0 (OpenAI, Inc) was provided with the same complaint email and tasked to generate a response. The complaints and the respective responses were uploaded onto a web-based questionnaire. Respondents were asked to rate both responses on a 10-point Likert scale for 4 items: appropriateness, completeness, empathy, and satisfaction. Participants were also asked to choose a preferred response at the end of each scenario.
RESULTS: There was a total of 188 respondents, of which 115 (61.2%) were health care workers. A majority of the respondents, including both health care and non-health care workers, preferred replies from ChatGPT (n=164, 87.2% to n=183, 97.3%). GPT-4.0 responses were rated higher in all 4 assessed items with all median scores of 8 (IQR 7-9) compared to human responses (appropriateness 5, IQR 3-7; empathy 4, IQR 3-6; quality 5, IQR 3-6; satisfaction 5, IQR 3-6; P<.001) and had higher average word counts as compared to human responses (238 vs 76 words). Regression analyses showed that a higher word count was a statistically significant predictor of higher score in all 4 items, with every 1-word increment resulting in an increase in scores of between 0.015 and 0.019 (all P<.001). However, on subgroup analysis by authorship, this only held true for responses written by patient relations department staff and not those generated by ChatGPT which received consistently high scores irrespective of response length.
CONCLUSIONS: This study provides significant evidence supporting the effectiveness of LLMs in resolution of patient complaints. ChatGPT demonstrated superiority in terms of response appropriateness, empathy, quality, and overall satisfaction when compared against actual human responses to patient complaints. Future research can be done to measure the degree of improvement that artificial intelligence generated responses can bring in terms of time savings, cost-effectiveness, patient satisfaction, and stress reduction for the health care system.