目的:搜索在线健康信息是一种流行的方法,被患者用来提高他们对疾病的认识。最近开发的AI聊天机器人可能是这方面最简单的方法。该研究的目的是根据癌症患者中最常用的放射性核素治疗来分析AI聊天机器人反应的可靠性和可读性。
方法:基本患者问题,大约三十RAI,PRRT和TARE治疗和29个关于PSMA-TRT,在2024年1月被一个接一个地问GPT-4和巴德。使用DISCERN量表评估回答的可靠性和可读性,Flesch阅读轻松(FRE)和Flesch-Kincaid阅读等级(FKRGL)。
结果:GPT-4和GoogleBard对RAI的响应的平均(SD)FKRGL得分,PSMA-TRT,PRRT和TARE治疗为14.57(1.19),14.65(1.38),14.25(1.10),14.38(1.2)和11.49(1.59),12.42(1.71),11.35(1.80),13.01(1.97),分别。就可读性而言,GPT-4和GoogleBard对RAI的响应的FRKGL得分,PSMA-TRT,PRRT和TARE治疗高于一般公众阅读等级。核医学医师评估GPT-4和Bard对RAI的反应的平均(SD)DISCERN评分,PSMA-TRT,PRRT和TARE治疗为47.86(5.09),48.48(4.22),46.76(4.09),48.33(5.15)和51.50(5.64),53.44(5.42),53(6.36),49.43(5.32),分别。根据平均DISCERN分数,GPT-4和GoogleBard关于RAI的响应的可靠性,PSMA-TRT,PRRT,TARE治疗范围从一般到良好。GPT-4,Bard和核医学医师评估的DISCERN评分的评分者间可靠性相关系数,PSMA-TRT,PRRT和TARE治疗为0.512(95%CI0.296:0.704),0.695(95%CI0.518:0.829),0.687(95%CI0.511:0.823)和0.649(95%CI0.462:0.798),分别(p<0.01)。GPT-4,Bard和核医学医师评估的DISCERN评分的评分者间可靠性相关系数,PSMA-TRT,PRRT和TARE治疗为0.753(95%CI0.602:0.863),0.812(95%CI0.686:0.899),0.804(95%CI0.677:0.894)和0.671(95%CI0.489:0.812),分别(p<0.01)。Bard和GPT-4关于RA的响应的评分者间可靠性,PSMA-TRT,PRRT和TARE治疗中度至良好。Further,GPT-4和GoogleBard都很少强调对核医学医师的咨询,GoogleBard的一些回应中包含了参考文献,但在GPT-4中没有参考。
结论:尽管AI聊天机器人提供的信息在医学上可能是可以接受的,它不容易为公众阅读,这可能会阻止它被理解。使用“提示工程”的有效提示可以以更易于理解的方式完善响应。由于放射性核素治疗是核医学专业知识特有的,为了指导患者和护理人员获得准确的医疗建议,核医学医师需要在回应中作为顾问。就寻求信息的患者和护理人员的信心和满意度而言,参考意义重大。
OBJECTIVE: Searching for online health information is a popular approach employed by patients to enhance their knowledge for their diseases. Recently developed AI chatbots are probably the easiest way in this regard. The purpose of the study is to analyze the reliability and readability of AI chatbot responses in terms of the most commonly applied radionuclide treatments in cancer patients.
METHODS: Basic patient questions, thirty about RAI, PRRT and TARE treatments and twenty-nine about PSMA-TRT, were asked one by one to GPT-4 and Bard on January 2024. The reliability and readability of the responses were assessed by using DISCERN scale, Flesch Reading Ease(FRE) and Flesch-Kincaid Reading Grade Level(FKRGL).
RESULTS: The mean (SD) FKRGL scores for the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT and TARE treatmens were 14.57 (1.19), 14.65 (1.38), 14.25 (1.10), 14.38 (1.2) and 11.49 (1.59), 12.42 (1.71), 11.35 (1.80), 13.01 (1.97), respectively. In terms of readability the FRKGL scores of the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT and TARE treatments were above the general public reading grade level. The mean (SD) DISCERN scores assesses by nuclear medicine phsician for the responses of GPT-4 and Bard about RAI, PSMA-TRT, PRRT and TARE treatments were 47.86 (5.09), 48.48 (4.22), 46.76 (4.09), 48.33 (5.15) and 51.50 (5.64), 53.44 (5.42), 53 (6.36), 49.43 (5.32), respectively. Based on mean DISCERN scores, the reliability of the responses of GPT-4 and Google Bard about RAI, PSMA-TRT, PRRT, and TARE treatments ranged from fair to good. The inter-rater reliability correlation coefficient of DISCERN scores assessed by GPT-4, Bard and nuclear medicine physician for the responses of GPT-4 about RAI, PSMA-TRT, PRRT and TARE treatments were 0.512(95% CI 0.296: 0.704), 0.695(95% CI 0.518: 0.829), 0.687(95% CI 0.511: 0.823) and 0.649 (95% CI 0.462: 0.798), respectively (p < 0.01). The inter-rater reliability correlation coefficient of DISCERN scores assessed by GPT-4, Bard and nuclear medicine physician for the responses of Bard about RAI, PSMA-TRT, PRRT and TARE treatments were 0.753(95% CI 0.602: 0.863), 0.812(95% CI 0.686: 0.899), 0.804(95% CI 0.677: 0.894) and 0.671 (95% CI 0.489: 0.812), respectively (p < 0.01). The inter-rater reliability for the responses of Bard and GPT-4 about RAİ, PSMA-TRT, PRRT and TARE treatments were moderate to good. Further, consulting to the nuclear medicine physician was rarely emphasized both in GPT-4 and Google Bard and references were included in some responses of Google Bard, but there were no references in GPT-4.
CONCLUSIONS: Although the information provided by AI chatbots may be acceptable in medical terms, it can not be easy to read for the general public, which may prevent it from being understandable. Effective prompts using \'prompt engineering\' may refine the responses in a more comprehensible manner. Since radionuclide treatments are specific to nuclear medicine expertise, nuclear medicine physician need to be stated as a consultant in responses in order to guide patients and caregivers to obtain accurate medical advice. Referencing is significant in terms of confidence and satisfaction of patients and caregivers seeking information.