简介人工智能(AI)聊天机器人有可能成为患者青光眼信息的主要来源,描述聊天机器人提供的信息是至关重要的,这样提供商就可以定制讨论,预测患者的担忧,识别误导性信息。因此,这项研究的目的是评估来自人工智能聊天机器人的青光眼信息,包括ChatGPT-4,巴德,还有Bing,通过分析响应精度,全面性,可读性,单词计数,以及与青光眼相关的美国眼科学会(AAO)患者材料进行比较。方法将AAO青光眼相关患者教育手册中的章节标题改编成问题形式,并向每个AI聊天机器人询问五次(ChatGPT-4,Bard,和Bing)。每个聊天机器人的两组响应用于评估AI聊天机器人响应和AAO手册信息的准确性,与AAO手册信息相比,人工智能聊天机器人响应的全面性,由三名独立的青光眼训练的眼科医生得分为1-5分。可读性(使用Flesch-Kincaid等级(FKGL)评估,对应于美国学校年级),单词计数,并确定所有聊天机器人响应和AAO手册部分的字符计数。结果AAO的准确性评分,ChatGPT,宾,和巴德分别为4.84、4.26、4.53和3.53。直接比较,AAO比ChatGPT更准确(p=0.002),而巴德是最不准确的(巴德与AAO,p<0.001;巴德与ChatGPT,p<0.002;Bard与Bing,p=0.001)。ChatGPT的反应最全面(ChatGPT与Bing,p<0.001;ChatGPT与Bardp=0.008),有了ChatGPT的全面性分数,宾,和巴德分别为3.32、2.16和2.79。AAO信息和Bard响应处于最可访问的可读性水平(AAO与ChatGPT,AAO对抗Bing,Bard对ChatGPT,吟游诗人对宾,所有p<0.0001),具有AAO的可读性级别,ChatGPT,宾,和巴德分别为8.11、13.01、11.73和7.90。Bing回答的单词和字符数最低。结论AI聊天机器人反应的准确性各不相同,全面性,和可读性。准确性得分和全面性低于AAO手册,可读性水平提高,AI聊天机器人需要改进,才能成为患者更有用的青光眼信息补充来源。医生必须意识到这些限制,以便向患者询问现有的知识和问题,然后提供明确和全面的信息。
Introduction With the potential for artificial intelligence (AI) chatbots to serve as the primary source of glaucoma information to patients, it is essential to characterize the information that chatbots provide such that providers can tailor discussions, anticipate patient concerns, and identify misleading information. Therefore, the purpose of this study was to evaluate glaucoma information from AI chatbots, including ChatGPT-4, Bard, and
Bing, by analyzing response accuracy, comprehensiveness, readability, word count, and character count in comparison to each other and glaucoma-related American Academy of Ophthalmology (AAO) patient materials. Methods Section headers from AAO glaucoma-related patient education brochures were adapted into question form and asked five times to each AI chatbot (ChatGPT-4, Bard, and Bing). Two sets of responses from each chatbot were used to evaluate the accuracy of AI chatbot responses and AAO brochure information, and the comprehensiveness of AI chatbot responses compared to the AAO brochure information, scored 1-5 by three independent glaucoma-trained ophthalmologists. Readability (assessed with Flesch-Kincaid Grade Level (FKGL), corresponding to the United States school grade levels), word count, and character count were determined for all chatbot responses and AAO brochure sections. Results Accuracy scores for AAO, ChatGPT,
Bing, and Bard were 4.84, 4.26, 4.53, and 3.53, respectively. On direct comparison, AAO was more accurate than ChatGPT (p=0.002), and Bard was the least accurate (Bard versus AAO, p<0.001; Bard versus ChatGPT, p<0.002; Bard versus Bing, p=0.001). ChatGPT had the most comprehensive responses (ChatGPT versus Bing, p<0.001; ChatGPT versus Bard p=0.008), with comprehensiveness scores for ChatGPT,
Bing, and Bard at 3.32, 2.16, and 2.79, respectively. AAO information and Bard responses were at the most accessible readability levels (AAO versus ChatGPT, AAO versus Bing, Bard versus ChatGPT, Bard versus
Bing, all p<0.0001), with readability levels for AAO, ChatGPT,
Bing, and Bard at 8.11, 13.01, 11.73, and 7.90, respectively.
Bing responses had the lowest word and character count. Conclusion AI chatbot responses varied in accuracy, comprehensiveness, and readability. With accuracy scores and comprehensiveness below that of AAO brochures and elevated readability levels, AI chatbots require improvements to be a more useful supplementary source of glaucoma information for patients. Physicians must be aware of these limitations such that patients are asked about existing knowledge and questions and are then provided with clarifying and comprehensive information.