■要评估四种大型语言模型(LLM)的性能-GPT-4,PaLM2,Qwen,和百川2-对中国患者关于干眼症(DED)的询问做出回应。
■两阶段研究,包括第一阶段的横截面测试和第二阶段的真实世界临床评估。
■8名获得董事会认证的眼科医生和46名DED患者。
聊天机器人“对中国患者的反应”对DED的询问进行了评估。在第一阶段,六位资深眼科医生使用5点Likert量表在五个领域对聊天机器人的回答进行主观评价:正确性,完整性,可读性,乐于助人,和安全。使用中文可读性分析平台进行客观可读性分析。在第二阶段,46名DED代表性患者询问了在第一阶段问题中表现最佳的两种语言模型(GPT-4和百川2),然后对答案的满意度和可读性进行了评分。然后,两名高级眼科医生评估了五个领域的反应。
■五个领域的主观得分和第一阶段的客观可读性得分。患者满意度,可读性分数,以及第二阶段五个领域的主观得分。
■在第一阶段,GPT-4在五个领域表现出优异的性能(正确性:4.47;完整性:4.39;可读性:4.47;有用性:4.49;安全性:4.47,p<0.05)。然而,可读性分析表明,GPT-4的反应是高度复杂的,平均得分为12.86(p<0.05),而Qwen的得分为10.87、11.53和11.26,分别为百川2和PaLM2。在第二阶段,如五个领域的分数所示,GPT-4和百川2均擅长回答DED患者提出的问题。然而,百川2的回答的完整性相对较差(4.04与GPT-4为4.48,p<0.05)。然而,百川2的建议比GPT-4的建议更容易理解(患者可读性:3.91vs.4.61,p<0.05;眼科医生可读性:2.67vs.4.33).
■这些发现强调了法学硕士的潜力,特别是GPT-4和百川2,对中国患者关于DED的问题提供准确和全面的回答。
UNASSIGNED: To evaluate the performance of four large language models (LLMs)-GPT-4, PaLM 2, Qwen, and Baichuan 2-in generating responses to inquiries from Chinese patients about dry eye disease (DED).
UNASSIGNED: Two-phase study, including a cross-sectional test in the first phase and a real-world clinical assessment in the second phase.
UNASSIGNED: Eight board-certified ophthalmologists and 46 patients with DED.
UNASSIGNED: The chatbots\' responses to Chinese patients\' inquiries about DED were assessed by the evaluation. In the first phase, six senior ophthalmologists subjectively rated the chatbots\' responses using a 5-point Likert scale across five domains: correctness, completeness, readability, helpfulness, and safety. Objective readability analysis was performed using a Chinese readability analysis platform. In the second phase, 46 representative patients with DED asked the two language models (GPT-4 and Baichuan 2) that performed best in the in the first phase questions and then rated the answers for satisfaction and readability. Two senior ophthalmologists then assessed the responses across the five domains.
UNASSIGNED: Subjective scores for the five domains and objective readability scores in the first phase. The patient satisfaction, readability scores, and subjective scores for the five-domains in the second phase.
UNASSIGNED: In the first phase, GPT-4 exhibited superior performance across the five domains (correctness: 4.47; completeness: 4.39; readability: 4.47; helpfulness: 4.49; safety: 4.47, p < 0.05). However, the readability analysis revealed that GPT-4\'s responses were highly complex, with an average score of 12.86 (p < 0.05) compared to scores of 10.87, 11.53, and 11.26 for Qwen, Baichuan 2, and PaLM 2, respectively. In the second phase, as shown by the scores for the five domains, both GPT-4 and Baichuan 2 were adept in answering questions posed by patients with DED. However, the completeness of Baichuan 2\'s responses was relatively poor (4.04 vs. 4.48 for GPT-4, p < 0.05). Nevertheless, Baichuan 2\'s recommendations more comprehensible than those of GPT-4 (patient readability: 3.91 vs. 4.61, p < 0.05; ophthalmologist readability: 2.67 vs. 4.33).
UNASSIGNED: The findings underscore the potential of LLMs, particularly that of GPT-4 and Baichuan 2, in delivering accurate and comprehensive responses to questions from Chinese patients about DED.