背景技术大型语言模型(LLM)已经成为能够处理和生成类似人类文本的强大工具。这些LLM,例如ChatGPT(OpenAI公司,任务区,旧金山,美国),谷歌吟游诗人(AlphabetInc.,CA,US),和微软必应(微软公司,WA,US),已经应用于各个领域,展示他们在帮助解决复杂任务和提高信息可访问性方面的潜力。然而,它们在解决生理学案例中的应用还没有被探索。本研究旨在评估三个LLM的绩效,即,ChatGPT(3.5;免费研究版),谷歌吟游诗人(实验),和微软Bing(精确),回答生理学中的小插曲。方法本横断面研究于2023年7月进行。由两名生理学家准备了总共77个生理学案例,并由另外两名内容专家进行了验证。这些案例提交给每个LLM,并收集了他们的回答。两位生理学家根据LLM的准确性对LLM提供的答案进行了独立评估。根据观察到的学习结果的结构(前结构=0,单结构=1,多结构=2,关系=3,扩展抽象),以0至4的等级进行测量。通过弗里德曼检验比较LLM之间的分数,并通过组内相关系数(ICC)检查观察者之间的一致性。结果ChatGPT的总分,宾,和研究中的Bard,总共77例,分别为3.19±0.3、2.15±0.6和2.91±0.5,p<0.0001。因此,ChatGPT3.5(免费版)获得了最高分,Bing(精确)得分最低,和巴德(实验)在性能方面介于两者之间。ChatGPT的平均ICC值,宾,和巴德为0.858(95%CI:0.777至0.91,p<0.0001),0.975(95%CI:0.961至0.984,p<0.0001),和0.964(95%CI:0.944至0.977,p<0.0001),分别。结论ChatGPT在回答生理学中的案例插图方面优于Bard和Bing。因此,学生和老师可能会考虑为他们的教育目的选择LLM,以进行基于案例的生理学学习。需要进一步探索他们的能力,以采用医学教育和临床决策支持。
Background Large language models (LLMs) have emerged as powerful tools capable of processing and generating human-like text. These LLMs, such as ChatGPT (OpenAI Incorporated, Mission District, San Francisco, United States), Google Bard (Alphabet Inc., CA, US), and Microsoft
Bing (Microsoft Corporation, WA, US), have been applied across various domains, demonstrating their potential to assist in solving complex tasks and improving information accessibility. However, their application in solving
case vignettes in physiology has not been explored. This study aimed to assess the performance of three LLMs, namely, ChatGPT (3.5; free research version), Google Bard (Experiment), and Microsoft Bing (precise), in answering cases vignettes in Physiology. Methods This cross-sectional study was conducted in July 2023. A total of 77
case vignettes in physiology were prepared by two physiologists and were validated by two other content experts. These cases were presented to each LLM, and their responses were collected. Two physiologists independently rated the answers provided by the LLMs based on their accuracy. The ratings were measured on a scale from 0 to 4 according to the structure of the observed learning outcome (pre-structural = 0, uni-structural = 1, multi-structural = 2, relational = 3, extended-abstract). The scores among the LLMs were compared by Friedman\'s test and inter-observer agreement was checked by the intraclass correlation coefficient (ICC). Results The overall scores for ChatGPT, Bing, and Bard in the study, with a total of 77 cases, were found to be 3.19±0.3, 2.15±0.6, and 2.91±0.5, respectively, p<0.0001. Hence, ChatGPT 3.5 (free version) obtained the highest score, Bing (Precise) had the lowest score, and Bard (Experiment) fell in between the two in terms of performance. The average ICC values for ChatGPT, Bing, and Bard were 0.858 (95% CI: 0.777 to 0.91, p<0.0001), 0.975 (95% CI: 0.961 to 0.984, p<0.0001), and 0.964 (95% CI: 0.944 to 0.977, p<0.0001), respectively. Conclusion ChatGPT outperformed Bard and
Bing in answering
case vignettes in physiology. Hence, students and teachers may think about choosing LLMs for their educational purposes accordingly for
case-based learning in physiology. Further exploration of their capabilities is needed for adopting those in medical education and support for clinical decision-making.