背景大型语言模型(LLM),例如ChatGPT-3.5,GoogleBard,和微软Bing,在各种自然语言处理(NLP)任务中显示出很有前途的能力。然而,它们在解决特定领域问题方面的性能和准确性,特别是在血液学领域,没有被广泛调查。目的本研究旨在探索LLM的能力,即,ChatGPT-3.5,谷歌吟游诗人,和微软Bing(精确),在解决血液学相关病例并比较他们的表现。方法这是一项在生理学和病理学系进行的横断面研究,全印度医学科学研究所,Deoghar,Jharkhand,印度。我们策划了一组50例血液学病例,涵盖了一系列主题和复杂性。数据集包括与血液疾病相关的查询,恶性血液病,实验室测试参数,计算,和治疗选择。每个案例和相关问题都准备了一组正确的答案进行比较。我们利用了ChatGPT-3.5,GoogleBard实验,和MicrosoftBing(精确)用于问答任务。答案由两名生理学家和一名病理学家检查。他们以1到5的等级评定答案。通过弗里德曼测试和邓恩的事后测试比较了三种模型的平均得分。通过单样本中位数测试,将LLM的表现与2.5的中位数进行了比较,因为策划问题的课程具有50%的及格分数。结果三种LLM的得分差异有统计学意义(p值<0.0001),ChatGPT得分最高(3.15±1.19)。其次是Bard(2.23±1.17)和Bing(1.98±1.01)。ChatGPT评分明显高于50%(p值=0.0004),巴德的得分接近50%(p值=0.38),Bing的评分明显低于及格评分(p值=0.0015)。结论LLM在解决血液学中的病例样本方面存在显着差异。ChatGPT表现出最高分,其次是GoogleBard和MicrosoftBing。观察到的性能趋势表明,ChatGPT在医学领域具有广阔的潜力。然而,没有一个模型能够准确回答所有问题。语言模型的进一步研究和优化可以为医疗保健和医学教育应用提供有价值的贡献。
Background Large language models (LLMs), such as ChatGPT-3.5, Google Bard, and Microsoft Bing, have shown promising capabilities in various natural language processing (NLP) tasks. However, their performance and accuracy in solving domain-specific questions, particularly in the field of hematology, have not been extensively investigated. Objective This study aimed to explore the capability of LLMs, namely, ChatGPT-3.5, Google Bard, and Microsoft Bing (Precise), in solving hematology-related cases and comparing their performance. Methods This was a cross-sectional study conducted in the Department of Physiology and Pathology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. We curated a set of 50 cases on hematology covering a range of topics and complexities. The dataset included queries related to blood disorders, hematologic malignancies, laboratory test parameters, calculations, and treatment options. Each
case and related question was prepared with a set of correct answers to compare with. We utilized ChatGPT-3.5, Google Bard Experiment, and Microsoft Bing (Precise) for question-answering tasks. The answers were checked by two physiologists and one pathologist. They rated the answers on a rating scale from one to five. The average score of the three models was compared by Friedman\'s test with Dunn\'s post-hoc test. The performance of the LLMs was compared with a median of 2.5 by a one-sample median test as the curriculum from which the questions were curated has a 50% pass grade. Results The scores among the three LLMs were significantly different (p-value < 0.0001) with the highest score by ChatGPT (3.15±1.19), followed by Bard (2.23±1.17) and Bing (1.98±1.01). The score of ChatGPT was significantly higher than 50% (p-value = 0.0004), Bard\'s score was close to 50% (p-value = 0.38), and Bing\'s score was significantly lower than the pass score (p-value = 0.0015). Conclusion The LLMs reveal significant differences in solving
case vignettes in hematology. ChatGPT exhibited the highest score, followed by Google Bard and Microsoft Bing. The observed performance trends suggest that ChatGPT holds promising potential in the medical domain. However, none of the models was capable of answering all questions accurately. Further research and optimization of language models can offer valuable contributions to healthcare and medical education applications.