目标:基于人工智能(AI)的聊天机器人在各个领域都表现出了准确性,包括医学,但研究尚未证实其准确性和临床相关性。我们评估了AI聊天机器人对治疗计划会议期间提出的问题的答案。
方法:病理学居民,病理学系,AI聊天机器人(OpenAIChatGPT[2023年1月30日,发布])回答了泌尿生殖系统亚专科治疗计划会议策划的问卷。结果由2名盲目的裁决者评估:临床医生专家和病理专家。评分基于准确性和临床相关性。
结果:总体而言,教师得分最高(4.75),其次是人工智能聊天机器人(4.10),研究准备的居民(3.50),和毫无准备的居民(2.87)。AI聊天机器人的得分在统计学上显着优于未准备好的居民(P=.03),但与研究准备好的居民(P=.33)或教职员工(P=.30)没有统计学差异。居民在研究后没有统计学上的显著改善(P=0.39),和教师的表现在统计上显着优于两个居民类别(无准备,P<.01;准备的研究,P=0.01)。
结论:AI聊天机器人对医学问题的回答在准确性和临床相关性上与病理学教师相当,提出进一步发展的承诺。严重的担忧依然存在,然而,如果没有提供参考支持的能力,人工智能将面临如何将其纳入医疗决策的合法审查。
OBJECTIVE: Artificial intelligence (AI)-based chatbots have demonstrated accuracy in a variety of fields, including medicine, but research has yet to substantiate their accuracy and clinical relevance. We evaluated an AI chatbot\'s answers to questions posed during a treatment planning conference.
METHODS: Pathology residents, pathology faculty, and an AI chatbot (OpenAI ChatGPT [January 30, 2023, release]) answered a questionnaire curated from a genitourinary subspecialty treatment planning conference. Results were evaluated by 2 blinded adjudicators: a clinician expert and a pathology expert. Scores were based on accuracy and clinical relevance.
RESULTS: Overall, faculty scored highest (4.75), followed by the AI chatbot (4.10), research-prepared residents (3.50), and unprepared residents (2.87). The AI chatbot scored statistically significantly better than unprepared residents (P = .03) but not statistically significantly different from research-prepared residents (P = .33) or faculty (P = .30). Residents did not statistically significantly improve after research (P = .39), and faculty performed statistically significantly better than both resident categories (unprepared, P < .01; research prepared, P = .01).
CONCLUSIONS: The AI chatbot gave answers to medical questions that were comparable in accuracy and clinical relevance to pathology faculty, suggesting promise for further development. Serious concerns remain, however, that without the ability to provide support with references, AI will face legitimate scrutiny as to how it can be integrated into medical decision-making.