背景:美国代谢和减肥手术学会(ASMBS)教科书是减肥手术的综合资源,涵盖最新进展和临床问题。使用此权威来源测试人工智能(AI)引擎可确保准确和最新的信息,并提供其对外科教育和培训的潜在影响的洞察力。
目的:确定质量并比较不同的大型语言模型(LLM)对与减肥手术有关的教科书问题的回答能力。
方法:远程。
方法:提示进入LLM是多项选择题,见于“ASMBS减肥手术教科书,第二版。提示被查询到3个LLM中:OpenAI的ChatGPT-4,Microsoft的Bing,和谷歌的吟游诗人。根据总体准确性评估生成的响应,根据主题正确答案的数量,以及根据问题类型正确答案的数量。进行统计分析以确定正确的每个类别的每个LLM的响应数。
结果:使用了200个问题来查询AI模型。答案的准确性总体上存在显着差异,ChatGPT-4的准确率为83.0%,其次是Bard(76.0%)和Bing(65.0%)。亚组分析显示,模型在问题类别中的性能存在显著差异,ChatGPT-4在与治疗和外科手术(83.1%)和并发症(91.7%)相关的问题中,正确答案的比例最高。不同题型的表现也有显著差异,ChatGPT-4在包容性问题中表现出卓越的性能。Bard和Bing无法回答某些问题,而ChatGPT-4没有任何问题未回答。
结论:LLM,特别是ChatGPT-4,在回答与减肥手术相关的临床问题时,显示出有希望的准确性。需要不断的AI进步和研究来阐明LLM在培训和教育中的潜在应用。
BACKGROUND: The American Society for Metabolic and Bariatric Surgery (ASMBS) textbook serves as a comprehensive resource for bariatric surgery, covering recent advancements and clinical questions. Testing artificial intelligence (AI) engines using this authoritative source ensures accurate and up-to-date information and provides insight in its potential implications for surgical education and training.
OBJECTIVE: To determine the quality and to compare different large language models\' (LLMs) ability to respond to textbook questions relating to bariatric surgery.
METHODS: Remote.
METHODS: Prompts to be entered into the LLMs were multiple-choice questions found in \"The ASMBS Textbook of Bariatric Surgery, second Edition. The prompts were queried into 3 LLMs: OpenAI\'s ChatGPT-4, Microsoft\'s Bing, and Google\'s Bard. The generated responses were assessed based on overall accuracy, the number of correct answers according to subject matter, and the number of correct answers based on question type. Statistical analysis was performed to determine the number of responses per LLMs per category that were correct.
RESULTS: Two hundred questions were used to query the AI models. There was an overall significant difference in the accuracy of answers, with an accuracy of 83.0% for ChatGPT-4, followed by Bard (76.0%) and Bing (65.0%). Subgroup analysis revealed a significant difference between the models\' performance in question categories, with ChatGPT-4\'s demonstrating the highest proportion of correct answers in questions related to treatment and surgical procedures (83.1%) and complications (91.7%). There was also a significant difference between the performance in different question types, with ChatGPT-4 showing superior performance in inclusionary questions. Bard and Bing were unable to answer certain questions whereas ChatGPT-4 left no questions unanswered.
CONCLUSIONS: LLMs, particularly ChatGPT-4, demonstrated promising accuracy when answering clinical questions related to bariatric surgery. Continued AI advancements and research is required to elucidate the potential applications of LLMs in training and education.