■先前的研究评估了大型语言模型(LLM)在医学学科中的能力;然而,很少有人专注于图像分析,没有特别针对心血管成像或核心脏病学。
■本研究评估了四个LLM-GPT-4,GPT-4Turbo,GPT-4omni(GPT-4o)(开放式AI),和Gemini(GoogleInc.)-在回答2023年美国核心脏病学会委员会准备考试的问题时,反映核心脏病学认证委员会(CBNC)考试的范围。
■我们使用了168个问题:141个纯文本和27个基于图像的问题,分为四个部分镜像CBNC考试。每个LLM都有相同的标准化提示,并应用于每个部分30次,以说明随机性。对除了GPT-4o之外的所有模型评估六周内的性能。McNemar测试比较了正确的反应比例。
■GPT-4,双子座,GPT4-Turbo,GPT-4o正确回答了56.8%的中位数百分位数(95%置信区间55.4%-58.0%),40.5%(39.9%-42.9%),60.7%(59.9%-61.3%)和63.1%(62.5-64.3%)的问题,分别。GPT4o的表现明显优于其他型号(p=0.007与GPT-4Turbo,p<0.001vs.GPT-4和双子座)。与GPT-4,双子座相比,GPT-4o在纯文本问题上表现出色,和GPT-4Turbo(p<0.001,p<0.001,p=0.001),而双子座在基于图像的问题上表现较差(全部p<0.001)。
■GPT-4o在四个LLM中表现出卓越的性能,达到的分数可能在通过类似于CBNC考试的测试所需的范围之内或之外。尽管需要改进医学图像解释,GPT-4o显示出支持医生回答基于文本的临床问题的潜力。
UNASSIGNED: Previous studies evaluated the ability of large language models (LLMs) in medical disciplines; however, few have focused on image analysis, and none specifically on cardiovascular imaging or nuclear cardiology.
UNASSIGNED: This study assesses four LLMs -
GPT-4,
GPT-4 Turbo,
GPT-4omni (
GPT-4o) (Open AI), and Gemini (Google Inc.) - in responding to questions from the 2023 American Society of Nuclear Cardiology Board Preparation Exam, reflecting the scope of the Certification Board of Nuclear Cardiology (CBNC) examination.
UNASSIGNED: We used 168 questions: 141 text-only and 27 image-based, categorized into four sections mirroring the CBNC exam. Each LLM was presented with the same standardized prompt and applied to each section 30 times to account for stochasticity. Performance over six weeks was assessed for all models except
GPT-4o. McNemar\'s test compared correct response proportions.
UNASSIGNED: GPT-4, Gemini, GPT4-Turbo, and
GPT-4o correctly answered median percentiles of 56.8% (95% confidence interval 55.4% - 58.0%), 40.5% (39.9% - 42.9%), 60.7% (59.9% - 61.3%) and 63.1% (62.5 - 64.3%) of questions, respectively. GPT4o significantly outperformed other models (p=0.007 vs.
GPT-4Turbo, p<0.001 vs.
GPT-4 and Gemini).
GPT-4o excelled on text-only questions compared to
GPT-4, Gemini, and
GPT-4 Turbo (p<0.001, p<0.001, and p=0.001), while Gemini performed worse on image-based questions (p<0.001 for all).
UNASSIGNED: GPT-4o demonstrated superior performance among the four LLMs, achieving scores likely within or just outside the range required to pass a test akin to the CBNC examination. Although improvements in medical image interpretation are needed, GPT-4o shows potential to support physicians in answering text-based clinical questions.