目的:评估GPT-3.5,GPT-4和微调的GPT-3.5模型在将FleischnerSociety建议应用于肺结节的准确性。方法:我们根据Fleischner协会指南,为12个结节类别中的每一个生成了10个肺结节描述,将它们纳入一个虚构的报告(n=120)。GPT-3.5和GPT-4被提示根据报告提出后续建议。然后,我们将完整的指南纳入提示中并重新提交。最后,我们将提示重新提交给经过微调的GPT-3.5模型。结果使用R中的二元准确性分析进行分析。结果:GPT-3.5应用Fleischner协会指南的准确性为0.058(95%CI:0.02,0.12)。GPT-4的准确度提高到0.15(95%CI:0.09,0.23;准确度比较P=.02)。在推荐PET-CT和/或活检时,GPT-3.5和GPT-4的F评分均为0.00.在提示中明确包括Fleischner协会准则之后,GPT-3.5和GPT-4的准确性显著提高到0.42(95%CI:0.33,0.51;P<.001)和0.66(95%CI:0.57,0.74;P<.001),分别。GPT-4仍显著优于GPT-3.5(P<.001)。微调的GPT-3.5模型精度为0.46(95%CI:0.37,0.55),与包含指南的GPT-3.5模型没有不同(P=.53)。结论:GPT-3.5和GPT-4在应用广为人知的指南方面表现不佳,从未正确推荐活检。错误的知识和推理都导致了他们的糟糕表现。虽然GPT-4比GPT-3.5更准确,但其不准确率在临床实践中是不可接受的。这些结果强调了大型语言模型对于基于知识和推理的任务的局限性。
Purpose: To evaluate the accuracy of GPT-3.5, GPT-4, and a fine-tuned GPT-3.5 model in applying Fleischner Society recommendations to lung nodules. Methods: We generated 10 lung nodule descriptions for each of the 12 nodule categories from the Fleischner Society guidelines, incorporating them into a single fictitious report (n = 120). GPT-3.5 and GPT-4 were prompted to make follow-up recommendations based on the reports. We then incorporated the full
guidelines into the prompts and re-submitted them. Finally, we re-submitted the prompts to a fine-tuned GPT-3.5 model. Results were analyzed using binary accuracy analysis in R. Results:
GPT-3.5 accuracy in applying Fleischner Society
guidelines was 0.058 (95% CI: 0.02, 0.12). GPT-4 accuracy was improved at 0.15 (95% CI: 0.09, 0.23; P = .02 for accuracy comparison). In recommending PET-CT and/or biopsy, both GPT-3.5 and GPT-4 had an F-score of 0.00. After explicitly including the Fleischner Society
guidelines in the prompt,
GPT-3.5 and
GPT-4 significantly improved their accuracy to 0.42 (95% CI: 0.33, 0.51; P < .001) and to 0.66 (95% CI: 0.57, 0.74; P < .001), respectively. GPT-4 remained significantly better than GPT-3.5 (P < .001). The fine-tuned GPT-3.5 model accuracy was 0.46 (95% CI: 0.37, 0.55), not different from the GPT-3.5 model with
guidelines included (P = .53). Conclusion:
GPT-3.5 and
GPT-4 performed poorly in applying widely known
guidelines and never correctly recommended biopsy. Flawed knowledge and reasoning both contributed to their poor performance. While GPT-4 was more accurate than GPT-3.5, its inaccuracy rate was unacceptable for clinical practice. These results underscore the limitations of large language models for knowledge and reasoning-based tasks.