{Reference Type}: Journal Article
{Title}: From Bytes to Best Practices: Tracing ChatGPT-3.5's Evolution and Alignment With the National Comprehensive Cancer Network® Guidelines in Pancreatic Adenocarcinoma Management.
{Author}: Bresler TE;Pandya S;Meyer R;Htway Z;Fujita M;
{Journal}: Am Surg
{Volume}: 0
{Issue}: 0
{Year}: 2024 Apr 26
{Factor}: 1.002
{DOI}: 10.1177/00031348241248801
{Abstract}: <B>BACKGROUND: </B>Artificial intelligence continues to play an increasingly important role in modern health care. ChatGPT-3.5 (OpenAI, San Francisco, CA) has gained attention for its potential impact in this domain.<BR><B>OBJECTIVE: </B>To explore the role of ChatGPT-3.5 in guiding clinical decision-making specifically in the context of pancreatic adenocarcinoma and to assess its growth over a period of time.<BR><B>METHODS: </B>We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the Management of Pancreatic Adenocarcinoma and formulated a complex clinical question for each decision-making page. ChatGPT-3.5 was queried in a reproducible fashion. We scored answers on the following Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. We repeated this protocol at 3-months. Score frequencies were compared, and subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5).<BR><B>RESULTS: </B>In total, 50-pages of the NCCN Guidelines® were analyzed, generating 50 complex clinical questions. On subgroup analysis, the percentage of Acceptable answers improved from 60% to 76%. The score improvement was statistically significant (Mann-Whitney U-test; Mean Rank = 44.52 vs 56.48, P = .027).<BR><B>CONCLUSIONS: </B>ChatGPT-3.5 represents an interesting but limited tool for assistance in clinical decision-making. We demonstrate that the platform evolved, and its responses to our standardized questions improved over a relatively short period (3-months). Future research is needed to determine the validity of this tool for this clinical application.