From Bytes to Best Practices: Tracing ChatGPT-3.5's Evolution and Alignment With the National Comprehensive Cancer Network® Guidelines in Pancreatic Adenocarcinoma Management

Division

Far West

Hospital

Los Robles Hospital and Medical Center

Document Type

Manuscript

Publication Date

10-1-2024

Keywords

Humans, Pancreatic Neoplasms, Adenocarcinoma, Practice Guidelines as Topic, Clinical Decision-Making, Artificial Intelligence

Disciplines

Digestive System Diseases | Medicine and Health Sciences | Neoplasms | Surgery

Abstract

INTRODUCTION: Artificial intelligence continues to play an increasingly important role in modern health care. ChatGPT-3.5 (OpenAI, San Francisco, CA) has gained attention for its potential impact in this domain.

OBJECTIVE: To explore the role of ChatGPT-3.5 in guiding clinical decision-making specifically in the context of pancreatic adenocarcinoma and to assess its growth over a period of time.

PARTICIPANTS: We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the Management of Pancreatic Adenocarcinoma and formulated a complex clinical question for each decision-making page. ChatGPT-3.5 was queried in a reproducible fashion. We scored answers on the following Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. We repeated this protocol at 3-months. Score frequencies were compared, and subgroup analysis was conducted on

RESULTS: In total, 50-pages of the NCCN Guidelines® were analyzed, generating 50 complex clinical questions. On subgroup analysis, the percentage of

CONCLUSION: ChatGPT-3.5 represents an interesting but limited tool for assistance in clinical decision-making. We demonstrate that the platform evolved, and its responses to our standardized questions improved over a relatively short period (3-months). Future research is needed to determine the validity of this tool for this clinical application.

Publisher or Conference

The American Surgeon

Link to Full Text

Share

COinS