Artificial Intelligence in Surgical Education: A Pilot Study Using ASCRS Guideline-Derived Questions

Division

Far West

Hospital

Los Robles Hospital and Medical Center

Document Type

Manuscript

Publication Date

11-1-2026

Keywords

Pilot Projects, Humans, Practice Guidelines as Topic, Artificial Intelligence, Large Language Models, Surveys and Questionnaires, United States, Rectal Fistula

Disciplines

Digestive System Diseases | Medical Education | Medicine and Health Sciences | Neoplasms | Surgery

Abstract

BackgroundLarge language models (LLMs) have demonstrated strong performance on general medical and surgical examinations; however, their capacity to accurately interpret and apply subspecialty clinical practice guidelines remains incompletely characterized.ObjectiveTo evaluate and compare the accuracy and consistency of two contemporary LLMs-Google Gemini and OpenEvidence-using multiple-choice questions (MCQs) derived directly from the 2022 American Society of Colon and Rectal Surgeons (ASCRS) Clinical Practice Guidelines for anorectal abscess, fistula-in-ano, and rectovaginal fistula.MethodsThirty guideline-based MCQs were developed and independently validated by surgeon reviewers. Each question was presented to both models under identical conditions without additional prompting. Accuracy was calculated with 95% confidence intervals and compared against chance performance (p

Publisher or Conference

The American Surgeon

Link to Full Text

Share

COinS