Background Automated adjudication of cardiogenic shock (CS) from unstructured clinical documentation can facilitate clinical case finding and research trial/registry enrollment. While Large Language Models (LLMs) hold considerable promise, commercial cloud-based solutions raise data privacy and cost concerns. We assessed the performance of an open-source, locally run LLM for the adjudication of CS from unstructured progress notes.
Methods Patients from a 22-hospital healthcare system from 2021 to 2024 were identified using ICD-10 codes to form an enriched CS evaluation cohort with a ~1:1 case/non-case ratio. We selected gpt-oss-20b, an open-source LLM capable of running locally on a consumer-grade computer, and tested it without task-specific fine tuning. A total of 105 de-identified cardiology progress notes from distinct patients were independently reviewed by two physicians and the LLM to determine whether the patient had CS or not. Inter-rater agreement was assessed pairwise among reviewers.
Results Inter-reviewer agreement for CS classification between the two physicians was 90% (94/105 notes). Agreement between the LLM and two physicians were 79% and 82%, respectively, in the whole cohort. In a consensus subset (n = 94) where both physicians agreed, the LLM achieved 84% agreement with the human consensus. Among these cases, physicians identified CS in 44 (47%) patients, whereas the LLM classified 49 (52%) as having CS, indicating a slight over-estimation.
Conclusions An open-source, locally run LLM without fine-tuning demonstrated strong concordance with human consensus in identifying CS from unstructured clinical notes. This approach offers a scalable, privacy-safe, and cost-effective alternative to manual adjudication for clinical and research case finding and enrollment.