1082 — Adapting a Natural Language Processing Algorithm to Support Stroke Cohort Generation

Mowery DL, Salt Lake City VA Health Care System; South BR, Salt Lake City VA Health Care System; Garvin J, Salt Lake City VA Health Care System; Franc D, Los Angeles VA Health Care System; Ashfaq S, San Diego VA Health Care System; Zamora T, San Diego VA Health Care System; Cheng E, Los Angeles VA Health Care System; Chapman BE, Salt Lake City VA Health Care System; Keyhani S, San Francisco VA Health Care System; Chapman WW, Salt Lake City VA Health Care System

Objectives:
Significant internal carotid artery stenosis is a strong stroke risk factor when present. Because there is not a way to identify such patients in administrative databases, manual chart abstraction is required. We aimed to develop and evaluate a natural language processing algorithm, ConText, that filters carotid ultrasounds of patients with no or insignificant internal carotid stenosis to focus chart review efforts to only patients with significant carotid stenosis.

Methods:
From the Office of Quality and Performance Stroke Special Study dataset, a cohort of 5000 veterans with a primary diagnosis of ischemic stroke (FY 2007), we randomly selected 60 patients (n = 308 reports) for algorithm development and 34 patients (n = 110 reports) for algorithm evaluation. We adapted and validated the ConText algorithm leveraging the development set and performed a summative evaluation against the evaluation set. We assessed how well the algorithm correctly asserted whether a report had no stenosis, insignificant stenosis, or significant stenosis applying accuracy, Cohen's kappa, sensitivity (recall), positive predictive value, specificity (true negative rate), and negative predictive value.

Results:
On the evaluation set containing 206 abstractions (one each for the right and left carotid artery), we report an overall accuracy of 89% and Cohen's kappa of 69%. For no stenosis, we observed high sensitivity (99%), high positive predictive value (92%), moderate specificity (73%), and high negative predictive value (97%). For insignificant stenosis, we observed moderate sensitivity (61%), high positive predictive value (100%), high specificity (100%), and high negative predictive value (92%). For significant stenosis, we observed moderate sensitivity (42%), low positive predictive value (36%), high specificity (95%), and high negative predictive value (96%).

Implications:
Our initial results suggest ConText could aid health service researchers to generate a stroke cohort by filtering ultrasounds negative for significant carotid stenosis. We are actively adapting and applying the algorithm to a comparative effectiveness study for stroke prevention in the Veterans Administration.

Impacts:
Our research supports Stroke QUERI goal: Develop, evaluate, and integrate interventions to improve risk factor control among veterans at high risk of stroke by enabling researchers to design and conduct feasible studies that assess comparative effectiveness of stroke treatments for veterans.