BACKGROUND/RATIONALE:
VAMC: Tennessee Valley Healthcare System VISN 9
Solicitation/Targeted Area: HSR&D Priorities for Investigator-Initiated Research, Research Methodology.
Background/Rationale: Adverse events (Aes) are injuries that occur to patients as a result of their medical management and not their underlying disease process. The Veterans Health Administration's (VHA) commitment to deliver the safest medical care possible for its veterans hinges on its ability to detect patient injuries and make systematic improvements. The primary purpose of this proposal is to evaluate a concept-based indexing tool that electronically identifies adverse events within clinical narratives and compare this method to other adverse event surveillance methods.

OBJECTIVE(S):
The objectives of this study are to:
Objectives: The study aim was to implement natural language processing (NLP) to codify the free text of electronic health records and develop algorithms using concept extraction of text combined with structured clinical data to identify VASQIP post-operative complications (acute renal failure, sepsis, deep vein thrombosis, pulmonary embolism, myocardial infarction, cardiac arrest, pneumonia, urinary tract infection, and wound infection). We will determine the sensitivity and specificity of our Post-Operative Event Monitor (POEM) at identifying the target post-operative complications. Several non-VA organizations have recommended using administrative data, such as PSI, to electronically screen for post-surgical complications and thus comparing this novel screening technique to administrative screening might identify the most accurate electronic screening methodology. The objectives of this study are to:
Objective 1: To determine the feasibility of using concept-based indexing as an event finder for surgical complications (Post-Operative Event Monitor POEM).
Objective 2: To determine the test characteristics (sensitivity and specificity) of the POEM using the trained NSQIP nurse reviewers as the gold standard.
Objective 3: To determine the test characteristics (sensitivity and specificity) of the Patient Safety Indicators on matching NSQIP surgical complications and compare the performance of POEM with PSI screening modalities for detecting specific types of post-operative adverse events.
Objective 4: To design a composite event monitor tool combing the POEM system and PSI and compare the composite tool to the singular POEM and PSI monitoring systems.

METHODS:
Cases were from the VASQIP registry for VISN9 and controls were patients in VASQIP without complications, stratified by medical center and date of procedure. Data comprised the VASQIP for surgical complications, corporate data warehouse for administrative and clinical structured data, and narrative clinical notes such as discharge summaries, progress notes, nursing care notes, operative notes and outpatient visit notes from the Veterans Health Information System and Technology Architecture (VistA) electronic health record. Computer algorithms were rules according to VASQIP definitions using NLP coded SNOMED CT concepts, ancillary keyword strings and structured data. The VASQIP nurse reviewer was the reference standard for estimating sensitivity and specificity.
Sensitivity and positive predictive value will be derived from the surgical cases in the NSQIP database for VISN 9 from 1999 through 2006. The POEM and PSI algorithms will be applied to the narrative records and administrative data of these cases and then evaluated against the gold standard.

FINDINGS/RESULTS:
There were 8,123 patients in the development set, with mean age of 63 ( 13 years) and 95% male. The procedures were 73% inpatient with an average of 16 days hospital length of stay and 74% had an American Society of Anesthesiologist pre-operative score of 3 or higher. The sensitivity and specificity were 68% and 91% for acute renal failure, 69% and 90% for sepsis, 54% and 95% for DVT, 80% and 97% for pulmonary embolism, 86% and 89% for acute myocardial infarction, 83% and 92% for cardiac arrest, 80% and 90% for pneumonia, 95% and 79% for urinary tract infection, and 48% and 87% for wound infection, respectively.
Conclusions
Using NLP, our electronic queries produced respectable sensitivity and excellent specificity for automated identification of post-operative adverse events from the electronic health record. An NLP approach to identify surgical complications is more sensitive but less specific compared to a strategy using the Patient Safety Indicators. Future iterative rule development should help improve the performance of the rule algorithms.


IMPACT:
Ensuring the highest quality of care for veterans cared for by the VHA requires treating safety as an utmost priority. Successful quality improvement programs like NSQIP still rely on manual chart review, a costly approach, and much of this information could be electronically identified. Mining the electronic record would enable the redistribution of resources into interventions, allow for performance measures that evaluate the impact of those interventions and provide new sources of information that would stimulate innovative applications of implementation research. This tool could afford VHA the ability to monitor and identify a broad range of patient safety concerns and quality indicators. Electronically obtained clinical information can result in marked improvement in the accuracy of clinical surveillance strategies and could result in gains in the overall efficiency of manual adverse event abstraction strategies thus making this an attractive means for routine adverse event detection.

---

External Links for this Project

---

PUBLICATIONS:

None at this time.

DRA: Health Systems
DRE: none
Keywords: Cost, Informatics, Research measure, Research method, Safety
MeSH Terms: none