BACKGROUND/RATIONALE:
Early illness detection is essential to minimize morbidity and mortality in the event of infectious disease outbreaks of public health significance e.g. severe influenza, SARS or bioterrorism. The automated monitoring of electronic health information offers the potential to enhance disease surveillance compared to manual case reporting. The selection of electronically extractable data elements affects an automated system's ability to detect events at the earliest stage. Today's most sophisticated systems are limited to a narrow range of electronic data types, often obtained from unrelated sources. A system that could relate a broad array of clinical data around unique individuals could identify both the incidence and the time course of illnesses, and thus be helpful not only in the recognition but also in the management of epidemics.

OBJECTIVE(S):
The overall objective was to automate the use of CPRS data to enhance efficiency, timeliness, and validity of detection of outbreak of acute respiratory infections (ARI). We hypothesized that expanding the scope of an automated surveillance system to include illness progression and severity would shorten the time it takes to recognize disease outbreaks that pose a significant threat to veterans and public health.

Our specific aims were to:
1) Refine and validate automated methods to interpret ARI progression and severity;
2) Develop and validate statistical signals for surveillance that combine disease incidence and severity;
3) Use simulated outbreaks of ARI to evaluate the sensitivity and specificity of statistical models that account for disease severity compared to models that do not;
4) Vary simulated ARI outbreak characteristics so as to optimize the automated surveillance methods developed in Aims 1 and 2 for timely, sensitive and specific outbreak detection;
5) Initiate real-time electronic transmission of health record data to support public health disease surveillance.
In August 2009, a project modification added the following aims:
6) Apply previously validated ARI case-detection algorithms (CDAs) that utilize CPRS data to create time-series of daily counts of patients with possible ARI;
7) Develop seasonally-adjusted auto-regressive integrated moving average (SARIMA) forecasting models for each of the above time-series;
8) Compare the performance of ARI surveillance systems that use alternative aberrancy-detection methods: a) one which utilizes long-term historical data for forecasting (SARIMA); or b) one which utilizes data from the recent past only.

METHODS:
1 Development of ARI case-detection algorithms (CDAs).

CPRS data was extracted and transferred to relational databases. To build CDAs, relevant data were grouped and classified along different dimensions of respiratory disease severity: diagnostic and procedures codes, type of health care delivery, therapeutic and monitoring efforts, physiological, laboratory and imaging results, and the free text of clinical notes and XR reports. CDAs were created using iterative regression and tested against a reference manual structured chart review for their ability to identify ARI. The most successful ARI CDAs combined free-text analyses and structured, and have been published in PLoS ONE, 2010 5(10) e13377 and in PLoS ONE, 2012 7(12):e51147. Different aspects of the text analyses have also been published (AMIA Annu Symp Proc 2008:692; AMIA Summits Transl Sci Proc Mar 1; 2008:36; AMIA Summits Transl Sci Proc Mar 1; 2010:56).

2 Development of whole-system simulations (WSS).

We recreated historical background case count time series by applying the most successful CDAs to past CPRS data. We injected factitious influenza cases to CDA-specific backgrounds using an age-structured influenza epidemic model. From the time of this injection, aberrancy-detection statistics were applied each successive day (n = 50-80) on paired background+injection vs. background-only time series. Each injection-prospective-surveillance cycle was repeated 52 times, each time shifting the injection to a different week of the study year. We computed three whole-system benchmarks: 1) the "Detection Delay", the average time from injection to the first true-positive signal, defined as a statistical alarm originating in the background+injection dataset but not present in the background-only dataset; 2) the "False-Alarm Rate" (FAR), defined as the number of unique false-alarms originating in the background-only dataset during the study year, divided by 365 days; 3) the "Caseload", the yearly number of cases included in the false alarms. System performance was compared using activity monitoring operating characteristic (AMOC) curves relating Detection Rate or Delay to FAR or Caseload.

FINDINGS/RESULTS:
1 Aims 1, 3 and 4.
We identified three ARI subgroups who may have more severe disease: 1) ARI patients with fever; 2) ARI patients who seek emergent care; 3) outpatients with pneumonia.
A) Targeting ARI patients with fever. Seven (7) percent of our ARI patients had fever, compared with 70% of influenza patients. Thus, monitoring for febrile-ARI reduced sensitivity by 30%, but also eliminated 93% of afebrile ARI background cases thereby improving signal-to-noise ratio (S/N). Eight ARI CDAs that regrouped information about diagnostic codes, cough remedies and a computerized text analysis seeking ARI symptoms in free-text of the clinical note were compared against otherwise identical CDAs that also required patients to have an elevation in body temperature when seen. AMOC curves revealed that for any given FAR within a practical range (0-10%), CDA that targeted febrile patients yield lower detection delays than CDA that targeted all ARI patients. Surveillance caseload was also greatly reduced at any given detection delay.
Status: Completed. These results are being prepared for publication.

B) Targeting ARI patients seeking urgent care. We observed that two-thirds of the cases with ARI were seen in urgent care areas, but that these areas received only 15% of total outpatient visits. Because of a favorable S/N ratio, we hypothesized that an ARI surveillance system that focused on urgent visits would outperform one that monitored all outpatient visits, even though that system would ignore a significant number (1/3) of outpatients with ARI. To test this hypothesis, we applied one of eight different ARI CDA to CPRS entries related to "All" or to "Urgent-only" outpatient encounters. WSS was then used to generate paired AMOC curves of otherwise identical surveillance systems that included either "All" or Urgent-only visits. In each case, simulated surveillance systems aimed at Urgent-only visits resulted in Detection Delays that were lower at any given FAR or Caseload.
Status: Completed. These findings are being prepared for publication.

C) Targeting ARI patients with pneumonia. We applied a sensitive ARI CDA (99%) to documents related to all outpatient visits from 01/01/2004 to 12/31/2006 (n = 2.7 million). All associated chest imaging (CXR) reports (n = 2,747) associated with CDA-flagged encounters (n = 22,960) were manually reviewed and classified as "non-negative" if findings could possibly support the diagnosis of pneumonia. We then reviewed all related CPRS entries to determine if the patients had a "Possible Pneumonia" (non-negative imaging report with either cough, sputum, fever/chills/night sweats, dyspnea or pleuritic chest pain) or "Pneumonia-in-Plan" (non-negative imaging report with pneumonia stated as a likely diagnosis by the provider). The review identified 370 reference cases with Possible Pneumonia and 250 with Pneumonia-in-Plan. CDAs aimed at detecting these reference cases using ICD-9 codes, chest imaging requests, elevated temperature and other structured data elements from CPRS were sensitive (90-100%) but inaccurate (positive predictive values (PPV) of 13-22%).
To improve CDA accuracy, the manually classified chest imaging reports served to train machine-learning software that produced an imaging report text classifier. When combined with EMR-based structured data and with text analyses aimed at ARI symptoms in clinical notes, the x-ray report text classifier increased the positive predictive value pneumonia by 20-70%, while sensitivities of 58-75 % were retained.
Status: Completed. These results have been published (PLoS One, 2013 8(8) e70944).

2 Aims 2 and 3.
To begin developing statistical signals for surveillance that combine disease incidence and severity, we have used a simplified categorization framework, assigning ARI cases as either outpatients (cat1), hospitalized (cat2), or dead (cat3). Even when using the most sensitive ARI CDA, there were only three patients who died within 24 hours of their ARI encounter (cat3) in the last 8 years in the Baltimore dataset. Thus, all of cat3 instances were set to generate an alert. To develop a daily score that incorporates information about disease severity, we assigned a range of weights by which cat2 counts were multiplied before being added to cat1 counts. We used WSS to evaluate this daily score over a broad set of weights using hospitalization rates consistent with severe influenza. The resulting AMOC curves indicated that system performance worsens with increased cat2 weight.
Only a small proportion of patients with ARI exhibit markers that suggest a more severe disease. The proportion of ARI patient who: 1) are febrile is 7%; 2) have an abnormal CXR is 2.5%; 3) are admitted to the hospital is 2.5%. Thus, daily case counts of patients who exhibit any one of these markers are low, with many days showing no single instance. To incorporate disease severity into ARI surveillance, we therefore had to develop statistical aberrancy detection approaches that focus on sparse datasets. We chose to develop a general yet parsimonious method that could account for both large and small shifts in daily counts: a generalized likelihood ratio (GLR) control chart for zero-inflated, Poisson-distributed (ZIP) data. Using a range of purely simulated datasets, we have now shown that GLR-ZIP approach outperforms a recently published alternative, CUSUM-ZIP. We have implemented GLR- ZIP detection models in software, determined model parameters and control limits over a range of simulated data compatible with our authentic background datasets of patients with pneumonia.
Status: The results comparing GLR-ZIP to CUSUM-ZIP are being prepared for publication. In a recent grant application, we propose to evaluate the relative performance of these new aberrancy detection methods for pneumonia surveillance.

3 Aim 5.
The above work begins to highlights the priorities for CPRS data extractions and processing and the most fruitful surveillance strategies. We have now succeeded in developing a JAVA application that automates the sequence of database queries, text processing and statistical analyses necessary to perform surveillance using operational CPRS data.
Status: Proof-of-concept programming completed but not fully tested. We are currently updating software documentation before release to the NLP and surveillance community.

4 Aims 6, 7 and 8.
To test the hypothesis that aberrancy-detection methods that incorporate seasonality and other long-term data trends would reduce the time required to discover an influenza outbreak, we created 8-year authentic background time series of daily case counts by applying either of two ARI CDA to CPRS-derived data: 1) CDA1, ICD-9 codes set; 2) CDA2, ICD-9 codes combined with text analyses seeking ARI symptoms. Specific seasonal autoregressive integrated moving average (SARIMA) models were developed for each background. We used WSS to compare systems that used either: a) CDC's EARS W2c, which makes statistical predictions using 4 weeks of past data; b) SARIMA. At any given FAR within a practical range (0-10%), the SARIMA methods yielded lower detection delays than EARS W2c. These data suggested that forecasting approaches that incorporate long-term data trends can significantly improve the performance of surveillance systems.
Status: Completed. These results are now being prepared for publication.

IMPACT:
Our work indicates that information from a comprehensive EMR can be leveraged to improve both the effectiveness and the efficiency of automated surveillance systems for ARI. Fruitful sources of information are diagnostic codes, prescriptions, and the text of clinical note and chest imaging reports. Our work show that ARI surveillance systems should preferentially seek accurate rather than sensitive case-detection algorithms, and target subsets of ARI patients who may have more severe disease, such as those who are febrile and those who are seen in the emergency room. This work highlights the usefulness of developing simulation-based empiric approaches to guide the development of CPRS-based disease or event surveillance systems, in general. New pneumonia case-detection methods were developed that could serve not only to improve surveillance performance, but could also help evaluate the quality of care of patients with community-acquired pneumonia.

---

External Links for this Project

---

PUBLICATIONS:

Journal Articles

1. Rattinger GB, Mullins CD, Zuckerman IH, Onukwugha E, Walker LD, Gundlapalli A, Samore M, Delisle S. A sustainable strategy to prevent misuse of antibiotics for acute respiratory infections. PLoS ONE. 2013 Jul 11; 7(12):e51147. [view]
2. South BR, Shen S, Chapman WW, Delisle S, Samore MH, Gundlapalli AV. Analysis of False Positive Errors of an Acute Respiratory Infection Text Classifier due to Contextual Features. AMIA Summits on Translational Science proceedings. 2010 Mar 1; 2010:56-60. [view]
3. Gundlapalli AV, South BR, Phansalkar S, Kinney AY, Shen S, Delisle S, Perl T, Samore MH. Application of Natural Language Processing to VA Electronic Health Records to Identify Phenotypic Characteristics for Clinical and Research Purposes. Summit on translational bioinformatics. 2008 Mar 1; 2008:36-40. [view]
4. DeLisle S, South B, Anthony JA, Kalp E, Gundlapallli A, Curriero FC, Glass GE, Samore M, Perl TM. Combining free text and structured electronic medical record entries to detect acute respiratory infections. PLoS ONE. 2010 Oct 14; 5(10):e13377. [view]
5. South BR, Shen S, Jones M, Garvin J, Samore MH, Chapman WW, Gundlapalli AV. Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease. BMC bioinformatics. 2009 Sep 17; 10 Suppl 9:S12. [view]
6. South BR, Chapman W, Delisle S, Shen S, Kalp E, Perl T, Samore MH, Gundlapalli AV. Optimizing A syndromic surveillance text classifier for influenza-like illness: Does document source matter? AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 2008 Nov 6; 692-6. [view]
7. DeLisle S, Kim B, Deepak J, Siddiqui T, Gundlapalli A, Samore M, D'Avolio L. Using the electronic medical record to identify community-acquired pneumonia: toward a replicable automated strategy. PLoS ONE. 2013 Aug 13; 8(8):e70944. [view]

Journal Other

1. Price CS, Gritzer L, Mickiewicz T, Gundlapalli A, Delisle S, Davidson A, Perl TM. Evaluating a BioSense binning algorithms for utility in detecting category A agents. [Abstract]. Society for Healthcare Epidemiology of America Proceedings. 2009 Jul 1. [view]

Conference Presentations

1. South B, Shen S, Chapman W, Delisle S, Samore MH, Gundlapalli A. Analysis of False Positive Errors of an Acute Respiratory Infection text Classifier due to Contextual Features. Poster session presented at: American Medical Informatics Association Annual Symposium; 2010 Mar 13; San Francisco, CA. [view]
2. South B, Gundlapalli A, Kim B, DuVall SL, Samore MH, Delisle S. Automated Classification of Pneumonia Cases using Chest X-Ray Reports for Hospital and Public Health Surveillance. Poster session presented at: International Society for Disease Surveillance Annual Conference; 2009 Dec 3; Miami, FL. [view]
3. Delisle S, Fang M, Sun BC, South B, Smith R, Samore MH, Perl. Automated surveillance to detect an influenza epidemic: Which respiratory syndrome should we monitor? Paper presented at: International Society for Disease Surveillance Annual Conference; 2008 Dec 3; Raleigh, NC. [view]
4. Delisle S, Tian F, Sun P, South BR, Smith G, Gaff H, Samore M, Perl TM. Automated surveillance to detect and influenza epidemic: which respiratory syndrome should we monitor? Paper presented at: VA HSR&D National Meeting; 2009 Feb 1; Baltimore, MD. [view]
5. LaFleur J, Ginter T, Hayden C, DuVall SL, Nebeker JR. Bone Mineral Density Screening and Osteoperosis rates in veterans. Poster session presented at: American Society for Bone and Mineral Research Annual Meeting; 2011 Sep 18; San Diego, CA. [view]
6. Campbell B, South B, Gundlapalli A, Delisle S, Samore MH, Perl. Concept level evaluation of negotiation processing for an influenza-like illness text classifier. Paper presented at: International Society for Disease Surveillance Annual Conference; 2008 Dec 3; Raleigh, NC. [view]
7. DuVall SL, Butler J, LaFleur J, Nelson RE, Kamauu A, Shuerch M, Foskett N. Determining multiple sclerosis subtype from electronic medical records. Poster session presented at: Pharmacoepidemiology and Therapeutic Risk Management Annual International Conference; 2013 Aug 25; Montreal, Canada. [view]
8. Delisle S, South B, Phansalker, Perl, Gundlapalli A, Samore MH. Free-text processing to enhance detection of acute respiratory infections. Paper presented at: International Society for Disease Surveillance Annual Conference; 2008 Dec 3; Raleigh, NC. [view]
9. Kim B, South B, Samore M, Delisle S. Free-text processing to enhance surveillance of acute respiratory infections. Paper presented at: American Thoracic Society Annual International Conference; 2009 May 15; San Diego, CA. [view]
10. Nelson RE, Butler J, DuVall SL, LaFleur J, Xie Y, Shuerch M, Foskett N. Multiple sclerosis subtypes and serious infections resulting in a hospitalization in the Veterans Health Administration. Poster session presented at: Pharmacoepidemiology and Therapeutic Risk Management Annual International Conference; 2013 Nov 25; Montreal, Canada. [view]
11. Campbell B, South B, Gundlapalli A, Samore MH, Delisle S. Natural Language Processing: Can it help detect cases and characterize outbreaks? Paper presented at: International Society for Disease Surveillance Annual Conference; 2008 Dec 3; Raleigh, NC. [view]
12. Onukwugha E, Lee MT, Shinogle JA, Zuckerman IH, Godish M, Delisle S. Preventable hospital admissions for congestive heart failure in a Veterans population. Paper presented at: AcademyHealth Annual Research Meeting; 2010 Jun 27; Boston, MA. [view]
13. Delisle S, Fang, Hongzhang, Sun BC, South B, Gaff, Samore MH, Perl. Using biosurveillance-system facsmiles to compare aberrancy-detection methods: Should Biosense use SatScan? Paper presented at: International Society for Disease Surveillance Annual Conference; 2008 Dec 3; Raleigh, NC. [view]

DRA: Health Systems, Infectious Diseases
DRE: Technology Development and Assessment, Research Infrastructure, Epidemiology
Keywords: Bioterrorism, Research method
MeSH Terms: none