Patient Safety Indicators Do Not Always Identify True Safety Events in VA Hospitals

BACKGROUND:
As increasing numbers of healthcare systems are adopting the Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) for public reporting/pay-for-performance, it is essential that the PSIs accurately reflect hospital safety performance. Although the PSIs underwent a rigorous development process, concerns about how well the indicators identify true events and accurately reflect hospital performance have heightened with their increasing use in public reporting. This retrospective study examined the positive predictive value (PPV – proportion of flagged cases confirmed by chart review to have PSI event) of 12 selected PSIs using data from VA’s electronic medical record as the gold standard. Investigators also sought to determine reasons why flagged PSI cases did not represent true events, and to recommend modifications to the PSI algorithms to improve PPVs. The 12 PSIs included: decubitus ulcer, foreign body left in during procedure, iatrogenic pneumothorax, central venous catheter-related bloodstream infections, post-operative hip fracture, post-operative hemorrhage or hematoma, post-operative physiologic and metabolic derangement, post-operative respiratory failure, post-operative pulmonary embolism or deep vein thrombosis, post-operative sepsis, post-operative wound dehiscence, and accidental puncture or laceration. Using VA data from FY03 through FY07, investigators identified PSI-flagged cases from 28 representative VA hospitals, focusing on hospital discharge information, e.g., diagnosis and procedure codes, and patient demographics.

FINDINGS:

• Despite evidence on the accuracy and completeness of VA data, all PSIs misidentified true events to some extent, with considerable PPV variation across PSIs. PPVs ranged from 28% for post-operative hip fracture to 87% for post-operative wound dehiscence. This variation was due to coding inaccuracies or limitations (e.g., lack of precise or meaningful codes, poor documentation).
• PSI rates were generally low. Ulcer and respiratory failure were the most commonly flagged PSIs, suggesting clinical areas for targeting and opportunities for hospital improvement.
• VA PSI rates will be reported on both the VA and CMS Hospital Compare websites in the near future. However, results suggest that additional coding improvements are needed before the PSIs evaluated in this study are used for hospital reporting or pay-for-performance.

LIMITATIONS:

• Investigators were unable to report other aspects of criterion validity, e.g., specificity, sensitivity, or negative predictive value of the PSIs.
• The sample size was small, preventing investigators from examining whether there was variation in coding accuracy across hospitals.

AUTHOR/FUNDING INFORMATION: This study was funded by HSR&D (SDR 07-002). Dr. Rosen is part of HSR&D’s Center for Organization, Leadership and Management Research, Boston, MA.

Rosen A, Itani K, Cevasco M, et al. Validating the Patient Safety Indicators in the Veterans Health Administration: Do They Accurately Identify True Safety Events? Medical Care January 2012;50(1):74-85.