BACKGROUND/RATIONALE:
Crucial to the ongoing drive toward improving quality of inpatient care and outcomes is the ability to assess hospital performance. While VA has made great strides in automating and systematizing patient data, there has been little research in evaluating statistical methods that accurately identify excellent, average or poor performers.

OBJECTIVE(S):
Despite increased use of hospital profiling, there is concern regarding the appropriateness of the statistical profiling method used. We examine two commonly used methods: traditional logistic regression leading to observed to expected ratios and the random effects (RE) hierarchical logistic regression method used in the VA/CMS HospitalCompare program. The two methods result in different profiles of VA hospitals when examining 30-day mortality for acute myocardial infarction (AMI) patients. There is no gold standard for identifying which profiling method "gets it right". To evaluate the two methods, we used simulated data for which the true hospital performance was predetermined. Implications of findings for VA hospital profiling are assessed.

METHODS:
The simulated data consist of randomly generated measures of patient risk, hospital performance and patient dichotomous outcomes. We developed a series of simulated scenarios by incrementally varying hospital volumes and inter-hospital variation in performance (intra class correlation [ICC]). For each simulation we obtained standardized hospital risk-adjusted mortality rates (SMRs) using traditional and RE methods and classified hospitals into above-average, average and below-average SMR categories. These were contrasted with the true SMR category to calculate sensitivity, specificity and positive predictive values (PPV +/-).

FINDINGS/RESULTS:
Overall performance indicated higher discrimination from the traditional method, compared to the RE method, in identifying above-average SMR hospitals (c-statistic: 0.89 vs. 0.81, p-value<0.001) and below-average SMR hospitals (0.90 vs. 0.88, p-value=0.02). Both hospital volume and ICC were important factors in this performance differential. Traditional method indicates higher discrimination for the lowest quartile of hospitals by discharge volume; no such difference was noted for other volume quartiles. Similarly, for ICC of 1% and 2%, discrimination was higher for traditional method; this difference was negligible for ICC of 5% or greater.

IMPACT:
To improve quality of hospital care we need valid methods for identifying the high and low performers. Our work provides a basis for evaluating current methods for identifying high and low performers after consideration of tradeoffs between sensitivity and specificity. It contributes to the growing literature on pros and cons of the current profiling methods.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX000404-01
Link: https://reporter.nih.gov/project-details/7998299

---

PUBLICATIONS:

Journal Articles

1. Hanchate AD, Ash AS, Borzecki A, Abdulkerim H, Stolzmann KL, Rosen AK, Fink AS, Pugh MJ, Shokeen P, Shwartz M. How pooling fragmented healthcare encounter data affects hospital profiling. The American journal of managed care. 2015 Feb 1; 21(2):129-38. [view]

Conference Presentations

1. Hanchate AD. Evaluation of Statistical Methods for Profiling Hospital Outcome Performance: The Case of Acute Myocardial Infarction 30-day Mortality. Poster session presented at: AcademyHealth Annual Research Meeting; 2012 Jun 24; Orlando, FL. [view]
2. Hanchate AD. Evaluation of Statistical Methods for Profiling Hospital Outcome Performance: The Case of Acute Myocardial Infarction 30-day Mortality. Paper presented at: VA HSR&D / QUERI National Meeting; 2012 Jun 16; National Harbor, MD. [view]

DRA: Health Systems Science
DRE: Treatment - Observational, Research Infrastructure
Keywords: none
MeSH Terms: none