2056. Measuring Quality of Care: Is There a Role for Peer Review?
TP Hofer, VA Center of Excellence-Ann Arbor, S Asch, VA Center of Excellence-Los Angeles, M Hogan, VA Center of Excellence-Ann Arbor, J Adams, Rand Corporation, EA Kerr, VA Center of Excellence-Ann Arbor

Objectives: We sought to develop a more reliable structured implicit chart review instrument for use in assessing the quality of care for chronic disease. We hypothesized that we would be more successful for clinical conditions that have a substantial evidence base for treatment in the clinical literature. We also illustrate a method of analysis that allows one to determine the measurement questions to which the instrument is generalizable.

Methods: As part of a larger study comparing implicit and explicit measures of quality across 26 sites in the VA, we developed a structured implicit review instrument for four clinical conditions: COPD, diabetes(DM), hypertension(HTN) and acute conditions. Twelve reviewers carried out 180 reviews of 60 medical records for the care given in up to all four of the conditions over a 13 month period, including inpatient and outpatient care. We estimated the reviewer, record and noise variance for the calculation of reliabilities in a multilevel analysis.

Results: The reliability of a single review for detecting differences in quality of hypertension care and diabetes care across patients is .45. This estimate, which is generalizable to the population of reviewers and patients, is substantially higher than most prior estimates of implicit review reliability. However, for COPD and acute care, two clinical areas where the body of evidence supporting specific interventions is substantially less, the reliability is poor (.26 and .11 respectively). The lower reliabilities are due to decreased signal rather than increased noise.  With a careful reliability study the reviewer effects can be removed raising the reliability for the review of DM and HTN as high as .53.

Conclusions: It is possible to obtain reliable ratings of chronic disease and that higher reliability is achieved for reviews in clinical areas that have a more developed evidence base. With a less developed evidence base the reviewers appear to have more difficulty in making systematic distinctions in the quality of care between patients. This analytic approach allows for generalizable conclusions about the use of these instruments and reasons for low or high reliability.

Impact: Sampling relatively few records for structured implicit review may be a reliable alternative or complement to assessing quality of care by checklist approaches.