Using Administrative Data to Measure Treatment for Veterans with PTSD May Overestimate Delivery of Psychotherapy
BACKGROUND:
In recent years, studies have used various methods to characterize the quality of care for PTSD delivered in VA outpatient clinics. One common method is counting the number of individual encounters administratively coded as psychotherapy as a measure of treatment adequacy. However, it is possible that this method results in an over-estimate of psychotherapy treatment, and that manually reviewing patient records, while more labor intensive, is more accurate. This study sought to determine whether using administrative data to determine the number of psychotherapy sessions Veterans receive is equivalent to manual record review. Investigators identified 100 consecutive Veterans who presented to one VA mental health clinic and screened positive for PTSD. All notes administratively coded as individual psychotherapy in the six months subsequent to a positive screen were identified. Two psychologists then independently reviewed the notes and classified whether or not the encounter was psychotherapy, using criteria from another published VA study. The manually-classified notes were used to develop an automated coding protocol using the Automated Retrieval Console (ARC), a VA-developed natural language processing program. ARC was then used to independently code the notes, and the performance of the automated coding program was compared to manual coding.
FINDINGS:
- Of the 221 notes that were administratively coded as individual psychotherapy for PTSD, 126 (57%) were coded as individual psychotherapy after manual review of records. Thus, nearly half of the encounters that would have been counted as the provision of psychotherapy in large administrative studies appeared to be records of services other than psychotherapy (e.g., intakes, psychological testing).
- Findings suggest that using counts of administrative codes over-estimates the amount of psychotherapy delivered to Veterans with PTSD. This suggests a potential limitation in current studies of the quality of care for PTSD in VA.
- The ARC program replicated the performance of the manual coders in classifying psychotherapy notes very well. This suggests that ARC may help bridge the gap between the accuracy of manual coding and the scope of administrative coding. However, additional research is required on larger datasets prior to drawing statistically significant conclusions.
LIMITATIONS:
- These findings are based on observations made at a single VA site.
- The number of cases reviewed was not large enough to draw statistically significant conclusions.
- Multisite validation of these findings and further development of ARC for this application is required before these findings can be considered generalizeable.
AUTHOR/FUNDING INFORMATION:
This study was partly funded by HSR&D’s Consortium for Healthcare Informatics Research (CHIR). Dr. Shiner is the recipient of a VA New England Early Career Development Award and is part of the VAMC in White River Junction, VT.
Shiner B, D’Avolio L, Nguyen T, et al. Automated Classification of Psychotherapy Note Text: Implications for Quality Assessment in PTSD Care. Journal of Evaluation in Clinical Practice 2011 Feb 14;e-pub ahead of print.
