Using Natural Language Processing to Evaluate Evidence-Based Treatment for Veterans with PTSD
BACKGROUND:
Given the high prevalence of PTSD among Veterans who use VA healthcare, the impairment associated with PTSD, and the amount of care VA provides for PTSD, it is essential that this care be as effective as possible. A major limitation in the ability to track the use of evidence-based therapies on a regional or national level has been the need to manually review the content of treatment notes to determine whether the type of psychotherapy being delivered follows practices shown to be effective for PTSD. This study examined whether automated coding of treatment notes using natural language processing could match the accuracy of manual chart review for classifying evidence-based PTSD treatments. Investigators analyzed the content of clinical notes using the automated retrieval console (ARC). ARC "learns" from a set of gold standard interpretations made by human chart reviewers and combines natural language processing and machine learning technology to replicate their judgments. Using this automated process, they evaluated the use of evidence-based psychotherapy treatment for over 1900 Veterans newly enrolled in six VA outpatient PTSD clinics in New England during FY10, following extensive national efforts to increase use of evidence-based psychotherapies for PTSD.
FINDINGS:
- ARC identified psychotherapy notes and classified the type of therapy performed in a treatment session as reliably as expert human raters.
- Evidence-based psychotherapies appear to be infrequently used as an initial treatment in VA outpatient PTSD clinics in New England. Of the Veterans in this study, ony 6% received at least one session of one of the evidence-based psychotherapies outlined in the VA/DOD guidelines for PTSD during the initial six months of treatment.
LIMITATIONS:
- It is possible that patients in this study were receiving other evidence-based treatments, such as pharmacotherapy. Examination of pharmacy records was beyond the scope of this study.
- This study was limited to Veterans receiving treatment in specialized PTSD clinics in New England.
IMPLICATION:
- The ARC method uses a clinican-friendly interface, does not require custom programming, and is freely available to others, so could be used to evaluate treatment for other mental health disorders and in other healthcare settings that use electronic medical records.
AUTHOR/FUNDING INFORMATION:
This study was partly supported through VA/HSR&D's Quality Enhancement Research Initiative (QUERI: RRP 11-432) and through the VA Consortium for Health Informatics Research (CHIR). Drs. Shiner and Watts are part of the White River Junction VAMC and the VA New England Veterans Engineering Resource Center (VERC), White River Junction, VT. Dr. Shiner is also part of the VA National Center for PTSD, White River Junction, VT.
Shiner B, D’Avolio L, Nguyen T, Zayed M, Young-Xu Y, Desai R, Schnurr P, Fiore L, and Watts B. Measuring Use of Evidence-Based Psychotherapy for Post-Traumatic Stress Disorder. Administration and Policy in Mental Health and Mental Health Services Research April 26, 2012; Epub ahead of print.
