2023 HSR&D/QUERI National Conference

1129 — Improved prediction of opioid-related adverse health outcomes from electronic medical record data

Lead/Presenter: Ralph Ward,  COIN - Charleston
All Authors: Ward RC (Health Equity and Rural Outreach Innovation Center, Ralph H Johnson VAMC, Charleston SC), Weeda E (Health Equity and Rural Outreach Innovation Center, Ralph H Johnson VAMC, Charleston SC) Taber DJ (Health Equity and Rural Outreach Innovation Center, Ralph H Johnson VAMC, Charleston SC) Gebregziabher M (Health Equity and Rural Outreach Innovation Center, Ralph H Johnson VAMC, Charleston SC)

Veterans suffer disproportionate health impacts from the opioid epidemic, including overdose, suicide, and death. Prediction models based on electronic medical record data can be powerful methods for identifying patients at greatest risk of such outcomes. The VHA implemented the Stratification Tool for Opioid Risk Mitigation (STORM) in 2018, a model that has become an important tool for identifying Veterans at greatest risk for suicide or overdose events related to opioid use. Our objectives were to develop and apply alternative statistical and machine learning methods with superior prediction performance when compared to the original STORM model.

Prognostic study designed to produce an improved model for predicting opioid overdose and suicide-related events among Veterans. CDW data in the VA Informatics and Computing Infrastructure (VINCI) was used with corresponding CMS Medicare data. The study cohort was a 20% random sample of Veterans between 2013-2018 who had an opioid prescription, an OUD diagnosis or a prescription medication overdose during the study period. Patients with cancer or hospice care were excluded. Predictor variables for each year were used to predict patient outcomes (overdose, suicide-related events) in the following year. We applied advanced statistical modeling techniques (joint outcome multi-trajectory generalized linear mixed model (mGLMM), elastic-net penalized regression) and machine-learning methods (random forest) to support improved prediction of opioid-related overdose and suicide.

All of the attempted modeling techniques benefited from the incorporation of additional longitudinal data sources and new predictor variables types compared to those used in the original STORM model. One of these proposed models used a multivariate generalized linear mixed modeling (mGLMM) approach to produce separate predictions for overdose and suicide-related events (SRE) from a joint model rather than a single prediction for combined outcomes. Compared to a modified version of the STORM model with the same outcome, predictor and interaction terms, the mGLMM model has a significantly better prediction performance in terms of AUC (84% vs. 77%) and sensitivity (71% vs. 66%). The mGLMM performed particularly well in identifying patients at risk for SREs, where 72% of actual events were accurately predicted among patients with the 100,000 highest risk scores compared with 49.7% for the modified STORM model. Other methods (elastic-net penalized regression and random forest) provided similar results.

The mGLMM’s strong performance in identifying true cases (sensitivity) among this highest risk group was the most important improvement given the model’s primary purpose for accurately identifying patients at most risk for adverse outcomes such that they are prioritized to receive risk mitigation interventions. Some predictors in the proposed model have markedly different associations with overdose and suicide risks, which will allow clinicians to better target interventions to the most relevant risks. The other models (elastic-net and random forest) provided strong results as well, and further work is warranted to develop an ensemble model that develops predictions for a patient based on ‘votes’ from the 3 methods described here. This ensemble model is expected to further boost prediction performance.

This approach involves innovative steps not seen in current VA risk models for opioid overdose and suicide-related events. New data sources, variable types and advanced statistical and machine learning methods were applied to boost prediction performance. This work has the potential for life-saving impacts by driving improvements in VHA clinical decision support tools used in making evidenced-based decisions to help prevent overdose or suicide-related events.