Session number: 1040
Abstract title: Development of an Automated Scoring System for Computerized Clinical Vignettes
Author(s):
J Luck - VA Greater LA Health Care System
JW Peabody - SFVAMC
B Lewis - RAND
Objectives: Computerized clinical vignettes are a valid, low cost, case-mix adjusted measure of physician competence and actual practice that have the potential to be used widely to measure quality. For computerized vignettes to realize their full efficiency, an automated scoring system is necessary. We measured how well an automated scoring system compares to a trained abstractor’s manual scoring
Methods: We administered computerized clinical vignettes to 120 randomly selected primary care physicians in two VA and two non-VA medical centers. These vignettes measured quality for 4 conditions: COPD, diabetes, vascular disease and depression. Physicians completed a simple and a complex case for each condition, resulting in 926 completed vignettes. Scoring was done by a trained abstractor. Approximately 65 explicit evidence-based criteria were scored per case. We concurrently developed an automated scoring algorithm that scored exactly the same criteria. The algorithm examined text responses to vignettes for the presence of patterns that corresponded to correct or incorrect answers. One half of completed vignettes were used to develop initial patterns; the other half were used to test the scoring engine and make comparisons with the trained abstractor. Percentage agreement was calculated between the test set and the manual scoring results (considered the gold standard).
Results: Agreement between manual and automatic scoring ranged from 86 per cent (Vascular disease) to 94 per cent (COPD) across the 8 cases, with a mean value of 90 per cent. Agreement of 85 per cent or greater was observed for 75 per cent of individual criteria scored. Agreement of 95 per cent or greater was observed for 42 per cent of individual criteria. Development of automated scoring patterns also identified an error rate of 1.2% in the manually scored items.
Conclusions: An automated scoring engine can score physician responses to open-ended computerized clinical vignettes with 90 per cent accuracy.
Impact statement: Vignettes, which have been shown to be valid methods for measuring quality, can be used more widely to measure quality. Automated scoring reduces costs of scoring and provides a rapid method for evaluating the quality of practice after interventions are used to improve clinical practice and to make comparisons between VA facilities.