Methods for evaluating mobile applications for test ordering and diagnosis

Meyer A, Thompson PJ, Desai S, Taylor JR, Wilson KK, Zhang L, Redmond N, Singh H. Methods for evaluating mobile applications for test ordering and diagnosis. Poster session presented at: Diagnostic Error in Medicine Annual International Conference; 2016 Nov 7; Hollywood, CA.

Related HSR&D Project(s)

• CRE 12-033 – Automated Point-of-Care Surveillance of Outpatient Delays in Cancer Diagnosis

Background: There has been a rapid increase in both availability of medical mobile applications and adoption of mobile devices by physicians for clinical decision support. However, most applications are not evaluated in terms of their usefulness as clinical decision support or educational tools. We sought to establish and pilot test methods for evaluating effects of mobile applications on clinical decision making related to test ordering and diagnosis, as well as their perceived usefulness as learning tools. We focused this evaluation on PTT Advisor, a CDC-developed application that assists physicians with follow-up test ordering and diagnosis related to certain types of coagulation and bleeding disorders (when patients have normal prothrombin and abnormal partial thromboplastin times). Methods: We created and refined the evaluation materials, including 8 paper-based patient vignettes and a questionnaire. The vignettes consisted of challenging clinical scenarios prompting physicians to make laboratory test ordering and diagnosis decisions in two conditions: with the help of the mobile application or with the help of usual clinical decision support. With these vignettes, we assessed impact on clinical decision making by collecting percent of test ordering and diagnostic decisions made correctly, reported confidence, and time taken to assess the vignettes. The questionnaire consisted of questions meant to assess physicians' perceptions about the application's usefulness for clinical decision making and learning using a modified Kirkpatrick Training Evaluation Framework. We pilot tested the evaluation materials with physicians specializing in internal medicine or hematology/oncology. Each physician solved 4 random vignettes in each condition. Results: Preliminary results with 6 physicians suggest advantages for using PTT Advisor over usual clinical decision support on mean accuracy (85.5% vs 80.2% correct; p = .40) and mean confidence in decisions made (8.8 vs 7.0 out of 10; p = .06), as well as a reduction in mean vignette completion time (3:14 min. vs 3:42 min.; p = .58). Additionally, physicians reported positive perceptions of the application's potential for improved clinical decision making and learning. However, a desire was also noted for the application to address broader medical issues. Conclusion: Given enthusiasm for mobile applications, evaluating their effects on physicians' decision making is essential. We propose our evaluation method is a useful model for evaluating mobile clinical decision making and educational applications. Additionally, our preliminary evidence shows that the mobile medical application, PTT Advisor, can contribute to better clinical decision making and serve as a learning tool. We are presently conducting a larger evaluation study to verify these findings.