Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

Health Services Research & Development

Go to the ORD website
Go to the QUERI website

IIR 12-065 – QUERI Project

IIR 12-065
A Novel Approach to Measuring Costs and Efficiency: Lung Nodules as a Case Study
Steven B. Zeliadt PhD MPH
VA Puget Sound Health Care System Seattle Division, Seattle, WA
Seattle, WA
Funding Period: May 2014 - April 2018

Over 400,000 diagnostic chest CT imaging tests are performed each year in the Veterans Health Administration. These tests can identify findings that are highly suspicious and require immediate follow-up or identify indeterminate lung nodules that require surveillance. The goal of this project was to understand the performance of chest CT imaging nationally including the population frequency of highly suspicious and nodule findings identified across VHA, and assess patterns of necessary (and unnecessary) follow-up care and costs associated with that care.

Aim 1. Use natural language processing to operationalize a measure of adherence to clinical guidelines for managing lung nodules, and assess variation in guideline adherence across VHA.

Aim 2. Measure utilization and costs of managing lung nodules, and assess variation in utilization and costs across VHA.

This was a retrospective cohort study of Veterans who received diagnostic chest CT imaging in FY2011. The findings from each diagnostic chest CT were categorized using a combination of diagnostic radiology codes, ICD-9 diagnostic codes, and natural language processing (NLP). A training (n=1,073) and validation (n=270) process was used with the WEKA (Waikato Environment for Knowledge Analysis) machine learning tool, version 3.8. Each patient's index/first CT in FY2011 was classified into one of five mutually exclusive categories: 1) Malignancy suspected; 2) Pulmonary nodule requiring follow-up; 3) Pulmonary nodule not requiring follow-up; 4) Other finding unrelated to lung cancer; and 5) No notable finding. Clinical and demographic characteristics were extracted from the medical record. Because exposure to histoplasmosis, an airborne fungal infection that is endemic to areas with geographic features such as river valleys, has been hypothesized to be associated with frequency of pulmonary nodules, we developed a county-specific measure of histoplasmosis endemicity related to patterns of outpatient diagnostic data available in VHA. Multivariate mixed logistic regression models were used to examine variation in diagnostic performance of CT imaging adjusting for patient characteristics. The identified cohort was then followed for 12 months, capturing all of the care they received within VHA. This follow-up care included activities considered to be directly related to findings identified on the initial chest CT, such as additional CT and PET imaging tests, biopsy, bronchoscopy, mediastinoscopy and surgical procedures. Procedures after a lung cancer diagnosis were excluded. Following identification and categorization of the follow-up care received by the cohort (stratified by initial findings on chest CT), we assessed the costs attributed to this follow-up care using standardized unit prices for each applicable care activity.

Aim 1. Use natural language processing to operationalize a measure of adherence to clinical guidelines for managing lung nodules, and assess variation in guideline adherence across VHA.

We identified 91,677 individuals across 123 medical facilities who received a chest CT in 2011 who had no history of malignancy or existing pulmonary conditions requiring routine imaging surveillance. Administrative and diagnostic codes were available for 28.3% of CT studies. Natural language processing was used to classify findings for the remaining CT studies. The machine learning validation procedure performed best at identifying category 1 (malignancy suspected) and category 5 (no notable findings), with 75% and 73% correct, respectively. Categories 2, 3, and 4 performed at 22%, 54% and 47% correct, respectively, demonstrating that the machine learning tool had difficulty distinguishing between nodules needing additional follow-up and nodules noted by radiologists that did not require further surveillance. Extensive revision and modifications to the WEKA tool, as well as other NLP tools including the Automated Retrieval Counsel (ARC) tool were conducted; the highest accuracy achieved with machine learning was 53.3%.

Using all available information, 16% of the CTs for the cohort were classified as category 1 (malignancy suspected), 28% as category 2 (pulmonary nodule requiring follow-up), 14% as category 3 (pulmonary nodule not requiring follow-up), 22% as category 4 (other finding unrelated to lung cancer), and 22% as category 5 (no notable finding). Among patients with category 1 findings, 17.5% (2,506/14,296) were diagnosed with lung cancer, and among patients with category 2 findings, 3.1% (807/25,669) were diagnosed with lung cancer. The frequency of each category of findings varied considerably across the 123 medical facilities. The interquartile range across facilities was 10%-18% for category 1 findings, and 20%-33% for category 2 findings. Multivariate adjustment accounting for patient characteristics did not significantly reduce the amount of variation in frequency of suspicious findings observed across medical facilities. Focusing on the 25,668 patients identified with category 2 (nodule requiring follow-up) findings, a total of 77,868 subsequent follow-up procedures were identified, with 35% of patients receiving some type of surveillance within 12 months, consistent with guidelines. The percentage of individuals receiving follow-up procedures within 12 months varied across facilities from 5.9%-67.7%.

Aim 2. Measure utilization and costs of managing lung nodules, and assess variation in utilization and costs across VHA.

Patients with category 1 (malignancy suspected) findings received 4,864 invasive diagnostic procedures (0.3 per patient) and 34,621 subsequent imaging procedures (2.4 per patient). Patients with category 2 (nodule requiring follow-up) findings received 2,686 invasive diagnostic procedures (0.1 per patient) and 48,472 subsequent imaging procedures (1.9 per patient). Patients with category 3 (nodule not requiring follow-up) findings received 1,025 invasive diagnostic procedures (0.8 per patient) and 21,536 subsequent imaging procedures (1.7 per patient). The average cost per patient associated with managing category 1, 2, and 3 findings was $1,041, $784, and $665, respectively. Notably, the average cost per individual in each category varied substantially across facilities.

Identifying pulmonary nodules that require surveillance is challenging. Coding of these findings is not routine, and radiology reporting of nodule findings, even in traditional dictated text reports, lacks specific information to systematically determine what type of follow-up care is recommended by clinical guidelines. As a result of the lack of systematic assessment of diagnostic chest CT imaging findings, there is wide variation across medical facilities in the performance of chest CT imaging and adherence to guideline-concordant follow-up care. Quality improvement initiatives are needed to improve surveillance of pulmonary nodules, which can substantially reduce costs associated with unnecessary and guideline-discordant follow-up care for Veterans.

External Links for this Project

NIH Reporter

Grant Number: I01HX000940-01A2

Dimensions for VA

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

If you have VA-Intranet access, click here for more information

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.
    Search Dimensions for this project


Journal Articles

  1. Farjah F, Halgrim S, Buist DS, Gould MK, Zeliadt SB, Loggers ET, Carrell DS. An Automated Method for Identifying Individuals with a Lung Nodule Can Be Feasibly Implemented Across Health Systems. EGEMS (Washington, DC). 2016 Aug 26; 4(1):1254. [view]
  2. Makarov DV, Hu EY, Walter D, Braithwaite RS, Sherman S, Gold HT, Zhou XH, Gross CP, Zeliadt SB. Appropriateness of Prostate Cancer Imaging among Veterans in a Delivery System without Incentives for Overutilization. Health services research. 2016 Jun 1; 51(3):1021-51. [view]
  3. Greene PA, Sayre G, Heffner JL, Klein DE, Krebs P, Au DH, Zeliadt SB. Challenges to Educating Smokers About Lung Cancer Screening: a Qualitative Study of Decision Making Experiences in Primary Care. Journal of Cancer Education : The Official Journal of The American Association For Cancer Education. 2019 Dec 1; 34(6):1142-1149. [view]
  4. Knerr S, Hu EY, Zeliadt SB. Incidence of Neutropenia in Veterans Receiving Lung Cancer Chemotherapy: A Comparison of Administrative Coding and Electronic Laboratory Data. EGEMS (Washington, DC). 2017 Mar 13; 5(1):1269. [view]
  5. Joseph AM, Rothman AJ, Almirall D, Begnaud A, Chiles C, Cinciripini PM, Fu SS, Graham AL, Lindgren BR, Melzer AC, Ostroff JS, Seaman EL, Taylor KL, Toll BA, Zeliadt SB, Vock DM. Lung Cancer Screening and Smoking Cessation Clinical Trials. SCALE (Smoking Cessation within the Context of Lung Cancer Screening) Collaboration. American journal of respiratory and critical care medicine. 2018 Jan 15; 197(2):172-182. [view]
  6. Graf SA, Zeliadt SB, Rise PJ, Backhus LM, Zhou XH, Williams EC. Unhealthy alcohol use is associated with postoperative complications in veterans undergoing lung resection. Journal of thoracic disease. 2018 Mar 1; 10(3):1648-1656. [view]
Conference Presentations

  1. Zeliadt SB, Feemster L, Hammond K, Au DH, Takasugi J. Development of a Natural Language Processing Tool to Categorize Radiology Findings and Assess Appropriateness of Follow-up. Paper presented at: AcademyHealth Annual Research Meeting; 2015 Jun 10; Minneapolis, MN. [view]
  2. Zeliadt SB, Backhus LM, Reinke LF, Hebert PL, Liu C, Hu E, Au DH. Intensity of Diagnostic Chest CT Imaging and False Positive Lung Cancer Rates Before the Introduction of Lung Cancer Screenings in VA. Poster session presented at: AcademyHealth Annual Research Meeting; 2014 Jun 10; San Diego, CA. [view]
  3. Zeliadt SB. Using electronic medical record collected smoking status to assess performance of providers in delivering cessation support. Paper presented at: AcademyHealth Annual Research Meeting; 2015 Jun 10; Minneapolis, MN. [view]

DRA: Cancer, Lung Disorders, Health Systems
DRE: Treatment - Observational
Keywords: none
MeSH Terms: none

Questions about the HSR&D website? Email the Web Team.

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.