Impact of different cover letter content and incentives on non-response bias in a sample of Veterans applying for Department of Veterans Affairs disability benefits: a randomized, 3X2X2 factorial trial.

Murdoch M, Clothier BA, Beebe TJ, Bangerter AK, Noorbaloochi S. Impact of different cover letter content and incentives on non-response bias in a sample of Veterans applying for Department of Veterans Affairs disability benefits: a randomized, 3X2X2 factorial trial. BMC medical research methodology. 2022 Mar 6; 22(1):61.

Related HSR&D Project(s)

• IIR 14-004 – Biased Survey-Based Recruiting in Longitudinal Studies: Extent and Remedies

Dimensions for VA is a web-based tool available to VA staff that enables detailed searches of published research and research projects.

If you have VA-Intranet access, click here for more information vaww.hsrd.research.va.gov/dimensions/

VA staff not currently on the VA network can access Dimensions by registering for an account using their VA email address.    Search Dimensions for VA for this citation
* Don't have VA-internal network access or a VA email address? Try searching the free-to-the-public version of Dimensions

Search for Abstract from PubMed

---

Abstract:

BACKGROUND: Non-random non-response bias in surveys requires time-consuming, complicated, post-survey analyses. Our goal was to see if modifying cover letter information would prevent non-random non-response bias altogether. Our secondary goal tested whether larger incentives would reduce non-response bias. METHODS: A mailed, survey of 480 male and 480 female, nationally representative, Operations Enduring Freedom, Iraqi Freedom, or New Dawn (OEF/OIF/OND) Veterans applying for Department of Veterans Affairs (VA) disability benefits for posttraumatic stress disorder (PTSD). Cover letters conveyed different information about the survey''s topics (combat, unwanted sexual attention, or lifetime and military experiences), how Veterans'' names had been selected (list of OEF/OIF/OND Veterans or list of Veterans applying for disability benefits), and what incentive Veterans would receive ($20 or $40). The main outcome, non-response bias, measured differences between survey respondents'' and sampling frame''s characteristics on 8 administrative variables, including Veterans'' receipt of VA disability benefits and exposure to combat or military sexual trauma. Analysis was intention to treat. We used ANOVA for factorial block-design, logistic, mixed-models to assess bias and multiple imputation and expectation-maximization algorithms to assess potential missing mechanisms (missing completely at random, missing at random, or not random) of two self-reported variables: combat and military sexual assault. RESULTS: Regardless of intervention, men with any VA disability benefits, women with PTSD disability benefits, and women with combat exposure were over-represented among respondents. Interventions explained 0.0 to 31.2% of men''s variance and 0.6 to 30.5% of women''s variance in combat non-response bias and 10.2 to 43.0% of men''s variance and 0.4 to 31.9% of women''s variance in military sexual trauma non-response bias. Non-random assumptions showed that men''s self-reported combat exposure was overestimated by 19.0 to 28.8 percentage points and their self-reported military sexual assault exposure was underestimated by 14.2 to 28.4 percentage points compared to random missingness assumptions. Women''s self-reported combat exposure was overestimated by 8.6 to 10.6 percentage points and military sexual assault exposure, by 1.2 to 6.9 percentage points. CONCLUSIONS: Our interventions reduced bias in some characteristics, leaving others unaffected or exacerbated. Regardless of topic, researchers are urged to present estimates that include all three assumptions of missingness.