2031. Inter-coder Reliability
of ICD9 Assignment: Flaws in an Important Diversity Assessment Tool
JF Hurdle, VA Salt Lake City Health Care System, CR Weir, Salt Lake City Health Care System, JR Nebeker, Salt Lake City Health Care System, JM Hoffman, Salt Lake City Health Care System
Objectives: During a
large-scale, HSRD-funded study of adverse drug events (ADEs), we found evidence
that ICD9 coders were non-uniform in assigning codes that we conclusively had
shown to be documented in the chart. Concerned
about overall ICD9 reliability, the objective of this study was to measure
reliability across coders at several different hospitals for all diagnoses in a
set of cases.
Methods: We randomly
selected 80 admissions from 237 (34%) known to have one or more ADEs in our
previous study (we limited the study to ADE cases as a convenience sample for a
later study). We hired three certified (RHIA or CCS) coders employed separately
at nonVA hospitals to independently review the 80 admissions (for a total of
four coders including ours). We calculated inter-case agreement (three-digit
code match) and aggregate statistics. Coders were limited to a maximum of 10
codes per case (following local practice).
four coders logged a total of 1,994 ICD9 codes (mean 6.2 codes/case). Overall,
the sum of unique codes across all cases was 901; of these 168 (18.6%) agreed
4-ways (unanimous); 141 (16.5%) agreed 3-ways; and 171 (19%) agreed two-ways.
The remaining 45.9% were codes that only one coder found in a case. The overall
average agreement was very low at 2.2 codes/case.
Conclusions: ICD9 codes
affect diverse populations of veterans in profound ways. For an admission, they
are arguably THE most important metric of clinical outcome and resource
consumption, usually reported by population subgroup. Decisions about a host of
issues for these subgroups range from billing-reimbursement to resource
allocation to clinician productivity profiling to DRG assignment to station- and
VISN-performance measures to VA versus nonVA care provision, all hinging on ICD9
code sets. While many studies have assessed how accurate a set of codes is for a
given admission, or show how certain diagnostic groups defy accurate coding,
this study is unique in focusing on coding reliability between institutional
Impact: These results suggest that a concerted effort needs to reconcile cross-institutional differences in this crucial metric; at a minimum such an effort will clarify the quality and quantity of care provided to diverse veteran populations.