advantages of automating vae detection 829
rechecked by a subset of the authors (E.S.S., E.S.R., M.B.W., E.E.R., N.S.), who were aware of the automated surveillance and manual surveillance interpretations, until consensus was reached. The reasons for each final determination were summarized and presented to senior IC staff. Adjudicated classification of each VAE event had to be signed off by senior IC staff before being considered final. Errors detected during the development cohort through the process of determination of the gold standard resulted in algorithm programming updates. The updated algorithm was then applied to the validation cohort.
Reported Outcomes
Clinical and demographic characteristics of the study popula- tion were extracted from the EHR, including age, gender, admitting diagnosis, duration of mechanical ventilation, and length of stay. Confusion matrices and 2×2 tables comparing manual surveillance and automated surveillance to adjudicated classifications were created. The sensitivity, specificity, and PPV of manual surveillance and automated surveillance compared to adjudicated classifications were calculated. For detection errors generated by manual surveillance, study
staff categorized the errors by type: missed events (failure to detect the VAC, IVAC, or PVAP); misclassified events (detected but misclassified VAE), which was further classified as underclassification (eg, classified an IVAC event as VAC) or overclassification (eg, classified an IVAC event as PVAP); and false detections (ie, manual surveillance detected a VAE that did not meet the NHSN definition). The automated surveillance system produced a visualization
of each VAE that provided ease of interpretation for how criteria were met.
results Cohort Characteristics
There were 1,325 and 1,234 admissions to the 4 ICUs during the development and validation periods, respectively. The cohort characteristics are summarized in Table 1. The cohorts were similar in age and gender. Among ventilated patients in the development cohort (N=1,325), the median duration of ventilation was 1.9 days. Among ventilated patients in the
table 1. Cohort Characteristics Characteristic
Age, median y, (IQR) Female, no. (%)
Ventilated, no. (%)
Length of ventilation among ventilated admissions, median d (IQR) Length of stay, median d (IQR) VAE detected (adjudicated)
NOTE. IQR, interquartile range; VAE, ventilator-associated events. aJanuary–March 2015. bJanuary–March 2016.
validation (N=1,234) cohort, the median duration of venti- lation was 2.2 days. The length of ICU stay was 7.7 days in the development cohort and 7.5 days in the validation cohort (Table 1). During the development period, a total of 47 VAEs were identified in the development cohort, including 28 VACs (60%), 12 IVACs (26%), and 7 PVAPs (15%). During the validation period, a total of 56 VAEs were identified in the validation cohort, including 44 VACs (79%), 12 IVACs (21%), and no PVAPs.
Performance of Manual and Automated Surveillance
In the development cohort, for manual surveillance, sensitivity was 40%, specificity was 89%, and PPV was 70%. For auto- mated surveillance sensitivity was 100%, specificity was 100%, and PPV was 100% (Figures 2A and 2B). In the validation cohort, for manual surveillance, sensitivity
was 71%, specificity was 98%, and PPV was 87%. For auto- mated surveillance, sensitivity was 85%, specificity was 99%, and PPV was 100% (Figures 3A and 3B). During the validation period, a temporary interruption of data archiving occurred, resulting in loss of data. In all cases with data available, the algorithm made no errors.
Classification of Detection Errors
Manual surveillance yielded 73 detection errors in the combined development and validation cohorts. Of these, 47 (64%) were missed detections; 12 (16%)weremisclassifications, and 14 (19%) were false detections. Among the missed detections, 37 of 47 (79%) were due to misapplication of the PEEP or FiO2 criterion. Amongmisclassifications, 8 of 12 (67%) were underclassified and 4of 13(33%)wereoverclassified. Among false detections, 8 of 14 (57%) were due to errors in applying the PEEP/FiO2 criterion (Table 2). “Misapplication” refers to the incorrect application of VAE definitions rather than errors arising from missing data.
Case Detection Visualization
Each case reviewed by automated surveillance produced a detection visualization. These figures provided a visual depiction of the criteria used by the algorithm to detect and classify cases.
Development Cohorta (N=1,325)N=1,325
63 (51–72) 606 (46) 479 (36)
1.9 (0.6–5.4) 7.7 (4.0–13.8) 47
Validation Cohortb (N=1,234)
62 (49–72) 550 (45) 431 (35)
2.2 (0.7–6.5) 7.5 (4.1–14.7) 56
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84 |
Page 85 |
Page 86 |
Page 87 |
Page 88 |
Page 89 |
Page 90 |
Page 91 |
Page 92 |
Page 93 |
Page 94 |
Page 95 |
Page 96 |
Page 97 |
Page 98 |
Page 99 |
Page 100 |
Page 101 |
Page 102 |
Page 103 |
Page 104 |
Page 105 |
Page 106 |
Page 107 |
Page 108 |
Page 109 |
Page 110 |
Page 111 |
Page 112 |
Page 113 |
Page 114 |
Page 115 |
Page 116 |
Page 117 |
Page 118 |
Page 119 |
Page 120 |
Page 121 |
Page 122 |
Page 123 |
Page 124 |
Page 125 |
Page 126 |
Page 127 |
Page 128 |
Page 129 |
Page 130 |
Page 131 |
Page 132 |
Page 133 |
Page 134 |
Page 135 |
Page 136 |
Page 137 |
Page 138 |
Page 139 |
Page 140 |
Page 141 |
Page 142 |
Page 143 |
Page 144
