406
Katherine E. Goodman et al
Table 4. Comparative Strengths and Limitations of Logistic Regression-Derived Risk Scores and Classification and Regression Tree (CART) Analysis-Derived Decision Trees for Predicting Drug-Resistant Infections in Clinical Settings
Risk Scores Decision Trees Notes
Data Characteristics High dimensionality
Collinearity Interaction effects − − þ þþþ þþþ þþþ
Decision trees are well suited to high-dimensional data, which possess high predictor-to- outcome ratios. Logistic regression-derived risk scores impose more stringent sample size requirements (a general requirement is 10 expected cases per predictor).
Logistic regression-derived risk scores require minimal collinearity among independent variables, unlike decision trees.
Logistic regression can accommodate interaction effects, but it requires moderately large sample sizes and a priori evaluation. CART decision trees can detect simple and higher- level interaction effects without user specification.
Rare outcome(s) þþ Rare outcomes pose challenges for both models. In logistic regression, rare outcomes limit the number of evaluable predictors. CART analysis may require parameter adjustment and/or case
oversampling before model fitting and validation to improve sensitivity if outcomes are rare.
Model development Ease of development
Robustness to overfitting þþ þþ þ −
Decision trees for standard applications are relatively straightforward to develop, but logistic regression-derived risk-score methodology is more well known in the infectious disease literature and more widely available on all common statistical computing platforms.
Both methods require validation, but decision trees are particularly prone to overfitting, in which they fit the data “too well” and may consequently perform poorly on new data. Methods to combat overfitting include imposing branching-stop criteria and “pruning” back terminal branches.
Implementation and usage Intuitiveness Ease-of-use
Adaptability
End-user adjustment of sensitivity and specificity
Addition of new variables over time
þþþ þþ − þ
By changing the score cutoff point, individual users can tailor risk scores’ sensitivity and specificity. A decision tree possesses a fixed sensitivity and specificity that, following model development, cannot be modified.
New variable(s) can be evaluated for risk score inclusion (eg, by comparing Akaike’s information criterion (AIC) values of the original and expanded models).27 Variable addition may change coefficient values and, accordingly, risk score points but will leave original score variables intact. Because decision trees are built “top-down,” new variables require tree refitting and may substantially alter nodes and branching patterns.
þ þ
þþþ þþþ
Decision-tree branching logic is highly intuitive.
Decision trees generally do not require calculations, making them user-friendly for bedside application.
imperative that the table of cutoff-point sensitivities and specific- ities, and an understanding that an institution’s disease prevalence will affect the positive and negative predictive values, guides deci- sions about score thresholds for ESBL infection. In contrast to risk scores, classification trees provide binary pre-
dictions (eg, “ESBL” or “not ESBL”), with a single sensitivity and specificity value for the tree as a whole. Terminal node percentages (eg, “37% probability of being ESBL positive”) can quantify these predictions but do not provide a formal mechanism for prioritizing sensitivity versus specificity. For research applications in which sensitivity is the priority, methods are available to impose a greater “cost” for case misclassification during the tree-fitting process.23 The limitation, however, is that these mechanisms are not adjust- able by end users after a tree is built. In other words, whereas the CART approach provides flexibility to optimize sensitivity or specificity, once a single, final tree (output) is developed and pro- vided to clinicians, the ability to adjust sensitivity and specificity is limited.
Although these considerations can help researchers to evaluate
whether a risk score or a decision tree is preferable for a given research question (Table 4), a decision is rarely clear cut. In cases in which each model would at least partially meet stated goals, we
encourage investigators to develop both support tools in parallel to compare their performance metrics. In particular, although model performance was comparable in this case study, other applications with more challenging data (eg, high-dimensionality, higher-order variable interactions) might more clearly favor a machine learning approach such as CART. Our study has several limitations. This study was conducted in a
single center, and although we internally validated our models, it lacked an external validationcohort. In addition, datamay have been missing for patients treated outside of the EpicCare Everywhere net- work, althoughwedonot expect suchoccurrences tohave differedby ESBL status. As such, any resulting exposuremisclassificationwould likely reduce predictive performance, and yet risk score discrimina- tion remained robust, including in cross-validation. Nevertheless, we encourage others to evaluate and validate the risk score in their own patient populations, particularly for settings that differ from our academic, tertiary-care hospital cohort. Importantly, however, because study characteristics were constant across analyses, we expect decision tree and risk score comparisons to be unbiased. Finally, this case study intended to offer a practical, high-level introduction to a relatively simple machine learning approach, but we note that many machine learning methodologies (eg, random
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84 |
Page 85 |
Page 86 |
Page 87 |
Page 88 |
Page 89 |
Page 90 |
Page 91 |
Page 92 |
Page 93 |
Page 94 |
Page 95 |
Page 96 |
Page 97 |
Page 98 |
Page 99 |
Page 100 |
Page 101 |
Page 102 |
Page 103 |
Page 104 |
Page 105 |
Page 106 |
Page 107 |
Page 108 |
Page 109 |
Page 110 |
Page 111 |
Page 112 |
Page 113 |
Page 114 |
Page 115 |
Page 116 |
Page 117 |
Page 118 |
Page 119 |
Page 120 |
Page 121 |
Page 122
