BACKLOG MAINTENANCE MODELLING
Understanding the ‘problem’ with the linear scale
In order to appreciate the problem with this linear scale it is necessary to provide the following basic demonstration. Assume that the impact of a simple cut on a finger requiring the application of a plaster has been assessed as generating a ‘Minor’ impact, and is being compared to an incident where the legs have been amputated, attracting an impact rating of ‘Major’ (NB, ‘Catastrophic’ is rated as death). The resulting ratings for these two incidents would be 2 and 4 respectively. The linear scale will assume that the cut finger has half the impact of that of the amputation; this is clearly wrong. To correct this, the scale of impact must be calibrated to show the difference between the two incidents.
There are two types of information – quantitative and qualitative. The former consists of numbers, e.g. financial costs, but the latter relates to a person’s perception of a particular situation. Within the example above we are dealing with a qualitative problem where numbers do not exist. We must therefore rely on what is termed ‘Expert Judgment’. Where measurements, observation, experimentation, or simulation are unavailable, Expert Judgement is used within the decision analysis; it is also known as expert opinion, subjective judgement, expert forecast, best estimate, educated guess, and, most recently, expert knowledge.
Repeatable and robust outputs Much research has been devoted to developing the elicitation techniques and methodologies for extracting information from experts to produce repeatable and robust outputs.10–13
By applying these
techniques to the impact scales, the calibrated non-linear convex blue line was formed within Figure 2, with the corresponding rating (ordinal) scores. The academic term given to the
25 25
20
Uncalibrated Calibrated
15
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Data points
Figure 2: Rank comparison effects between the calibrated vs. uncalibrated risk rating scales.
calibrated assessment is a ‘Utility Function’. A number of people (experts) have been involved in producing a series of utility functions under controlled conditions, and have all developed calibrated scales very close to that presented within Figure 1. The slight variation between these calibrated scales has had negligible effect on the outputs of this study. Apart from possessing the same rating for the impact of catastrophic (death), all of the other four ratings are below that formed from the linear uncalibrated rating (ordinal) scores. It must be noted that the maximum rating (ordinal) score for both the uncalibrated and calibrated scales is 5, in both cases corresponding to catastrophic (death), and that the minimum rating (ordinal) score for the uncalibrated scale cannot fall below 1 due to the integer nature of the design, but the calibrated scale can (and does) fall below 1.
A ‘crude’ tool
Readers should also note that the resulting calibrated scale developed for this demonstration is crude, as it only
grades an amputation as being seven times more severe than a cut figure. However, although crude, the effects of comparing a calibrated vs. uncalibrated scale are demonstrable within this example.
Probability of failure
In order to simplify this example, it has been assumed that the bands of probability shown in Table 2 have been assigned to the ratings of the five descriptors, and that the scale of the probability ratings is linear (i.e. a straight line).
Comparison of the calibrated vs. uncalibrated risk scores By superimposing the calibrated and uncalibrated rating scores for impact and probability of failure into one matrix, Table 3 was developed.
20
Uncalibrated Calibrated
15
Each of the 25 boxes (data points) within the matrix contains the product of the impact and probability of failure risk scores for both the calibrated and uncalibrated rating (ordinal) scales. By comparing the risk profiles of both rating (ordinal) scales Table 4 was formed. The table shows the calibrated data points in rank order against the ranking formed from the uncalibrated scales. It can be seen that there is an overestimation of risk with the uncalibrated integer scale when assessed against the calibrated scale. The table also shows that 12 of the 25 data points (48%) are subjected to Rank Reversal.
10 5
Figure 2 was developed from Table 4, and gives a pictorial view of the overestimation of risk and the effects of Rank Reversal.
Practical application 0 Data
Figure 3: A comparison between the risk profiles created from the calibrated and uncalibrated risk rating scales.
42 Health Estate Journal May 2019
In order to demonstrate the scale of the issues caused by the mathematical fundamental flaw within the NHS numbered 5x5 risk matrix design, the outputs from the calibrated and
Risk score
Risk score
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76