BACKLOG MAINTENANCE MODELLING
Table 4: A comparison between the calibrated vs. uncalibrated numbered 5x5 Risk Matrix rating scales for impact and probability of failure.
Impact 5
5 5 4 5 4 4 5 4 3 3 4 3 2 3 2 2 3 2 1
2 1 1 1 1
Five descriptors
The scale of the impact of a failure and accompanying probability of failure are determined from reference to five descriptors that are assigned an integer rating (ordinal) score of 1-5. By multiplying these two rating (ordinal) scores together, an assessment of the severity of the risk is created from the resulting risk score, where the greater the risk score, the greater the risk. For example, from reference to Table 1, assuming that a particular project was assessed as creating a ‘Moderate’ impact, that is ‘Likely’ to fail, this would create rating scores of 3 and 4 respectfully, whose product gives a risk score of 12. This design looks simple and intuitive, and suggests that the mathematical manipulation of the scales may be meaningful and provide some quantitative basis for the ranking of projects. However, the scales are actually just raw (uncalibrated) rating (ordinal) scales, reflecting only relative standing between scale levels, and not actual numerical differences.3–5, 7, 8
Any
mathematical operations performed on results from uncalibrated rating (ordinal) scales provide information that will at best be misleading, if not completely meaningless, resulting in erroneous ranking and severe rank reversal.
Uncalibrated scale Probability
5 4 3 5 2 4 3 1
2 5 4 1
3 5 2 4 3 1
2 5 1
4 3 2 1
Risk 25
20 15
20 10 16 12 5 8
15 12 4 9
10 6 8 6 3 4 5 2 4 3 2 1
Impact 5
5 5
2.1 5
2.1 2.1 5
2.1
0.597 0.597 2.1
0.597 0.264 0.597 0.264 0.264 0.597 0.264 0.059 0.264 0.059 0.059 0.059 0.059
Calibrated scale Probability
5 4 3 5 2 4 3 1
2 5 4 1
3 5 2 4 3 1
2 5 1
4 3 2 1
Therefore this design has been condemned by the academic world and international organisations since the early 1990s, as discussed below.
Explanation of the mathematical ‘fundamental flaw’
Within the mathematical world there are four types of number9
– the first is termed
a ‘group’ number. These, as the name suggests, create a group of like objects, e.g. apples, oranges. No mathematical computations can be applied to group numbers. The second type is an ‘ordinal’ number; these create lists of items in ‘rank order’ only, e.g. shoe size 4 and shoe size 8. Again, no mathematical process can be applied to these numbers. The third type is known as an ‘interval’ number; typical examples often quoted use degrees centigrade. These can be subtracted or added, e.g. 10 degrees is half of 20 degrees. However, as they do not possess an absolute zero they cannot be multiplied or divided. It is for this reason that within thermodynamic equations units of Kelvin are employed. The final and fourth type of number is known as ‘real’. These have an absolute zero, and can be subjected to all mathematical computations. The numbered 5x5 risk matrix within Table 1 is composed of the second type of number, uncalibrated
Rank Reversal
Risk 25
20 15
10.5 10
8.4 6.3 5
4.2
2.985 2.388 2.1
1.791 1.32
1.194 1.056 0.792 0.597 0.528 0.295 0.264 0.236 0.177 0.118
0.059
rating (ordinate) scales. Thus the numbers only form a rank order of assessed impact and probability of failure. The scales are termed ‘uncalibrated’ because the numerical difference between, say, a rating of 2, is not equal to half the rating of 4. When these uncalibrated rating (ordinate) scales are multiplied together to form a risk score for various projects, the resulting list of risk scores does not form a ranked list of projects.
A simple example
This is demonstrated in the following simple example, where the ratings of Impact and Probability of Failure are firstly calibrated, then comparisons made between the calibrated and uncalibrated ratings within the NHS numbered 5x5 risk matrix design, and, finally, these two scales applied to actual data from a large acute NHS Trust to demonstrate the scale of the problems due to the mathematical fundamental flaw within this design.
The formation of ‘Calibrated scales’ Impact
The rating scores for impact and probability of failure within Figure 1 are linear (a straight line) between points, e.g. the rating score of 4 is twice that of the rating score of 2. This is shown as a straight red line within Figure 1.
May 2019 Health Estate Journal 41
x x x x x x x x x
x x
x
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76