BACKLOG MAINTENANCE MODELLING
A copy of an e-interview with Dr Conrow can be freely downloaded from
http://www.risk-services.com/
eintvw1.htm, where he discusses these ‘fundamental flaws’, and the scale of the impact they have on the assessed risk when compared against actual measured assessments.
Risk matrix ‘condemned’ by DoD guidance
In 2003 the DoD published guidance supporting Dr Conrow’s work, and condemning the (what is now) NHS numbered 5x5 matrix design. This document, Risk Management Guide for DOD Acquisition, can again be freely downloaded.3
Table 2: The percentage probability range assigned to each Risk Rating. Failure Descriptors
Rare
Unlikely Possible Likely
Certain
cognitive biases that impair most people’s ability to assess risk.
Inconsistent interpretation of the verbal labels used in ordinal scales among different users and by the same user.
Page 19 states that: ‘There is
a common tendency to attempt to develop a single number to portray the risk associated with a particular event. This approach may be suitable if both probability/likelihood (probability) and consequences/impacts have been quantified using compatible cardinal scales or calibrated ordinal scales whose scale levels have been determined using accepted procedures (e.g., Analytical Hierarchy Process). In such a case, mathematical manipulation of the values may be meaningful and provide some quantitative basis for the ranking of risks. ‘In most cases, however, risk scales are actually just raw (uncalibrated) ordinal scales, reflecting only relative standing between scale levels, and not actual numerical differences. Any mathematical operations performed on results from uncalibrated ordinal scales, or a combination of uncalibrated ordinal and cardinal scales, can provide information that will at best be misleading, if not completely meaningless, resulting in erroneous risk ratings.’
‘Discredited’ by numerous books and papers
During the intervening years, right up to the present day, numerous books, academic papers, and journal articles have discredited the numbered 5x5 risk matrix design. A simple Google search using the words: ‘Risk Matrix Problems’, will produce over 51 million ‘hits’. One such academic paper, published by a very well-respected scientist, Louis Anthony Tony Cox, of the University of Colorado UCD Department of Biostatistics and Informatics, can again be downloaded free.4
To summarise: the research and case histories that have taken place over the past 27 years by numerous scientists have shown the problems generated through the use of the numbered 5x5 Risk Matrix, and that these issues are rarely known by the user. Consequently the user does not compensate for these fundamental failures, namely: Accounting for the findings of psychological research concerning the
40 Health Estate Journal May 2019
Treating these scales as if they are ratio scales, with the result that they draw invalid inferences.
Simple scoring methods that do not consider correlations that would change the relative risks.
Multiplication of uncalibrated ordinate scales is mathematically forbidden.
IEC/ISO guidance
In recognition and acceptance of this research the international community in 2009 published IEC/ISO 31010, Edition 1.0, 2009, Risk management – Risk assessment techniques.5
On page 82, in
relation to ‘Risk indices’, the document warns that it is: ‘Strictly not valid to apply mathematical formulae to rating (ordinal) scales, and that in many situations where indices are used, there is no fundamental model to define whether the individual scales for risk factors are linear, logarithmic, or of some other form, and no model to define how factors should be combined. In these situations, the rating is inherently unreliable’.
‘Scope’
The scope of this document does not address the problematic issues relating to Heuristics and Biases, etc.6
experienced by individuals and/or groups when
Risk Rating 1
2 3 4 5
% Probability Range <=20
>20 - <=40 >40 - <=60 >60 - <=80 >80
attempting to use such a risk matrix design, as it was considered that the arithmetical approach was sufficient to present the case for the replacement of the recommended NHS numbered 5x5 risk matrix design. Nor does it discuss the possible (probable) legal ramifications for individuals, organisations, or institutions found to deliberately ‘covering up’ or suppressing such information, or knowingly continuing to use these internationally discredited methodologies for the assessment of risk.
The NHS numbered 5x5 risk matrix The NHS Estates guidance document relating to backlog maintenance recommends the use of a numbered 5x5 risk matrix to assist in the identification and prioritisation of projects, as shown in Table 1.1
Each project is formed from an assessment of the condition of the organisation’s buildings, infrastructure, plant, and equipment, and the impact that would be experienced should a failure occur in service. This is combined with an estimation of the probability of such a failure to form a judgement of the level of risk that would be generated when assessed against the following four criteria: Health & Safety. Environment. Business. Operational/building/engineering element.
Table 3: The calibrated vs. uncalibrated numbered 5x5 risk matrix. Probability of Failure
1 Insignificant Minor Moderate Major 1 0.059 2 0.264 3 0.597 4 2.1 Catastrophic 5 5 2 2 3 3 4 4 8 5
Rare Unlikely Possible Likely Certain 1
5
0.059 0.118 0.177 0.236 0.295 2
4 6 8 6
0.264 0.528 0.792 3
9
0.597 1.194 4
2.1 5
5
4.2 10
10
1.791 12
6.3 15
15
1.056 12
10
1.32 15
2.388 2.985 16
20
8.4 20
20
10.5 25
25
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76