BACKLOG MAINTENANCE MODELLING
cases the overall backlog maintenance expenditure was the same, at £171.56 m, and that the difference between the outputs of the two rating (ordinal) scales was a reallocation of expenditure within the four silos of Low, Moderate, Significant, and High risk categories. The next section relates to the effects of Rank Reversal between the two rating (ordinal) scales.
Rank reversal
When the same data was analysed to determine the scale of the ‘rank reversal’ problem, Figure 5 was developed. This clearly shows the different ranking assigned between the calibrated vs. uncalibrated rating (ordinal) scales for the assessment of risk and the prioritisation of backlog projects.
The outputs from this practical application confirm the findings within all of the references cited – that the information created from using uncalibrated rating (ordinal) scales as recommended within the two NHS guidance documents cited is misleading and completely meaningless.
Discussion
Although this discussion paper has used a crude methodology to calibrate the rating (ordinal) scores, it has amply
demonstrated the problems of employing uncalibrated rating (ordinate) scales within the NHS numbered 5x5 Risk Matrix design. For a more detailed study the reader is encouraged to consult the references presented.
It was only from the year 2010 that the Institute of Healthcare Engineering & Estate Management (IHEEM) made public the mathematical fundamental flaw within the NHS numbered 5x5 risk matrix design through a series of presentations and two national publications within Health Estate Journal (2010 and 2011). Between 2010 and 2012 two pilot projects were undertaken on behalf of NHS Wales Shared Services Partnership (Facilities Services) that again confirmed the issues with the NHS numbered 5x5 risk matrix, as described within this document. In 2014 HTM 00, Policies and principles of healthcare engineering, was published,2 which recommended the use of the numbered 5x5 risk matrix reproduced within Table 7.
Although this design employs different numbers for the rating (ordinate) scales, they are still uncalibrated rating (ordinate) scales, and suffer the same mathematical fundamental flaw as described throughout this document relating to the NHS numbered 5x5 risk matrix within the NHS HTM backlog maintenance guidance. Both designs are thus not favoured by the academic, industrial, and international standards community, which warn against their use.
44 Health Estate Journal May 2019 120 100
Uncalibrated Calibrated
80
60
40
20
0 Low Moderate Significant High
Figure 4: A graph showing the estimated Financial Profile within each category of risk between the calibrated and uncalibrated risk rating scales.
HTM 00 (2014) goes on to suggest that ‘a consistent method of assessment should be engaged’. Unfortunately this particular approach does not work, which explains why the international standard IEC/ISO 31010, Edition 1.0, 2009-11 Risk management – Risk assessment techniques5
lists almost 30 risk
assessment techniques in recognition that each potential risk scenario must be assessed employing the most suitable methodology applicable, in the knowledge that ‘no-one-shoe-fits-all’. If it could be possible to effectively ‘shoe- horn’ risk assessments into such a simple configuration, then the international community would all be operating to this system.
A ‘groundswell of concern’ With the demise, in its earlier form, with a very sizeable workforce, of NHS Estates as it once was, and various reorganisations at a national level throughout the UK, it is taking time for the problems described within this document
25
to become common knowledge throughout the NHS; that stated, there is now a groundswell of people expressing concern and asking for change. Nevertheless, it is, in my view, indefensible for an organisation (or person[s]) to ignore the failings within the current national policy when known, particularly when the ramifications and scale of these problems are so large. I would thus suggest that individual organisations do not wait for a change in national policy to be published, as this may not in fact happen for some considerable time.
Conclusions
This article – based on a document that I had written – was developed to highlight the problems with the numbered NHS 5x5 risk matrix, by employing a crude example relating to the subject of backlog maintenance to illustrate the issues and their scale, thereby showing that the NHS recommended methodology generates totally misleading information that is completely meaningless. On this basis,
20
Uncalibrated Calibrated
15
10
5
0 Data
Figure 5: The effects of rank reversal between the calibrated and the uncalibrated risk rating scales.
Risk score
£ millions
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76