COMMENT
In contrast, our evaluation of the Advancing Quality PfP scheme in the NHS North West used robust data and methods. Publishing the fi ndings from the fi rst phase of the evaluation in the prestigious New England Journal of Medicine (NEJM) required rigorous review by subject experts who demanded further analysis and our research paper underwent much scrutiny before it was fi nally accepted for publication.
Advancing Quality is a voluntary programme that provides fi nancial incentives for improvement in the quality of care provided to patients. It has been implemented in the north west of England since 2008. The programme is based closely on a PfP project implemented in the USA.
Advancing Quality was initially designed and supported by a non-profi t US organisation (Premier Inc.) and involved similar quality indicators and fi nancial incentive structures. However, it diff ered from the US variant in a number of respects. Importantly, it involved universal participation of eligible providers and implementation in a diff erent health system.
The detail of the Advancing Quality scheme
The Advancing Quality scheme involved incentives to improve care and reporting of performance on quality measures for fi ve clinical conditions: acute myocardial infarction, heart failure, pneumonia, coronary- artery bypass grafting and hip and knee surgery. The fi rst year was run as a pure tournament. Hospitals were ranked according to performance and those in the top half (i.e. 12 of 24 hospitals) received a bonus payment. For the next six months, fi nancial incentives were awarded based on performance ranking, but also providers whose performance was above the median score from the fi rst year were awarded an ‘attainment’ bonus. There were no penalties for poor performers during these fi rst 18 months.
Knee surgery is one of the fi ve clinical conditions under Advancing Quality
number of unique indicators, which often made benchmarking of performance impossible. Additionally, indicators were interpreted diff erently by diff erent stakeholders.
In contrast, Advancing Quality involved standardised data defi nitions and bespoke software, underpinned by data assurance provided by the Audit Commission to ensure that comparisons (before and after, between trusts and with the rest of the country) were on a like-with-like basis.
A gruelling struggle
But aside from these technical aspects, collaborative events brought together staff from all 24 participating organisations to share their learning and work through common problems. In addition to shared learning, the development of this Advancing Quality ‘community’ appears to have been really important in providing emotional support for what was a gruelling and often uphill struggle for the staff involved.
We compared mortality in the north west for Advancing Quality incentivised conditions with the rest of England. We also chose a number of conditions for which performance improvement was not incentivised and compared these in both groups.
Findings
Our evaluation of what happened during the fi rst 18 months of Advancing Quality found that it was associated with a reduction in mortality in incentivised conditions and that this reduction was signifi cantly greater than in hospitals in the rest of England. Mortality reduced for the non-incentivised conditions in both the north west and the rest of England. But this reduction was not signifi cantly diff erent between the north west and the rest of England. Advancing Quality apparently saved almost 900 lives across the north west during its fi rst 18 months. Based on the fi rst 18 months of the scheme, Advancing Quality appeared to be a worthwhile and cost-eff ective intervention.
However, when we analysed results after 42 months, the picture was somewhat diff erent. Mortality for the incentivised conditions continued to fall, but the reduction in mortality was greater in the rest of England than the north west. So does that mean that the scheme was a failure? Well that’s one way of looking at it. But it could be more complex than that. After the initial success of Advancing Quality, two other regions in England implemented a variant of the scheme. It might be that the improved performance in the rest of England was due to this.
Spillover eff ects
In our CQUIN evaluation we found that quality indicators were often developed locally, changed annually and introduced in a hurried manner. One result of this was the large
Furthermore, when we looked at the non- incentivised conditions, we found that the reduction in mortality for these conditions was
signifi cantly greater in the north west than in the rest of England. Again, we published our results in the New England Journal of Medicine after rigorous peer review, which suggests that our analyses are correct.
So if it’s not fl awed research that is to blame, how can we account for our fi ndings? One interpretation is that there were positive spillovers from the Advancing Quality initiative. Changes to care delivery for conditions incentivised under Advancing Quality may have resulted in more general improvements in care which explains the superior performance in the north west in relation to these conditions. However, it may be that the observed improvements in mortality in the non-incentivised conditions within hospitals participating in Advancing Quality were unrelated to Advancing Quality.
During the study Advancing Quality payment rules changed following the introduction of CQUIN. But we don’t know whether, how and to what extent this impacted on mortality.
Challenging context
The changing context of the NHS presents challenges for researchers attempting to evaluate policy innovations. In our bid to funders we outlined our plan to combine quantitative data analysis with more in-depth investigation, talking to people and observing what they did during the fi rst phase. We would then negotiate with funders and decide on where to focus our resources for the subsequent phase. Having shown an impact in the initial evaluation phase, with an accompanying explanation of how changes were made, the funders were keen for us to focus our resources on measuring and quantifying impact in the follow-up phase of the study. This meant that although we used interviews and observation during the initial period to understand how and why things happened, we did much less of this in the second phase. The result is that we don’t know whether Advancing Quality was a roaring success or a failure. What we need now is further research to help us answer that question.
The paper
The paper, entitled ‘Long-Term Effect of Hos- pital Pay for Performance on Mortality in England’, was authored by Søren Rud Kris- tensen, Rachel Meacock, Alex James Turner, Ruth Boaden and Matt Sut- ton of the University of Manchester;Ruth McDon- ald of Warwick Business School; and Martin Ro- land of the University of Cambridge.
Ruth McDonald
TELL US WHAT YOU THINK
opinion@nationalhealthexecutive.com
national health executive Sep/Oct 14 | 19
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84 |
Page 85 |
Page 86 |
Page 87 |
Page 88 |
Page 89 |
Page 90 |
Page 91 |
Page 92 |
Page 93 |
Page 94 |
Page 95 |
Page 96 |
Page 97 |
Page 98 |
Page 99 |
Page 100 |
Page 101 |
Page 102 |
Page 103 |
Page 104