Feature


the reach, influence, mobilisation of knowledge and


there isn’t a simple formula to measure


“Unfortunately, impact in society’”


There have been huge changes in scholarly communication in recent decades, with significant implications for how it is measured.


Not only has the digital transformation


of communications enabled a rapid expansion in the amount of data that can be collected, but there have also been changes in the research itself. As Jonathan Adams, chief scientist at the Institute for Scientific Information (ISI), noted, there are now many more countries publishing in Anglophone journals, new journals, new fields and new subfields have emerged, and there’s more international collaboration and a shift towards the North American research culture, with shorter journal articles focused on one key element. This all has implications for measuring research. The big change, however, has been a widening in what people want to measure. There has been a move from research excellence in purely academic terms, towards research achievement in more societal and economic terms as well. As Tony Roche, CEO of Emerald Publishing, put it: ‘Funders and policy makers are putting more than academic influence on the table. They want to see more from research than great research. Great research is absolutely critical and fundamental, but researchers also need to demonstrate how their work moves beyond the academy.’


Beyond bibliometrics and citation analysis The desire to measure the impact of research beyond academia is perfectly


www.researchinformation.info | @researchinfo


understandable. governments and taxpayers want to know that money is being spent wisely, and in the best way possible. The problem, however, as Adams


explained, is there are no good indicators for measuring societal impact analogous to citations, and the timescale for impact is much longer: ‘There really is a paucity of appropriate indicators, and part of that is the nature of research projects themselves. Each project is individual in the way in which it delivers outcomes. What research does conveniently is it gets reported by one academic to other academics in the form of papers, and those papers if used get referenced and cited, and so that’s been a very convenient currency for decades. ‘Many policymakers have assumed a similar currency can be found in other areas of research activity, but we don’t have the history of evaluation through those other routes that have established a proper grounding for what the indicators might be. If you consider the pipeline between research and its broader outcomes, then it can be a very long one, and very variable. When we look at the initial research associated with the impact case studies reported in the 2014 REF, some of that research was 20 years old.’ Of course, that is not to say there


have not been useful and interesting metric developments, moving beyond traditional bibliometrics, it’s just they are a long way from providing the sorts of robust indicators that governments,


policymakers and institutions increasingly want for evaluative purposes. As Roche explained, there’s not a


one-size-fits-all solution in these desired metrics, but we need to find the right metrics for the particular research community: ‘Publishers are now reporting on speed of publication, usage of content, social media attention, and more recently a number of service providers and publishers are looking at policy implications and uptake of research. ‘All of these things are useful, and as a


sector we are supporting our customers, our authors and our researchers with more tools and more services. The key point is we need to work very closely with the research community itself. It’s all about creating the right resources for the right domains, because unlike the impact factor, which is this generic one size fits all, a lot of these research metrics need to be honed in their appropriate domain. ‘Unfortunately, there isn’t a simple


formula to measure the reach, influence, mobilisation of knowledge and impact in society. The journey to impact is far more heterogeneous than something measured by an impact factor calculation. ‘We’re all looking for the same sort of


solution we got with bibliometrics, and I don’t think it exists. We need a far more nuanced and sophisticated approach using both quantitative and qualitative approaches. A combination of the classic quantitative measures, combined with storytelling narratives around how research makes that journey to broader society. That can’t be done in a number.’


Summer 2022 Research Information


g 5


Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32  |  Page 33  |  Page 34