Analysis and news: Metrics
Measuring the impact of OA content Knowing how much effect agreements have is critical for developing sustainable OA business models, writes Tim Lloyd
For an increasing number of funders, an important ingredient of impact is the audience. Yet existing analytics don’t do a good job of understanding how communities are engaging with OA content. With it accounting for the majority of journal articles published1, we need a better understanding of which stakeholders rely on OA analytics, and how they want to use them. To this end, LibLynx and PLOS partnered
last year to develop the next generation of OA analytics. The first step was to engage with PLOS stakeholders – institutions, funders, and consortia – to see how they wanted to assess OA publishing’s value.
The limitations of traditional approaches Traditional quantitative metrics, such as citation analysis or the number of downloads, provide easily comparable numbers but little depth – was the usage from the communities targeted, or a bunch of bots and pirates vacuuming up free content? Altmetrics add a valuable sense of the attention that articles receive, but underplay value for communities that don’t engage with media in the same way. Counter reports seem an obvious
answer, and one libraries are already very familiar with, but they are engineered for a very specific use case – helping librarians understand and compare usage of paid content to make informed decisions on acquiring content to meet learning and research goals. This traditional Counter use case rightly focuses on the aggregate numbers, as the audience is already defined as the subscribing institution. But there is no standard for how to
attribute usage more broadly, or offer alternative methods of analysing that audience. This does not mean Counter has no role. Far from it, because ‘metrics are the gold nugget at the heart of Counter’2. Think of Counter metrics as a defined set of lego bricks we can re-combine to create new analytics reporting.
Who needs OA analytics? Back to our partnership with PLOS. Our research in recent months has identified a
10 Research Information June/July 2021
wide range of industry stakeholders: • Research institutions that are typically generating published research. Most obviously, we’ve got the traditional library role, which sits at the core of the Counter reporting use case. Librarians are already tracking usage of OA content as part of licensed collections, and those that also play a role in pure OA publishing will be equally interested in monitoring that usage. There are also institutional roles that sit outside the library and are focused on research management, such as the senior research officer3. These roles are more interested in understanding how usage of OA content ties into institutional research priorities.
• Publishers that publish OA content. These can be the same research institutions or dedicated publishing organisations like PLOS. There are
“It’s clear that the value of consistent, credible, comparable underlying metrics is still important”
development roles that need to understand which organisations are getting value from OA content to identify potential future sources of funding, as well as editorial roles that want to understand the subjects and topics that are engaging the community.
• Authors of that research. Authors want to understand the impact of the research they publish. This information can influence their choice of publisher.
• Funders that pay for the research to be published. These can be institutional budgets or separate entities, such as Wellcome or the Gates Foundation. Funders also want to understand the impact of the research they funded. Did it reach the communities they were targeting, or perhaps it also got engagement from new communities that they previously weren’t aware of? Various intermediaries perform a variety
of functions that support the publishing workflow, such as service providers like KU, consortia like Jisc, and distributors like JStor. Last, but not least, the broader community that is interested in reading that research – a group often overlooked, but of key importance to funders vested in delivering benefits outside of narrow research interests, such as publicly- funded institutions.
And what metrics are they interested in? While we’re still digesting the results of our research, some clear themes emerge. Counter metrics are highly valued. While the nature of Counter reports will develop over time to incorporate OA content, it’s clear that the value of consistent, credible, and comparable underlying metrics is as important as ever. Institutions want to understand the value they get from their publishing relationships, regardless of whether the content is subscription or one of the increasing flavours of OA, and across both the content they pay to publish AND the content they consume to support learning and research. Understanding your research audience
is essential. While data privacy rightly ensures individual anonymity, it’s still possible to quantify usage by geography and (where IP address matching allows) by organisational name and category. This can identify new, valuable communities engaging with research that have previously been ignored. We need to support more diverse use
cases. OA stakeholders want far more granular detail, in addition to traditional aggregated reporting; visually-rich layouts that are easy to consume in addition to tabular formats, and real-time reporting plus periodic reports.
Tim Lloyd is founder and CEO of LibLynx
References 1
2 3
According to
Lens.org data 51 per cent of journal articles published in 2020 are available through OA.
Jeremy Morse, Director of Publishing Technology, Michigan Publishing.
Read Roger Schonfeld’s fascinating December Scholarly Kitchen post to better understand this developing role.
@researchinfo |
www.researchinformation.info
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38
