Analysis and news


A look at prediction markets Jacqueline Thompson, Marcus Munafò (pictured), and Ian Penton-Voak introduce a tool to help assess research quality


Assessing the quality of research is difficult. Jisc and the University of Bristol are partnering to develop a tool that may help institutions improve this process. To attract government funding for their


crucial research, UK universities are largely reliant on good ratings from the Research Excellent Framework (REF) – a process of expert review designed to assess the quality of research outputs. REF scores determine how much government funding will be allocated to their research projects. For instance, research that is world- leading in terms of originality, significance and rigour, will be scored higher than research that is only recognised nationally. Considerable time is spent by


universities trying to figure out which research outputs will be rated highest (4*) on quality and impact. The recognised ‘gold standard’ for this process is close reading by a few internal academics, but this is time-consuming, onerous, and subject to the relatively limited perspective of just a few people. It’s far better to include the insights of


more people – which is where prediction markets come in. This online, crowd- sourcing mechanism has been gathering steam in assessing academic research, and has, for example, been remarkably accurate at predicting which social science experiments will replicate, or how various chemistry departments would rank in the REF.


How prediction markets work Prediction markets capture the ‘wisdom of crowds’ by asking large numbers of people to bet on outcomes of future events – in this case how impactful a research project will be in the next REF assessment. It works a bit like the stock market, except that, instead of buying and selling shares in companies, participants buy and sell virtual shares online that will pay out if a particular event occurs – for instance, if a paper receives a 3* or above REF rating. Markets usually run over the course of


a few days or weeks, during which time participants can update their bets and


18 Research Information December 2019/January 2020


compete to earn points by buying low and selling high. After the market closes, the final output is a list of ‘market prices’ (one for each paper). A paper’s market price represents the group’s collective confidence that the paper will achieve a certain threshold of ratings.


Benefits over other assessment methods Prediction markets have several advantages over other assessment methods. Crucially, the fine-grained market prices allocated to various elements of the research assessed allow the papers to be ranked against each other. And, in comparison to most other assessment methods, such as surveys or close-reading panels, prediction


“Prediction markets have several advantages over other assessment methods”


markets have a built-in mechanism for weighting participants’ confidence in their own ratings. Namely, participants can choose to bet (or not bet) on whichever papers they like, plus they see real- time information on the group’s overall confidence in each paper, which they can use to inform their bets.


Our Jisc pilot project Over the past six months, Jisc ran three pilot markets at the University of Bristol, in the psychology, biology and chemistry departments. The pilots showed promising results: the outcomes (market prices) from all three correlated highly with the ratings that were given by the internal REF panel. The psychology market was also


compared against a machine learning algorithm trained on various metrics; the machine learning results correlated at similar levels with both the prediction market and the internal REF panel. Crucially, these levels of correlation suggest that all of these methods are


picking up relevant information, but the underlying information that each reflects is somewhat different. Judging by our discussions with the


REF co-ordinators from these Bristol University departments, we envision that the results of the prediction markets will not take the place of the traditional close- reading approach, but instead will be most useful as an extra source of information for uncertain or borderline cases. We also measured participants’


feedback on the experience of taking part in the markets. After all, these are busy academics, who are often deluged with requests to fill out surveys or help with assessment exercises. We were pleased to find that, overall, participants reported that they felt engaged with the process and found it enjoyable – one even reported playing the prediction market for fun, instead of checking football scores!


Future directions We are expanding our series of pilots beyond Bristol to explore how the prediction market tool works in various types of institutions and departments. We still have bandwidth to include more institutions in the pilots, so do contact us if your institutions may want to take part. Once these are complete, we plan to publish results from the full set of studies in a paper, with our collaborators at the University of Innsbruck and Stockholm School of Economics. In the next year we also aim to develop


a more specialised and optimally user- friendly interface for the prediction market tool through Jisc, based on feedback. Ultimately, we hope the tool may


be useful for other areas of research assessment outside the REF – after all, the REF isn’t the only context where research quality is difficult to assess! We’re betting that in all sorts of areas, the old adage may prove to be correct: two heads are better than one.


Jacqueline Thompson is a research associate at the University of Bristol; Marcus Munafo is a professor of biological psychology in Bristol’s School of Psychological Science; Ian Penton-Voak is a professor of evolutionary psychology at the University of Bristol.


@researchinfo | www.researchinformation.info


Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32