NEWS AND ANALYSIS
PEER-REVIEW DEBATE SHOULD INCLUDE SOFTWARE
Richard Padley argues the case for scrutinising software submitted with scholarly articles within the peer-review process
I
t’s not often that coding errors make the news. But in April one particular slip-up with formulae on an Excel spreadsheet caused worldwide repercussions. It emerged that Harvard economists Carmen Reinhart and Kenneth Rogoff had made mistakes in the data underlying their influential 2010 paper, Growth in a Time of Debt that appeared to undermine the paper’s main contention: that countries with debt-to-GDP ratios above 90 per cent see markedly slower growth. The data was not published with the paper, and only showed up when PhD student Thomas Herndon requested the spreadsheets directly from the authors. After a thorough debug, Herndon rubbished their results as lead author of his own paper, Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff.
At a time when scholarly publishing is debating the issue of data being published alongside papers, this makes an interesting test case. Reinhart and Rogoff’s errors could not have been detected by reading the journal article alone, so proper scrutiny in this case ought to have included the dataset. But I would argue that the terms of the debate should go beyond data: we ought also to be thinking about software. Reproducibility is one
of the main principles of the scientific method. Initially, Herndon and his Amherst colleagues found that they were unable to replicate Reinhart and Rogoff’s results. This was what caused them to request the underlying data, resulting in their subsequent discovery of errors. Three areas gave concern, but the most highly-publicised flaw was an Excel coding error.
What was flawed was not the base data they drew on, but their use of the software tool Excel to analyse that data. Without the use of
What was flawed was not the base data, but their use of Excel to analyse that data
the same software that Reinhart and Rogoff had employed in reaching their conclusions, Herndon, Ash and Pollin would likely never have got to the bottom of why those results were not reproducible using the same dataset. It
Friedel Grant is communications officer at LIBER
becomes a central artifact in the presentation of scholarly results.
follows that not only the data ought to be open to the scrutiny of peer review in an instance like this, but also the software. Since everything that involves data nowadays involves software to some degree, the software
Excel is a universally familiar piece of software but other more specialised tools such as MATLAB are routinely used within the scientific community and by researchers in disciplines as diverse as engineering, economics and social science to perform operations on data that result in published science. MATLAB goes beyond Excel in that its output might not be just a set of numbers, but an algorithm. If you want to look under the hood of that algorithm, for the purposes of peer-review scrutiny or reproducibility, you might well need to access the software that produced it. And you might also want to see the algorithm in action. There are many ways to represent algorithms – including as formulae, flowcharts and in natural language. However, arguably the best way is by using the programming languages that were written specifically for the purpose and, of course, programming languages were created not just to represent algorithms, but to actualise them. It is logical therefore that MATLAB produces not only algorithms but also executables – i.e. software. Clearly, not every item of published research needs to include a piece of software. But if we restrict our vision of scholarly publishing to just articles and data we risk ignoring the other digital bits and pieces that now rightfully belong in the scholarly record – and without which it cannot properly be understood and scrutinised.
Richard Padley is managing director of Semantico. He thanks to Dave De Roure for his paper raising these issues in a panel discussion chaired by Padley at the APE 2013 conference
KEEPING RESEARCH IN STEP WITH POLICY
Many recent policies favour openness, and studies show that researchers as readers value this too. However, as authors, researcher priorities are different. Rachel Bruce and David Prosser consider this gap and what can be done to address it
I
n June, under the UK’s presidency of the G8, ministers and research academies from the G8 nations met at the Royal Society, London and jointly endorsed the need for open data and open access (OA) to research. This was in recognition that this will have a positive impact on global economies and challenges. This is one of many recent policies that demonstrates the tide has turned firmly towards OA research. Publishing research openly and
8 Research Information AUGUST/SEPTEMBER 2013
engaging in online sharing is what the current policy environment demands of academics. This makes sense in terms of value for money, impact of research and democratisation of knowledge. There is actually now a range of dissemination and delivery options available and it’s not only articles and monographs that can be shared openly but also datasets and software. Recently, a joint Jisc and Research Libraries UK (RLUK) survey of 3,500 UK academics,
conducted by Ithaka S+R, uncovered rather conservative behaviour in terms of the sharing of research in an OA way. When asked to rate a number of factors that influence their publication decisions, fewer academics rated free access on the web as important in comparison to other factors and a fairly high number rated the ability to publish for free, without article charges, as an influential factor in their publishing choices.
@researchinfo
www.researchinformation.info
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32