Two steps toward a culture of reproducibility in science

  1. 1.  iResearchNetwork

Abstract

The following is a short, possibly humorous essay providing two concrete ideas to increase reproducibility in science research. The two ideas are united under the notion of a “culture of reproducibility”. The first idea is hiring different kinds of professionals in the research ecosystem whose job is to ensure reproducibility and impact. The second idea is to require reproducibility risk-management plans in funding applications. Together, these actions are an investment into the infrastructure of research for increased impact and accountability, along with a stronger conception of reproducibility. It will also solve the world energy crisis for under half a million USD by June. No citation software was harmed in the production of this essay.

Introduction

“Can we do it again?”

“I don’t know. Do you have enough money?”

Such is a typical conversation between two scientists concerning how they could replicate the ground-breaking experiment they just cleverly devised, likely improvised, statistically analyzed, curated, realized, summarized, related, criticized, revised, digitized (CC-BY), closed their eyes, and surprise: PUBLISHED! In next month’s issue. With corrections. It was totally the <citation software>’s fault.

And I’ve done the math: Replicating a worthy scientific study should cost at least as much as it did the first time: stipends and salaries of people involved, extra costs of which can be reduced to almost zero if you include “mentoring opportunities for graduates” a.k.a. “summer co-ops for undergraduates” a.k.a. “volunteer hours for med school”; costs of materials and instrument maintenance—especially after having put the instruments through hell the first time a hundred times; and inflation.

But who wants to write a grant proposal that says:

            Our current budget allots $100,000 for the initial phase of experiments, based on the a) aforementioned blah blah and ( $100,000 + inflation ) X 3 to do it b) again and c) again and d) again. If it works, we will be one step closer to creating tangible “societal impact” with our research, and you’ll gladly give us more funding for whatever Tolkien narrative we can drum up for the next phase. IF it doesn’t work, and it will work, then ‘you still learn something’ right? References attached.

No, you want this, beautifully formatted, each point expanded:

We realistically can solve the world energy crisis by June. We calculated how much of a stretch this is and it is not much of a stretch. We will start tomorrow. We’ve solved the energy crisis before and we’ll do it again. We need around $400,000 for now. References attached.

Pails of money chucked onto your lap and all society swooning in the tangibility of your impact.

If it is left to money, no reproducibility will be the reality until enough money appears in pockets and business gains are on the horizon. And since “the research is not in yet”, neither is the money for “making sure”.

 

Research impact is tied to reproducibility: create a culture

Speaking of money, last week the Canadian “Advisory Panel for the Review of Federal Support for Fundamental Science” published their mandate. Although I’ve never been on a mandate, it was a pretty good read. A culture is brewing that is different from brewing culture: it is a culture of responsible funding. However, this requires measuring impact! Now we see that responsible funding of any sort will need to be tied more and more to metrics and tangible impacts!

Professors are already too busy to do it on their own, while funding processes grow increasingly more complex, legislating to favour “concrete societal impacts” over fundamental research, and, to top it all off, those who will be the benefactors of the impacts of research are more difficult to locate and communicate effectively with. Thankfully, there exist professionals whose job is to ensure that research is converted to impact: right form, right people/communities, right time. These people are the knowledge mobilization (KMb) people, including knowledge translation (KT) & implementation science (in health sciences), knowledge brokers and K*.

“Reproducibility!?” you cry. If research is not reproducible, it is too risky to invest in. Research that will have put in all the effort and dollars to be reproducible will need to be assessed more harshly for its potential impact in communities and businesses by KMb professionals whose job is impact. These people are a budding part of the reproducible research ecosystem and culture but are sorely underutilized at present. Hire them!

 

(Ir)reproducibility Risk-Management Plans: Not all research is equally (ir)reproducible

Why should my fMRI patterns on Monday after tikka masala ever match up well to the one on Tuesday? Why should what a small homogeneous group of people think be the same as another remote, homogeneous group of people? Some things are just not reproducible and some things are more or less reproducible, within defined boundaries. Neither does reproducibility always come down to the skill of the scientist in controlling variables, since a scientist only controls foreseen variables and rarely interprets variable coupling accurately on first glance—especially in extremely complex systems, which is almost every system. Another way of saying “extremely complex system” could be “non-equilibrium system with possible local equilibria which contain innumerable diss\nipative structures”. And while we learn everything in books as static and at equilibrium, almost everything we experience in the world is time-dependent and at non-equilibrium. Fluctuations in this kind of system give rise to anomalies or even new stable states and this contributes heavily to irreproducibility, not to mention fantastic research!

Therefore, I suggest an (ir)reproducibility risk-management plan in applying for research grants. This boils down to a kind of risk management plan in the event of irreproducibility. It would begin with a general feel of reproducibility: “How reproducible is your field of study, generally?” (Here you could compare group psychology to particle physics) and progress to “How reproducible do you expect your particular studies to be in that context?” giving an outline of factors expected to affect the reproducibility of the major experiments (instrumental, expertise of personnel, other physical factors). Could you expect anomalous fluctuations to create new stable states or cause outlying data frequently? Next would be plans to compare results within the lab and between labs, design of experiments for finding optimal sets of parameters and also strategies for redefining and finding multiple and multi-component optima, even on-the-fly. While robustness testing is often done down the line, providing a more comprehensive (ir)reproducibility risk-management plan, right from the start, will be a big step toward a research culture of reproducibility, impact and transparency.

 

Footnote

It will not be a clean technological fix that will boost scientific research reproducibility, but the adoption of a kind of “reproducibility culture” where everyone recognizes that reproducibility works in the best interests of society and science. New people need to be hired in new positions and new requirements need to be asked. A new ecosystem needs to exist in the research funding and publishing organism. It is not a cost, but an investment in a kind of structural update. The societies we live in are evolving, our relationships are evolving and our needs are evolving. Research itself is always open to the future, but the structures in place to disseminate, fund and mobilize research serve the dynamic public and it’s up to us scientists to guide them in their facilitatory roles.

 

Onward!

Look, I don’t know about you, but I love reading research that has never been replicated. It makes me feel in my guts that I am on the pulse! Like I have a pulse in my gut! Maybe I just ate too much gluten…

I believe it will be replicated and that makes me excited. When it is not replicated, it just means that it should never be used in court, or taught in schools, or spoken about in Middle Earth. I mean, there’s still a lot of latitude: You can practice text mining...

All joking aside, I don’t know the surefire way of ensuring scientific reproducibility, but we have to ensure that science is reproducible proper to its expected reproducibility! I decided to propose a tough route: Build a culture of reproducibility. Hire professionals who exist to ensure impact is forged by trusting scientific results; have grants ask for a risk management plan in the event of experiment irreproducibility. Who is making sure that science is being replicated? Weren’t scientists (like me) supposed to make sure things were being replicated before we published? I know: times are tough. Well, if I have to hire a gaggle of scientists to do the deed then I will. I just have a few bucks in the bank, but I’m waiting for a possible $500 payout.

 

Appendix

License

This article and its reviews are distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and redistribution in any medium, provided that the original author and source are credited.