Topic: Errors in Science
Comments by Lorenzo Federico
MD Anderson Cancer Center
6/9/2015
Proposed Actions
The Reproducibility Project (http://validation.scienceexchange.com/#/) is an incredibly important initiative that aims at reproducing results of several high impact papers in cancer biology and other research areas. Recently, Sigma-Aldrich made a donation of $50K in products to support this challenging endeavor and there is no reason to believe that funders cannot act along these lines. A more stringent control system over research quality and reproducibility should be introduced. Ideally, scientists should be principally rewarded according to the reproducibility of their research. In theory, published research can be scrutinized and reassessed by trained scientists/inspectors in collaboration with principal investigators and postdocs that made the study. Each year a number of high profile papers can be selected for such inspection. Experiments should be performed and original da ta reexamined under funders supervision in the PI laboratory. Under such scenario, researchers must be able to provide acceptable explanations if data cannot be reproduced. If a satisfactorily explanation cannot be provided, the work in question should be flagged with a RED tag which can be posted alongside the paper in the NCBI’s (or any other) public database. Under this system, a GREEN tag would indicate reproducible work, whereas irreproducible data for which researchers were able to provide an acceptable justification should be flagged in YELLOW and corresponding paper annotated/updated accordingly. Clearly not every paper in literature can be scrutinized, but a careful choice of testable experiments of key publications is unquestionably feasible. I believe that this system could work as it is implementable and relatively cheap, when considering how big the burden of irreproducible research on US taxpayers could be. Additionally the psychological pressure associated wi th the prospect of a potential inspection should naturally raise the level of transparency, honesty, and data accuracy in any funded laboratory. The academic enterprise is the key diver of discovery and must be urgently fixed. The time is now.
Comments on the Problem
Reproducibility is the most important aspect of science, much more important than scientific “impact”, because it is the basis from which the entire scientific enterprice can be efficiently built on. In contrast, impact is extremely difficult to predict, because science advances one step at a time, slowly, only occasionally experiencing a big jolt that shakes scientific understanding and accelerates progress. But the jolt is just like an earthquake: it comes randomly; it can’t be predicted. Enrico Fermi used to say that when we perform an experiment that confirms a previous finding we have made a measurement; otherwise we have made a discovery. This concept simply cannot be applied to today’s biomedical research, because we have no way to timely know whether a paper is reproducible or not. High profile retractions and sloppy mistakes in papers are undermining the trust in s cience, slowing down discovery, and, even worse, keeping patients suffering and waiting for a cure. The standard for quality assessment of research output remains poor, in fact, nearly absent. It is imperative that funders begin to make their best efforts to ensure that the scientific output of laboratories is reliable and reproducible. This is not an easy task, but the implementation of policies aiming at the improvement of research reproducibility must be considered by NIH and other funders an absolute priority. I believe that only a different incentive system along with a more stringent control over research output can revolutionizethe present and the future of biomedical science. The real value of what we do as scientists is to help society and improve the life of people. New policies must be introduced to raise the level of transparency and honesty in science. This is, in my opinion, not only the best way to ensure a faster scientific progress but it is, above all, a mo ral duty.