Sparks of Hope for a More Open Approach to Scientific Research and Publishing
Transparent peer review, replication studies, and journals of negative results all suggest change is on the horizon
As I was writing this editorial, I learned about yet another scientific paper being retracted. This time it was a genetics paper in Science, one of the hundreds of retractions that the blog Retraction Watch has covered in the last year, and it served as another reminder of the flaws in our scientific process.
Errors are to be expected: To some extent, this is the nature of science. Scientific findings represent the truth, but with a caveat. It is the truth as captured in that moment of time with the available knowledge and technologies. New developments change our understanding. Think of how DNA testing has provided new insights into criminal investigations, and in some cases, altered the fate of those who had already been convicted. New information can lead to new conclusions.
These are not the types of errors that concern me. What worries me are the types of errors that are reported in the “Error!” article in this issue of Biomedical Computation Review and that are responsible for some recent retractions—the flawed data analysis or worse, the falsely manipulated data; incorrect assumptions that lead to wrong conclusions. How did these studies get through our peer-review process? And why has it taken so long to correct some of these mistakes?
The outlook might appear bleak, especially according to the recent popular press—the New York Times ran the article “It’s Science, but Not Necessarily Right” in June of this year, and the Wall Street Journal followed that up in August with the article “Mistakes in Scientific Studies Surge.” There is clearly room—lots of room—for improvement. But I also see sparks of hope as the scientific research and publication process tries to correct itself.
Take the journal Biology Direct, for example, with its novel peer review system. Authors choose their reviewers from the journal’s editorial board, and reviewers can optionally prepare comments that are publicly available on-line. The idea of an open, or even a partially open, review process is attractive. Reviewers will likely feel greater responsibility with this type of process, and the reviews themselves provide readers with additional insight into the work. Think of how your view of a study would differ if you knew the reviewers had concerns about the statistical analysis that had not been addressed. And the critiques might even change the way you approach other studies you read.
There also appears to be a growing awareness that replication studies and negative results may be valuable in the scientific process. Though science prides itself on discovery—think of how many grant proposals and papers include the words “new,” “novel,” or “groundbreaking” in them—the research path consists of more failures than successes. So knowing what not to do is incredibly useful, especially when there are no clear directions on what the right next step is. And the infrastructure to motivate and promote this different kind of research is beginning to develop.
The “Facilities of Research Excellence in Spinal Cord Injury” (FORE-SCI) program funded by the National Institute of Neurological Disorders and Stroke (NINDS) is a noteworthy example. This group within the National Institutes of Health concluded that replication studies were needed in order to turn research findings based on in vitro or animal experiments into clinical trials. And, very importantly, they decided to provide the funding for replication studies, a real paradigm shift for an organization that emphasizes novel research.
There are also a few journals dedicated to publishing negative results: the Journal of Negative Results in Biomedicine, the Journal of Negative Results—Ecology and Evolutionary Biology, the Journal of Articles in Support of the Null Hypothesis, All Results Journals. And there are some mainstream journals that are open to publishing replication studies. The FORE-SCI-sponsored studies, for instance, are being published as part of a special issue of the journal Experimental Neurology and provide unique insights into the issues and usefulness of replication studies.
Efforts like Biology Direct and FORE-SCI show us what’s possible, but we have a long way to go. The community needs to support these efforts to ensure that these sparks of hope are fanned into a new reality for scientific research and publishing.