http://blogs.scientificamerican.com/information-culture/2013/10/29/you-cant-read-just-one-reproducibility-and-multiple-sources/
There are lots of ways to mess with the heads of undergraduate students. Giving them a research assignment and failing to specify a minimum number of references needed is just one example.
“Include as many sources as you need to make your point and illustrate your thesis.”
For students, finding one scholarly article on their topic often
seems to be enough. Researchers did an experiment, got some results,
and answered the research question the student started with. All done,
all set, time for dinner.
But science doesn’t work that way. One experiment may suggest
something interesting, but it doesn’t prove anything. In fact, it is
quite easy to point to many examples of intriguing scientific studies
that were either proved false or that couldn’t be reproduced later on.
Scientific ideas that are true should be reproducible: other researchers
should be able to repeat the experiments and get similar results or use
other methods to arrive at the same conclusions. You can’t say that
you discovered something new if someone else can’t reproduce your
result.
This fundamental scientific idea, reproducibility, may be in crisis.
A recent article by Vasilevsky et al. in the journal PeerJ suggested
that many scientific journal articles don’t provide the information that
other scientists would need in order to replicate their results. Key
information about chemicals, reactants or model organisms is often
missing, despite journal requirements to include such information
(Vasilevsky et al., 2013). And a recent item in
The Economist
suggests that this might not matter that much. The emphasis placed on
new research (by funding agencies and tenure and promotion committees)
means that few scientists even attempt to replicate the work of others
(“Unreliable research: Trouble at the lab,” 2013).
All of this means trouble from the very beginning of a research
project, before an experiment is even designed, when scientists start to
do background research on their topics. In the same way that
experimental scientists can’t rely on the results of just one experiment
to prove something, relying on just one information source for
knowledge is a sure way to end up with unreliable information.
Journalists look for corroborating sources, wikipedia flags articles
that need a wider variety of citations, and scholars need to find
multiple scholarly articles to support their ideas.
Some innovative people, companies and publishers are trying to sort
this mess out. A collaboration between PLOS ONE, Mendeley, Figshare and
the Science Exchange will be attempting to replicate the results of
selected projects as a part of the
Reproducibility Initiative. The
Reproducibility Project is a crowdsourced effort to evaluate the reproducibility of experimental results in psychology. And the
Reproducible Science
project aims to make the results of computational experiments
reproducible by ensuring the sharing of code and data and by making that
information available to reviewers who can test the results described
in a manuscript they are reviewing.
Unfortunately, these innovative programs are just a drop in the
bucket of modern science. Funding agencies, publishers and tenure and
promotion committees still value original work more highly than
verification work. Scientists who concentrated on replicating the work
of others would risk their careers.
As a result it is important for students and scholars to be aware of
the challenges facing the reproducibility of science. We teach students
in introductory science classes that reproducibility is one of the
hallmarks of science. As they learn more about their disciplines, they
need to be aware of the practical challenges involved in reproducing the
work of others, and the importance of finding multiple sources about a
topic needs to be emphasized.
As a librarian, part of my job is to help students find additional
sources related to their research topics, even if there isn’t a
published reproduction of an original source. This isn’t about which
database to use or whether to put quotes around a phrase. It is about
getting them to think critically about their topics. For example, while
there might not be a second study that repeated the experiment of the
first, students can look for:
- Studies that examined the same topic in a different way
- Studies that used the same methodology on a different species, geographic area, etc.
- Background studies on individual aspects of their research question, including the statistical analyses used
- Studies that cite the original study (even if no one has tried to
reproduce the results, other scholars might express doubts about their
conclusions when they cite the original).
The issues surrounding reproducibility in science won’t be solved
overnight, and it will take a concerted effort from scientists at all
levels of the modern scientific enterprise to steer this very big ship.
In the meantime, students and scholars can make special efforts to
ensure that they are using the highest quality information available as
the basis of their original studies.
Works Cited:
“
Unreliable research: Trouble at the lab.” (2013, October 19).
The Economist,
409(8858), 26-30.
Vasilevsky, N. a, Brush, M. H., Paddock, H., Ponting, L., Tripathy,
S. J., Larocca, G. M., & Haendel, M. A. (2013). On the
reproducibility of science: unique identification of research resources
in the biomedical literature.
PeerJ,
1, e148. doi:
10.7717/peerj.148.
