What is science without replication?

Dr. Mina Bissell published a commentary in Nature in November (Reproducibility Risks) which was in response to questions raised by a study by Amgen scientists where they could only reproduce 11% of work published in high impact journals. She took what I view as a very peculiar tack on these observations. Her major concern was that the drive to improve the reproducibility of research findings could serve as a stumbling block to  promising research and researchers. She believed her concerns were justified based upon the time required and the technical difficulties associated with reproducing studies:

"People trying to repeat others' research often do not have the time, funding or resources to gain the same expertise with the experimental protocol as the original authors, who were perhaps operating under a multi-year federal grant and aiming for a high-profile publication. If a researcher spends six months, say, trying to replicate such work and reports that it is irreproducible, that can deter other scientists from pursuing a promising line of research, jeopardize the original scientists' chances of obtaining funding to continue it themselves, and potentially damage their reputations."

"Many scientists use epithelial cell lines that are exquisitely sensitive. The slightest shift in their microenvironment can alter the results — something a newcomer might not spot. It is common for even a seasoned scientist to struggle with cell lines and culture conditions, and unknowingly introduce changes that will make it seem that a study cannot be reproduced."

My question is how biologically relevant are observations if they are only reproducible under very narrow and difficult to replicate (impossible to replicate?) circumstances? Shouldn't one's reputation be somewhat dependent upon doing and reporting scientific studies which others can replicate? Substitute the term magic for science and wizard for scientist. Magic and alchemy became discredited because it was not reproducible and it was dependent upon special proprietary talents held as secrets by the purported practitioners. Where have we gone if our models begin to resemble magic tricks or at best difficult to duplicate scientific parlor tricks?

What brought this to light was the biotech company Amgen invested a huge sum of money and the product of their investment was not what they expected (Begley). The rest of us in the biomedical establishment should understand what they uncovered. Prior to their work, we had no way of measuring the quality of biomedical research work in terms of a metric which is central to research in general: reproducibility. One must assume that when they undertook this endeavor, they were not intentionally testing the reproducibility hypothesis. I can only imagine this was believed to be a given since it is so central to research activities.

To be faced with a reproducibility percentage which almost reached into single digits must have been staggering to the Amgen scientists. How can this be?  The assumption that scientific work is self correcting does not appear to be valid.  Thus, Dr. Bissell's response to this is to find all sorts of reasons that under the current circumstances with complex systems that reproducible findings are hard to generate. She misses the point. If we have arrived at a place where only one in ten studies published in the most reputable journals can be reproduced, we have an awful quality control problem in science. It is likely worse than these numbers reveal since these results come from the most reputable journals.

Given the extent of the problem highlighted by the Amgen work, one must also wonder how much of this problem may be due to scientific fraud. I came across a report of an investigation in Germany published in 2000 (Hermann and Brach) regarding two cancer scientists who operated in Germany in the 1980's and 1990's. The commission that investigated them and found rampant issues. However, little structural action has been taken to address the underlying problems. It is sad to admit but humans functioning in science will be representative of people in general. If money, fame, power are involved, some will stretch the rules and in the absence of mechanisms to identify undesirable activities, the culture will devolve into whatever it takes to be successful in the increasingly competitive funding climate.  The Wikipedia page on scientific misconduct continues to expand (WIKI).

This is simply unacceptable. The fact that any attempt to address these observations will be disruptive of current operations should be taken as a given, not as a barrier. How valuable is this multi-billion dollar enterprise if the models only work is a few hands and there are no consistent quality control tools? The arguments placed are highly reminiscent of the points put forth when the safety and quality movement came on the scene in health care delivery. Prior to the late 1990's it was difficult to track quality in health care delivery. Thus, consistent with the mantra that if you can't measure it, you cannot improve it was operational. Since there are few incentives to measure reproducibility in biomedical sciences, few focus on this. What the field values in general is productivity, whether valid and reproducible observations are made or not is of secondary value. I fear that the absence of  reproducibility feedback loops has taken us to a very undesirable place from the stand point of those investing financial resources in the research.

Dr. Bissell offers a few anecdotes from her lab to refute the concerns. When sophisticated scientists in industry spending hundreds of millions of dollars cannot duplicate your work, they might have a problem in that they burned through resources that could have been better invested elsewhere. However, if they cannot reproduce your results you have a bigger problem.  The fact that under the current system that this issue can be viewed as someone else's problem is a HUGE PROBLEM. One cannot and should not deflect the onus on someone else. Magicians and wizards can resort to secret spells and incantations. That is not science.