Editor's note: This article was originally published by The Hill on December 2, 2018.
America is suffering from a crisis of irreproducible science. In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology, but could only replicate six. That same year Janet Woodcock, director of the Center for Drug Evaluation and Research at the FDA, “estimated that as much as 75 percent of published biomarker associations are not replicable.”
The federal government bears some blame. According to a 2015 study, government funds two-thirds of preclinical research in America and half of that research is irreproducible. Of the $28 billion our country wastes each year in irreproducible preclinical research, the government share is $19 billion.
The federal government makes policy based on research that can’t reproduce. The EPA used the National Ambient Air Quality Standard (NAAQS) for fine particulate matter (pm25) to justify many costly regulations, such as the Clean Power Plan. When Dr. James Enstrom acquired the original data and conducted an independent re-analysis, he “found that there was no significant relationship between pm25 and death from all causes.”
Earlier this year the National Association of Scholars (NAS) published The Irreproducibility Crisis in Modern Science, which analyzes the effects of the improper use of statistics, arbitrary research techniques and political groupthink. Together these have produced a reproducibility crisis that afflicts a wide range of scientific disciplines.
The irreproducibility crisis spreads across a host of departments and agencies. They make regulations based on scientific research, trusting in peer review, but not requiring that the research be reproducible. Peer review has proven to be a weak protection against bad science. The recent study proclaiming — falsely — that the oceans are heating 60 percent faster than expected passed multiple layers of peer review. Just days later it was exposed as having a fundamental statistical error and the headline claim was withdrawn.
Political advocates often prefer to rely on irreproducible research, since that makes it easier to come up with a “scientific” justification for the regulations they want to impose.
Some parts of the government are now trying to restore reproducibility to science. The National Institutes of Health directs grant applicants to make reproducibility a top priority. The Environmental Protection Agency has proposed a rule to strengthen transparency in its own use of scientific research.
But there are dozens of federal agencies that are moving too slowly. The federal government needs to make a global change to its reproducibility requirements.
Congress can do something now — in the weeks before the new Democratic majority takes over the House. Democratic members of Congress have so far opposed all reproducibility reforms. But one simple change by the current reform-minded majority will strengthen the federal government’s reproducibility standards.
Congress should first state that “best available science” must be “publicly accessible research,” defined as research whose registered report (including protocols), research data, associated protocols, computer codes, data analysis scripts, recorded factual materials, statistical analyses and algorithms are archived on an online digital repository in a manner sufficient for continuing independent inspection, replication, reproduction and verification.
Congress should then require that all grant programs that fund “best available science” must fund only “publicly accessible research.”
Congress should also require that any significant regulatory action (SRA) that depends on “best available science” must also depend on “publicly accessible research.”
A congressional requirement that all federal agencies that use “best available science” rely on “publicly accessible research” will have an immediate effect throughout the tangle of federal government—but it will only affect those agencies that already rely on “best available science.” This change will be both simple and tailored.
The federal government will need to make far greater changes to solve the irreproducibility crisis. Each agency needs to take responsibility for instituting specific measures to improve reproducibility standards in its own grants and regulatory processes. Some agencies, for example, might need to require that SRAs rely on science that isn’t just publicly accessible, but which has actually been reproduced. It isn’t unreasonable to ask that a major piece of regulation depend upon two independent works of scientific research, not just one.
But Congress can make a great step forward now. If it does, it will champion America’s longstanding commitment to constantly improve its scientific standards. It will also act as a responsible steward of American taxpayers’ money and a prudent governor of our regulatory state.
David Randall is director of research at the National Association of Scholars.
