Mice, P-hacking, and More Irreproducible Science

Peter Wood

Further evidence continues to accumulate about the extent of the irreproducibility crisis—the combination of flawed statistics, arbitrary research techniques, and groupthink that afflict modern scientific research. A recent article at Nature Communications, “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons,” appears to be a poster-child for the irreproducibility crisis, for it states in the discussion of its procedures that “we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable—you are supposed to determine your sample size first, not just stop collecting data at the moment that you happen to get a statistically significant result. Is this what happened? Nature claims to be committed to “substantive steps” to address the irreproducibility crisis, and we trust that the commitment also applies to Nature Communications. We have sent an inquiry to Nature Communications—our letter is at the bottom of this post.

We found out about this article from People for the Ethical Treatment of Animals (PETA), whose primary concern is that the article required painful, fatal animal research. The NAS’s fundamental commitment is rather to freedom of (scientific) inquiry. Yet we don’t relish the thought of p-hacking compounded by hacking apart mice. If animals are going to die for science, we’d rather it was for science done right.

It is possible that the articles’ authors simply explained badly what was actually a properly done scientific experiment. But we believe Nature Communications ought to clarify the facts at issue.

***

Dear Nature Communications,

Nature recently published “Scientific Rigour and Reproducibility” (https://www.nature.com/collections/byblhcfwhw), which stated that “There is growing alarm about results that cannot be reproduced.  Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions, and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community.” Yet you recently published “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons” by Yang Dan, et al. (Nature Communications 9, 354, https://www.nature.com/articles/s41467-017-02765-w) , which contain this statement: “For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable. Is this what happened? I urge Nature Communications to re-examine this article to see whether it meets Nature’s standards—and to take appropriate measures if it does not. I also urge Nature Communications to provide a public explanation to its readership as to how it came to publish an article that seems openly to avow p-hacking.

 

Respectfully yours,

Peter W. Wood

President

National Association of Scholars

 


Image: Spark // Creative Commons

  • Share

Most Commented

October 29, 2024

1.

The Looming Irrelevance of Middle East Study Centers

Today’s Middle Eastern Studies Centers are facing a crisis due to the winds of change in the Middle East and their own ideological echo chamber....

November 19, 2024

2.

Lee Zeldin Should Reform EPA Science Policy

NAS welcomes the nomination of Congressmen Lee Zeldin to lead the Environmental Protection Agency....

November 20, 2024

3.

NAS Welcomes Administrator McMahon's Nomination to Serve as Education Secretary

With McMahon, the new administration has a chance to drastically slim down and depoliticize the Education Department....

Most Read

May 15, 2015

1.

Where Did We Get the Idea That Only White People Can Be Racist?

A look at the double standard that has arisen regarding racism, illustrated recently by the reaction to a black professor's biased comments on Twitter....

October 12, 2010

2.

Ask a Scholar: What is the True Definition of Latino?

What does it mean to be Latino? Are only Latin American people Latino, or does the term apply to anyone whose language derived from Latin?...

May 26, 2010

3.

10 Reasons Not to Go to College

A sampling of arguments for the idea that college may not be for everyone....