NAS Public Comment on Strengthening Transparency in Regulatory Science

The National Association of Scholars has just submitted public comment to Secretary Scott Pruitt in support of the Environmental Protection Agency's proposed rule Strengthening Transparency in Regulatory Science.The new rule requires that “When promulgating significant regulatory actions, the Agency shall ensure that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation.” The NAS has long been concerned about politicized distortions of dose-response science, and our just-released report The Irreproducibility Crisis: Causes, Consequences, and the Road to Reform urges that the Federal government require that only properly reproducible science be used as a basis for regulatory action. We are delighted that Secretary Pruitt’s proposed action addresses two of our priorities. Below is our full public comment, which includes our suggestions for how Secretary Pruitt might build upon this rule to improve reproducibility throughout the EPA--and model larger reproducibility reforms throughout the Federal Government.

 

May 4, 2018

 

The Honorable E. Scott Pruitt
Administrator
Environmental Protection Agency
1200 Pennsylvania Avenue, NW
Washington, DC 20460

Re proposed rulemaking – Strengthening Transparency in Regulatory Science- Docket ID No. EPA-HQ-OA-2018-0259

Dear Administrator Pruitt,

I support the Environmental Protection Agency’s (EPA) proposed rulemaking, “Strengthening Transparency in Regulatory Science.” The current state of science about dose-response supports this measure as do larger concerns about reproducibility in scientific research. Indeed, the larger concerns about reproducibility suggest that this measure should be applied generally within the Environmental Protection Agency and across the Federal Government.

I write as President of the National Association of Scholars (NAS). NAS is a network of scholars and citizens united by our commitment to academic freedom, disinterested scholarship, and excellence in higher education. As part of our mission, we support the highest standards of truth-seeking in the sciences, and seek to have government policy support and rely upon science that eschews political advocacy and subjects its own procedures to the strictest scrutiny.

The NAS is pleased that the EPA has chosen to prioritize the application of reproducibility reforms in the area of dose-response regulation. The NAS has long been concerned about politicized distortions of dose-response science. A notable example is the status of the linear no-threshold (LNT) dose-response model for the biological effects of nuclear radiation.  The prominence of the model stems from the June 29, 1956 Science paper, “Genetic Effects of Atomic Radiation,” authored by the NAS Committee on the Biological Effects of Atomic Radiation.  This paper is now widely questioned and has been seriously critiqued in many peer-reviewed publications, including two detailed 2015 papers.  These criticisms are being taken seriously around the world, as summarized in a December 2, 2015 Wall Street Journal commentary.  This is a consequential matter that bears on a great deal of national public policy, as the LNT model has served as the basis for risk assessment and risk management of radiation and chemical carcinogens for decades. A reassessment of that model could profoundly alter many regulations from the Environmental Protection Agency, the Nuclear Regulatory Commission, and other government agencies.[1] 

In December 2015 the NAS called upon the National Academy of Sciences and the American Association for the Advancement of Science to revisit the 1956 Science paper. Both declined. In the Winter 2017 issue of our journal Academic Questions, Professor Edward J. Calabrese published a detailed account of the origins of the 1956 Science paper, raising serious questions about ethical improprieties that may have contributed to flawed and overstated conclusions in the paper.  These matters are fairly well known. Yet due to the complacency of the scientific establishment and, no doubt, the prestige of some of the scientific bodies implicated in the cover-up, no one in a position of authority until now has been willing to challenge the linear no threshold dose-response orthodoxy.[2] 

The NAS is pleased that the EPA has chosen to address increasing concerns about the irreproducibility crisis of modern science. The NAS has recently written a long report on how the  improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results together have produced a reproducibility crisis that afflicts a wide range of scientific and social-scientific disciplines, from epidemiology to social psychology. Many supposedly scientific results cannot be reproduced in subsequent investigations. We have recommended extensive changes to scientific procedures and to the way government judges the science it uses to make policy—including measures such as this proposed rule, to require that government make policy only based on scientific research whose data and procedures are available for other scientists to reproduce.[3]

In response to the EPA’s solicitation for comment on its proposed rulemaking “Strengthening Transparency in Regulatory Science,” we respectfully provide the following suggestions on ways to implement the principles of scientific reproducibility into the administrative practice of the EPA, and into the administrative practice of the Federal Government as a whole.

1. We recommend that the EPA draft a Reproducible Regulatory Science Guidance Document (RRSGD) that governs all administrative processes (regulatory decisions, proposed rules, promulgations, individual party adjudications, enforcement activities, permit proceedings, site-specific permitting actions, non-binding regulatory determinations, and all other agency actions with precedent-setting influence on future actions) undertaken by the EPA. The RRSGD should:
   1. Define “best available science” to include only scientific research done using pre-registered protocols, whose research data, associated protocols, computer codes, recorded factual materials, and statistical analyses are archived and publicly available in a manner sufficient for continuing independent verification;
   2. Explicitly rescind the “weight of evidence” standard for justifying regulatory policy, and replace it with a “best available reproducible science” (BARS) standard, which meets the definition of (a) above;
   3. Require all regulatory decisions, meaning “significant regulatory actions” as defined by the Office of Management and Budget pursuant to Executive Order 12866, to be based on:
      1. Scientific research that meets the “best available reproducible science” standard;
      2. At least one study on the effect of publication bias on this scientific research;
      3. At least one meta-analysis of this scientific research; and
      4. Explicit consideration of whether the corpus of scientific research has accounted for different aspects of the irreproducibility crisis, including flawed statistics, arbitrary research methods, publication bias, and disciplinary and/or political groupthink.
   4. Direct all existing EPA risk assessments, guidance documents, etc., to be explicitly updated to include the RRSGD’s standards;
   5. Restrict all grant money for scientific research disbursed by the EPA to scientific research that meets the BARS standard;
   6. Provide a set procedure for the EPA Administrator to waive the regulatory decisions requirements in (c) above on a case-by-case basis, and a set procedure for private individuals and organizations to challenge the waiver;
   7. Provide strict criteria for the case-by-case waivers in (f) above, focused on the EPA Administrator’s judgment that he must prevent immediate dangers to the health or life of American citizens;
   8. Be drafted so that it may be used as a model for all Federal agencies that use scientific or social scientific research; and
   9. Be drafted so that it may be used as a model for Federal legislation to introduce reproducible science requirements throughout the Federal Government.
2. We recommend that the EPA call on Congress to enact a Reproducible Science Reform Act (RSRA) that codifies the principles and policies embodied by the RRSGD.
3. We recommend that the EPA prioritize its funding toward upgrading existing research data that does not meet the BARS standard; e.g., by anonymizing research data so as to preserve privacy, confidentiality, etc.
4. We recommend that the EPA also provide substantial funding for a “reproducibility architecture” of hardware and software to facilitate the production of reproducible science by all American scientists whose research informs the EPA’s decision-making.
5. We recommend that the EPA also provide substantial funding so as to make it possible—and then required—for all EPA notices, proposed rules, regulations, etc., to include easily accessible links to all scientific materials used to justify these EPA actions. These links should include all relevant scientific research that meets the BARS standard, but that does not support the proposed EPA action.
6. We recommend that the EPA, as it determines how best to implement the methodologies and technologies of a “reproducibility architecture” to facilitate the adoption of a BARS standard, consult with representatives of the Center for Open Science, the Meta-Research Innovation Center at Stanford (METRICS), and the Laura and John Arnold Foundation Research Integrity Initiative.[4]
7. We recommend that the EPA consult with the American Statistical Association about how to institute standard procedures that will ensure that all scientific research used or funded by the EPA is conducted according to the highest standards of statistical practice.
8. We recommend that each EPA granting program establish a funding category, with funding priority over all other categories, for meta-analysis and research into publication bias.
9. We recommend that the EPA institute a process by which to rescind existing regulations based upon irreproducible science. This process should include:
   1. The establishment of a permanent investigatory commission to examine existing regulations and determine which are based on irreproducible science; and
   2. The establishment of a process to rescind regulations based on irreproducible science, which provides a reasonable amount of time for researchers to make their science reproducible before the regulations are rescinded.

We at NAS are aware that significant numbers of scientists and prominent scientific organizations have voiced their disagreement with “Strengthening Transparency in Regulatory Science.”  Responding in detail to their points is beyond the scope of this letter, but I will add that we have read their public statements carefully and we do not find them persuasive.  Many of those who characterize the proposed rulemaking as unnecessary or counterproductive are the same people and organizations implicated in publishing non-reproducible results and then standing in the way of reasonable and responsible independent efforts to verify important claims derived from those results.

To a woeful extent, the scientific establishment in the United States has proved to be a poor guardian of the quality of scientific research, especially in situations where (A) a large amount of federal research funding is in play, (B) significant public policy decisions hang in the balance, and (C) fixed ideological goals have been set forth.  The combination of these three factors has given rise to advocacy-driven work that claims the authority of science while disregarding scientific methods and safeguards. “Strengthening Transparency in Regulatory Science” is a major step towards rectifying these problems.

NAS believes that these reforms will strengthen the Environmental Protection Agency’s longstanding commitment to using only the most reliable science to inform its decision-making. We also believe these reforms will strengthen American science, by prompting researchers to incorporate and make routine in their practices the highest standards of reproducibility.

 

Sincerely yours,

Peter Wood
President
National Association of Scholars