The Environmental Protection Agency (EPA) has just issued a final rule on Strengthening Transparency in Pivotal Science Underlying Significant Regulatory Actions and Influential Scientific Information. The National Association of Scholars (NAS) applauds the EPA for taking a significant first step toward strengthening reproducibility requirements in the science the agency uses to inform its regulations.
Strengthening Transparency is not as ambitious as the first draft of the rule. Nor is it as ambitious as what the NAS recommended: in our publiccomments, we called for even greater transparency and reproducibility requirements. The policymakers at the EPA, however, had to confront a wave of hostility by science activists and limited support from the EPA’s permanent staff. We imagine that, in the final stages, the EPA policymakers also had to consider that the EPA administration was about to pass into the hands of anti-reform establishmentarians, and that they crafted a final rule so narrowly defined that it might be acceptable to their successors. Some criticisms of the first drafts, moreover, did constructively address looseness and imprecision in the proposed rule. The result is a modest reproducibility reform, which can serve as a model for larger reproducibility reforms in the future—and which might survive the change of administration.
The EPA has confined the scope of its reform to dose-response research, and within that body of research, to research that analyzes “the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect.” The scope of the reform largely confines itself to data, rather than models. The EPA now says that dose-response research with publicly available data should be given greater consideration than research lacking such data—but not that such research should be categorically excluded. The reform only applies to future regulations, not to past ones. The EPA Administrator also may grant an exemption to the requirement to give greater consideration to research with publicly accessible data, if the circumstances warrant. In other words, Strengthening Transparency allows the EPA to continue to use un-transparent data.
Yet the reform will have a significant effect. First, Strengthening Transparency refines the “weight of evidence” standard, which facilitates arbitrary judgments by scientific experts. Strengthening Transparency now establishes and defines a category of pivotal science—“the specific dose-response studies or analyses that drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information.” The EPA must now identify which precise pieces of dose-response research justify its regulations, rather than rely more vaguely on the “weight of evidence.” The emendation not only provides a useful focus to the requirement to give greater weight to research with publicly available data but also, and more radically, requires the EPA to justify their regulations by reference to specific research and to explain why they rely on those individual studies. The requirement to explain ultimately may be more revolutionary than the requirement to provide public data.
Second, Strengthening Transparency takes a first step to reduce the unwarranted authority of journal peer review. Until now, the EPA has accepted too readily the authority of journal peer review—even though such peer review has not generally included independent inspection of the underlying data, and even though journal peer review has proven susceptible to capture by group-think. Strengthening Transparency allows, but does not require, the EPA to commission a new independent peer review of pivotal science and to “ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied and the implications of those assumptions for the results.” This portion of Strengthening Transparency will have little practical effect until a reform-minded EPA Administrator dedicates himself to commissioning such independent peer reviews on a regular basis. Yet it is important for shifting the EPA in principle away from its unwise credulity in the reliability of journal peer review.
Third, Strengthening Transparency requires the Administrator to explain his rationale when he grants an exemption to the requirement to give greater consideration to research with publicly accessible data. Just as the requirement to explain should change the way the EPA uses research to justify regulation, it should also inhibit activist Administrators from intervening arbitrarily to short-circuit the new transparency requirements.
Fourth, Strengthening Transparency provides administratively actionable definitions of several concepts essential for reproducibility reform: data, dose-response data, independent validation, influential scientific information, pivotal science, publicly available, reanalyze, science that serves as the basis for informing a significant regulatory action, and significant regulatory actions. Reproducibility reformers cannot use language casually when they attempt to transform their principles into regulations; they must use carefully defined language, capable of consistent administrative application, and without untoward effects. Strengthening Transparency provides a model in the realm of dose-response research for how this should be done.
Fifth, Strengthening Transparency does give preference to research that considers “a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.” Given the substantial controversy about whether current dose-response models are correct, it is a significant improvement to prefer research that at least considers which of the several models best fits the dose-response data.
All this, of course, is in addition to the central effect of Strengthening Transparency—to increase the transparency requirements of dose-response regulation. While the activist opponents of Strengthening Transparency have exaggerated the consequences of this reform, it will presumably have a considerable effect (if not rescinded) to restrict the EPA’s capacity to impose regulations that rely on scientifically unfounded dose-response research. That change alone should significantly improve our country.
We may judge Strengthening Transparency to be a modest but significant success simply in terms of the reforms it enacts. But we should also consider what the enactment of Strengthening Transparency tells us about the larger political landscape and the tactics of reproducibility reform.
Perhaps the most important thing to note about Strengthening Transparency is how long it took for it to emerge—to be drafted, submitted for comment, revised, and finally published in the very last weeks of a four-year administration. The skeleton of EPA policymakers strove valiantly, but they were in the end forced to produce a fairly limited rule. They have made a virtue of necessity and are advertising Strengthening Transparency as a model reproducibility reform. This is true—but it is also the maximum reform that could emerge, against the resistance of EPA staff and the scientific establishment.
We should also note the overwhelming and hysterical response of the scientific establishment to these reforms—which, even in their earlier drafts, were scarcely the stuff of revolution. True, thoroughgoing reproducibility reform would have a revolutionary (and beneficial) effect on scientific regulatory policy—but the scientific establishment has proven itself unwilling to countenance even this fairly minor reform. Future reform efforts will have to expect equally fanatical resistance.
We may further note that some of the critiques of the initial rule did correctly focus upon its loose, imprecise language. EPA policymakers should have been prepared to provide a more polished document from the very first stages. The lack of full support from the EPA staff presumably explains some of this unpreparedness. Nevertheless, reproducibility reformers should work harder in the future to provide more expertly drafted proposals. Indeed, they would be advised to do the work to provide such expert draftsmanship in the interim before reproducibility reformers again lead the EPA, so that such proposals can emerge more swiftly from the EPA’s regulatory process.
Then, Strengthening Transparency reveals the absolutely massive effort necessary for thoroughgoing reproducibility reform. Even this rule will require further bureaucratic action, to integrate it properly into the EPA’s regular procedures. The experience of Strengthening Transparency argues that dozens or hundreds of such new rules will be needed to adequately reform the entire EPA—and the EPA is not the only agency that relies on scientific information to justify its regulations. The rate by which the federal government produces new regulations based on scientific research would seem to outpace the rate of reproducibility reform by a substantial margin. This incremental approach has some effect—but probably not enough to produce thoroughgoing reproducibility reform.
Purely administrative reform at the EPA would be more effective if a substantial fraction of the scientific community, and of the EPA’s permanent staff, were to shift in favor reproducibility reform over inertia and activism. We do not need a majority of scientists and administrators to be won over—just enough to provide the needed professional and bureaucratic expertise to make the EPA’s machinery run smoothly for reform. The strengthening power of the reproducibility reform movement makes this a possibility—a growing number of scientists have become disaffected from activist pseudoscience. Reproducibility reformers should make it a strategic priority to create such an alliance. But the political and cultural bonds between most scientists and the activist left are so strong that we should not be optimistic about this project’s chances for success.
This should return reformers to the possibilities of legislation, along the lines of the Secret Science Reform Act. Such legislation has failed repeatedly in the past to advance beyond Bill stage—but it would seem that only such legislation would have the power to increase the rate of reproducibility reform beyond a snail’s pace. Moreover, the EPA was only able to enact Strengthening Transparency by use of its internal “housekeeping” power. While this was legitimate, it is untoward to rely on such powers to achieve policy change. Reproducibility reformers ought to be able to base their policy on the democratic consent of the people, expressed by means of legislation. This is a difficult goal—but one for which it is worth aiming. Reproducibility reform will not likely be effective without that popular sanction.
The two strategic priorities of reproducibility reform are 1) to initiate a political and cultural shift that detaches a critical minority of reproducibility-minded scientists from their activist left allies; and 2) to assemble a political coalition capable of passing reproducibility reforms into law. But what we need immediately is to draft all the dozens or hundreds of reproducibility reforms that should accompany Strengthening Transparency, and which apply reproducibility reform to every specialized domain of the EPA and other federal agencies. Reproducibility reformers should think now how precisely to translate their principles into precise, actionable administrative language. They should also consider how to proceed from these modest beginnings to more rigorous requirements, in equally precise and actionable administrative language. If they draft such reforms now, then they can be enacted at a far more rapid pace the next time that reproducibility reformers have a chance to set EPA policy.
Boldness and principle are not enough. Reproducibility reformers must also be prepared.
David Randall is Director of Research at the National Association of Scholars.
