Introduction

Modern science’s professional culture prizes new research far above replication studies. Ph.D.s, grant funding, publications, promotions, lateral moves to more prestigious universities, professional esteem, and public attention all depend upon research that seems to reveal something new. Scientists therefore avoid doing replication studies. But replication studies are needed to provide the essential check that original research actually is correct in its conclusions. This is particularly true of research that depends on statistics, with levels of statistical significance that guarantee that a large number of research results will be incorrect. Federal grant programs contribute significantly to this problem, since they provide a very large financial and reputational incentive to doing original research at the expense of replication studies.

Our model bill dedicates ten percent of the funds for each research grant program to replication studies, especially to replication studies of government-funded research and to replication studies of research that justified or is proposed to justify a significant regulatory action or a highly influential scientific assessment. Our model bill also funds research that improves best practices of research replicability, including improving best practices of transparency in primary research to facilitate either conceptual replication with different data or actual replication with the same data.

Our model bill refers to the {National Science Foundation}. The bill should be adjusted by policymakers to refer to some or all individual federal agencies funding science grants. Similar references to the {Director of the National Science Foundation}, the {Executive Branch Office}, the {Senate Committee}, and the {House Committee} also should be adjusted appropriately by policymakers.

Model Legislative Text

1. Research Grant Funding for Replication Studies. The {National Science Foundation} shall direct at least ten percent of the funds for each research grant program it operates to replication studies.
2. Funding Priority. The {National Science Foundation} shall give priority to funds granted in subsection (1) above to:
   1. replication studies of research funded by {National Science Foundation}; and
   2. replication studies of research that justified or is proposed to justify a significant regulatory action or a highly influential scientific assessment.
3. Replication Studies Best Practices Grant Funding Program. The {National Science Foundation} shall direct {$X,000,000} to fund research that improves best practices of research replicability, including improving best practices of transparency in primary research to facilitate either conceptual replication with different data or actual replication with the same data.
4. Coordination. To prevent unnecessary duplication of research, activities under this section shall be coordinated with similar activities supported by other Federal agencies, including research funded by {Agency} and {Agency}.
5. Reports. On an annual basis, starting no later than {Month Day, Year}, the {Director of the National Science Foundation} shall submit to the {Executive Branch Office}, the {Senate Committee}, and the {House Committee} a report cataloguing and describing all grants funded in subsection (1) above.
6. Metrics. The {Director of the National Science Foundation} shall develop metrics to measure the success of all grants funded in subsection (1) above.
7. Office of Reproducibility Research. There is established within the Office of the {Director of the National Science Foundation} an office to be known as the Office of Reproducibility Research. The Office shall be headed by a director, who shall be appointed by the {Director of the National Science Foundation}. The Office of Reproducibility Research shall be responsible to the {Director of the National Science Foundation} for executing all responsibilities in this section.
8. Severability. If any provision of this chapter, or the application of any provision to any person or circumstance, is held to be invalid, the remainder of this chapter and the application of its provisions to any other person or circumstance shall not be affected thereby.
9. Definitions.
   1. “Replication studies” shall mean studies for which:
      1. any outcome would be considered diagnostic evidence about a claim from prior research, such that outcomes consistent with a prior claim would increase confidence in the claim, and outcomes inconsistent with a prior claim would decrease confidence in the claim; and
      2. the subject matter and the procedures meet best professional standards in their attempt to replicate the subject matter and the procedures of the prior research.
   2. “Significant regulatory action” shall be defined as any regulatory action that is likely to result in a rule that may:
      1. have an annual effect on the economy of $100 million or more or adversely affect in a material way the economy, a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or tribal governments or communities;
      2. create a serious inconsistency or otherwise interfere with an action taken or planned by another agency;
      3. materially alter the budgetary impact of entitlements, grants, user fees, or loan programs or the rights and obligations of recipients thereof; or
      4. raise novel legal or policy issues arising out of legal mandates, the President’s priorities, or the principles set forth in Executive Order 12866.
   3. “Highly influential scientific assessment” shall be defined as any scientific assessment, as determined by the OMB Office of Information and Regulatory Affairs Administrator or the {Director of the National Science Foundation}, which could have a potential impact of more than $500 million in any one year on either the public or private sector, or whose dissemination is novel, controversial, precedent-setting, or has significant interagency interest.