New York, NY, July 12, 2022 – The National Association of Scholars (NAS) has released a new report examining the consequences of irreproducible science on government policy and regulation. The second report of Shifting Sands: Unsound Science and Unsafe Regulations, Flimsy Food Findings, focuses on Food Frequency Questionnaire and irreproducible research in the field of nutritional epidemiology, which informs the U.S. Food and Drug Administration’s (FDA) policies and regulations. The report finds compelling circumstantial evidence that the nutritional epidemiology literature—specifically the literature that relies on Food Frequency Questionnaires (FFQs)—has been affected by statistical practices that have rendered the underlying research untrustworthy.
NAS’s report on the FDA continues the work of the first report of Shifting Sands, which studied PM2.5 Regulation and irreproducible research in the field of environmental epidemiology, which informs the U.S. Environmental Protection Agency’s (EPA) policies and regulations. The first report found compelling circumstantial evidence that the environmental epidemiology literature on PM2.5—specifically for mortality, asthma, and heart attack claims—also had been affected by statistical practices that had rendered the underlying research untrustworthy.
“The scientific world incentivizes the publishing of exciting research with significant association claims—but not reproducible research,” explained NAS Director of Research David Randall. “This encourages researchers, wittingly or negligently, to use a variety of statistical practices to produce positive, but likely false, claims.”
Shifting Sands argues that federal agencies would benefit from more rigorous and transparent statistical processes in laying the foundation for regulations. If these regulatory regimes are founded on faulty science, it could threaten to undermine public policy initiatives, and especially public trust in government-funded research. This is an outcome Americans can ill afford when trust in policymakers is already at an all-time low.
Randall continued: “Replication makes for sound science. Without it, our government is relying on potentially faulty research to create and enforce regulations that have real consequences on the quality of life for millions of Americans. I and my colleagues, Warren Kindzierski and Stanley Young, offer 12 recommendations to bring the FDA’s methodologies up to the level of best available science.” These include:
- Adopt resampling methods as part of the FDA’s standard battery of tests applied to environmental epidemiology research.
- Rely exclusively on meta-analyses that take account of endemic biases, including question research procedure, HARKing, and p-hacking.
- Require pre-registration and registered reports of all research that informs the FDA’s regulations.
- Require public access to all research data used to justify regulations.
- Place a greater weight on reproduced research for informing FDA regulations.
- Require the FDA in its assessments of scientific studies to take account of endemic HARKing, p-hacking and other questionable research procedures.
- Establish systematic procedures to inhibit research integrity violations.
America’s government agencies must shore up the science that underlies its regulatory regime. Failing to do so will have economic consequences and further erode public confidence in the nation’s leading policy experts.
NAS is a network of scholars and citizens united by a commitment to academic freedom, disinterested scholarship, and excellence in American higher education. Membership in NAS is open to all who share a commitment to these broad principles. NAS publishes a journal and has state and regional affiliates. Visit NAS at www.nas.org.
###
If you would like more information about this issue, please contact David Randall at [email protected].
