Making Government Policy Reproducible

The National Association of Scholars (NAS) has just sent a friendly letter to Secretary Elaine Chao of the Department of Transportation (USDOT), urging her to strengthen reproducibility standards for research funded by the Department.

The USDOT funds disparity studies, which are used to determine the existence of racial discrimination in the transportation business sector, and to justify government intervention to counter that discrimination. Many of these disparity studies are published without publicly accessible data or methods, on the grounds of ‘proprietary methodology’ and ‘confidential data.’ Because the data and methods are not publicly accessible, they are not reproducible—and may well present false findings. The NAS is urging the USDOT to draft a Transparent Research Guidance Document (TRGD) to update its guidance on disparity studies. The TRGD should require that the USDOT only fund or use disparity studies based on publicly accessible research. Research cannot be reproduced unless it is publicly accessible

The NAS is generally interested in fixing the irreproducibility crisis of improper use of statistics, arbitrary research techniques, political groupthink, and a scientific culture biased toward producing positive results—both by urging reforms in modern science’s professional culture and by urging the government to strengthen its reproducibility standards. This summer we sent in a public comment supporting the Environmental Protection Agency’s proposed rule Strengthening Transparency in Regulatory Science. We now also urge the USDOT to take a step toward strengthening its reproducibility standards.

The USDOT’s existing guidance on how to use scientific research was drafted before scientists became aware of the full gravity of the irreproducibility crisis. We think the USDOT should strengthen its reproducibility standards because the American government has always wanted to use the best available science to inform public policy. Now that scientists are aware that best available science means publicly accessible science, it’s only natural the government should also adopt that definition.

***

                                                                                                                        November 19, 2018

Secretary Elaine L. Chao

US Department of Transportation

1200 New Jersey Avenue, SE

Washington, DC 20590

CC: Steven G. Bradbury, General Counsel

CC: Judith S. Kaleta, Deputy General Counsel

CC: Andrew R. Kloster, Legal Advisor to the General Counsel

Dear Secretary Chao,

I urge the United States Department of Transportation (USDOT) to reform its procedures to ensure that it relies upon “disparity study” research that meets strict standards of reproducibility.

I write as President of the National Association of Scholars (NAS). The National Association of Scholars is a national organization of scholars and citizens committed to academic freedom, disinterested scholarship, and excellence in higher education. As part of our mission, we support the highest standards of truth-seeking in the sciences, and seek to have government policy rely upon research that eschews political advocacy and subjects its own procedures to strict scrutiny.

The NAS has recently reported on how the improper use of statistics, arbitrary research techniques, political groupthink, and a scientific culture biased toward producing positive results together have produced a reproducibility crisis that afflicts a wide range of scientific and social-scientific disciplines. Many supposedly scientific results cannot be reproduced. We have recommended extensive changes to government procedures, to ensure that only scientific research whose data and procedures are available for other scientists to reproduce can be used to support government policy.[1] If executive agencies, legislatures, and courts rely on irreproducible research, they risk enacting harmful public policies, wasting taxpayer dollars, and forfeiting public confidence. 

Every year the USDOT funds numerous disparity studies to examine whether there is a compelling interest and narrow tailoring for the use of race and gender classifications in awarding USDOT-supported transportation contracts, particularly in Disadvantaged Business Enterprise (DBE) programs. These disparity studies are supposed to be performed according to the highest professional standards. USDOT guidance states that:

• research “should ascertain the evidence of discrimination for each separate group presumed … to be disadvantaged”;
• “Recipients should exercise caution in drawing conclusions about the presence of discrimination and its effects based on small differences”;
• “In calculating availability of DBEs, the study should not rely on numbers that may have been inflated by race-conscious programs that may not have been narrowly tailored”; and
• “the [disparity] study should rigorously determine the effects of factors other than discrimination that may account for statistical disparities between DBE availability and participation. This is likely to require multivariate/regression analysis.”[2]

This guidance predates the current professional awareness of the irreproducibility crisis’ gravity, and does not require that disparity studies be performed using data and methodologies that are transparent and reproducible. As a result, many private firms, on the grounds of ‘proprietary methodology’ and ‘confidential data,’ produce disparity studies without publicly accessible data or methods. The USDOT therefore has funded studies that turn out to have insufficient scientific foundation and may well present false findings.[3] Both the USDOT’s commitment to rigorous sciences and its prudential guardianship of taxpayer money argue that it institute rigorous reproducibility standards.

The NAS recommends that the USDOT draft a Transparent Research Guidance Document (TRGD) to update its guidance on disparity studies. The TRGD should require that the USDOT only fund or use disparity studies based upon publicly accessible research. Publicly accessible research should be defined as research whose registered report (including protocols), research data, associated protocols, computer codes, data analysis scripts, recorded factual materials, statistical analyses, and algorithms are archived on an online digital repository in a manner sufficient for continuing independent inspection, replication, reproduction, and verification.

NAS believes that this reproducibility reform will strengthen the USDOT’s longstanding commitment to use only the most reliable science to inform its decision-making. We urge you to continue USDOT’s tradition of steady improvement to its high professional standards.

                                                                                                Sincerely yours,

                                                                                                Peter Wood

                                                                                                President

                                                                                                National Association of Scholars