Author: David Randall

In 2018 the Environmental Protection Agency (EPA) dismissed its existing Clean Air Scientific Advisory Committee (CASAC) and established a new panel. Ex-CASAC member John Balmes took to The New York Times to argue that the new CASAC review of the EPA’s Integrated Science Assessment for Particulate Matter is a tool to allow freer emissions of the microscopic particulate matter PM 2.5.

But pollution standards aren’t really what’s at stake.

What’s really at issue is how the government should constrain the discretionary power of scientific experts.

In the heart of the CASAC review, new CASAC Chair Louis Anthony Cox criticizes the way the government allows scientific experts to decide what scientific research should be used to shape federal regulations. Cox argues that the government gives scientific experts too much discretionary power.

Cox graphically describes all the ways that EPA experts misuse their discretionary power. They don’t properly define key terms, such as causal relationship. They rely on research that doesn’t formulate predictive, falsifiable hypotheses. They use arbitrary methods to select which scientific literature will provide the “weight of evidence.” They don’t require that research use publicly accessible, independently reproducible data and algorithms. They don’t require that researchers account transparently for how their research hypotheses, statistical models, and parameterizations may artificially limit their research results. They aren’t transparent about modeling uncertainties, measurement error, and the difference between modeling assumptions and empirical results.

The Irreproducibility Crisis

These aren’t minor matters. Modern science as a whole faces a major irreproducibility crisis of improper research techniques, groupthink, and a scientific culture biased toward producing positive results. John Ioannidis, who in 2005 was the first to begin reckoning with the full scope of this irreproducibility crisis, argues that half of modern published scientific research is probably wrong. Since then it has become clear that irreproducibility is at the heart of a major breakdown in Western scientific practice, as recently argued in The American Mind by Glenn Ellmers and J. Eric Wise. Reproducibility reformers are fighting to reduce scientists’ discretionary power in their labs. Cox is striking a major blow against an irresponsible establishment by extending reproducibility reforms to government policymaking.

I don’t use the word “irresponsible” lightly. Balmes, for instance, has used his own discretionary power in a questionable fashion while serving on the California Air Resources Board (CARB). In 2008, whistleblower James Enstrom informed CARB that the lead author of their staff report on PM 2.5 pollution had lied about having a statistics Ph.D. from the University of California Davis. The author actually received it from a diploma mill. Balmes chose not to disclose this information to his fellow board members, and voted in favor of regulations based on the staff report. In his expert judgment, the report drafted by a fraud adequately represented the state of the field.

Balmes’ actions illustrate precisely why the government should establish strict rules to govern the exercise of scientific expertise. The government needs to form rules to ensure that the exercise of scientific expertise can itself be independently reproduced and verified.

Raising Our Standards

Cox outlines new requirements that scientific experts must use standardized vocabulary and standardized means of selecting only relevant scientific literature with falsifiable hypotheses. This should only be the beginning. The government needs to charter a new committee that will draft standard requirements to make scientific expertise reproducible throughout its agencies. We need to focus on expertise standards without getting the question mixed up with pollution standards.

This committee should include an interdisciplinary team of experts who collectively will command bipartisan assent among policymakers and professional authority among scientists. Committee members might include experts in risk analysis, reproducibility studies, research procedures, statistics, complex and uncertain systems, experimental design, hypothesis and falsifiability, applied ontology, pragmatic science policymaking, groupthink, expertise studies, and political philosophy.

Since shared interests and groupthink may create a feedback loop that improperly narrows the range of scientific research deemed acceptable for grant funding and incorporation into government policymaking, the committee’s remit should include a formal investigation of whether the government should prohibit grant recipients from also acting as the experts who judge research quality. (Balmes, for example, has received a large number of grants from the EPA. These grants both “qualify” him as an expert and raise the possibility that interest or groupthink may affect his expert decisions.)

The committee’s recommendations should draw on the best existing practices within the federal government. If the National Institutes of Health has better rules governing the exercise of scientific expertise than the EPA does, then the NIH rules should be applied to the EPA. The committee should also think creatively about how to import best practices from the larger world of American science. For example, they could translate the Center for Open Science’s institutional support for the principles of open science into government regulations.

Since this will also be a committee of experts, the group should establish rules to govern its own exercise of expertise before it begins work. That way the committee can act as a model for the government as a whole.

The new rules governing scientific expertise should be general enough to be applied to each government agency and scientific discipline, but specific enough to prevent foot-dragging by scientists who don’t want to accept constraints on experts’ discretionary power. Scientists themselves can figure out how best to restrain their own arbitrary exercise of expertise and apply that model across the government.

Done right, this committee’s work won’t devolve into another partisan fight—and the Federal government can deepen its longstanding commitment to make sure it only uses the best science to inform its regulations.