Restoring Gold Standard Science is Far More than Stopping Fraud

Some brief thoughts on the call for Gold Standard Science

Look at this post from Dr. Joseph Marine. He comments on the new executive order about “Restoring Gold Standard Science.”

I write to clarify one particular issue about flawed science. The executive order highlights the matter of science that does not replicate, likely because it is fraudulent.

Fraud is not the biggest problem with medical science. Not even close. The biggest problem is what I call “middle-of-the-curve” lousy science. If there are 10 problems with medical science, numbers 1-7 are simply poorly designed and inconsequential projects, which never had a chance of advancing knowledge.

The analogy for the middle-of-the-curve problem is modern clinical practice. While the criminals giving chemo or placing stents without reason are real, downright malfeasance is a rarity. Far more common is everyday low-value non-beneficial care that is performed as standard operating procedure. I call it therapeutic fashion. Coronary angiography after every positive stress test; screening elderly people for cancer; coronary artery scans for people on statins. I could make a long list.

I have no doubt that some small fraction of science is falsified, but far more common is the everyday lousy study.

Take observational comparison studies that are fatally biased from the beginning. Most of these studies have zero chance of eliminating bias. The attempts to match groups is a charade. Journals publish oodles of such studies every week. Sometimes these are popular papers, covered in big news sites.

The authors try not to couch the results in causal language, though they often fail. Yet it would not matter if a large language model scraped every causal verb from the manuscript; the truth is that the goal was to find causation. And it was hopeless. It shouldn’t have been attempted.

Yet incentives to publish are strong. The saddest thing is that the young people assigned to do these worthless studies could have spent this time learning clinical medicine (or cycling or reading or doing just about anything else).

Another example: the 29th substudy of an already underpowered randomized controlled trial. The general results of a trial have wide 95% confidence intervals, leading to uncertainty about the overall outcome. Then authors post-hoc break up smaller subgroups which have even less power. The most likely outcome is being fooled. These may be worse than worthless.

But perhaps the absolute worst type of middle-of-the-curve bad science are trials designed to be positive before the first patient is enrolled. These are marketing trials wherein nobody involved is interested in answering a question about nature.

The ATLAS trial compared two types of defibrillator and the endpoint had 4 of 5 components that could only happen in one arm. The OPTION trial compared safety of a stroke prevention device (which requires an invasive procedure) against a simple anticoagulant and used an endpoint that excludes bleeding from the procedure. This list, too, is long.

I know many people worry that the new administration has cut science funding. I don’t. Because every week, when I review medical evidence, I see tremendous amounts of waste.

The solution of course is to foster science that has every characteristic shown in the slide below:

I don’t know if Vinay, Marty and Jay can reduce the amount of worthless science, or hold industry to higher standards, but the state of medical evidence is really sick. It is need of big changes. Incremental is not going to cut it.

I’m happy to give a new group a chance. JMM