Tell Me How I Should Feel About This Study
Evidence is what separates doctors from palm readers. But the evidence has to be trustworthy
Here is the second half of my two-part series on curious studies. The first part is here. Thanks again for your support. JMM
This series is about two things: clinical trials and trust.
In an ideal world, clinicians (and patients) should trust that scientists strive (neutrally) to answer an important question.
Trust is necessary for the system to work. If there is suspicion that a scientist endeavors for anything else besides evidence, then the system breaks down.
And…If a major journal, with its acclaimed editors, peer reviewers and statisticians, publish an obviously biased study, then what should readers think of subsequent studies in that journal, or other similar journals?
Before I tell you about this biased study, I need to say that no single trial can ever flawlessly answer a question. That is not how science works. A good study raises as many questions as it answers.
Yet, the users of medical evidence need scientists to be fair in the design, conduct and interpretation of the trial. We also expect medical journals to enlist neutral peer reviewers and editorialists to put the trial results in context.
The MONITOR-HF trial assessed the efficacy of a special tool in the management of patients with chronic heart failure.
The CardioMEMS device looks like a paperclip. A cardiologist inserts the device into the pulmonary artery (the PA connects the right ventricle and lungs). The device measures pressures in the PA, and wirelessly transmits the data to doctors.
The idea makes sense—for three reasons: A) It’s hard to manage patients with chronic heart failure. Not enough medicine and fluid builds up in the lungs; too much medicine can harm the kidneys and cause passing out from low blood pressure. B) Clinicians manage this delicate balance now with simple things like asking about symptoms, signs (swelling) and measuring weights—with a scale! C) An invasive and wireless monitor should be much better.
But regular readers of Stop and Think know that things that make sense don’t always pan out. This is where you need randomized controlled trials.
In fact, in the matter of the CardioMEMS device, a large blinded randomized trial called GUIDE-HF found that patients with heart failure managed with the device did not have a statistically significantly lower rate of death or heart failure events than those managed with standard of care. (How these non-significant trial results led to regulatory approval and widespread acceptance of this pricey device is also a sad story—but I will now tell you an even sadder story.)
In the MONITOR-HF trial, Dutch scientists randomized nearly 350 patients with heart failure to be managed with the device vs standard of care.
Unlike GUIDE-HF, the MONITOR-HF trial was not blinded. One group got a fancy wireless and invasive monitor, the other group did not. That is vital information because of the authors choice of endpoint—how patients graded their health.
The Kansas City Cardiomyopathy Questionnaire or KCCQ is patient-reported and includes questions about physical limitations, self-efficacy, social limitations, and emotional well-being. Responses are on Likert scale, so the output of qualitative sensations are quantified. KCCQ has been well validated in previous studies. But it is a subjective measure.
The difference in mean change in KCCQ overall summary score at 12 months was 7·13 (95% CI 1·51–12·75; p=0·013)
Boom. The MONITOR-HF was positive. But there was more. The authors also reported and emphasized a statistically significant 44% reduction in heart failure hospitalizations in the CardioMEMS arm.
They also included a table of outcomes pictured below. Notice that every clinical outcome that had a p-value ≤ 0.05 had heart failure hospitalizations as a component.
Comments:
I hope you see the problems.
It defies common sense to measure subjective outcomes where one group gets an invasive procedure and a fancy device, and the other group does not. This is why we have placebo controls.
The problem of lack of blinding also biases the decision to admit a patient for heart failure. In MONITOR HF, patients and doctors knew their treatment assignment. The decision to admit a patient to the hospital is a subjective one. Trials are done at specialty centers. Doctors at specialty centers are often proponents of the latest device.
You don’t have to imply any degree of malfeasance to think knowledge of treatment assignment might affect the decision to hospitalize (or discharge) a patient in a trial with heart failure symptoms. This is why trials have blinding. Because of human nature. The fact that every clinical outcome with a heart failure hospitalization as a component had a significant p-value suggests bias.
The third problem is that the authors and the editorialist even note the clinical outcomes. This trial enrolled 1/3 rd of the patients in GUIDE HF—which was an outcomes trial. Even if MONITOR HF was blinded, it would not be enough patients to sort signal from noise in clinical outcomes.
Now I want to connect the trial results and trust. Let’s use pictures.
Here is a picture of the conclusions of this unblinded under-powered trial in the prominent journal The Lancet:
One more picture from the manuscript. JJB are the initials of the primary author.
Here are two quotes from the editorial:
MONITOR-HF showed substantial benefits in patients discharged after a heart failure-related hospitalization…
Scientific evidence supports the use of the CardioMEMS-HF system to enhance remote patient management in heart failure care.
Here is another picture from the editorial:
What am I to think about the state of medical evidence if this can happen?
To review:
The trial was biased from its design. The worst part is that these are experienced scientists. They knew an unblinded study with subjective endpoints would be hopelessly biased. They did the study anyways.
They note the limitations in the discussion section, but this does not stop them from concluding benefit and suggesting that these results should influence guidelines.
The peer reviewers surely knew this was a biased study. Nothing happened. The editors of The Lancet knew it as well. But there it is, in print.
And the chosen editorialist also offers a few sentences on limitations, but goes on to give the device glowing reviews. But she, too, has major conflicts of interest.
Whether or not a low-value high-cost device gets accepted in the care of heart failure will not bring the end of the world.
But the presence of such a story shreds trust in science. How can we believe in any medical science when we see this?
I am asking. Let me know what you think.
I don’t disagree with your assessment. You are highlighting the limitations of trials with medical devices, which are not easily placebo controlled. In this case, it is also not practical to have them blinded. The bigger question is whether this intervention does more harm than good. Evaluating benefit-risk of a medical device is not straightforward, it requires a lot of judgment. The risk I think you are highlighting is that claims made from such studies end up distorting our judgment. We need to acknowledge the level of uncertainty in benefit-risk. If we still decide to approve/clear a medical device with a high level of uncertainty then we should require a post-approval study, increased labeling control and safety information to guide the physician’s decision.
I was intrigued by the FDAs decision to approve Leqembi, a new type of drug that claims to slow down the progression of Alzheimer’s. I would love to hear your perspective on this example. Thank you.
How do we press the journals to have some conviction? I am very frustrated and I am losing faith in the discipline I have practiced and pushed for over 20 years. It makes me wander what else I have believed and or been pushed to do because of “best practice or guidelines”. I really makes me sick.