“But perhaps the absolute worst type of middle-of-the-curve bad science are trials designed to be positive before the first patient is enrolled. These are marketing trials wherein nobody involved is interested in answering a question about nature.”
How do we see “failed” trials that never get published funded by pharma? How can I trust biased Pharma studies designed to profit?
Another way to frame / summarize the list characteristics listed here is: "actionable". Is the outcome of the study something likely to be information that can ground further evidence building or justified shifts in medical practice? All these failures that you rightly focus on make the study results mostly unusable (aka 'useless').
I disagree that the funding cuts are not a problem, because so far they have cut or paused many very valuable and actionable trials-- but I agree with many of these critiques and that in theory there could be more done with less.
I think you’re downplaying the fraud aspect too much. Fraud doesn’t have to be a cartoon evil plot for gain. When a researcher introduces bias that confirms their pre-existing opinion that is fraud, not just poor quality work. A few of them may be so ignorant they don’t understand what they’re doing but most know perfectly well and do it anyway. It’s human nature, they convince themselves that their motive is good and so they can’t be doing anything wrong. I find this more harmful that blatant altering or inventing data because the people that do it have deceived themselves as well as others.
I hate to tell him, they are still placing unnecessary stents and doing a lot of unnecessary surgery. I am not sure why. I think it's the call to do "something". Let's just start with useless prescriptions. I am not sure how these changes because we the public want a pill to fix things, but life doesn't work like that. I think there's plenty of low value research. I know someone that is studying sunflowers. Is that really necessary? Other than a cosmetic use, why is there a government grant for that? Couldn't a company pay for that research? Do we really need all these snack foods with weird oils?
I agree with Dr. Marine’s tweet. What the EO wants to improve upon is hard to quibble with.
What remains to be seen is whether the “solutions” actually solve the problems.
As for the problems Dr. Mandrola identifies, I’m hopeful Drs. B, M, and P can greatly raise the bar. Sponsors will only design trials to clear the FDA bar, and the FDA has failed, in its fiduciary duty as gatekeepers, for years. If the FDA insists upon better pre-approval evidence, I suspect industry will suddenly sponsor better trials (like sham controls in device studies as a no-brainer).
But I see no solution for journals publishing useless observational studies or substudies of RCT. The incentive structure in academia rewards quantity over quality, and the explosion of the numbers of journals directly supplies space for “quantity”. I’m not sure how you break that vicious cycle.
Excellent description and summary of the issue(s). I share his concern with mediocrity. Meta-analysis has value but it seems to have wandered away from any skepticism about the outcome of the analysis, i.e. vs common sense, controlled data, and the fact that as the author points out, the fact that bias and confounding are almost unavoidable. This problem extends beyond medical research. There is a danger of becoming a data laundering operation with worst case being preconceived outcomes being validated by flawed data sets. This is not a knock on statisticians who have worked very hard to develop newer and better methods and to save researchers from pitfalls of poor design, under-powered tests, etc. etc.
I was once given list of life guidelines to follow ; number six was "never attribute to malice that which is better explained by incompetence " ......
Excellent article.
“But perhaps the absolute worst type of middle-of-the-curve bad science are trials designed to be positive before the first patient is enrolled. These are marketing trials wherein nobody involved is interested in answering a question about nature.”
How do we see “failed” trials that never get published funded by pharma? How can I trust biased Pharma studies designed to profit?
Another way to frame / summarize the list characteristics listed here is: "actionable". Is the outcome of the study something likely to be information that can ground further evidence building or justified shifts in medical practice? All these failures that you rightly focus on make the study results mostly unusable (aka 'useless').
I disagree that the funding cuts are not a problem, because so far they have cut or paused many very valuable and actionable trials-- but I agree with many of these critiques and that in theory there could be more done with less.
I think you’re downplaying the fraud aspect too much. Fraud doesn’t have to be a cartoon evil plot for gain. When a researcher introduces bias that confirms their pre-existing opinion that is fraud, not just poor quality work. A few of them may be so ignorant they don’t understand what they’re doing but most know perfectly well and do it anyway. It’s human nature, they convince themselves that their motive is good and so they can’t be doing anything wrong. I find this more harmful that blatant altering or inventing data because the people that do it have deceived themselves as well as others.
I hate to tell him, they are still placing unnecessary stents and doing a lot of unnecessary surgery. I am not sure why. I think it's the call to do "something". Let's just start with useless prescriptions. I am not sure how these changes because we the public want a pill to fix things, but life doesn't work like that. I think there's plenty of low value research. I know someone that is studying sunflowers. Is that really necessary? Other than a cosmetic use, why is there a government grant for that? Couldn't a company pay for that research? Do we really need all these snack foods with weird oils?
Amen Amen Amen
I agree with Dr. Marine’s tweet. What the EO wants to improve upon is hard to quibble with.
What remains to be seen is whether the “solutions” actually solve the problems.
As for the problems Dr. Mandrola identifies, I’m hopeful Drs. B, M, and P can greatly raise the bar. Sponsors will only design trials to clear the FDA bar, and the FDA has failed, in its fiduciary duty as gatekeepers, for years. If the FDA insists upon better pre-approval evidence, I suspect industry will suddenly sponsor better trials (like sham controls in device studies as a no-brainer).
But I see no solution for journals publishing useless observational studies or substudies of RCT. The incentive structure in academia rewards quantity over quality, and the explosion of the numbers of journals directly supplies space for “quantity”. I’m not sure how you break that vicious cycle.
Excellent description and summary of the issue(s). I share his concern with mediocrity. Meta-analysis has value but it seems to have wandered away from any skepticism about the outcome of the analysis, i.e. vs common sense, controlled data, and the fact that as the author points out, the fact that bias and confounding are almost unavoidable. This problem extends beyond medical research. There is a danger of becoming a data laundering operation with worst case being preconceived outcomes being validated by flawed data sets. This is not a knock on statisticians who have worked very hard to develop newer and better methods and to save researchers from pitfalls of poor design, under-powered tests, etc. etc.
Well said- thank you!