Is the reason money? I'm pretty sure the reason is money.. plus the hijacking science to wield its authority to push propaganda because religion doesn't scare people enough anymore.
In summary: because it's now being used as a control device.
Causal inference from secondary or archival data is not inherently flawed - if studies are designed well. Volumes have been written by social scientists on inference from non-experimental research, and almost all good policy research has drawn on studies of this type. Good epidemiological research often draws on these methods. The problem is not RCT vs observational data. It is good vs bad research design in either case.
Drawing causal inferences from a cohort study is pretty egregious, and when major journals publish it, it simply adds to the mistrust of the public towards the profession, and of skeptical physicians towards the “establishment “ types.
It seems every day I’m adding dollops of cynicism into my baseline of healthy skepticism.
We have known about this problem for at least a generation. (See citations below.) Scientists use data. We may have become confused into thinking the use of data makes one a scientist. Has observational publication replaced the wearing of white coat and stethoscope as expertise signaling? Is the disease of "scientistics" curable and can physicians heal themselves? Is this too strong a question?
Maybe the data hold back solution proposed by Young and Karr over a decade ago would help save observational studies.
Granger, C. W., & Newbold, P. (1974). Spurious regressions in econometrics. Journal of Econometrics,
2(2), 111-120.
Hendry, D. F. (1980). Econometrics-alchemy or science?. Economica, 47(188), 387-406.
Mayes, L. C. , Horwitz, R. I. and Feinstein, A. R. (1988) A collection of 56 topics with contradictory results in case-control research. International Journal of Epidemiology, 17, 680–685.
Feinstein, A. R. (1988) Scientific standards in epidemiologic studies of the menace of daily life. Science, 242, 1257–1263.
Ioannidis, J. P. A. (2005) Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294, 218–228.
True and can't argue that incentives are part of it. But a more optimistic take - it's not the studies that are bad necessarily, EVERY study has limitations. Rather, I think it's the conclusions drawn. And the classic one is the one you posed where observational data is used to make causative conclusions. At the same time, it took 40 years to establish a link between smoking and lung cancer, none of which is from randomized data so there is a "don't throw the baby out with the bathwater" thing going on. Also, if one is going to invest millions of dollars in an appropriately powered randomized study, you can also argue that some less costly observational studies that see an association help increase the value of the investment.
Thank you for this thought-provoking article. I work in the medical device industry to help manufacturers improve safety of their devices. I am curious to know about the reasons for lead extraction. Is this procedure needed due to any malfunctions of the device?
Is the reason money? I'm pretty sure the reason is money.. plus the hijacking science to wield its authority to push propaganda because religion doesn't scare people enough anymore.
In summary: because it's now being used as a control device.
Causal inference from secondary or archival data is not inherently flawed - if studies are designed well. Volumes have been written by social scientists on inference from non-experimental research, and almost all good policy research has drawn on studies of this type. Good epidemiological research often draws on these methods. The problem is not RCT vs observational data. It is good vs bad research design in either case.
Drawing causal inferences from a cohort study is pretty egregious, and when major journals publish it, it simply adds to the mistrust of the public towards the profession, and of skeptical physicians towards the “establishment “ types.
It seems every day I’m adding dollops of cynicism into my baseline of healthy skepticism.
Over many years I have read SO many studies. [I am 70]
When I would mention them to my husband's kidney doctor, he always asked: "Was it a real study?"
Sometimes he was kidding and other times he was trying to get me to LOOK more.
That helped me to realize that I needed to do really in depth study to find out the so called 'truth'!!!
Thank you for your essay.
We have known about this problem for at least a generation. (See citations below.) Scientists use data. We may have become confused into thinking the use of data makes one a scientist. Has observational publication replaced the wearing of white coat and stethoscope as expertise signaling? Is the disease of "scientistics" curable and can physicians heal themselves? Is this too strong a question?
Maybe the data hold back solution proposed by Young and Karr over a decade ago would help save observational studies.
Granger, C. W., & Newbold, P. (1974). Spurious regressions in econometrics. Journal of Econometrics,
2(2), 111-120.
Hendry, D. F. (1980). Econometrics-alchemy or science?. Economica, 47(188), 387-406.
Mayes, L. C. , Horwitz, R. I. and Feinstein, A. R. (1988) A collection of 56 topics with contradictory results in case-control research. International Journal of Epidemiology, 17, 680–685.
Feinstein, A. R. (1988) Scientific standards in epidemiologic studies of the menace of daily life. Science, 242, 1257–1263.
Ioannidis, J. P. A. (2005) Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294, 218–228.
Young, S.S. and Karr, A. (2011), Deming, data and observational studies. Significance, 8: 116-120. https://doi.org/10.1111/j.1740-9713.2011.00506.x
True and can't argue that incentives are part of it. But a more optimistic take - it's not the studies that are bad necessarily, EVERY study has limitations. Rather, I think it's the conclusions drawn. And the classic one is the one you posed where observational data is used to make causative conclusions. At the same time, it took 40 years to establish a link between smoking and lung cancer, none of which is from randomized data so there is a "don't throw the baby out with the bathwater" thing going on. Also, if one is going to invest millions of dollars in an appropriately powered randomized study, you can also argue that some less costly observational studies that see an association help increase the value of the investment.
Why so many bad studies? One word: $.
Thank you for this thought-provoking article. I work in the medical device industry to help manufacturers improve safety of their devices. I am curious to know about the reasons for lead extraction. Is this procedure needed due to any malfunctions of the device?
excellent article!!