Why You Can’t Trust Every “Research Paper” You Find
Research papers are published every day. The majority of research data is inconclusive. There just isn’t enough data to draw conclusions from. In most cases the best a researcher can do is suggest that “this topic is worth further investigation.” Yet that isn’t very exciting to the fat cats signing the grants and funding the projects.
They want something substantial they can use or generate press over. The research that really benefits mankind tends to be incremental. It gets limited funding and even more limited coverage. The research papers we see picked up by the media tend to be fragile claims with sensationalist conclusions.
So what makes a research paper “credible?”
The Pyramid of Evidence
In the world of research, there is a hierarchy of what constitutes good work and what constitutes fabrication. High school taught most of us that any good application of the scientific method needs to be free of bias. Researchers should have no agenda other than to obtain knowledge.
Yet validity is determined by far more than whether or not a study is biased. There are many different kinds of studies. For instance, did you know animal testing is considered the bottom of the barrel in terms of applicable research? It’s actually viewed with such scrutiny that it doesn’t even appear on modern evidence pyramids.
At the bottom of the pyramid you simply have opinions from experts. These are people that presumably have a background in the topic in question. It’s dangerous to consider the opinion of experts, no matter how qualified, as there is no filter to determine if their opinions are accurate or valid (and as opinions go, they tend to be biased by experience).
Moving up from that you have case studies, case series and cohort studies. The former might best be described as a doctor writing up a report about the care he provided and the results he received. It may occur with several patients. Yet as there is no control, they’re only working off of observation and hoping their conclusions are correct.
Cohort studies are over a longer period of time; they’re planned observations. Above those is randomized controlled trials (what we hopefully heard about in schools). These are the classic experiments that involve a control group and an experimental group.
However, even running this kind of trial doesn’t mean the research has been appraised by peers. Drawing a conclusion takes multiple studies and the critical appraisal of the rest of the world’s researchers.
At the very top, you have systematic reviews, which are studies done comparing other studies. This is how researchers determine how valid the body of research is and whether or not it should be applied. Without this process, you get the kind of thing you see on TV.
New Studies Show Coffee Causes Cancer, Cures Cancer…
Without critical appraisals and peer reviews, you get the kind of garbage you see on the news every night. Remember when fat was bad for you? Turns out that study was bogus. Newer, more credible research actually shows very little link between fat and heart disease (excluding trans-fat, which is basically processed poison).
The media doesn’t need research to be backed by peer reviews. At the same time that a paper was published, numerous other studies could have offered contradictory results. Yet you’d never know it from a news report.
Follow the Money
One of the most expedient ways to discern a research paper’s validity is to follow the funding. A great example would be the tobacco industry. Companies such as Philip Morris personally funded studies to determine whether cigarettes cause cancer. There couldn’t be a clearer example of a conflict of interest
The results were a negative (surprise surprise). Yet a later study by Stanford University on the researchers themselves found that they were basically being paid to fudge the results. They provided their “expert opinions” so people could feel safer about the company’s products.
Rest assured, things are no different in the food industry. Corn and soybeans are okay because they’re the top crops in the world (not the other way around).
Don’t be fooled. If you see a research claim, look the paper up online and see if it’s peer-reviewed. You might be surprised how many of these “studies” are just part of some company’s sales scheme.
Do you have any thoughts on the matter? Have any “studies” made you suspicious as you saw them in the media? Leave a comment below and tell us what you think.