How to Read & Question Scientific Literature
Prevent brainwashing and promote logic with these 10 easy steps!
1. Read media post
(pro-tip: just start with Pubmed if you’re dying for scientific knowledge and skip this step)
2. Find the journal article referenced & download it
If the information costs money - welcome to the first problem with research - there can be a socioeconomic catch.
Note* Quality of journals vary
3. Scan title, authors, and oh! CONFLICTS OF INTEREST
Does the title match the conclusion by the media? Who are the authors and where are they coming from? And importantly, who funded the research?
*The image used is a real-life example of a paper on the effectiveness of the Pfizer vaccine against COVID-19. You can see an author reported funding from Pfizer on this paper. Some authors do not report it, even though they should.
4. Figure out the type of study
Experimental study [1]: a procedure carried out to support or refute a hypothesis (hypothesis driven vs data-driven).
Animal research: [2] [3] the use of non-human animals in experiments that seek to control the variables that affect the behavior or biological system. This approach can be contrasted with field studies in which animals are observed in their natural environments or habitats.
Anecdotal study/Case report [4]: a detailed report of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient.
Observational study [5]: where the independent variable is not under the control of the researcher. One common observational study is about the possible effect of a treatment on subjects, where the assignment of subjects into a treated group versus a control group is outside the control of the investigator.
Randomized control trial [6]: an experiment that aims to reduce certain sources of bias when testing the effectiveness of new treatments; this is accomplished by randomly allocating subjects to two or more groups, treating them differently, and then comparing them with respect to a measured response. One group—the experimental group—receives the intervention being assessed, while the other—usually called the control group—receives an alternative treatment, such as a placebo or no intervention.
Meta-analysis [7]: performed when there are multiple scientific studies addressing the same question, with each individual study reporting measurements that are expected to have some degree of error.
5. Question how and why they did what they did
Methodology matters. Question and be aware of the following:
“Generally, the higher the quality a scientific study has, the more specific the researchers will be about exactly what they have done and how. If they are not being specific, what are they trying to hide?
- Sebastian Rushworth
COVID: why most of what you know is wrong.
Who/what was sampled
demographics of human population (age, sex, race, weight, dietary and exercise habits…).
other species used (cell-culture, rodent models, non-human primate).
What was controlled for? What wasn’t?
keep track of the variables mentioned and pay close attention to the ones not mentioned.
Ask, “could there be another reason why we are seeing this data?”
Nutritional studies are notoriously bad with this -hypothetical here - one study may find meat “causes cancer” however they did not control for smoking (they don’t show the number of people within the sample who were smokers) and therefore, meat cannot be stated to cause cancer because there could be another player involved that’s skewing the data.
6. Question what was measured and what was not measured
Surrogate endpoints: alternate endpoints that “stand in” for the thing that actually matters.
E.g1. “looking at whether a drug lowers cholesterol instead of looking at the thing that actually matters, overall mortality” - Rushworth
E.g2. focusing on specific mortality (cardiac) versus overall mortality - “it is perfectly possible for a drug to decrease cardiac mortality rates while at the same time increasing overall mortality.” - Rushworth
Combined endpoints: adding two or more end points together to get a bigger total number of events.
Eg. “a study might show that there is a statistically significant decrease in the combined endpoint of overall mortality and cardiac stenting … [look] to see if the decrease was actually in mortality, or stenting, or a combination of both. In fact, it’s perfectly possible for overall mortality to increase and still have a combined endpoint that shows a decrease.” - Rushworth
Following adverse events: what are the stated adverse events? what are the rates at which those adverse events happen? are they not including an adverse event?
Eg. “Say a drug decreases the number of people having heart attacks, but increases the number of people developing dementia… this can make the drug seem much safer than it is.” - Rushworth
Per-protocol: only including the results from those who finished the study. People who dropped out due to no effect of treatment, bad side-effects, or mortality are not included in the data. YIKES.
Intention to treat: including everyone who started the study, regardless if they dropped out or not. Ideal.
Double-blind used? to eliminate experimental bias, the subject nor the experimenter knows which trial group they are in (treatment vs control).
7. How did they reach their results?
If a study has a certain outcome that the researchers want, they may work really hard to direct you towards that data. It is not uncommon to “cherry-pick” the evidence that supports their desired claim.
Basically, examine all the data, not just what they choose highlight
What statistical methods were used?
What was statistically significant and why?
What was not statistically significant and why?
8. What are they selling in the discussion section?
*Spoiler Alert*
Science is the scientist’s livelihood, therefore their work has to sell in order for their career to continue. If you’ve made it to the discussion section, be mindful of what they are trying to sell and why. A few motivations for the science may be:
1. Research on a specific subject may be a “hot topic” which could lead to receiving some serious funding for your lab.
COVID-19 (NIH grants $82 million to Centers for Research in Emerging Infectious Diseases)
Alzheimer’s disease (“Salk $19.2 million by the American Heart Association-Allen Initiative to study Alzheimer’s and again in the brain).
2. The product (drug) tested could stand to make serious money.
”The [Pfizer] vaccine brought in $3.5 billion in revenue in the first three months of this year, nearly a quarter of its total revenue, Pfizer reported. The vaccine was, far and away, Pfizer’s biggest source of revenue.” - New York Times
9. Publishing Predicaments
Publishing data in journals is the current distribution route for scientific knowledge. There are a few hiccups that can happen at this stage before the science sees the sun (if it ever does).
Publication Bias [1] occurs when the outcome of a research study influences the decision on whether to publish or not. Often times, the powers that be support experiments that have “positive findings” which is simply statistically significant data.
Conversely, studies that have “null” findings do not get published as frequently. This promotes not only an unbalanced field but wastes a lot of time because multiple labs have to find out on their own that what they were testing does not work.
Peer-review [2] is the evaluation of work by one or more people with similar competencies as the producers of the work. It functions as a form of self-regulation by qualified members of a profession within the relevant field. Peer review methods are used to maintain quality standards, improve performance, and provide credibility. In academia, scholarly peer review is often used to determine an academic paper's suitability for publication.
Peer-review + human nature = possible problematic scenarios
What if I’m a peer reviewer and am reviewing a manuscript that is exactly what I’m working on; ultimately they are competition so I may just reject it (because I “don’t think the science is good enough”).
The paper submitted contradicts the current dogma and therefore should be rejected.
Peer reviewers are anonymous while the authors of the paper are not (this should make you say, “WTF”).
Since this process is unpaid and a donation of time, if I’m spread too thin and don’t have the appropriate amount of time to assess the paper, I may accept or reject it without proper critique.
Replication Issues. If no one has the money to spend on replicating the study, it may go unchallenged.
“The bigger, and the higher quality a scientific study [is], the more expensive it is. This means that most big, high quality studies [randomized control trials are $$$$] are carried out by pharmaceutical companies”
-Rushworth
