A typical study has six major parts. They generally begin with an abstract, which briefly describes the question the researchers were trying to answer, what data they collected, and what the results were. Then the introduction and literature review sections set the stage and tell readers more about the ideas the researchers were exploring and what previous studies have found. The methods section explains exactly how the study was conducted, which allows other researchers to repeat the experiment to see if they get the same results. Then the results, discussion, and conclusion sections break down what the scientists found and what that might mean. The authors might also bring up any problems or questions they encountered, and suggest avenues for further study. When reading the conclusions, it’s important to understand that the scientists’ data set might support or contradict a hypothesis, but it won’t definitively prove or disprove a hypothesis.
Studies are not meant to be read linearly from start to finish. Instead of being organized chronologically or to create a narrative, the papers are organized by section to make it easier for other scientists to find certain kinds of data or information. For readers who aren’t experts, some sections, like the one outlining methods, can be pretty impenetrable, says Horii.
For the lay reader, she recommends starting by spending some time with the abstract. “Often it’s the most concise and clear articulation of what they were testing, what they actually did, and what they found,” Horii says. When she reads studies, Horii will underline exactly what the researchers say they found and refer back to that claim as she works through the article.
Next, she advises skimming the introduction and literature section to get a sense of the background before skipping to the results, discussion, and conclusion. See what the researchers found, she says, and then bounce it back to what the press coverage or what the abstract are saying. Do the article’s claims actually line up with its results?
If Horii has more detailed questions, then she might dive into the methods section. For instance, if the study claims a drug will be a great treatment for Covid-19 patients, she might look at who was included in the study. Was it tested on a young or old population? On women and men? Was it done in a lab setting, a clinical setting, or out in the world? It also matters how big the study was. Anecdotal evidence is important, but if the paper makes a huge claim and the data only comes from 10 people, that might be a red flag.
Stuart says to watch out for overgeneralizations. “The fundamental challenge there is extrapolating: taking a piece of evidence, a piece of data, which might be perfectly valid, but then assuming that that leads to these much more general conclusions,” she says. What works for very sick patients may not work for those who have less severe cases, and what works for younger patients might not benefit older people, whose immune systems work differently.
Readers should be especially wary of extrapolating the results of studies done in animal models to human populations. Some researchers have struggled to even find the right species for Covid-19 studies, because not all animals react to the pathogen the same way humans do. The NCRC team only reviews preclinical animal studies for vaccines, because they feel that for other interventions the differences between humans and other animals are too significant.
“I think there’s certainly information we can learn from animal studies,” says Grabowski. “But I think it’s always really important to understand how these things work in human populations, particularly when it comes to things that might be considered a behavioral intervention.” She points to examples of studies that try to examine the benefits of wearing a mask, which is a human social behavior that can have many variations. People have to wear the masks properly and in the right situations in order for the intervention to work or even be accurately studied. Those behavioral aspects can’t be measured in animals.