It seems every week, there’s a new study about the effects of e-cigarettes and vaping. And when studies come out, headlines pop up which sensationalise them, often taking what the study actually found out of context. Knowing how to interpret a scientific study is something that average journalists, with the exception of science journalists, are unclear about - sometimes they just want something to write a good story about, and interpreting the facts properly can take too long in today’s 24-hour news cycle.
This applies to studies that cast vaping in a good light and those that have negative results about ecigarettes - or a scientific study on anything at all. Here are a few tips to distinguish the significant studies from the ones that may have questionable results.
1. Source of Funding
To find a study’s source of funding, look at the “Acknowledgements” section of the study. The problem with this is that the funding sources are rarely available in the abstract, which is essentially the front page of the study. Some journalists write their articles based on the study abstract alone, and do not download or review the entire study - including its funding sources. This is one of the most important things to look for in a study and it can even be hard to tell the actual backers of a study as they may funnel the money through a lobbying group or a foundation set up that has an innocuous-sounding name. The only way to truly get the sources of funding is to research each funding source to see what their ties are to industry.
2. Where was the study done?
If the study was done by a publicly funded governmental organization, it is more reputable since the funding isn’t tainted by potential donors with a vested interest. Generally, very recognizable institutions such as Ivy League universities in the U.S. are considered to be reputable, but funding sources should always be checked for any study coming out of an institution.
3. Where was it published?
In the scientific community, a study is legitimised by being published in a peer-reviewed journal. If a study is trying to appear legitimate, but can’t stand up to peer review, the backers will have it published in an open-access journal. The general public doesn’t know the difference between a peer-reviewed journal and an open-access one, so this tactic can often work. For a list of questionable journals, see the Scholarly Open Access website.
4. Sample size and p-value
Even if a study has amazing results, if it has a small sample size (e.g. a small number of test subjects) those results should be called into question. A reliable paper will contain a p-value, which measures the probability that some results happened by chance. A p-value of 0.05 or less suggests that the study is sound. If there is no p-value listed with a small sample size, you should automatically question the results.
5. Researchers citing their own work
If you look at the “Citations” section of the study, it should not be full of the researcher’s own papers and work. This would signify that they are operating inside of a bubble of their own conclusions and biases, rather than relying on the rest of the scientific community for support.
If you want the facts on any vaping study in particular, Vape Ranks curates a good page on the more popular ecigarette studies that are being listed in the headlines.