Usually it all starts out innocently enough. I first read the abstract (a short summary provided by the authors) of a scientific “peer-reviewed” paper in order to decide if I want to commit myself to finding and downloading the pdf file of the full text.
Sometimes the abstract is as far as you need to go if all you want is a quick reference. The “I have a slide presentation that needs to look like I’m not making stuff up” or “One sentence in a grant that needs 3 or 4 references” or “I need a bunch of references that suggest that A and B are linked in some fashion” types of references.
For example, I write, “Alcoholism causes liver disease.” I do a pubmed or google scholar search for “alcoholism liver disease” and find over 6000 articles. Because this is such a general statement and unlikely to be covered extensively in one particular study, it’s okay for me to cite a recent review article, which covers the topic. I find in the first page of my search, an article by Rehm in “Alcohol research and health” called, “The risks associated with alcohol use and alcoholism.” I scan the abstract, and they have written the word “liver”, so it looks good to me.
I write, “Alcoholism causes liver disease (reviewed in Rehm, 2011).” But, to be fair, we’re scientists, so what we really would write is something like, “Alcoholism has been linked with liver disease (reviewed in Rehm, 2011).”*
Anyway, quick and dirty referencing aside, you’ve got yourself a paper now. Maybe your supervisor sent it to you to read. Or you’re part of a journal club. Or you’re genuinely interested in reading it (…you haven’t been in grad school long, have you?). You’ve read the abstract, so you know what the authors think their data proves. Now you get to decide for yourself if it actually does, which is kind of fun.
If this is a primary paper, meaning one that is data-driven and not a summation of many papers (like the review that I cited above), my first stop is the figures. I don’t read the introduction – it’s my area of expertise, so I don’t need to read about how alcohol is addictive and likely causing my liver to melt. I don’t even touch the paragraphs of text in the results section, since I don’t want to read yet what the authors think their data says.
This is because in a well-written paper, the figures by themselves should create a story. If they have a figure that shows that people who drink 3 martinis a day have less liver damage then someone drinking 6 martinis a day, I will write in the margins of the figure, “More martinis = more liver damage.” This doesn’t mean that martinis cause liver damage necessarily.** Although the correlation is strong, I am pleased to see that in their next figure, they show that in non-smoking, germ-free mice who have only been given a diet of gin, they get jaundice after a few days and then die from acute liver disease. They also have control groups where mice are given normal mice food, or given a diet of non-alcoholic gin, and these mice don’t seem to be dying off. So I write, “Alcohol in gin results in mice dying from liver disease.” I try and think of other controls that they might be missing. Does this happen with all types of alcohol? Am I safe if I drink vodka martinis instead? Humans aren’t mice. Maybe humans can withstand the toxicity of alcohol, and that’s why they think they can drink and drive all the time?
After looking through the figures and coming to my own conclusions on what they say, I then read the results. If I’m confused about anything in the figures or have any questions about the experiments, I’ll consult the methodology or supplemental figures to see if they offer any sort of clarification. For instance, I’m wondering how they collected their data on the number of martinis a person drank a day and how they recruited people for their study. I look to at the methodology section and find that they went to a Hooters bar and had the waitresses hand out surveys, and offered free body-shots for anyone willing to do a liver biopsy.*** Obviously, that would bias their findings towards a certain demographic and would have me question the validity of their results.
Only after I’ve figured out what the figures are saying to me, do I go to the results section and read what the authors think their data says to them. If we agree, then I give the paper a stamp of approval, and walk away feeling happy and optimistic about science and the world at large.**** This doesn’t always happen though. Sometimes the results or discussion section will say, “Based off our findings (refer to the figure with mice drinking gin), we conclude that drinking less than three martinis a day will protect you from liver disease.” And then I get sad at the world, and go and drink a martini.
*However, alcoholism isn’t my field of interest (just something I do on weekends). When you’re writing about your specific field of study, you do get to the point where you can regurgitate data without having to search for it anymore. But then, often what happens to me is that I remember the lead investigator’s name because his/her group has done a series of papers on the same subject. So now I’m stuck trying to remember what paper I need to find that says exactly what I need it to say (and hope it isn’t something that my brain hasn’t just made up with the intention of frustrating me to pieces).
**I had a professor in undergrad that gave an excellent example of how correlation doesn’t always mean causation. “We did a study and found that in the months of June to August, there was an increase in ice cream intake, as well as sun burns. Therefore, we concluded eating ice cream causes sun burns.” Next time you read about how something correlates with something else, I urge you to keep this in mind. The problem is that most media reporting on science doesn’t cite the papers they’re referring to, and all you get is the headline “Scientists report ice cream causes sun burns”. You have to dig to get to the bottom of things and need to come up with your own conclusions.
***And yes, I know you wouldn’t be able to get ethics approval this. But maybe a reality tv show would…?
****Of course, the ultimate validation is if I try and successfully replicate the results shown in the paper. And if establishing the impact of gin drinking on mice livers is important enough to my current research interests, I will go off and do just that.