How we got duped by fake chocolate science

The public, probably thanks to how the media covers science, often regards the findings in peer-reviewed papers as irrefutable fact. But peer review is highly imperfect. As Bohannon points out in io9, his study was published just two weeks after he and his co-authors paid the journal’s publication fee. The journal claimed it has a rigorous peer review but that’s doubtful. “Not a single word was changed,” he wrote.

Having peer-reviewed papers myself, that’s not the norm. At least it shouldn’t be. If you’re submitting to a respected journal like Nature, Cell, Neuron, PNAS, Science, or the Journal of the American Medical Association, a paper will have to pass, first through a staff editor, and then through a handful of reviewers, other scientists who are specialists in the area of research the paper deals with. (Yes, a tiny group of people get to decide what is good science and what isn’t.) Usually, good reviewers tease a paper apart, looking for holes in the data. Often, they’ll ask for extra experiments to round out a study, or suggest ways in which an experiment could be done better. This is supposed to serve as a filter, a quality-control to weed out bad studies. Of course, that doesn’t always work: the latest Science retraction is testament to that. But the problem is bigger. A recent study, for instance, found that 100 of psychology’s most seminal papers were not reproducible.

For now, peer review is the best we’ve got, which is to say we need to do better. Some journal startups, like PeerJ, are trying out a more crowdsourced approach to peer review, but the verdict is still out on whether this model works.

Part of the problem is that reviewers aren’t paid. It’s a volunteer gig. You get an email from a journal asking you to review a paper. Can we really expect overworked scientists to dedicate the necessary amount of time to dissect each paper they’re asked to review? Not always.

Also, what passes peer review can reflect people’s biases rather than a valid assessment of the research. Take, for instance, deep learning—the AI technique that’s en vogue right now at all of the biggest tech companies. In the early aughts, top researchers in this field, like Facebook’s Yann LeCun and Google’s Geoff Hinton, had a hard time getting their papers published if they as much as mentioned deep learning because most computer-vision experts thought the technique was subpar, despite some promising results and commercial applications, like reading digits on checks. Now, things have shifted because the science politics have shifted. Hinton and his grad students, for instance, blew competitors away in a computer vision competition some years ago, and the rest of the field had no choice but to take notice. Now deep learning summits are standing-room only and journals like Nature are publishing articles on the topic.

Science journals can fall prey to trends, and even chase Internet memes. What we see in journals isn’t absolute truth or reflective of the entirety of what’s happening in the scientific world. The gatekeepers’ biases play an important part in what gets published where. Keep that in mind.

Enter rat race to the top.

Because we place such a high premium on high-impact, peer reviewed papers, scientists are judged almost exclusively by their publication record. As Adam Marcus and Ivan Oransky wrote in their editorial about scientific fraud in The New York Times, “scientists view high-profile journals as the pinnacle of success — and they’ll cut corners, or worse, for a shot at glory.”

A paper in Science or Nature as a grad student can set you on the road to academic success. If you can keep that up through your post-doc years, you might even get a shot at a tenure track position. And getting one of those is increasingly difficult. Graduate programs are pumping out more and more grad students, but very few actually manage to climb to the top of the ivory tower. That has bred a highly competitive culture in which “sexy” results are rewarded more readily than meticulous research.

I think part of the problem is funding and job security. Federal funding for science has been drying up in recent years. Pay for young scientists is just atrocious. The baseline salary for a recently minted PhD, as set by the National Institutes of Health, is a measly $42,000 — and that’s for working 80-100 hours a week. Post-docs and grad students are basically cheap, high-quality labor. So the way to work your way up is to come up with interesting data because that can translate into grants, a raise, and perhaps down the line a coveted tenure-track professorship.

Show me the money.

And with that comes the promise of even more money. Many researchers have funding from industry. They get paid to give talks or to do consulting, which can lead to conflicts of interest. When publishing studies, corporate-funded researchers are supposed to disclose this, especially if the way they interpret the data could be biased by company interests, but that doesn’t always happen. Again, something to keep in mind.

We need better ways to make science and its shortcomings more transparent. But this month’s news isn’t all bad: we found out things went awry and how, thanks to those Berkeley students and Bohannon. The next step is doing something about it because these things won’t just fix themselves. Scientists should work together to devise a better way to screen research and the public should be more critical. If it sounds too good to be true, wait for a second study before filling your larders with chocolate.

Daniela Hernandez is a senior writer at Fusion. She likes science, robots, pugs, and coffee.