Was It Right to Fool Millions of People into Thinking Chocolate Helps You Lose Weight?

If you opened a newspaper at the end of March, it is possible that you may have noticed an article telling you that chocolate makes you lose weight. The more skeptical amongst you probably already know that these kinds of nutritional claims are very often not worth the paper they are written on; nutrition is by its very nature an inexact science requiring very large, very well-controlled studies that sadly are the exception rather than the rule. As John Ioannidis once wrote in the British Medical Journal:

“Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome”

The paper cited in the coverage in May however was different from the countless similar studies we see reported in newspapers on a regular basis; this time the misinformation was intentional. Journalist Johannes Bohannon (real name John) conducted the study to demonstrate how statistically significant — but yet completely meaningless — findings can be extracted from a poorly conducted study. The experiment wasn’t really about chocolate at all; it was a test to see whether professional journalists would buy an obviously flawed paper.

Unsurprisingly perhaps, they bought it in spades. The paper made the cover story of Bild – Europe’s most read newspaper, and also featured in “the Daily Star, the Irish Examiner,Cosmopolitan’s German website, the Times of India, both the German and Indian site of The Huffington Post, and even television news in Texas and an Australian morning talk show“. While the study was submitted to a newswire distributed to 5,000 journalists, not one journalist sought to expose the obviously ridiculous paper or the headlines created by it.

Though not at all dissimilar to a great deal of poorly conducted nutrition research that makes big claims based on small sample and effect sizes; this particular paper was actually published in a bogus “pay to play” journal that conducted no peer review whatsoever. This is far and away my biggest criticism of the sting, which, had it also been submitted to bona fide nutrition journals, could have truly shaken the field, rather than merely rattled journalists. I made a very similar criticism of the same journalist’s “sting operation” against open access journals in 2013, in which a bogus paper was submitted to hundreds of open-access journals but no traditional paywalled journals.

The response online has ranged from outrage to glee. Like many, my initial reaction was one of mixed feelings. On the one hand, the study knowingly put misinformation into the public domain on a grand scale, misinformation that will likely impact the lives of countless people who never had the opportunity to consent and who may well never know that they were part of an enormous social experiment. For all we know, the readers of the news reports may continue eating chocolate bars with reckless abandon until they die of a heart attack, obese and alone. Full disclaimer: I am being facetious.

On the other hand, can we really honestly say that this one news story could have any discernible negative impact amongst the continual Niagara Falls of junk science stories just like it? Anyone who might have believed this story enough to change their diet because of it, frankly, has bigger problems. Problems that this sting may help tackle. One British newspaper, the Daily Mail, known for both its cavalier attitude to misreporting science and for running the most widely read news website on the planet, has suggested over the years that so many hundreds of different things both cause and prevent cancer that an entire website is dedicated to tracking its progress. Chocolate by the way, is already in there nestled in between chillis, Chinese medicine, chips, Chlamydia, chlorine, cholesterol, and circumcision:

If the Bohannon paper wasn’t just like hundreds of other papers that get published every month and the stories covering it weren’t just like the stories we see in the papers day in, day out, then the paper might be cause for greater concern. Unlike the sea of poorly conducted science we see in the papers every day, the fallout from Bohannon’s paper at least has the potential to shock people into thinking differently and have a real impact.

On balance, it seems reasonable to assume that the overall impact of Bohannon’s paper will be resoundingly positive. Scientists who read the story may decide to design experiments that are well-controlled enough and with big enough samples to provide valid results that don’t risk becoming an embarrassment. Journalists may recognise that their industry has a problem and learn to treat scientific findings with an appropriate level of skepticism; pigs might fly, but that’s a topic for another blog post (see you next week). 

Most importantly of all, within only 48 hours of publication, Bohannon’s explanation for his study had already been read hundreds of thousands of times. We can only hope that the people reading his article will take home lessons that are far more valuable and meaningful than any two-bit breaking news story ever could be: That just because a study is published in a scientific journal doesn’t mean its results are valid. Just because a study’s finding is statistically significant, doesn’t mean it didn’t occur by chance. That a single study on its own should never cause us to drastically change our behaviour, especially if its sample is small and the effect size is weak. 

Even if the only thing people take away from this affair is that they cannot trust newspapers to communicate science to them, then this will be a very good thing. For so many people it seems that the only time they are genuinely skeptical of what they read in the papers is on the first of April. For this reason I would argue that claims that the sting is “shameful” for “undermining (the public’s) trust in both journalism and science” couldn’t be wider off the mark. The last thing we want is a public that blindly trusts journalists to report science accurately and uncritically. A culture of informed skepticism where people are inclined to actually check what they read in the science pages is far more healthy. 

None of this changes the fact that the study crossed a big red line in research ethics, a point that, like so many things in life, was well-made by Ed Yong on Twitter:

It is entirely unethical for a journalist to deliberately introduce false info into the media to prove a point, IDGAF what that point is

— Ed Yong (@edyong209) May 28, 2015

The discussion that followed included some wonderful analogies: "It's like shouting 'haha fooled you' at a train that departed several hours ago" on the one side and "I see it like adding a dye to a water inlet to prove that the pipe is leaking" on the other.

While this is certainly an ethical gray area, I do feel that on this occasion, when it comes to science, the fact that the popular media is currently a cesspool of false information already is a point worthy of consideration. While some have argued that the "point has been made for years through far more ethical means," I would argue that this is in fact an argument for rather than against work like Bohannon's. The argument has been made for years and still hasn't resulted in change. Perhaps it takes something like this to make people listen.

Check back in next week when we'll look at how you can defend yourself against bad science.

Follow Neurobonkers on Twitter, Facebook, Google+, RSS, or join the mailing list to get each week's post straight to your inbox.