Most New Psychology Findings Can’t Be Replicated. So Now What?

The field of psychology has been shaken by a massive replication effort, which has found that out of 98 papers published in the top three psychology journals only 39 could be replicated.

From much of the coverage you’d be forgiven for thinking that this means we should discard psychology as a field. I certainly don’t believe this is the case for the simple reason that well-studied areas of psychology are full of findings that have been repeatedly tested in various different ways. Instead of resorting to cynicism, we should take this opportunity to embrace a culture of skepticism and fundamentally change the way we interpret new research findings.

Whenever you hear the words “new study,” alarm bells should ring. It isn’t new studies that you should base your opinions on; it is old studies that have been replicated again and again, and the results reported in meta-analyses and systematic reviews.

… taking a scientifically informed view of the world requires a great deal more than looking at any one study.

In that last sentence I was careful not to restrict that claim to psychology, for the simple reason that we have every indication to believe that new research is similarly difficult to replicate across many areas of science. For example, 30 percent of the most widely cited randomized controlled trials in the world’s highest-quality medical journals have later been found to be wrong or exaggerated and that number rises to five out of six for non-randomized trials — a number that is in fact worse than the rate found by the psychology reproducibility project.

So what does this mean to you, the reader, who is simply trying to distinguish fact from fiction? Obviously, it is the responsibility of researchers to make their research considerably more reliable, but this may take a very long time, and may never happen.

The solution is to take a skeptical approach to the world around us, to treat every new claim not as a problem solved, but as an open question. Questions that are best answered by looking with a critical eye at all the evidence we have. In a perfect world, I’d wholly agree with psychologist Steven Pinker that much of this responsibility lies with science journalists:

I’ve long argued that sci journalists should stop reporting single newsworthy studies = recipe for error. Only meta-analyses, lit reviews.

— Steven Pinker (@sapinker) August 30, 2015

But this is the real world, where this is unlikely to ever happen whilst the dominant forces in the news business are clicks and sales. Sexy findings sell newspapers for the same reasons that they get published in top journals. In the increasingly cutthroat age of digital journalism, few journalists outside specialist publications have the time or inclination to critically assess new findings or place them in appropriate context. 

So what are you to do? The first step is to learn the basics of how to evaluate research critically for yourself. The next step is to seek out informed, critical voices who can present balanced and evidence-based analysis of new research. If the publication you are reading reports a study as fact without offering any criticisms or bothering to reference any other studies, seek out an alternative.

The simplest way to avoid being misled is to stop getting your psychology and science news from, well, the news. Being the author of a skeptical psychology blog, I am of course biased, but for those of us who follow science critically, these new psychology replication failures came as no surprise. They represent the culmination of plenty of the things that we have long been saying are endemic problems including over-reliance on statistical significance, the prevalence of p-hacking, the ubiquity of the file drawer problem, and the issue of underpowered research.

When it comes to psychology and science, the news is simply very often the single worst place to get informed. There are far better places where you can read informed perspectives, where claims are placed in the context of the literature rather than treated as standalone nuggets of truth. At the top of any list should be specialist publications that employ experts to evaluate new findings against what we already know. The British Psychological Society's The Psychologist magazine looks at the breadth of research on various issues, rather than focusing on breaking news. For a balanced view on health news more generally, check out Behind The Headlines, a feed of critical commentary on health news from the UK’s National Health Service. Another reliable destination is The Conversation where critical commentary on science news is written exclusively by academics. Similarly, Health News Review and Research The Headlines are great places to gain a skeptical perspective on the headlines informed by experts.

The skeptical bloggersphere is another fabulous place for informed commentary. A few of my favorites are: Neuroskeptic, a pseudonymous neuroscientist who offers a skeptical look at his field. The Mind Hacks blog, where psychology and neuroscience news is discussed by neuroscientist and psychologist Vaughan Bell and cognitive scientist Tom Stafford. Another reliable science writer is Ed Yong who offers incisive commentary on science news, variously at Nature, National Geographic, The Scientist and now The Atlantic. His article on the replication effort is not to be missed. It is this kind of critical, informed commentary that we need more of; anything less and you are simply wasting your time.

How can you keep on top of reliable sources? Twitter is your friend. Alternatively, simply pop their URL into any RSS reader such as Feedly and next time you are waiting for a bus, instead of reading yet another dubious headline that your friend posted on Facebook, you can simply open an app and get the full story from sources you can trust to examine the bigger picture.

If you only remember one thing from this, make it this. Science is messy and studies get things wrong all the time. That doesn't necessarily mean science is broken; it means taking a scientifically informed view of the world requires a great deal more than looking at any one study.

Follow Neurobonkers on Twitter, Facebook, Google+, RSS, or join the mailing list.