An easy way to solve the problem of garbage in scientific journals

Prestige hierarchies hold a strong grip on life and business in academia. A small number of universities and departments dominate the academic world. In fact, most universities perceive their own PhD graduates as being not good enough to hire and instead lust after those from more renowned institutions. 

Does a similar bias permeate peer review, an important process for evaluating whether research is good enough to be accepted into the canon of scientific knowledge? According to a 2023 study investigating bias in peer review, the answer isn’t quite what you’d expect. But the research does demonstrate a potential way to alleviate the problem.

Reviewing peer review

Before we get to the research, here is peer review in a nutshell. 

Researchers perform a study and write a formal report on their methods, analysis, and results. They submit this paper to an academic journal where the editors likely will reject it on the spot. If it stands among the best, however, the editors send it out for peer review, an intensive critique by a group of anonymous experts in the field. Many papers are rejected outright by these reviewers, too. The rest are sent back to the authors with detailed requests for revisions and additional work. Being asked to revise and send the paper back is good news; quality editing at this point often leads to the study being accepted and published.

Traditionally, the peer review process is single-blind — that is, the reviewers are anonymous but the authors are not. This means reviewers can see the authors’ names and their academic affiliations. Conversely, a double-blind review makes both parties anonymous to each other.

Turning a double-blind eye

Back to the research: The journal Functional Ecology assigned nearly 3,700 new manuscript submissions to either single- or double-blind peer review. Most of the papers were rejected (79% in aggregate). The double-blind papers were, on the whole, 15% less likely than the single-blind papers to receive good news leading to possible acceptance. The journal’s standards rose under double-blind review, as well.

The researchers then compared the results of single- and double-blind peer reviews, looking for statistical correlations with the gender, geographic location, and language of the authors. Immediately, it was clear that women did slightly better in all cases. Papers by women were more likely to be picked by editors for a full peer review. Women’s review scores were likely to be a few percent better than men’s, and their papers were ultimately more likely to receive a good news result. Single- or double-blinding did not affect this particular (and small) bias.

The study then classified the authors based on their country’s socioeconomic status. Countries with a very high Human Development Index (HDI) score include about 65 countries, such as the U.S., most of Europe, and parts of Asia, Oceania, and South America. Notably, China, India, Brazil, and Indonesia fall below the line.

Under single-blind review, papers from the top HDI countries were likely to receive better reviews and 28% more likely to be invited for resubmission after editing. When the review was double-blinded, this difference disappeared almost entirely. Double-blind review outcomes were the same whether the research came from a country with a very high HDI or a lower one. 

A similar phenomenon appeared when the authors compared the results of single- and double-blind reviews on studies submitted by authors in countries whose primary language was English compared to those with other mother tongues.

Here, however, lies a lesson in statistics: When the study applied a statistical covariate model that considered both very high HDI and the English language as variables, English became an insignificant factor while HDI remained very significant. Hence, it seems likely that whether a country is predominantly English-speaking was not important. The effect is likely driven by HDI, and it happens that English-speaking countries generally have a very high HDI.

Single-blinding didn’t seem to make reviewers negatively biased against authors from lower HDI countries. Rather, it appeared to make reviewers think more highly of authors from very high HDI countries. Double-blinding didn’t affect papers written by authors in lower HDI countries, but it did significantly lower the reviews and outcomes for authors in very high HDI countries.

A prestige bias?

At this point, it’s natural to ask whether this happens because the reviewers know the specific researcher or, at least, the prestige of their university or institution. The study raises this possibility but cannot answer it. Still, it’s a compelling hypothesis (or at least speculation). Just as graduates from a handful of universities dominate faculty hiring, papers by famous authors or those from prestigious universities may be treated with significant positive bias. After all, most of those famous authors and prestigious universities are in very high HDI nations.

It’s also important to acknowledge the limitations of statistical correlation studies. As the threadbare but true saying goes, correlation does not imply causation. As we saw, a strong statistical correlation (English) became insignificant once the variable of HDI was taken into consideration. It’s entirely possible that further variables, known or unknown, drive these results.

Still, double-blinding the peer review process appears to have clear benefits. It ups the quality standards of more prestigious institutions — the “haves” — in more developed countries without harming the less prestigious “have-nots.” What’s more, it accomplishes this without explicitly picking or favoring one group over the other. It’s truly blind.