The good reasons scientists are so hostile to new ideas

Every few months, a novel headline will fly across the world, claiming to revolutionize one or more of our most deeply held scientific ideas. The declarations are always sweeping and revolutionary, ranging from “The Big Bang never happened” to “This idea does away with dark matter and dark energy” to “Black holes aren’t real” to “Maybe this unexpected astronomical phenomenon is due to aliens.” And yet, despite the glowing coverage of the novel proposal, it most frequently languishes in obscurity, attracting little mainstream attention other than a myriad of dismissals.

Commonly, it’s portrayed that scientists in this particular field are dogmatic, wedded to old ideas, and close-minded. This narrative might be popular among contrarian scientists or those who themselves hold fringe beliefs, but it paints a disingenuous picture of the scientific truth. In reality, the evidence supporting the prevailing theories are overwhelming, and the new headline-grabbing proposals are no more compelling than the scientist’s equivalent of playing in the sandbox. Here are the four big flaws that commonly occur with new ideas, and why you’ll never hear about most of them again after they’re first put forth.

A visual history of the expanding Universe includes the hot, dense state known as the Big Bang and the growth and formation of structure subsequently. The full suite of data, including the observations of the light elements and the cosmic microwave background, leaves only the Big Bang as a valid explanation for all we see. As the Universe expands, it also cools, enabling ions, neutral atoms, and eventually molecules, gas clouds, stars, and finally galaxies to form.

1.) When you work, every day, with “the real McCoy,” you can immediately spot an impostor’s shortcomings. In science, we have accumulated an enormous body of knowledge — a set of experimental and observational data — and a set of theories that provides a framework to accurately describe the governing rules of our reality. Many of the results that we obtained were initially bizarre and counterintuitive, with multiple theoretical possibilities proposed to explain them. Over time, further experiments and observations winnowed them down, and the most successful theories with the greatest degrees of validity were the ones that survived.

Proposals that attempt to revolutionize one (or more) of our accepted theories have a large suite of hurdles to overcome. In particular, they must:

• reproduce all the successes of the prevailing theory,
• explain a phenomenon more successfully than the current theory can,
• and make novel predictions that can be tested that differ from the theory it’s attempting to supersede.

It’s very rare that all three of these criteria are met. In fact, the overwhelming majority of these grand proposals fail on even the first point.

The Sun’s actual light (yellow curve, left) versus a perfect blackbody (in gray), showing that the Sun is more of a series of blackbodies due to the thickness of its photosphere; at right is the actual perfect blackbody of the CMB as measured by the COBE satellite. Note that the “error bars” on the right are an astounding 400 sigma. The agreement between theory and observation here is historic, and the peak of the observed spectrum determines the leftover temperature of the Cosmic Microwave Background: 2.73 K.

Attempts to explain the Universe without a hot Big Bang fail to explain the existence and properties of the Cosmic Microwave Background: an omnidirectional pattern of radiation that’s been known for some 55+ years. Contentions that gravitational wave detectors are seeing noise, rather than signals, ignore the large suite of evidence that links electromagnetically observed events with their gravitational wave counterparts. And the idea that gravity might emerge from another entity, like entropy, yield absurd results for the problem of dark matter, failing to maintain the necessarily constant ratio of dark matter to normal matter.

It’s not enough, by scientific standards, to simply propose a wild idea that explains one property that the prevailing, presently accepted theory has difficulty with. One new observation can always be explained by one new “free parameter,” which is a kindhearted way to say “invoking something brand new.” If that new theoretical addition lacks the power to explain other phenomena as well, however, it’s not likely to gain serious traction of any type.

The proton isn’t just made of three valence quarks, but rather contains a substructure that is an intricate and dynamic system of quarks (and antiquarks) and gluons inside. The nuclear force acts like a spring, with negligible force when unstretched but large, attractive forces when stretched to large distances. To the best of our understanding, the proton is a truly stable particle, and has never been observed to decay, while the quarks and gluons composing it show no evidence of compositeness.

2.) Many “new ideas” are unoriginal repackagings of old, discredited ideas that don’t merit reconsideration. Most of us, if we have any sort of imagination at all, have played the “what if” game about some aspect of reality at some point. Perhaps you’ve wondered about this yourself, and have had ideas such as:

• What if you traveled in a straight line through the Universe for a long enough distance; would you ever come back to your starting point?
• What if the particles that we think of as fundamental today — quarks, electrons, photons, etc. — are actually composite particles made up of more fundamental components?
• What if there’s some sort of extra, new field in the Universe that permeates all of space, and that’s the explanation behind what we’re currently calling “dark matter” and “dark energy?”

All of these ideas are good ideas. There are many papers that have been written about them, and explored them in great detail.

In a hypertorus model of the Universe, motion in a straight line will return you to your original location, even in an uncurved (flat) spacetime. Without access to a higher-dimensional view of what our 3D world appears to be like to us, we cannot know or measure its true extent and shape in space.

But each and every one of them has difficulties that led to them being abandoned, and no new evidence has come in to favor them over the prevailing theories. For example, the idea that the Universe might have a non-trivial topology continues to be interesting, but if it does, the evidence demonstrates that whatever the “size” of the Universe is, it must be significantly larger than the entire observable Universe. If any of our fundamental particles are composite particles, they don’t exhibit that behavior under any of the experimental conditions we’ve ever probed.

And if there is no dark matter or dark energy, but rather a field explanation instead, then that explanation requires at least two new free parameters: a “clumpy” one that behaves like dark matter, and a “smooth” one that behaves like dark energy. You don’t gain anything by these reformulations, and in many cases, you’ve just added more complexity to explain a puzzle in an inferior fashion. There’s no reason you can’t explore these avenues, but unless you can either explain something the prevailing theory cannot or you can reduce the number of free parameters required by your theory, you haven’t done anything more than playing in the sandbox.

Perhaps the most famous depiction of the ‘Creation of Adam,’ from the ceiling of the Sistine Chapel. Although this might be a fascinating metaphorical story, we have ample evidence that indicates this is a picture at odds with what science understands today.

3.) It’s fundamentally unscientific to begin with an ideologically motivated conclusion. This is one of the most dangerous pitfalls that scientists — particularly young and inexperienced scientists — can fall into. If you have a puzzle or problem that vexes or fascinates you, you might have a thought along the lines of, “Wouldn’t it be fascinating if ____________ explained what we were seeing?” There’s absolutely nothing wrong with having that thought, and there isn’t even anything wrong with exploring the theoretical consequences of what your idea would imply for the Universe we have the capacity to observe.

But there’s a line that, once you cross it, pushes you over the line from legitimate scientist into crackpot territory: when you become convinced that your idea must be correct. As soon as you take that leap, you’ve decided that “I know what the conclusion is,” and that means you’ll fiddle with your theory until it gives you the conclusion you know you need to reach. This type of model-building-by-working-backward might give you the result you want, but it won’t be a scientific result.

Niels Bohr and Albert Einstein, discussing a great many topics in the home of Paul Ehrenfest in 1925. The Bohr-Einstein debates were one of the most influential occurrences during the development of quantum mechanics. Today, Bohr is best known for his quantum contributions, but Einstein is better-known for his contributions to relativity and mass-energy equivalence. Both were known for thinking long and hard about the most difficult puzzles the Universe had to offer.

Many scientists have fallen prey to this pitfall. Fred Hoyle became convinced that the Universe must be in a steady state and could not have a hot, dense origin, despite the overwhelming evidence supporting the Big Bang. Arthur Eddington was convinced that the stars in the Universe could never achieve properties beyond certain limits, despite the observational evidence that those limits were frequently exceeded. Even Einstein himself became convinced that quantum “randomness” must have a deterministic explanation and that gravity and classical electromagnetism would lead to a unified force; these avenues produced no consequential results over the last 20+ years of Einstein’s scientific life.

In many ways, these influential scientists held back progress in their field substantially until their deaths, with the lesson being that your physical intuition — regardless of who you are or what you’ve accomplished — is no substitute for the legitimate information that we gain by asking the Universe questions about itself. It’s why Johannes Kepler, who threw away his “beautiful” theory of nested spheres and perfect solids for the “ugly” theory of elliptical orbits that fit the data better than any other, remains such a spectacular role model for how to do science right.

Tycho Brahe conducted some of the best observations of Mars prior to the invention of the telescope, and Kepler’s work largely leveraged that data. Here, Brahe’s observations of Mars’s orbit, particularly during retrograde episodes, provided an exquisite confirmation of Kepler’s elliptical orbit theory. Kepler put forth his 1st and 2nd laws of planetary motion in 1609, with his 3rd law coming 10 years later: in 1619.

4.) A scientist’s job is to rigorously attack their own hypothesis, and “new idea proponents” often fail to do that very job. Did you have an idea, and did you fall in love with it? Many of us do, and this is a tremendous problem for us. In science, it falls to us to be the harshest critics of our own ideas, as we will be the first to explore them in-depth, before we present our findings to the world where they will be evaluated by others. If you fail in the attempt to knock down your own ideas — to find its weak points, to expose where its range of validity ends, to identify where it compares unfavorably with the theory it seeks to replace — others will do that work for you.

That is not cruelty. That is not close-mindedness. And that is certainly not adherence to dogma. That is a necessary part of science: to subject any new hypothesis to rigorous scrutiny and evaluation. Though it may be unfortunate, most “new ideas” will fall apart under the weight of the evidence that has already been collected, just as most of the ideas that are originally proposed to explain a novel phenomenon turn out to fail spectacularly at describing the full suite of evidence that the Universe provides.

Compared with a number of other known objects with Solar System origins, the interstellar objects 1I/’Oumuamua and 2I/Borisov appear very different from one another. Borisov fits in extremely well with comet-like objects, while ‘Oumuamua is completely depleted. Discovering why is a task that still awaits humanity, but it’s almost certainly not because it’s an alien probe.

It’s easy to understand why, if you have an idea that you love, you’d want others to love it, too. But it’s very difficult to convince other scientists — particularly the scientists who embrace the notion of having appropriate levels of skepticism for ideas — that your idea is worth loving if you haven’t subjected it to the necessary scrutiny. If you want to propose a theory where the speed of light is different for different wavelengths of light, it had better not disagree with any of the multiwavelength observations we’ve already collected about light from distant objects, for example.

If you have an idea that falls outside of the mainstream, there are some questions you’ll definitely want to ask.

• What is the problem you’re considering that motivated this idea?
• How does this idea compare to the prevailing theory when applied to this specific phenomenon?
• How does this idea compare to the prevailing theory when applied to the other major successes of the prevailing theory?
• And what are some critical tests that you can legitimately perform (with current or near-future technology) to further discern your idea versus the prevailing theory?

As Richard Feynman once put it so eloquently, “The first principle is that you must not fool yourself — and you are the easiest person to fool.”

On the largest scales, the way galaxies cluster together observationally (blue and purple) cannot be matched by simulations (red) unless dark matter is included. Although there are ways to reproduce this type of structure without specifically including dark matter, such as by adding a specific type of field, those alternatives either look suspiciously indistinguishable to dark matter or fail to reproduce one of the many other observations in support of dark matter.

It is not an act of cruelty, dogmatism, or close-mindedness to demand scientific rigor. Instead, it is a mark of integrity and a commitment to finding the scientific truth surrounding whatever issue or phenomenon you’re investigating. There are many great, brilliant ideas that have been relegated to the historical dustbin of failed theories for the best reason of all: because they did not successfully line up with our observed reality. No matter how fanciful or compelling an idea may be, if it disagrees with experiment, measurement, and observation, it’s wrong.

There are plenty of compelling, interesting, and viable ideas that are out there, and there will always be plenty of room for speculation about the unknown. But whenever we consider a novel, alternative idea, we have to do it through the lens of scientific rigor. We cannot simply pick-and-choose the phenomena we wish to pay attention to while ignoring the aspects of reality that are inconvenient for our pet ideas.

In the end, the Universe will always be the ultimate arbiter of what is real and what theories best describe our reality. But it’s up to us — the intelligent beings that conduct the enterprise of science — to rigorously uncover those truths. Unless we do it responsibly, we run the risk of fooling ourselves into believing what we want to be true. In science, integrity and intellectual honesty are the ideals to which we must aspire.