No matter how you interpret the data, dark energy remains

When it comes to the Universe, it’s easy to make the incorrect assumption that what we see is an accurate reflection of all that’s out there. Certainly, what we observe to be out there really is present, but there’s always the possibility that there’s far more out there that’s unobservable. That extends to radiation outside of the visible light spectrum, matter that neither emits nor absorbs light, black holes, neutrinos, and even more exotic forms of energy. If something truly exists in this Universe and carries energy, it will have non-negligible effects on quantities that we can actually observe, and from those observations, we can go back and infer what’s truly there. But there’s a danger: perhaps our inferences are incorrect because we’re fooling ourselves somehow. Could that be a legitimate worry for dark energy? That’s what this week’s question-asker, Bud Christenson, wants to know:

“As one who has studied physics, I have been able to wrap my brain around some ideas that at one time were considered crazy… But dark energy is the most whacked idea I’ve heard. I know I’m not the sharpest knife in the drawer, and not getting any smarter as I age. But if so many of you are convinced that this intuitively impossible idea is valid, maybe I need to investigate instead of rejecting it out of hand.”

Regardless of our estimation of the way the Universe ought to be, all we can do is observe it as it is, and draw our conclusions based on what the Universe tells us about itself. Let’s go back to the very beginning when it comes to dark energy and see what we learn for ourselves.

There is a large suite of scientific evidence that supports the expanding Universe and the Big Bang. At every moment throughout our cosmic history for the first several billion years, the expansion rate and the total energy density balanced precisely, enabling our Universe to persist and form complex structures. Today, dark energy dominates the Universe, while early on, prior to the onset of the hot Big Bang, a phase of cosmological inflation occurred, preceding it and setting it up.

Our Universe — at least as we know it — began about 13.8 billion years ago with the hot Big Bang. At this early stage, it was:

• extremely hot,
• extremely dense,
• extremely uniform,
• filled with every allowable form of energy that could exist,
• and expanding at an extremely rapid pace.

All of these properties are important, as they all affect not only one another, but the evolution of the Universe itself.

The Universe is hot because of the amount of energy inherent to each particle. Just as if you heat a liquid or gas up, the particles it’s made up of move more quickly and more energetically, the particles in the early Universe take this to an extreme: moving at speeds indistinguishable from the speed of light. They collide with one another, spontaneously creating particle-antiparticle pairs in every allowable permutation, leading to a veritable zoo of particles. Every particle and antiparticle allowed in the Standard Model, as well as any other yet unknown particles that may exist, did exist in copious quantities.

This simplified animation shows how light redshifts and how distances between unbound objects change over time in the expanding Universe. Note that the objects start off closer than the amount of time it takes light to travel between them, the light redshifts due to the expansion of space, and the two galaxies wind up much farther apart than the light-travel path taken by the photon exchanged between them.

But this hot, dense, almost-perfectly-uniform Universe wouldn’t remain this way forever. With so much energy in such a small volume of space, the Universe absolutely must have been expanding at an incredibly rapid rate at these early times. You see, there’s a relationship in general relativity, for a largely uniform Universe, between how spacetime evolves — expanding or contracting — and all the combined matter, radiation, and other forms of energy present within it.

If the expansion rate is too small for the stuff within it, the Universe will rapidly recollapse. If the expansion rate is too large for the stuff within it, the Universe rapidly dilutes so that no two particles will ever find one another. Only if the Universe is “just right,” — and I hope you’re saying “just right” the way you would when you tell the tale of Goldilocks and the Three Bears — can the Universe expand, cool, form complex entities, and persist with interesting structures within it for billions of years. If our Universe, at the earliest stages of the hot Big Bang, were just a tiny bit denser or just a tiny bit less dense, or conversely expanded just a tiny bit more or less quickly, our own existence would have been a physical impossibility.

If the Universe had just a slightly higher matter density (red), it would be closed and have recollapsed already; if it had just a slightly lower density (and negative curvature), it would have expanded much faster and become much larger. The Big Bang, on its own, offers no explanation as to why the initial expansion rate at the moment of the Universe’s birth balances the total energy density so perfectly, leaving no room for spatial curvature at all and a perfectly flat Universe. In regions that are overdense, the expansion can be overcome.

As the Universe expands, however, a number of things evolve.

• The temperature drops, as the wavelength of any photons traveling through the Universe get stretched along with the expansion of space.
• The density drops, as any species of energy that’s quantized into a fixed number of particles will see the volume expand while the number of particles remains constant.
• The types of particles that exist simplify, as all of the massive, unstable particles (and antiparticles) in the Standard Model require large amounts of energy to create them — via E = mc² — and once there’s no longer enough energy present, they simply annihilate away with their antimatter counterparts.
• The level of uniformity drops, as all of the forces in the Universe push and pull on the various forms of matter and energy within them, leading to the growth of gravitational imperfections and, eventually, a cosmic web of large-scale structure.
• And the expansion rate itself also evolves, since that rate is directly related to the total energy density of the Universe; if the density drops, the expansion rate must drop as well.

The law of gravity, general relativity, is so thoroughly well-understood that if you could measure what the expansion rate is today and you could determine what all the different forms of matter and energy in the Universe are, you could calculate precisely what the size, scale, temperature, density, and expansion rate of the observable Universe was at every point throughout our cosmic history, and what those quantities will be at any point in the future.

While matter and radiation become less dense as the Universe expands owing to its increasing volume, dark energy is a form of energy inherent to space itself. As new space gets created in the expanding Universe, the dark energy density remains constant.

The reason we can do this is simple: if we can understand what’s in the Universe, and we understand how the expansion (or contraction) of the Universe affects what’s in it, and how those changes in turn cause the expansion rate to change, we can learn precisely how any type of matter, radiation, or energy will evolve along with the separation scale between any two points in the Universe. Some cases of note include:

• normal matter, which drops as the inverse of the scale of the Universe to the third power (as the volume of our three-dimensional Universe grows),
• radiation, like photons or gravitational waves, which drops as the scale factor to the negative fourth power (as the number of quanta dilutes and as the wavelength of each quantum gets stretched by the expanding Universe),
• dark matter (which behaves identically to normal matter in this regard),
• neutrinos (which behave as radiation when things are very hot and as matter when things are cold),
• spatial curvature (which dilutes as the inverse second power of the scale of the Universe),
• and a cosmological constant (which has a constant energy density everywhere in space, and remains the same regardless of the expansion or contraction of the Universe).

Components of the Universe that dilute the most quickly are the most important early on, while components that dilute more slowly (or not at all) will require that more time passes before their effects can be observed, but then — if they exist — they’ll be the ones that become dominant.

Various components of and contributors to the Universe’s energy density, and when they might dominate. Note that radiation is dominant over matter for roughly the first 9,000 years, then matter dominates, and finally, a cosmological constant emerges. (The others, like cosmic strings and domain walls, do not appear to exist in appreciable amounts.) However, dark energy may not be a cosmological constant, exactly, but may still vary with time by up to ~4% or so. Future observations will constrain this further.

Although this framework is incredibly powerful, we have to take extraordinary care to ensure that we’re letting the observations guide us, and that when they come in, we don’t let ourselves be fooled by what they say. As the Universe expands, for example, the light emitted by a distant galaxy gets stretched to longer, redder wavelengths, and so it appears red by the time it reaches our eyes. But the light from intrinsically redder (as opposed to bluer) objects is also red. The light from an object speeding away from us is also shifted toward the red. And the light from an object obscured by dust will also appear preferentially reddened compared to an identical object located along a dust-free line of sight.

The way we try and account for these sorts of errors are threefold.

1. We demand multiple, independent lines of evidence when drawing a conclusion about the Universe, so that even an unidentified error with any particular set of objects won’t bias us toward an incorrect conclusion.
2. We do our best to identify every conceivable source of error or uncertainty and to quantify it, so that we can study each aspect of every phenomenon that might affect our inferred results and what they mean.
3. And we concoct alternative possibilities for everything that we observe, so that we can perform independent tests of these various hypothetical ideas to see which ones can be ruled out and which ones still remain valid.

So far, this has proven to be a tremendously successful approach.

This graph shows the 1550 supernovae that are a part of the Pantheon+ analysis, plotted as a function of magnitude versus redshift. The supernova data, for many decades now (ever since 1998), has pointed toward a Universe that expands in a particular fashion that requires something beyond matter, radiation, and/or spatial curvature: a new form of energy that drives the expansion, known as dark energy. The supernovae all fall along the line that our standard cosmological model predicts, with even the highest-redshift, most far-flung type Ia supernovae adhering to this simple relation. Calibrating the relation without substantial error is of paramount importance.

We’ve known for a long time that our Universe must contain matter and radiation, but we’d often wondered if that was all there was. Could there be exotic forms of energy out there: topological defects like monopoles, cosmic strings, domain walls, or textures? Could there be a cosmological constant, or perhaps some type of dynamical field? And would all those forms of energy add up to a certain critical value determined by the expansion rate, exactly, or would there be a mismatch, meaning that there was (either positive or negative) spatial curvature to the Universe? Without sufficiently accurate and compelling data, many viable possibilities remained on the table.

Throughout the 1990s, multiple teams working with the best ground-based telescopes at their disposal set out to measure the most distant, brightest objects in the Universe that always displayed regular, known brightness properties: type Ia supernovae, triggered when massive white dwarf stars explode. In 1998, enough supernovae had been built up at a variety of distances and with quantifiably observed redshifts that two independent teams noticed something remarkable: these explosions were appearing fainter than they ought to from beyond a certain distance.

It was possible that there was something other than matter and radiation in the Universe, stretching the light from these supernovae by more than the expected amount, and pushing them out to greater distances than if the Universe were populated with matter and energy alone.

Given enough time, light that was emitted by a distant object will arrive at our eyes, even in an expanding universe. However, if a distant galaxy’s recession speed reaches and remains above the speed of light, we can never reach it, even if we can receive light from its distant past.

But there were other possible explanations for why these supernovae would appear fainter-than-expected besides having an unexpected composition to the Universe’s energy budget. It could be that:

• these supernovae, thought to be the same everywhere, were actually evolving with time, causing the recent ones and the ancient, distant ones to have different properties,
• that the supernovae wasn’t evolving, but their environments were, and that was affecting the light,
• that there was dust polluting some of the more distant supernovae, and that was causing them to appear fainter than they actually were by blocking a portion of their light,
• or that there was some non-zero probability that these distant photons were oscillating into some other sort of invisible particle, like axions, causing distant supernovae to appear fainter.

So either there’s some effect at play that’s the cause of these distant objects appearing as though the Universe has expanded by a greater amount than we’d otherwise expect, or there’s some sort of alternative scenario at play.

Thankfully, we have ways to test these ideas against one another, and see which one fits not just the supernova data, but all the data together.

A plot of the apparent expansion rate (y-axis) vs. distance (x-axis) is consistent with a Universe that expanded faster in the past, but where distant galaxies are accelerating in their recession today. This is a modern version of, extending thousands of times farther than, Hubble’s original work. Note the fact that the points do not form a straight line, indicating the expansion rate’s change over time. The fact that the Universe follows the curve it does is indicative of the presence, and late-time dominance, of dark energy.

It didn’t take long to rule out supernovae evolving or their environments evolving; the physics of atom-based matter is very sensitive to these scenarios. Photon-axion oscillations were ruled out by detailed observations of light coming from different distances; we could see these oscillations were not present. And the changes in the light occurred equally across all wavelengths, ruling out the dust possibility. In fact, an unrealistic type of dust — grey dust, which would absorb light equally across all wavelengths — was also tested to such great precision until it, too, could be observationally ruled out.

Not only did the addition of a cosmological constant fit the data incredibly well, but completely independent lines of evidence also pointed to the same conclusion. We have:

• other objects to look at besides supernovae at great distances, and although they go out reliably less far and have greater uncertainties to them, they also appear fainter at great distances, as though they’ve been moved to greater distances than a matter-only Universe would indicate,
• the large-scale structure of the Universe, which indicates that the Universe is filled with only about ~30% matter and a negligible amount of radiation,
• and the temperature fluctuations in the cosmic microwave background, which place tight constraints on the total amount of stuff, indicating that the Universe is spatially flat so that the total amount of energy is ~100% of the critical density.

Three different types of measurements, distant stars and galaxies, the large-scale structure of the Universe, and the fluctuations in the CMB, tell us the expansion history of the Universe and its composition. Constraints on the total matter content (normal+dark, x-axis) and dark energy density (y-axis) from three independent sources: supernovae, the CMB (cosmic microwave background) and BAO (which is a wiggly feature seen in the correlations of large-scale structure).

By the early 2000s, it became clear that even if you were to ignore the supernova data entirely, you would still be compelled to conclude that there was an extra type of energy present within the Universe that comprised this “missing” ~70% or so, and that it had to behave in such a way that it was causing distant objects to have a redshift that increased over time, rather than decreased as expected in a Universe without some form of dark energy.

Although the evidence that dark energy behaved as a cosmological constant had initially large uncertainties, by the mid-2000s that was down to ±30%, by the early 2010s it was ±12%, and today it’s down to ±7%. Whatever dark energy is, it sure does look very much like its energy density remains constant in time.

An illustration of how the radiation (red), neutrino (dashed), matter (blue), and dark energy (dotted) densities change over time. In a new model proposed some years ago, dark energy would be replaced by the solid black curve, which is heretofore indistinguishable, observationally, from the dark energy we presume. As of 2023 in an expanding Universe, dark energy can deviate from a “constant” by about ~7% in the equation of state; any more is too tightly constrained by the data.

In the near future, observatories like the ESA’s Euclid, the NSF’s Vera Rubin Observatory, and NASA’s Nancy Roman Observatory will improve that uncertainty so that if dark energy departs from a constant by as little as ~1-2%, we’ll be able to detect it. If it strengthens or weakens over time, or varies in different directions, it would be a revolutionary new indicator that dark energy is even more exotic than we currently think.

Sure, the idea of a novel form of energy inherent to the fabric of space itself — what we know today as dark energy — is a wild one, nobody doubts that. But is it truly wild enough to explain the Universe that we have? The only way we’ll learn is to keep asking the Universe questions about itself, and listening to what it tells us. That’s how good science is done, and in the end, our best hope for learning the truth of our reality.

Ethan is on medical leave until May 6th. Please enjoy a republication of this article from the Starts With A Bang archives!