How to fool the world with bad science

Did you hear about the EmDrive, the “impossible space engine?” Here are the red flags you should have looked for!

Image credit: © 2012 — Twentieth Century Fox Film Corporation.

“You can’t cheat an honest man. Never give a sucker an even break or smarten up a chump.” –W.C. Fields

You’ve no doubt heard by now about the EmDrive, the impossible space engine that runs off of microwave power reflected inside a cavity, creating thrust on the outside without having any mass or energy escape. Sure, it violates the known laws of physics, but at its core, physics is an experimental science, and so if the experiments are producing robust results that fly in the face of what we think we know, it’s the theory that has some explaining to do, not the experiment.

Unless, of course, there are legitimate problems with the experiment, whether that’s due to fraud, carelessness, malfunctioning equipment or a whole host of other errors. We have a saying in science — extraordinary claims require extraordinary evidence — and what we mean by that is the following:

• When an experiment gives a result that your theory didn’t predict or admit, you must be extremely vigilant to ensure that your results are not in error.
• That means you must account for and quantify all sources of uncertainty, including statistical errors (which depend on how many data points you have) and systematic errors (which are inherent to your setup).
• It is important and necessary to independently verify and reproduce these results, meaning that if you detail the setup and methods, anyone else can achieve these results for themselves with the proper equipment.

Even then, you must be very careful not to fool yourself, which is often an easy thing to do, particularly if you have a stake in the outcome turning out a certain way. Historically, there’s an example of this you may not have heard of that goes back a little more than 100 years: N-rays.

You’ve heard of X-rays, no doubt: high-energy electromagnetic radiation that can pass directly through human skin-and-flesh, but not bone. Back in the early 1900s, many physicists were performing experiments on light of various wavelengths, trying to understand its behavior and properties in a variety of situations. One such study attempted to polarize X-rays, setting up a spark gap as part of the experiment. The way a spark gap works is simply that you have two electrodes separated by a certain distance, and when the voltage between them exceeds the breakdown voltage of the medium between them (including air), you get a spark that “jumps” the gap.

What experimenters found was a very strange and unexpected result: when they placed the gap in an X-ray beam, the brightness of the spark increased for no discernible reason. The experimenters attributed this effect to a hitherto undiscovered form of radiation named N-rays, for the University of Nancy where they were discovered.

Now, 120 scientists managed to detect these effects, although there were a few peculiarities that some of the more astute observers began to notice among these publications.

1. The magnitude of these effects varied tremendously from experiment to experiment.
2. The threshold of measurement — the difference between a detection and a non-detection — was always extremely close to the actual claimed detection.
3. Many attempts at confirming the experiments by some of the leading scientists of the day, including Lord Kelvin, Heinrich Rubens and Robert Wood, all produced null results.
4. And finally, even if you restricted your data sets to the positive the experimental results, their claims were inconsistent with one another.

Yet despite the dismissals of many prominent scientists, N-ray researchers continued to claim that what they were seeing was real. Finally, the journal Nature commissioned Robert Wood to go visit Blondlot’s lab at the University of Nancy, and witness the demonstration himself.

Wood, however, had the inkling that what he was witnessing was scientific chicanery of the highest order. Before Blondlot had a chance to perform his demonstration, Wood was able to stealthily remove a prism from the experimental apparatus, and replace a metal file (a supposed emitter of N-rays) with a piece of wood, one of the materials claimed not to be an N-ray emitter.

When Blondlot performed the demonstration, however, he claimed that N-rays were being detected in the spark gap. Shortly thereafter, in September of 1904, Wood published his results in Nature, stating:

“After spending three hours or more in witnessing various experiments, I am not only unable to report a single observation which appeared to indicate the existence of the rays, but left with a very firm conviction that the few experimenters who have obtained positive results, have been in some way deluded.”

And now, we return to the present day.

So what are the claims of the EmDrive?

• A team in China, an inventor in Britain and a third person, Guido Fetta, have built three separate versions of the EmDrive. All three versions supposedly give different results varying by a factor of ~500 from one another.
• The “test” performed at NASA was sensitive to a minimum thrust threshold of about 10-to-15 microNewtons, and the “positive result” claimed detection of somewhere between 30-to-50 microNewtons of thrust.
• And finally (and most damning), there was a “true” version and a “null” version of the EmDrive that were both tested at this facility, with the anticipation that the true version would produce this thrust and that the null version wouldn’t. But both versions produced the same thrust.

From the abstract itself:

“Thrust was observed on both test articles, even though one of the test articles was designed with the expectation that it would not produce thrust. Specifically, one test article contained internal physical modifications that were designed to produce thrust, while the other did not (with the latter being referred to as the “null” test article).”

In other words, despite headlines like Space Engine Breaks Laws of Physics?, NASA validates ‘impossible’ space drive, and EmDrive Is an Engine That Breaks the Laws of Physics and Could Take Us to Mars, the EmDrive is none of those things.

It is not a space engine; it is an apparatus that does nothing different from an empty container.

It was not validated by NASA; it was shown by NASA to be indistinguishable from a ‘null’ device.

It does not break the laws of physics; it obeys them.

And the only way it could take us to Mars is in the Stone Soup way: if something else legitimately and causally took us to Mars, and we simply brought the EmDrive along for the ride.

I want to emphasize that although there are some incredibly powerful theoretical objections to the EmDrive, I am not calling this “bad science” on account of any of those objections. Like I said, physics is, by its nature, an experimental science, and one good experiment can disprove or invalidate even the most elegant of theories. On the contrary, this is bad science because:

• The results are not robust, in that they are not identically-or-similarly reproducible by different teams.
• The results are not significant, in that they are not distinguishable from a setup that should give a null result.
• And even if the results were significant (and they are not), they are too close to the minimum threshold of detection to warrant the claims of “discovery.”

So what are we left with? Every legitimate scientist’s nightmare: false information posing as science, eroding the public trust in science itself. Don’t let it! If there’s bad science going around, the only cure for it is more and better science, just as was the case for N-rays.

For more on this fiasco, see Corey Powell’s excellent piece, and the next time a story like this comes up, don’t forget these signs of bad science, and look for them before you share the next example of bad science reporting!

Leave your comments at the Starts With A Bang forum on Scienceblogs!