> > Each experiment quotes a likelihood of very close to “5 sigma,” meaning the likelihood that the events were produced by chance is less than one in 3.5 million.

> These words come from Lawrence Krauss, a highly respected theoretical physicist, and they are wrong.

I don't get what is wrong with it, "likelihood that the events were produced by chance" sounds exactly like "P(D|H_0)" i.e., the probability that we see given data (or more extreme) assuming null hypothesis (no systematic error produced the data, just random chance).

Numbers also correct one-in-3.5 million is what we expect from "5 sigma" (one-tail):

    import statistics

    print(f"{1 - statistics.NormalDist().cdf(5):.3g}, {1/3.5e6:.3g}")    
    # 2.87e-07, 2.86e-07

The issue lies in the way the signal was actually measured. The first observation was in the digamma decay channel, where it basically amounted to an excess photon count over the background at a specific energy (i.e. invariant mass). So the idea of a simple "null" hypothesis was not really well defined here, because the excess could have also been at other energies. But the five sigma value was given only for the bump over background at ~125 GeV. In that specific sense, Krauss' statement is very much correct, but the overall hypothesis of whether H0 exists - yes or no - given arbitrary data is a bit more complicated when doing the statistics rigorously. That's also why you really want 5 sigmas, because as you widen the parameter space you're searching in, even 3 sigma results (= less than half a percent chance of statistical error) at specific energies regularly turn out to be flukes.

For those curious, this is often described as the look-elsewhere effect.

https://en.wikipedia.org/wiki/Look-elsewhere_effect

I seem to recall that they didn't declare victory until it hit 6 sigma.

It always tickles me that a hedge fund is called Two Sigma. Who would want that on their CV?

For reference, 6 sigma is one-in-a-billion case (one-tail):

    print(f"{1 - statistics.NormalDist().cdf(6):.1g}")
    # 1e-09

and 2 sigma is two-in-a-hundred (~2%). ~5% for two-tail case.

Sigmas do not protect you from systematic errors in your measurements, right?

correct.

I don't see where Lawrence Krauss claims otherwise.

I always bring this up in discussions with social scientists and economists who maintain 99% or 95% significance level as the standard above which results from their studies get accepted. It's ridiculous; we can expect 1 or 5 in a hundred studies to be wrong just from statistics.

Even 5% is much better than these people would have you believe. There are fields where 70-80% of published research is not reproducible.

https://en.m.wikipedia.org/wiki/Replication_crisis

Author discusses precision, doesn't get to the point. There's a joke in there somewhere.

Good article. Feels like it has been written by a Bayesian though:)