The method that can "prove" almost anything - James A. Smith
In 2011, a group of researchers conducted a scientific study to find an impossible result: that listening to certain songs can make you younger. Their study involved real people, truthfully reported data, and commonplace statistical analyses. So how did they do it?
The answer lies in a statistical method scientists often use to try to figure out whether their results mean something or if they’re random noise. In fact, the whole point of the music study was to point out ways this method can be misused. A famous thought experiment explains the method: there are eight cups of tea, four with the milk added first, and four with the tea added first.
A participant must determine which are which according to taste. There are 70 different ways the cups can be sorted into two groups of four, and only one is correct. So, can she taste the difference? That’s our research question. To analyze her choices, we define what’s called a null hypothesis: that she can’t distinguish the teas.
If she can’t distinguish the teas, she’ll still get the right answer 1 in 70 times by chance. 1 in 70 is roughly .014. That single number is called a p-value. In many fields, a p-value of .05 or below is considered statistically significant, meaning there’s enough evidence to reject the null hypothesis. Based on a p-value of .014, they’d rule out the null hypothesis that she can’t distinguish the teas.
Though p-values are commonly used by both researchers and journals to evaluate scientific results, they’re really confusing, even for many scientists. That’s partly because all a p-value actually tells us is the probability of getting a certain result, assuming the null hypothesis is true. So if she correctly sorts the teas, the p-value is the probability of her doing so assuming she can’t tell the difference.
But the reverse isn’t true: the p-value doesn’t tell us the probability that she can taste the difference, which is what we’re trying to find out. So if a p-value doesn’t answer the research question, why does the scientific community use it? Well, because even though a p-value doesn’t directly state the probability that the results are due to random chance, it usually gives a pretty reliable indication.
At least, it does when used correctly. And that’s where many researchers, and even whole fields, have run into trouble. Most real studies are more complex than the tea experiment. Scientists can test their research question in multiple ways, and some of these tests might produce a statistically significant result, while others don’t.
It might seem like a good idea to test every possibility. But it’s not, because with each additional test, the chance of a false positive increases. Searching for a low p-value, and then presenting only that analysis, is often called p-hacking. It’s like throwing darts until you hit a bullseye and then saying you only threw the dart that hit the bull’s eye.
This is exactly what the music researchers did. They played three groups of participants each a different song and collected lots of information about them. The analysis they published included only two out of the three groups. Of all the information they collected, their analysis only used participants’ fathers’ age—to “control for variation in baseline age across participants.”
They also paused their experiment after every ten participants and continued if the p-value was above .05, but stopped when it dipped below .05. They found that participants who heard one song were 1.5 years younger than those who heard the other song, with a p-value of .04.
Usually, it’s much tougher to spot p-hacking, because we don’t know the results are impossible: the whole point of doing experiments is to learn something new. Fortunately, there’s a simple way to make p-values more reliable: pre-registering a detailed plan for the experiment and analysis beforehand that others can check, so researchers can’t keep trying different analyses until they find a significant result.
And, in the true spirit of scientific inquiry, there’s even a new field that’s basically science doing science on itself: studying scientific practices in order to improve them.