How We’re Fooled By Statistics

Which is most effective for helping people learn: punishment or reward? Well, consider the case of Israeli fighter pilot training, because instructors there found that negative feedback was far superior to positive feedback. If a cadet performed a particularly poor maneuver, they would reprimand him. But they noticed that on follow-up attempts, his performance invariably improved. In contrast, if they praised a cadet for executing a skillful maneuver, his performance on subsequent attempts typically declined.

So naturally they concluded that positive feedback is ineffective or even detrimental, whereas negative feedback is what works. The problem is this seems to contradict a body of research that shows positive feedback is actually more effective than negative feedback. For example, in studies involving teachers, it was found that if a teacher increased the ratio of positive feedback to negative feedback, that actually increases the percentage of times students spend on task.

In another study, rugby players were given a video feedback session following a game, except half of them were shown their highlights, and they were praised for what they had done, and the other half were shown their biggest blunders, their mistakes. And they were scolded for doing things incorrectly. Now a week later, at the next game, it was the players who had received the positive reinforcement who performed the best. And, in fact, the difference was not just psychological. Blood tests revealed that the players who received positive feedback actually had higher levels of testosterone than players who had received the negative feedback.

I think that is quite remarkable that even a week post-feedback, you could still see a physiological difference between the two types of feedback. Wow. Well, so why didn't this work for the Israeli fighter pilots? Well, maybe there are cultural differences and in some cultures, negative feedback works better. Or maybe it is task-dependent. Perhaps some skills, like learning to fly a fighter jet, require more negative feedback.

Or maybe the feedback had nothing to do with the performance of the fighter pilots at all. Maybe they would have performed in the same way, regardless of the feedback they received. Go with me on this. Imagine you have 100 students taking a test, which consists of 100 true or false questions on a subject they know nothing about. Now, assuming they all answer randomly, we know that the resulting distribution of scores will have an average of around 50. But just by chance, some students will have scored significantly better or worse than the average.

Oh, man. >> If you select the top 10 students, whose scores were all above 50, and gave them a second very similar test, you would find that the average of their scores would drop back to around 50. Similarly, if you selected the bottom 10 students, their score on a subsequent test would rise to 50. This is just regression to the mean.

Regression to the mean is the reason why if you have a really good round of golf today, your round tomorrow will not be as good. Yes. That is because random chance plays a role in virtually everything that we do. So the outcomes of events are influenced by both our skill and a little bit of luck. So if you have especially good luck on one day, chances are your level of luck will not be as good the next day.

Now that sounds a little bit like the gambler's fallacy, which states that past events influence future probabilities. For example, if you flip a coin and it comes up tails a couple of times, the gambler's fallacy is to expect the probability of heads to increase for the next flip. In reality, it doesn’t. It is still 50-50. The idea with the gambler's fallacy is that probabilities change so that overall things even out in the long run. That is the idea.

With regression to the mean, it is not that things are evening out, it is just that extreme events are becoming diluted by the average events, which happen much more commonly. Now with these students, they were completely guessing on every question. But even when you know something about the subject, there is still going to be that element of chance. And so regression to the mean always occurs, just to a lesser extent. This is really important to consider in research.

Imagine you are trialing a new drug to help prevent heart disease. So from a sample of patients, you select those in the bottom 10 percent of heart health indicators, people with high blood pressure, high cholesterol, that sort of thing. Now, after a month of being on a drug, you test them again to find that their scores have improved. Well, great, the drug is working, right? Well, maybe not.

See, the trouble is that although blood pressure and cholesterol are more stable than, say, your score on a random true or false test, there is still some inherent variability caused by, say, your level of stress on the day, or your recent diet, or even the uncertainties introduced by the measuring apparatus. So the people who ended up in the bottom 10 percent on the first test likely had these factors all count against them. They were particularly unlucky.

But you should not expect them to be as unlucky when you test them a second time. So their scores should improve just based on random chance. That is why it is so important that clinical trials use control groups drawn from exactly the same population so that you can see whether the drug improves scores more than random chance alone would.

Or what about assessing the impact of speed cameras? When they are first installed, they are normally put in locations that have had a high volume of accidents in the previous year or two. Makes sense. Now those accidents are likely due, at least in part, to bad road design, but also due, at least in part, to bad luck. And so you shouldn’t be surprised that after a speed camera is installed, the number of accidents goes down. It would go down anyway simply due to regression to the mean.

Meanwhile, somewhere else that previously had a small number of accidents will likely have more. So the overall accident rate may not change, and yet we will feel as though our investment in road safety has paid off. But perhaps what is most troubling about regression to the mean is how it influences our perception of feedback. For example, with the fighter pilots, after they execute a maneuver particularly poorly, chances are the next attempt is going to be better, regardless of the feedback they receive.

Similarly, after an especially successful display, chances are the next attempt is not going to be as good. And that has nothing to do with the feedback and everything to do with the statistical nature of our universe. But we are hardwired to see patterns and causality everywhere, which is why the instructors felt that positive feedback was detrimental and negative feedback is what works.

And this is really unfortunate because that is the exact opposite of what well-controlled psychological studies show. You know, I think it is really unfortunate that if you give negative feedback, chances are you will be rewarded. They will do better. And if you give positive feedback, chances are you will be disappointed because it is difficult to maintain that level of luck. But that is the way our world is.

So think about that next time before you tell someone off. If you stay positive, it may just work out for the best in the long run. This video was inspired by the book "Thinking Fast and Slow" by Nobel Prize-winning economist Daniel Kahneman. So that was the first time I heard of the Israeli fighter pilot training and really started thinking about regression to the mean.

It was pretty awesome, and I downloaded this book from Audible, so I was listening to it on my travels. I am currently at the Perimeter Institute, which you can see has some beautiful blackboards. Now if you want to listen to this book, you can download it for free by going to Audible.com/Veritasium, or you can listen to any other book of your choice for a one-month free trial.

So thanks to Audible for supporting me. Thanks to Perimeter Institute for putting me up in this amazing location. And I will see you next time with some theoretical physics ideas. So stay tuned for that.