Reversion to the Mean

It’s a cold winter morning. You wake up fresh and rested. You get ready and go to the nearest cafe to start the day with a cup of coffee. You order your favorite: an iced caramel latte. As you wait for your coffee, you make small talk with the stranger next to you in line. You two quickly hit it off and the conversation just oozes chemistry. Oh, and the coffee is here too. Right on queue. You just know it’s going to be good as always, and it really is. Just the right amount of milk, not too sweet. The caramel is there too, but it isn't overpowering. “How awesome that the barista got it right this time?” you think to yourself. The conversation continues, and in the back of your mind, you just know: “Today can’t get any better.”

… and then slowly the day comes to an end. You wake up the next morning still buzzing from the energy of the day before, your mind all excited to try and recreate that perfect experience, that perfect conversation, that perfect cup of coffee. You get ready just the same and head out to the cafe, only to realize your favorite spot has been taken. Oh, and there’s a staff shortage so the queue is extra long. Everyone’s kinda doing their own thing, kinda cranky. Even the coffee today is a bit too sweet. There was no refreshing aroma this time, and certainly no kind stranger willing to make eye-contact, let alone small talk. The whole experience is just so… mediocre. And really, that’s the story of most of our days.

What happened? What changed? Is it something you did? Maybe the way you did your hair in the morning? Maybe it’s the faulty coffee machine? You try coming up with reasons, but you can’t seem to figure it out. What prevented you from recreating that “perfect” experience? Statistics might have an answer for you. You see, whenever we have a random variable that could be almost anything, a phenomenon called “reversion to the mean” tells us that an “extreme” instance of that random variable will be followed up by a less extreme instance if that measurement were to be taken again.

In the cafe example I just mentioned, how well your experience in that cafe goes is a random variable, or rather a bunch of random variables. And having the ‘perfect’ experience entails that all of the random factors that could have gone your way did. It would be an “extreme” instance in that mornings are rarely as perfect as this one. Reversion to the mean suggests that if this experiment were to be conducted again, if you were to go to the cafe again, your experience would tend more towards the average, leading to a mediocre experience. This is because all those factors being lined up the right way the first time was just a result of very good luck, and it’s not likely to reoccur a second time.

Now, when we’re having average days, or even bad ones, it’s normal to think that there is some negative force of the universe causing things to be this way. But, in truth, there is no such cryptic reason as to why your morning coffee wasn’t so great. In fact, reversion to the mean is a general statistical tendency. It happens to everyone and everything, all the time. Whenever you have a variable whose behaviour is accounted for partly by randomness, you are bound to see reversion to the mean. If your variable doesn’t depend on randomness, well, there you have it, you will have the same result over and over again - there won’t be any variance to begin with.

But reality is rarely like that. In fact, reality is never like that. It’s riddled with randomness and factors that are too complicated for us to predict. As a result, we will always see extreme events from time to time in both directions; but over the long run, they seem to cancel each other out, and the outcomes tend towards a mean. Now given the universal nature of this phenomenon, one would expect people to generally have an intuition for it. But you’ll be surprised as to how often reversion to the mean can be overlooked.

For example, one of the more famous examples about reversion to the mean is from Daniel Kahneman’s book, “Thinking Fast and Slow” where he details a meeting with Israeli fighter pilots. Daniel was arguing that positive feedback was more beneficial to cadets in terms of their performance in-flight. However, some of the commanding officers that were present disagreed. They said that Daniel’s rosy take on feedback wasn’t realistic, and that based on their experiences, cadets who underperformed reacted best to harsh criticism and those that performed well ended up performing worse if they were praised.

So, clearly, positive feedback wasn’t good for them, right? Well, not quite. You see, someone who performed really well on a maneuver probably had some luck on their side. Of course, these are trained professionals and their abilities have a part to play in what they do, but so does luck. And so, on subsequent attempts, despite the same ability, they had simply run out of luck and ended up performing worse than before. Their low-performing colleagues, however, probably had bad luck the first time, and lightning never strikes twice. Over the next few attempts, they had better luck.

Point is, in both instances, the pilots regressed towards the mean. For the pilots that did well the first time, this meant a decline in performance, and for the pilots that did poorly the first time, this meant an improvement. This, of course, implied that the commanding officers were, in truth, overemphasizing the contribution of their intervention. But how do we know that for sure? How do we know what interventions work and which don’t? Answering that question becomes particularly important once you realize the consequences of reversion to the mean, especially in healthcare.

How do we know whether any medical intervention is effective at all, given that patients with most illnesses tend to feel better with time? How do we know they are responding to treatment and not simply regressing to the mean? That’s where control groups come in. And really, that’s where the concept of Placebos comes in, which is a topic I have covered on this channel before. Once you realize the effect of reversion to the mean, it becomes imperative to have control groups where the intervention being tested won’t be used so as to distinguish its effect from that of the reversion. If a medicine works better than reversion to the mean, we can be reasonably certain about its effect.

This has become a crucial part of the medical process and the greater scientific process, as, again, reversion to the mean is everywhere and you have to isolate its impact in your conclusions. The consequences of not doing so can be dangerous. It can lead to unnecessary suffering from ineffective interventions, and could even cost lives. Reversion to the mean is also seen in the electoral process. More extreme candidates tend to be followed by less extreme candidates. Then there are financial markets, where prices can stall or skyrocket, but generally stay in the ballpark of some market average. In fact, moving averages are used by traders around the world every day.

Of course, most of us are naive to this, despite its regularity, and routinely extrapolate the immediate past into the indefinite future. Reversion to the mean also applies to individual circumstances. If you did really well on a test, chances are, you won’t do so well the next time. Exemplary performance is rarely sustainable. At the same time, if you did poorly the first time around, you are likely to do better the second time around. As I mentioned with the pilots, your ability definitely has a role to play in what you do, but the overall statistical tendency of ebb and flow remains.

You can see it in sports too. Athletes who do really well on their rookie season, rarely live up to the expectations in the subsequent seasons. In fact, knowingly or unknowingly, there are many references to this tendency in the form of the “Sports Illustrated jinx” or the “commentator’s curse.” Of course, the commentator isn’t casting some dark spell over the players. It’s just that given the spectacular nature of what a player did to earn the adoration, they are simply not likely to recreate it again. Of course, there are exceptions to all rules, and even with this one. Some people were just too good.

Michael Jordan, for example, rode the covers of Sports Illustrated over 50 times with no significant decline in his performance. But even this elusive athlete wasn’t entirely free from the effects of reversion to the mean. Michael Jordan was an exceptional basketball player, needless to say. And his talents were justifiably extreme. If reversion toward the mean is indeed correct, it would predict that his sons, despite inheriting some of that talent from their dad, were not likely to soar to the heights their father did. And that’s exactly what happened. Even though they had increased attention and the privilege that came with being Michael Jordan’s sons, they never really made it. They were successful college athletes, sure, but they were no NBA players, let alone one of the greatest athletes to have ever been.

Chances are, they worked really hard. But, genetics is just a tad too random, and that makes the effects of reversion to the mean that much more pronounced. And that might make Jordan a bit sad. But, in some sense, he should thank genetics for indeed being so random. After all, his dad was 5’9” and his mom was 5’5”. Michael was 6’6”, and without that seemingly “lucky” boost in height, who knows, maybe he would never have become the greatest basketball player of all time.

So, wait, you’re telling me reality tends towards mediocrity? And there’s nothing I could do about it? Now, I don’t know about you, but I find that hard to accept. I thought there’s gonna be Hans Zimmer playing in the background as my life plays out. I thought there were gonna be cool cut scenes here and there where the wind hits just right so that it ruffles my hair and makes me look cool. To me, the concept of reversion to the mean seems almost boring, and a wee bit disheartening if I’m being honest. Because it takes away from the notion that hard work is what gets you success, not chance events.

It further casts a shadow on the already poorly lit facade of free will. If we are all indeed ebbing and flowing between spectacular and awful, only to end up in the mediocre, what control do we really have? Besides, it seems as though statistics is almost forcing us to be mediocre, to be less than, and nobody likes being mediocre. As Lieutenant Commander Phillip F Queeg, captain of the USS Caine in World War II said, “Aboard my ship, excellent performance is standard, standard performance is substandard, and substandard performance is not permitted to exist.”

I think a lot of you feel the same way. There is something fundamentally nauseating about “mediocrity.” It’s so… well… mediocre. Of course, “average” should by definition be a neutral word, but it’s hard to disassociate the negative connotation it seems to carry. That is until you fall sick. Or until you lose a loved one, or don’t have access to food, water, or shelter. Then, well, being average is not too bad. Then, being normally healthy seems priceless.

Ask someone who has an empty seat at the dining table - a “normal” family is all they could ask for. Being normal doesn’t seem so bad now does it? And yes, while we can’t control the statistical tendencies that cause reversion to the mean, we can, nonetheless, gradually push the mean closer to where we want it to be. With consistent but ever so slight strides, and with hard work, we can all become better averages.

This is not a denial of how unpredictable life is, and how little control we have over it sometimes. Rather, it is taking control of whatever little we can control and not getting too preoccupied with what we can’t. It also serves as a reminder that we may not be as responsible for our success as we might like to take credit for. Maybe the person next to me deserved it just as much, but while the random variable that is my life was basking in its fleeting luck, that of someone else's was regressing to its unspectacular average.

They were just out of luck. And as far as free will is concerned, well, maybe, the height of free will is to simply be aware of its limitations. But enough talking, life is finite. You haven’t got all day, and uh, your coffee’s getting cold.