The Universe is Hostile to Computers

A plane plummets out of the sky, a speed runner inexplicably jumps to a higher platform. What the? What the?! And an election recount is triggered. All because of the same invisible phenomenon that permeates the universe.

On May 18th, 2003, voters in Belgium went to the polls. In many regions, voting was done on a computer, something the Belgians had been experimenting with for over a decade. But the system had a backup. Each voter would insert a magnetic card into the machine and make their selection on screen. Their vote was saved both to the computer and the magnetic card, which they dropped into a box for redundancy.

Late that night, as the votes were being tabulated, one of the election officials detected a problem with the results from Schaerbeek, a municipality in central Brussels. Maria Vindevogel, a little known candidate with her own party, received more votes than was mathematically possible. They knew this because of the way the preferential voting system works. So they took out the magnetic cards and started a recount. One by one, they fed each of them through the machines again, and after several hours, the recount was complete.

The vote totals for every candidate were exactly the same as before, except for Maria Vindevogel. In her case, the recounted number of votes was less than the original by 4,096. So what went wrong? How had her original tally been inflated by over 4,000 votes? Computer experts were brought in to run extensive tests on the software. They combed through the code, but could find no bugs. They got the computer that had made the initial erroneous tally and tested the hardware again and again, but they could not replicate the error.

Everything about the hardware seemed to be in perfect working order. And this left only one possible explanation, and it is seriously weird. The clue comes from the excess number of votes Vindevogel received: 4,096. Computers work using binary, strings of zeros and ones, each corresponding to a power of two. So somewhere inside the computer tabulating all the votes was a string of bits representing the number of votes Maria received.

It started the day all zeros, and then as each vote for her came in, it would increment by one. Physically, this is done by turning on a transistor for one and turning it off for zero. What's remarkable about the number 4,096 is that it is exactly a power of two: two to the power of 12. That is the 13th bit. So for Maria Vindevogel to receive an extra 4,096 votes, only one thing needed to happen. The 13th bit had to flip from a zero to a one. But why would that happen?

Computers work precisely because bits don't flip unless we want them to, or do they? Looking into the problem, Belgian investigators found reports of similar issues from big computer companies starting in the 1970s. In 1978, Intel reported some strange errors popping up in their 16 kilobit dynamic random access memory or DRAM. Ones would spontaneously flip to zeros with no apparent cause. The problem turned out to be the ceramic packaging the chip was encased in.

With the demand for semiconductor packaging skyrocketing in the 1970s, a new manufacturing plant was constructed on the Green River in Colorado. Unfortunately, this site happened to be just downstream of an old uranium mill. Radioactive atoms made their way into the river and then into the ceramic packaging for Intel's microchips. Intel scientists investigating the problem found that even trace amounts of uranium and thorium in the ceramic were sufficient to cause problems.

In their DRAM, memory was stored as the presence or absence of electrons in a semiconductor well. The alpha particles emitted by uranium and thorium were energetic and ionizing enough to create electron hole pairs in the silicone. If an alpha particle is struck in just the right place, it could create a large number of free charge carriers causing electrons to accumulate in the well, flipping a one to a zero. This is known as a single event upset, a type of soft error.

The error is soft because the device hasn't been damaged. The bit has changed, but you could erase it and rewrite it with no problems. Investigators exposed the chips to alpha emitters with different levels of activity. And just as you'd expect, they found the number of bit flips directly correlated with the number of alpha particles the chip had been exposed to.

The reason this problem was identified in the 1970s was because chip components had been miniaturized to the point where a single alpha particle could produce enough charge to flip a bit. Immediately, these findings attracted a lot of attention. Before the paper was published, it was widely circulated in the industry. And as a result, chip manufacturers were a lot more careful to avoid radioactive materials when producing their microchips and packaging.

Therefore, the bit flip that gave Maria Vindevogel 4,096 extra votes wasn't caused by natural radioactivity in the computer. So where did it come from? After Henri Becquerel discovered radioactivity with uranium in 1896, scientists sought a way to measure it. How radioactive were different materials? And one way to do this is with a gold leaf electrometer. When it's charged, the leaf is repelled and you can measure the amount of charge by the angle of the leaf.

Now, if ionizing radiation enters the chamber, it rips electrons off air molecules, creating positive and negative charges. Opposite charges are attracted to the leaf, discharging it over time. The higher the level of ionizing radiation, the faster the device discharges. In 1910, Theodore Wolf took his electrometer to the top of the Eiffel Tower. Since radioactivity was found in the soil and rocks of earth, he expected that 300 meters up, the radiation would be just a few percent of the ground radiation.

Instead, he found only a slight decrease. In 1911, Austrian physicist Victor Hess decided to take this experiment further. Literally. He loaded electric scopes into the basket of a hydrogen balloon. In his first two flights, he observed the same thing as Wolf. Up to an altitude of 1100 meters, both trips revealed no fundamental change in radiation compared to the values observed on the ground.

But the next year, he conducted seven balloon flights up to an altitude of 5,200 meters. And here he discovered something remarkable. While there was an initial drop in radiation for the first several hundred meters, above one kilometer or so, the level increased with increasing altitude. At his maximum height, the level of radiation was several times greater than it was on the ground. The radiation appeared to be coming, not from the earth, but from the sky.

Hess scheduled one of his balloon flights during a solar eclipse. And as the moon passed in front of the sun, he carefully watched his instruments. But the readings were unaffected. Even with the sun half-covered, the level of radiation was the same. Since no influence of the eclipse on the penetrating radiation was noted, we conclude that even if a part of the radiation should be of cosmic origin, it hardly emanates from the sun.

Victor Hess had discovered cosmic rays. High energy radiation from space. But what were these rays exactly and where were they coming from? Well, today we know they aren't electromagnetic rays, as many suspected, but particles. Around 90% are protons, 9% are helium nuclei, and 1% are heavier nuclei. Some of them are from the sun, but they have comparatively low energy. High energy cosmic rays moving very close to the speed of light come from exploding stars, supernovae in our own galaxy and in others.

And the highest energy particles are thought to come from black holes, including the super massive black holes at the centers of galaxies. But it's hard to pin down exactly where cosmic rays come from because, as charged particles, they're deflected by magnetic fields in space. So they can wind their way through the universe for billions of years. A cosmic ray detected on October 15th, 1991, had an energy of 51 joules. That is a single subatomic particle with the energy of a baseball going a hundred kilometers per hour.

For obvious reasons, it was dubbed the OMG particle. But primary cosmic rays like these don't make it down to earth's surface. Instead, they collide with air molecules around 25 kilometers above the ground and create new particles like pions. These collide and decay into other particles like neutrons, protons, muons, electrons, positrons, and photons, which in turn collide with other air molecules in one long cascade.

So from a single primary cosmic ray, comes a shower of particles streaming towards the earth. It is one of these particles that investigators suspect struck a transistor in a computer in Belgium, flipping the 13th bit from a zero to a one and giving Maria Vindevogel 4,096 extra votes. But how often do things like this happen?

In 1911, Charles Wilson made it possible to see the cosmic rays all around us when he perfected his cloud chamber, an enclosure with supersaturated water or alcohol vapor. When a cosmic ray passes through the chamber, it ionizes the gas, causing vapor to condense into tiny droplets on the ions, revealing the path of the particle. Alpha particles, helium nuclei, leave short thick tracks while beta particles, electrons, leave long skinny trails.

In 1932, Carl Anderson identified a trail that looked like it was made by an electron, but in the applied magnetic field, it curved in the wrong direction. Implying it had a positive charge. Anderson had found the anti-electron or positron. It was the first confirmed sighting of anti-matter. Four years later, also using a cloud chamber, he discovered the muon, again in cosmic rays. For his discovery of the positron, Anderson was awarded the Nobel Prize in Physics in 1936. He shared the prize with Victor Hess, the man who discovered cosmic rays in the first place: invisible particles that affect our lives in ways most of us are oblivious to.

This is possibly the rarest thing to ever happen in a video game. In 2013, user DOTA_Teabag was speed running Super Mario 64 on the console. In the level Tik Tok Clock, he suddenly up warps onto a higher platform. - [DOTA_Teabag] What the? - [Player] Did you get invisible wall? What? - Please say you got the--- - [DOTA_Teabag] No, that was the craziest thing I've ever seen though.

• [Narrator] The move shaves off a few seconds and it seems like a newly discovered glitch in the game that could give speed runners an edge. User PenandCook12 put out a $1,000 bounty for anyone who could replicate the up warp. But so far, after six years, no one has been able to.

The best explanation anyone can come up with is that a cosmic ray caused the glitch. It's been shown that a single bit flipped in the first byte of Mario's height coordinate could have caused the effect. On the main level, the byte was 1 1 0 0 0 1 0 1. But if you flip the last one to a zero, it changes his vertical position. And just by chance, this new height coincides with the higher floor. PenandCook12 wrote a script to manually flip the bit at the right moment and was able to achieve the same up warp.

This is a particularly visible bit flip, but the truth is cosmic rays are triggering bit flips all the time. - An upset there, transient there can alter the function of these devices and we call that a single event functional interrupt. So an entire process can hang. So a blue screen of death that you get might actually have been a neutron or whatnot. - [Derek] When people see the blue screen of death, could that be caused by a cosmic ray? - Absolutely.

These days, there are a number of ways computer chips are made resilient in the face of bit flips, like error correction code or ECC memory. But you can't totally prevent bit flips from happening. In 1996, IBM estimated that for each 256 megabytes of RAM, one bit flip occurs per month. And the main culprit seems to be neutrons created in the shower of particles from cosmic rays.

Starting in 2009, Toyota recalled millions of vehicles due to the problem of unintended acceleration. - We were in the fast lane driving at about 70, and he said that the car was continuing to accelerate and he couldn't bring it to a stop. - Many speculated that cosmic ray induced bit flips in the electronic control system were the cause. So much so that NASA was called in to help with the investigation. But it turns out, cosmic rays were probably not the culprit.

NASA identified as the main causes, sticky accelerator pedals, poorly fitted floor mats, and most commonly, drivers pushing on the accelerator thinking it was the brake. But cosmic rays have caused crashes of supercomputers, especially at higher elevations. Los Alamos National Labs, located 2200 meters above sea level, is constantly dealing with neutron induced supercomputer crashes. So the software auto-saves frequently, and neutron detectors have been installed throughout the facility.

If you go even higher, like climbing up to cruising altitude in a plane, you can see the radiation from cosmic rays increasing on a Geiger counter. To .5 microsieverts per hour at 18,000 feet. Up to one microsievert per hour at 23,000 feet. Over two microsieverts per hour at 33,000 feet. And over three microsieverts per hour at even higher altitudes and towards the poles. At cruising altitude, this increases the chance of a single event upset by 10 to 30 times.

This isn't critical if it happens in your laptop, but what if it occurs in the flight computer? On October 7th, 2008, an Airbus A330 took off from Singapore to Perth. Just over three hours into the flight, the plane suddenly pitched down, diving 200 meters in 20 seconds. Inside the plane, everyone experienced negative 0.8 Gs of acceleration. It would have felt like the plane had flipped over. - The G-Force was enough, even with our three-point harness, to lift us both out of the seat and push us forward as well.

• Minutes later, the plane dropped another 120 meters. 119 people on board were injured, many from bumping their heads into the ceiling. So the pilots decided to make an emergency landing in Learmonth. In the investigation that followed, the fault seemed to occur with the first air data inertial reference unit, or ADIRU for short. This computer supplies critical data like airspeed, angle of attack, and altitude.

The way it supplies each of these pieces of information is in a 32-bit binary word. The first eight bits identify the type of information, and bits 11 to 29 encode the actual data. What seems to have happened is that a bit flip in the first eight bits meant altitude information was mislabeled as angle of attack information. Inside the cockpit, alarms went on for overspeed and stall simultaneously, something that should be impossible. But the plane nosed down sharply to correct what it thought was a stall, throwing passengers and crew into the ceiling.

Investigators looked into software bugs, software corruption, a hardware fail, physical environment factors, and electromagnetic interference. But each of these possibilities was found to be unlikely based on multiple sources of evidence. The other potential triggering event was a single event effect resulting from a high-energy atmospheric particle striking one of the integrated circuits within the CPU module.

One of the challenges with single event upsets is that they are soft errors. They don't leave a trace, but interestingly, the Airbus A330 was built in 1992 when there were no specific regulatory or aircraft manufacturer requirements for airborne systems to be resilient to single event effects. With the space shuttle, redundancy was built in from the start. For navigation and control, there were four computers simultaneously running identical software. If one computer had a soft error, the other three would overrule it.

And using this setup, they could also track the frequency of bit flips. On one five-day mission, STS 48, there were 161 separate bit flips. Above the atmosphere, cosmic rays are so energetic sometimes you can even see them. - Once in a while, you have your eyes closed, and you're not asleep yet. And if you wait a little while, you occasionally will see a flash of light.

And we think it is heavy particles or individual bursts of energy coming from radiation that are either going through the eyeball itself or going through the optic nerve. And the way that your body registers radiation going through it is amazingly enough by showing you a little flash in one of your eyes, just to remind you that you are subject to the radiation of not only our sun, but every star of the universe that is radiating at you.

I picture back to the first astronauts who must have closed their eyes and seen that radiation and gone, "I'm not going to tell anybody about this because no one's told me about it. I'm not talking." I can just imagine the first two guys that said, "Hey, I am, sometimes I see flashes of light. Do you see flashes of light?" And then it's, "Oh, we all see flashes of light." "Oh, okay, well, that's all right then." (narrator laughing)

• [Narrator] For missions to other planets, protecting electronics is critical. - If a single bit of information controls a critical function on your spacecraft, let's say your thrusters, and that goes from a one to is zero, from an on to off, or vice versa, you could lose the mission. - [Narrator] That's why the computer on the Perseverance Rover that just landed on Mars is 20 years old.

It's a power PC launched in 2001 with only 256 megabytes of RAM and two gigabytes of flash storage. But it is radiation hardened, meaning the design, materials, circuits, and software are built to withstand 40 times the radiation of an ordinary computer. It's been used on over a dozen space missions going back to 2005.

• In fact, when we first started doing the power PC testing years ago, the way we did it, we just simply stuck an operating system processor in a beam line where we generate these particles on the planet and look for blue screens of death. You can kind of figure out what's going wrong and try to undo that. So you don't get to the blue screen of death because a spacecraft that gets into that mode is basically unrecoverable.

• As the Voyager 1 spacecraft departed the solar system, one of the ways we could tell was by an increase in the flux of cosmic rays it experienced. Although on earth, the particles streaming from the sun and the solar wind are a source of radiation, further out these same particles maintain a protective bubble: the heliosphere. It helps deflect and slow cosmic rays from interstellar space, protecting the solar system from ionizing radiation.

But the sun has an 11-year activity cycle. So this protection fluctuates. The flux of cosmic rays on earth is much lower when the sun is active than when it's dormant. In the history of our planet, cosmic rays may have played an even larger role flipping bits, not in electronics, but in the genetic codes of living organisms, providing some of the variation on which natural selection acts.

Maria Vindevogel is now a member of the Belgium Chamber of Representatives, elected by people, not a particle. But her story is a reminder of the zillions of particles winding their way through the universe for millions or billions of years, one of which might at any moment change your life by passing through a tiny transistor and cra.... (electronic disturbance)

Discovering the origin of cosmic rays or where 4,096 extra votes came from or how to protect the Mars Rover from radiation requires problem-solving, which is a perishable skill. And the best way to keep your problem-solving skills sharp is to solve a diverse range of problems, which you can do with the sponsor of this video, Brilliant.

Brilliant is a website and app that teaches you STEM concepts in an interactive way. They have courses on topics from applied computer science to special relativity. Two courses I would recommend if you enjoyed this video are computer memory and algorithm fundamentals. Brilliant has really upped their interactivity recently. So instead of reading about algorithms, you're taught how to write algorithms yourself.

Brilliant engages you in active learning. Instead of telling you the right answer, they allow you to figure it out for yourself with plenty of helpful hints and guidance along the way. And for viewers of this channel, Brilliant is offering 20% off an annual subscription to the first 200 people to sign up. Just go to brilliant.org/veritasium. I'll put that link down in the description.

So I want to thank Brilliant for supporting Veritasium and I want to thank you for watching.