The Battle for the Soul of Artificial Intelligence | Podcast | Overheard at National Geographic

[Music]

I'm a sci-fi nut and one of my favorite books is The Caves of Steel by Isaac Asimov. It's all about this hard-boiled grizzly detective who gets assigned a strange new partner, a robot. I've always wanted a robot partner, and now through the magic of text-to-speech and radio drama, hello! I can finally have one.

My name is Natalie. I'm here to help you host the show.

Hi! I'm glad you're here. This is my first time hosting, and since I'm taking over for our regular hosts, Amy Briggs and Peter Gwyn, I'm sure I'll need all the help I can get.

In the book, this robot detective is the perfect crime-solving machine, like a calculator but for crime, so I'm glad to have the perfect partner.

Pretty good?

Uh, what? Not perfect, but pretty good. Sometimes we make mistakes.

Ah, little mistakes?

Not exactly.

Well, I'm sure it's not that bad. At least nothing serious, like using humans as batteries, or turning the universe into paper clips, or subtly perpetuating systemic racism.

No, no, and yes.

Ah, well, how bad can it be? Research shows that voice-enabled assistants are likely to have a higher accuracy rate when understanding white American male voices as compared to any other identity.

Okay, subtle and annoying, but not too bad. What else?

In 2015, Google's photo categorization software was found to be labeling black people as gorillas.

Oh, that's pretty bad, but it was six years ago. I'm sure we've worked out all the kinks by now.

Since 2019, at least three black Americans have been arrested because facial recognition used by police misidentified them.

Oh boy, that's what this episode's about, doesn't it?

Correct! Like in your detective story, humans are teaming up with AI systems all over the world. These systems are pretty good but not perfect.

Correct! If we're not careful, the machines trained by humans will inherit the same flaws, biases, and imperfections as humans.

I'm Brian Gutierrez, and I'm Natalie from a text-to-speech program. I'm a podcast producer for National Geographic, and you're listening to Overheard—a show where we eavesdrop on the wild conversations we have here at Nachio and follow them to the edges of our big, weird, beautiful world.

This week we look at some of the biases showing up in artificial intelligence and how we can do better. More after the [Music] break.

On the afternoon of January 9th, 2020, Robert Williams, an African-American man who lives in a Detroit suburb, arrived home from work. Waiting for him were two unexpected visitors: Detroit police officers who arrested him on his front lawn.

I was completely shocked and stunned to be arrested in broad daylight in front of my daughter, in front of my wife, in front of my neighbors. It was one of the most shocking things I ever had happen to me.

This audio comes from an interview with the ACLU. It's sadly a familiar story: a black man arrested with no idea why. But this scenario is a little different. Williams was held overnight in a crowded cell for 18 hours before he was pulled into a room for questioning, and that's when he finally learned why he was being detained.

A detective turns over a picture of a guy, and he's like, "This is you." I look, I said, "No, that's not me." He turns another paper over. He said, "This isn't you either." I picked that paper up and held it next to my face, and I said, "This is not me! Like I hope you all don't think all black people look alike." Then he says, "Well, the computer says it's you."

His accuser was a cold hard algorithm. Facial recognition software had incorrectly matched Williams to a blurry surveillance photo.

He laid the papers out on the table, and at that time, I was still guilty in their eyes. "This isn't you either," until the pictures don't match, and they left them on the table, and they looked at each other like, "Oops."

Police released Williams later that night and eventually dropped the charges against him, but that came after thousands of dollars in legal fees, missing time at work, and the humiliation of being arrested in front of his family and neighbors.

The ACLU says Robert Williams was the first known wrongful arrest in the United States based on an incorrect facial recognition match. Robert Williams's case is not an isolated event. We looked into it and found at least three examples of people of color who have been wrongfully arrested based on flawed facial recognition software.

There is clearly something funny going on here. [Music]

Well, when we get into bias, it truly has to do with the image processing that's going on behind the scenes.

That's Gloria Washington, an assistant professor at Howard University, where she teaches computer science classes and directs a lab on artificial intelligence.

When you have these fuzzy videos and you get these fuzzy image stills and you're comparing it against a high-quality mug shot, sometimes you're going to have some false positives.

She's careful not to blame those in law enforcement for the misuse of AI in identifying criminal suspects.

You have policemen and people who work in criminal justice who are very busy, and they don't have time on a granular level to take a really good look at the images or the recommendations that the system may say is the person, and they act on that information.

Am I correct in thinking that these kinds of facial recognition software are more biased towards black Americans?

A computer, from the perspective of a computer scientist, cannot be biased. When it gets to darker melanin, it really starts to fail because the features that are present, you really can't see a lot in images.

Like even if you look at myself, like right now, I have hair that is covering part of my eye, and I am a darker-skinned individual, and if the lighting is not correct, sometimes parts of my features can be occluded or can be fuzzier. It can be noisier.

Okay, so darker skin shows up differently in pictures and might be the source of some of these errors. So it's not really biased; it's just the image processing behind the scenes.

There needs to be more techniques that focus on darker-skinned individuals, and how do we pull out those features that are more prevalent in darker-skinned individuals?

Facial recognition algorithms learn to recognize faces by seeing lots of examples.

Gloria explained that these programs can develop blind spots if it wasn't shown enough of them.

So in order to teach the AI something, you have to show it a bunch of pictures?

Yes, and it seems like you're saying that that original group of pictures you show tend to favor one group or another.

Why are so many of these data sets to start with skewed one way or another?

Well, I think it's like an academic problem.

Gloria explained that a lot of this data tends to come from college students who are not particularly diverse in terms of age and ethnicity.

How do you know if a data set is diverse?

You know, my impression is that there's millions and millions of images. How do you know if there's an issue?

Well, for me, I had no choice but to look at these databases of millions of images, where my entire day was looking through these databases to determine how diverse they were.

How many images do you think you looked at to code with this information?

Well, there was a data set from Hong Kong that had a minimum of a million, and I was there for three years. So I, at a minimum, it was a million.

Because I had to truly— you looked at a million images?

Yeah.

You kind of—when I speak about it, it was tedious, but I kind of got really good at doing it.

Spending three years to manually review a million images is one solution, but it's not necessarily practical.

Is there an easier way to figure out if a data set is biased?

I reached out to Patrick Brother, a computer scientist with the National Institute of Standards and Technology, or NIST, to find out.

Would you mind holding your phone kind of like you’re doing a phone call for one hour?

Really? I have to do that?

Brian, did you make him hold the phone for an hour?

Part of what Patrick does with NIST is publish a monthly report on the accuracy of facial recognition software. For the first time in 2019, Patrick and his team decided to check not just how accurate these programs are generally, but to break it down by race.

Various colleagues across the U.S. government were interested in, well, how serious is this problem and what is this problem? The problem was obvious.

After Patrick and his team evaluated software from 99 developers, a majority of the industry, they concluded that these programs misidentified African Americans and Asians anywhere from 10 to 100 times more than Caucasians.

So even with essentially pristine photos of the kind that appear in your passport or the kind that the police take in a booking mug shot setting, good photos would give you false positive variations, particularly by race, also by age, and less so by sex.

Given that there have been maybe three or four false arrests based on these facial algorithms, do you think it's too soon for us to rely on them in this way?

So yeah, that should be concerning. It occurred in fingerprint recognition from latent fingerprints in the past also.

And yeah, this is not an easy call to be made by people who do that for a living, but the overarching policy should be to make the investigators aware that the algorithms can make mistakes, that the algorithms are merely producing candidate identities and not saying anything definitive about whether it's the same person or not.

So it could be helpful in terms of testing the algorithm. You could show somebody a picture of the face you fed to the algorithm and then see if it's actually that person.

There are limits to this. In the case of Robert Williams, remember he was the Detroit man who was falsely arrested.

The Detroit police had been told that the match was only a lead, so in that case, at least a disclaimer wasn't enough.

Patrick also explained that human reviewers might not even be able to tell if a fuzzy surveillance photo actually is a suspect recommended by the algorithm.

Face recognition algorithms are now a lot better than humans are.

Wow, that's really surprising. I think, you know, we have the sort of sense that computers aren't that great at recognizing images.

Like today, I had to do a captcha where I was supposed to take out stop signs out of a series of pictures. You know, and it's surprising to me that computers would have difficulty picking out stop signs but be able to recognize faces better than I could.

You've got to remember that there's been an enormous amount of research put into face recognition, both in the academic world and in the commercial world.

And so, you know, because there's money to be made, algorithms have been developed to excel at this task.

My big takeaway from speaking with Patrick was that facial recognition AI works really well, in general, but the mistakes it does make tend to disproportionately impact people of color, and that has led to the false arrests of people like Robert Williams.

[Music]

This isn't you! This is not me! This isn't you either! Like I hope y'all don't think all black people look alike!

Well, the computer says it's you.

So how do we fix these problems for the long term?

Gloria Washington says it all comes down to who is making the algorithm in the first place.

When you look at the actual numbers of the number of skilled workers who work for Google or Facebook or these big tech companies who are black, it's not even close to the percentage of black people who are in the U.S. population—it's less than three percent.

This lack of diversity creates blind spots.

Yeah, there's not enough diverse people at the table to identify the things that are happening with the bias that's going on, and it's continued because it's an old boy and/or frat kind of environment.

So we've allowed it to continue, but they really need to open the door to everyone. If you have input and you are knowledgeable in AI, you should be able to contribute to the algorithms and the techniques that are being built.

Google, in particular, has been struggling recently. According to Google’s annual diversity report, just 1.6 percent of Google staff are black women.

There's definitely no secret that Silicon Valley, in general, has, you know, a diversity problem.

There's no two ways about it.

That's Tiffany Deng, a program manager at Google working in algorithmic fairness and AI.

I think that we should all approach AI with a healthy dose of skepticism at all times.

It can make our lives easier; it can make us safer, but it also has the potential to reinforce negative stereotypes and make things harder for people and exclude people.

Right?

Tiffany pointed out that AI systems tend to reflect the qualities of the people who build them—both the good and the bad.

I think it's really important to understand that AI learns from us; it learns from our behaviors.

I always, you know, like to say that there are no engineers in the background that are very insidious and want to make things like really bad for people—that's not it.

The most likely scenario is that, you know, there just aren't people in the room that can give a different perspective on how things could go wrong.

A Google spokesperson says they have hundreds of people working on responsible AI, and they will continue expanding their work in this area.

But Google also found itself at the center of a firestorm in December 2020 after one of Tiffany's colleagues, a top Google AI ethics researcher and black woman, was allegedly forced out.

Google CEO Sundar Pichai issued a memo apologizing for the way the company handled the case.

The long-term solution is to help more people of color become computer scientists, but there's a more immediate problem: one in four law enforcement agencies has access to these algorithms today.

So what should we do?

[Music]

The committee will come to order!

Now it's May 2019, and the House Committee on Oversight and Government Reform is holding the first of several hearings during the session of Congress to examine the widespread use of facial recognition technology.

The room is packed with lawmakers, academics, and computer scientists who are all concerned with the technology's impact on civil rights and liberties due to the consequences of failures of this technology.

I decided to focus my MIT research on the accuracy of facial analysis systems: vast National Geographic Emerging Explorer Joy Buolamwini.

Joy is the founder of the Algorithmic Justice League, which works to fight bias in machine learning.

These studies found that for the task of guessing the gender of a face, IBM, Microsoft, and Amazon had errors of no more than one percent for lighter-skinned men.

In the worst case, those errors rose to over 30 percent for darker-skinned women.

Given such accuracy disparities, I wondered how large tech companies could have missed these issues.

A year before this hearing, Joy published a study with more than 1200 faces, showing three facial recognition software programs from leading companies misclassified darker-skinned faces, particularly those of women.

Joy's research is among the first to explore the errors with facial recognition technology with regards to race.

Patrick Brother’s study on racial bias in AI was partly inspired by her work.

We all have biases, but technology should be better than humans.

We've all been trained to trust computers to be these accurate, fair, and flawless machines, but without trying, human biases have turned up in the software.

Civil liberties activists say it might be time to rethink those applications.

At a minimum, Congress should pass a moratorium on the police use of facial recognition, as the capacity for abuse, lack of oversight, and technical immaturity poses too great a risk, especially for marginalized communities.

Several cities around the country have taken heed. In 2019, San Francisco became the first major U.S. city to ban the use of facial recognition technology in law enforcement, and soon cities around the country followed, including Boston, Portland, Oregon, and Springfield, Massachusetts.

Oh no, does that mean humans are taking jobs from robots?

Uh, I'm sorry to say it, Natalie, but I think this is a job for humans.

Please explain.

Uh, don't get me wrong—AI is great, and it generally works pretty well, but we need to be careful when the consequences of making a mistake are really high.

Around 10 million arrests are reportedly made each year. There are only a handful of known false arrests.

Well, just one is too many, right?

But those are just the ones we know about. There could be many, many more, and we already know that facial recognition tends to make more mistakes with people of color—almost without anyone knowing it.

Systemic injustices are finding their way into these algorithms.

So for now, I just don't think the world is ready for AI detectives.

I understand.

Goodbye, partner. Now I need to find a new friend.

Hey Siri, do you think AI is biased?

Hmm, I don't have an answer for that. Is there something else I can help with?

I don't think there's anything you can do. [Music]

Facial recognition is everywhere.

For full disclosure, National Geographic Partners—co-parent company the Walt Disney Company—is beginning to test using facial recognition instead of tickets for admission to the Magic Kingdom at Disney World.

2020 has been an especially strange time for facial recognition because so many people are wearing masks or COVID-19.

We've included an article in the show notes about how one San Francisco company is checking whether or not people are wearing masks using AI. [Music]

As we found out in our interview with Patrick Brother, most humans are not very good at identifying faces of people they don't know.

But London police have been working with gifted individuals called super recognizers to help ID suspects in high-profile cases.

I took the test and I scored 8 out of 14, so I'm definitely not a super recognizer, but you might be.

And subscribers can read our cover story, "The Robot Revolution Has Arrived." It's all about the latest advancements in robot hardware and software and where things are going next.

That's all in your show notes; you can find them right there in your podcast app.

Overheard at National Geographic is produced by Jacob Pinter, Laura Sim, and Alana Strauss.

Our senior producer is Carla Wills, who produced this episode.

Our senior editor is Eli Chen.

Our executive producer of audio is Devar Ardulon, who edited this episode.

Our fact checkers are Julie Beer and Robin Palmer.

Our copy editor is Amy Kolzak.

From Teen Aeros Bluey Sound designed this episode, and Hans Dale Sue composed our theme music.

This podcast is a production of National Geographic Partners. Whitney Johnson is the director of visuals and immersive experiences.

Susan Goldberg is National Geographic's editorial director, and I'm your host, Brian Gutierrez.

Thanks for listening, and see you all next time. [Music]