Biases in algorithms | Intro to CS - Python | Khan Academy

Algorithms increasingly control many areas of our everyday lives, from loan applications to dating apps to hospital waiting lists. As responsible consumers and now creators of algorithms, we need to think critically about how the success of an algorithm gets measured. What assumptions and biases may be influencing the results? And how should that impact how we use them?

Consider a recommendation algorithm. Your newsfeed doesn't ask you to rate your satisfaction with each result, so it doesn't truly know how you feel. Instead, many algorithms rate success based on more quantifiable indicators. For example, how often users click on a specific result. But this can lead to a false feedback loop, where algorithms are more likely to recommend articles with sensational headlines or cute pictures of cats because they catch users' attention. Even if, when they do open the article, they're extremely dissatisfied with the result. Ultimately, this algorithm is measuring success based on engagement and not actual usefulness. That's not necessarily wrong, but it'll certainly influence the types of results that users see.

So, as you consume technology, keep a healthy skepticism and ask yourself: Why is the algorithm offering me this result? Okay, so humans are biased, but what if I just have AI design my algorithm for me? Well, AIs are trained on data that's created by humans, often content on the internet. Remember that computers see all data, whether it's textual, audio, or visual, as a sequence of numbers. They don't understand what's happening in a photo in the same way that a human does.

Most of the time, AI-generated algorithms are just looking for patterns in the data, whether there's a causal relationship or not. For example, an experimental hiring algorithm found that the AI favored male applicants, downgrading resumes that included terms like "women's" or referenced all-women's colleges. The existing pool of engineers at the company was predominantly male. So when the AI trained on previous hiring data, it had found a pattern that it thought was meaningful: don't hire women.

In practice, AI algorithms can actually amplify historical biases because available training data tends to lag behind the current cultural moment and heavily skews English. This means other cultures and languages are less represented. That's not to say that human-generated algorithms are always better than AI-generated ones. But AI algorithms tend to be less transparent, so it's more important that we hold organizations accountable for monitoring their bias.

As programmers, how can we limit the bias in the algorithms we design? It's impossible to perfectly model the real world in a program. We'll always need to make some assumptions and some simplifications. We just want to make sure we recognize the assumptions that we're making and we're comfortable with how that impacts our results.

Let's evaluate this content moderation algorithm we wrote. What assumptions did we make? Well, here we're favoring older accounts—people who have been on the site for a while. Our algorithm assumes that those users are more trustworthy. Now, we looked at historical data and did find a correlation here, and we don't think account age correlates strongly with any protected class like race, gender, or religion. So, I've decided I'm comfortable with this assumption. I'm willing to accept the slight unfairness toward well-intentioned new users.

What about this word count check? We're assuming posts with a lot of words are less useful. Now, this might have an unfair impact based on the language of the post because some languages tend to need more words to express the same idea. For example, French is often wordier than English. That's a bias I'm perhaps not willing to accept. Our site has a lot of users from all over the world, and I don't want to favor one language over another. So, I might go back to the drawing board with this one and either find a different criteria to use or try to fairly adjust the word count limits based on the language.

This evaluation process is ongoing. As this algorithm runs on our site, we want to monitor trends in which posts are featured and which are flagged, and adjust the algorithm accordingly in response to new data.