yego.me
💡 Stop wasting time. Read Youtube instead of watch. Download Chrome Extension

Examples of bias in surveys | Study design | AP Statistics | Khan Academy


4m read
·Nov 11, 2024

We're told that David hosts a podcast, and he's curious how much his listeners like his show. He decides to start with an online poll. He asks his listeners to visit his website and participate in the poll. The poll shows that 89 percent of about 200 respondents love his show. What is the most concerning source of bias in this scenario?

And, well, like always, pause this video and see if you can figure it out on your own, and then we'll work through it together. Let's think about what's going on. He has this population of listeners right over here. I'll assume that the number of listeners is more than 200, and he says, "Hey, I want to find a sample. I can't ask all of my listeners. Who knows? Maybe he has 10,000 listeners. They don't tell us that, but let's say there's 10,000 listeners here, and he says, 'Well, I want to get an indication of what percent like my show, so I need a sample.' But instead of taking a truly random sample, he asks them to volunteer. He asks his listeners to visit his website."

So that's classic volunteer response sampling. This is non-random because who decides to go to his website and listen to what he just said? He maybe even has access to a computer. That's not random. In fact, the people more likely to do that are those who already like David or like to listen to what he tells them to do. The people, the listeners who aren't into David, or don't want to do what he tells them to do, well, they're unlikely to say, "Oh, I'm not really into David, and I don't like him telling me what to do, but hey, I'm going to go to his website anyway. I'm going to fill out that poll." That's less likely.

Or you might get extreme cases. For example, people who really don't like him might say, "I'm going to definitely go there." But in this case, I would say that it's more likely your fans are going to do what you ask them to do and go to your website and spend time on your website. Because of that, that 89 percent is probably an overestimate. Eighty-nine percent is probably an overestimate of the number of listeners who really love his show because you're more likely to get the ones who love him to show up and fill out that actual survey.

Now, these other forms of bias—response bias—this is when you're asking something that people don't necessarily want to answer truthfully, or the way that it's phrased might make someone respond in a biased way. Classic examples of this are like, "Have you lied to your parents in the past week?" or "Have you ever cheated on your spouse?" or "Do you smoke?" Any of these things that people might not want to answer completely truthfully or they might be hiding from the world. They might not just want to answer that truthfully on a survey, and so you're going to have response bias. But that's not the case right over here.

And undercoverage is when the way that you're sampling, you're definitely missing out on an important constituency. You know, voluntary response; we're likely missing out on some important constituencies, on some people who might not be into going to your website. But undercoverage is where it's a little bit more clear that that is happening.

Now let's do another case. Let's do another case—maybe an alternate reality—where David's trying to figure this out again. He's still hosting a podcast, and he's still curious how much his listeners like his show, but he tries to take a different sample. He decides, in this case, to poll the next hundred listeners who send him fan emails. They don't all respond, but 94 out of the 97 listeners polled say they loved his show. What is the most concerning source of bias in this scenario?

Well, this is a classic, "Hey, I have a group. I have a sample sitting in front of me. It's in my inbox on my email. Let me just go to them." Isn't that convenient? So this is classic convenience sampling, and this isn't just like, "Hey, these are the first hundred people to walk through the door," and there's a lot of times you could argue why that might be not so random, but the next hundred listeners who sent him fan emails—these are, this is a convenient sampling.

And the sample that you happen to use out of convenience is one that's going to be very skewed to liking you. So once again, this is overestimating the percent that love his show. Now, non-response is when you ask a certain number of people to fill out a survey or to answer a questionnaire, and for some reason, some percent do not fill it out, and you're like, "Well, who were those people? Maybe they would have said something important, and maybe their viewpoint is not properly represented in the overall number that actually did fill it out."

And there is some non-response going on here. He asks a hundred people who sent fan emails to fill out the survey to say whether they love it or not. Ninety-seven fill it out, so there were three people who did not fill out the survey. So there is some non-response going out here that would be a source of bias, but it's not the most concerning. You know, right over here, they're asking us to fill out the most concerning source of bias, and the convenience sampling is definitely the biggest deal here.

There were three people who didn't respond, but that's not as big of a deal. Voluntary response sampling—well, he didn't ask people, like in the last example of, "Hey, if you can go here and fill it out." I guess there is actually—actually no, I'll take that back. There is a little bit of voluntary response here where he goes to these hundred people and he asks them to respond. And so you have the 97 people who choose to respond.

But once again, that could be a source of bias, but most of the 97 of the 100 are responding, and once again, the most concerning thing is the convenience sampling, which will once again, based on this sample that he's happening to use out of convenience, is going to be a very significant overestimate in terms of representing the entire population of his listeners.

More Articles

View All
15 Expensive Things That Are NOT Worth the Money
You dream about becoming rich so you can afford everything you ever wanted, only to find out that you hate having to take care of so many things. Most expensive things are just a clever way to separate rich people from their money. If last Sunday, we look…
What Lies Beneath | Primal Survivor
Oh my God, it is a blue ringed octopus! See those beautiful blue circles? Those aren’t to make it look pretty; that’s warning coloration. Believe it or not, this tiny little creature is one of the most venomous marine animals in the entire world. The blue…
10 Things I Wish I Knew Before Investing
Hey guys, welcome back to the channel. In this video, I’m going to be going through 10 things I wish I knew before I started investing, so hopefully we can get through these 10 in around about 10 minutes. So, time is on, let’s get stuck into it. The firs…
Existence theorems intro | Existence theorems | AP Calculus AB | Khan Academy
What we’re going to talk about in this video are three theorems that are sometimes collectively known as existence theorems. So the first that we’re going to talk about is the intermediate value theorem, and the common thread here, all of the existence t…
The Rescue | Official Trailer | National Geographic Documentary Films
Breaking News. Right now, out of Thailand. Rescue teams are working through the night to save 12 boys and their coach, trapped inside a cave. The monsoon had come early. The conditions in the cave were impossible. There was a very strong feeling that the …
Polynomial special products: difference of squares | Algebra 2 | Khan Academy
Earlier in our mathematical adventures, we had expanded things like ( x + y \times x - y ). Just as a bit of review, this is going to be equal to ( x \times x ), which is ( x^2 ), plus ( x \times \text{negative } y ), which is negative ( xy ), plus ( y \t…