yego.me
💡 Stop wasting time. Read Youtube instead of watch. Download Chrome Extension

Introduction to power in significance tests | AP Statistics | Khan Academy


6m read
·Nov 11, 2024

What we are going to do in this video is talk about the idea of power when we are dealing with significance tests. Power is an idea that you might encounter in a first year statistics course. It turns out that it's fairly difficult to calculate, but it's interesting to know what it means and what are the levers that might increase the power or decrease the power in a significance test.

So just to cut to the chase, power is a probability. You can view it as the probability that you're doing the right thing when the null hypothesis is not true. The right thing is you should reject the null hypothesis if it's not true. So it's the probability of rejecting your null hypothesis given that the null hypothesis is false. You could view it as a conditional probability like that, but there are other ways to conceptualize it.

We can connect it to type two errors, for example. You could say this is equal to one minus the probability of not rejecting the null hypothesis given that the null hypothesis is false. This thing that I just described—not rejecting the null hypothesis given the null hypothesis is false—that's the definition of a type 2 error. So you could view it as just the probability of not making a type 2 error or one minus the probability of making a type 2 error. Hopefully, that's not confusing.

So let me just write it the other way. You could say it's the probability of not making a type 2 error. So what are the things that would actually drive power? To help us conceptualize that, I'll draw two sampling distributions: one if we assume that the null hypothesis is true and one where we assume that the null hypothesis is false, and the true population parameter is something different than the null hypothesis is saying.

For example, let's say that we have a null hypothesis that our population mean is equal to, let's just call it mu1, and we have an alternative hypothesis, so H sub A, that says, "Hey, no, the population mean is not equal to mu1." So if you assumed a world where the null hypothesis is true—so I'll do that in blue—what would be our sampling distribution?

Remember, what we do in significance tests is we have some form of a population. Let me draw that. You have a population right over here, and our hypotheses are making some statement about a parameter in that population. To test it, we take a sample of a certain size, we calculate a statistic—in this case, we would be the sample mean.

We say, "If we assume that our null hypothesis is true, what is the probability of getting that sample statistic?" If that's below a threshold, which we call a significance level, we reject the null hypothesis. In that world, that we have been living in, one way to think about it in a world where you assume the null hypothesis is true, you might have a sampling distribution that looks something like this: the null hypothesis is true, then the center of your sampling distribution would be right over here at mu1.

Given your sample size, you would get a certain sampling distribution for the sample means. If your sample size increases, this will be narrow, or if it decreases, this thing is going to be wider. And you set a significance level, which is essentially your probability of rejecting the null hypothesis even if it is true. You could even view it as—and we've talked about it—you can view your significance level as a probability of making a type 1 error.

So your significance level is some area. Let's say it's this area that I'm shading in orange right over here; that would be your significance level. So if you took a sample right over here and you calculated its sample mean, and you happen to fall in this area or this area or this area right over here, then you would reject your null hypothesis.

Now, if the null hypothesis actually was true, you would be committing a type 1 error without knowing about it. But for power, we are concerned with a type 2 error. In this one, it's a conditional probability that our null hypothesis is false. So let's construct another sampling distribution in the case where our null hypothesis is false.

Let me just continue this line right over here, and I'll do that. So let's imagine a world where our null hypothesis is false, and it's actually the case that our mean is mu2. Let's say that mu2 is right over here, and in this reality, our sampling distribution might look something like this. For once again, it'll be for a given sample size. The larger the sample size, the narrower this bell curve would be.

So it might look something like this. In which situation—in this world—we should be rejecting the null hypothesis. What are the samples in which case we are not rejecting the null hypothesis, even though we should? Well, we're not going to reject the null hypothesis if we get samples in. If we get a sample here, or a sample here, or a sample here—samples where, if you assume the null hypothesis is true, the probability isn't that unlikely.

The probability of making a type two error when we should reject the null hypothesis but we don't is actually this area right over here. The power—the probability of rejecting the null hypothesis given that it's false—so given that it's false—would be this red distribution. That would be the rest of this area right over here.

So how can we increase the power? Well, one way is to increase our alpha, increase our significance level. If we increase our significance level, say from that—remember the significance level is an area—so if we want it to go up, if we increase the area, it would look something like that.

Now by expanding that significance area, we have increased the power because now this yellow area is larger. We've pushed this boundary to the left a bit. Now you might say, "Oh, well, hey, if we want to increase the power, power sounds like a good thing. Why don't we just always increase alpha?" Well, the problem with that is if you increase alpha—so let me write this down—so if you take alpha, your significance level, and you increase it, that will increase the power.

That will increase the power, but it's also going to increase your probability of a type 1 error. Because remember, that's one way to conceptualize what alpha is—what your significance level is. It's a probability of a type 1 error. Now what are other ways to increase your power? Well, if you increase your sample size, then both of these distributions—these sampling distributions are going to get narrower.

If these sampling distributions—if both of these sampling distributions get narrower, then that situation where you are not rejecting your null hypothesis, even though you should, is going to have a lot less area. There's going to be, one way to think about it, there's going to be a lot less overlap between these two sampling distributions. So let me write that down.

So another way is to, if you increase n, your sample size, that's going to increase your power. This, in general, is always a good thing if you can do it. Now other things that may or may not be under your control are well—the less variability there is in the data set, that would also make the sampling distributions narrower, and that would also increase the power.

So less variability—and you could measure that as variance or standard deviation of your underlying data set—that would increase your power. Another thing that would increase the power is if the true parameter is further away than what the null hypothesis is saying. So if you say, "True parameter far from null hypothesis," what it's saying, that also will increase the power.

These two are not typically under your control, but the sample size is, and the significance level is. There's a trade-off, though. If you increase the power through that, you're also increasing the probability of a type 1 error. So for a lot of researchers, they might say, "Hey, if a type 2 error is worse, I'm willing to make this trade-off. I'll increase the significance level." But if a type 1 error is actually what I'm afraid of, then I wouldn't want to use this lever.

But in any case, increasing your sample size, if you can do it, is going to be a good thing.

More Articles

View All
Finding decreasing interval given the function | Calculus | Khan Academy
Let’s say we have the function ( f(x) = x^6 - 3x^5 ). My question to you is, using only what we know about derivatives, try to figure out over what interval or intervals this function is decreasing. Pause the video and try to figure that out. All right,…
Comparing P-values to different significance levels | AP Statistics | Khan Academy
What we’re going to do in this video is talk about significance levels, which are denoted by the Greek letter alpha. We’re going to talk about two things: the different conclusions you might make based on the different significance levels that you might …
Climbing Asia’s Forgotten Mountain, Part 1 | Nat Geo Live
It was harder than we anticipated and it was much, much colder. We’re a team of six people. Our goal is to determine what the highest peak in Burma is and then climb it. Like to solve this fantastic geographical mystery. It never let up, just taken down t…
2015 AP Physics 1 free response 1a
Two blocks are connected by a string of negligible mass that passes over a massless pulley that turns with negligible friction. It is shown in the figure above. We see that the mass M2 of block 2 is greater than the mass M1 of block 1. The blocks are rele…
Types of RICH PEOPLE
You know, Alex, so many people think that rich people are all the same, but it’s just not quite true. Not all wealth is created or spent equally. So today, we’re talking about the 15 types of rich people. Welcome to Alux, the place where future billionair…
Cosine equation algebraic solution set
The goal of this video is to find the solution set for the following equation: negative 6 times the cosine of 8x plus 4 is equal to 5. And like always, I encourage you to pause this video and see if you can have a go at this before we do it together. A re…