Efficacy of Khan Academy

As a teacher thinking about using a tool inside or outside of your classroom, the first natural question is: well, does that tool work? That's also very important to us here at Khan Academy, with our mission. We don't want just people to use it; we want to feel good that that usage is actually driving real learning.

What I'm going to give an overview of in this video is a very large study conducted in the state of Idaho. It was across the entire state, understanding whether Khan Academy is driving real learning. This is the largest study that has ever been done on a personalized learning tool, and so I'm going to summarize this. We'll also give a link to the study later on in this tutorial. It was done and funded in conjunction with the JA and Katherine Albertson Family Foundation, and so I’m just going to summarize the report here.

So just as a big picture, let me go right down here, and there's a lot of really cool stories from teachers, educators, and district officials here talking about how they implemented the pilot. But the big picture? This was 47 schools, 33 districts, 173 teachers, and several thousand students throughout the state of Idaho. What they did in the study is the students took the NWEA MAP exam at the beginning of the year, they worked on Khan Academy, and then they took the NWEA MAP exam at the end of the year.

The way that efficacy was measured is: well, the MAP exam can see for any student whether they are a student who is more advanced or a student who's remediating. They can see what their expected growth is in a traditional model, and then they can compare that expected growth to what the actual growth was using Khan Academy.

So, let's just cut to the chase and see what the results are. There's a lot of videos that we have on our educator portal that talk about how the different teachers used Khan Academy and what they found valuable, but I'll go straight to the data here for the sake of this video.

If we go down here, this is just the distribution of students by grade level. You can see kind of late elementary and middle schools where the bulk of the students in the study were, but even in third grade, we're talking about over 200 students. So this is a very large-scale study.

This first one says: Mission Completion versus Spring Percentile. Mission completion represents the grade-level personalized learning experiences that the students can pick. So there's a third-grade mission, there's actually an early learning mission, there's a third, fourth, fifth—all the way up to algebra, geometry, trigonometry; there's even a calculus mission.

These are aligned to the important state standards, especially the Common Core standards for K through pre-calculus, and they're aligned to the AP standards for calculus. What you see here is, depending on how much a student completed of their mission over the course of the pilot, it correlates very strongly with how they performed on the MAP exam during the spring assessment.

Now, you might say, "Well, of course, naturally a student who completes more is... that's maybe a student who's a little bit more advanced, and so they're going to be in a higher percentile." I would agree with you; that makes sense. So this right here is not clear evidence of efficacy just yet. But you definitely want to see this trend. If for whatever reason this was a flat line, or if this had a downward slope for some reason, then we would have an issue.

But this is the direction that you want to see. We want to see more evidence; we want to see for a given student, regardless of where they started, that usage of Khan Academy is not driving just expected learning. For all we know, this might be just what you would expect for different students. It's driving learning above and beyond the expectations, and so that's where we see this data down here.

They bullet it out: students who completed 0 to 10% of their mission on average grew as expected. These students barely used Khan Academy, so once again, grew as expected. This means if you are an advanced student and you're learning at a fairly quick pace, you grew as expected; you grew at that quick pace. If you're a student who took a little bit more time, well, you also grew as expected. So when you didn't use Khan Academy, just what you would expect happened.

But then when you see students who had significant usage of Khan Academy, students who completed 40% or more of their mission on average grew 1.5 times more than their expected growth in one year. Students who completed 60% or more of their mission grew an average of 1.8 times their expected growth. So they almost had two years of growth in one year.

Once again, I want to be very clear: this—what we're plotting on the vertical axis here—isn't just growth. This is average actual growth divided by target growth. In a normal model, if you just look at the MAP data on a nationwide basis, whether a student is strong or weak, if Khan Academy had no effect, you would expect just a flatline right over here. You would expect everything here to be just at one.

If Khan Academy was actually less effective than whatever students were doing before, then you'd say, "Well, these students who are spending a lot of time on Khan Academy, then it would be taking away from those other things." You'd see a downward slope. But over here, we see that the students who are actually using Khan Academy deeply—and you can imagine in any large-scale study like this, you have variation in the fidelity to which folks are using Khan Academy, how integrated it is in the classroom—but the classrooms that are using it more deeply, more regularly, are not just seeing expected growth.

They're seeing a significant multiple of what you would expect their growth to be. This is nearly two years of growth in one year of learning time, and this is, once again, at 60% plus. You can only imagine what it would be if students complete 70%, 80%, 90% of the mission, or if it's used in an even deeper way in different classrooms.