Khan Academy Ed Talks featuring Brooke Mabry - Wednesday, December 16

Hi everyone, Sal Khan here from Khan Academy. Welcome to our Ed Talks Live, this new flavor of homeroom that we're doing. We have a very exciting conversation with Brooke Mabry about learning loss, summer slide, and actually our partnership with NWEA as well.

Before we jump into that, I'll give my standard announcements. A reminder that we are philanthropically funded, and we can only do the work we do through philanthropic donations from folks like yourself. So if you're in a position to do so, please go to academy.org donate. Donations of all levels make a huge, huge difference. Every dollar donated to Khan Academy creates many hundreds of dollars of social impact, one of the most scalable ways that you could create social impact in the world.

I also want to give a special shout out to operations that have stepped up, especially during the COVID crisis, realizing that as the load on Khan Academy has increased by a factor of two or three, our costs have as well. We've wanted to accelerate a bunch of programs to support parents, teachers, and students. So, special thanks to Bank of America, Google.org, AT&T, Fastly, and Novartis for stepping up and helping us fill this gap. But once again, we still need help, especially as we go into 2021, to fill that gap.

Last but not least, a reminder to everyone that there's a version of this live stream that you can get in podcast form: Homeroom with Sal, the podcast. So check that out wherever you get your podcasts.

With that said, I'm excited to introduce Brooke Mabry. Am I pronouncing it right? Is it Mabry or Mayberry?

No, you got it right, Sal. Mabry.

All right! Well, Brooke, thanks for joining us. You are the head of strategic content design coordinator at NWEA, and NWEA is one of our closest partners here at Khan Academy. But why don't we, for those folks who don't know what NWEA is or what a strategic design coordinator is, maybe that's a good place to start.

Absolutely! So, NWEA is a non-profit educational organization that's really grounded in our mission of partnering to help all kids learn. We work in all 50 states and 140+ countries around the world, providing assessment solutions and educational solutions to really think about how we can harness student learning and grow those students as far forward as possible.

My role in the organization is on the professional learning design team, and so my work is really around how do we create content that supports teachers helping their students to learn. How do we help them use the assessment data that they get from our flagship product, MAP Growth? But also, how do we deepen their formative assessment practices and their assessment literacy? So, specifically around strategic content, that's really our largest partnerships. We're both involved together with Clark County School District in Las Vegas, and we also work with New York City Public Schools and Chicago Public Schools. So I'm really in there to help support our consultants and our teams who partner with some of our largest districts across the nation.

Yeah, and just to remind a lot of folks, NWEA is the administrator, as you just said, of the MAP Growth assessment, which 20% of all kids in America take. And if you look, you know, there's been over 50 efficacy studies on Khan Academy, and most of them have been using your assessment, the MAP Growth assessment. It's a real differentiator; it's really the first and still the major assessment that can measure growth. So a lot of assessments just measure where you are or whether you're at grade level, but this measures where you are and how much you grew from one point to another point and how does that compare to how much other people grow, which is really valuable because we know there might be a sixth grader who started the year at a fourth grade level and got to a fifth-grade level. They had a year of growth. But on a sixth-grade assessment, they might not see a lot. Likewise, you might have a sixth grader who starts the year at a seventh-grade level and stays at a seventh-grade level. They'll do great on the grade-level assessment, but they saw no growth, which would be a problem. So that's what's really cool about the MAP Growth assessment and why it's so powerful to use it for things to measure efficacy on things like Khan Academy.

And as we just saw on the screenshot, we have this partnership where now the MAP Growth assessment acts as a placement and a diagnostic for personalized practice on Khan Academy, this MAP Accelerator, which we're doing with school districts like Clark County, which is Las Vegas, and many others. Many hundreds of thousands of students are now using this inside of their classroom in conjunction with their teachers. And so it's connecting the assessment. The assessment helps figure out what you need to work on on Khan Academy, and then because everything is linked, you can figure out how much that Khan Academy work has driven, hopefully, and we expect better than expected growth.

So it's a really exciting partnership that we're doing together. So, you know, we're in very unusual circumstances right now. I think, what is it, month eight or nine of COVID? I know NWEA has been studying what the effect of COVID has been or what it could be and also how people have been transitioning or not to distance learning. What have you been seeing both on the research side in terms of how it's affecting students' performance and other ways that people are trying to cope?

Well, if you've read our latest research brief, we're cautiously optimistic about what we found. You know, we made some projections back in the spring based on our summer learning loss data that we have accumulated over the years. But when we looked at this fall data and really compared what was happening, we didn't quite see the same slides that we projected. That's really good news! Reading was around typical of where we typically see students coming in in the fall. We did see five to ten percentile points of a drop in the math area, but it's not as deep as we thought it would be.

So we really want to give a shout out to teachers and say you really stepped up, and you really adapted well and found ways to lean in and make sure that that loss was not as large as we thought it might be. So that's the initial results. But we do still have some places where we don't fully understand what's happening in the data because there are missing students. We do have about a quarter of students that aren't typically represented in our data and not there, and what we have found is those are our students who come from the most underserved communities. Students who tend to be students who live in impoverished areas, students who tended to start at a lower percentile to begin with, and are students of color. So we don't have any real conclusions; our data is too early right now to make any conclusions in that area. But we do want to say that right now we're seeing some positives and some things to hang our hope on, but we also know that we want to understand deeper what's happening in some of our places where students have tended to be underserved.

So to make sure I'm understanding this, what we're seeing, at least based on the data, is you're seeing the reading loss is not as much as expected. If anything, it looks like your data is showing that kids are tracking similar to what they would have done in a normal year; math, there is a bit of a regression compared to a normal year. And then there's a big asterisk on all of this because the number of students that you can assess, there's been some percentage of the population, whether it's five or ten percent of the population, that is just not engaging right now over distance learning. You couldn't assess them right now. But there's a lot of reason to believe that those are the kids that were already maybe facing not as fast growth as we would hope, and the fact that they haven't been engaged in their schools would argue that they're probably the worst off in terms of growth of their learning, and they're probably atrophying. Because that's not in the data, it might be making the data seem a little bit less negative than it might be.

That is very possible, and that's absolutely where we're standing right now, hoping that we'll have a more complete data set as we progress through the year.

Are you seeing that? Are you seeing that, you know, this five or ten percent of students that have just not been showing up? Are districts getting better at finding them? When you do your January MAP Growth administration, do you think you're going to get more of those kids in the assessment?

Well, I think it's that. But it's also some of our partners decided not to test remotely. They were busy trying to uplevel their remote instruction and just didn't feel that they were ready to administer the test remotely. We had a large turnout for remote testing, but we did have fewer districts who made a choice not to test their whole district. But you're right; there is a percentage of students out there who have not been engaged in school and have not obviously been engaged in the assessment portion too.

I think anecdotally, you know, we're hearing from organizations that there's a real struggle. I think the technological struggle—if you look at how we've defined interrupted learning to be about the physical space, the technological space, the social emotional, the cognitive—but that technological divide is real in our country. I know that that is certainly something that the district that my son belongs to. I know that I have teacher friends who've said that they have difficulty getting a hold of students, and students have difficulty accessing the internet. So I think that there are a number of things that are happening that are preventing those pieces.

But we really have to lean in and figure out how we can address those systemic inequities in the platforms and tools that students have access to if we're going to continue this remote learning. And so the sense is maybe the January administration at least more schools will be ready to do this type of administration. We still don't know what might happen with some of the kids who've disengaged, and that's kind of a deeper problem that everyone is facing right now.

You know, what are the things outside? You just mentioned, essentially, the digital divide access, you know how that's a clear issue that we've talked a lot about on this forum. What are the other things that you're seeing that are really tough for the education system right now, and what are you seeing that might be categorized as silver linings?

Yeah, I think one of the challenges that we often hear from the partners that we work with, teachers who are in the field when we're doing our workshops, is they often talk about the way the day is structured. When you're in the physical space in the classroom, it is so much easier to be able to run your small groups and to be able to bounce around to the different students who need you. And when you're in that virtual space, it is much more challenging. Especially when you think about the kinds of tools that we have and whether or not we're teaching whole group or small group and how you have that set up for your students to meet.

So, I think that the scheduling is one thing that has certainly presented a challenge to teachers. And I think the other piece is engagement, really looking at— we're asking our students and our teachers to engage in ways that haven't been the norm.

So there's this comfort level that we also have to address in this school climate and community. It's one thing to build community when you're in person with all of your learners, but to build community when everyone is sitting behind a screen and trying to facilitate those connections presents a totally different challenge.

I think the silver lining there is that what we're finding is that those in-person techniques work virtually as well with some modifications. But the premise of creating that strong connected learning team is still the same thing that we want to go for. And so we know that the foundation of all learning is strong relationships. I'm hearing teachers say over and over the return on investment that I'm getting for the way that we're building community and climate is really paying off, and now I can really dive into some of those deeper learning pieces. So teachers are really investing on this, you know, they're on video conference, whatever, but they're making sure that there's time for them to get to know each other, build trust between the students and the teachers, and the students with each other. And that's what's one of the things that's paying dividends, which makes a ton of sense.

Are you seeing pockets of teachers doing things exceptionally well right now? Do you think there are certain elements of what's going on right now that we should try to retain post-COVID?

You know, one of the things that I think really stands out is there’s been so much student choice and voice that I've seen in my own son's education as he has been remote for part of the school year as well. And when I'm talking to teachers across the country, when I'm partnering... So I think one of the silver linings here is that education doesn't have to look the same that it always has.

I think we've been given this opportunity to innovate and to see things differently and to know that perhaps when we come back to the brick and mortar in a more permanent way, that we can continue to honor the choice and voice that we've allowed students to have and really empower student-directed and self-regulated learning in a way that continues to meet students where they are, but also gives them that motivational factor to engage in ways that maybe they haven't before. And I think that's the really exciting piece that I'm seeing is that a lot of teachers are saying, "You know what? We're learning some really good lessons about what works for students," and we want to make sure that we want to capitalize on these and continue these practices when we get back to what used to be the normal way.

So there's a sense that, you know, some of what COVID's done is it's maybe unbundled or personalized experiences a little bit more, whether it's the teachers doing it or sometimes families have done it on their own. And obviously, I believe strongly in this as well; if we can keep a lot more of that momentum around personalizing to the needs of students, allowing them to fill in gaps, learn that they're...

Yeah, I know you've written a lot about, you know, the zone of proximal development, which, you know, it's just the idea that you should be learning the things that are just ready for you to learn— not too easy for you, not too hard, just right.

You know, I'm curious about the assessment. You know, I think a lot of folks, especially people who aren't super deep in the education world, might be confused. Like, there's so many assessments! You know, we have interim assessments, which is what a MAP Growth is, which can measure growth; you do it three times a year. Then you have your state assessments, which are usually done at the end of the year and kind of give a sense. And the state assessments are oftentimes the things that are used to measure school performance and all of that.

I'm curious how you would tell everyone listening how these assessments fit together and where you think assessment is going to go. You know, you all have been doing digital administration of the assessment because you've, well, it's always been digital, but you've had to do distance administration of the assessment this year because of COVID. Do you think some of that's going to stick? What's assessment going to look like in five years?

That's a really great question, and I think that it's something that we're constantly thinking about too. You know, we were one of the first organizations to ramp up and offer our partners the opportunity to assess remotely. We know how valuable the assessment is because, as you said, an interim assessment is really a snapshot in time.

I like to think of those— I used to say Polaroids, Sal, but some of my teachers I'm working with are getting younger and younger, so I have to say like it's an Instagram post, right? So you line up three Instagram posts, and you can look at changes over time. But at that snapshot in time, we can really see where students are ready to learn relative to what I'm trying to teach.

And so when you think about MAP Growth in that interim space, we're an instructional readiness assessment, not a mastery assessment, which is what your state summatives are. State summatives are mastery assessments. When we look at giving a mastery test to students, we're looking at them to get high levels of accuracy— so as many right as they can— because a mastery test is all about certifying what you know and can do relative to the standards that are being assessed.

An instructional readiness instrument, which is what MAP Growth is, is the opposite of that. We're actually looking for every single student to get about half the items right. And people, that blows their minds. They're like, "Why would you design a test where students are only going to get about 50% right?" That's where the zone of proximal development is.

When we think about instructional readiness, what we're really trying to find is not high levels of mastery and not low levels where they have no background knowledge; we're trying to find that sweet spot right in the middle where they have some background knowledge but mastery hasn't emerged yet. So we can use that information to create a plan, a pathway forward to mastery.

Because what we know from Lev Vygotsky's work, which is where we got this idea of the zone of proximal development, is that when we teach in that zone, what we're doing is we're helping move a student from where they are, closing that metaphorical gap between what they can do and what they can't do yet.

And so with an interim assessment, we at NWEA have linking studies where we look at a group of students in a state who've taken both the state summative and the MAP Growth assessment, and we do a correlational study where we can see if this is how they score on MAP Growth, how do they tend to score on the state assessment? And then we can provide information to teachers to help them understand, based on a student's MAP Growth score, what kind of trajectory they're on relative to that state summative.

So in a way, the MAP Growth interim assessment becomes a predictor where I can use that information to change a student's trajectory. So the interim assessment is such that it helps us to monitor growth, as you said.

But I like to think about how John Hattie talks about assessment, and he says we should look at assessment as information about our impact. And so if a student is growing, we really need to figure out what is it that we're doing that's helping get that growth and how can I replicate that with more and more students? But if the growth isn't there, why not? What are those factors that are preventing the most growth possible forward, and how can we really get in there and change up things before we get to that state summative?

If you think about a balanced assessment system, we need summative assessments to understand how students are performing relative to those grade-level expectations. We need interim assessments that help us monitor progress along the way, but it's really formative assessment practices that drive our day-to-day instruction.

That's one thing that doesn't get spoken about enough when people talk about NWEA because they're so familiar with our flagship product, MAP Growth, that they don't realize how deep we are in the formative assessment space. And so that's where we really get in there and help teachers think about their instructional practices and what am I going to do with that data that I get from MAP Growth and my classroom formative data to keep that learning moving forward every day between interim assessments.

So it's the macro down to the micro.

You know, what's interesting— I know we've talked a lot about this as we're kind of visioning the future. You know, you're using the interim assessment, and just explain to folks why you actually need kids to get things wrong. Is that you're zeroing in; it adapts. You know, you might start a student at, let's say they're a sixth-grade student, you might start them at a fourth-grade concept. If they get it right, you can give them a little harder concept, a little harder. Maybe you're getting to more advanced sixth-grade type concept. But then they get something wrong, and then you give them an easier question, and then they get a few right, then they get something wrong.

So you're zeroing in on exactly where that learning edge is, that zone of proximal development. And that's also what allows you to measure growth beyond just one grade level. And to your point, there's formative— and you know, these when I first learned these words, I was like, wow, there's a lot of types of assessments! But formative is really just another way of saying assessment in the spirit of helping you learn how well the kids are learning in the moment so that the teachers or whoever else is trying to, even the students, can better diagnose what can be done to make sure they're learning it well.

What's interesting with the MAP Accelerator that we're doing together, obviously, is using the interim assessment to figure out a personalized learning plan that the student is ready for. But then, while they're practicing using the MAP Accelerator on Khan Academy, that actually also helps inform the teacher on what they need help in.

So it is doing some of this kind of a shorter cycle loop, and then, you know, every three or four months, you then get another interim assessment to understand overall how much that's been driving the growth, and then you can replace—I say that not replace, replace students again.

I mean, it's a fascinating space of where we're going with assessment! You know, we're getting a bunch of questions here. One is— well, you know, there are families that are worried about— actually, there was a question where it went about parents with students. You know, there's reports that we're seeing an unusual number of kids who are failing this year, even kids who typically were getting A's and B's before. Any advice you would have for parents and I guess a broader question and students and teachers is, you know, how do we keep the kids engaged right now, and how do we keep their motivation high so that these types of things don't happen?

Boy, what a loaded question, Sal! I have some really strong feelings about grades, and I think that that’s the first question that you have to ask. What is that grade measuring? Is that grade measuring progress toward mastery in the subject, or is it measuring completion of assignments?

And so really having conversations about what grades mean with teachers is, I think, really important, and that's the place that teachers can create clarity with their students and with the families that they serve. Is to really help them understand what a grade means and what they're trying to collect.

And so that's one piece. But there's the motivational piece. Our students have to understand— and this is part of that formative assessment cycle— what it is that our expectations are. So what are we aiming for in the learning? And really give them a pathway for understanding where we're going but also understanding where they are relative to where we're trying to go and then strategies for closing the gap between those two things.

I think when students are empowered to see the trajectory, they're empowered to figure out where they are, and then they get the supportive practices in the classroom— supportive assignments, scaffolding if you will. I've written extensively about that on our NWEA blog, Teach Learn Grow.

Then I think that's where the motivation comes from. A lot of learned helplessness with students comes when they've tried and failed, tried and failed, and tried and failed. And so I like to point back to the educational system sometimes and say if we have—I don't want to say we have unmotivated learners; I think that we always have motivated learners. They may just not be motivated to do what we want them to do.

So we have to really tap in and ask ourselves what are we doing to help students see the return on investment to be successful. Because that learned hopelessness happens when students don't see that return on investment or they fail over and over after trying. So it's important to get early and easy wins. Students have to experience success. The greatest predictor of future success is past success. Success breeds success.

So as teachers and as parents, we have to help our students get a win, and then they know what success feels like, and then they can crave it. I always like to say that chocolate and red wine are two of my favorite things, Sal, but I never knew to crave those things until I tried them. And so the same thing is true for students. We have to give them a little taste of success before they'll want it on their own.

And we might have to spoon-feed it for a little while, and then eventually they’re going to take the spoon away, and they’re going to feed themselves because they know what it feels like and they know what it takes to achieve it.

Now, what I love is your statement about there's no such thing as unmotivated students. I hear that all the time. "Oh, this might be good for motivated students, but what about all of these unmotivated students?" It's like it's in their DNA or something! There's not a person on the planet who is not— you know, I mean the video game industry has figured out how to motivate people! There are a lot of things, and I haven't met a four- or five-year-old who is incredibly curious and motivated. You have to fight them off! You know, they want to explore everything; they want to ask you ten questions. You're like, "I'm done answering your questions! Please give me a break!"

Every four- or five-year-old in the world... And so what your point is is that they can become demotivated in certain domains, but once again, it's not quote "their nature." It's just a kind of learned helplessness. Ideally, that doesn't happen in the first place, but there are ways to deprogram it by giving them some wins.

You know, we have a lab school, Khan Lab School, and there was an open house that was— and the teachers, the parents are always— the prospective parents are always like, "Wait, this is a school without grades?" You know? And I remind them, "No, no, we have grades! We don't call them A, B, C, D, F. We could have, but we call them advanced, mastery, mastery approaching, mastery." Because they're for mastery!

And the key is that there's always opportunity. If a student is at approaching mastery, which you can kind of view as a B or even a B minus, that there's always opportunities for them to improve it. So they never get the message that it's somehow their innate, you know, beingness or "seenness" of themselves.

But it's just, you know, that's just where they are right now. If you're practicing basketball, you're a 60% free-throw shooter. It doesn't mean you're doomed to never be better! It's when you keep practicing, you get to 80%, get to 90%. So I couldn't agree with that more.

You know, I was just on CBS this morning yesterday, and everyone said, "Hey, this McKinsey study about all these kids failing classes." I was like, "The key thing is we have to give opportunities for these kids, especially because— especially because they're probably failing it based on this kind of more completionist mindset. Are they completing the assignments or not?"

We have to give these kids opportunities, frankly all kids, when we get back for them to make up the work and show that they have mastery, so that it doesn't do the opposite of what you just described.

I agree completely, and I think what we've got to do as an educational system in the United States is challenge this idea of the average as well. For me, I think that zeros are problematic, and I think averages are problematic. And I think that what we're really trying to communicate is mastery.

Then we should be thinking about then how do our grades reflect mastery? And there's an interesting research study out there that actually shows that then the minute you put a number grade on something, the learning stops for students. So I always, when I'm working with teachers, really encourage them, as long as students are in the learning cycle, withhold that number. Think about what kind of descriptive feedback— what kind of actionable feedback can you provide that will feed the learning forward?

And how can you make time in your classroom for students to use that feedback? I hear all the time from teachers, "Well, I write tons of things on kids' papers, but they don't ever use— they don't ever use it." My question is, one, are you also including a grade? And two, are you making explicit time for students to dig in and improve? And in what ways?

So if you're out there and you're struggling— as a student, or you are a parent who has a child who's struggling, or you're a teacher who wants to dig in and think about these things— that's where I'd ask you to start: to really start thinking about what is it that I believe about grades? What is it that I believe about student learning? What am I trying to accomplish, and how can we do that in a way that empowers the learner to have more of the locus of control? And how do we empower the learners at goals that matter and are meaningful and attainable?

And then how can we be a support for that? And I really think that's our job as parents, and I think that's our job as teachers, is to help cultivate these individuals who will eventually be empowered and can self-regulate and can take charge and have the kinds of futures that they want for themselves.

No, it's a very, very powerful idea. Well, Brooke, these conversations always go much faster than I expect. You know, thanks so much for joining, and I look forward to going on this journey with you and NWEA because I think together we can help a lot of parents, teachers, and students stay in that zone of proximal development, understanding what's working, not working, make sure kids have a chance to master concepts, not just complete a few things without mastery. So it's really been an honor to talk to you about this.

Thank you, Sal. I appreciate the work, and I appreciate the partnership that we have.

Thank you! Well, everyone, thanks for joining. Hopefully, you enjoyed that conversation. Some really interesting things to think about. Actually, I'm going to think a little bit about that notion that as soon as you see the number, it might in certain ways stop some learning. We might have to modify some things on Khan Academy. We'll see about that.

We'll see you tomorrow for a show. We're going to have Beth Gonzalez, who's the Assistant Superintendent of Detroit Public Schools, and you can imagine there's a lot of interesting things to talk about when you're talking to folks who are administrators of large public school systems like Detroit. So start thinking of your questions, and I will see you tomorrow at 12 p.m. Pacific time, 3 p.m. Eastern.