How to take AI from vision to practice. Part 1
Welcome everyone! We are so excited for you to be here, uh, to join this amazing conversation. My name is Danielle Sullivan. I'm the senior regional manager of the Northeast Partner District Partnerships at Khan Academy, and I'm really thrilled, uh, that you get to learn from Dr. Kristen D. Bo, which I will let you say a little bit more about yourself.
Thanks! Hi everyone, and I will also extend my welcome. So for folks who don't know me, I'm the chief learning officer here at Khan Academy. What does a chief learning officer do? I lead our content team, our product team, our design team, our community support team, and our assessment team. So lots of teams. But most of what you see as you interact with Khan Academy has mostly come from, uh, the designs and the efforts of a lot of the folks that I lead.
But my background is in educational psychology. I started out initially actually as a school psychologist, but have spent about the past 20 years building and designing learning technologies, building in what we know about how people learn into those digital experiences.
All right! Well, let's get right into it. So I want to start off with how your perspective on AI for education has changed over the last two years.
Yes, so many of you probably also had this crazy experience when you first started interacting with ChatGPT or any of those kinds of similar things, where suddenly AI, that had been a thing that you kind of knew, was something that all of a sudden burst into the forefront. I had actually been working on a variety of different AI kinds of projects, thinking about how these might work.
Interestingly, I was actually part of a project in 2015, where I was working—um, I was at PI at the time, and I was working in partnership with IBM, if you all know Watson. We were trying to figure out if Watson could be a tutor for students. At the time, the technology required us to train it on hundreds of student examples on every single question we wanted to ask it, and eventually, we decided, you know what? This isn't going to work; we can't do this.
So when we got access to these new generative AI models that suddenly can have conversations with students, that for me was a big eye-opener and started me thinking, "Hey, maybe we can have some of this conversational dialogue as part of the AI-based tutors that we have." And that’s really exciting as we think about what that can mean for student learning.
Oh, absolutely! I mean, when we think about learning in general, having conversations, being able to process, and talk in classrooms is really important. So I want to have you speak a little bit more about the science of learning in particular.
How does Khan Academy integrate the science of learning into this educational platform?
Yes, even if we take AI out of it for a second, there are some foundational ideas that Khan Academy is built on. So the first is mastery learning. This is the theory and practice that you work on skills until you can demonstrate sufficient mastery of those skills before you proceed on to the next skill to learn. It's particularly important in math because math builds on itself so much, and if you don't have those foundational skills, every time you try to build on top of it, it's like when you're building a house on a shaky foundation—all of a sudden, that's going to be hard and a struggle because you don't have those strong foundations.
So mastery learning has always been key to our approach at Khan Academy. Fundamentally, we believe that kids need, in order to learn, to have lots of opportunities to practice with immediate feedback and the ability to be able to get unstuck when they're practicing. So that they can continue to practice has always been a bit of a challenge, but it's really fundamental to what we're doing.
Then finally is the motivation piece—which I know is a struggle for lots of kids. Learning is hard, and it requires a lot of mental effort. So we also want to think about what the research tells us about how kids are motivated. We know kids are motivated when they have an expectancy that they'll be successful at the thing they do. So who wants to do something when you think, "You know what? If I do this, I'm just going to fail at it"? Like, that's not—no one wants to engage in that.
The second thing is when they see value in the thing they're doing. And so that is always the next challenge. How do we help kids see the value of the learning activities we're asking them to do? Those two things actually have an official name: expectancy-value theory. But that's some of the things, as we design our experiences, that we think about—how can we help kids expect that they're going to be successful and also value the things they're doing?
That's a little bit of what we've been doing, and then I'll build on that now. AI comes in!
Oh, AI can help with that practice! AI can help give immediate feedback, and AI can help kids see some of the value in what they're doing. In a lot of ways, so that it's really been about how can the technology help us do these things that we already knew were important for learning?
I mean, that's really, really important to have tools to address those, especially as you're talking about motivation. So many educators, like you said, struggle with this, or even they know what the science of learning is but sometimes barriers get in the way to actually apply that.
So, speaking of barriers and AI, right? We are starting to think differently at Khan Academy, but what are some misconceptions that you're hearing about AI that you would like to address?
Certainly, the big one—and it happens sometimes when we even show demos of Kigo, our AI tool—is the idea of, "Oh my gosh, is this trying to replace teachers?" I want to be clear that we don't think this is ever going to replace many of the key fundamental human activities that, uh, that teachers do in classrooms. That's why we frame it as a teacher assistant, for example.
We know that human relationships are really important for learning. There's research that shows that if a kid thinks there's someone in the school who cares about them and their outcomes, we see better graduation rates and better post-secondary attendance. That's so important to have that relationship, and AI is never going to be that thing that is able to accomplish that, uh, that feeling that someone cares about you.
So I think that's, you know, one of the big misconceptions or a worry that people have about AI in terms of being able to replace teachers. The other piece that I want to be clear on is that AI is not some kind of all-knowing, all-powerful system knowing all the answers. In fact, the generative AI models make mistakes, and we need to increase our overall AI literacy so we understand when it's likely to be making mistakes, how can we work with it best to understand when it's most helpful, and the kinds of things that it can do and the kinds of things it can't do.
So what are you seeing in your travels? How are you seeing that teachers and students are using AI in classrooms?
For the past year, we have been working a lot with about 45 districts who were engaged with us in our pilot of Conmigo. Some of those also partnered with us and let us come in and interview students and teachers and do some observations of what's going on in classrooms. We are so thankful to districts who want to engage with us in that way and help us really improve the offerings that we have for classrooms.
So some of the things that we found—on the teacher tool side—when we first launched Kigo in March of 2023, I think we had four or five teacher activities. Every time we started talking to teachers, they would say, "Oh, could it do this? Can it help me unpack standards into learning objectives? Can it help me write a rubric?" All of these—"Can it help me write multiple-choice quiz questions? Because, oh my gosh, coming up with those is really hard!"
So we just kept thinking, "Well, let's see; let's give it a try." And because of that, we now have, uh, 25-plus activities for teachers just to work with Kigo on. And the things that we see most are helping to write lesson plans, which is not totally surprising. We know that's a big administrative task that teachers spend a lot of time on.
We also see one of our most popular activities is "Refresh My Knowledge," which is where you can interact with Kigo around content like, "Hey, I'm teaching gravity in the solar system tomorrow. Can I just get a refresh on that?" especially if, "Hey, I've got pulled into a classroom to sub for a teacher who's out, and I haven't taught this in forever!"
But it might just be, you know, last year, "What did you teach?"—all of those kinds of things. So, um, I think that was interesting for us to see what teachers were doing with the various activities.
Then on the student side, we certainly see the idea of the math and science tutor as our most commonly used activity. My personal favorite is "Create a Story with Me," which is where you can go back and forth and create a story together. But the tutoring step sees the most use, and what we've heard from, um, a lot of teachers and students is that it's that "helping me get unstuck" piece of things.
So if the teacher is in the classroom, maybe they're doing a centers kind of setup where they're working with a group of students, and there's students over here engaged in independent practice, there's a group over here on, you know, another project—those students on independent practice now can get unstuck on their own without needing the teacher to come over here and help them get unstuck.
The same thing happens for students who are, uh, able to access from home, doing homework. Those kinds of things are really helpful. And the other thing that I wasn't expecting so much was that Kigo is available in Spanish and Portuguese as well, and, uh, we also heard from a lot of students who are English language learners that being able to talk to Kigo in their home language to clarify some things maybe they're a little puzzled on, um, is a big help for them. That was one of the surprising things for me in the findings as we talked to students as well.
Well, that's—I mean, you're painting a very exciting picture of all the ways that this can be used! So I want us to shift thinking about a little bit of data and analytics. Like, how are we measuring this? How are we measuring learning in this new age of AI? Can you kind of discuss the role of how data and analytics are helping to understand outcomes, first off in Khan Academy, but then is it changing with AI? Is it making it more effective? Just speak to a little bit of that, please.
Yeah, so I do want to highlight again if you take the AI out of things, we have been doing large-scale efficacy studies particularly on our product that uses MAP growth scores to place students into their appropriate learning area. So for three years now, we have been able to look at the 200,000 to 300,000 students who use that offering and are able to look at what gains they get from Khan Academy on the MAP growth assessment based on those students. And we consistently find that students who use Khan Academy at least 30 minutes a week—which I know it can be a lot to find that in the classroom, but it's 30 minutes a week—that comes out to about 18 hours a year see significantly greater gains than expected based on norm growth gains from that usage.
Again, we've repeated that over and over, um, three years in a row to see that finding. So just to be clear, even before the AI, doing 30 minutes a week practice on Khan Academy—we know improves those math scores. So again, that comes back to the practice, the feedback—um, so important for learning.
So now the question, of course, is how does Kigo build on top of that? And I want to be clear that this first year, we were definitely focused on more questions of how is this best implemented in classrooms? How do teachers include it in their workflows? What are the best ways to help teachers and students learn how to use it? So we're not quite at the place yet where we can say we can do the study with the external assessments because first we want to make sure that we understand how it's working in classrooms and how to set students and teachers up for success.
That said, we've been able to do some things with, like, analysis of the transcripts that students or the conversations that students have with Kigo and be able to look at what are the kinds of tutoring moves that Kigo makes. So human tutors do a whole lot of things. They summarize information, they ask questions, they probe for further information, they give a hint—all of those are—there's research about what good human tutors do, and we can actually, we've built Kigo to do those things, but then we can evaluate if it's actually doing them, and then we can look at what are the student responses and are students responding in that way.
All right, here's one of the findings, which I'm sure for teachers in the audience is probably not going to be a surprise, but one of the things we found is that students aren't always good with, uh, asking questions. They sometimes struggle to know what they know and know what they don't know to articulate that in the form of a question that makes sense. So sometimes in these conversations, we see students just, you know, typing—not even "I don't know"—they're just typing "IDK, IDK." So because—and as all teachers know, it's not just that they're not good with interacting with AI; it's that students struggle to ask good questions a lot generally to teachers, to their friends.
So we've been thinking about how can we help Kigo extract more from students so that they're getting—they're going to get more out of it if they're able to do those kinds of skills, and that will hopefully improve not just their interactions with AI but their learning overall and their ability to reflect on their learning and those kinds of things. So those are the kinds of things we're learning in our first year here.
The other thing I will say is that we are always—and this kind of goes back to a point I'm making—we are always interested in districts that want to partner with us on thinking about data and analytics and ways to share data so that we can all understand how these tools are working.
Yeah, absolutely! And I mean, as a former special education teacher myself, I know the pain of having students be able to ask and construct questions. So what are some ways—you spoke that we're thinking through that—but does Kigo today sort of help guide students in that way? Have we already put some of those pieces in place?
Yes, so this is getting a little into the detail, so I, uh, I hope folks find that kind of interesting. What we're doing—so when we think about why are students, you know, just kind of having maybe less meaningful or in-depth conversations, there's—there are two explanations. One is they don't know how to ask good questions; they're not sure what good questions look like. The second is they're kind of being a little lazy, uh, not putting forth all that effort that they should.
So to address the first piece, we have done what we call action bubbles. So if you're interacting with the AI, we can suggest things that you might want to say to the AI—the prompts. Yeah, exactly! So those can be models of "Here's what a good question looks like." "Here are three good questions, and you might want to say which of these is a question that you have." So that's the idea of helping to, you know, helping teach students what good questions look like, offering some of those possibilities.
So I think that's one of the big things. The next question is what do we do to increase students' effort levels and where those are? So there's some things to come there, but certainly, one of the levers we can pull on motivation is offering rewards and things. One of the things we already have is Kigo can get different hats depending on what you're doing. So we may start linking up, you know, with some different hats for the ways you’re able to interact with Kigo and what that looks like.
So we're playing around with some of those ideas as well. And also, I know that our professional development, our professional learning team does a great job partnering with educators to help, um, help students understand how to use the tool to kind of overcome some of the learned helplessness or, uh, resistance or the easy button, if you will. Because, I mean, if I was growing up, you know, I have a tutor right here; I don't have to think!
Exactly! Yes! Our professional learning team definitely does that, and we can see in the data when, uh, schools have taken advantage and participated in that professional learning that we actually see the results in the data on the back end of students having those more meaningful and extended conversations that we would like them to have to really take advantage of the tool. So yes, and that professional learning cannot be understated.
Yeah, and that's the dream, right? For students to take advantage of the learning tools in the classroom. So I want us to think a little bit towards, uh, the future—the future of AI, even at Khan Academy. Are there any recent or upcoming projects that you're really excited about that we're thinking of doing?
Yeah, so you know, we're really known for math, um, but we have launched a writing feedback tool that we've had that, really, I was surprised how good the feedback it can give on student writing—things like how to evaluate your evidence from the argument, evaluating tone and style, evaluating your introduction, does it grab the reader's attention—all of those kinds of things.
And that's been really, I think, interesting for me to, to see the development of. We are going to be launching—so how that works right now is that students can paste in an essay and get that feedback. We're going to be launching a bigger writing coach experience so that it'll walk students through the writing process more and be able to then also provide that feedback, and that teachers will be able to then have access to that whole transcript of the entire experience and see what the student wrote and how they interacted with the AI.
So I'm really excited about that because we know that writing is so hard for teachers to grade. Oh my gosh—of getting that stack of essays—that you know, that is your whole weekend! Oh my gosh! And so because of that, students don't write as much as they should to get the practice they need in writing, and so we hope that this will increase the frequency with which students are practicing writing and make them better writers in the end.
Yeah, and that's really exciting! One of the times I was burning out as a teacher was when I was struggling getting my students to write, so having a tool to support students, um, is a total game-changer. So I would love—we only have a couple more minutes left, and then, um, please keep asking any questions in the chat. We'll leave a little bit of time at the end for next steps.
But thinking about the future, right? Thinking about what's next—the future of AI and education, because you hear the buzzwords. What do you envision the future will look like, and what role do you think Khan Academy can play in that future?
Yes, so first, it's so dangerous to predict the future. I could never have imagined that we'd be here two years ago—like, what does that look like? And I would be remiss if I didn't mention the fact that our boss Sal just wrote a book that envisions some of this future. If you're interested, "Brave New Words," check it out!
But I think that, so I think there's a couple of things that I think are coming in the near future, and then further out there are some, uh—their organization called Gartner that has something called the hype cycle, and it describes what happens with technology. And basically, the part we're in now—there's a—when new technology comes, you launch up and you get to this peak of inflated expectations, they call it, when everyone's like, "Oh my God, this is the thing! Can you believe it? It's amazing!"
And then you drop down into what they call the trough of disillusionment, where you're like, "Oh my gosh! This is terrible! I thought it could do everything; if can't do anything!" And then you move back towards the slope of the plateau of productivity where you start to realize these are the things AI is good for and these are the things it's not good for.
So many people right now are at that peak of inflated expectations, and I'm worried that we're all going to fall into the trough of disillusionment and not come back out. So I would encourage everyone who's, uh, you know, listening and thinking about this to think about, you know, what are the reasonable things that this technology can do and the things that it can't do.
So I think that’s really important, but I really fundamentally think that this idea of having a partner for students that can converse with them, that knows them, but that continues to have the safety guardrails—the links to what's happening in their classroom, their teachers' eyes on what they're doing—is really important and can finally potentially get us to this idea of individualized instruction and individualized support that we've been talking about for a long time but have not really been able to make happen yet, um, because there's one teacher and so many kids.
So I think this individualized support, for me, is the most exciting potential area that we can continue to build on. And before we move on to the final logistics of what's next, I would love to know just any parting thoughts on how where educators can go for more information, or if they're—I mean, right now, they're learning from you, but what are some of your favorite places to stay up to date on any upcoming news with AI?
So my biggest suggestion is to get in and play with it yourself! Because there's only so much reading you can do, and it really is, I think, you make a huge leap forward if you just kind of get in and try to get the AI to do things that you want it to do in your life and to help you, um, see. And you'll see where it's not doing well! You'll get into that, that, uh, idea that, hey, it's almost like talking to an assistant. It's going to not necessarily give you exactly what you want the first go-around, but you'll get better at giving instructions on how to get what you want, and it will start to give you the things you're looking for and is really helpful.
So that's my first suggestion! Whatever AI you choose—C-CLI—I mean if you go to AI, you can play with ours, but I think it's also good to play with, you know, ChatGPT to see the difference between an app that's made specifically for education and a more general app. So those are some of the things that I really suggest in terms of, um, just getting out there and getting your feet wet.
Great! Well thank you so much for this conversation! It's always wonderful learning from you, with you, and just hearing your perspective on the science of learning, AI, and how educators can just maximize their effectiveness in classrooms.
So AIV, if you wouldn't mind putting the slides back up, uh, we appreciate all of you who are here and watching this! So if you had questions we did not answer, or if you are watching this and you would like to ask us questions, you can go ahead and click on the QR code to the left on my screen, but it might be on the right if you're watching.
We also do have Sal coming to this webinar series to talk about his new book, so you can please continue, uh, sign up there! And then if you want to learn more about how you can bring this amazing tool to your school, we have a link to sign up for, uh, the district so you can reach out to us, and we're happy to reach out for any answer any questions.
But like we said in the chat, um, we put links in the chat. You can sign up for free today and just start playing with the tools! As Kristen said, the best way to learn was everything that we shared today and to do—and to get in there, so we appreciate it! Thank you so much, uh, Kristen for being here, thank you, everybody for joining us, and have a wonderful rest of your day!