How to take AI from vision to practice. Part 2

Uh, my name is Danielle Sullivan. I am currently the senior regional manager of Northeast District Partnerships at KH Academy, and I have been in education for a really long time. Lifelong learner. I was a former special education teacher and have been with edtech, oh gosh, about 15 years. I'm so excited to be here. Uh, with Sal at KH, so Sal, go ahead and introduce yourself to the world of our audience.

I'm also excited to be here, Danielle. I'm the CEO and founder of KH Academy. Great! So let's start us off. What is your broadest, most aspirational vision for how AI can transform education in the coming years?

Yeah, big, big question. Um, well, I think it's always important. Well, I'll say a few things that I've said well before AI came onto the scene because obviously folks have always associated me and Khan Academy with AI in education. Let me apologize. I literally bit my lip before, so I'm like a little bit – I'm talking a little bit funny. But the, um, I think there are a couple of clear things. I've been very clear and we've been very clear - if we ever had to pick between an amazing teacher and amazing technology, we'd pick the amazing teacher every time.

Now ideally, you don't have to make that trade-off. Ideally, you can use the best of both. The other thing that we're very clear on - we've always been - is that it should never be about technology first. It should never be about, "Hey, there's a cool technology; how do we use it?" It should be about what are the problems we're trying to solve and then what are the simplest ways of solving it. Maybe it involves technology like AI.

So to answer your question, the big opportunity – and this is something, you know, our mission statement – we talk about world-class education for anyone, anywhere. We've never imagined that it's us doing it by ourselves. That's arguably the mission of the global school system as well.

But the reason why we think it's possible now is that we're getting closer to it. World-class to us means more student-centered, more personalized. I always give the example of, you know, Alexander the Great had Aristotle as his personal tutor. How can we get closer to that ideal? Every teacher knows that there are 25, 30, or 35 kids in their classrooms. They're all at different places, and they have different needs. That's gotten even worse after the pandemic – more grade levels in the same classroom. Every teacher wishes that they could provide more support, more tutoring, etc.

And at the same time, every teacher wishes they had more time for themselves. We have record-high teacher burnout, spread thin. There's a lot of work that they're doing that isn't student-facing, like writing lesson plans and grading papers and progress reports, and you know, we could go on and on and on.

So I think the highest order bit is how do we make more classrooms personalized in every way possible – supporting students better, making it more relevant to what they're doing, supporting teachers, acting as a bit of a teaching assistant for them so that they have more time and energy for themselves and for their students. They can even customize lesson plans and personalize that more.

So that's the big high level. KH Academy has been doing those things for many years, and we've been approximating it with technologies like software and teacher tools, on-demand video, and district dashboards. But now there's an opportunity to go that much closer with generative AI, which is exactly some of the themes that you talked about in your new book, "Brave New Words."

Can you share how some of those themes in your book are connecting to the work that KH Academy is doing with school districts to solve some of the things you just shared?

Yeah, there's a bunch of interesting things here. I mean, and, and well, I’ll start even pre-AI. Again, when KH Academy, in the early days of Khan Academy, you know, the teacher tools were always part of Khan Academy. So it was – it was a lot of people imagined KH Academy as this thing that started only for students, but the first teacher tools were made in 2007.

Um, actually no, 2005 for myself while I was tutoring my cousins, and then later for a teacher in Washington, D.C., in 2007. So it was always part of the picture.

But one thing we appreciated, I would say, about six or seven years ago is we had all these efficacy studies. There are 50 plus efficacy studies on KH Academy; I believe we are the most studied edtech platform of any kind. Almost every study has a narrative: hey, if students are able to put in, call it, 60 minutes a week of personal practice, they’re accelerating, depending on the study, anywhere between 20% and 50%. Some of them even have much larger numbers.

And so we would go to school districts and we would say, “Look, here are the studies.” And, you know, almost every superintendent or chief academic officer would say, “Oh yeah, we believe you. You know, my niece swears by Khan Academy, or you got me through grad school,” or whatever else. That’s always great to hear.

But the districts – we said, “Look, a lot of your teachers are already using it. Why don’t you use it more systemically?” And a lot of the – every district pretty much told us, “We would love to, but you’ve got to give us support, training, integration with our rostering systems, district-level dashboards.”

And so that was the moment, about six or seven years ago, before generative AI came on the scene, that Khan Academy decided, “Hey, we have to build this very important muscle if we really care about moving the dial for districts, for states, for countries over time.”

And so now that we are in this generative AI world, we are obviously working very hard, making these tools as effective and usable and engaging as possible for students and teachers on all those dimensions I just talked about, and there are many more that I haven’t even just touched on.

But I really... the way that we’re going to get maximum impact and maximum traction is by working through school districts. Um, that’s the way; that’s the only way that we can make sure that we’re not only reaching the precocious students or the, you know, highly engaged families or the teachers who are only at the kind of bleeding edge, always willing to try new things.

There’s enough evidence now that we think if we do this together, we can move. And there are many reasons why people might want to use KH Academy now. It could be, um, accelerating academic achievement, and it’s not just math anymore. We have, as you know, math, science, SAT, um, humanities, increasingly reading comprehension.

Another reason is for a district to be able to even monitor that progress on how they're doing, and now there’s this whole new universe of a district also wanting the productivity and the learning tools around AI. But we realize that only by going through the districts are you able to reach all students, which is absolutely true.

So I want you to really dig in and let’s think of some specific ways that at KH Academy we’re thinking through these district partnerships using AI to address some of the challenges and pain points. Um, so if you could give some examples that leaders can start to think through of how you’re approaching this problem.

I'll start with some things that people already associate with Khan Academy, and then I’ll move to some new things that we’re launching literally as we speak, and then we can talk a little bit about the future.

You know, the classic use case of Khan Academy is a teacher assigning practice on Khan Academy. Most people associate math, but as I just mentioned, it goes well beyond math. Math is where we have the most fleshed-out progression from pre-K through the core of college, and teachers would both assign a daily practice according to what’s in their scope and sequence, but they would also assign goals like, “Hey, I want you to get mastery or proficiency in this unit by this date or in this course by that date.”

That use case, which, as I mentioned, we’ve always seen really good efficacy results from, can help support the student as a tutor. It doesn't cheat for them; it won't answer the questions, but it can give them CTIC nudges to go in the right direction, engaging them. It helps students see relevance. Students can say, “Why do I care about this?”

And it will remember what the students care about, but we make all of that transparent.

Then on the teacher side, first of all, it can report all of that to the teacher. So the teacher doesn’t just know, “Oh, Danielle did her assignment; she got eight out of ten,” you know, right? It could say, “Well, she needed a little help with this question, and this is what we talked about. Did you know that she wants to become a doctor? When I made this system of equations about medicine dosing, you know, Danielle really got excited.”

So a teacher can see that.

Classically, our dashboards were like what you would expect – these things that kind of look like spreadsheets. I mean, we think they're better than that, but you have to be a little bit of a data analyst, and it’s not just KH Academy; any edtech tool, to figure out exactly who’s where and what.

But now, teachers can talk to the AI like it’s a teaching assistant. “Where have my students been? What are they working on? What do you, the AI, recommend for assignments for the students?”

Um, that’s already there.

Then there’s all of the productivity tools that I’ve been alluding to. There’s a teacher being able to write a lesson plan, getting drafts of, you know, the teacher portion of IEPs that they need to write, create lesson hooks.

Um, you know, there’s like – I lose track of the number – there are 30 plus activities that teachers can engage in that directly improve their own productivity, including helping refresh their own knowledge.

Um, and even going into simulation as a student or as a teacher with an AI student to see how good they are at being able to, um, go through the concepts.

So that’s and all of that is already in place. What we're launching as we speak – and this is really, you know, most people don’t associate KH Academy with writing, but I think a year from now, hopefully people will associate as strongly in writing as they do in math and science and reading comprehension and other things.

Uh, but we’re – we’re, in the next couple of weeks, going to launch something called Writing Coach, and by September, uh, teachers in our districts are going to be able to create writing assignments with the AI, assign it through the AI, and then the AI will act as a coach with the student. Not doing it for them, but hey, let’s work on the thesis statement. Let’s do some outlining.

And there’s a whole tool where they do outlining together, so it’s not just a, you know, plain vanilla chat interface like you might see on ChatGPT. Then the student writes the essay, and it highlights parts of it, starts threads, and it’s discussing it.

And then when the student’s ready to submit, it submits to the teacher, and this is, I think, going to be very powerful from a student’s and teacher support point of view, but it also addresses a major pain point that educators are facing right now.

So classically, a teacher just got the final output, and I have a whole chapter in the book about cheating. I started – cheating isn’t new to ChatGPT; it existed well before that.

Students were using services on the web that would write your papers for you; they literally hire someone in like Nigeria or Kenya with a PhD in English to write your paper. So this kind of stuff already existed, but now when you do this workflow I just described, it’s going to be – literally, it’s going to exist in September.

Teachers aren’t just going to get the final output; they’re going to be able to see the process—the transcript of how they got there. They’re going to be able to talk to the AI, and the AI is going to be able to communicate, “Yes, I’m confident this is Danielle’s work. We worked on it; here’s the transcript consistent with her other work.”

And if Danielle went to ChatGPT or one of these other sources of cheating, our AI will be able to point that out: “That looks shady; you should look into it.”

But even more important, so that’s a guardrail to prevent some of the fears which I know a lot of folks have. But it’s going to support the student way better.

Uh, most students have to write in a vacuum; most students don’t have the supports at home. Teachers don’t have the supports to really get transparency into themes that their kids are struggling with while they’re going through the writing process.

Um, and this will streamline all of that as we go. Um, actually in about that same time frame, we’re also going to be able to – we’re also allowing teachers to create custom AI activities that they’re going to be able to assign to their students.

So, imagine how powerful that’s going to be for exit tickets or just polling students on their current understanding of something. You know, you could say, “Hey everyone, I want you to have a conversation about how you think a plant gets its mass from before we cover how photosynthesis works and carbon fixation and all of that.”

Um, so, and I imagine teachers are going to come up with really creative activities that they can then assign and have facilitated by the AI.

And then if you go to the future, um, you know, all of that’s going to get much better. You can start, you know, we’re integrating with mainstream learning management systems. The audio, being able to see visuals, all of that’s happening probably in the next, you know, one, one to three years.

I mean, thank you so much. All of that is very exciting. After this webinar, we’ll give you a link of where you can get a demo account and see Writing Coach for yourself or sign up for a demo.

Um, but a couple of questions are coming through the chat. Keep them coming! But I’d love for you to talk about safety and responsible use of AI. There are a couple of questions coming in, um, asking if you have an example of what policy we’re sort of recommending around AI use. And then even talking a little bit more about what we’re thinking through at Khan Academy on the safe and responsible use of AI.

Yes, and there is a lot there. Many, many folks here know that, and I read about it in the book. Um, OpenAI reached out to us months before ChatGPT came out, months before anyone knew that this was about to happen.

And you could imagine we were both excited about the potential of the technology, but we were concerned. You know, we’re a not-for-profit; our bottom line is serving students and teachers and families.

Um, and, and you know, we’ve always been very careful that yes, data can be valuable if you’re personalizing it, if you’re measuring your efficacy, if you’re reporting back to important stakeholders like teachers or parents. But it can also be dangerous if it’s in the wrong hands. It could be used to, you know, violate privacy, sell information, advertise, etc., etc.

Uh, so immediately, and then when we saw generative AI, we immediately saw that, hey, this could be used as a cheating tool; this could, in many ways, undermine what’s going on. So, and students might have dangerous conversations with it or inappropriate conversations with it.

So the guardrails that we put in place from day one—well, I would say the first one, and I’ll say kind of what we did, but also what I recommend any district or educator or decision maker to think about when they’re looking at any tools, especially if it’s using AI, is first, it should be using what in the field is called a Frontier Model.

Uh, the frontier models are the ones that are the most sophisticated ones. Uh, be very suspicious of someone offering you a free, no-strings-attached model.

Um, and if they can’t – we are offering some things for free for teachers, but that’s because we got major philanthropic grants from folks like Microsoft. But the frontier models cost more to operate, but they’re the ones that make the fewest errors, are the least biased, most accurate, etc., etc. So you want to be using these frontier models, things like GPT-4.

Um, and, and you know, there are other similar models from Anthropic and others, but those are the costlier models to run. Anyone who’s not doing that, I think, is being a little bit irresponsible, especially in an education use case.

The second one is make sure it’s not a cheating tool. Um, and that’s the other thing some of these frontier models you need because you can make them act like an ethical tutor, while some of the more basic models are just going to give you the answer, even though that answer will oftentimes be incorrect.

Um, so with mine, it acts as a Socratic tutor; it will nudge you with Socratic questions, but it will not give the answer.

Um, also everything is transparent to the educators, to teachers, to districts, so they can see any conversation that an under-18 user is having with the AI. If the student decides to do something shady with the AI, “Hey, help me make a bomb,” or “You know, I hate these people,” or whatever, then it’s going to shut down those conversations and actively notify not just the teacher, but the district official, so that they can monitor that.

So that’s a major and I think an important table stakes feature that if you’re going to use it with under-18 students to have that.

And then on the, uh, data privacy—and this gets a little bit wonkier, but it’s important—I would say first and foremost make sure any software provider that you use, even before AI, but even more sensitive with AI, has their SOC 2 audits.

SOC 2 are these data privacy, data security audits to make sure that that organization can protect your data. Because the last thing you want is a breach of that data, and it’s either being used to train some shady LLM or it’s being used to just be out there on the dark web—your teachers’ and your students’ data.

So that’s number one. Number two, make sure that, um, there are proper safeguards to ensure that the large public general models are not training off of data from your students.

The large language model providers do take safeguards, but you just don’t want to risk that it knows something about your student that it shouldn’t, and that some random people are going to be able to leverage that.

Um, so that data should be held in a sandbox by the provider that you trust, um, that hopefully has a bottom line that’s around trust, not necessarily about maximizing profit.

I would scrutinize data privacy agreements. A lot of data privacy agreements these days say, “Hey, we’re going to keep your data secure, but if we get acquired or go bankrupt, all bets are off; the acquiring company will now have access to your data.”

Um, if we go bankrupt, it’s kind of an asset that goes for sale. Um, so pay very close attention to that type of thing.

Oh, and we also mask personally identifiable information as it goes between Kigo and any large language model.

Thank you. I mean, so yes folks, we are definitely doing a lot at KH Academy to make it safe and responsible. So another question that came in is that administrators are trying to figure out how to fit everything together with what they’re currently trying to do with AI.

So how would you recommend to leaders to balance investing in AI literacy and AI tools while still keeping everything else that’s on their plate? Where do they prioritize their time?

Yes, I mean, what I'm telling everyone is just try to start using these tools. Uh, you know, Kigo obviously in an education setting makes a ton of sense, but even in your own life where appropriate, use other AI tools. I think it’ll just make everyone more comfortable with it.

But I think the lowest hanging fruit is start leveraging or working with your faculty, with your teachers, to use our teacher tools. One of the problems about edtech—and KH Academy’s guilty of what I'm about to say—is that we would show you these efficacy studies, we’d show you these great tools and, you know, I present to thousands of teachers at a time and they're like, “Yes, this is cool, personalized; I need this!”

But then it’s one more thing for a teacher to learn. They already have a full plate, and it’s hard for them to squeeze it in. A lot of teachers have done it, but it’s a high activation energy now, especially with our teacher tools.

Yes, it is one more thing to learn, but immediately it can start saving a lot of time for teachers. We’re getting anecdotal reports that it’s saving teachers 5 to 10 hours a week—and these are from partner districts that we’re hearing this from. So they’re even using that as a recruiting and a retention tool for teachers, because imagine now if you can take at least 5, 10 hours off of their plate; that can go to their own time or more time with the student. That’s a win for everyone, and they are pretty easy to use.

So teachers, they don’t have to change everything they were doing; they can use it to just customize or modify their existing lesson plans or get nice lesson hooks or exit tickets or whatever it might be.

Um, so I would say that’s the first step. And then as they get more comfortable with it, you know, think about using Kigo holistically with the teacher and the student. A lot of what we’ve talked about, you know, using Kigo to support students, and it can support them not just in their KH Academy work but in any work they have.

It can support them in any classroom—it could be a math classroom; it could be an art history classroom; it could be whatever. Um, they can get that Socratic tutoring, um, and then, um, and then, you know, we’d love folks to kind of go on that journey with us as we go deeper and deeper.

Another thing that I’m imagining, we are all imagining, over the next, let’s call it two-year time frame, one to two years, is that Kigo will increasingly have all of the context of the district, being able to know what the suggested curriculum is, what the scope and sequence is, even things like the district handbook, etc.

So that as the teachers are working through their lessons, it is anchored on what the district cares about and can even, you know, do some reporting back to the district.

Um, but yeah, I would start using it in your own life, have the teacher start using our teacher tools, which, because of that Microsoft grant, the majority of those are free now, and they’re, even though they’re using frontier models.

Um, and then, you know, let’s work together to get in front of your students and do all the reporting, etcetera.

Exactly! And somebody—I did type in the chat—yes you can work with us, and we will put up a link in a minute. But before we close out, Sal, we just wanted any last-minute things to think through the future.

I mean, you shared some exciting things that we’re thinking about, but where – how do you see the future of our educational partnerships in five years? What would be a dream of yours? Like in five years, if we could do X...

Yeah, I’ll say one thing before I go five years, which now feels like forever. You know, the reality is that teachers and students are already using AI, but they’re just using it in completely off-label ways and sometimes shady ways, and we don’t know what’s going on.

So I would say there is an imperative to get control of it, and hopefully we can help everyone here think about how to do that in a safe and ethical, but at the same time, pedagogical and productive way.

But when you imagine it in the five-year time frame, I mean, even three years feels like it’s a very long time from now. I’m imagining the AI, as I just alluded to, will have full context of everything that the district cares about. We have to be very careful with that data that it’s in a very safe place.

Um, but it can use that information to help develop the lesson plans with the teacher, and then also help the teacher administer the lesson plans in certain ways.

So, hey teacher, here’s the lesson plan; click here and I’ll launch the bell ringer for you, and then the bell ringer is an AI-driven bell ringer, and the kids do it.

I could even imagine a world where it’s not necessarily that the students have to take out their laptops to, you know, some of the activities, yes, they will take out their laptops to work with Kigo or to work on KH Academy or other tools, and the AI will be there.

But you can imagine a world – and you know, we have to be very careful on safety, privacy, etc. – but as long as we can manage those risks, some very positive things happen.

Where, you know, there’ll be the equivalent of an Alexa device but for your classroom, and it will be able to observe the classroom and even, you know, facilitate breakouts.

But once again, the teacher is in charge there. It’s able to report back to the teacher, use that information to revamp assignments, revamp lesson plans.

Um, that the AI is aware of all of the different tools that a teacher would use. One of the biggest problems of tech over the last 10, 15 years is how fragmented it’s been.

Um, and for everyone, the reporting’s fragmented; everything’s fragmented. Hopefully, the AI can look at things that are structured in different ways or even unstructured and make sense of those.

So you can have one report, one conversation with the AI about everything that the student has been doing across platforms.

Um, but you know, I think if we do that, we’re going to get to a world where teachers are spending a lot more time doing the rich and the fun things that they wanted to be teachers for, a lot less time on the non-student facing side of the house.

Classrooms themselves are this optimized, really engaging, really relevant experience and really active learning experiences for everyone involved.

Well, that sounds just – it’s a little sci-fi but also really doable too. Like, it just sounds – it’s very exciting.

So thank you so much, Sal, for taking the time to have a conversation.

Um, AIV, would you mind putting up where people can find out more information? So if you wanted Sal to answer any more of your questions, you can absolutely click on that QR code if you want.

We’d love for you to join us for our third session. Please scan that to sign up to join us for the final session with our director of professional learning, director of district success management.

And if you want to work with us, we’d love to work with you! We have plenty of time between now and back to school, so go ahead and – we also drop the link in the chat.

And then, if you would like to buy Sal’s book, you can go to this QR code.

So we appreciate you for joining! Thank you so much for just being in education, and Sal, thanks again for being here. We appreciate everybody.