Setting AI Policies for your School Districts: Part 2 of 2

So hello everyone. I'm Kristen Desero. I'm the Chief Learning Officer at KH Academy, and I'm going to, uh, let our two other panelists do quick introductions of themselves, and then we'll get into discussions. Chris, you want to start first?

Sure, I'm Chris Hegel. I'm the Executive Director of Digital Learning for the Peninsula School District in Gig Harbor. We're a school district of about 9,000 kids. We're about 45 minutes south of Seattle. Um, we kind of jumped in head first last year, um, with AI back in January of last year, um, after seeing quickly the impacts that it could have for some of our teachers and some of our students. And so, um, happy to share anything about that or where we're going or what we've been doing. So awesome.

Bobc?

Yeah, sure. Hey everybody, I'm Bobc Moimi. I'm an Assistant Superintendent with Gwinnett County Public Schools. Um, we're the 12th largest school system in the country, so about 180,000 kids, um, just north of Atlanta. Um, and we've been on the AI journey since December of 2017, actually, um, and have built out our own sort of learning framework and three-course pathway, uh, and really looking at embedding it for every kid. Um, although we're starting obviously with a K12 pilot in one of our sort of feeders of schools, which is about 7,000 students. Um, so really glad to be here, uh, today and to be part of this conversation.

Great, thank you so much. I really appreciate the quick overview of where you are. Um, Chris, could you tell me a little bit more about what you've done in your district and maybe how that relates to some of the principles?

Sure. Um, last year our kind of our technology department, our teaching and learning, we were merged together. Um, we actually did that about four years ago, um, into the Department of Learning and Innovation. And so when ChatGPT came out, we'd been doing some small like little pilots around, um, AI with things like the SD Explorations course or some other, um, just small learnings with some small cohorts of teachers, six to ten teachers, um, for the last couple of years. Um, but when ChatGPT came out, we really decided, oh boy, we better, we better do something about this. Um, you know, my technology engineers were sitting right next to our like teaching and learning specialists, um, and having these conversations back in December last year saying, oh boy, we better get to work on doing something. Um, and so we met with our all of our high school English teachers in early January last year and, you know, of course there was some anticipated freaking out, um, a little bit, you know, oh my God, what are we going to do? But we'd been focusing for the last few years on implementing Universal Design for Learning in all of our classrooms, and so teachers were already having, you know, doing the thinking behind I need to change my lessons, I need to change my instruction anyway, and so this just catapulted them to do it quickly. Um, and so they were able to, our English teachers to their credit, were able to, you know, kind of pivot even in that first meeting saying, all right, well there's some things that I really didn't, I knew I shouldn't probably be teaching and doing this this way. Um, I'm just going to move on. This is the thing that's going to make me move on from that right now. And so, um, that's how we started. We put together a team of about 30 teachers where we have about 600 total in our district. Um, but we put together a team of about 30 teachers. Um, we called it AI Action Research last year. Um, they started experimenting with different tools, different techniques, um, working with their kids, all of their kids, using different generative AI tools, um, just to kind of understand what how kids understood what they were doing or what did kids know just to kind of collect data. And that allowed us to, um, build out what do we believe as a district? What are our principles around AI and our beliefs? And so that's how we've kind of gotten to where we are this year, where we're fully embracing AI in our classrooms. Um, of course not every teacher is there yet, but it's growing day by day. But, um, yeah, that's how we kind of got to where we are right now.

Great. Can you tell me what fully embracing AI in your classrooms looks like? Like give us an example of what that might be.

Um, like I said, it's not every teacher, but, um, you know, we just recently invited the public into, um, two of our middle schools for American Education Week, and what we did is we focused, we brought together teachers from all over the district, and they talked about what they're doing in the classroom, um, with AI. And so it kind of just depends on the subject. Um, we had a really amazing resource teacher who is using AI to provide grade level content to students who may not be able to read at that level. So, um, she was—for giving me an example or giving the public an example—of how she asks kids to take, she—I think she was teaching an 11th grade, um, history course at the time, and it was a Supreme Court case that they were talking about, and the kids—some of the kids in that classroom were reading at a fourth or fifth grade level. And so she just had them take and use, um, one of the generative AI tools to take that case, put it in there, summarize it, and rewrite it down to a level for that they understood. And so those kids who may not have the ability to read at that level were still able to get the grade level content that she was teaching in that class. Um, we've seen quite a few teachers, um, with our emergent multilingual students doing the same thing; you know, rewrite this into a language that you understand, into your primary first language. Um, and it still allows them to get the content that they are teaching in class and understand. Um, the first classroom I went into that was testing K—the first girl I talked to, the first question she asked me was, "Can I speak Swahili?" And I was like, "I don't know, let's see!" Yeah, yeah, we've had a couple of Science and Math teachers doing the same thing; you know, some of the science concepts, like in AP Chemistry or AP Astronomy in the high schools, can get into some pretty technical terms, and sometimes kids don't understand those words, and so they're using it to kind of help me restate this or take this content and explain it to me in language that I understand. So those are some of the more powerful things that I see happening in our classrooms right now.

Yeah, great. All right, uh, Jim, I see your hand up. I'm going to go to Bobc, and then we'll open it up for questions. So I'll go to Jim's question first, but if other folks have questions, you can either type them into the chat or raise your hand as well, and we'll get you next. Bobc, tell me a little bit more about how you have started approaching this in Gwinnett and what it looks like now.

Sure. So back when we started was December 2017, so people thought we were absolutely crazy. Um, and, like, because at that point, like, the public concept of AI was like Terminator and that about it. Um, and so at that—what happened with us was we were reading some of the new sort of news out of the McKenzie Global Institute and other places about how the future of work was shifting rapidly, um, and we realized, "Oh my goodness, we better get on this pretty quickly." Um, given how large of a system we are and how long it takes to shift practice. Um, and so we started looking at AI in terms of two different ways. And remember this was pre sort of public understanding of generative AI. Um, we started looking at it from like an embedded and a discrete model. So like the discrete model was the sort of like three-course pathway that we built out with the state of Georgia, um, where we're like, "Okay, for kids who want to sort of deep dive into AI, let's make sure that we have like a strong deep pathway," and that it's not focused just on the technology or just on the coding, but also has aspects around like ethics, and creative problem solving, and like applied experiences and user design and those sorts of things, uh, to make it so that the kid can be, um, sort of ready to go deeply in that work.

Um, at the same time, what we realize is that this AI learning can't just be a high school thing, um, and it can't just be for those sort of deep divers, so we talk about our swimmers and our snorkelers a whole lot, um, with this idea of like everyone is going to have to work in this world of AI, just like today; like you can't be like, "I don't do the internet," right? Like, well, sorry, like one way or the other, um, if you're engaged in sort of modern society, like the internet's going to either do something to you or you're going to use it for something, right? Um, and so we really jumped into the principles around like the knowledge building and the balance piece where we're really promoting AI literacy, we're really doing it across all our subject areas, um, and looking at how, for example, like in a social studies lesson in, um, about like the Civil War, um, how do you actually build in some data science components into that so that kids are building those skills.

Um, in our like English classes, how can you build in like a coding program as one of the assignments that a kid could do to show that they like understood a novel and those sorts of pieces? So the idea would be that through the accumulation of sort of the K12 experiences, a kid would get all those different tools and knowledge and those, this, you know, interactions that they'll need, um, for it to build up their sort of ability. So we're essentially sort of piloting this idea of like content as like the substrate through which we, uh, are able to teach really important skills and like deep thinking abilities, um, as opposed to the content being like the final outcome and this sort of only outcome.

Um, and then on the balance side, that fourth principle is super important, right? Of like, to your point earlier, Kristen, like you can't just like say, "Hey AI, go do for me," and then like it takes over sort of your own agency and those sorts of things. We've really started thinking about like the generative AI outputs as like the new floor, right? Like that's the new floor now; what's the human element that comes on top of that? And we mean that not only for kids but also for our teachers, right? So to your point, like if you're building out a lesson plan, like the first lesson plan is that first one that's generated that's maybe 60 or 70% of the way there if you have a great prompt. Now how do you as the human in the loop, like create that extra 30, 40, 50% of value, uh, to make it great for everybody?

Um, so it's been really neat and we've been really focused on our students of color, our women and young girls in particular, trying to sort of rewrite the narrative. Um, and thankfully there is no narrative, at least locally, on like who does AI, uh, so we're like, "All right, let's like start from scratch at the beginning," uh, literally and figuratively, right? Um, and get people there.

Awesome, thank you. Jim, go ahead, what's your question?

Thank you. Um, one of the things that I am worried might be missing from many of these conversations that I've been observing is rigorous analysis of traditional preest post test against a control group, uh, to measure effect sizes. And I say this from the position of producing, um, a language learning app for all ages—ESL primarily for pronunciation remediation—but as I've been incorporating large language models, I've been able to cover things like grammar and, uh, vocabulary that it was not my traditional field because I've been so focused on intelligibility remediation. Uh, so can you talk a little bit about how you measure, uh, the introduction of AI into a curriculum? Are you, uh, and your users, uh, your customers, are they using pre-test post-test analysis against a control group for effect sizes, or is there a different way to measure that?

Um, the motivation for this question is I have had surprising results in both directions—things that I thought would be extremely, uh, useful had fallen, have fallen flat, and vice versa. Some things that I thought weren't going to work had a huge effect size, so I'm fascinated to learn more.

Right, so I'll talk a little bit about what we're doing and then would love, uh, the two of you to talk about how you're evaluating, uh, the success of these programs. So for us, uh, when we released in March, we had a couple of partner districts and we did very much qualitative research. We observed what was happening in classrooms. We did interviews with teachers and students because everyone was just trying to figure out how to use this, and in my opinion, that's too early to do your, you know, randomized control trial of whether things are working or not. Uh, this fall we have a plan where we want to actually compare students, uh, in districts that are already strong Khan Academy users without Kigo and then having students in some of those schools, uh, or potentially classrooms use Kigo and some who don't.

Uh, and then we can then look at results on other external assessments to see whether we see that improvement. But that's a big ask for us of districts, so here's a little plug: if you're a district that's a strong Khan Academy user and you'd be interested in helping do some of that kind of evaluation work, um, we would love to hear from you so, uh, that we can do that in more places and, uh, be able to make sure we're getting that kind of evidence.

Chris, BC, what are you folks doing?

It's interesting that you have this question because it was just something we were talking about. I was talking with our Director of Research and Assessment last week. Um, it was mostly around the fact that, um, last week there was a research paper that came out around, um, using large language models in math education, um, and the positivity that they were seeing in that research paper around that. And so, um, we haven't done anything substantial yet as far as pre- and post-utilization, um, because it's still—we're still figuring it out. I think it's still really early for us and so, but yeah, it is something to look at, and I think there are people researching it and studying it right now too from a, you know, at the university level, or this one was written—a lot of it was written by Microsoft Research, so yeah.

Yeah, just to jump in there quickly as well, um, we've, similar to you Kristen, have been looking mainly at the sort of qualitative data and feedback since we haven't been focusing on like a specific AI tool and its impact, and we've really been thinking about like AI literacy as a whole and how to get a kid ready for their future.

Um, we've actually just recently embarked on a partnership where we're really looking at like how do you measure like the creative problem solving? How do you measure like the ethical understanding and those sorts of pieces in students, um, related to these technologies? So in the interim, like our sort of feedback has been from teachers talking about highly engaged students, from students having better work outputs that they're actually creating, and like better understanding and deeper understanding.

Um, I'll give a really quick example: we had like a class where a biology class where they were looking at research papers, and they had, um, ChatGPT generate one or two as well, and the kids had to actually like look through it and go, "Hey, like what's right? What's wrong about this like scientific paper?" Um, and it actually got the kids to think way more deeply than just giving them an already sorta sorta vetted paper and looking at it that way. Um, so lots of opportunity and lots of need still to look at the effect sizes and impacts.

Well, I'm going to use my moderator privilege and ask you a follow-up question. When you create those three courses that are helping on the technical side students who are really interested in the deep dive, did you make up your own learning objectives for that or are you using some set of a learning framework?

Yeah, so we built our own framework back in 2017, uh, that we call the AI learning framework, uh, and it essentially has like six components, um, around programming, data science, math reasoning, creative problem solving, ethics, and applied experiences. Um, and then based on those, we have like what we call a K12 continuum of learning, which is sort of grade band level what we think kids are going to need to know and how to like sort of prepare them for 12th grade, with the understanding that that's going to shift and be revamped every year, um, as kids start coming in more and more ready, right?

Um, for these types of experiences, um, and so the, the three-course pathway was built off of that, and we had a lot of friends, um, from Google, from HP, from, um, you know, Microsoft, from, um, UGA, from Georgia Tech, all sorts of people, a lot of local industry that came together alongside teachers to go, "Okay, how are we going to put this into a course that's usable?" Um, or a set of courses that's usable across our state. Um, so it's really cool, and if you haven't seen them, you can check out the standards for those courses on the, um, Georgia Department of Education website. They have them posted for everybody to be able to use.

Awesome, great, thanks for sharing those. All right, I'm going to take one more question. Jamie, I'm clicking allow to talk here. Great!

Um, so I'm really interested in this AI discussion, specifically around the role of human judgment and critical thinking. I hear it a lot, um, in webinars and in sessions around AI that we have to center the human perspective here. So I'm curious, in a two-pronged approach from an AI product perspective, Kristen, how do you and your company think about, um, the necessary shifts in pedagogy that are coming due to AI, and like how do you keep up with that to ensure that students are pushed to do that judgment and critical thinking that's so necessary?

Um, especially with, like, well, you know, currently so much of the curriculum can just be knowledge-based, but like how do you look ahead and get ahead of that? And then on the school and district side, I would be really interested in how, um, from a curriculum perspective, teaching and learning perspective, you two are thinking about avoiding this like judgment and critical thinking being taught in a vacuum and ensuring it's truly integrated into the practice of the classroom.

Great, thanks for your question. I will, uh, start off with the answer from, from our side. So one of the first things about using artificial intelligence as a tutor is that it can be wrong, and so part of critical thinking is actually teaching students not to just accept the answer from the artificial intelligence and to think to what are the questions they should be asking themselves about whether this is the right answer or not, and how would they go about assessing whether that's the right answer or not.

And so that kind of critical thinking, that—I mean that's no different from the internet. When you do a search on the internet, you may get wrong information too, and we've been really struggling, I think, as a whole, to help, uh, society figure out how to judge information and to process information and to make their own—be able to think critically about whether it's right or wrong. But I think AI actually provides us a tool, uh, for helping to, to do some of that, and I was in a classroom where the teacher actually made it almost a game to figure out if the, uh, if the AI's response was correct or not, and if a student could spot it being incorrect, that was, you know, a big exciting thing that happened.

Um, so I think that's one place as we think about AI literacy that, uh, brings in then helping to, uh, help teach some of those critical thinking skills as students evaluate information. Bobc, you want to go next for thinking about critical thinking?

Yeah, I'll jump in and I'll, I'll be, uh, really transparent here. I know Jamie and if anybody on this call doesn't know Jamie, you should. She's on the exact opposite side of the continent as I am, but I wanted to put that plug out there, and I think her question is fantastic. So, for us, we really think about the critical thinking and the judgment pieces starting in kindergarten, right? And it starts off as like easy as like, "Hey, two people might have completely different opinions about something, and that's okay."

Um, and like here's how to like talk through that, here's how to understand it. That then gets translated ultimately like later on in their careers into like, "Hey, your lived experiences cause you to have certain perceptions and biases and like ways of thinking about decisions, but there are other ways of making decisions as well," right? And then we eventually get to the place of like, "Hey, these human biases can actually show up in the data sets that are behind a lot of these algorithms that are, that are pumping out the results, uh, whether it's generative or not."

Um, in terms of the AI, um, so then, then what happens essentially is within like math classes, for example, our conversation is less around like, you know, a random data set that you're, that you're trying to plot and more of like, "Hey, before we get there, where did this data come from? Um, could it be inaccurate? Like what could it be potentially biasing us towards?" and those sorts of things. So those sorts of conversations happen across the curriculum trying to get kids to realize that, "Hey, there could be many philosophical perspectives to something, like how do we empathize the other side, um, so that we can see what they're thinking and why?"

Um, and then sort of building that all the way through so within every class, classroom there's space for judgment, there's space for critical thinking, and in fact, that's really the most important thing, uh, because if you can just like get the answer from ChatGPT or Bard, uh, then that really wasn't deep or critical thinking, that was just a sort of fancy regurgitation. Um, and that's been a really, really hard move, uh, and pivot for us to make as an organization, as teachers, as, as all those things.

Great, Chris?

Yeah, I'll also chime in. Um, I was just having this discussion with a teacher team from one of our high schools last week or the week before, um, and they had started moving to, um, you're encouraged to use some of these AI tools, but I want to see the entire conversation. So like with some of them now you can share the link to the conversation you had, so they're looking more at the thought process and the thinking that went into the questions that they asked and not as much onto the actual output that they generated.

And so I think that's a fascinating shift for the teachers to be able to just see, okay I can kind of understand where you're going now and I understand I can see that you know what you're talking about or you're maybe you're struggling in this area, um, and thinking about that a little bit. And so I thought that was a really interesting way to kind of flip the way that they're looking at, you know, what kids are learning, um, is to just kind of watch the actual thinking process and the thought process to get to where they got to.

So I see a couple of their hands; we don't have time for more questions at this point. I do want to highlight going back to, if we think about what those principles are, um, that this was very much aligned to that first one, what are our educational goals to create critical thinkers? All right, how do we use AI to help achieve that goal? Um, so keep these in mind; I do find them helpful to drive some of the conversation that we're having.

Let me offer our panelists the last word, uh, if you were going to, uh, just provide some advice for folks that are starting out in their districts trying to wrangle, uh, the artificial intelligence and think about what they should do, what's the first step they should take? Chris, go ahead.

I get this question feels like daily now. Just call, ask, or you—I did a presentation at the conference yesterday. Um, a lot of people are getting hung up on we need to write policies. We need all of these things in place. We need somebody to tell us what to do, and I would just say you've got to just get started. Just try something. Um, there are, you know, the TJJ toolkit is amazing. It has a ton of work in there. Um, there's some other stuff put out by Council of Great City Schools and Coen. Um, look at those—steal ours! Like that's what I tell a lot of people, just take the one off our website and use that to start, um, that we've developed in our district.

So like just get started, get something, you know? It doesn't have to be, you don't have to get into this big formal process yet. Just get your teachers moving. Uh, if you don't know where to get started, just try it and get going, and then you'll be able to learn and adapt and adjust from there.

Yeah, I'll jump in. I agree with that completely and I'd actually go back to the why. Um, and I think if our entire point in public education or any education is to prepare kids for their future, our kindergarteners today aren't going to graduate until about 2036, which means that our kindergarten classrooms today should be preparing them with the skills and the belief systems and the knowledge and the mindsets and things that they'll need to be, uh, prepared in 2036.

And what that really means is that all of us have to admit that we don't know much, um, and that that's okay. We can learn this work together. Uh, we can do the AI literacy work and learn ourselves while our kids are learning as well, um, and make that push forward. Um, and similarly to Chris, find a framework that you're interested in and then just start there and then tweak it for your local conditions and what people know and what people think and believe.

Um, you know, I think communities are at a place where they're recognizing how important these tools will be in the future, uh, so the more we can make sure to prepare our kids, the better off we'll be. Great, thank you so much both for being here today and for the just trailblazing work you're doing on this stuff and sharing it with all of us, so thank you very much.

Thanks to those in, uh, who came and participated today. Please do, as Chris and Bobc said, they have materials out there on their websites. T AI has plenty of materials. I would echo, just get started, just do something, and it's better than kind of the inertia. Just getting over that inertia to do something, um, I think starts you moving in the right direction.

So thank you everyone. We appreciate your time today and good luck with your AI work and moving forward.