Given the widespread availability of generative AI tools, traditional assessments of learning conducted in an unproctored environment have become unreliable indicators of student learning. In this episode, Jason Gulya joins us to discuss the potential benefits of shifting our focus to the process of learning.
Jason is Professor of English and Media Communications at Berkeley College, where he also chairs the AI and Academic Integrity Committee. He is a co-author of Artificial Intelligence, Real Literacy: a practical guide to using AI for evidence-based literacy practices in education, and is currently writing a book on process-focused teaching in the age of AI, for Oklahoma University Press.
Show Notes
- Matthews, Paul; Gulya, Jason (2025). Artificial Intelligence, Real Literacy: A Practical Guide To Using AI For 10 Evidence-Based Literacy Practices in Education. Independently published.
- Jeanne Beatrix Law
- Perkins, Furze, Roe and MacVaugh (2024). The AI Assesment Scale
- PAIRR
Transcript
John: Given the widespread availability of generative AI tools, traditional assessments of learning conducted in an unproctored environment have become unreliable indicators of student learning. In this episode, we explore the potential benefits of shifting our focus to the process of learning.
[MUSIC]
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer…
Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.
[MUSIC]
John: Our guest today is Jason Gulya. Jason is Professor of English and Media Communications at Berkeley College, where he also chairs the AI and Academic Integrity Committee. He is a co-author of Artificial Intelligence, Real Literacy: a practical guide to using AI for evidence-based literacy practices in education, and is currently writing a book on process-focused teaching in the age of AI, for Oklahoma University Press. Welcome, Jason.
Jason: Thank you so much. It’s such a pleasure to be here and talk to you both.
Rebecca: Today’s teas are:… Jason. Are you drinking some tea with us today?
Jason: I’m not. I usually drink coffee over tea. I have my own preferences when I do drink tea, but right now I don’t have any. But if I were drinking tea, it would either be English breakfast or Earl Grey.
Rebecca: All right, classics. You’re not a complete tea denier. [LAUGHTER] How about you, John?
John: And I am drinking Prince of Wales tea today.
Rebecca: I’m drinking English breakfast today.
Jason: In some ways, I prefer the taste of tea. I don’t know why I don’t drink it more. I probably should.
Rebecca: I wasn’t trying to make you feel guilty, Jason.
John: Yes, she was. .[LAUGHTER] We’ve invited you here today to discuss process-focused teaching, which is the subject of your new book. Can you tell us a little bit about the origin story of this book project?
Jason: Yeah, it really, in some ways, started way before generative AI. I think for a lot of us, generative AI was a push to really think about process over product. I’m more than happy to talk about what that is. But really, a long time ago, I felt like just looking at a final product, which typically… I’m a writing professor, so it’s typically a paper in front of me… I felt like it wasn’t really giving me enough insight into how a student was thinking, how they were thinking about their own thinking, how they were learning, what they were learning, and so a long time ago, I felt very disillusioned just looking at a final product. And I did a lot of the process-focused teaching that in writing we do, multiple drafts and things like that, but I started to get a sense, and this really hit a head probably three or four years ago, right around the time ChatGPT hit everyone. I started to think about how to move past that disillusionment and another way of teaching, and trying to get more of an access into what my students’ processes were like, because beyond a rough draft and a final draft, I didn’t really know. And so when ChatGPT came out for many of us, it put a lot of pressure on us. And I started to redesign. That’s where the book really starts, trying to think about what that push meant to me, and trying to figure out giving students agency over the process. One of the things that I think a lot about is that there are so many different kinds of processes, and now especially that process-focused teaching has kind of been picked up by so many people who’ve tried to redesign their assessments for an age of AI or whatever you want to call it. But as I dive into it more, there’s a lot to say about processes that I create versus ones that my students create versus ones that we co-create, and so when I started to do the research for this book and talking about generative AI and what it meant for college education, I just did more and more of a deep dive, and as is always the case, it just became more and more nuanced, more and more complicated as I went along, and that became sort of invigorating because it forced me to really start to flesh out what process actually is, when I think that many times we lose touch with just how much of a black box that term is, even though everyone uses it and it’s become very much a term for the moment.
Rebecca: I heard you say that generative AI was a little bit of a catalyst to kind of move you further in this direction, in this moment, but a lot of folks have been doing some of this work over time. So you mentioned a little bit of digging deep into this literature. Can you talk a little bit about some context prior to generative AI [LAUGHTER] that you might have been looking at?
Jason: This is something that I think is really important because if you look at the broader conversation about generative AI in education, it almost makes it seem like this is a new thing. But there’s this really long history going back 60, 70, years that has looked at teaching process, and so a lot of this comes out of writing studies. Like there are individuals who really, in the 1960s, early 1970s, already started to think about how to teach process, what that looks like. There’s all these different pieces of information, and there’s also a lot of research, even starting late-70s, early-80s, into something like think-aloud protocols. Now, this is not only in English; it’s big in sociology and other fields as well. But getting someone to talk about something as they do it, it’s kind of difficult to do. It takes a little bit of practice, but think-aloud protocols in the 1980s, people really looked at when a student is writing something, and they were forced to talk about what they were writing, what they were thinking as they did it, what do you learn? And we’re actually starting to see Jeanne Beatrix Law at Kennesaw State University also has done some research in the AI version of this. I think one of the best things we can do in terms of invigorating this conversation about AI and process-focused teaching is to go back to that. Look at how process was talked about in writing studies and sociology and in other fields, because there is this really, really long research into process in the abstract, that’s one kind of research, and then also like specific strategies. This is think-aloud protocols. This is revision practices, revision literacy, like all this is linked, and there’s this really, really long history of it. And I feel like the conversation about AI in education, especially higher education, tends to jump over that. It’s such a shame because I think it’s really, really rife with a lot of ideas. That no, we can’t transfer all of them, but I think it’s surprising how much you can actually use, because I’m a firm believer that generative AI broke some things, but more often than not, it showed us things that were broken for a while, maybe we shouldn’t have been as product-focused as many of us were in our disciplines, and we can go back, look at that literature, and kind of reinvigorate it and adapt it if we need to.
John: So while there have been many people who’ve been exploring and in many disciplines exploring process-focused teaching, it’s still not the norm, although there’s certainly been a lot of push for people to move in that direction in response to generative AI. When only the final product is evaluated, you can see where students ended up, but it’s hard to detect how much they’ve learned and their process for learning, and help them develop better learning skills. What aspects of learning become more visible when you’re engaged in process-focused teaching?
Jason: Yeah, I think that one of the big benefits is that teaching process is a way to make thinking visible. One of the things that’s happened, especially now, when we’re very product focused, we can look at something that looks like it has the skills in it. It used to be 5 to 10 years ago, I could read a paper from a student, and I could see well, there was a good argument. I’m pretty sure they could argue, or they use evidence and analyze quotes. Well, I’m pretty sure they can analyze quotes. And then one of the things that’s happened with ChatGPT, Claude, Gemini, all these programs that have just been given to students and thrown in institutions, is that it’s getting harder and harder to bridge that gap. So, the idea behind process-focused teaching is it gives you a way to bridge it, to kind of figure out what’s happening with that gap. What are they actually working through? And one of the things that I’ve learned over the last few years is, as I’ve done a deep dive into process-focused teaching, there are so many different ways of doing it, and I think we need to make distinctions. If you’re talking about one form of process-focused teaching, which is giving students an interactive oral assessment, so that students come in, they have a product, something they’ve created, and then in class you can do this online too. It’s a little bit harder, it takes some creativity, but you can do it. So the whole idea is, it puts a little bit of pressure on the students, so they need to explain things, and so that’s where process comes in, because it’s something they couldn’t fully rehearse for, and it forces them to pivot. And so here you’re testing something pretty specific. You’re trying to get a sense of whether the information stuck, whether they can pivot, they can move because they can’t fully plan for a conversation, and so that’s what you’re really taking a look at. Whereas other forms, it can be very different. So if you ask a student to, for example, do a think-aloud protocol, and I’ve done this with my students at Berkeley College, I’ll ask them: for this week what I want you to do, keep writing, keep going forward with your project. But I want you to take five minutes. I want to take five minutes, and as you are writing, I want you to talk about it. And so when I look at that, what I’m really looking for is how they’re processing things as they write. So this also focuses on process, but it’s a very, very different side of it. And then another way to get a process is to really allow students to create their own process. So this we can do, and I often do in a fully asynchronous learning environment. That instead of telling a student: write a five-page paper about blank, instead of doing that, I have them, if I’m able to, if I have the leeway, to design their own project, and we do it together. Some of them will say, “I know what I want to do.” Some will say, “I don’t,” and I work with them. They design a project. They design a process. So you want to create blank? What steps are going to get you there, do you think? Do you think generative AI can help with that? Why or why not? Do you think they’re going to help with the outline? Do you think they’re going to help with reformatting?… whatever it is. And some students will just say, “No, I don’t want to use generative AI. I’m just going to go through my process.” And others will say, “I think it’s going to help here and not here.” And that’s a part of the conversation. So for that, again, it’s process-focused teaching, but the skill set is a little bit different. It’s about design thinking. It’s about can a student create their own process and manage their own process, and sometimes even evaluate their own process. Halfway through, I might say, “How’s it going? Do you need to pivot? What do you need to do?” And those are three examples of process-focused teaching, but sort of splinter in terms of how they’re approaching process. And I think one of the things that’s happened in the conversation about AI in education is that process-focused teaching has become one umbrella term that’s trying to capture all these different things, and it’s just really hard to do. And I don’t think we should have to do that. It’s very, very nuanced. So those different strategies, they focus on different skills, even though they all focus on process.
Rebecca: I come from a field that focuses on process a lot, the design field, so we’ve, over time, required students to do things like process books, or we might do a case study at the end where they talk about how they came to their final product, which is similar to your oral exam, but maybe is in a written format or has audiovisual components to that. It makes me think a little bit more about what you were talking about earlier, in that there’s advantages and disadvantages when an instructor designs the process that’s being looked at, maybe a process that’s happening over time versus discussion of process in a fixed amount of time versus a student designing their process, and it ties, I think, to some scaffolded learning too. Those are some processes that faculty design. Can you talk a little bit about some of the strengths and differences between those different kinds of examples that you just mentioned.
Jason: I think the big question is: What is the intentionality? What are we trying to teach? And I can say things like teaching process makes things visible, makes thinking visible. But the question is: To whom? Who needs to see it? And once we have those questions, what is the intention here, and also who should the process be visible to, then that helps us start to design things. Sometimes I want to see the process out in the open so that I can give feedback on it. This is what we’re talking about when we ask the students to go through the design process. What did you do? Let’s talk about it. So it’s a form of collaboration. Sometimes I want to do that. Sometimes, really, the most important thing is for students to see it, to see that kind of process and see it documented. And so when I do that, the big benefit here is metacognition, making students aware of how they’re thinking through something, and sometimes it’s more for them than it is for me. And this is where we get to reflections and journals. Sometimes I read them. Sometimes I may even evaluate them. But more often than not, ideally that is for them. And I think that those are some of the big questions that allow us to start to design things because I think it’s about balance. I think it’s really all about balance. If we go all in on teaching process for every single assignment, we’re gonna lose it. You can’t do it. On the one hand, it’s not really ideal because it’s just going to balloon everyone’s work. Students have to do more documenting. I have to look at more assignments. It can very quickly become what I call product bloat, where instead of sending in one thing, which is what they might have done 5, 10, years ago, now they’re sending in eight things, and they’re doing that full scaffold. And so you can see how product bloat isn’t good for them, and it’s not good for me. And so I think it’s okay to sometimes say I just want to see the overall process here, or maybe I don’t. There are examples of what I call process-bearing products that you can take a look at. So, for a quick example, a student writes, say, a two pager. That’s a product, but if you ask them to go through and annotate it, now it has something about the process built into it. Annotate the process you went through. Do it by pen. Do it by Google Doc. And so I think it’s okay to try to balance it. This is based off of discipline. This is based off of how many students we have. I think that one of the mistakes we can sometimes make is thinking that something that might work for teaching process in a 20-person classroom might not work in 100-person classroom, and so we make that determination and we start to narrow it down from there. And I think it’s okay to just focus on product for a little bit, and then you get the process in there later. So I think it’s all about like balancing that. A lot of it is institution, what we’re teaching, asynchronous online versus in person can be very very different too in terms of how much leeway we have.
John: And going back to that issue you just raised of scale, I can see this working really nicely in small classes. But say, hypothetically you were teaching a class of between two and four hundred students in introductory economics. Are there any ways in which process-focused teaching could be introduced into that?
Jason: Yeah, I think there can be. You know more about the field than I do in terms of how this would work out pragmatically, and I think the big decision that needs to happen at that level is how assessment is going to work. You can have students walk through the process of something in the course, a project that you maybe have had for a while, and they can walk through the process of it. But then, where it becomes a problem is if everything has to be evaluated, everything has to be graded. That’s a big decision. So I know people who’ve experimented with self-assessment as a way to get around that. Now, depending on where you teach, depending on your discipline, you might have a different approach or maybe non-approach towards that, but self-assessment is one way. Alternative assessment is another way. Students assessing each other, depending on your course, if you feel like that’s going to work. So I think a lot of it is trying to figure out how to be creative with it and what we want grading to be. So one of the things that ends up happening when we talk about the scalability of process teaching, this is where we have the conversation about AI grading or AI assisted grading, and I know people have very different feelings about this. So one way that people have done this is yes, I have a 200 person class. I want to teach process, but I can’t do it, especially if it’s a 200 person online course, I can’t manage that, and so that is where some people have leaned into AI grading as a way to approach scale, and people might have different ideas about AI-generated grading versus AI-assisted grading, but I think it’s something that has helped a lot of people. And then again, it’s just a matter of like, is it worth the trade off? There’s always a trade-off. You’re going to move away from product and towards process. Try to figure out: is that trade-off worth it? Is there another way to do it? Now, one of my hopes is that in the next couple years, we will have a way to start approaching that scale in another way that isn’t AI grading. I’m not aware of too much yet, but my hope is that we’ll continue to be creative as a community and try to figure out how to do that in a way that doesn’t just hand off grading to an AI bot. I think that many of us would balk at that, myself included. But, there is a middle ground potentially here that helps us with the scalability issue.
Rebecca: I think in terms of scalability, those different kinds of process assignments that you were talking about have different kinds of scalability. So a student that’s designing their own process that’s an interactive process with the instructor to design that process-seems a lot more labor intensive and would probably only work in a small context. But even that, if you have 20 students and they’re all doing a different process, that can seem a little overwhelming as well. [LAUGHTER] Can you talk a little bit about how you might implement something like that in a class?
Jason: Yeah, and so this is where I think we need to really decide what we need to have access to. We don’t need to have access to everything. So, for example, if I were doing a think-aloud protocol for my students, so they film themselves for five minutes, and I take a look at it. Now, 10-person class, 20-person class, cool. 50-person class, even a 20-person class, that’s a lot. That’s a lot of work hours. And so I tell them, and this takes a little bit of trust building, that actually I’m not that interested in watching the video. Sometimes I don’t watch the video. I want them to take a look at it. It’s for them. The video is for them, and instead of getting a five-minute video, maybe I get a quick write-up that I can look at very, very quickly. Sometimes I can look at it in 30 seconds, and so I think we make that determination. And I think it’s okay too to have this ad hoc approach, so that if I go and I’m looking at those think-aloud protocols, and I read one, and I say, “Well, I’m not quite sure the learning is happening here. I’m not seeing that level of engagement.” Then I can, in an ad hoc way, go and look at that video, and that allows me to use my time a little bit more. And I think it’s okay to do that. I think it’s okay to have things that are just for them. And the other thing, and again, I know this is so discipline-based, but teaching self-assessment has saved me a ton of time. Having students go through and give their own grade for themselves, I obviously come in and take a look, and if something seems really, really off, then I can step in and intervene. But in general, giving them the ability to self-assess themselves really really does help, and I think it’s okay. I don’t think that’s even cutting corners. I think that’s allowing us to have a conversation about how they’re assessing themselves. If there is a gap between how they assess their work and how I’m assessing their work. If I have a student send in a project and their self-assessment, I might be able to review that in five minutes, maybe even shorter, depending on the project, and I can see: “Oh, I think their self-assessment is pretty good. I think it’s kind of on point here.” Or conversely, that maybe I think it’s really off. Maybe I would give a student a D, but they’re giving them an A, and I want access to that, and that allows me to have a conversation with the student in a way that also does guard my time. So I think, for me, what has really helped is those different strategies, and I think that we can pick and choose them and adapt them for our own discipline. That’s in my experience, anyway.
John: So one of the things you talk about is having students design their processes individually. Could you walk us through an example of that in terms of a particular assignment, let’s say, or to achieve a particular learning objective?
Jason: Yeah, so I’ve done this in my art of film class. I’ve also done this in our writing-intensive courses. So for the class I’m teaching right now, I’m teaching a foundational literature course, and so it’s all about just engaging with story and just how expansive that can be as a category. So for that one, as their midterm project, I want them to create a product, create a project, that shows how they’re working through the course material. I can give students more specific directions if they’re stuck with that. But what I really do is I ask them, let’s design this together and show how it connects. So, for example, I will have a whole range of projects in that literature class. Some students want to do a creative writing assignment. Some students want to do an analytical assignment. So, right now we’re reading Persepolis. So, I might have some students who say write a prequel to Persepolis. They want to do that sort of work, and then others they want to do more of a traditional essay. So I work with them, and I do this as complete and incomplete, or try again. As they’re going through, and they have that project, we have a back and forth. I give the okay. That’s something to review pretty quickly. If they’re connecting it to the course, it’s something that they’re interested in. I usually give them just a complete, and then they start to build from there. And so once they have that done, once they have that project, and again, some will be creative, some will be analytical. Then we have the conversation about what’s going to help you get there. And this is where it is 90% for them, 10% for me, that they’re going to go through and say, “This is what I think I’m going to do. I want to write a prequel to Persepolis.” Okay, cool. So I’m going to go and I’m going to write a rough draft on my own. Then I’m going to use AI if they want to. I’m going to create an outline or whatever it is, whatever the process is. And for that, again, thinking about scalability, it’s all about do you have a process that makes sense? I can usually review this in about 30 seconds. You get a sense very quickly some students who are really struggling with creating a process, and others who like this is fine. This is not bothering me at all. And so, even in that part in the process itself, I call a process folio. I get a sense of how students are doing with creating processes and thinking about processes. The course learning objective that this comes down to, and this is where I anchor everything. For me, managing a writing process is a foundational core skill of the course, what a lot of it is about. To give a sense of how I start to create that, so when I look at a class and I’m trying to think about a process-focused assignment, first thing I do, I look at the course goals. These have been established for a while. What are the course goals that I need to hit? Then I break it down into the core skills. I usually pick like four of them. How can I boil it down even more? And then I link this assignment to that. And so that’s a way for me to make sure I’m actually hitting the course objectives, the learning objectives that I need to hit, even as I’m taking a somewhat unconventional approach, and I think that’s the sort of balance, right? You want to do both. If you’re going to focus on process, you can be creative, you can push things a little bit, but you do want to make sure you’re hitting those course objectives because that student needs to have certain skills when they graduate, go on to the next course. So I think doing that sort of work, starting with the learning objectives and working back from there, has really helped me out a lot. It also prevents me from starting from just step zero or step one. It gives me something to work with and make sure that if a student gets through my class, they’re ready for the next writing class, even if they’re doing something a little bit different. That’s been my process anyway.
Rebecca: Can you talk a little bit about the difference between students experimenting with process in like a 100-level class, where they maybe don’t have [LAUGHTER] a general sense of the processes used in the field or discipline, or maybe one a little further along in their academic journey, where they might be more deeply rooted into a discipline and might have more exposure to different kinds of processes they might use for a project.
Jason: I’m a big fan of the gradual release model, so scaffolded appropriately. And AI can actually help a lot with this, and it can allow us to give more scaffolding to some students and less to others. So, for example, for a lot of my classes, for the first project, I give a lot of scaffolding. It walks them through. This is how you create a project. For me, it’s kind of it may seem self-evident, but for a student who’s just learning the discipline, not at all, right? If someone tells you, oh create a project that’s connected to the course. So, what does that look like? And so, my impulse is to always give a lot of scaffolding for the first project, and then gradually give a little bit less as we go along. The other big way that I do this is with self-assessment. If I ask the students to self-assess themselves, some of them will just be like, all right, I kind of know what you’re talking about. Others have no idea, and so I actually give my students a couple of ways to do it. First way is the AI-free way, so they think about this is what I want to create, these are the skills I’m working on, this I’m going to self-assess. And in any individual class, I probably have about 40 to 50% choose that one, and then the other thing I do is for students who need more scaffolding and need more support, and they just need it in the moment asynchronously. I give them a custom chatbot that walks them through the process. So it asks them, “What are you trying to create?” All right, cool. Build from there. “What are the skills?” And it actually can create a rubric with the chatbot. Now suddenly, this is an activity that is more scaffolded for the students who want that scaffolding, and it’s something they can do in five minutes. Whereas without that scaffolding, it might have taken them an hour, maybe more. And then I can see that, and then when they send me the overall chat, I can quickly scan through it, and I can see the process work itself out as they’re dialoguing with this, and so my rule of thumb is to always give them more and more agency over the process as they go along. So that means a lot of scaffolding early on, and if I have two projects or three projects, I let them create the process a little bit more. And now for foundational courses, I really lean into that. If I have a student in writing and research, like I really try to carefully scaffold it. If it’s say an upper-level elective course, very very different. You might not need a ton of scaffolding early on, so I do personalize that for level and discipline. A writing-intensive course versus an elective course might be a little bit different, and it’s different in terms of course learning objectives that you need to hit. And so I’m kind of informed by that as well as I design. And then, the other thing that’s going to be interesting for us is that my college this semester has taken on AI literacy as a college-wide objective that we need to hit, and so we’re going to start to see that work its way into the conversation too. If a student is taking, say, our writing and research course, we need to have a way to tie that course to that overall college-wide learning objective. So that affects the scaffolding, that affects what you do, that affects if a faculty member wants to create a custom chatbot and link it to that, and so that’s going to be another additional wrinkle, I think. And we’ll see it cutting across different departments because students in every department are going to have to show some evidence that they worked on that competency, that literacy, whatever you want to call it.
John: One concern that many faculty have is that students are using AI to offload their cognitive processing and strategies such as designing chatbots where students are interacting with that to help design a process can be really productive because the research is suggesting that when AI is used intentionally in a manner to support student learning, it can help students learn. But we don’t always see that. One of the things I know some faculty here have been doing is they had been using in asynchronous courses metacognitive reflection discussions and logs and so forth, and they’ve been seeing more and more students turn in reflections that appear to be generated by AI rather than the authentic voice of their students. What can we do to help ensure that students are using AI appropriately. Having students share their interactions is a really good way of doing that, so you can observe it as part of the process. But how do we deal with those cases where students turn in something that doesn’t appear to have been created by them individually?
Jason: I do think getting full transcripts when you can really helps. It gives you a sense of their process, how they’re dialoging with it. My approach has been this: I front load early on in a semester, demystifying it. So custom chatbots are a great way to do this. I also design for my students pressure bots, and I have my students doing this next week. So they did some of the reading. They got the page 70 or whatever it is in the book, and I’m giving them a pressure bot that is designed to put pressure on them. I told them it might frustrate them a little bit. It’s not going to give them answers. It’s going to ask them for clarification. Going to ask them to go back to the book. I want the book in front of them. They can do some of it from memory, but in the end, I want that student to go back to the book and take another look. And so I gave it instructions for doing that. And I’ve done this for a while. One of the things that really changed for me was… it seems so simple now… show them the prompt and so they come back and we have this conversation, I say, oh, this is how AI helped them learn, and I kind of say, well, AI did, but it’s because I did this to it, and you can like show them I had to change it. I had to basically, in these custom instructions, change it from a sycophantic chatbot that’s going to say you’re awesome and great and just keep going, and towards something that’s going to challenge you, and there’s all of this knowledge that needed to happen in the background. And showing them has been really, really helpful for demystifying how this works and making it clear. And some students, I think, already knew this, or it seemed very obvious, and some not so much. If you go and you open ChatGPT and try to do this, it will not work because it doesn’t have this background. It doesn’t have this push for it. So that has been very, very helpful in demystifying it, and then getting transcripts really help. And then also the other thing that’s changed in my approach is after it’s all said and done, a student has gone through the process, they’ve created the thing, asking for a transparency statement that really has them walking through what they did and why they did it. So I have students at the end of a paper. I ask them, “What are the steps you took? Did you do a draft? Did you do an outline? Did you draw something out? …whatever it is. And I have them go to the AI Assessment Scale. So this was created by Leon Furtze and Mike Perkins, and so very simply, for a step in their process, they give it a one if they didn’t use AI. I will have students who say, “I didn’t want to use AI. I didn’t work it into my process. One’s all the way down.” A one if they didn’t touch it. A five if it was full on AI. I rarely get those. Every once in a while, they just say I created an AI outline and went from there. So go through the process, give it a number from one to five, and think about your intellectual ownership. Did you feel like that actually served you as a learner? Were you handing something off? And so, it’s not foolproof by any means, but it tries to get at how, when we use this technology, there’s always a trade-off. What we might gain in efficiency, we might lose in intellectual ownership, or anything like that. And so that has sort of helped. And over the years, the other thing that has helped me is… and this just sounds so obvious, but I think it’s kind of hard to do… really leaning into asking questions. It seems so obvious, but say, “I want to talk to you about your process. What did you do?” I think for a lot of my students, it’s hard because that feels accusatorial, and I’m learning to lean into it a little bit more. I took a look at what you did, can you walk me through the process? Did you happen to use AI? Why or why not? And just trying to really make that conversation normal. It’s not even just that I’m policing it or accusing it, but try to say “I really want to talk about this. I think something happened. Can you work with me to try to figure out if that happened or that didn’t happen?” And I always felt like that was going to automatically feel like I was catching them, and all that. But I’ve actually found exactly the opposite. Depending on how we approach it, it is a way to collaborate with the students. And I try to make it clear, this is us working together because I want to talk about your process and figure it out. It takes some ironing over to do that, but having that transparency statement, having that demystifying conversation early on has helped me scaffold for that conversation because then I talk a lot with my students about how AI could be a learning tool, but only if you use it in very specific and very intentional ways.
Rebecca: I know you were calling showing the prompt that you wrote, demystifying a little bit, but I also think it’s modeling process. Your teaching process, your modeling process, you’re talking about process. So can you talk a little bit more about the role of modeling some of these skills for students?
Jason: I think modeling is one of the most important things we can do. So when I create a scaffold for my students, I walk them through it. Why did I create that scaffold? If I’m telling students for that first project, this is what I want you to do. That is a model process, and I think sometimes I myself I never leaned into just how that works. So if I ask them, all right, I want you to free write first. Cool. Why did I do that? Why did I feel like, oh, you can’t just free write with ChatGPT? I had you free write, then I had you create an outline. Whatever the scaffolding is, and that is a model process that the instructor has created. And I think that a lot of it is showing students that step back and being very honest when we learn something about our process. So one of the example I give to my students is after ChatGPT came out and other programs started to come out. There was a few months when I was generating my PowerPoints with AI. I’d have like notes, I’d have a document, I would generate it and then I would change it. I would revise it, all that sort of stuff. And I end up walking through my students, this is the process I’ve created, and then I junked it because I felt like generating things with the slide made it so that I couldn’t present on it. I had a hard time presenting, pivoting, doing that sort of work. And so talking to them about the processes we create, why we change them, how we go forward. I think it’s probably one of the most valuable things we can do in the classroom, and it helps them to see how important process creation actually is. I think that many of my students, when they come in, they’ve been taught by the education system that what really matters is that final product, and that final product does matter, but I try to use modeling as a way to get them to think about how does scaffolding or any sort of laying out of the process, how does it help us? Why are we doing things? I think that my impulse is always just to have the scaffold there and not really talk about why I designed it that way. And I’m trying to be better about that, just showcasing the intentionality because I think it helps them think about things a little bit differently.
Rebecca: I feel like one thing that I’ve also done in that same space that has been effective is pointing out parts of the process that maybe as a student I thought were a waste of time, or I didn’t fully understand why they were there, or I would skip over it or shortcut it in various ways, and why I don’t do that now. But I found those conversations to be really powerful conversations about what is actually the point of this step.
Jason: That was my experience in high school. I hated, I hated writing in high school because of how it was scaffolded. Because I was told do this, do this, do this, and do this. And even when I was a high school junior and high school senior, someone would come in and say, “All right, the first thing I want you to do is to create a thesis statement.” No, that doesn’t help me. I need to go through the actual writing process to figure out what it is. And then they would say, “Do a thesis statement, do an outline, do this.” And I started to realize at that point that, like, “Oh, this process does not work for me. It’s great that it works for someone else. Great that it works for several of my classmates. Does not work for me.” And this again is something that we can take from process studies. In the early 1990s, there was a whole post- process movement. So this is something that was almost like 35 years ago that people were thinking about, “Oh, we all have different processes. It doesn’t work for everyone.” There isn’t a linear process we can do that gets us from point A to point B to so on and so forth and final product. The processes that we use are all different, and they should be different. And I think that sometimes we lose how fragmented that can be. But again, there’s all this scholarship about how to teach in that context, and sometimes we just need to go back to it and look at how much of that carries over to the current moment. And my experience has been that quite a bit of it does carry over.
John: So, for those who are thinking about adopting process-focused teaching, you’d recommend that they not try to impose a process on all their students.
Jason: So yeah, find ways in which, if you’re teaching process, make it conversational if you can. Now there are certain limits to this if you’re teaching online or teaching a large number of classes, but find ways to make it as conversational as you can. And this actually can be a way to also manage your course load so that you’re not designing a process for everyone. You’re working with them as they create their processes. There are ways to manage that, and again, it’s about scaffolding. For that first project, I think it’s okay to control the process a little bit. You just want them to see it. You want them to see what that process looks like, and then gradually release more agency so they can think about their own process. Think about what works for them, and make those sort of selections. That also, for individual students, might help cut down those seven steps to three steps. If I do a scaffold for students, it might be like seven steps or so. I’ve never had a student who, when I ask them to create their own process, had a 10-step process. It’s always like two or three or four. It’s always on the lower end. I think it helps us in terms of scalability too.
Rebecca: One thing that has been helpful when I’ve thought about this and providing flexibility is, for example, we need to do something to come up with an idea. We need to do some sort of ideation, but there’s lots of ways to do that. [LAUGHTER] But you need like a component of a process that gets some ideas out, and maybe offering like a menu of options sometimes can be really helpful too.
Jason: Gives them that agency, and I think that is so key, and it’s one of the reasons too why I give students as much agency as I can while also hitting those objectives over the project they’re creating. This is one of the first principles of teaching process from the 1970s. If you’re going to teach students to own the process, they have to have some ownership over the product. That’s like principle number one or principle number two, when it was laid out. I think that’s just so key, and also it pushes against the incentive to overuse AI or misuse AI, to own the project you’re creating. No, it’s not going to make everyone be honorable and do all the work. But I think it helps when you are creating a project that you have very little buy-in, you didn’t help create, you didn’t help put that sort of work in, AI becomes an even more attractive option for bypassing the learning. So I think giving students agency, giving them ownership in the material is one of the best ways to push against that and get them to do the learning, which sometimes requires hard work.
John: You’ve suggested some strategies that can be used in any class, including asynchronous courses. But are there some aspects of process-focused assessment that works well with asynchronous courses when you don’t get to see the students individually on a regular basis.
Jason: Two things: one, really think about products that you can take a look at that have something about the process built into them. Those comments, some sort of combination that you can look at a product pretty quickly and see what actually was built into it, how was it designed, that sort of thing. And then the other is lean into asking questions. I think that if I ask for every process along with every assignment, especially in an online course, it blows up so fast it’s overwhelming for everyone. But if I’m looking at say 20 projects and I ask a question about the process with one or two of them, that’s doing my due diligence. And again, it can be conversational. I think there are ways to phrase it so that it becomes a part of a collaborative exercise to figure out what that process was, how it can be improved. And those have been ones that really, really helped me because if you’re teaching a pretty large online asynchronous class, you need to do it in a way that guards your own mental health, if you can. And those have been the two that really, really helped me. So that I’m not asking for a page-long process statement from 20 students for one week. I can’t do that, but I can do ad hoc questions, make them conversational, have a back and forth. That’s much more imaginable to me. And I make it very clear to students, I ask you for clarification on a comment on Canvas, the LMS that we use, that is a part of the assignment. Writing back to me is a part of the assignment, so we can have this back and forth, which they’re not usually used to early on. But if I do it a couple times, they become very used to it pretty quickly.
Rebecca: We talked a little bit earlier about your pressure bot, and then also a bot to help design a process. Can you talk a little bit about ways that we might use AI to do some critique of our work or challenging students to interrogate their work a little bit more before submission? [LAUGHTER]
Jason: The model that has been really, really helpful is the PAIRR model. So it’s P-A-I-R-R. It’s peer and AI review and reflection, I think i’s the acronym. So this is out of the University of California San Diego, and so what they found was the big thing was you use AI assistive feedback, and it’s very very helpful for students, and you sandwich it. Basically, you start with peer assessment. I also sometimes start with self-assessment. How do you think you did? Then you can get feedback from an AI program. You can ask for the transcript, however you want to do it, and then you have them review and reflect on it. And so it’s really about anchoring that feedback. They go and they take their paper and they put it into a feedback generator or an AI grader, which my students are increasingly doing. Do you agree with it? Would you change it? Do you think that grade is accurate? And that allows them to anchor it a little bit more. That’s the key to it. They can get that feedback, but I also want them to engage with it critically, because I think the impulse for my students is to just take all of the feedback from an AI bot and just implement it. And I don’t want them to do that. And I’ll tell them, if I even go through something like Grammarly, it’s on my own work. I’m constantly saying yes, no, yes, no, yes. You use your judgment, and so I think that’s the muscle that needs to be developed. And a lot of it is contextualizing the feedback rather than just letting it stand on its own. Yeah, and you compare it. What did the AI bot tell you? What did your peer tell you? If they’re different, why do you think they’re different? What am I telling you? And so it allows them to kind of see the link, which is exactly what a lot of times we’re being asked to do in the workforce. We go through, and we have a project. I do this all the time. You have a project, you send it out there, you get three sets of feedback, and you have to like judge. And they don’t complement each other a lot of times. They contradict each other. You can’t hit all those sets of feedback. So I think that there’s an argument to be said that this is part of workforce training. How do you implement that feedback? And AI just becomes one set of feedback among others, which I think is exactly where it should be, so that it’s not getting too much gravity and becoming the center of attention, and it contextualizes it a little bit, and also starts to demystify the feedback in a way that’s just one source among many.
John: Suppose someone wants to begin implementing process-based assessment. Can it be something done at a small scale before they completely revise the design of their courses, or is it something that should be done throughout the course as a complete change in structure?
Jason: Start small, please start small. Make it easier for yourself. Do it for an assignment. Do it for a mini assignment. So start small. Build from there. In my experience, if you try to make the entire course process based. It’s a lot to deal with at once, and especially because I do believe that a lot of teaching process is discipline-based or discipline-sensitive, I’ll say. And so if you have that assignment, you have that scaffold, that’s one way of teaching process. Take what you’ve already learned, move it towards something else. Try something a week. Maybe you try to think-out-loud. Maybe it works. Maybe it doesn’t work. You can sort of pivot from there. So I would say you definitely try small, iterate, figure out what you’ve learned from that, and then you can design from there. I don’t think you have to do everything all at once or do all of your classes. I say if you’re just getting started on that journey, start small in one course and then build as you feel like you learn more about it.
Rebecca: Well, we always wrap up by asking: “What’s next?”
Jason: Yeah, so I think the big thing is trying to figure out how grading works in this conversation. This is an ongoing project for me. I have a lot of provisional ideas, and I’ve changed my grading over the last couple of years. But if the process-focused classroom, and I think it is, is about experimentation, it’s about you try something, you try that process, and it fails. How does grading fit into that? Because the students get to me, and they’ve had 12 plus years, usually in a formal education system where everything is made visible, everything is graded, everything is given a score, and sometimes, especially when students go on their phone now, they go into the LMS and they’re greeted by an AI bot that offers to do something for them. I think a lot about the incentive structure, and this is big-picture stuff. Trying to figure out if we’re going to teach process in this way, where generative AI is just going to be embedded in everything. Do we need to think about grading again? What sort of grading systems helps? Is there an incentive structure that students are surrounded by that we need to start to challenge and poke again? So that’s increasingly what I’m thinking about because it’s really hard to get students to experiment and play in a classroom when it’s like, oh, they’re gonna get a 70 instead of 100. So that’s the current part of my project that I’m always stuck on because grade reform is really hard, and we’d have to wait a really really long time for if it ever happens, and so trying to figure out how we can push against the incentive structures of transactional education, even within our very pretty limited spheres of influence. That’s where my mind is right now.
Rebecca: Well, thank you so much for joining us. It’s a great conversation and plenty of things for everyone to think about.
John: And we’re really looking forward to seeing your book.
Jason: Thank you so much.
[MUSIC]
John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
John: Editing Assistance provided by Madison Lee.
[MUSIC]