407. Covering or Teaching

New faculty often begin their teaching careers by emulating the teaching methods that they have observed, but these practices are often not consistent with evidence on how students learn. In this episode, Chris Hakala joins us to discuss the role that educational developers can play in shifting faculty focus from presenting to teaching.

Show Notes

Transcript

John: New faculty often begin their teaching careers by emulating the teaching methods that they have observed, but these practices are often not consistent with evidence on how students learn. In this episode, we discuss the role that educational developers can play in shifting faculty focus from presenting to teaching.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Chris Hakala. Chris is a professor of psychology and is the Director of the Center for Excellence in Teaching, Learning, and Scholarship at Springfield College. He is the author or co-author of numerous books and articles on the science of learning, assessing learning, large-class instruction, and on teaching centers. His most recent work, co-authored with Leslie D. Cramblet Alvarez is Understanding Educational Developers: Tales from the Center, released in late December 2024. Welcome Chris.

Chris: Well, thanks. Thanks for having me. I’m really excited to be here and have this conversation with the two of you. So I appreciate it.

Rebecca: Today’s teas are: … Chris, are you drinking tea?

Chris: So this is an interesting thing. I’m routinely a coffee drinker. However, my wife and I every morning get up in the morning and we share time together. We actually have a station in our bedroom where we make coffee and tea. And so in honor of my wife and the tea, I’m drinking her tea, which is a Zen green tea, which I will sometimes drink as well. So yes, I’m having Zen green tea this afternoon.

Rebecca: Excellent. Thanks for ditching the coffee for us.

Chris: Happy to do so, happy to do so. I’ve had enough of it already today.

John: And I am drinking a spring cherry green tea, even though it’s late summer here in Oswego.

Rebecca: It does feel more like a spring day today, though, after being so hot for so many days. I have a weird mix of random black teas because I was cleaning out my tea cupboard, [LAUGHTER] so it’s an eclectic mix.

Chris: Whatever gets you through the day, right?

John: I imagine no cherry teas, though.

Rebecca: No, they are black teas without any weird flavors, John.

John: We’ve invited you here today to discuss the relationship between pedagogical practice and what research tells us about how students learn. First, though, could you tell us a little bit about your path to becoming an educational developer.

Chris: Sure, it’s an interesting path that actually we talk about a little bit in the book, but one of the things that is fascinating to me about this field of educational development is it’s not a field people often go directly into out of college. And so I went to college in Vermont, at a small school in Vermont, went to grad school, and in grad school, I earned a PhD in cognitive psychology from the University of New Hampshire. And I know we’re going to talk about this a little bit more later, but I just want to point out that during that time, we had a course in a seminar in practicum and teaching of psychology, and that course was taught by a faculty member named Victor Banassi, who was doing teaching center work at that time. So it was the first time I began to hear about educational development. It was back in the mid 90s, and so I thought it was kind of interesting, but I went on, I became a faculty member, I received tenure, and then as I was working through my career, I began to notice that many schools had teaching centers. Many schools had the opportunity for faculty to talk to other faculty about what they’re doing and to share ideas. And so I approached our Provost and asked him if we could start a teaching center. And he said, “No, there’s not enough money.” And so I approached him again, and he said, “No, we don’t have enough money.” And finally, I was able to steal an idea from Todd Zakrajsek, who is a name you are probably aware, and Todd recommended me just buying coffee mugs that said Center on it. So I did. I bought coffee mugs that said Center for Teaching and Learning, and I had them branded, and then I handed them out and gave one to the President, and the President said, “Do we have a teaching center?” And the Provost said, “Well, I guess so.” And so they gave me a little bit of money, and so I had this very small teaching center. I would do a little bit for faculty orientation, I’d run some workshops, and it became really interesting for me to see how sharing ideas with faculty about teaching and hearing what they’re saying and what they’re doing and what they’re experiencing with their students was a really wonderful way to engage with my colleagues, to go across disciplines and to really sort of work to improve our students’ learning. So I left there and became a full-time center director in 2014, and I’ve have been doing that ever since. It’s been an interesting process, because as we were speaking off-mic, in 2020 our world changed dramatically, and teaching centers became very important. And it was a really interesting time to see all that work ramp up. And I think now we’re in a different sort of era of teaching centers, as people began to realize the value of educational development when they needed us. Yeah, so it’s been an interesting journey.

Rebecca: I see some sales of teaching center mugs going up [LAUGHTER] with the sneaky strategies for starting teaching centers, moving forward. [LAUGHTER]

Chris: That’s all Todd. It’s all Todd.

Rebecca: He’s always good for a good idea.

Chris: He is good for a good idea.

Rebecca: Most faculty begin their teaching careers without the kind of training that you mentioned that you had as part of your graduate program in evidence-based teaching practices and kind of rely on what they experienced with their own instructors. What are some of the disconnects between learning science and pedagogical practices?

Chris: So it’s a great question. It was really interesting. I’m gonna go back to go back to my experience and sort of talk through that, because I think it’s a great question. I went into my first experience teaching in 1992… by the way, I still have a video of it. I had spectacular hair. It was wavy and flowy… and I went in with this idea that I was now the disseminator of knowledge. I was going to share what I knew with these students, and I did what I had experienced the faculty that I admired do. I lectured. I had an overhead projector, I wrote on the board, and my students were doing the tasks I was asking them to do. They were studying. They were doing well on quizzes and all of that. But what became really apparent to me over time was student learning is not as simple as they just remember things, because there’s more value in students taking what we teach them and expanding that out to other concepts. And so I began to think about it in lots of different ways, and my teaching now looks very different. One of the things that becomes problematic, though, is, as you brought up, Rebecca, many faculty end up in a teaching experience without having any kind of guidance or support. They have very little in the way of reflective teaching, and they think their job is to just share what they know with their students. I hear “covering content” so much from faculty with this idea that they have to make sure they say everything. And I think that helping faculty understand that the magic words coming out of your mouth doesn’t mean a student learns it is a really important thing. And one of the distinctions I draw up with my faculty all the time is this distinction between presenting and teaching. And so I think a lot of faculty are good at presenting, but they haven’t really thought through teaching. And so a lot of times I spend time working with our faculty to talk to them about who they are and what they bring to the table as an instructor, who their students are and how they can begin to find what is going to leverage what we know about student learning in the classroom in the context of who they are and who their students are. So for example, I tend to be pretty expressive in class. I tend to move around a lot. I tend to do lots of interactive things, but that’s also part of who I am. You don’t have to be that to be successful. You can be lots of ways in the classroom, but it’s a matter of understanding that the processes I’m engaged in are to help students focus attention, to help engage them, to draw them in with different anecdotes and stories, to try to tie those things together, and then have them reflect it back to me so we can work through that. People have different approaches, but you have to sort of to be aware of that and know that the student learning is essential. The key to student learning is that they do the work. It’s not that you’re doing all of the work. They’re doing the work. And so you have to help faculty sort of navigate that. And it is fortunate for me. I’m at a small college where teaching is really emphasized and faculty care about it, that I get to work with wonderful people and help them understand some of the things that we know about human learning, and help them sort of integrate that into their classes.

John: And we have lots of faculty who are very interested in improving their teaching, and we also are a teaching focused institution, but there are some faculty who are a little reluctant to change their practices, or consider changing their practices. What are some strategies that you’ve used to help reach out to those faculty who may not see the benefits of changing the way in which they’re teaching.

Chris: Yeah, the recalcitrant faculty problem. It’s a big one. I have had a mixed success. When I arrived on campus, I was fortunate to have someone here who handed me a list of people. On one side was “these people will come to everything you offer. Don’t worry about it. You could roll out a boring thing, they’ll show up.” “These people would come if it’s interesting.” “These people will never darken your door.” And so I went right after the people who would never darken my door and went and had a conversation with them about their teaching. Now, this doesn’t scale up to a big institution because there’s too many people, but in a small institution, it was really effective for me to go talk to faculty, find out what it was they were doing in the classroom, ask them if they were happy with it, and start to look at things in their classes they might want to work on. Now, again, you scale it up to a big institution, you start with departments and ask the department, “what would you like support with?” But one of the things that’s been really helpful for me as a center director is showing faculty things that they can implement, that they can see the impact on, and it’s been simple things. One of the things that our faculty in our health science really like is that I taught them about retrieval practice. They had not really thought about retrieval practice, and so now they’re using tools, including some AI, to build out many little quizzes the students can take on their own that seem to be benefiting the students’ ability to do well on these very high-stakes exams. So it’s a matter of finding the niche where the faculty member’s issue and your expertise can come together. I also strongly, strongly encourage faculty to watch each other teach, which is a really interesting thing. I have an open-door policy. Faculty can show up in my class anytime. It doesn’t matter when, they don’t have to tell me, they can just come. And I have 5 to 10 faculty a semester show up in my class and they just are there, and we talk, and then I go to lots of other faculty members’ classes, and we have an open-door week where we open up classes for faculty to go to each other’s classes. I think that’s really helpful for people to see lots of different ways of doing things. I have a line that my associate director always gets annoyed with when I say, but I believe this. I have faculty who say to me, “Chris, I’ve heard that if you’re funny, students really like it.” And I say “that’s true, but only if you’re really funny. If you’re not funny, it’s terrible.” And I’ve seen faculty try to be funny and they’re not and it fails. So one of the big messages is, “lean into who you are.” It’s okay if you’re a bit of a dork. I love statistics. My students, I get them on board eventually, but I come in with the enthusiasm of someone who loves statistics. So it’s a challenge, and we’re humans. So John, you know, humans don’t always respond well, when you say, “we can help you do something better.” So there’s a lot of nuance to it. I will say teaching centers should never be where bad teachers are sent. I push back against it.

John: We do too.

Rebecca: I think one of the things that’s interesting about teaching centers, and what you said about faculty not having support, is that actually a lot of institutions have teaching centers, so there is the availability of support, but getting said support is not built into requirements or time is not necessarily allocated by the institution to participate in teaching center things. It’s voluntary. It feels extra sometimes. So you talked about your list and individually reaching out to folks. But what are some other strategies to get over this hurdle of helping folks realize that this is a good investment of their time.

Chris: We have some events that are campus wide, and we use these events as a way to try to promote options of how you can interact with the center. So to just give you a couple of examples. Our President is incredibly supportive of the center, so twice a year, we host an evening event at the president’s house, it has wine, that goes over well, and we bring in speakers, and we try to get speakers who will hit a variety of different areas of interest for faculty. And we often will get faculty at those events that don’t come to anything else, but then may come to other events, as they’ve seen the value in those events. So that’s been a good one. Diversity of programming has been really helpful, and I’m super proud of this. We run a summer conference. It’s a week after commencement, all faculty are off contract, and we will get roughly 40 to 50% of the full-time faculty showing up at our Summer Conference off-contract because we structure it in such a way that faculty are talking to each other. We’ve been very fortunate having some really, really amazing outside speakers, and then we give faculty the opportunity to do some chatting with their colleagues, get to see people that they don’t normally see, and that’s been another way. Part of it is having a big footprint on campus, Rebecca, and being involved in lots of things. I go to department meetings once a year. My associate director goes to department meetings. But I will say this, our office also has this footprint, because we not only oversee educational development, we oversee all of instructional design, we oversee all online course builds. We also oversee all student research. We also oversee the IRB. We also oversee sponsored research with our staff of two. So it’s hard to have something on campus that doesn’t involve reaching out to one of us once in a while, and that’s been really helpful. But it is, it’s challenging, because those resources are there and they’re often not used as much as they could be.

Rebecca: I think it’s important to ask how you went from a mug to overseeing that many things.

Chris: I switched schools, that helped, and then… I said this to John… this is 33 years of me being in higher ed, and you learn a lot when you’re just around in higher ed. And I’ve been tenured full professor for a long time, and I still teach, and that actually buys me credibility among the faculty, but I also manage budgets, that buys me credibility among the administrators. So that’s all been really helpful. It’s a needle you have to thread sometimes, and teaching centers are interesting because if you look at it, I mentioned before how we were able to show the utility of teaching centers. But as budgets get tighter and things begin to change across the landscape of higher ed, you’re starting to see teaching centers get shrunk and cut again. I know of at least two teaching centers at major institutions that were simply cut and all the people were fired because of budgetary crises. And I’m actually down four staff members right now, which I may or may not get back. So it is a very difficult time financially, and you work for a state system, so you know what that’s like. You look at the state budget. We were private. Stuff just happens in a private institution. It just happens. It’s not easy.

John: One of the things we found has been helpful to reaching out to more faculty is getting faculty who were in that wave of people who were interested in attending more workshops, having them do things and then present on it to other faculty, because the more faculty hear from their peers about how things have worked and made their classes more enjoyable, the more likely they are to show up and to try these things as well. There’s still some areas, though, that it’s difficult to reach and we keep trying.

Chris: Yeah, John, it’s a great concept you’re talking about, and that’s the Summer Conference. I do nothing at the summer conference but serve food. I simply show up and all my colleagues talk to each other. And that’s really the joy of it, because they see other people, and we have a SOTL fellowship program, and so they’ll present their scholarship of teaching and learning work, and the opportunity to do that is often gonna draw people in. It really is a grassroots thing. And one of the interesting things that Leslie and I saw when we were working on this book is that there’s no one way to run a center. It’s deeply contextualized into the context of the institution and the structure and culture of the institution. And some places have had a long culture and a long history. I was the first full time director at my place here, and by the way. I didn’t say this before, but we’re Springfield, Massachusetts. I know there’s Springfields in every state, and I feel like I’m contractually obliged to say, if you know anything about Springfield College in Massachusetts, we are the home of basketball. Kansas wants to claim it, but we are where basketball was invented, so I just have to say that. My president will be proud if she hears this.

Rebecca: These are important details.

Chris: These are important details.

Rebecca: Can you talk a little bit more about how teaching centers can continue to help faculty align their practices so that we’re really teaching students how to learn and building that into the work that we’re doing, and help faculty see value in spending some time or allocating time in their classes to that kind of work.

Chris: Sure, the change in teaching practices over the last 30, 40, years has been pretty remarkable, and a lot of it has to do with the changing demographics of students entering college and the changing demographics of people teaching them. And what we’ve learned as we’ve looked at the science of learning is that human learning hasn’t shifted as much, but there are other outside variables. And I like to talk about, with faculty as we discuss these things, the notion that a student is going to walk into your class, any given student with a number of variables that are impacting what they’re doing in the classroom. Some of them you have some control over. Many of them you don’t. One of the things that you can help them with is a simple concept, such as attention. And I always start with attention, because it’s the one that faculty can begin to sort of wrap their head around. And I say you can help by using the classroom space in a way that is not going to cause them to have other distracting variables pull them away. And then I get the same comment, “well, students’ attention spans have changed.” And my response always is, “Nope, hasn’t. Hasn’t changed at all. What has changed is how many things they can be distracted by.” So you have some control over that. What are you going to do about that? And so it’s a way in, and then I can talk to them about… and because I’m a practitioner who does this research too… I can talk about data. Now, by the way, the data always get the bench scientists. They love that, because I’m talking about we have results that show this. Here’s the data, here’s the results, here’s the interpretation of the results. And things like I mentioned, retrieval practice, things like interleaving, or worked problems, the data are pretty strong that these kinds of procedures are effective. Now what is really, for me, the joy is working with a faculty member to see how that particular process works within their class, with their content, with what they’re trying to do, and try to really personalize it to their approach. One of the ones that my faculty seem to really like is this notion of interteaching. It’s a way to help students sort of structure their reading and so students get questions before they read, and they answer the questions, but they come in with two copies. One copy goes to you, one they sit and they discuss with a partner, and they look at where they answered the questions the same way, they answered them differently. They talk about it, and then everyone comes back together and talks about it. It seems like they’re going to not say much to each other, but they actually get pretty competitive about who said what on these things. But the real joy is not that they get it right or wrong, but it’s rather that they’ve thought through these things and now they’re engaging with the concepts in a different context. And I always talk to faculty about the key is to get the students to not just hear it, but to do something with it, to engage with it. And so how you structure that is the most important piece in terms of that they’re doing it, but how you do it is going to vary. So I think that that’s where I get some buy in. But I will say, Rebecca, it causes people to have to change their classes and they don’t like it, because you have your slides and you have your presentation, and you know what you’re going to do, and this opens up the probability, and I say, probability, not possibility, that class is going to be derailed sometimes. That’s okay. I love to tell faculty I teach two sections of intro sometimes back to back, one at eight and one at nine. I might be two chapters apart in those two classes because they’re going at different rates, and I’m okay with that.

John: So you encourage faculty to build in some forms of retrieval practice and interleaved practice, spaced practice and so forth. But sometimes students react poorly when you build that into your courses. Do you have any strategies you share with faculty in terms of getting buy-in from students? Because students, in general, in a number of studies, prefer to be lectured at, but we do know they learn more when they use evidence-based strategies. How can we get students to buy into that?

Chris: I’m going to tell a story while I talk about this. Part of it is being incredibly transparent about why students should be doing these kinds of things. And I think that’s pretty obvious, but I’m gonna tell you a story. I did this study at a different institution, and we never published it. We had two sections of neuroscience, and they both had 60 students. So we had 120 students total. And for the students, for exam one, the students were told in class A, we’re going to teach you how to concept map for studying. So we taught them how to concept map. Class B, study however you want. For exam two, we flipped it and we flipped it all the way through, and what the students found was when they concept mapped, they tended to do better than the other class, and then on their next exam, they did better when they concept mapped then when they didn’t concept map and we showed them the data. We showed them students tend to do better. For the final exam, we told the students, “Pick whatever you want. It doesn’t matter. You study however you want. We’re done with the study. Just study however you want.” They took the exams, and then we just asked them, “What approach did you take?” And the idea was, many of these students would adopt concept mapping, because it worked. Well, much to our chagrin, the number was so low that it was actually zero. None of them concept mapped. They all said, “I know it looked like it worked there, but my way is better.” So you’re absolutely right, John, it’s really difficult sometimes to convince people of these things. The approach I tell faculty to do is to introduce these things slowly into your classes, to build up and get small gains. So I’m a firm believer in formative assessment at various times throughout the semester, including my intro psych class, we do an assessment every class. It’s simple. It takes a minute or two, but it’s a way for me to gauge where they are. And I share the results with the students, and I talk to them about why the results might look the way they look, and I talk to my fellow colleagues about how to try to incorporate that in terms of how they’re doing it. The students, they do not like when you do retrieval practice in class, yet, if you give them the opportunity to build a Quizlet outside of class, they’re all over it, which is really retrieval practice. And so it’s a matter of trying to have that conversation. And it’s one of those things about human behavior, and you’re an economist, you know this. It’s so frustrating. People don’t do things that are good for them sometimes, and it’s frustrating, but you just keep working at it in a variety of ways. I know we’ve had success by hosting student faculty conversations, where the students and the faculty will talk to each other about what is going on in class, and it helps the students understand why the faculty members do things. It helps the faculty understand why students do things. But at the end of the day, it’s incredibly challenging, and I think it’s one of the things that makes this work. We’re going to be around for a while, because we really need to continue to work. And I will say one more thing, it also doesn’t help that faculty are incredibly resistant. And they’re resistant because you try something, the students don’t like it, you back off of it because the students don’t like it, even though it might take two or three iterations for them to like it. So there’s a lot of challenges there.

John: And we’re also coming up against students who’ve had 12 to 13 years of practice before entering college without using those techniques with high stakes exams and so forth. So there’s a lot to overcome there.

Chris: Yeah, and the high-stakes exam thing, we have a large group of physical therapists, physicians assistants, and OTs (occupational therapists), and they all have this national exam they have to sit for. And so they just want the stuff. They just want to know stuff. And I always try to tell them, you could know a lot, but you have a patient in front of you, and this is the example I give. I say, “If a patient comes to you and says, ‘My arm hurts when I bend it up’ and you say, ‘Okay, here’s what you’re gonna do. You’re gonna do this.’ And then the patient comes back says, ‘I did that and it doesn’t help.’ You can’t look at the patient and say, ‘Well, boy, that’s terrible.’” You have to have another approach. You have to have something else to try. And I try to help them sort of work through that. Again, we’re fortunate here. I do go in and talk to almost all the grad students when they go from undergrad to grad, to talk about the application of knowledge. I also talk to all the undergraduates when they show up on campus to talk about some of these things too, with the idea that if we repeatedly talk to students about it and try to help show them, we may get more buy-in. There’s no panacea to this.

Rebecca: I think what you’ve both highlighted is that there’s resistance on both sides. There’s resistance from students and resistance from faculty, because change requires effort, and establishing new habits takes time and effort. But I like the idea of the dialoguing so that you’re working on change from both directions.

Chris: Yeah, and it’s interesting, I always ask faculty, especially like, if they give an assignment and say, “Why did you give that assignment?” And I’ll sometimes hear, “Well, it was time.” Well, that’s not a good enough answer. If you can’t relate that back to what you’re trying to get students to learn, then the students don’t know why they’re doing it, and they’re just doing it, which is the big issue with AI right now, is faculty are like, “Why are students AI-ing my assignments?” They see zero value. You work towards things that you value. You create value in education, students are more likely to work. It’s not guaranteed, but it’s more likely they’re going to do the work.

John: Earlier you suggested that one strategy you encourage is transparency, so that students understand why you’re asking them to do these things. Might that help in getting students to break the habits they’re developing of using AI for so many assignments now.

Chris: Yeah, I feel pretty strongly about this. I’ve done quite a bit of work in the AI field, working with AAC&U as well as with the American Psychological Association. And I think that the transparency with AI, we just have to do it at this point. What we need to do is lean into teaching AI literacy, teaching students about what AI can do, the limits of AI, showing them some of the limits of AI, and allowing them to see this as a tool, not as learning, but as a tool to help them learn more effectively. John, you were teaching at the time. I remember in 1995 we got essentially a graphic browser called Mosaic, and I could actually see the internet versus just typing things into gopher. And I thought my life has just gotten better. It’s just gotten better, and I was able to do things I couldn’t do before. Well, AI has done the same thing. I can’t tell you how many times n ow, when I’m generating a data set for statistics, I give it the parameters, and it just generates the data for me and it comes out the way I want it to. I used to calculate those numbers, and it was terribly difficult to do. Now it just does it, and that takes the time from that for me to give students feedback or do other things. Well, the students, just like with the calculator, can use it in a way, and I’m not saying it’s a calculator. It’s much different than that. I’m not saying that, but it’s a tool that we can use, if we can figure out how to leverage it. But transparency is key. We are talking right now about creating a guidance document through the center that’s going to be shared with faculty and students about how the center and how academic affairs views AI and the value of it, as well as some of the pitfalls and concerns and some of the caveats that need to be around it. Because I think, even if you hate it, and we have faculty who truly hate it, they truly hate it, it’s not going anywhere, and the students have access to it on their phones. It’s there, so we have to figure out what to do with it.

Rebecca: One of the things I want to underscore about what you just said is that you’re going to develop a guide for faculty and students and that you’re going to share it to both parties. And I think that’s a practice that institutions could just do more generally, so that both sets of individuals are getting the same information, because especially around AI like literacy is necessary for all members of the campus community, not just students.

Chris: Yeah, yeah. We have a task force right now we’re running with my associate director, which involves five staff members and five faculty, and we’re going to cycle in some students so that we can sort of figure out what we need to be talking to people about. Our staff, many of them are afraid to use it because they’re afraid that it’s going to work them out of a job. But we say “No, this is gonna make your lives better if you use this.” So we’re trying to work through some of that. But the transparency is really key. I would argue that for everything, Rebecca. I’m a firm believer in being incredibly transparent with my students about all sorts of things that I think help move the learning process along, and that’s been really valuable for me as an educator and with my colleagues as a faculty member, as an educational developer.

John: And one of the things we’ve been advocating is that faculty have these conversations at the start of the semester with their students about what uses of AI are appropriate to see if they can get some degree of buy in. Do you think that might be helpful in convincing students that they should use it in ways that support their learning, rather than ways that serve as shortcuts to their learning?

Chris: Yeah, my associate director, Matt Citron, developed a system for one of our majors because the faculty really wanted it. And it’s not a new system, people are using it, but they just have a system built into Brightspace where an assignment will have either red or green or yellow light saying: red, don’t use AI; yellow, use it with caution; green, do what you want with it. But even before they see that, there’s a whole conversation about what “use it with caution” or “go crazy with it” means, how you’re going to involve that, and what you’re going to do with it, and how you’re going to report what you got out of it to the instructor, so they know exactly where this is coming from and how it looks. Again, It’s a transparency. But, you know, I’ll say this, I am continuously amazed when faculty are trying to sneak things in and trick students, because learning is not about being tricked, man, it’s about being able to do things more effectively. And I even do this, and faculty don’t love this. But when I give multiple choice… I’d really moved away from it…but a few years back, when I gave multiple choice, I told students, look, if you don’t like my answers, give me your own. If I like it, I’ll give you points for it. Or if you think my answers aren’t good enough, and you could write it better, fine. Write it underneath the multiple choice, I’ll read it. I mean, students don’t often do it, but every once in a while they do. It’s not like I have the magic knowledge, that’s the thing. I just know more about this than you do. I’ve learned more, I can think about it differently than you can, but let’s talk about it. And I have conversations with students after exams. I have conversations with students about their papers. I’ve taken to shooting video feedback to all my students on assignments and their responsibility is to give me video feedback, telling me how they understood my feedback and what they intended to do with it. So it becomes this virtual conversation. I talked to my faculty about that. Teaching and learning is an innately humane relational process, and the more we can continue to build that in, the better off our students and our faculty are going to be. But Rebecca, it’s work. It’s work.

Rebecca: It is work.

Chris: It is. I have two classes. I have 67 students this upcoming semester, as well as running a teaching center. It’s a lot of work. I grew up as a first kid to go to college, first one to graduate. I feel pretty lucky to do this job. I’ve worked other jobs, man, I built fences. This is way better than building fences. This is way better than building fences.

Rebecca: Sounds like we’re trying to take them down, actually.

Chris: Yeah.

Rebecca: One of the things that we’re seeing with budget crises across higher ed is sometimes increases in class sizes, and you just talked about how deeply humane and relational [LAUGHTER] teaching and learning is. Do you have some strategies for student engagement and some of that relation building in those larger classes.

Chris: Yeah, it’s a great question, and it’s interesting because at Springfield, and I’m not sure about the size of Oswego, but for us, a big class is like 60 students. Faculty freak out when they hit 60. And the fact is, I think that relational piece is going to look different depending on the size of your class and how you’re able to navigate that. I think part of it is understanding a variety of ways to give feedback. So I tell faculty, in particular with large classes that I would encourage them to do small assignments and give feedback to percentages of students every class to ensure that every student gets written feedback across time, but they don’t have to get all the same written feedback every time. You would never not read a student’s work, but you would give more substantive written feedback to a percentage of your students, but work in ways to develop those kinds of touch points. It’s not always easy. Sometimes it involves using the LMS, sometimes it involves getting there early, but being aware that your students, if they identify you as a human, they’re much more likely to be kind to you. So it’s really a matter of trying to figure out a way to allow yourself to be perceived as a human in students’ eyes. That’s idiosyncratic to the instructor. My students know a lot about my life. Part of it was I was a single dad for six years. I taught with my kid in a Snuggie. There was no way for my students to not be aware of what I was doing. I had a child strapped to my chest. She fell asleep. I told my students, come closer. She’s sleeping. I can’t wake her up. So I’ve always been pretty transparent with my students about lots of things. It makes me human, and I think part of it is allowing the students to see that you’re human too at whatever level you’re comfortable, whatever level people are comfortable, some people are more private. I get that, but I think you have to allow that humanity to kind of show through. I’ll say one more thing about that. If nothing else, you went and got a degree in this discipline. Show your passion for your discipline. Let your passion for your discipline come through however it does for you. I can’t talk about some psych topics without being jazzed. I’m a psycholinguist by training. I can’t talk about language without being super excited about language. It’s fun. I was just in Europe. I was looking at things written 1000 years ago. My camera is filled with pictures of stone tablets. My wife’s like, “What is this?” I said, trust me, “That’s linear B, man, that’s good stuff, right there.”

John: One of the issues that we do hear a lot of concerns from, especially since the pandemic, although it seems to be a little bit less last year than in the previous years, is students seem a little bit less engaged with some of their classwork, that students are more likely just to blow off some assignments and not complete them. And also, many faculty complain about having lower attendance in their classes. You mentioned using polling as a strategy. Are there other strategies that faculty can use to help increase that connection, or to get students a bit more engaged so they don’t disappear on us.

Chris: Yeah. I mean, the mental health crisis has really had its fingers in lots of this, and students’ anxiety has led them to be less willing to chat in class, less willing to engage in class, and with the devices they get to class, and they used to talk to each other, they sort of look at their phones, and so that’s an issue. I am pretty transparent about what I expect my students to do, and I talk to faculty about stating your expectations clearly and being open to knowing that students are going to sometimes struggle with this. I’ve got a couple of simple little tips. So for example, I give this tip to faculty. Have students answer questions and just write them down on a note card without their name, and then they take their note card and they stand up and they walk around the room, they hand it to someone, they take theirs, and then they take that note that note card and hand it to someone else, and then they take that, and then they just pass them around. And then I can call on any student, and all they’re doing is reading the note card, and so that helps them sort of see that it’s okay. I’ll tell you another secret… my students, whenever they find out about this… they get really mad, but I do this. I collect note cards at the beginning of the semester with all my students’ names on it, and I tell them, sometimes I’m going to randomly call on them to see if they know an answer. And they’re always softball, simple, very easy questions. So I’ll shuffle the cards and call someone. I will almost always call a name that is not on the card. I’ll call the name of the student who I know hasn’t said much, but knows what’s going on. So they have a chance to do something in class, be successful, and shine, and then they’re more likely to talk. And students don’t know what name I draw off the card. It doesn’t matter to me. It could be blank cards for all they know, but it’s a way to get the students… and again, as a psychologist, I look for ways to sort of shape that behavior up. Also, I tell faculty: “Get the class early, greet students when they come in, just talk to them.” I don’t know if this happens at Oswego, but this is a Springfield College thing. Our students thank us. At the end of every class, I get fist bumps as my students are walking by after every class. And I’m not unique. Lots of our students do that, and part of it is a culture that has built up over campus, over time, this is before my time. This is sort of who our students are. And so you work with that culture, okay, if the students are gonna do that, then I know they’re polite. Okay, good. Then I can work with that. Your students might be different. You know what they are. You try to work with that. It’s like I say to people all the time when faculty say, “Well, what can I do to be a better teacher?” I can’t pull a book off the shelf and say, “Here, do this.” I have to know who you are, who your students are, and all of that. And so it’s why the field is so much fun. I get to know lots of faculty, and I get to work with lots of faculty.

Rebecca: One of the things that we’ve been talking a little bit about, are some budget cuts in higher ed and teaching centers being on the chopping block? What are some of the things that you’re concerned about in this space?

Chris: So you don’t want to open that door too much. I really appreciate this in part, because I think there is a sense with teaching centers right now that if we let them go, we’re going to be okay. And I think faculty need the resource, even if it’s a perceived resource. My wife and I talk about it all the time. Perceived help is often as useful as the help you get, and so us existing is really important, and I worry that teaching centers are going to start to shrink, and it’s going to make faculty struggle in their teaching. So I hope that we continue to talk about these things and I appreciate having the opportunity to talk about it.

Rebecca: So we always wrap up by asking, what’s next?

Chris: So what’s next for me? I have an idea for another writing project. I want to look at intentional teaching. I want to talk about barriers that make it difficult for faculty to be as effective as they can be in the classroom, starting with things like lack of pedagogical understanding and going through variables such as student variables, classroom size, physical space, etc. Because I think we need to take a harder look at what we are doing, knowing the stakes are higher than ever at being able to be effective at what we do. We need to be incredibly intentional about creating really strong learning environments for our students to succeed. And I think our livelihood depends on it. Listen, I teach at a small liberal arts college in the Northeast, and have either you ever been to Springfield, Massachusetts?

Rebecca: Yeah.

Chris: So from my office, if there were no trees, I could see two other colleges less than a mile away from me. So there’s lots of schools and fewer students. We just need to be really good at what we do. So I’m thinking about that. The other thing is, I am excited to be building out some additional resources for our faculty that I think are going to support them with short burst sort of faculty development that we’re building with our media team, which has been really fun. So we’ve got lots of stuff coming. We’re excited to get the year started and to do these things and to share them out. I’ve lots of ideas.

John: Well, thank you for talking to us. We very much appreciate the conversation and look forward to future conversations.

Chris: Yeah, thank you both so much, and I appreciate having this conversation. I know that you folks talk to lots of people. I went through, and I just have you know I run every day. You’ve gotten me through many runs, because your times is just about right, so I really appreciate that, and you’ve talked to some people that I have deep affection for. So I really appreciate being part of this, and thank you for all the work that you’re doing for teaching and learning. I think it’s super important for people to hear from these other people, so thanks.

Rebecca: Yeah, thank you. We’re looking forward to sharing out this episode with our audience, and we’re grateful for all the people that we’ve gotten to have on the show, you included.

John: And that’s why we keep doing it. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

406. AI: A Student Perspective

In past podcasts, we’ve often explored faculty concerns about student use of generative AI. In this episode, Kaija Hoyt joins us to discuss the evolution of her AI use. Kaija graduated from SUNY Oswego in May 2025 with a major in Human Resource Management and a minor in Mathematics. After graduation, Kaija provided a presentation on this topic to Oswego faculty and staff during an AI symposium.

Show Notes

Transcript

John: In past podcasts, we’ve often explored faculty concerns about student use of generative AI. In this episode, a student describes the evolution of her AI use.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Kaija Hoyt. Kaija graduated from SUNY Oswego in May 2025 with a major in Human Resource Management and a minor in Mathematics. After graduation, Kaija provided a presentation during an AI symposium on our campus in which she discussed the evolution of her use of AI tools. Welcome back, Kaija.

Kaija: Thank you for having me again.

John: Today’s teas are:… Kaija, are you drinking any tea?

Kaija: I am not today because the room that I’m in right now is too small for me to store my tea. So it’s really sad because for this entire month I’ve been tealess. Yeah, it’s very unfortunate.

Rebecca: That is very, very horrible.

Kaija: Yeah.

Rebecca: I have English tea time today, John

John: And I’ve been without tea, but I picked up some tea at a Whole Foods across the street from Duke where I’m recording from, and I’ve got some ginger peach black tea, again, restoring one of my favorites.

Rebecca: Oh, look how it doesn’t matter where you are, [LAUGHTER] you can still drink the same tea. [LAUGHTER]

John: It’s so convenient. So we enjoyed that conversation we had with you in last week’s podcast on microaggressions, and we thought we’d bring you back for another discussion. And once again, this is based on a presentation you gave during a workshop for faculty on our campus, except this time you were talking about how your use of AI tools evolved. ChatGPT arrived on the scene in late November of your sophomore year and was rapidly adopted by many students. What was your initial reaction to ChatGPT and AI tools in general?

Kaija: I hated it. I was so mad that they had kind of become a thing. Because for me, who’s always been like a strong writer and a strong student, it kind of devalued that in a way, because a lot of students were using that, and their writing was about at the same level as mine, or stronger. And some professors were starting to maybe not realize that it was AI doing that, and expect me to write more of that standard. And I also didn’t like the idea of using it. I’ve always really wanted my work to be fully my own, and so when it was first coming about and other people were using that, I was just very strongly against it, and still to this day, to some extent, like, I’m not a huge fan of AI and the use of AI in a lot of environments, but for me, I wish it hadn’t come about when it first did.

Rebecca: I hear, though, that your position or reaction or response to generative AI has shifted a bit over time. Can you talk a little bit about what changed your opinion?

Kaija: Yeah, so I was very firm in the AI use all the way until, I would say, honestly, like the end of junior year, so pretty much like a solid year and a half of being anti AI. I had a professor, Dr. Friedman, who loves AI. He’s kind of obsessed with it in a good way. So in our classes with him, he really encourages the use of AI and gives us some ideas on how to use it. He was really passionate about “You could use it to help you with this part of an assignment.” Or, like, for example, with writing assignments, he would say, “you could use it to draft an outline for you and then build off the outline.” Or when we were doing our compensation project, he said, “help it come up with competencies for you, help it come up with the weights, it can help you with all of that.” So he pushed it pretty strongly. And that compensation class I took with him junior year in the beginning. And even though he was saying that and pushing that, and my group members were like, “Yeah, let’s use it. Let’s use it.” I was like, “Listen, if you want to use it, you can, but I’m not going to” like,” if that’s what you choose to do, fine, whatever. I don’t care, but I’m going to still steer away from it.” And then he also became my honor thesis advisor at about that same time, and that was when he started pushing AI at me even harder. And I kind of was like, “okay, whatever like, just to get you off my back, I’ll try it this one time.” And then I was like, “oh, you know what? That actually did help a lot.” So I started realizing, like, “okay, it’s really useful with things that are, like, the small work that you have to put into a project that doesn’t really have to come from you, like things that AI can do for you that will make it easier to do the project and will also leave more time for you to, like, brainstorm and be creative.” So it takes that monotonous part out of it, which ended up being really helpful, and it also is really good with writer’s block, I feel like, so that was kind of the thing that switched me, for the most part, to being okay with using AI.

John: Could you tell us a little bit about how you used it in your writing?

Kaija: Yeah, so in my honors thesis, I used it when I was having a hard time, like merging two ideas together with like getting it to flow, just having it give me ideas. I very rarely used what it gave me, because basically what I did was like, I would have it help me, and then that would spark an idea for me. Maybe it would give me a transition word, and then I’d be like, I should use furthermore here, that would work really well, or something like that. So it was more of like a jumping off point for brainstorming, but it also helped me a bit if I was struggling to paraphrase something, like getting it to what is another word that fits here? Because sometimes I know there’s a word, and I think I know what the word is, but I just can’t remember it in that moment. So it did help me a lot with that. I also used it with restructuring my thesis. Because when I got to a certain point, I don’t even remember what point this was, but there was a point when I was like, wait a minute, my outline doesn’t make any sense for what this has turned into. So I needed to, like, move things around. And that was really stressful, because at that point I was definitely over halfway done. So there was, like, a lot of paragraphs, there was a lot of information, and I kind of knew how I wanted to move it around, but I also wanted that feedback of, does this make sense? Are these transitions better? worse? the same? And I was trying not to put too much work on my advisors, because I knew they were both very busy at the time and I also really prefer to get their feedback after I know what I want, so that I don’t feel like I’m just doing what they tell me to do. I like to know this is how I want it to be, and then hear their feedback, and then go from there. So I used it to help me switch up that outline a little bit, and I didn’t take exactly what it told me, like this paragraph should go here. For some of it, I was like, “Yeah, that makes sense.” And for others, I was like, “Actually, I feel like this is a better transition,” but it helps me kind of be okay with making that change. So basically, it’s helped me in a lot of ways with writing, but it’s mostly just that starting point and getting me comfortable with what I’m doing, and then later on, it basically looks like AI didn’t help me at all because of how many changes I ended up making.

Rebecca: It sounds to me like some of the ways that you’ve been using AI are to find out what you don’t want, like to bring clarity. [LAUGHTER]

Kaija: Yep, a lot of the time, a lot of it is because there are things that AI wants or thinks that I hate, or just, I’m like, “That is not my writing style. I’m not going to do that.” So I do definitely use it in that way. Or like, sometimes if I give it a paragraph to proofread, and then I see how it wants to make that paragraph, and I’m like, that’s not what I’m trying to say. And it helps me be like, this is what I need to change to make it work. So I definitely used it in that way.

Rebecca: One other thing you mentioned was the writer’s block. And I don’t know any faculty or students that get writer’s block ever. So I’m asking for a friend. How were you using it to overcome writer’s block? You hinted at a little bit, but could you give some more specifics?

Kaija: Yeah. So one of the big things is if, for example, I at one point was going to talk a bit about some psychological theories that relate to microaggressions, and Dr. Friedman had suggested that I cover that at some point in the thesis. He’d given me some brief explanations of different theories he thought would apply, like he gave me a lot to build off of, but I didn’t feel great about writing it, because I’m not really a psychology person, and so I asked AI for some help. One, it helped me find sources, but it also helped me with figuring out how to work that into my thesis and starting the writing. So I told it what Dr. Friedman wanted, and then it kind of gave me, like, “you could do this.” And so then I was able to figure out, okay, from that, what do I want to do? It’s also really good with brainstorming outlines and ideas for projects, because sometimes you just get stuck and you can’t think outside the box at all. Usually, in my experience, like I’ll ask it to brainstorm something, and I won’t really love the answers it gives me or the ideas, but it’ll spark something else. It’s like, if you talk to someone else about it and they gave you a bunch of ideas, except you don’t have to find someone who wants to talk to you about it. You can just ask it, and it’ll word vomit some things at you, and then you can be like, “Okay, I know what I want to do now.”

Rebecca: An introvert’s dream.

John: And also, if you’re doing some work at two or three in the morning, it might be easier to have that conversation with ChatGPT than to wake someone up and ask them to give you feedback. One of the other things you mentioned was that the type of feedback you were getting on your writing was different than the type of feedback that people get from Grammarly and other tools, because with Grammarly, which students have been using for quite a while now, it’ll tell you what to change, but it won’t explain why it should be changed. I think you were getting a little bit more information from ChatGPT in terms of the types of recommendations it would use, and that also ended up affecting the writing that you did later during your thesis. Could you talk a little bit about this?

Kaija: Yeah, so one of the main differences for me personally was occasionally I was looking for grammar checks, but that wasn’t my main focus, because grammar and spelling, like, I’m good on that, and mine is more like, “How can I just make it even better?” So I feel like Grammarly can do that, but yeah, like you said, It never tells you why it’s doing it. I also find Grammarly a little bit annoying, because it just yells at you about everything, and that’s not what I want. Like there’s specific things that I’m looking to fix, and you can’t tell it that, whereas with ChatGPT, you can be like, “Hey, check this paragraph to see if I used also 50 times instead of mixing it with different words.” Or one of the things that I did with it was in my thesis on microaggressions and talking about the trainings that I was proposing, I wanted it to be trainings (plural) consistently, because that’s what made sense for what I was writing. But a lot of the time Google and Grammarly, whatever, really wants it to be singular, which makes sense, but in the context that I was using it, it needed to be plural, even though it thought it was wrong. So I’ve used it in one way with that to be like, “Hey, if you see it, say training singular anywhere like highlight that or call that out so I can fix it.” Another thing, this isn’t a real example for me, but I know some people will start a sentence “although this” and then they never finish, like the last part, so you can tell it look for those situations, and it can highlight them for you. Another thing that I did was there’s like, active/passive voice, I tend to forget to check that, and also really struggle with like when I reread something to check for that, sometimes I just get distracted by something else and I forget what I’m looking for. So I tell it, like, call out those moments, and then I could see it. And then over time, like, the more it was like, “You need to fix this, you need to fix this, the less I made that mistake. So it was really more of a learning thing for me with figuring out, like, “Okay, what are the problems in my writing and what is the way to do it better?” And then, like, I would say by the end of the thesis, like, I wasn’t really using it that much, because most of the problems that I was having I figured out what I was doing wrong and how I should do it better.

Rebecca: Based on some of your examples, I’m thinking other examples that fall into the same category, or maybe, like, checks for first and third person, other things like that that we all make mistakes on when you’re initially writing a draft.

Kaija: Yeah, for sure. I think if it’s one of those things where you maybe are someone who writes… this is really good for students, because I know there’s a lot of students who like to write in the first person… I think it would be awesome if professors who notice, like, “Hey, you’re someone who has a really hard time getting rid of those ‘I’s’ like you love to write. ‘I think that,’ or ‘I believe,’ showing them how to use ChatGPT to help you reword it could be really helpful,” because it’s not that they want to submit it with the ‘I’ in there. It’s just that they don’t realize there’s a better way to do it.” And the more that you have something telling you this is the better way, like, the more likely you are to catch on and to learn it. And a lot of times, professors don’t really take the time to do that with students. They just say, like, “don’t use I,” but students don’t understand how to not do it. So they really need that assistance with the teaching on that part.

Rebecca: One thing, Kaija, that you mentioned earlier was telling ChatGPT, like, “No, I don’t agree with you. [LAUGHTER] I want you to focus on this in a way that you felt like tools like Grammarly couldn’t do that.” And I know that that’s something I enjoy doing too, like telling the machine, “No, do it this way instead. Try this instead.” Can you talk a little bit more about some of the iterations you were able to accomplish using ChatGPT?

Kaija: Yeah. So with different iterations, I would definitely say there were, especially with the structure piece. So trying to restructure it, I think I changed the structure a couple times. And one of the things that I started with, figured out the main ideas of my paragraphs, gave it that, and said, “What order makes the most sense?” And then it did whatever it did. And I was like, “Actually, you don’t realize what the paragraph really includes. So you don’t understand. I don’t agree.” So then I gave it more information. So instead of just like this paragraph defines microaggressions, I really summarized it a little bit deeper. So that was one of the things where I kind of went through a couple different attempts with it and got it to understand what I wanted. Another thing that happened, this is, like, less related to it proofing my writing, and more related to, like, the output it gave me. So one time, I gave it a paper because I had proofread it twice, and I was tired of staring at it, and I knew that there were mistakes, and I just couldn’t find them, so I gave it the paper, and I said, “Don’t edit it,” because I gave it the paper the first time and said, proofread this, whatever prompt I used for it, something along the lines of like, proofread this. And then it gave me the essay rewritten how it would write it. And I was like, “No, don’t do that. Don’t rewrite it. Just tell me what is wrong with it. Tell me what needs to be fixed.” And then I don’t even remember what that output was, but I was like, that’s not really what I want. So then I said, “Okay, without changing anything, just tell me what the strengths of this are, what the weaknesses are, and what things are just clunky.” And then it gave me a very broken down, “These are the parts that are strong. And then a couple of quotes of examples. And then these are the areas that could be improved.” And then if, with the improvements, I needed help with, like, how do I do that? Then I can ask it that, like, “Well, how do I make this more concise?” And it would give me examples.

John: When you gave that presentation to faculty, one of the other things you talked about was using this to help prepare you in your math classes. Can you talk a little bit about how you use ChatGPT to support your learning in math.

Kaija: Yeah. So it was in my introduction to algebra class, which started off a little rough for me. I was having a really hard time paying attention in class for the first, I want to say, like a month, and that first test came around, and it’s not go well at all, which is very strange for me, because I usually do really well in math. So that was, like, a big blow for my ego. And I was like, “Yikes. Okay, this is a problem.” So I started paying more attention in class, and that helped. And I was doing the homeworks and starting to understand them, but the first stuff that we had learned, like, kind of I knew, like, “I just really don’t get this, and I don’t know if I’m going to if I just keep going in the class.” So we had a textbook, but it was one of those textbooks where it’s like, the professor uses the textbook, but not really, so it doesn’t directly match up. It’s not like Calc I and II, where you are literally doing your homework out of the textbook, and the textbook follows the same path that the class does, like this class in the textbook were different enough that it made it kind of challenging to find practice problems in the textbook. And the other thing is that practice problems in the textbook are not always what you’re looking for, and there’s no way to like, filter them out, like, you just have to stare at it and find the one you want. You can’t be like, “Well, I’m looking for how to do mod eight.” You have to find that in the massive answers that there are. And the professor did give homework, but I think we had like one a week, and there was only like six to eight questions each. And I’m very used to math classes where you have like 20 questions for a night at minimum, although those are usually optional, but I do them because I love math. So it was kind of odd for me. And I was like, “Why are we not doing more?” And I also, in addition to the fact that I was struggling, like I also wanted to do more with it, I really enjoyed it. So I went to ChatGPT, and I was like, “I feel like this could help me.” So I kind of told it what the class was, told it what I was learning, and then said, “Do you understand what this topic is?” And it was like, “Yes, I can give you practice questions.” So then I would tell it, “Okay, can you give me a question on Z/8?” and then it would give me something, and then I could do it. This is the one that I had a problem with. It was that I told it like, “I just want you to say whether it’s right or wrong. That’s it. Give me the question, I’ll tell you the answer, and you say ‘yes’ or ‘no,’ and then if you say ‘no,’ then I’ll be like, ‘Okay, walk me through it.’” But it just kept giving me the whole thing anyway, even though I told it, like five times, so I just gave up. So that was kind of how I used it for mostly getting practice problems. And there were also sometimes, if there was a concept that I was struggling with, I would have it kind of walk me through it. And honestly, it’s pretty impressive how well it did with this class, because this is like a higher level math class. I did not expect it to understand it, but it really was good at walking me through like how to solve it. And then from that, I was able to create my own understanding of what to do with some of these problems.

Rebecca: Kaija, what class was this in?

Kaija: Introduction to algebra, which sounds easy, but it’s Math 330 and it’s not like solving simple algebra equations. It’s like abstract algebra concepts.

Rebecca: I kept thinking that doesn’t sound like intro to algebra. [LAUGHTER]

John: When you were getting these problems, one of the things you noted when you gave a presentation describing this was that you learned how to modify your prompts a bit to get more useful feedback. Could you talk a little bit about what sort of modifications you made to get more useful results from ChatGPT?

Kaija: I, for one, kind of learned how to get better at telling it what kind of problems I wanted, because I started off with being like, “Can you give me an example of something like asking whether or not it’s a homomorphism?” I gave it no context, so it didn’t know what types of groups we learned about and what types we didn’t. So one time it gave me this group that I honestly was like, “I don’t even know what that is. Never heard of it.” So then I was like, “Okay, I’m gonna start telling it like, these are the groups we know. Give me an example from there.” So that was for the practice poems it gave me, and then when it gave me the feedback on whether it was right or wrong. If it was a time when I was like, “Okay, I actually need this feedback because I didn’t do it right or I could just then I would read through what it gave me,” and then I would give it follow up questions of, like, “Okay, here’s where I’m stuck with your explanation, L I don’t know what this means.” Or “Can you clarify this more,” which sometimes it would just regurgitate the same thing. So then I had to figure out a different way that I can phrase this to get the part that I’m confused about out of it.” So asking, like, “Okay, I don’t understand where this Z came from. Why is there a z here?” instead of just, like, “What does this mean?” because if you just say, “What does this mean?” it’s just going to throw the same thing at you, and it’s not going to help.

Rebecca: So I’m kind of wondering, given the experiences that you’ve had in using ChatGPT in some really different contexts, how would you like to see faculty talk about AI in their classes, or what parameters or ideas do you think would be helpful for other students to understand how it might actually be a productive way of learning in a class?

Kaija: I have very strong feelings about that, so I’m glad you asked. For me, I think that it was really helpful having a professor who said, “This is how you should use it. This is the ways that it can help you in this class specifically, and I’m not going to penalize you for using it in those ways.” That really is what helped me, like, be open to using AI. And I know, not for me because I was anti-AI, but for other students who can’t seem to get away from using it, when a professor is just like, blanket no AI policy. You’re not allowed to use it like they just decide to entirely ignore the AI policy because they don’t care. They don’t know how to learn without it. And they’re so dependent on it that if you don’t provide them a way to use it productively and without plagiarizing, then they just can’t move away from it, and obviously some students are always going to use it to write their papers for them. I’m not saying that this is going to solve that problem totally, but I think that there are a lot of students out there who, if you say to them, “Hey, I’m okay with you using it to make the outline,” or “I’m okay with you using it to help you proofread,” or something like that, and showing them like these are the ways that you can use it without it becoming an academic integrity problem, then they are going to be more likely to actually put that work in alone, because a lot of it is like, they just don’t know where to start. They don’t know what to do. They’ve got writer’s block, and there’s a deadline, and so they’re just like, I’m just going to have it write it for me. But I don’t know for sure, because I wasn’t super close with the people that were in my senior seminar HR class, but Dr. Friedman said in that class for our paper that we had like “You can ask ChatGPT for an outline, and you can build off of that. You can ask it for assistance in writing this paper. You can ask it for sources.” And I’m pretty sure that a lot of students in the class really took that and did that and used AI responsibly and still submitted mostly their own work, whereas with other professors, I noticed them just kind of having it write it for them, or things like that. And I know that, like a lot of the time in the tutoring center, if I have a student come in for writing who is clearly someone who’s really dependent on AI, I try to give them that overview of like, if you use it in this way, A) you’re safe and B) this will help you in the future as well. And it seems to work. So that’s a small example. So I don’t really know, but it does seem to be really helpful to be clear about like, these are good ways to use it, these are bad ways to use it. Instead of just saying all of it’s bad, you can’t use it at all.

John: So during your time at Oswego, I think you had five semesters during which AI has been available to all students. How often, though, did professors talk to the students about whether AI could be used, when it should be used, and why it might be inappropriate to use it in other areas? Was that a conversation that happened very often?

Kaija: So I would say I had the one professor who talked about it lot. I had probably one or two professors who really consistently called out, like, please do not use AI in my class. If I catch you using it, I will fail you. And then I would say it’s hard with math, because math professors don’t really talk about AI very much, which is mostly because it can’t really do that much with math, because it’s really easy to catch it, because it hallucinates a lot when it’s math, and honestly, it will take more of your time to use it in a math class to, like, do your homework for you if you’re trying to get a good grade. So they don’t talk about it a ton, but they do say they have their policy of don’t cheat, and that includes AI use, things like that. And then in my other classes, I think most of them had it in the syllabus, like you are not allowed to use AI in this class, that that would lead to a failing grade and being reported to the dean. But if they even said anything about it, they only addressed it, like on the day that they went over the syllabus, and then they would just not talk about it again. So I think a lot of them, like, just put the policy on there because they feel more comfortable with that and they don’t really want to have to address it. That’s kind of the vibe that I’ve gotten. So I would say it’s like, mostly just this thing that, like, you know you’re not supposed to do, but you also don’t really know what’s going on with it, at least in the School of Business.

Rebecca: You’ve talked a bit about math and you were just pointing out the hallucination issue with math problems, and because you were doing practice problems and things with it, how often did you find the hallucinations bubbling up in your use?

Kaija: So surprisingly, it didn’t happen as much because it was more conceptual things than computational, but there were a couple things that were computational that I used it for, and that’s when I found it hallucinating. One of them was I was practicing the product of cycles, and I’m pretty confident in this, but I just wanted to practice it because the final was coming up, and you never know. So I was going through, I think it was the second or third example with it, and I did the multiplication the way that I knew was correct, and I got the answer that I knew was correct, and I told it, this is what I got. And it said, that’s wrong. So then I looked through the output that it gave me of it’s step-by-step, and it’s step-by-step didn’t make any sense. I don’t know what it was doing. I don’t know what happened, because it was working fine before that. I do suspect that it’s possible that it had been right when I’d run out of the like higher-level questions, and it switched me to the lower level, and that’s why it happened, but I don’t know that for sure. And so I got to the end of it, and I was like, “No, I’m definitely right. I don’t know what you’re doing, but it’s wrong.” So I basically told it that. I said, “Your answer is wrong, my answer is right. I don’t know what you did, but that’s what it is.” And so then it responded to that, and was like, “Yeah, you’re right. You got the right answer.” But then, like, it’s calculation, it had the answer, it had before, and then it had my answer, and then it said, “Yeah, so you’re right.” It was like, 123, so 345, and I was like, [LAUGHTER] that doesn’t make any sense. So I did stop checking my answers with it at that point with those because I was like, you don’t know what you’re doing, but that is the only time that happened with the higher-level algebra. I have seen it do funny things with lower-level math classes, like regular algebra or calculus, because I’ve used it for fun with that, just to, like experiment and see if I can get it to hallucinate, because I think it’s funny. [LAUGHTER] And it definitely has happened a couple times with that, which is a little bit interesting, that it can just kind of end up in places. And I’ve also found that if you tell it like it’s wrong, it doesn’t even care if it’s actually wrong, it just decides that it is. And so that is probably part of the problem, is people being convinced that they’re right, and so they’re training it that this other thing is true and it’s not. So it’s pretty confused on things like that. And I think that’s the other thing that I think is why it worked better for introduction to algebra, because I don’t think there’s a lot of people going to ChatGPT for that. So I think it really just had that database of whatever it found when it was first started, and less of that user input that can make it more incorrect.

John: One of the things you described just a few minutes ago was using ChatGPT to get more practice problems, especially in classes where there’s not enough practice problems to make you feel comfortable. Is this something that perhaps students should do more regularly or more often? Use it to help develop skills when they’re not getting practice problems that are relevant to their own areas. When instructors give assignments with a group of problems they’re generic problems that are designed to address the needs of all students, but every student comes in with very different backgrounds in terms of what they understand and what they don’t. Were you able to use this to help develop those areas where you weren’t as comfortable with and not having to spend that much time on those areas that you were more comfortable with?

Kaija: Yeah, so it’s totally adaptable to what I needed, and with what I said about I fell behind because I wasn’t with it at the beginning. So it was something where it’s like, we’re not going to have more practice problems. We left this way back there. It can still show up on the final but we’re not talking about it anymore, whereas he’s still giving us practice problems for whatever we’re working on now, and so I would do that homework, but that was something that I was like getting. So instead of spending extra time on that, I would just go do the areas that I needed, and I didn’t, in this context, but you can have it figure out on its own what areas it is that you’re struggling with, and then it can just throw those questions at you, because you don’t have to tell it: “Give me a question on homomorphism.” So you can say, “Give me a question that would appear on an introduction to algebra test.” And for the most part, it’s good, especially the more information you feed it. And then if you work through, and you work through, it starts to catch on, like, this is where you’re struggling, and this is where you’re set. So it can really modify to you in the same way that your professor would at office hours, without you needing to go to office hours and figure out that schedule with them, because our times usually don’t match.

John: When you presented this, a math instructor asked, “Why didn’t you just go and talk to the instructor with questions?” And you gave a really good response that I think would resonate with a lot of people. How did you respond to that question?

Kaija: So I basically said that for me as a student, I’m a very independent learner, and I also don’t like to ask for help, and so going to office hours is really difficult for me from that side of it, and I personally am someone who would rather spend two hours on a math problem than have anyone or anything help me with it, like I really want to figure it out on my own. That’s something that’s really important to me, and so the more help that I get with it, the less satisfied I am when I solve it. And one of the things that I really like about AI versus a human in that is like with your professor, they don’t really know what pace you learn at, and they don’t really know what specific spot you’re stuck on. And so with a professor like they might just either show me the whole problem or do it in that way where they’re trying to get me to figure it out, which are both great methods, but it’s not necessarily what I’m looking for in that situation all of the time. And so with AI, what I can do is I can do my steps and say, “Okay, this is where I’m stuck,” and then just see what’s the next thing. Or, like, “give me a hint on something small,” which professors sometimes are really good about that. The professor for this class was actually great with that. There was a couple questions that I went to him with, because I’m not going to use AI to solve the math problems. Also, it doesn’t know what it’s doing. So I would go to him and be like, “I’m really stuck, can you give me a hint?” And he would do that, which is great. But when it comes to a practice problem, I don’t want to go in and be like, “Can you give me a bunch of practice problems and sit there and solve them?” So there was that. I also feel like, for me, with the way that my schedule has been because I’m an overworker, so I tend to be like, overloading on credits, working two jobs and running a bunch of student organizations. I don’t have time to go into a professor’s office hours. I could send them an email, but a lot of the time, I find that communication to just be a little bit ineffective. We’re on different time schedules, so when you reply to me, I might not be able to reply for a bit. So there’s that side of it too, where, like, I’m not available during your office hours. I’m honestly not available when you’re awake, most likely. So I can’t really make that work. And so a lot of the time, it’s easier to have something that’s available whenever I have that five minutes to work on a problem, instead of like, “Oh, I have five minutes, let me email them and see if, in this five minutes, they can give me practice problems.” So there was that as well. And then, especially for students like me who struggle with anxiety, going to talk to your professor is like, one of the hardest things that you will ever have to do in college. Like, it’s just really intimidating. And if you don’t feel close to them already, it’s really hard, at least for me, to go into their office and have that one-on-one interaction. With professors that I bonded with throughout the class, throughout the year, like at the end of the year, sure, I’ll go to your office hours, but at the beginning of the year, when I don’t really know you yet and I’m not comfortable around you, that’s just too difficult for me, and so having AI there as a backup and as an extra resource is really helpful.

Rebecca: Thanks for sharing those. I think those are hard things sometimes for students to articulate and say, but I think it’s a common experience for a lot of students. I think one thing that you’ve pointed out, and I’m really curious how you got there, was being able to fully articulate what exactly you needed. So because you were able to say, “this is what I need” to the tool you could get what you needed. But sometimes people aren’t quite at that point, and you hinted at maybe there was times when you weren’t quite sure what you needed. Can you talk a little bit about how you develop some of those skills to be able to articulate what you need?

Kaija: Yeah, so I like to say that I’m a pretty self-aware person, so that certainly helps, like, I am a little bit better at being like, “Okay, this is an area that I’m struggling” or I know what I need right now, but there are obviously times when I’m like, “I have no idea.” So sometimes I had AI help me with it, like I would tell it, maybe just throw things at it, just like word vomit into it, and then have it be like, “Okay, this is what it looks like your problem is,” because it somehow can pick that out. But also sometimes it was like, just using my resources, so looking back through tests or the homework and figuring out where did I get stuck there, and really diving into it and thinking deeply about like, “Okay, I know I can’t solve this problem, but why can’t I solve it? What is it that I’m confused? Do I just not know the definition? Or is it I’m halfway through the problem and I just can’t get to the next step?” So it’s really a lot of kind of talking to yourself. For me, I do that out loud, and it really helps. But just like, having that conversation with yourself about, like, “Okay, let’s really think about this problem, and let’s really think about what part of it is confusing,” or “Let’s think about this class as a whole, and why are my grades not as high as they should be? Where is that problem?” So, like, part of it for me, like I said before, was really just that I wasn’t paying attention in class, and that is something AI can’t help me with. AI cannot help me study for a class if I don’t even know what’s going on in the class. So that was something that I had to be like, “Okay, this is a me problem, and I need to figure out a way to fix it so that then I can use AI as a supplemental resource.”

John: One of the things that’s been in the news a lot are student and sometimes parent reactions to faculty using AI to generate questions, problem sets, and other things. How do students react, in general, to faculty use from what you’ve heard from other students and your own reactions?

Kaija: Honestly, I haven’t heard a crazy amount about it. I think that for the most part we get it. We’re like, “hey, whatever.” I mean, as long as you’re still teaching us, as long as we’re still learning like, “it’s okay that you’re using it,” but where we get stuck is, “how come you can use it to teach us, but we can’t use it to teach ourselves?” That is the part that bothers me. I’m totally okay with you using AI to help you generate the syllabus, generate the questions that you’re going to ask, the learning objectives, like, that’s fine. I don’t really care. But if you can do it, then why can’t I? And to me, it makes that gap between us more. I understand that the professor is above me and standing like, that’s fine. I’m okay with that, but I want, at the same time to feel as respected by them as I’m supposed to respect them. And so when you say, like, “Well, I can use it because I know how to use it responsibly, but I don’t trust you to use it responsibly.” That’s where I kind of have a problem, where it’s like you need to respect students in the same way that you would expect them to respect you. So if you want your students to be okay with you using AI to assist you with the learning objectives, the curriculum, whatever it is, then you also need to be okay with students using it to assist them.

Rebecca: Well Kaija, this has been really great, but we always wrap up by asking, “what’s next?”

Kaija: Yeah, so who knows? So, as I said on the last one, I’m moving to North Dakota. I do have an apartment set up. So moving in on the 15th of August, which is exciting, still looking for a job, which is terrifying. Currently, hopefully I’m going to have one. I’m going in for like, the third round of interviews. So that’s hopefully a good sign with one company, and otherwise, I’m just really, really hoping to do something in human resources out in North Dakota for a couple years and then hopefully come back here.

John: Well, it’s been great talking to you, and we really appreciate your willingness to join us for these two conversations. And we hope everything goes well.

Kaija: Thank you.

Rebecca: Yeah. Best wishes on your job search.

Kaija: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

390. Critical Teaching Behaviors

Student feedback is important to improving teaching, but may not be aligned with evidence-based teaching practices. In this episode, Lauren Barbeau and Claudia Cornejo Happel join us to discuss a midterm student feedback instrument focused on critical teaching behaviors, an AI-assisted tool for analysing the feedback, and strategies for debriefing with students.

Lauren is the Assistant Director for Learning and Technology Initiatives at the Georgia Institute of Technology. Claudia is the Director of the Center for Teaching and Learning Excellence at Embry-Riddle Aeronautical University. Lauren and Claudia are the authors of Critical Teaching Behaviors: Defining, Documenting, and Discussing Good Teaching as well as a whole series of other resources related to this book.

Show Notes

Transcript

John: Student feedback is important to improving teaching, but may not be aligned with evidence-based teaching practices. In this episode, we discuss a midterm student feedback instrument focused on critical teaching behaviors, an AI-assisted tool for analysing the feedback, and strategies for debriefing with students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Lauren Barbeau and Claudia Cornejo Happel. Lauren is the Assistant Director for Learning and Technology Initiatives at the Georgia Institute of Technology. Claudia is the Director of the Center for Teaching and Learning Excellence at Embry-Riddle Aeronautical University. Lauren and Claudia are the authors of Critical Teaching Behaviors: Defining, Documenting, and Discussing Good Teaching as well as a whole series of other resources related to this book. Welcome, Lauren and Claudia.

Lauren: Thanks, John.

Claudia: Thank you.

Rebecca: Today’s teas are:… Lauren, are you drinking any tea today?

Lauren: I sure am. I’m always drinking tea. So I am drinking a passion fruit peach tea, courtesy of my friend and colleague Karen, who went to the SOTL conference in Savannah and brought this back for me from the Savannah Bee Company. So if you’re listening, Karen, thank you.

Rebecca: That’s a nice little story. How about you, Claudia?

Claudia: I am also drinking tea that comes to me by way of a conference, mine was in Albuquerque, New Mexico, and the tea that I’m having is a Rajasthani chai tea.

Rebecca: Oh, yum. How about you, John?

John: I am drinking tea that came from a Wegmans, [LAUGHTER] which is a black raspberry green tea from the Republic of Tea.

Rebecca: And I have a Blue Lady tea from Scotland.

John: Very nice. We’ve invited you here to discuss the critical teaching behaviors framework that you have jointly developed and some recent extensions of that framework. Could you tell us a little bit about how this project got started?

Lauren: Sure. So this project really started when we were working together at Georgia Southern University, and at the time, we were very much in need of a campus language in which to discuss what good teaching is. What happened after that is we found out lots of other people were in the same place. They also needed some sort of a common language in which to ground their conversations about teaching on campus. And so what started off as meeting a campus need ended up expanding to meet a broader need in higher ed.

Rebecca: Can you provide a little bit of an overview of the framework?

Claudia: Sure. So the critical teaching behaviors framework is based on sn extensive literature review that Lauren and I did, starting with the Scholarship of Teaching and Learning educational research, primarily in the context of higher education, but also some K through 12 insights found their way into it as well, and based on that literature review, we narrowed it down to six important behavioral categories that all of us as we teach, should ideally engage in in some way, shape, or form to support student success, and those six behavioral categories are align, making sure that our course materials activities align with each other; include, making sure we’re creating an environment and opportunities for all students to be successful; engage, making sure that students are taking an active role in their learning journey; assess, making sure they have a chance to self-assess, to get feedback on their learning; integrating technology, we cannot really think about teaching and learning without technology anymore, and so because it’s such a big part of teaching and learning, we need to think about it intentionally; and then finally, reflect, which is what comes back to us and the importance that we play a role also in continuously growing and improving our teaching.

John: You’ve also developed a number of instruments for faculty use in alignment with this framework. Could you tell us a little bit about these tools and why faculty might be interested in using them?

Lauren: So, this book was designed to be a one-stop shop for faculty. The first half of the book is an overview of the research we did on those six categories of behaviors, as well as reflection questions to help faculty readers think about how they can implement those behaviors. The second half of the book is really all about helping faculty document the fact that they’re doing those things. So we start the second half with some narrative starter templates and some deep reflection to identify your core teaching values, because our values should guide the principles that we use when selecting strategies to implement in our classrooms. That should frame everything that we’re doing. After that chapter, we do a deep dive into peer observation of teaching, which, coincidentally, is our next project. I know I’m jumping ahead to the end where we talk about what’s next, but we do have a book under contract with Routledge to explore peer observation of teaching further. But in this chapter, you get just the basics of what you need to go out and do this right now. So that includes email templates, logistical information collecting templates, some reflection starters for the faculty being observed. It also includes a note-taking instrument for faculty who need help figuring out what I should take notes on when I go into the classroom. And finally, it includes a report form for faculty to collect what they saw in the classroom, with space for the faculty being observed to write down their own reflections. That way, we’re giving faculty agency and voice in the process of creating this artifact. Chapter after that moves into thinking about student feedback, and how we collect student feedback from our students, and how we use that to make decisions about our teaching as well. That chapter really focuses on student feedback, in particular at the midterm, not necessarily at the end of course, which may be what most of our listeners think of when they think of student feedback, but a lot of the work that we’ve done more recently has been about collecting midterm student feedback. As we know from research, this can help boost your end of course evaluations when you debrief it correctly. I always want to add that caveat when we say this, it isn’t a magic wand that we can wave and ensure that we get better evals at the end of the semester, but when we debrief it with our students correctly and meaningfully, one of the things we found when we looked at the research on it is that, yes, it can boost your end-of-term evaluations.

Rebecca: So we’re hoping we can dig into that midterm feedback aspect a bit more. In the Fall 24 issue of To Improve the Academy, you published a formative midterm feedback instrument that you developed and validated. Can you talk a little bit about this instrument and why faculty might want to use this instrument and midterm feedback just generally?

Claudia: Yeah, happily. The critical teaching behavior, midterm feedback instrument, we started implementing at Embry-Riddle in 2021 and we have since collected data on its use across the different departments and across different instructors. And one thing that I’m super excited about is that from the first year that we implemented it, we had about 20 or so faculty using the instrument per year, and now we are reaching about 20% of the Embry-Riddle faculty with that tool, which is pretty amazing. It has been very well received. It combines differently than some other midterm feedback processes. It combines multiple-choice rating questions with open-ended feedback. Generally, midterm feedback often focuses on only the qualitative responses, so open response feedback from students, and that has really resonated with the STEM faculty, specifically at our institution, because it gives them a place to look at, okay, here’s where we started, here is where we are next semester. This is what we had at midterm. We made changes, we talked to students. Now, looking at end-of-course feedback, we can see that there has been growth or not. So that’s been really helpful for a lot of faculty here, because we’ve had such a lot of participation with it, it also gave us really good data to look at: “Okay, does the tool that we’re using, is it actually statistically reliable and valid?” And so that is the study that we published in To Improve the Academy, we validated the tool and did a confirmatory factor analysis, and it was valid and it was reliable, but importantly also, as part of this study, we asked faculty on their perception of the tool, and we asked students to also tell us, “Okay, these are the questions that we’re asking in the survey, how do you interpret it? What does that mean to you when we’re asking this question? What would you look at in a course to give you information on how to answer the question?” And so triangulating all of that different information and data, we found that the instrument is working really well for students, for faculty, and also the data that we have shows that it is reliable and valid. So that was exciting to see.

Rebecca: It’s always nice when we take on a project like that, and it works out, right? [LAUGHTER]

Claudia: Absolutely

John: A lot of the teaching evaluations used in many departments often are not very well designed, and they often include many categories that students may not be the best source of judgment. One of the nice things about your evaluations are that they focus on observable behaviors that are evidence based and they are things that should be observable to students, and if they’re not observable to students, the instruments would reveal that in ways that suggest maybe the faculty needs to address those. So one of the things you suggest is that this instrument might also have some impact on the bias, which has been demonstrated to show up in student evaluations of teaching. Could you talk a little bit about that?

Claudia: Yeah, and you see me nodding along and getting really excited, because that is another thing that, when we first published the book and the article, we had some preliminary data that suggested, yes, it reduces bias broadly, but we actually had an opportunity to gather additional data on that and to look at that, and we are still in the final stages of preparing the data and analyzing it, but we have done some analysis on this now that shows that if we’re looking at multiple factors that all impact teaching and learning and student feedback in some way, the questions that we are asking with a critical teaching behaviors midterm feedback instrument actually show now that statistically, gender in particular, is no longer a significant factor. So that means that, basically, gender is not one of the demographic categories that seems to be influencing the student responses to the questions that we’re asking. We do see, in the data analysis that we have now, that teaching experience, so years of teaching, as well as age, are still statistically significant, but gender is one of the categories that is no longer statistically significant. So that was exciting to see.

John: You mentioned that age and years of teaching experience were still significant. What was the nature of the impact of teaching experience and age on the evaluations?

Claudia: So if we’re looking at the data that we have, it shows that teaching experience and age, so they go parallel, the longer teaching experience that you have, as well as the older you get, it generally signifies an increase in the rating that students give you on how frequently you are engaging them, on how frequently you are creating an inclusive teaching behavior in the classroom, on how frequently you are providing feedback that is meaningful to them. So overall it increases until around the plus 55 where it falls off. But that is also probably a limitation, because we had very few respondents in that age group. So, that’s what we found.

Rebecca: I can imagine at this point in the conversation, audience members might be like, yes, this sounds good, but what does it look like? Can you talk a little bit about what some of the questions are? You mentioned that there’s some multiple-choice questions, some open-ended questions. Obviously midterm implicates when in the semester you might implement this tool. Can you talk a little bit about what those questions actually say?

Lauren: So let me talk first about what midterm student feedback is. Obviously this is something we collect at midterm, specifically between weeks five and 10 is what research recommends. There are a lot of tools out there [LAUGHTER] to collect midterm feedback. The standard forms are all open-ended questions. Typically they’re two to three questions long, maybe a little bit longer if you as an instructor choose to add an open-ended question or two. The most common questions that are asked are: “What’s working well in this class? What’s not working? What additional feedback do you want to have or want to share?” There are also versions of this that look like, “What should I start doing? What should I stop doing? and What should I keep doing?” So I just want to put that out there that we are not the originators of midterm feedback. This has been around for a while, and there are lots of different ways of doing it, although they’re fairly similar and standard in the fact that they are qualitative, open ended. What makes ours different is that we did choose to depart from the purely qualitative aspect of this by introducing some quantitative questions. So with our feedback tool, there are still the standard questions: “What’s working well, what’s not working well?” From a data analysis perspective, I think it’s interesting to note that we chose to put the quantitative questions first on our survey for students. So we have on the validated tool 15 quantitative questions that students answer on a never to very frequently scale, and those questions are broken down by each of the CTB categories, except for reflect, because students really can’t see us reflecting on our teaching, so that’s not something we ask them to give feedback on. But an example question would be… so from the aligned category, “The instructor states the learning outcomes, development of specific skills and knowledge to be accomplished in the course assignments and activities.” And then students would answer never or very frequently. So each of these questions is designed to capture how often an instructor engages in particular behaviors that we know from the framework research are proven to promote student success. When the instructor gets this report, they see a breakdown of their behaviors by category, so that they’re able to look at the quantitative data and see overall strength by category. So for instance, you might see that you’re doing really well in “engage,” but students said that you only rarely or sometimes gave feedback that was helpful for them. So maybe the “assess” category is an area for development. So what’s nice about these quantitative questions is that it’s allowing you to get the nuance of individual behaviors while also capturing overall pictures of what’s going on in these categories from the student perspective. Now, I mentioned that we put the quantitative questions first on the survey. There’s some interesting stuff going on there in terms of what students end up putting down in their qualitative feedback, and this is something we would like to study in more depth, is how much does the language of the quantitative questions prime the students in terms of what they give feedback on and the language that they use. This is not something we’ve run a study on yet, because we’re running all sorts of other studies, but it is a study that we do plan to run on the data we’ve collected through midterm feedback, because if this bias study has shown us anything, the tool does help mitigate bias to some extent, if the tool also does some priming of students to help direct their feedback to particular areas of teaching and take them away from comments about “I love her shoes” or “I hate his shirts,” then it has an additional use factor, which is that the quantity of questions are doing some work to teach students how to give good feedback. That’s our hope, anyway.

Rebecca: So let’s imagine that I’m an instructor, and I use the midterm feedback form and I get some feedback about areas that maybe I’m doing well in, and areas that I might need to improve upon. What do you suggest are the instructor’s next steps?

Lauren: Well, I would suggest that you consult with your Center for Teaching and Learning. Oftentimes, this is a service that Centers for Teaching and Learning offer, and there are trained consultants like me and Claudia who can help you make sense of what you get out of that experience. In fact, we can help facilitate that experience as a neutral third party, which can help students feel like they’re not necessarily going to be judged on their responses. Even if you make it completely anonymous, there are always those students who say, “Well, but if you really want to, you can figure out who said what.” But if you have a neutral third party who is facilitating this experience, it can make it a better experience for students overall. Not everybody has the luxury of access to a Center for Teaching and Learning. And when we were designing our tools, we were thinking about that. So the next thing I would say is, if you don’t have a Center for Teaching and Learning and trained consultants, you do have colleagues. One of the things that is very important to us in this project is that we’re creating dialog around teaching and learning. So if you have a friend or a colleague who would also like midterm feedback, maybe you can conduct this for each other and have conversations with each other about this instrument. Now, of course, you can always just administer the tool yourself, collect the data and sit down to analyze it. If you’re going to sit down to analyze your own data, here’s what we have for you. If you request access to our folder of materials, you’ll see in there an Excel spreadsheet that Claudia, who is an Excel guru compared to me, has created that will help you decipher the quantitative data. It’ll spit out some really nice graphics for you to be able to analyze, but I’m super excited to be able to share that we’ve also developed a way to help you make sense of the qualitative feedback on teaching. This is where we are debuting the CTB FIT, critical teaching behaviors feedback insight tool that is an AI-powered platform that we developed in conjunction with a student at Georgia Tech, Elliot Roe, and his partner in crime, Duncan Johnson, who’s at Tufts. And these students helped us train the AI platform on the critical teaching behaviors categories, again, not including “reflect,” because it’s not something that students can give feedback on. And what this platform will do is you feed it the student feedback, and it will output an Excel spreadsheet for you where it tags the qualitative data with the five CTB categories. It’ll pull out snippets and code them into each of the CTB categories. Then it will do a count of the number of times it tags the category and it will produce a nice bar graph, so you get a visual of how many times something shows up, for instance. This has been really useful to the faculty who are conducting midterm feedback. We’ve gotten some great preliminary insights from faculty saying that this helps them make sense faster of the data that they have collected from students, and it helps them focus less on that one-off comment, you know, the one, the one that gets under your skin, and you have all these other wonderful comments praising your teaching, but there’s that one that sticks with you, because we just can’t shake the one negative comment. This is why it’s so helpful to have a consultant do this, or to have a peer do this, because they can move you past the one negative comment. But if you’re doing it for yourself, it’s so easy to get hung up and miss the bigger picture. What FIT does is it organizes the data, provides you AI summaries in each of the CTB categories that it tags so that you’re not focused in on that one negative piece of feedback, you can now focus on the big picture and take away insights that allow you to report back to your students and make informed decisions about how you’re going to teach your class. Claudia, what would you add?

Claudia: Just reiterating, I think the importance of using this also as a tool for conversation. Lauren already mentioned that colleagues can play a really important role in helping us make sense of this. We have the tool that’s really available for anybody to use who’s interested. But I think the other important thing to mention also is when we’re looking at the feedback that we are receiving, one important conversation partner is also our students, and so making sure that whatever we’re learning from the feedback, whether that is positive or room for improvement, we go back to the students and say, “Hey, I learned this from you. I appreciate you for providing the feedback. And here’s something that I can change, maybe in response, here’s something that I will not change because I have a good reason. Let me explain that.” But yeah, so I think just reiterating the importance that students have in this process, not just in providing the feedback, but also being a conversation partner with us.

Lauren: And I would close that thought by adding to do this recent study that we’re working on, we ran a couple of focus groups with faculty. One of the really interesting insights that came out of that focus group for me was that faculty who do midterm feedback and debrief it with their students, go back and they talk to them about what’s not working in the class, what they can do to change it, and what they don’t have the power to change. But there was this big revelatory moment in the conversation where a faculty member said, “I have never talked to my students about what they say is working well. I’ve never gone back and said, ‘Thank you for letting me know.’ I have never debriefed that with them.” So if there’s anyone else out there who is realizing that they also only focus on what’s not working and what they can and can’t change, this was a really great moment where faculty came together and said, “Oh, we also need to talk to our students about the positive feedback that they’re giving on our teaching and what is working. So that’s part of the loop that we need to close as well.”

John: Now, you mentioned that this is an AI tool. Was it created as a GPT in ChatGPT, or did you use some other system for creating it?

Lauren: This is a question primarily for Elliot Roe, who’s not here today, but Elliot created a pipeline that is trained on the critical teaching behaviors, and it goes to ChatGPT. So what we were careful to do thinking about: this is student data, this is faculty data, we want to protect privacy. We don’t store your data. First of all, that costs a lot of money, and second of all, that gets into all sorts of privacy concerns. So if you run any sort of data through the platform, you need to download that report immediately, or the next time you go to run a report, it will be gone. So it’s actually stored in your cache, it is not stored on our site, so that data belongs to you. As we’ve set it up, it’s not saving the data in any way, shape, or form, and it’s not being used to train ChatGPT on anything, as far as we can tell from talking with Elliot, who is our technological guru here.

John: We have encouraged faculty to consider using ChatGPT on anonymous student surveys on our campus, because it has all those nice advantages that you’ve talked about, that it can summarize things very nicely, and it can focus on the major themes that occur, and it can also offer some suggestions on ways in which faculty might even address that if that’s programmed into the chatbot. So it sounds like a really great tool. Are you sharing this more widely beyond your campus?

Lauren: Yes, that is the goal. It’s free to use where we want to keep it free to use, because our goal is to improve teaching in higher education, and we believe that this is a tool that can do that. To be clear, you’re not going to sign into ChatGPT when you do this, CTB FIT is its own platform that you sign into and put the data into and it will output the Excel spreadsheet for you through that platform. We do ask you to sign up. One of the ways that we’re trying to protect privacy and security is to have sign ups, but that’s not because there’s a pay wall, it’s just because that’s an additional security measure to have a sign up on it. So we’re putting it out there. We hope that faculty use it. Our idea behind this is that, yes, you can run your data through any AI platform you want, and it will give you some insights, and it will probably do some analysis for you. The problem I’ve seen when I run data through it is that it’s not trained on any sort of teaching database in particular, so depending on its mood that day, [LAUGHTER] you will get all sorts of different answers out of it, and some of them might be good, but as we know, some of them might not be. We think that the value of FIT is that it’s grounded in that common language. So if you get your report back and you see that students are seeing a dearth of behaviors, for instance, in the “Engage” category, and you want to know what behaviors you could implement in the “Engage” category, well, we’ve got a whole book on that for you. You can go to the chapter, you can reference the research that we’ve done. You can dig deeper into that category. There is a real knowledge base behind this platform that allows faculty to dig deeper into their own teaching. You can dig as deep as you want. You can go reference the chapter in the book. You can start looking at the references and go look at the research for yourself, and then the book, of course, is structured to produce reflection, so that the reflection that faculty engage in produces growth. So if you then want to set a goal for growth, we’ve also got tools in the book to help you do that. So really, the idea of FIT is that it can be as surface-level analysis of data as you want it to be, but because there’s this knowledge base behind it, you can dig deeper and deeper and deeper as much as you want.

Rebecca: One of the things that you’ve talked about is that debrief, but I imagine the setup of the midterm feedback is equally as important. So can you talk a little bit about what the setup is as you present the opportunity for feedback to students, as well as what that debrief ideally looks like? You’ve hinted at some of those things, but I know people are always looking for very concrete, tangible ways of doing things.

Claudia: Yeah. And so when we are facilitating midterm feedback as a third-party facilitator from the Center for Teaching and Learning, we usually go into the classes, if it is a face-to-face class, and give that introduction to students, telling them that this is something that’s voluntary for the faculty. They are not required to do that. They are engaging in midterm feedback because they really want to know how things are going for the students and whether the strategies and materials that they have chosen are meeting the students’ needs, and so emphasizing that it’s voluntary, but then also telling students that not all feedback is equal. And so if I’m just saying, “Oh, this class is awesome, that is great affirmation, but it does not help me really get better at my teaching.” And so also telling the students how important it is to think about providing feedback that is specific and actionable, pointing out, if they are saying it’s amazing presentations, giving some more feedback on well the PowerPoints are great, or it’s a great interactive lecture that helps me think along. So really providing some of that detail. If we are not in a face-to-face classroom, we do have that information also available as a script that faculty can use and then share with their students through the learning management system, as an announcement, again, mentioning it’s voluntary, it’s really helping me, as the instructor in your class, grow, it’s helping you as a student provide me feedback now, rather than waiting until the end of the term, please provide feedback that is as specific as possible. And so that is usually what the setup looks like. For the debrief for our campus, we are meeting with the faculty member one one-on-one to facilitate the debrief, and it is really about the conversation where we are looking at the data primarily and then make sense of it together. So for us, it is really important to not be prescriptive, even if we’re saying, “Oh yes, maybe engagement is a category where there’s room to grow,” it is not that we’re saying, “Oh okay, engagement is not good right now, let’s do think-pair-share every single day of the week.” So really thinking through also there are many ways in which we can engage our students. And what are some options that really work for your course, for your context, if you are online, face to face, what is your personality like? What is your teaching persona? And really finding strategies that are helpful for the individual, rather than a silver bullet that works across the board, which doesn’t exist.

John: One of the things I really liked about your book is part two. Part one, I really liked it was very comprehensive, and it provided some really good strategies that are evidence based. But the second part of it, and the appendices as well, are very helpful, because a lot of faculty, especially when they start their teaching career, but often continuing for quite a while, suddenly get hit with this need to write up a summary of their teaching activities and a reflection on their teaching that’s used for promotion and tenure. And there’s very few resources out there that address that, and your book does a really nice job of giving faculty information on what they may want to include in these reports and what they may want to submit as part of the reflection. This seems like a gap in the professional development literature that your book fills very nicely.

Lauren: First of all, thank you, [LAUGHTER] that’s a lovely compliment. When we were developing this framework, the need to have the conversations around teaching grounded in the same language was a paramount issue on the campus we were working at, but we were very attuned to the needs to document teaching, because this is part of the conversation. So we can have conversations with peers, we can have conversations with administrators, we can have conversations with students. Those are sort of the three different types of conversations we have, and they all need to be grounded in the literature of good teaching. But it’s really hard to have that conversation in any sort of personalized way if you have none of your own data to back it up. This is a very timely conversation, because just yesterday, I ran a two-hour session on reflecting on teaching for faculty at Georgia Tech, and part of what we did is we started with, “What do you think are your teaching strengths and what are your areas for growth?” We just put that out there into the ether and let them put something down. But after we had talked about the CTB framework, we went through a more structured teaching practices inventory and had faculty sit down and go through the framework and say, “What are you actually doing in each of these categories? Now look at it across categories. What is your teaching strength and what is your area for growth? How does that compare with what you set when we first started this session and you had no common language, you had no scaffolding to help you make that decision?” It was a really interesting conversation, because someone said “it was hard,” like, “well, it was hard in the sense that it was work. It was a lot of work to sit down and really look this closely at my teaching.” So I think part of the reason this doesn’t happen more often is because faculty are busy. We’re busy people already, and making time to formally reflect on teaching is difficult because it’s just another thing that we don’t have time for, and it seems like there’s always another fire that we need to put out. So it doesn’t become a fire to put out until you need to create your materials for promotion, and then it’s an issue because you haven’t created any of this documentation along the way. The goal of our book is to help faculty reflect as they go to build in just a little bit of time to do that structured reflection so that they can identify areas of strength and set intentional areas for growth. Why? Because a narrative of growth over time is persuasive, and when you go up for promotion, being able to show intentional goals, data you’ve collected in support of those goals and data doesn’t have to be a full-blown research study. It can be student feedback over time that shows you’re getting stronger in these categories. Having all of these aligned CTB tools gives you a common language to collect data from students, peers, and yourself, so that when you go to create that portfolio to showcase what you’ve done, you no longer have to reconcile “My peers are talking about my teaching this way, but students talk about teaching this way, and admin understands it this way. How do I put all these different perspectives together in a way that’s going to make any sense when they seem to all be speaking different languages?” Well, the tools we’ve developed are all aligned to the same common language, and when you go to put together your portfolio, it’s that much more powerful and persuasive to have the common language, to have collected the data over time, as opposed to be sitting down at the last minute saying, “I’ve got to put this together, and now I’ve got to figure out what I did,” as opposed to 10 minutes a week could allow you to create a much stronger teaching portfolio.

John: And you provide a number of resources in various appendices at the end of the book that faculty can use for some of these purposes. C ould you just briefly talk a little bit about what’s included in these appendices?

Claudia: Yeah, so in the appendices we have the framework, of course, but also we have the peer observation report form, as well as associated materials that help peers set up the observation as for information before we even go into the classroom, or before we go into the online class. It has a prior version of the midterm feedback, so there is an updated version that’s available on our online share documents that anybody can ask for, but it gives a nice preview. But it also has some reflection questions, as Lauren said, that help prompt some of the reflection for faculty to start putting together the material into a coherent format for a portfolio. Also you mentioned it is Creative Commons licensed, and that is true, that was also an intentional choice, because we recognize that while the critical teaching behaviors framework, we hope, provides a great starting point for everybody, there are institutional preferences. Sometimes, when it comes to language, there are institutional focus areas that may not necessarily be captured with what we are providing. And so it is intentionally Creative Commons licensed. So it is a starting point that people can build on, based on departmental needs, based on their individual needs based on institutional needs.

Lauren: It was important to us that these materials be Creative Commons licensed for meeting contextual needs. I think as much as we recognized that a common language was necessary in higher ed, we also recognized that institutional context matters, individual context matters, and that there can’t be one definitive way of doing things. The Creative Commons license doesn’t just allow you to adapt to your context. It allows you to make these multifunctional. So let’s say that you really want to do a dive into your assessed behaviors, and you want to have a peer observation report that’s just focusing on your assessed behaviors. You can do that if you have an institutional initiative. Down here in the south we have quality enhancement plans for our accreditation. Let’s say you wanted to focus on just one aspect of the critical teaching behaviors to align with institutional objectives. You can do that. The only thing that’s not Creative Commons licensed is the framework itself. So that still stands, but we’ve had a number of institutions take that as a place to develop their own institutional framework. So Harvey Mudd, in particular, has done a great job with this. Most recently, they even added a whole new category on mentorship, because that meets their campus need. But we’ve worked with a number of institutions now that have really done a good job of taking this not as: “we told you, this is the language,” but saying “somebody went out and did the research for us so we can develop the language that we use here at our institution.” We see more and more of this happening, but when it’s most effective is when it’s grounded in evidence-based practice. And not everybody has the time to go out and do that research, but apparently we did, [LAUGHTER]so we went out and provided that for people to go and build their own frameworks.

Rebecca: It’s always nice when there’s a starting point and you don’t have to [LAUGHTER] start from scratch, and that there’s templates and things that you can iterate on so that you can move forward faster. So we always wrap up by asking, what’s next? You’ve given a little preview, but…

Claudia: We just look forward to talking more about critical teaching behaviors no matter what, because we believe it can make a huge difference in fostering conversations across campuses. But then also, as Lauren said, we are really looking into peer observation, more specifically, as a next focus to really dig deeper, as a way to continue those conversations, as a way to apply critical teaching behaviors, but also as a way, again, to have a starting place to then customize something that meets somebody else’s context.

Lauren: So we always have a little bit too much in the way of what’s next. [LAUGHTER] In the way of what’s next. I think we would like to continue the research projects on midterm feedback in particular. So midterm feedback, we would like to do that study on whether the quantitative questions have any bearing on the qualitative responses we’re getting from students. That is an area we’d like to follow up on, but we do have that book contract, and we’re supposed to have that manuscript done by June of 2026, so that’s gonna have to take precedence over any other projects that we might want to delve into in the next year.

Rebecca: Well, thanks so much for joining us today and sharing some of your work and the framework.

Lauren: Thank you for having us.

Claudia: Thank you for having us. It’s been great.

John: Thank you for joining us. And again, this book is an excellent resource for faculty at any stage of their teaching.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

385. More Than Words

Many students use generative AI tools to complete writing assignments. In this episode, John Warner joins us to discuss what may be lost when they do so. John has twenty years of experience teaching college writing at five different institutions and is the author of 8 books encompassing a wide variety of topics including political humor, short stories, and a novel, including Why They Can’t Write: Killing the Five-Paragraph Essay and Other Necessities. He writes a weekly column on books for the Chicago Tribune and an associated newsletter, The Biblioracle Recommends. John is also a contributing writer to Inside Higher Ed. His most recent book is More than Words: How to Think About Writing in the Age of AI.

Show Notes

Transcript

John: Many students use generative AI tools to complete writing assignments. In this episode, we explore what may be lost when they do so.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is John Warner. He has twenty years of experience teaching college writing at five different institutions and is the author of 8 books encompassing a wide variety of topics including political humor, short stories, and a novel, including Why They Can’t Write: Killing the Five-Paragraph Essay and Other Necessities. John writes a weekly column on books for the Chicago Tribune and an associated newsletter, The Biblioracle Recommends. John is also a contributing writer to Inside Higher Ed. His most recent book is More than Words: How to Think About Writing in the Age of AI. Welcome, John.

John W.: Very glad to be here. I’m a longtime listener and fan.

John: Well, we’ve been a long time reader of your columns and your other work. So we’re very pleased to have you here.

John W.: That’s great.

John: Today’s teas are… are you drinking tea by any chance?

John W.: Not at the moment I was earlier, I have sort of a post-lunch tea to keep me going. And post lunch, it is green tea, breakfast, I’m a black tea guy, and I’m into tea because this is a strange true fact about me. I’ve never had a cup of coffee in my entire life. So I’m only a tea drinker as far as warm caffeinated beverages go.

Rebecca: There we go. A true fan, [LAUGHTER] a true fan, unlike all those pretend tea drinkers [LAUGHTER] who drink coffee all the time on our podcast.

John: …and Diet Coke. [LAUGHTER]

Rebecca: Yeah, and Diet Coke, right.

Rebecca: I have English Afternoon today, although I was drinking a Christmas blend earlier to match our winter blizzard that we’re recording in.

John: Yes, it’s mid February here, but it’s been snowing about a foot or so or more of snow every day for the last week, it seems. So, that is very fitting. And I have an Earl Grey today.

Rebecca: Classic.

John: Yes, from Tea Forte, so it’s in a nice little pyramid.

John W.: Oh, wow.

Rebecca: So John, we’ve invited you here today to discuss your most recent book, which focuses on writing in the age of AI. You begin the book by describing initial reactions to the potential impact of the introduction of ChatGPT on writing. What was your initial response to the release of ChatGPT?

John W.: This may sound strange coming from somebody who teaches writing and has taught writing for a long time. I was concerned, but I was simultaneously excited. I saw it as an opportunity rather than a pure threat, because for many years, and people can go back to my archives at Inside Higher Ed, I’ve been lamenting the way writing is taught, not just in college, but prior to college, as a sort of formulaic exercise in filling in prescribed templates. And my experience as a teacher mostly of first-year college writing was that my students came in not with deficient skills around writing, but cramped attitudes about writing, where it’s not that writing was hard, it’s more that it was boring. They’d done the same thing over and over again, and couldn’t find any sort of intellectual interest or stimulation in writing. And I saw writing as the opposite of this. And so the fact that ChatGPT can churn out a five-paragraph essay in seconds, and it’s highly proficient at these things, to me, was an opportunity for us to examine our priorities around the kind of writing we ask students to do in school, and maybe even more importantly, than the kind of writing they do, the way we assess that writing, the values we bring to how we assess that writing. Has that been borne out in the two plus years since? …not as much as one would like, but that’s one of the reasons why I wrote More than Words. And there actually are some encouraging signs around these conversations, that we recognize both the limits of asking students to write like algorithms, which is what a five-paragraph essay format does, and the limits of the syntax generated by a large language model like ChatGPT, that there is space for humans to still write. I’ll believe this until they drag me off the scene, [LAUGHTER] but I think more people are seeing this. They maybe don’t have the kind of background or experience that allows them to really think deeply about it, but they sense that there’s a problem associated with outsourcing all this stuff to ChatGPT.

John: One of the things you write about in the beginning of your book is how Daniel Herman, in an Atlantic article, argued that ChatGPT would be the end of high school English. And as you’ve already indicated, that’s something that you welcome. Could you talk a little bit more about what’s wrong with writing in the K-12 system, this sort of mechanical system that’s become really standard for the last few decades, I think?

John W.: Yeah, it’s not that I want to not have high school English class. I think we should have more of those. But it would be great if high school English got back to helping students learn to think and relate to written texts, including literature, including whole texts, like books, as opposed to extracts that can be tested against quite easily. And really this is the sort of core of the problem. The standardization of instruction around something like a five-paragraph essay is driven by the standardization of curriculum around demands related to accountability measures that have been a long-term aspect of the so-called school reform movement. In my previous book, Why They Can’t Write, I sort of date it to the A Nation at Risk Report, which came out during the Reagan Administration when I was in middle school, where they were worried about a rising tide of mediocrity in the nation, of which apparently I was a member, because I was in eighth grade, and they were certain that Japan was going to eat our lunch because of the Sony Walkman, basically. We’re not making the Walkman, they are, and so we need to get on with this. This is the Sputnik moment of my generation, and it sort of began what is not ill meaning, right, like the idea that we should be able to measure what students are learning and be able to know where additional interventions should go in terms of schools or districts or students, is all very sound thinking, but it became, over time, an exercise in a concept known as Campbell’s Law, which is essentially a social science precept where once the measurement becomes the goal, it’s no longer a good measurement. And that has happened writ large in education in all kinds of different domains, and writing is one of the sort of most obvious manifestations of this, where we allow the creation of a five-paragraph essay to stand in for writing ability and writing experiences, and it just isn’t. Writing a five-paragraph essay is not the same process as writing something for an audience inside a full rhetorical situation. And so we now have a couple of generations of students who have really, in some cases, only experienced this, by their own testimony. They’ve written these forms over and over. And then they would arrive in my first-year writing class in college, and I would pull the rug out from under them and say, “none of this is useful.” They had been told explicitly, you have to do this so you can do it in college. And then I come in and say, “That’s all wrong.” These disconnects were what motivated Why They Can’t Write, and More than Words is motivated by my concern that now that we have this tool to which these forms can be outsourced, we’re going to forget to actually teach students to write.

Rebecca: So as you’re talking about the evaluation of writing, and earlier, you mentioned the values in writing and how they have changed, maybe need to change [LAUGHTER] again. Can you talk a little bit about what we should be valuing in student writing?

John W.: Yeah, I have long believed, long before ChatGPT showed up, that students should write in genuine rhetorical situations of message, audience and purpose. That audience should not be the teacher unless the teacher is an authentic audience for the thing they’re writing, which could be the case. We could be communicating something to a teacher as an audience, rather than teacher as assessor or grader. I think this is a good idea for a lot of reasons, one of which is students are much more engaged by these challenges than they are when writing for agrade, they’re much more authentic to the work they will do in other classes or outside of school once they graduate. The goal, as I put it in Why They Can’t Write and another book called The Writer’s Practice, is to develop their writing practices, the skills, knowledge, attitudes and habits of mind of writers. These writing practices, they’re then transportable to new and novel writing problems that they’re going to need to solve. It’s just much more flexible. It’s much more useful. And really, what I realized over the course of my teaching career that I wish somebody had told me before I started teaching, was that I had been lucky enough to develop this writing practice, sort of on an ad hoc basis, through the course of my education as an undergraduate, as a rhetoric major, my graduate studies in creative writing and English literature, and then a job at a market research firm where I had to learn how to write things like focus group reports and questionnaires, forms that I had no prior knowledge of, but which I realized were amenable to the same process I used to also write a form that I was not familiar with when I was in graduate school, which was a 20-page explication of a single poem. I didn’t know how to do this, but I could fall back on my writing practice that allowed me to do it, and I realized I had been given this great gift of a writing practice without anybody knowing it. And so my mission became to try to help students see that they could develop a writing practice. So those values are really just around the things writing is: the act of thinking, the act of feeling, the act of communicating. This is going to sound like I’m simultaneously tooting my own horn and of being a little coy about myself, but I did an interview recently where the interviewer called my work groundbreaking, and I laughed. And he asked why I laughed, and I said, there’s no groundbreaking. I’m doing what I was taught in grade school. It’s just that we have gotten away from these values around writing and so that it feels groundbreaking is really ironic to me, given that I think they’re fundamental, but we have to agree at the values level that these things are fundamental, otherwise we will be stuck in the same rut we’ve had, only now we’re letting ChatGPT solve these problems for us, rather than making students write these sorts of formulaic essays.

John: I also went through a school system where there had been no discussion of a five-paragraph essay. I didn’t run into that until some colleagues started talking about how students were not very good at writing these five-paragraph essays. And I wondered where that whole concept came from, because I remember when I was moving through, a little bit earlier than either of you were, but we had a wide variety of writing assignments, and we also did an awful lot of reading. And one thing that I think has been lost along with a decline in writing or with a shorter form more formulaic writing, is that students are not asked to read as much, and I think that’s also been affecting their ability to write, because engaging with multiple styles of text in reading can be really helpful in developing as a writer yourself.

John W.: It’s absolutely necessary. Reading and writing are inextricably related to each other, like the skill of interpreting reading across all kinds of different genres is the internalizing of the message. Like this is how this is delivered, and then we’re going to do this in turn. We are of a certain age. Those of us who are of a certain age did not encounter the five paragraph essay. My first encounter was in high school when I was taking AP classes, and I was coached to use this form to pass the AP assessment. And my first thought was, “Well, this is dumb, like this is the easiest thing I’m going to do all year is write these AP essays.” And indeed, they were. I dedicate Why They Can’t Write: Killing the Five-Paragraph Essay and Other Necessities to my actual grade school teachers, because in hindsight, I realized how fortunate I was by the accident of my birth and my parents to grow up in a Chicago suburb and go to my neighborhood grade school at a time where the teachers were given the freedom to engage their students, and I still have my fifth grade writing portfolio that my mom saved. If this was a visual podcast I would show it, but it’s called 12 stories, and it’s laminated construction paper cover tied together with hole punches, tied together with yarn binding. And if you look at those artifacts, none of them are a five-paragraph essay. They are all in some way asking me to solve a writing problem in a rhetorical situation. There’s the classic Thanksgiving assignment where you draw a picture of a turkey by tracing your hand and decorate it, and then for each letter of Thanksgiving, like T, H, A, et cetera, you have to give something you’re thankful about. There’s a limerick. There’s a haiku. There is a number of different genres of fiction, science fiction, historical fiction, which required us to read those forms, to understand them, and then produce them in the image of the original form. This one totally cracks me up. There is a classified ad that you would read in a newspaper for my skateboard. And so it’s like “Used skateboard, blue works good. $2 or best offer.” [LAUGHTER] Actually, it doesn’t say “or best offer.” It says “OBO,” which is the shorthand you would use in a classified ad at the time, because you paid by the word. That is rhetorical problem solving, that is internalizing how to think about audience as you write. This was fifth grade, and sometimes I can’t believe what we’ve done to students in the name of helping them try to prepare them for quote, unquote, college and career. We’ve really, in a lot of cases, done the opposite. And I know it doesn’t need to be this way, because I lived through a different time. I know something else used to be done. When I wrote Why They Can’t Write and went back and did all kinds of research, and I found articles dating back to when I was in grade school and middle school about some of the things I would later do that don’t help you learn to write, things like diagramming sentences, which is a valuable exercise in critical thinking, but doesn’t help you learn how to write or research papers. There’s a report from 1984 by this guy, Richard Larson, who wrote about how asking students to do research papers in high school is a fake genre, because you can’t actually ask high schoolers to do a research paper. Now, if the goal is to help us learn to use the library, or the card catalog at the time, that kind of stuff, that’s one thing. But if you think they’re doing research, you’re kidding yourself. And these are the things we need to know so we know what are students actually experiencing when we ask them to write in our classes. We have to think through that. It doesn’t mean that we should abandon whole genres or not have them do things ChatGPT can do, but if we’re going to ask them to do it, it has to be rooted in something meaningful. And to your point about reading, my superpower, which I don’t think of as a superpower, because it’s just been something I’ve been able to do since I was quite young, is reading texts for extended periods of time. It helps me with concentration, helps with my mood, [LAUGHTER] it helps me learn things. And the fact that we have allowed school to get away from that clearly important underlying skill, which is also a practice, which also has habits, which also has attitude, also has knowledge involved, is sort of tragic, and it’s a real heavy lift to get away from it. But it is, I think, like the thing honestly, to tackle if we’re going to help students become the kind of people they wish to be. Like, honestly, I think if we still read whole books, the distractions of cell phones and these things we argue about would be significantly less. Like, don’t get me wrong, I can be distracted by my cell phone like anybody, but I also know how to put my cell phone away and get lost in a writing or reading task for hours at a time, because I have that embedded in my practice. If students have never had that in their practice, and we’re asking them to suddenly engage in this behavior, we’re looking at trouble

Rebecca: Reflecting a little bit on your portfolio, John, and I’m thinking about the one that I also have from fifth grade, except at that time, I think I was also really exploring my design skills, because it’s hard bound…

John W.: Wow.

Rebecca: …and I made a full cover and bound the edges with thread, and all the things,.

John W.: That’s awesome, I mean my fifth grade portfolio is shameful because I had horrible handwriting. I always got “needs improvement on my handwriting” in my report cards and my artistic abilities are non existent, so that part is awful. But to your point, the opportunity to engage in even that relatively small instance of self expression in the context of something you did in school just strikes me as vanishingly rare in today’s culture of schooling, and not for the better in any way.

Rebecca: I was also reflecting on your mention of reading to change your mood, and I’m realizing how much I’ve really read since the beginning of this year, probably for the same reason. You argue in your book that ChatGPT just does text generation and not writing, because writing involves thinking and feelings. How do we convey this to our students and maybe to our colleagues?

John W.: Yeah, the only way to do this is to experience it for yourself. And my hope is somebody teaching in higher education, regardless of discipline, has had the experience of writing as thinking, which I like to talk about as the simultaneous expression of an idea. You’re trying to capture something that’s in your head and put on the page, but it is also the exploration of an idea. So as we are writing, we have this notion that we’re trying to capture, and often that notion shifts as we are writing. That’s the exploration of the idea, and that is by itself, the best and clearest manifestation of critical thinking that I can imagine. In fact, as the way I assessed writing evolved, I would have assignments where that was the only criteria. It was have a thought [LAUGHTER] while you’re writing. So the thing students were writing, it wasn’t that it was unimportant, but what they were actually assessed on was the later reflection about the thought they had while they were writing. Could they articulate to me the experience of thinking they had while they were writing this particular artifact, because I knew once that had happened and they had had that experience, it was something they could return to again and again and again, recognizing that there’s a difference between I’m just sort of going through the motions to get words on a page to satisfy a grade, and writing as an act of thinking and development and not only capturing our thoughts, but developing those thoughts as we go. So that’s the only way it can be done. But if we’re going to do it in school, we have to prioritize it, and in terms of feeling, it’s kind of the same thing. And I write about some of the stuff in my book, where I write about my own instances of being like, as weird as this sounds, emotionally moved by my own writing, because I’m conjuring something for myself and my memories or my life that is meaningful to me, but it can even be something small that upon reflection, we realized this writing was meaningful. I think I have this anecdote in the book; if not, I should have put it in there. But it was during the pandemic, I was grocery shopping and on the floor, and we’re supposed to keep our distance and only go one way in the aisles, it was that period, and on the floor, I saw a yellow post it note that clearly had somebody’s shopping list on it. And I picked it up. It had a bunch of items, but one of them was “cherry Pop Tarts” in all caps with three exclamation points, underlined, and I just had this sort of moment to myself. I said, “those cherry pop tarts are very, very, very important.” And reflecting on it later, I thought of the person who’d sort of put that notion in the list writer’s mind. But I also thought of the list writer, sort of, particularly at that time… we can remember how fraught everything felt… going to the grocery store knowing that the central importance of that mission was to come home with these Cherry Pop Tarts to help soothe the spirit of the person in their household who needed them. Writing a grocery list can be an emotional experience if we attend to it, if we’re aware of it. Now, it’s not like we can go around 24/7 having emotions over everything we write. That would be disturbing and inefficient, but everything we write has that potential, and I think we should be cognizant of it, and every so often we should pause and reflect on it. And to the extent that school does not allow room for that, again, is to wall students off from some important aspect of their intellectual, emotional, social, let’s say even economic development, all these things matter, I think, long term, and so we should give students a chance to experience them in their education. Because why not? What else are we doing with our time? If not that?

John: Along those lines, you’d suggest that pretty much everything that ChatGPT does is essentially like hallucinations. There’s a lot of discussion of hallucinations in getting things wrong, but you suggest that essentially everything it does is just predictive text which is devoid of meaning to the system itself, that it’s missing the human context. One of the things I was thinking about is you also had that example where you mentioned cinnamon rolls, and that ChatGPT could write about cinnamon rolls, but it wouldn’t bring up the same sort of connotations that people would have with that. What are students missing when they rely on generative AI to generate writing or to summarize readings that they’ve been asked to do, which is an increasingly common use of AI tools.

John W.: Yeah, the reading, quote, unquote, I’m going to put air quotes around reading when you talk about ChatGPT or large language models “reading,” but that is a burgeoning and worrisome application of this technology. Now it’s not like we should never ask a large language model to summarize a source for us. That may be useful to us, but we have to be very, very cognizant of the difference between what a large language model does and what humans do. When humans read and humans write, we’re having an experience, and that experience is encoded in memory, it involves our senses, it involves sort of a metacognitive process. Large language models do none of that. They are syntax generators. They can’t think, they can’t feel, they can’t communicate with intention. So if we’re going to outsource that to a syntax generator operating on probabilities, we have to be aware of what is being compromised in the exchange. Now, in some cases, that’s fine. If I’m going to go to Wikipedia for something, because I need some kind of background about a source or an idea or a concept, maybe I’m going to go to ChatGPT instead, although the risk of hallucination can make that fraught, because Wikipedia, while it is the product of flawed human beings, has been checked and edited in ways that make it significantly more reliable than a large language model. You guys know this, reading is fodder for writing. So if we’re going to need to write, ultimately write about something, failing to read it leaves me ill equipped to write about it, because the reading itself is where my ideas, my thoughts are generated. It’s sort of ironic that we call this stuff generative AI, because it really is like reading is truly generative, like it creates the conditions under which our thoughts can occur. And so I’m disturbed by some of it. Like, the deep research release that can supposedly put together a PhD level dissertation is BS. I mean, it really is, the only way you believe that is if you ignore what it means to write a dissertation. No offense, I’ve never done a dissertation. I don’t have a PhD. No offense to the dissertation producers out there, we don’t write dissertations solely to be read. In fact, if somebody wants to publish their dissertation outside of an academic artifact, they have to turn it into a book. Dissertations are fundamentally used to demonstrate an extended engagement with a type of thinking and knowledge creation. And so when I see somebody on social media saying, use this to generate your dissertation’s literature review, my top of my head lifts off and my brain orbits around the sun and comes back, because I cannot believe that this is something anybody would tolerate for a knowledge producing exercise and industry, and yet, I have seen real academics recommend these behaviors and the imperatives of speed and efficiency and productivity override our good sense around what is the meaning behind what we’re trying to actually do. And this is my concern, that speed, efficiency, productivity, these are not educational values, yet, they are the things that are often valued inside of higher education institutional frameworks. So all this stuff needs to be examined if we’re going to make truly productive use of this. If it isn’t just going to be like I can now churn out six articles are necessary. I’m going to turn out 600 now, because I’ve got my large language model cranking this up. I just ask folks, “What is the point of all of that extra stuff?” It’s just slop. We don’t need more slop. We need good, reliable, thoughtful human products. And I said this at the top, this is the gift of this technology to help us understand what humans should be doing, but only if we take advantage of the gift. We could squander it. And in a lot of ways, I think we are squandering it, unfortunately.

Rebecca: So along the lines of productivity and efficiency, you do note in your book about automating parts of the writing process, like moving from cursive writing to typing and word processing are beneficial, including other things in different processes, like calculators, GPS systems, et cetera. There’s ways that technology has certainly made our lives more efficient. Can you talk a little bit about how ChatGPT is essentially, fundamentally different than those other kinds of automations?

John W.: I have to admit from the outset that even those quote, unquote good automations have trade offs. I’ve hugely benefited from these automations in my life and career. Talking earlier about my lousy handwriting, I really could not begin to write until I learned to type on a manual typewriter in quote, unquote, summer school in fifth grade, when my parents needed me out of the house and do something and I took a typing class at the junior high, typing island photography in fifth grade. It was great. I loved it, but I really loved typing because that I could do, and I could type so much faster and with more clarity than I could hand write that I now had a way to capture my thoughts roughly at the speed with which they occurred. So it was a great boon, and the noise the manual typewriter made was just a bonus, because it was super fun [LAUGHTER] to hammer away on that thing and then throw the carriage return across. Nobody’s done it. They should. Go find a manual typewriter somewhere and spend some time doing that. It is a good time. That said, like, I can’t handwrite. I can’t write cursive anymore. I recently had to write a paper check for the first time in months, and had to rewrite it three times in order to get it legible so somebody could actually cash my check. So automation is not unalloyed good without trade offs. That said something like automating the creation of human writing on a page from cursive to typewriting, doesn’t really alter the underlying labor of what’s happening. Now we do know that the typewriter and the word processor have changed the nature of writing. The existence of cut and paste as a technology that didn’t involve actually cutting paper and pasting paper changed the type of writing that we produce. So all this stuff is pretty fascinating, if you start to look at it. The analogy that people have drawn between the calculator and ChatGPT, it’s not that it’s entirely unlike that, it’s different from it in some very important ways. The most important difference is that when a calculator does its mechanical calculations versus a human doing the calculations, that underlying act is identical. Now a calculator does it differently. It uses sort of these binary calculation tools to do it while humans are using our brain, but the end result is identical through those different mechanisms, and the calculator has the benefit of ,unless we wrongly input a number, of being more accurate. Now, it doesn’t mean we shouldn’t still teach addition, subtraction and long division and this kind of stuff. We still have to be able to think about numbers in a way that makes them sensible to us. The difference though, is that a calculator is a labor-saving device for something that we would do not as well through our own calculation. ChatGPT, as a syntax generator, is not doing the same thing as human writing. Human writing is thinking, feeling, communicating. So when we outsource the production of text to a syntax generator, we are doing something fundamentally different in the production of that text, and that difference sands away the most interesting features of writing, which is the demonstration of a sort of spiky human unique intelligence, which is the thing that ultimately gets us interested in reading, particular writers write their unique intelligence. The misfortune of school is that we forgot this in how we assign and assess student writing, we’ve asked them to take away their unique intelligence, what they present to us, so the large language model output looks a lot like what gets good grades in school or has gotten good grades in school. But this is just the opportunity of resetting those values and keeping in mind these differences in the underlying labor is important, because again, if we want students to learn certain things, like writing, like critical thinking, like active reading, we have to have them do it. If they don’t do it, they can’t learn it. We learn through experiences, and if we outsource some critical part of that experience to a large language model, we are denying students the experiences to learn it. One of the early applications of large language models, people enthused about it, so I’m going to use it to do a first draft, and then I will edit the draft. This is ass backwards. [LAUGHTER] The first draft is where all the thinking happens. That’s the stuff. And so you can’t edit something using your thoughts that was not produced through thinking. We need to keep all this stuff in mind as we do it. Otherwise, we’re really going to spin our wheels around how this technology can be genuinely useful, rather than simply as a substitute for the stuff that we should be doing for ourselves, particularly in learning contexts. I can’t say this enough, like people write me like, “I’ll use this for the marketing copy I write for my business, and it’s fine.” I’m like, “Great, great. That’s fine, and that’s a great labor-saving device, but you have to remember all of the stuff you, the adult employed human, had to experience and learn, the skills, knowledge, attitudes and habits of mind in order to look at that output and say, ‘This is good,’ that can only be done through writing that cannot be done through becoming a prompt engineer with large language models.” I get heated up about this stuff, I’m sorry. [LAUGHTER]

Rebecca: You’re talking about the sound of the typewriter. Just recently, last week, my mom sent me a little letter that I had written to Santa when I was little, and I had asked for a typewriter. [LAUGHTER]

John W.: We’re gonna sound like a bunch of fogies here, but my work set up at my home office, I have a laptop and I have a large monitor, and because I like two screens, and I actually ordered a special Bluetooth keyboard for my laptop that was designed to purposely make noise, more like an electric than a manual, because a manual that really would be something, it was just a noisy mechanical keyboard. It was very satisfying to use it. I actually broke it because I worked it too hard, and the company went out of business. Maybe it wasn’t a great product, but I loved it. And loved it, and I had it while I was writing the book, and my wife would know when I had a good day by the sounds that had been coming from my office, [LAUGHTER] because it was like, clack, clack, clack, clack, clack and know if I had a bad day if it was mostly silent. I mean, this is real, like the experiences of our lives are our lives, and to the extent that we’ve sanded those experiences out of learning in education contexts is just kind of tragic and unnecessary, has no relationship to how much students learn or the value of their learning. In fact, it’s the opposite. And so I’m a true believer in this stuff.

John: Going back along the same line, I also, throughout my elementary school, got comments about my handwriting, and in fact, that continued into the next generation, because when my son’s teacher, and I think it was fourth or fifth grade, was complaining about his handwriting being illegible, apparently, he brought in something that he found that I had written at some point, and he said, “See, it’s genetic…” [LAUGHTER] but I benefit a lot, too, from those early automations. Getting a typewriter was a tremendous improvement in the ability to express myself more clearly. But then when word processing came out, it changed the way in which I wrote. Before that, I used to have very elaborate outlines and organized things very carefully, because there’s just so much you can do with Wite-Out and with correction tape. And my writing was much more efficient when I had the ability to move things around and to rephrase things and edit it before it finally was produced as a more final version. And I was at Stony Brook at the time, where dissertations had to be done on a typewriter, and I wasn’t about to do this was like 180 pages or so with all these math symbols. So I had a typewriter with a daisy wheel and an RS 232 interface from my computer, and I connected that to my computer to print that on a typewriter. It took 12 hours because I had to keep swapping the daisy wheel every time there were Greek symbols, but I certainly wasn’t going to go through and having to type it page by page, and the ability to edit using Word Processing made it so much more efficient. But as you note, students are not getting the same sort of efficiencies when they rely on ChatGPT and effective writing takes a lot of time and effort and thought and critical thinking, and we’re living in a time where there’s so much fake news out there, or distortions of reality, and so many conspiracy theories and critical thinking skills, I think, are perhaps more important than they have been in the more recent past. How can we help students develop those critical thinking skills when they do rely to such a large extent on generative AI to get summaries of content.

John W.: It’s a challenge. It’s an interesting challenge, and I’m gonna turn listeners on to some fascinating experiments being done by a guy named Mike Caulfield, who is training large language models to become essentially fact checkers, particularly around images, because they’re very good at this, because it can compare image features in ways that humans can’t. But the reason Mike can do this is because he has a career of thinking about the problem of fact checking, of how we think about information and how we check these things. I really recommend people checking out his newsletter, because his experiments are fascinating. Still, the only way to become critical thinkers is to require students to do a lot of thinking. The absence of thinking from large language model outputs should be sufficient warning for us to be wary of letting students use them to bypass parts of the process that may require thinking. Ferreting out misinformation is a practice. In fact, Mike Caulfield has a previous book called Verified, where he talks about a SIFT method for helping assess the veracity of online information. Once you get this method, you can employ it to help you understand the world, and this is sort of my pitch to students around writing, is that if they can develop the writing practices, if they can develop their powers of observation, drawing inferences from observations, and then drawing analytical conclusions from those inferences, they have a significantly increased ability to just go through the world, seeing, reading encountering things, and having that detector that allows them to say this thing doesn’t make sense. But it is something that has to be practiced and experienced to get good at. And writing is a kind of unique way of being able to do this, because it should require students to turn all of those senses on, but again, we have to make them do it when the assignment comes. And then we have to assess the assignment according to criteria that values those experiences. More and more as I wrote the book and as I think about the problems, it’s not so much the assignments themselves, although there are some assignments that are just sort of like we’re gonna have to put those away. Those are too vulnerable to the technology, and they’re too uninteresting to students to ask them to do these things, and so we have to do something else. But if the assignment is intrinsically interesting and engaging and gets students thinking deeply about these things, we then have to use assessment criteria that values that, as opposed to surface level features that so much of student writing is attended to. And some of this also involves convincing students that this is what we’re going to do. The first part of my first-year writing courses was always sort of fraught, because students would want to give me what they’d done before, and they were very conditioned to producing what I called pseudo-academic BS, which was a sort of high falutin fancy-languaged version of what they thought a smart student who had sort of read the assignment would sound like on paper, and they had been conditioned to this performance of good, smart studentness. And one of my rubric criteria was absence of pseudo-academic BS. And as soon as I saw it, I would just turn it back to the student and say, this is pseudo-academic BS, you need to revise, and they hated me for it, absolutely hated me for it, at least at the start. Some at the end also, but at the end, some would come around. But this is a reorienting around the values that we attach to learning, and if students have not been exposed to those values, or if those things had not been valued in their school context before, we have to help them reorient to them and not get overly frustrated, as I sometimes would when they’re not there already. When they enter my class and they’re not doing the stuff I need them to do, I have to remember that in a lot of cases, this was like new, it was unfamiliar. I think I used this analogy in my book, I don’t know if I did or not, but it was like the beagles that had only been raised in a lab and had only experienced like a cage, and then they took them to a park, and they opened the cage doors and say, “Run, be free, beagles,” and the beagles are afraid of the grass because it’s foreign. Maybe I’m going to regret reducing students to beagles. It’s an analogy, people. But the idea that we know the grass is the great thing, and that the beagles who are in the cages are going to love this grass as soon as they just get to experience it, but they might be afraid of it because it’s unfamiliar, and because the system within which they’ve been working has been rewarding these non-grass-like behaviors, they’ve been rewarding staying in the cage. They’ve been rewarding hitting the prescriptions. And so part of my work was always that reorientation, like “It’s okay, come out of the cage. There’s good stuff out here. You might get muddy, you might step in dog crap. There might be something that doesn’t go well, but we’re going to be okay, because ultimately on the other side of this is a genuine experience of learning in life, and you will appreciate this.” Again, no teacher bats 1000 with their students in the course of the semester. And I think, like a lot of instructors, I can remember every horrible end of semester comment a student ever wrote about me. But overall, the batting average was pretty good because students are thirsty with us. They’ve been really wishing for it, and as soon as they’re invited to do it, most students will embrace it pretty thoroughly.

Rebecca: We spent most of our time today talking a lot about students and learning, related to ChatGPT, the ethics around that, but you’ve also discussed several other ethical concerns related to the training and operation of generative AI, including copyright, exploitation of precarious workers, environmental issues, etc. Can you talk a little bit more about these concerns?

John W.: Yeah, it’s important to recall that these models are trained significantly on stolen merchandise, including things I’m willing to bet all of us in this virtual podcast room have had our work sucked into these models and is being regurgitated back to us without notice, without permission, without compensation. It is not impossible to create a system where the origins of these models is compensated. It’s not impossible. It’s incredibly unlikely to happen, though. So it’s not that I think we should sit around being sort of “Ah, whatever that ship has sailed.” I think we actually should keep it in mind every time we use it, that this stuff basically exists because of the use of material without permission and without compensation. Now our judicial system is going to work through this in terms of our actual copyright law, and I think it’s probably going to turn out in favor of tech companies. But I think this is primarily because our copyright law was written at a time where the thought of this sort of technology was not even in science fiction. So we can’t confuse the strict outlines of copyright law with justice and what is right. The environmental impacts of this technology is unknown. It could be incredibly severe. We already know that it’s using a lot of extra energy and a lot of extra water in the data centers that are online in order to train and deploy this technology. There’s a lot of arguments subsequent to me writing the book around like, well, it’s not as much as like watching a single series on Netflix or something like that. These comparisons I find interesting. But again, anytime we’re going to do this sort of stuff, we have to think, what is the cost to this benefit? If ChatGPT really is just kind of a novelty, then should we be using energy and draining aquifers in order to power these servers? And the exploited labor, the labor in third world countries that is used to train this is again, it’s the story of our Westerners’ use of tech over and over again. It’s nothing new, but it’s an intensified version of it, and it’s something we should be mindful of. We should be thinking about the ethical concerns of all technology we use. Like I have an iPhone. It has minerals that were mined under unconscionable conditions. I have Netflix that is using power that is cooling a server that allows that sort of stuff to happen. I still do these things. I also know that one of the biggest contributions to global warming is the beef industry. I still eat. I have to be able to recognize these harms even while I do these things. I don’t require anyone to like put on the hairshirt and repent for existing in modern society. We are where we are, but I think we do have an obligation to understand the truth behind these potential harms around our actions, just to be reasonable human beings in the world, like I’m about to go get on a plane to do some various events, to talk at institutions and promote the book. None of that is necessary. I could do everything over Zoom. I’m going to do it anyway, because I think it actually has a little better impact than it does over Zoom. And it’s an enjoyable part of my life. Being able to visit an institution and talk about these things, or work with faculty around how we adjust to this technology is fulfilling. And so I have to be aware of the balance between the life I want to live and the trade-offs and the costs to society. So it’s really just a matter of being thoughtful about in the same way we have to be thoughtful around student learning when we’re deploying this technology, we have to be thoughtful about these impacts. Otherwise, I think, as the technology develops, as greater demands are made around this technology, both funding, the money these tech companies want to develop something like artificial general intelligence or to deploy this technology in our institutions, often without our permission, without asking. We should be aware of these things so that we can use them as frames for thinking about how we want to use this technology in our lives, as individuals and as educators, and as educational institutions. We should never forget them.

John: So we’ve talked a lot about the challenges and the potential concerns associated with the use of generative AI, are there some uses that may actually benefit students’ learning and maybe even benefit students’ writing, perhaps doing some proofreading using AI tools, or perhaps evaluating a text that the students created to find some weaknesses in their arguments. Might there be some benefit for those types of activities?

John W.: Anything is possible, and there’s a lot of people doing some very interesting experimenting around the intersection of learning using these tools and writing, and I encourage all of that experimenting, even as I am not the person who is going to do it, because I don’t like to think of myself as a fundamentalist, but I do believe that this technology is primarily useful to people with well developed professional practices and to introduce it at times before those things are there, I’m dubious. That said, I’m encouraging of class contexts that are transparent and open… allow students to experiment with the technology as part of developing their practices. I think that’s all good, as long as it’s done mindfully and with important end goals around learning in mind. This is not my practice. I found this stuff increasingly less useful as I worked on the book. I set out to use it as I was working on it as a sort of demonstration to myself and to see what would happen. Over time, it was less and less interesting, because I realized that the predictive outputs of a large language model were not generative to me. I needed that unique intelligence in order to help me do my work. I do have some what I think are hard and fast lines around the use of this technology. If a student wants to sort of curiously and voluntarily engage with this technology for their writing, I think that’s great. I would never personally have a student write a piece for my course that was only going to be assessed and evaluated by a large language model, and that’s a betrayal of the relationship we should be establishing with our students. I don’t know why I would do it. If I’m teaching writing, the most important input I have for my teaching is my students’ writing. So the idea that I’m going to let a large language model grade that instead of me is sort of nonsensical. But even to ask students to use it to get feedback on their writing by my own urging, I don’t know. I mean, this is a probability machine giving feedback on writing. If I’m going to do that, I need to prove to myself that that output is more useful to learning, not to the outcome, not to the grade, but to learning, than, say, peer review or asking a student to read the work out loud to themselves, or do something that I call a reverse outline, which is just a technique for looking at something you’ve written, write a question based outline for each paragraph. What audience question is being answered by this paragraph, and does that reverse outline make sense? I’ve got a million things that students could and should do with their writing that are not interacting with a large language model in order to improve their writing. So personally, I would have to exhaust those before I would turn to them. But I’m not against experimentation. I know folks who are doing that. It’s not for me, as long as the experiment is predicated on learning, not just improving a student’s output for the purpose of a surface-level criteria grade, I’m for it, and I’m eager to see them, and people share them with me, and sometimes I’m taken aback, and I’ll have to eat some of my words and think, “Okay, maybe I still wouldn’t do this, but I get why this person is doing it.” But again, that’s just a mindful teaching practice; that is using technology that’s available to enhance the experiences of their students around learning. That’s the work. So I can’t criticize or gainsay that. When it is a substitute for what teachers should be doing, when it allows for an end run around our own labor, I think it’s dangerous. And I would just say, since instructional faculty are overwhelmingly your audience, I assume, we should be very, very, very, very, very, 12 more verys in there, careful about outsourcing our instructional labor to automation, because it is a encouragement for those who would like to automate faculty labor to do so, and for those of us who have worked at less than fully resourced institutions, we know what this sort of pressure is like. It’s constant, and the decisions that are made often have nothing to do with the quality of learning or instruction. And so if we’re going to allow these things to, like write our syllabi or grade our assignments or give peer student feedback or something like that, I would just say, be careful, because what you may be demonstrating is, for the purposes of the institution, to collect tuition and convey credits and credentials, you may be proving you’re not necessary. Now there’s no way that that process is the same as a good teacher, human interacting with other students, but that’s not a requirement for an institution to give grades and confer credentials, and so we have to be aware of these things. I’m not saying we have to be Luddites ready to destroy the looms owned by the factory, but we should understand the lessons of the Luddites and respect the actions of the Luddites, which were not just to protect their labor, but protect the quality of the goods that were produced for the public. The mass-produced cloths were not as good as what they did. The public ultimately decided mass production was more desirable than employing hand weavers, but we’re talking about education here, and when is mass produced education proved superior to individual, bespoke relationships?

Rebecca: Seems like a good note to move to our last question, [LAUGHTER] which is we always end by asking, what’s next?

John W.: That’s a good question. So whenever I write a book, I promise my wife to not write another book for a year, because, as much as I love writing a book, and I love writing books, because of what we’ve been talking about, like the depth of immersion that I have to do to write a book is just steeply, deeply pleasurable, but it also makes me checked out of my life in other ways. Now, fortunately, this book was written quite quickly relative to my other books, so that was good. But to answer your question, I’ve become increasingly interested in the idea of apprenticeship and how we learn to do what we do. I’m gonna guess both you guys either had formal apprenticeship programs to l earn to teach you had, like, literal mentors, or I, unfortunately didn’t have that. But what I did have was a lot of other people who are teaching around me who I would go bug and say, “This isn’t working. What do you do? [LAUGHTER] What can I do differently? Because here’s what’s happening, and it is very, very unsatisfactory for everybody involved in the equation, what else can we do?” And so I cobbled together a sort of apprenticeship model to help me learn how to teach and reading. I read a bunch of books. I’ll still never forget reading Ken Bain’s book What the Best College Teachers Do, where it was literally like the scales falling from my eyes. I’m like, “Oh my God, there’s people who care about this stuff out there, and I can read about it, and I’m going to start thinking about it too.” It was literally life changing. That book was a form of apprenticeship to better learn how to do the work of teaching. And the reality is, huge swaths of our society and work are based on an apprenticeship model without us recognizing it. And my concern over this technology is we’re going to lose it, either because we decide to outsource the work of the apprentice to the technology in the interests of increased efficiency and lowered cost, or when we ask apprentice workers to do the work, we will not be requiring them to have the kinds of experiences that allow them to become the mentor. If an associate lawyer is writing a brief using large language models, they will be missing out on the kinds of thinking that will make them the mentor lawyer one day. My older brother’s a lawyer, and he’s concerned about this stuff, like “How do I make me?” And what makes him is all the thinking he’s done over the course of his career to do this. So I’m fascinated by that. So I’m thinking of some kind of project around that, maybe not a book, maybe a podcast, so I can tell my wife I’m not writing a book, I’m doing a podcast.[LAUGHTER] But that’s what’s interesting me right now. That’s sort of, how do we learn to be what we are, and what does that entail, and how does this technology threaten those things, and what will happen if we allow it to dissolve those relationships in our workplace?

John: Well, this conversation and your book has provoked an awful lot of thought, and I’ve really enjoyed both listening to you and reading your book, and I’d strongly encourage anyone who is thinking about using generative AI tools to read through your book.

John W.: Oh, it’s my pleasure. I have been a long time listener, and so getting on Tea for Teaching is a career honor that I’ll always be glad for.

Rebecca: Thank you so much.

John W.: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

376. Students as Partners

Faculty members often design and revise courses with limited direct feedback from students. In this episode, Laurel Willingham-McLain and Jacques Safari Mwayaona join us to discuss a program in which faculty work with trained student consultants to improve the student learning experience.  Laurel is a consulting faculty developer at the Center for Teaching and Learning Excellence at Syracuse University. Jacques is a Faculty Development Fellow, also at Syracuse University. Laurel and Jacques both work with the Students Consulting on Teaching program at Syracuse University.

Transcript

John: Faculty members often design and revise courses with limited direct feedback from students. In this episode, we discuss a program in which faculty work with trained student consultants to improve the student learning experience.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Laurel Willingham-McLain and Jacques Safari Mwayaona. Laurel is a consulting faculty developer at the Center for Teaching and Learning Excellence at Syracuse University. Jacques is a Faculty Development Fellow, also at Syracuse University. Laurel and Jacques both work with the Students Consulting on Teaching program at Syracuse University. Welcome, Laurel and Jacques.

Laurel: Thank you.

Jacques: Thank you.

John: Today’s teas are:… Laurel, Are you drinking tea?

Laurel: I am drinking tea and I am drinking Throat Coat with lemon echinacea and honey, because I’ve been congested for about two months. Allergies are great in Maryland.

John: And Jacques?

Jacques: Well, I am drinking today a black tea with lemon lift. And as you can hear, it was a little bit hard for me to kind of pronounce the whole name. There is a fun thing that I found in American culture about food and the naming of the food. Most of the time they will give the full name and even the ingredients in the food that you are getting. And sometimes it will be like, ice cream. There’s so many variety of ice cream I cannot even name, so I really find that’s really, really interesting and really fun.

Rebecca: One of my favorite parts about the podcast is finding out all the interesting things that people are drinking, for sure. Today I have English afternoon, John.

John: And I have a spring cherry green tea today, continuing with the ongoing cherry tea flavor discussion.

Rebecca: Yeah,the tea smells good but tastes terrible, yes. [LAUGHTER] So we’ve invited you here to discuss your work on the Students Consulting on Teaching Program at Syracuse. Can you talk about when this program started and what the goal of the program is?

Laurel: So we piloted this program in the spring of 2023 with just two partnerships, and our goal is to have short-term partnerships between a faculty member and a student consultant who is trained to help faculty gather learner feedback. And so the idea is for the student consultants and the faculty together to imagine small, meaningful changes that they can make right away in their courses. We call it SCOT for short, and I just have to give a shout out. We built on the work of others. We took the name from Ursula Sorensen’s program at Brigham Young University, and we took a lot of input from Adriana Signorini, our dear colleague at UC Merced. And I also just want to say that if anyone’s interested in this work, the Special Interest Group Co-creating and Partnerships through the POD Network is your go to place. Jacques and I are so supported by them.

John: Suppose that a faculty member wishes to get some feedback on their class. Could you walk us through the steps of how this process works?

Laurel: So, the Center for Teaching and Learning excellence advertises that we have partnerships available through our newsletter, through New Faculty Orientation, through word of mouth. And so they get invited, and they register, they sign up at the very beginning of the semester to be a partner. And then Jacques and I meet with them very briefly, and our director, Jessamyn Neuhaus, gives a great shout out of her support. We work with them on the three methods that we’re going to be using, that they can choose from, and what their expectations are in terms of time commitment and their commitment to the student consultant.

Jacques: If I may add something there, I think the part of commitment and know what they are expecting, the expectations, is really, really important part. Because knowing what would be the time they will be giving to the program and what they could expect from student consultants is a very good, important part. Because sometimes you might have faculty, it would be maybe their first time they’re working with student consultants in a professional setting, let’s say, viewing students as collaborators. And it’s really, really important that we have that part of helping them to know what to expect from those students.

Rebecca: You mentioned that there’s three ways that a student consultant can work with a faculty member. Can you talk us through what those three ways are?

Laurel: Sure. So for each of these ways, we expect about three to four hours of faculty commitment over a month in the middle of the semester, and they can choose from among: observation of the class, small group instructional feedback, which we loosely call focus groups, even though it’s not technically a focus group, and early course surveys. For each of these methods, the student consultant meets first with the faculty partner to explore what their teaching interest or issue is and which method might be best. And together, they come to that decision, then they administer it, and then there’s a debriefing meeting, and then there’s a final report written.

Jacques: In our recruitment process for faculty, when we send them an invitation, we kind of propose them already those three methods, and they can choose from them, or they can choose that they’re not sure yet. And when they meet with the student consultant, since the student consultant is already trained to understand those methods and also to know what is the situation or to understand or to grasp the perfect method to use in a particular situation, during the first meeting that student consultant have with their faculty partners, they kind of confront their two views… I shouldn’t say, actually confront… it’s about mutual understanding. Discuss, exchanging ideas, and they come to an agreement which method they should use. So even though the faculty might have chosen to use class visits, for example, in the discussions they can discuss, and they say, okay, yeah, this will be just an early class survey, and they go with early class survey. So it’s always this understanding on where’s the need and try to meet the offer of method that we have for them.

John: How are students selected to participate in this program?

Jacques: Yeah, I can speak to that since I am the one working almost all the time with students, and Laurel dedicates more time to faculty, and I dedicate more time to students, like 70% to students, 20% to faculty for me, 70% to faculty for Laura, and 20% to students. So the way we try to select our students is we have two categories of students. We have the first-timers, those students who come for the first time to the program, and we recruit them through, most of the time, sending invitations or collaborating with other research units around the campus. And yeah, I can shout out to our first-year seminar program, which has sent to us very good candidates. And we also try to use, or maybe have our advocates, which are the past faculty who did this program, or past student consultants who found this program interesting, and they kind of recommended their friend or former student to come to the program. For those first-timers, normally, they go through an interview for us to try to understand, what are their class schedule, and if they are a good fit for the program. And there is a second category of students, which is mostly the returning students, and that is the biggest number that we have, because when students get into this program, they love it, and they stay until they graduate from Syracuse University. So the big number of student consultants are mostly those who are returning from past semester, and they stay and continue working with the program because of the value they get from it. And to date, we have had 24 unique student consultants. When I say unique consultants, I mean that counting once by name, but most of them return. And for those 24 unique students, we have had, like, more than 1000 of enrolled students impacted, which is kind of a ratio of one student consultant for more than 40 enrolled students impacted, which is huge for us.

Rebecca: What kind of training are you providing these students to get them up to speed and ready to work with faculty to give them this consulting capabilities?

Jacques: That’s a great question, actually, because of the diversity of students we get into the program, students come from all over the campus, and so far, we had had like students coming from the seven schools out of the 13 schools that we have here on campus, and most of them are double majors. So there’s a huge diversity of student consultants that we have. So now the training that we use to give them is two to four weeks of training, very compact training, and the training is about the methods that Laurel mentioned earlier, namely the early course feedback, class observation visits, small group instructional design, and we help them to rate, to understand the method, and also to practice. There’s so much role-playing that we do in the program for them to understand how this can work out, and most importantly, is also to train, and help them to shift their view from the relationships that they had with faculty which is mostly vertical, if I can say so, vertical relationship. Now they have to shift into a relationship, a more professional relationship, with faculty that they will be working with. So I think that is one of the most important training that we provide them with, so that when they go into the field, they start talking with the faculty as consultants, not only as students, but also as student and consultant at the same time to provide consulting to faculty.

John: How have faculty responded to this process?

Laurel: It’s a growing interest, largely through word of mouth, and then now that Jessamyn Neuhaus is our director, she is making this a focus of our center, that there will be different kinds of partnerships, of which SCOT (students consulting on teaching) is just one. And so many people are curious, because it’s a very different kind of relationship. It’s different from any other relationship they have at the university. So at first it was a little difficult to recruit, but I think we’re going to have more than we can accommodate soon.

Rebecca: How many faculty have you had participating in a typical year?

Laurel: There’s no typical year. Across the four semesters, we’ve had 32 unique faculty partners in 37 partnerships, and they represent, just like the students, they represent seven of the 13 schools and also a study abroad program. So we piloted with two in spring 2023. In fall 2024 we had the incredible privilege of having a cohort of about nine faculty from Design Studio in the School of Architecture, and together they debriefed, and together they planned on how to get student feedback. And that was really exciting. And then in spring 2024 we had eight separate partnerships, not a cohort model. And this fall, in 2024 we’ve had 16. So it has varied.

John: Participating in this as a student consultant takes a fair amount of time. Is there some type of compensation provided to the students who serve in this consulting role?

Jacques: Yes, actually, this is a paid position for student consultants, and there’s actually a good budget that is being allocated to this program. I think the university, and mostly we should be grateful to our new director who’s really working to expand this program, and the value that the university is giving to this program. We think the budget will continue to grow and we will be able to accommodate more students. And in addition to that financial or monetary benefit they get, there’s a huge professional experience they get from this, which is more than money. Many of them actually have reported, actually, growth professionally from when they got into this program, and when they leave the program. They’ve grown a lot, and that can be even seen in the reports that they do when they finish this program. It is really a very short program, but it’s really powerful in terms of the skills that it provides students in this small period of time and empowering the students to believe in themselves and what they can provide. And also so many students who have this consulting component in their programs, and when they come into the SCOT program, they kind of practice, they kind of learn or grow those skills that they need and put them into the consulting work. I see the students coming, for example, from human services, from business, from marketing, data analytics, or public policy and other fields who really need this consulting part of their training. And when they come into this program, they love it, because they start practicing, they start seeing faculty who are their “client”, and we can see how they interact with them. So this is kind of a push for them to practice the skills and already to be into the professional enivronment. So it’s not only finance, it’s also those skills that they get. Moreover, that also, I think, is really important to mention about those students who are former student consultants, who got good admission into good graduate schools after this program. Because I guess showing that you’ve been practicing your skills while you are still at university, kind of show how committed you are to your growth into academia and professional world. So yeah, that is also another important part that the students gain from this program.

Rebecca: One of the things that each of the student consultants is doing, no matter what method they’re doing, is some data collection. They’re either observing, they’re running focus group like activities, or they’re doing surveys, and then they’re consolidating that information into a report and providing that back to the faculty and debriefing. Is there some bigger data that’s being collected about the program itself, to measure the impact of the program?

Laurel: You know your timing is just beautiful, Rebecca, because I am elbow deep in data, in findings, in information from our people. It’s very emotional for me because of what they’re saying. So we keep track of the numbers. You saw that. 37 partnerships, 32 faculty, 24 unique student consultants and over 1000 enrolled students impacted, but most importantly, we gather the stories and the experiences. And so with the student consultants, we had a three-step process, a survey that they responded to individually, Jacques and I lifted out themes and places where we needed clarification and met with them in their weekly student consultant meeting to look at things that weren’t clear to us. For example, how Teams is organized, so it’s clear to them where to put their tracking and documentation from their consultations. Another one was what their final report should look like. And so we got clarification on two or three or four topics from the individual survey, and that was the program-level view. What should we keep? What should we change? How’s the onboarding? How’s the middle section? How’s the support? And then Jacques and I met individually with each of the 11 student consultants for about 15, 20 minutes to find out what the experience was like for them, how it fits their future goals, how we can be better in supporting them and who they recommend. And so Jacques already said that they emphasize communication skills, that just immediately comes to their lips when you ask, what has this program provided for you in terms of your personal growth. And then professionalism. Like he said, the opportunity to consult, getting a window into the life of faculty, which helps them as students in their own courses, as well. Collaboration and also this semester, a new thing was that their relationship with one another changed. So we had two different groups come into the SCOT program that were already very close friends, and they, on their own, brought up the fact that they see each other differently now. They’re not just football watching buddies or sorority sisters at breakfast together, but now they go to each other for professional consulting and mentoring for one another, like, how shall I solve this communication problem with Professor Jones? What’s a question that would get at this kind of issue, and that was exciting for us to see their relationships with one another growing. And then I could talk about faculty. So we debrief the faculty in pairs, again, 15, 20 minutes online, and we do that so they can learn from each other. So many times over the past few years, we’ve heard faculty say, “Oh, I thought I was the only one who was dealing with this issue with my students”, or “I was really curious to know what my student consultant partner was doing with another faculty member,” and they embrace the importance of the student perspective. For example, two or three of them talked about how they thought their enrolled students weren’t engaged in class because they were bored and they found out that they were nervous. There were things that they needed to do to prepare the soil for engagement. Another person found out that her group of students had been away on study abroad since the prerequisite course, and so they’d forgotten and so they weren’t ready to engage in the class in a way that would show that they had forgotten the basics. And so now she’s incorporating some basic review into that. A second way is that they can be vulnerable with a student in a way that they can’t be vulnerable with their faculty colleagues. And one woman even said, “You know, we tend to complain as faculty colleagues, but then I have this student consultant who’s here to help me and to affirm my work, and is in solidarity with me. And we don’t complain. We just work on getting things better,” that the student consultants help the faculty partners think of survey or small focus-group questions that they would never have thought of on their own, so they really collaborate in that. And then two or three people this semester talked about how they’d been wanting to make changes in their teaching, but they just never made time for it, but because they had this really short timeline of meeting, working, meeting, and report, they got stuff done that they’d never gotten done before, and now they have homework. One woman said “my slides need to be so much better,” and another faculty member said “my assignments need to be better.” And so I am going to find out if it would be okay this next semester for them to start at the very beginning of the semester and work on those two issues that they had talked about with previous student consultants. Two people had students look at their online materials, and one person gave us this beautiful quote that “their student consultant served, not only as a sounding board, but as someone that they could spell out their thoughts and doubts and questions too, but also a link between them and online learning.” For example, she found out, and this probably won’t surprise a lot of people, but our students don’t really read their emails. They don’t prioritize them, and in fact, if you’re working with a faculty member, you need to. So the student consultant helped the faculty partner be able to communicate with the enrolled students about how important communication would happen in that class. And then finally… you can tell, I’ve just been reading these… finally, they get a much stronger response rate on the early course surveys, and they can ask their own specific questions. And we have one faculty member and student consultant who have institutional review board approval to write about their project and to ask very specific questions about teaching techniques. So those are just some of the ways it’s been fruitful for faculty, and I’m excited to share those hot off the press findings with you.

Jacques: Just to add an idea about the exciting parts of the learning between faculty and students, I think Laurel mentioned some where the student could open a window into the faculty instructor’s world, and that really made a huge difference in the lives of the students. So the student understood how much work the instructors and everyone working to help them, put into the preparation of material, put into trying to help students learn. So even the worries that faculty have when they see their students not doing well, and although that actually helps students to understand how much faculty care about students. In the results, we saw some student consultants considering becoming themselves instructors or faculty in the future, which is very unusual for students who are still undergrad or the students studying at the university, to start thinking already how they could become faculty themselves.

John: You’ve already talked a little bit about plans to perhaps expand the program to reach more faculty and more students. Do you have any other plans in terms of future directions of this program?

Jacques: Yes, actually, with our new director, we have been discussing this a lot, and we are planning to expand more this program, making it efficient when we’re expanding, because when we expand, there will be so many challenges coming into the program. And one way we are thinking of making this program efficient is about time management. Since this is an early course mini course, one way of seeing what’s going on and see how we could improve the learning of the students. Time has always been a challenge, so starting as early as possible can help faculty to know what’s going on in the classroom and implement the results of the student consultant will be communicating to them. So this semester, and next semester, we’ll be focusing on how we can make this training efficient. And instead of having that two to four weeks training while the semester has already started, we are trying to move that training online within an asynchronous program, so that the students who are recruited this semester can start already training themselves during the winter break or summer so that they can start working when the semester starts. So when they finish the training, they can start working with faculty on day one, the first week of the semester, they can start already interacting with faculty. This semester, we are really blessed to have our student consultants coming back. So the student consultants who are in the program, most of them are staying because they want to do again next semester. So we are blessed because we’ll be using their expertise to go through the training that we have moved online. They can go through it, give us feedback on how we could improve it and make it better for the next semester. So this semester, the point that we are focusing on is time management, and we are moving all the training into the break period before the semester starts, and we have students who will be going through the program and test and give us the feedback as quickly as possible.

Rebecca: Sounds like some interesting developments. For sure, we always wrap up by asking: “What’s next?”

Jacques: Well, that’s a good question. Actually, it’s kind of reflecting on what we have achieved ourselves. And going through this program has been actually eye opening for me to see the impact of one student consultant impacting forty enrolled students, that’s huge. That’s a big impact that we are getting there. And I’ve been thinking a lot on how I could replicate the same experience somewhere else in so many other environments internationally as well. I think this model is mostly in the US and UK, and somehow in some countries in Europe, but there’s so many other countries that need this kind of model to change and bring innovation into their teaching and learning. And I’ve been thinking a lot about creating an NGO that would be really providing this service to other universities around the world, in Africa or Asia or somewhere else, and to implement this model, because it’s not everyone who have all the resources that we are blessed to have here at Syracuse University. So extending and providing help to other universities or professional training services. it’s not really only limited to university, it can be K-12, it can be professional training, and help them to implement this model. I think that will increase the impact, and that’s what I’m really thinking hard on doing next.

Rebecca: Sounds like some big plans for sure.

SAFARI: Yes, I think it’s really big [LAUGHTER] I still have to figure out how to plan this and make it happen.

Laurel: My plans are smaller [LAUGHTER] in scale, although I hope to be a part of whatever it is that Jacques does in the future. For many years, over a decade, another person who has directed the Center for Teaching and Learning and I have felt called to offer small retreats for women leaders. And we’re talking about the everyday kind of women leaders who teach and coordinate and manage and mentor. And so we recently met again and said, “Are we ever going to do this?” And I think we are, and they are small in the sense of just a few people, maybe in a house with catered food and lots of space for napping. Because if you don’t have napping in a retreat, it’s not a retreat. You have to have good food, idyllic place, various senses used: taste and smell and sight and so forth, and napping availability. And so just to make that kind of space and to bring some of our leadership expertise to bear, but mostly to help women reflect on their identity as leaders, their expertise, their calling, and just to deepen their understanding of their own vocation. So that’s my dream. I don’t know. We’ll see if it comes.

Rebecca: I like the true dreams in this “what’s next” for sure.

John: Thank you both for joining us, and we appreciate the information you’ve shared.

Rebecca: Well, thank you so much for joining us. We appreciate it.

SAFARI: Thank you.

Laurel: It’s been a delight.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

368. Pedagogy for Large Classes

Building a sense of community is an important component of inclusive teaching. In this episode, Anna Logan and Ann Marie Farrell join us to discuss strategies that can be used to cultivate belonging and increase student engagement in large classes. Anna is an Associate Professor in the School of Inclusive and Special Education and the former Dean for Teaching and Learning at the Institute of Education, Dublin City University. Ann Marie is an Assistant Professor, also in the School of Inclusive and Special Education at Dublin City University. Anna and Ann Marie are both recipients of multiple teaching awards. Since 2018, they have been running an annual Pedagogy for Higher Education Large-classes Symposium.

Show Notes

Transcript

John: Building a sense of community is an important component of inclusive teaching. In this episode, we discuss strategies that can be used to cultivate belonging and increase student engagement in large classes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Anna Logan and Ann Marie Farrell. Anna is an Associate Professor in the School of Inclusive and Special Education and the former Dean for Teaching and Learning at the Institute of Education, Dublin City University. Ann Marie is an Assistant Professor, also in the School of Inclusive and Special Education at Dublin City University. Anna and Ann Marie are both recipients of multiple teaching awards. Since 2018, they have been running an annual Pedagogy for Higher Education Large-Classes Symposium. Welcome Anna and Anne Marie.

Anna: Thank you.

Ann Marie: Thank you.

John: Today’s teas are: … Anna, are you drinking any tea?

Anna: I am. Generally I drink Irish Breakfast tea. But on Earth, this occasion, I got something special. I’ve never had it before. It’s called Emperors Clouds and Mist, so it’s a green tea, I don’t know. Hopefully it’ll work out alright. And I’m having it in a keep cup, a glass keep cup.

Ann Marie: Well, I did not go down the fancy route, so I am drinking Barry’s tea with, I don’t know what type of actual tea it is, except that it’s Barry’s tea, and it is a staple of pretty much every cupboard in every kitchen in Ireland. So it’s standard ordinary tea.

John: And we have some of that in our office, actually. We have a nice variety there.

Rebecca: On this occasion, I got some Irish Breakfast tea. [LAUGHTER]

John: And I should have, [LAUGHTER] but I am drinking a ginger peach green tea.

Rebecca: A good favorite of John’s, of course. So we invited you here today to discuss your work related to large-class instruction. Could you each tell us a little bit about your background and interest in large-class instruction?

Ann Marie: Well, I suppose we’re both teacher educators. All of our students are either student teachers or are qualified teachers who come back to do some of our postgraduate courses. But particularly in relation to student teachers, some of our classes are quite large, and we were anxious, I suppose, to model good teaching practices in that context, especially as our students will themselves be teachers. So a number of years ago, we began to look at our work in the large-class context, particularly on a four-year Bachelor of Education primary teaching program, which, in each year group, has over 400 students, and we were teaching all 400 together in a large tiered lecture theater. So that was, I suppose, my interest in that area.

Anna: And I suppose, just to add to that, for many years I’ve taught a small number of classes that are in a large cohort, that kind of 400-plus size, and always enjoyed it, always got a buzz from the experience. And then, I think a couple of years ago, I was one day teaching that large cohort at nine o’clock in the morning in that tiered lecture theater, a quick 50-minute lecture. And later that day, I was teaching a group of six doctorate in education students, and I remember being really struck by the kind of very differential model of resourcing there: these six, eight doctoral students, really experienced practitioners, getting a lot of support and so on, versus perhaps that introductory class. But then a number of years ago, Ann Marie, who we go back a long, long way, invited me also to co-teach some of her sessions with the large cohort. And I really enjoyed that work so much, so I’d already enjoyed the dynamic of working with the large group. I had questions about it, about the equity issues associated with it. We might come back to those a little bit later. And also, I just really enjoyed working with Anne Marie and the dynamic of working hard with that large group.

Ann Marie: The pupil-teacher ratio was still one to 225.

Anna: We go back a long, long way.

Ann Marie: We were in college together with our primary degree, and then 10 years later, just totally by chance, we ended up back in college together on a postgraduate diploma course. So we go back to the 1980s as students, and then later as colleagues. [LAUGHTER]

Anna: And we’re still talking to one another, which is good. Always helps. [LAUGHTER]

John: That’s wonderful. Can you tell us about your Pedagogy for Higher Education Large Classes project?

Ann Marie: Yeah, I suppose I can tell you a little bit about that. The Pedagogy for Higher Education Large-Class symposium (or PHELC) symposium, began really as a one-off event. We were invited to run a pre-conference workshop at a wonderful conference held annually in Valencia every year called the Higher Education Advances Conference, or the HEAD Conference. So in 2019 we ran that pre-conference workshop face to face, and again, a pre-conference workshop for the same conference in 2020 online when the pandemic hit. And in the course of that, we set up a website. I suppose, we set up a model for the symposium that then became an event in its own right. So we’ve run it now for six years, and we’ve attracted people both online… the last five have been entirely online… and we’ve attracted people from all over the world who are engaging with very large-class numbers and large-class cohorts, albeit in different contexts. But many of the challenges, and more importantly, many of the possibilities that have been experienced by those higher education teachers, is our focus at that symposium. So we look for papers based on research, but also based on reflective practice, with the aim of reducing the fear and angst around large-class teaching and giving people who are doing some really good work in that area, a platform to share that work with others. So we also set up a website, and most of the presentations and all of the papers, we asked presenters to write an accompanying paper along with their presentation. They are all available on phelc.ie, PHELC.ie, for anyone to look at. The Symposium is free, we usually get funding here in Ireland, and I suppose our main aim is just to enable access to anyone who wishes to participate, either to listen or to present, that there’s no barriers. So hence the online, and hence no conference fee for it.

Rebecca: But Anne Marie, you said it started as a one off, I think it’s still going.

Ann Marie: Oh, it’s still going. We’ve done six [LAUGHTER] and our initial intention was just to do the pre-conference workshop in Valencia in 2019 but it became evident that there was an interest in that area. The Conference Committee invited us back again the following year to do a second workshop on this topic, but we just got a lot of very good feedback from participants. We also had put in a huge amount of work into developing the website, developing the workshop or the symposium, and we love large-class teaching. We had done a lot of work in that area ourselves, and it just allowed for our pedagogical practice and that to be mapped onto the scholarship of teaching and learning, and to give people an opportunity to engage in scholarly discussions, be it at the symposium or in terms of the papers that they submit, that they could align their teaching practice along with their academic writing, researcher-type practice. So we’re still going. Now we are at a crossroads, Anna and I were just wondering if we will continue with our annual symposium, or maybe we might move to a symposium, perhaps that maybe every second year, and broaden out maybe to do some more focused publication and research, perhaps every other year, we’re not sure. So we’re due to carve out a day or two to work out what we’re going to do going forward, but we will be doing something going forward.

Anna: And we were always committed, John, to open education resources, so all the publications are available as OERs and so on. So again, I think reflecting Anne Marie’s observation, we tried face to face, we tried HyFlex, we tried online, and what we’re really interested in kind of equity and making sure there’s no barriers. So we have had people from all across the world, and we’re committed to the resources we produce being available to everybody.

John: Excellent, and we’ll include a link to the website in the show notes, so if people would like to check out those papers they can do so. One of the things we’ve noticed, in the U.S., at least, is that as we brought in more first-gen students and more low-income students, we see increased DFW rates and one concern that a lot of people have with large classes is that it might be easier for students to get lost. What’s your perspective on addressing that issue of students feeling disconnected from their class and their instructor, when they’re in such a large-class environment, and I teach a very large-class as well.

Anna: Yeah, that’s a really interesting question, John. And just thinking a little bit about it, and it’s interesting, it was actually some colleagues that I met on my sabbatical where I first heard that term “weed out,” because it’s not a term that would be much used, in the Irish context anyway. On the one hand, you know, looking at the data, I suspect that that notion of attrition or non-progression or drop out is potentially maybe slightly lower in the Irish context. The latest figures coming from our higher education authority are putting us at about 85% progression from first year, which is, I think, pretty good. That said, it’s interesting that the rates of progression are considerably lower among male students, but also among students who come from socio-economically disadvantaged groups. And coming back to your question and the question of relationships. I suspect that that failing to build relationships, or build connections with classmates or with lecturers or to feel that they belong in the community, whether it’s large or small, is very likely to be a significant factor in that dropout. Again, I think interestingly, it reminds us when we look at that, and it certainly was something I know we discussed at our most recent PHELC Symposium the importance of the relationships when teaching at scale, and that not making any assumptions. Sometimes, maybe as lecturers, we assume that students are well connected, that they have a good network. That’s very often not the case. So while, as I say, the data might not be quite as strong and we don’t have maybe that concept of a “weed out class,” I think it still doesn’t preclude the fact that a concern to foster a community that is relationship rich is important, regardless of the context.

John: I noticed in last summer’s PHELC symposium, there was a paper that you presented Ann Marie with Peter Felton on relationship-rich education in large-classes. Could you talk just a little bit about that?

Ann Marie: I suppose myself and Peter, in his context, would look at very large classes from the perspective of building a relationship, that it is possible to build a relationship, whereas sometimes, I think it can feel, sometimes, to some higher education teachers, that it’s almost impossible to have a relationship but you don’t know anyone’s names, or you might know very few names. But you can build a relationship with a really big class in different ways, even though you don’t know their name: very simple things like smile when people are walking in the room, say hi as they’re passing you heading up to the seat, have a wander around the room, don’t stay stuck down at the bottom, move. So like when I’m in a class, I share my proximity. I wear a headset. I move away from the podium. I’m up the back while I’m speaking, looking over the shoulders to make sure they’re not booking flights or watching YouTube while I’m chatting. And we use different teaching approaches and strategies in that large-class context to mix up how we relate with the students like Anna and I co-teaching. So when we co-teach, we tend to use a workshop format for at least some part of that session, which then allows myself and Anna to walk up the steps of the room. We chat to small groups. So we literally, physically bring ourselves closer to the students. You can have a bit of fun in class. If you do get to know some of the students, you can invite those students to help you change the dynamic, the colder dynamic, to make it warmer, then we can talk about that in a minute, in terms of chatting with Class Reps and that sort of thing. So I suppose in the first instance, it being the potential of having a relationship, like, if you take it completely out of teaching, really huge groups like Coldplay, Robbie Williams have a massive relationship with 10s of 1000s of people standing in front of them, and they don’t know any of them, but there is a feeling, there’s an empathy generated in that really large context. And I’m not saying that we have to be Robbie Williams or Chris Martin at all. That’s not what I’m saying, but it’s just really in the teacher’s perception of what is possible there. And I know with my work with large-classes, they are my favorite. My final-year students, I’d say I know names of 10 of them. There are 430 in the room. I love that class because there is such a two-way flow in communication in the room and between sessions on back channels. It could be something like we have, in the Moodle we use here, an in-built chat function, then a module page on Moodle, but something like Padlet where students can ask questions anonymously without having to put their hands up in a really big group, allows to build up that relationship as well. And it doesn’t just mean you answer the questions in class if they ask them in class, but it also allows them a way to reach you between classes, so if something might occur to them afterwards. So I would check that kind of a back channel between the sessions, and whether I answer the questions in class or not, I answer them in writing, one way or another, on that back channel. And you don’t even need technology in class. If people are a bit reluctant to speak, you can use a scrap of paper or post it, or something like that. It’s just to get the student voice and to capture that whatever way you can, so that you’re capturing their voice and their feeling that you’re listening and that they are engaging at that level in the class. So all of those sorts of strategies build up a teacher-student relationship. The way you would in small class of 20 may be a bit more naturally, but you just need to think about it in a different way. That’s about perception, around how you perceive or what you perceive as a possibility in that very large context. And it means that, in my own case, when I look at like I’m currently, this semester, working with 430 final-year Bachelor of Education student teachers, and they’re my favorite class. I love them, and I feel I have a relationship with them, and I feel I have a good relationship with them, even though, out of the 430 I probably could name 10 people outside, but I feel I have a relationship with them in spite of that. But just I think it is how we as teachers perceive numbers and what’s big. What does a large-class mean, actually? So for me, I think 400 is large, and then I visited somebody working in Sweden, who has over 600 in his classes. So mine suddenly seems very little compared to his.

Rebecca: I think your connection of using performance or concerts or theater as an example of where you can feel connection to performers is a good example of how students can feel connected to their faculty in these large settings.

Anna: Ann Marie has mentioned, and she’s right, that it’s about our perceptions of size and what’s possible and the possibilities. There’s also a little bit of it that is about students’ expectations and students’ perceptions, and sometimes a lot of our large-class teaching is in your traditional tier lecture theater with a podium at the front. And it’s no accident that Ann Marie has a great relationship with her final-year students, because she works with them in that large group in first year. But it’s also, I think, about over time, building their expectation that they are valued, that, as Ann Marie said, simple things like smiling, that there’s an expectation that we want to have a relationship with them, that we value that, that it isn’t that disembodied voice that’s standing at the podium and lecturing at them, that it is about relationships and it is about community and assessment is another way in which we can set that up, too.

Rebecca: We’ve talked a little bit about the differences in how you might build relationships in large versus small settings. And Anna, you also talked about the difference in your experience of going from a big class to a small, intimate doctoral setting. [LAUGHTER] Can you talk a little bit more about, from the perspective of faculty, the difference in instruction between teaching these large classes and teaching small classes?

Anna: Yeah, again, I think it is, apart from thinking about the equity challenges and the message that that conveys, sometimes, some of the literature talks about information transmission approaches in a large class, versus dialogic teaching and exploration and so on. And it sometimes strikes me, and having worked closely with Ann Marie over the years as well, that actually the difference in the large class, as Ann Marie said several times, it’s about kind of not just ruling anything out, but thinking, well, if I do want to create a relationship, or I do want to elicit learners voice in that context, how am I going to do it? Rather than just saying, “Well, it’s just not possible, and it has to be that information transmission piece.” For me, it strikes me that in order to do it well, the instructor, the educator, has to be much more deliberate, much more careful in their planning, not just thinking about their learning content, but thinking about the things we’ve spoken about, like the relationships, like, how am I going to elicit students voice? How am I going to build in formative feedback? Because it’s not, perhaps, as natural as in that context, where I was working with six or eight doctoral students who were coming to the session with me, having set prior reading and that they were anticipating and fully expecting. Their expectation was that was going to be dialog, lots of exchange of views, and that’s why I think two little things can make a difference that break down that barrier in the large-class, in terms of setting our expectations of what’s possible, for not just for ourselves as higher education teachers, but also for our students as well.

John: In large-classes, students are often somewhat reluctant to volunteer responses or to ask questions. You’ve already addressed that a little bit in terms of having a backchannel. Are there any other strategies that you use to help students become more comfortable participating in larger classes?

Ann Marie: Yes, so I usually, when I have a large-class, email the Class Rep prior to the module beginning, and I try to meet them, either online or face to face for 10 or 15 minutes, at least for the first couple of weeks. And when I meet them, I request that they will help me to teach in the first couple of weeks by asking and answering questions in class. In most cases, they kind of step back a bit in horror and because they don’t like speaking in the large-class context at all. And these are Class Reps who have put themselves out there anyway, but they’re still not comfortable in that context. So what I usually do is I go through the forthcoming session, I tell them the questions I’m going to ask, and I give them potential answers to those questions. And then for other parts of the session, I give them questions to ask me, and then I get one student to monitor the back channel so that I’m not reading out the questions that have been posted. It’s a student reading it out, one of the reps as well. And I have a little mic called a CatchBox that’s in a foam dice type item. And it’s a microphone that can be thrown around the room like a foam football, and it allows the students to speak in their normal voices without need to raise them. So the reps that I have set up then models answering and asking questions. They model the use of the CatchBox. And even in the first session, you find that some people that I have not planted who are on the edge of courage tip over and have the courage to ask the questions they’re dying to ask. And by the time they get to the third session, I don’t need to set up anyone anymore. There will always be a few people who will ask questions, and the fact that the student is monitoring the Padlet or the backchannel of the typed and posted questions by default, means you get student voice as well. So I have found that a really useful way of gathering the student voice, and also, if you do set a task in the class, or if Anna and I are co-teaching in the workshop, because they’ve spoken, we do something like a think-pair-share where they’ve chatted one or two others, they have a bit more courage to respond as well, because they’ve already sussed out the response. They’ve already sussed out the question. They’ve teased out a couple of things between themselves, plus we’re walking around the room answering questions. So sometimes when that happens, I might be chatting to a group, two or three, during the task, and we have a little chat, and then I say, “Right, I’m going to come back to you now with the CatchBox, and I want you to say the whole class exactly what you’ve just said to me,” and again, you get a little worried look, but one of them will do it. So again, I think Anna, what you said there is right, the very deliberate actions in that big setting really help to gather the student voice.

Rebecca: You’ve talked a bit about student reps. Can you talk a little bit more about what they are and how you’ve designated them,

Ann Marie: They either volunteer or sometimes there might be a vote amongst the student body. So every year group, on every program, has student representatives, and the number of representatives vary depending on the size of the year group. So in a year group with 400 students, there are a minimum of six class representatives, and they’re volunteers. They’re linked with the students union, and their role is, I suppose, as a liaison or a communication between the union, between the Program Chair, and the staff of the program, and the students. So they’re a conduit for information between staff and students and between the university and students.

John: You mentioned using a CatchBox, and I’ve always wanted to use one. I saw them at a conference in SUNY Oneonta a while back. Do you toss it to specific students, or do you just toss it in the crowd to get responses?

Ann Marie: So the first time we bought it, I was delighted. I walked into the large class, and I asked a question, and I threw it in to the middle of the crowd, and they threw it back at me, [LAUGHTER] a few times, [LAUGHTER] so I never did that again. [LAUGHTER] So that is why I model it, that is why I set up the first couple of sessions. So the students I choose, or the rest, or whoever’s helping me, they scatter through the room, and then I make it very clear that when I ask a question, they have to put their hand up high so I can see them. And then I will throw the Catchbox to them. They throw it back to me. If I’m on my own, if I’m the only teacher in the room, with two aisles, I’ll stand on one aisle, and I might ask a random student to stand on the other aisle, and I’ll throw it across to that student if I’m trying to reach somebody on the far side. So in my experience, I would caution against just randomly throwing it into the group, because people are very happy to use you as target practice. It doesn’t hurt if it hits you on the head, by the way. [LAUGHTER]

Anna: In those large-classes, we tend to use a headset mic because we want to do all that moving around the room in terms of building relationships. And when we first started that, before we had the Catchbox, we used to have hand-held microphones, the ones that move around. But I think we’re all the same. We tried to put that up to a student who we know has something wonderful to add from the work, but they all recoiled in horror, but I do think they enjoy the Catchbox. But again, as Ann Marie said, it’s about being deliberate and setting it up. That’s why it works.

Rebecca: So you both talked a lot about co-teaching and working with each other. Can you talk a little bit more about the benefits of this practice?

Anna: Co-teaching is interesting because you mentioned at the beginning, we’re both in the School of Inclusive and Special Education. And I know the language about university structures is different, but basically we’re in a department of 18 academics, and we’re all focused on inclusive and special education. And I know it’s the same in the United States and in many countries in the world, but one of the models of support for children in elementary and high schools, for example, that is very often used as a way to support more inclusive approaches, rather than withdrawing children from their mainstream classroom, is co-teaching. So the concept of co-teaching, as in, let’s define it, let’s use a standard definition of two or more adults in a single physical space delivering instruction. So that’s the definition of it that is used in the compulsory education sector, in elementary schools and high schools. We’ve looked a bit at the literature. The definition of what constitutes co-teaching is not at all as clear, it seems to us, in higher education, because co-teaching can be used just to describe a number of faculty members, sequentially teaching separately, within a module or within a course, for example. But, I suppose, coming back, co-teaching is an important concept. It’s kind of a threshold concept as part of looking at inclusion for a lot of our work with undergraduate students in those large courses and also larger courses. So a few years ago, I would have been responsible for teaching content in relation to working with special needs assistance, but also delivering some input on co-teaching. So it’s a concept that we want our students to grapple with, but we also enjoyed teaching the large class. So in the interest of modeling, walking the walk, talking the talk, Ann Marie invited me into the classroom to model that co-teaching. That was a really interesting experience for us. We did a little bit of research about it. We got students’ perspectives, but we also brought in some observers to observe us co-teaching. We know the literature about co-teaching, that for it to work well, it has to be very carefully planned. And we did plan. Initially, Ann Marie, I think we spent a lot of time really thinking, mapping out that 50-minute session. Typically, we’re talking about a 50-minute session, later we did longer sessions, but we really mapped it out, and what exactly I would be doing at a particular point in the lecture, what Ann Marie would be doing, what students would be doing. And we also thought about the kind of different models of co-teaching, because literature talks about lots of different models of co-teaching. So we thought about those, and what were we actually teaching? The really interesting thing was that the observers told us we needed to be more explicit, because, again, it’s no more than that kind of notion of maybe students come into the university and come into that tiered lecture theater, and they think about it like a theater, for example, and they think, “Well, I’m here to be passive and and I’m then going to get these wonderful thoughts from the professor on the podium at the front of the stage. So they also seem to be, in some instances, a little bit thrown by the fact that… Ann Marie, you might talk a bit more about that… about the two of us being in the room. It kind of threw them out of their comfort zone in terms of what was the expected model. So our observers noted that and reminded us that we needed to be explicit. This is why we’re modeling this. We’re modeling this because we think it’s really important for you to understand this concept and how it works, because you’re going to be a teacher. That’s taken me well away from the original question, which was about why we were using it. We were using it because we wanted to replicate workshop type activities that we would typically engage in with a much smaller group. And one of the supports for doing that was to have more than one teacher in the room.

Ann Marie: Anna said one of our reasons for using it was to model the process of co-teaching, because our students are going to be teachers. But other people have used co-teaching to model, let’s say, social workers. I can’t remember the author, but there is a paper on social work education and social workers using that co-teaching model to model different perspectives around a case conference or something like that. So it also enables, if you’re a higher education teacher, engaging in something that’s a contested concept or a contested idea. You can use the co-teaching model to show that contestation, because the two people are very likely to be coming into the room with perhaps different views and different insights. And you can model that carefully in that context. The most important thing, I think, around the co-teaching, is that you’re fairly comfortable with the person you’re co-teaching with, and that it is carefully planned. It’s a fair bit of work when you first do it, but once the planning is done for a session, like Anna and I started this about 14, 15, years ago, and where we have continued to co-teach for subsequent years, the work’s done. We know what we’re doing, but it was a heavy investment of time the first year we did it, but after that, it’s worth it, absolutely worth it, because it enabled us to do workshops with very large groups that I would have struggled to do on my own.

John: And it can provide a really rich experience for students. When I was a college student, two of my most memorable classes were team taught by two instructors. One was on bioethics, where there was a biologist and a philosopher, and the other one was on the philosophy of social sciences, where there was a philosopher and a sociologist teaching together with very, very different perspectives on all of it. And the ability to go back and forth and present different perspectives is really healthy. You had mentioned, though, doing some work with think-pair-share and some group activities. Could you talk more about what other type of small group activities you do in your large-classes?

Ann Marie: So usually what we do in those classes, it depends on the workshop, or it depends on the focus. So for example, on Monday, a couple of days ago, myself and another one of my colleagues, co-taught a workshop with the ED 4 group, and the focus of the workshop was on their assignments. I use universal design for learning in my own pedagogical practice with these students to model how to use it in teaching and learning, and part of the Universal Design is in the assignments. So I give them an awful lot of choice in terms of topics. I use authentic assessment. I’ve asked them to write for an audience that is not me. They choose their audience and so on. But they’re also allowed to work with who they want and how many they want, between two and five. Some people are very comfortable working in small groups, others are very happy to work with four others or whatever, but I give them all those choices. And then last month, the workshop we ran was for one and a half hours on planning for that assignment. So my objective was they walked out of there at the end of that session, before our reading week next week, with a plan of action to tackle that assignment. So the way that was structured was, I suppose, almost like a dance between myself and themselves. So I brought them through each criterion, they were supported by a writing frame that I gave per group, and I asked them to complete each step of the plan for their assignment based on the criteria and the expectation of the structure of that so that’s an example of a type of in-class task where they work together, in that case, choosing who to work with. And we stepped it so that we have six steps, but they completed one step, myself and the other teacher, Christina, wandered the room, and we came back after each step and summarized what we were seeing and filled in the gaps and then moved to the next step. That was a full tiered or stepped task. In other workshops or in other contexts, we might ask students to just talk about a key concept, they might not necessarily write anything. Give them five six minutes to talk about it. We’ll wander the room and then come back together and ask one or two to share with that. Sometimes, we might use the backchannel on our Moodle page for them to post what they’ve taught or things like, what does the word exclusion mean to you in a school context? So it just depends. We don’t structure the group because we can’t. They’re too big. The other thing I would say, though, is, in terms of very large-classes, it often, sometimes with me and often exclusively in other modules, they’re split into small groups, so they’re in workshop groups of 13, 14, or 15 workshop groups. In some modules, they’re never together as a big group, and there’s a bit of thinking, because it’s in a large-class, but what you’re managing there is the consistency across teachers’ input in class, formative feedback and so on. So yes, the group is smaller, but you’ve a whole curricular planning thing there to consider that I don’t have to consider when I have a big group. However, when the big class is split into those smaller groups of 20 or 30, you know who you have in your group, so you can plan for those groups of 30, whereas in my big class, I don’t know who’s talking to whom, so I can’t manage it at that level.

Anna: And we mentioned the think-pair-share, we would use that quite a lot, but using the same activity, but with lots of different kinds of materials. So it might be looking at samples of children’s work, for example, and within that group in the large lecture theater, we might roughly divide the room into four areas, and each quadrant, let’s say, would look at a particular piece of work, and then we’re looking for them to listen carefully when we take feedback from, it might be the same child’s piece of work, so they’re invested in kind of hearing of the other groups, while they might actually only be actively working in their small groups or in their pair on one piece of material, but they’re actively taking feedback from the activities that the other learners have been engaged in in the large-class. The other thing that Ann Marie uses really effectively is the panel idea within a large-class, bringing in a panel of speakers. I think that’s particularly effective in that large-class.

Ann Marie: It’s highly engaging. So with one of my groups, the module is on inclusion and inclusive pedagogy and inclusive practice in education, all contested terms. So to model the multiple perspectives, I bring in panels of guest speakers. So for example, I have one panel of parents and young people who talk to my students about their experiences of primary school. Now you could hear a pin drop [LAUGHTER] in those classes. It’s highly impactful, 10 times more impactful than me summarizing what the literature says about parental and student experience of inclusion. It literally demonstrates multiple perspectives. I have a second panel then, with teachers in different teaching contexts, and a third for a different team. So in other words, with the very large class, that’s this particular final-year group, every session is different. So I use three sessions where I have panels, I’ve two or three workshops where we have asked students to engage in a particular task, and a couple of sessions that are much more like a traditional lecture, albeit that we try and find a space for them to do something in that session, just to break it up a bit. So no two sessions are the same, and I try as well with those very large classes. Your question to Anna was about losing students, particularly in first year. I do wonder sometimes, is it not so much the class that’s large, but the program, it’s not just your module and your class is large, but lots of classes are large. There’s a multiplicity of experiences those students are trying to navigate. The large-class context can be the glue that brings some of those isolated learning experiences that are at a modular level back to the class. So for example, I think, after reading week next week, two of the co-taught sessions are with people who are teaching other modules with this same group. So they’re coming in to my session, we’re going to co-teach the session. They’re going to come in and make the links with what they’re doing in their modules, and I’m going to make the links within our module so that they’re getting an opportunity to see two of us working across the program, as well as somebody like myself and Anna, who are within the same module, but these are people in different modules. I do think that goes back to helping students make the links and the engagement for them to see that the two teachers in two completely different modules are now working together.

Anna: The other thing I would just add, in terms of Ann Marie’s panel, for example, in the panels that Ann Marie brings in, there is something particularly impactful of those real stories, real lived experiences. When they are heard in a very large context that enhances the impact of the experience for example. Because we were just talking there a couple of days ago, Ann Marie and I, and sometimes we talk about engagement, and I don’t know whether it’s the case in other universities, but a big focus on active engagement. There’s almost an assumption that that means that learners have to be up and about doing something, writing something, engaging with a student response system like Vevox or Menti or whatever it is, listening and listening to a powerful story like that, particularly when you’re part of a very large group, and it’s a powerful story, it enhances the impact of that, in fact, being in that large group.

Ann Marie: It does. And by the time we got to our third panel last week, one of the students in the group was on the panel. She came to me with something that had struck her, and she agreed, with a little bit of persuasion to be one of the panelists. And that’s powerful.

Rebecca: It always strikes me how, when I’ve had guest speakers come in and talk to students, that they can say the same thing that I’m going to say, but if someone else says it, it’s truth.

Ann Marie: Because it is their truth. So all of our speakers are speaking from personal or professional standpoint. It’s their lived experience, whether it’s personal or professional, whereas sometimes we might be talking about someone else’s experience sort of a step away. And I suppose that’s another thing, especially in the arts classes, is sharing yourself with the students and just giving them an insight into your experience. In my case, my past teaching experience, my own experience of my son’s engagement with school and little hurdles he’s had to hop over and those sorts of things and that, in terms of relationships in class, makes a difference as well. You suddenly were human, rather than the person who popped in for 50 minutes, dropped the pearls of wisdom and then runs out the door again.

John: What types of formative and summative assessment techniques have you found to be most successful in large-class instruction?

Ann Marie: I suppose some of the things we’ve mentioned are based on formative assessments and feedback. So a lot of those workshop type scenarios, and even when students are working on different case studies and different items and materials on the Moodle page, we will then provide feedback on all of those items, including the ones they’re not using. And sometimes I’ll use a little screencast, just a 3, 4, 5 minutes, to explain and fill in the gaps by feel from the class that they maybe they haven’t got it or whatever. Summative assessment can be trickier, and I think one of the challenges higher education teachers face with very large cohorts, the biggest one might be around the assessment and being fair, being manageable and so on. So I’ve grappled with this. I’m still grappling with it. I’ve two very, very large-classes, and I do something different with each and that’s because of the amount of content in the module that I’m part of. Other people who also have assessments in it and so on. With that fourth-year group, I have found that taking on the perspective of Universal Design for Learning in designing assessment and coupling that with authentic assessment approaches and viewing assessment as learning rather than of learning has changed how I experience the assessment process. So with that group that I have at the moment, for example, I have 430 students. I’m going to get, this December, about 140 assignments across all of them, a fair bit of work in that, in terms of marking and feedback and everything. I won’t deny that, but the most interesting set of assignments that I get, because they’re all doing something different, and they all present, so I’ll get websites…. They’re all creating a resort for a named audience, but they can create it in whatever way they like, and they really seem to buy into it, because they’ve got so much choice. So with that group that works, the multi-layered choices work, who to work with, what to do, who to write for, because they’re fourth years. But with my first-year group, which is 450 in it, I don’t offer those same range of choices. In that case, they have a fair bit of assessment in that particular module, so I have to cut my cloth to be fair to the student, and I’m not entirely happy with the assessment I have. It’s very applied. It’s based on case studies. We do similar, aligned case studies inside the class, in a workshop style format. They’re the co-taught sessions that Anna and I do, and then they do something similar themselves, but I still haven’t nailed that large-class first-year group, giving them options around what to do, who to work with, isn’t as fair as it is for the fourth years just because of where they’re at in the course of the size of the module they’re engaged with. So that’s my next task now, is to have a think about that and changing that before I meet that particular first-year group. It is challenging, but I would say particularly with my fourth-year group, the Universal Design for Learning Framework and applying that range of choices slowly and iteratively over time to see what worked, has completely changed my assignment/assessment marking experience, completely.

Anna: The other side of that coin, of course, is the feedback piece, which, as Ann Marie has said, the assessment piece at scale is particularly challenging. So is feedback. And again, it’s a bit like sometimes I do a lot of group feedback using exemplars as proxies for feedback. Sometimes, I think lots of higher education teachers have the same experience. Students nearly need to be hit over the head and told, this is feedback. This is relevant to you, that unless it’s actually on their assignment. And I know investing a lot of time, Ann Marie has said, in that first-year model where we’ve done the co-teaching, we model the task, we scaffold the task. There’s a lot of connections between the Moodle DLE in the classroom, so there’s lots of exemplars there. And that took an awful lot of time, sort of building that up when we started first, a lot of time, but it doesn’t change. I think of the example, Ann Marie, where we were giving feedback, asking students to write S.M.A.R.T. learning targets. That’s kind of a universal element, for example, in teacher education. And the investment in time in giving collective group feedback on the DLE, on various iterations of less and more specific, measurable, achievable, realistic, time-bound targets. It’s a lot of investment time initially, but it doesn’t change.

Ann Marie: Then it’s done. It took us hours the first year, then it’s done. You get the same type things back from following year groups.

Rebecca: So we always wrap up by asking: what’s next?

Ann Marie: Well, we are rethinking PHELC. So we’ll watch the space for that. We just need to think about where we might go with that, how we might expand it, maybe beyond us, and what we might do above and beyond our symposium. And then, as I said, in terms of the large-class teaching, it’s always what’s next. Like no two years are the same. We change. I love changing it. I’m basing the feedback from students as well. Each iteration of a module, the delivery of a module, might be a little bit different, and I am hoping, particularly with that fourth-year module, that the assessments that some of the really excellent ones, I’m developing a repository for those, which I’m hoping will become a site for publication at undergraduate level, in the first instance, within the university, but maybe further afield down the line. That’s a long term, big project that I’ve just dipped my toe in the water off. So I’ll let you know. [LAUGHTER]

Anna: And I suppose, just beyond our own work, one of the things we mentioned was the co-teaching and how much we have enjoyed over the years working together, and that has mainly been in the large class. So for me, and I mentioned that, one of the things we’ve done was we invited people to come in and observe. So one thing that I’m really interested in is that opportunity to be in other large classes, to be that observer, to maybe support other colleagues in their large-class work and so on, and any opportunity I can get to co-teach and to work with other people out there, embracing.

John: Well, thank you. This was a fascinating conversation, and we look forward to talking to you more in the future.

Anna: Thank you, John, and thank you Rebecca, for the opportunity. This was great.

Ann Marie: Absolutely. It’s been lovely to speak with you. Thank you for inviting us.

Anna: Indeed.

Rebecca: Well, thank you for some great ideas.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

367. Teaching Effectively with ChatGPT

AI tools can help faculty improve the quality of their teaching and assist students. In this episode, Dan Levy and Angela Perez Albertos join us to discuss a variety of ways in which ChatGPT can be used to support learning.

Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. Angela is a graduate of the MPA program in International Development at the Harvard Kennedy School, where she currently serves as a Teaching Fellow. Dan and Angela are the authors of Teaching Effectively with ChatGPT.

Show Notes

  • ChatGPT
  • Levy, Dan (2021) Teaching Effectively with Zoom: A Practical Guide to Engage Your Students and Help Them Learn. 2nd ed. LSC Communications
  • Levy, D. and Angela Pérez Albertos (2024). Teaching Effectively with ChatGPT: A practical guide to creating better learning experiences for your students in less time. LSC Communications
  • TeachFX
  • Teaching Effectively with ChatGPT Companion site

Transcript

John: AI tools can help faculty improve the quality of their teaching and assist students. In this episode, we explore a variety of ways in which ChatGPT can be used to support learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Dan Levy and Angela Perez Albertos. Dan is an economist and a senior lecturer in Public Policy at Harvard University where he teaches courses in quantitative methods, policy analysis, and program evaluation. Angela is a graduate of the MPA program in International Development at the Harvard Kennedy School, where she currently serves as a Teaching Fellow. Dan and Angela are the authors of Teaching Effectively with ChatGPT. Welcome Angela and welcome back Dan.

Dan: Thank you.

Angela: Thank you so much for having us

John: Today’s teas are:… Dan, are you drinking tea?

Dan: Moroccan tea.

Rebecca: That sounds nice.

John: And Angela?

Angela: I am drinking Chai Latin, my favorite.

John: And Rebecca?

Rebecca: I have English afternoon tea. It’s been a while since I’ve had that.

John: And I have English breakfast tea.

Rebecca: That’s unusual for you.

John: Yes, it is.

Rebecca: [LAUGHTER] Alright, yeah. We’ve invited you here today to talk about Teaching Effectively with ChatGPT. Can you talk a little bit about how this project got started?

Angela: Well, Dan and I had been working for a while already, when we started talking about this book, and we had been experimenting on different ways to integrate technology for more effective teaching and more effective learning. And I had personally always loved working with Dan, and he had written a book already about Teaching Effectively with Zoom that was very valuable for many educators during the pandemic, and he was thinking about doing something similar for ChatGPT. We saw that many educators were paralyzed by this technology. We saw that many students were already using it, but may not necessarily know the ways in which they could use it best for their learning, and there was just a big need to experiment better with this technology. We started with a small workshop that we did at the Kennedy School that was a massive success, and then from there, we saw there was a big need and a big desire from educators and students to learn more, and so we just decided to go ahead. And I think since then, it’s been a wonderful experience.

John: One of the things we’ve noticed in workshops is that faculty are often a little bit resistant to experiment with ChatGPT or other generative AI tools until they actually try it. So we thought maybe we’d start with that. What are some ways in which generative AI can be used by faculty to help design new courses or improve the design of existing courses.

Dan: I think generative AI can be used by faculty in a number of ways. And just to reinforce your point, John, I think you don’t have a good sense of how useful it can be to you until you actually start using and experimenting. So I think if we could leave faculty with one message today, it would be: “Start experimenting with the technology so you can understand the benefits and the potential limitations.” We can talk about ways in which we can use the technology to, as you said, design new courses or improve the design of existing courses, and I’m sure we’ll be doing it throughout the podcast. In terms of designing new courses, you can do things like help you refine your teaching goals, for example, or you can have it help you as a brainstorming partner in a class that you’re thinking about designing. You can ask it to provide you feedback with material that you’re developing. So there are a number of ways in terms of improving your course. You can ask it for feedback on slides that you have produced in the past. You can ask it to summarize documents, summarize course evaluations, to gather what are the main things that you need to do, but at a super broad level, maybe the way that I think our book tries to crystallize the use of AI, or ChatGPT in particular, is you should think of it in potentially two roles for you. One, it could be a brainstorming partner, a thought partner for you, and the second one, perhaps the most current use, is it could be an assistant. It could be a teaching assistant, a research assistant, a general assistant. So anytime that you are in front of your computer and you’re trying to get something done and you’re feeling stuck, I think you should just ask yourself, is this an activity where maybe spending three minutes with ChatGPT could help me either get unstuck or arrive to an output faster than I could in the absence of it.

Rebecca: I know as an instructor, I’ve used it to get past a blank page, just to get a jump start on something. In the second chapter, you describe ways in which ChatGPT can be used to customize your courses in the interests and needs of your students. Can you talk a little bit about how ChatGPT can create a more student-centered learning environment?

Angela: One of the biggest potentials that the education world has seen in ChatGPT is the opportunity to approach closer to the ideal of one tutor for every student. Research has known for many years that providing personalized learning can have huge impacts in each student’s learning and. And ChatGPT can enable part of that benefit. It’s not perfect, but it can get us closer to that outcome. Now, how can it do it for a professor? I like to think about it in three ways. One is that it allows educators to process the student information that has high volumes. So for example, in every class that I teach, I start by asking students to fill in a form where I ask them about their interests and what they’d like to learn. If with ChatGPT, I can get a summary of those interests and like the profile of my students. The second way is that it provides access to knowledge or to context that allows educators to tailor activities to their specific students. So for example, when Dan and I were updating one of our activities for the statistics class that Dan teaches, we were trying to think of topics that were relevant for the students in the class. And we asked ChatGPT to provide different ideas for topics. And these went from topics related to health to finance to arts, and many of these topics we were not experts in, but they were topics that our students were interested in learning more about. So ChatGPT allowed to tailor our activities and our context to make them more relevant to what our students wanted to learn. And then the third is that ChatGPT can be made as an avenue or as a tool to personalize student learning directly. So we can ask the students to learn about different topics directly on ChatGPT, so that the approach is the one that works best for them. And we’ve seen this in action as well, where, in one of the examples that the book covers, is how one of the courses that Dan was teaching, he asked his students to learn about signal theory, and then you have two students that take a completely different approach. One of them asked for more in-depth and more expert-level explanations, while the other one says, “I have no context about this, so I actually need to understand this as someone that has zero background knowledge,” and that is super powerful. There’s no way that an educator would be able to provide the two explanations in a class to different types of students. So in those ways, I think that the potential is huge.

John: And you mentioned using a form to gather information about students and feeding it into ChatGPT. One of the things you suggest, though, is that you don’t want to include personally identifying information, such as their name or other things. Could you talk a little bit about why it’s important not to do that?

Angela: Absolutely, so there are huge privacy concerns about what data we put in to ChatGPT, because basically, the algorithms are trained on the data that is existing on the internet, and right now, we have no good sense of what openAI is doing with the data that we input into ChatGPT, and so in that sense, we strongly encourage educators to not put any sensitive data, like student contacts, student grades or anything that would make the data that we are adding about the students identifiable. Now there are solutions that openAI is introducing for universities that are available and that we also talk about them in the book, but our general recommendation is to be very, very careful with the type of data that is used in conversations with ChatGPT.

Rebecca: Can you talk a little bit more about additional ways that faculty could incorporate ChatGPT to improve the active learning in their classrooms?

Dan: So in the book, we describe sort of a simple prompt that you can use, where you give it the slides or the materials that you use in a class a prior year, and you sort of say, “I want to incorporate more active learning into this class. Can you give me a few ideas?” and what ChatGPT would tend to do is provide you with a list of active learning activities. So maybe it would suggest a poll question for your slide number three, or a breakout room kind of activity for slide number seven. The reality is that if you’re just trying to inject a little bit of active learning into a class you’ve already planned, you don’t want to incorporate the 10 ideas that ChatGPT would give you, but it’s enough for you to have one or two good ideas come out of that literally three minute exploration for this to be, I think, worth the effort. I think often we think about, can ChatGPT be better than us?, or can AI be better than us? In the book, we emphasize that’s not the question that you should be asking. The question we suggest you ask is, can I plus ChatGPT be better than I alone? And so in this particular example, you might have a class from the previous year that you’re trying to inject some active learning activities. You ask ChatGPT for a little bit of help, maybe give me five ideas or 10 ideas of active learning that I can incorporate into this particular class. You get one or two that you like, and you might run with them. Or maybe it’s enough of a…not the exact thing that you want to do… but it’s enough of an idea that you can explore. The other interesting thing about AI is that it has infinite patience. So if you don’t like any of the 10 activities that it gave you, you can ask for 10 more, maybe 15 more. And if all you’re looking for is one or two, the spark of an idea might be there. You sort of were describing before, maybe there’s a blank page and I want to get some ideas. The same thing is for injecting active learning into the classroom. One more thing that I want to add is that both in terms of personalized learning and in terms of injecting active learning into our classroom, is that one of the things that has been very interesting is to observe, and I think Angela has been an amazing conduit to be able to see this, is to observe how our students are using ChatGPT, in the absence of us doing anything to use ChatGPT. And so just a couple of quick anecdotes. This happened, actually, after the book. This happened in the last week. So in one of them, I have a student in one of the classes who was hesitant to take the class because the class does require you to run simulations with Microsoft Excel. And he said, “I don’t know Excel. I’m willing to work hard, but I don’t know Excel.” And he, basically in a couple of weeks in the semester, designed simulations with the help of ChatGPT, in a way that he didn’t even think was possible. So to observe how he was able to pursue the curiosity, the intellectual wanting to learn about this and sort of struggle a little bit with ChatGPT: “Can you give me the Excel formula so I can do this?” And so it has been wonderful, and frankly, I couldn’t have spent the hours with him that it took him to do this, to the point that he’s now creating simulations of stuff that we’re doing in the class where we’re not even asking them to do simulations. He came back this week and said, I have created a bot for your course, and I have essentially put the materials from the class, and now I’m asking the bot some questions and getting answers, and that I think it would have been hard to do before. So those are just a couple of the examples of being able to see the students engage in this active learning themselves without us even doing much.

Rebecca: I think that underscores the idea that you suggested earlier, of this infinite patience. I love that idea.

Angela: In the book, we also talk about ChatGPT being empowering, because at the end of the day, it removes the barriers to learning, and so it gives them the encouragement that students need to be able to learn new things by themselves. And of course, critics have sometimes criticized that this only works for motivated students, because you need that motivation to learn to begin with, and that’s absolutely right. But is it not a benefit that at least those that are motivated to learn will be able to do things like this and feel that empowerment, which in turn, is going to feed more into their desire to learn new things?

John: And for those students who come in with a somewhat weaker background, it can help to equalize that background by providing a tutor that never gets tired, for the same reason, that it has infinite patience, and that’s something that human tutors may not have, or their friends, or other people in the class working with them may not have.

Rebecca: Or that lack of judgment.

Dan: There’s an additional benefit, John, to what you just said, which we discovered when interviewing students for the book. It’s not just that it has infinite patience. It’s when we asked the students, what do you see as the advantages of using a bot or just an AI tool to learn, two that came up, one we expected, the other one, in retrospect, we should have expected, but didn’t. The one that we expected is, well, it’s available to me. 24/7, an instructor is not. And the second one, which is very interesting, was summarized by a student who said, I can ask the tools questions that I would be too embarrassed to ask a teacher or a teaching assistant. So it’s not just the infinite patience, which, frankly, a human being won’t have after the third or fourth time that a student says, “I don’t get this.” You might sort of try in seven other ways, but at some point you might have difficulty understanding where they are. Or it’s not even infinite patience, it’s infinite time. You might want to spend time with a student, but you might not be able.

John: And students who may experience some imposter syndrome, perhaps that may help them get past that barrier of asking for help, because they don’t want to reveal what they’re not comfortable with.

Dan: Absolutely.

John: One other thing you talk about is course evaluations, and one thing that most faculty experience is that they may get hundreds of really positive ones and two or three really negative ones, and what they walk away with is a memory of those less positive comments, the negative comments tend to stick a little bit better. You suggest, perhaps using AI tools can help remove some of that. Can you talk a little bit about that?

Rebecca: How did you know they stick more, John?

John: I’ve heard rumors, anyway. [LAUGHTER]

Dan: Yeah, so I think you just pointed out one of the advantages of being able to use an AI tool to summarize course evaluations, which is that sometimes what you remember from reading course evaluations is that comment that was either offensive or hurtful or just like, mean, frankly. And what you can do with an AI tool is to just summarize and sort of say, “Give me what are three to four things that students seem to find effective in this course, and give me three or four things that students find ineffective,” which at the end of the day is probably the most important thing you want to learn. And you might even ask the AI tool to give me some specific suggestions for how to improve the course given these evaluations. Now, you might not agree with the suggestions. You might sort of say, “I know students are unsatisfied with this, but I’m not willing to do what ChatGPT is suggesting.” But again, it at least gives you a little bit of a sense of direction. The other thing I would say is that even the mere act of summarizing the themes is something that I think it’s hard for the human mind to do, like if you have 50 or 60 or 100 evaluations to read one after the other, you start thinking about that comment that was either very good or very bad, and you lose a little bit of track of what were the major themes. And so I think those are some of the benefits. I do want to say that I don’t think ChatGPT is as good as having a teaching professional read the evaluations and come to you and have a conversation with you about them. But you might not have such person at your disposal, and even if you did, that person might not be available when you need to do this, or maybe might not be available every semester. So I think there are some advantages of doing it in this way.

Rebecca: It almost seems like a possible benefit of: “Here’s your evaluations, and also here’s the summary.” [LAUGHTER]

Dan: Yeah, well, I think academic institutions should be thinking about doing things like that, where you automate and by the way, we had a meeting two weeks ago with our faculty, and in half an hour, I created a simple bot that our faculty could use to upload their course evaluations, and it gave them sort of the equivalent to what I was just describing. I even added like a little instruction for the bot: take the most favorable quote so you end reading the evaluations with a nice quote from a student. And this is just like half an hour investment in something that could be useful to many faculty members. You could imagine thinking more deliberately about, “Okay, what are the things that we want the faculty to be working on, and how do we design a simple bot to bring those things.”

John: And I think you also suggested that if a department chair or Dean is also reviewing evaluations, this could make it a much more efficient process to summarize them, rather than them having to go through and read them one by one, especially in larger classes or departments.

Dan: Absolutely, I think in general, the ability of AI to both summarize documents and to allow you to interrogate documents really far exceeds the human capacity to do that. So if you have 100 pages of student feedback or course evaluations, there’s no way that you can do it in the speed at which ChatGPT, or an AI tool similar to it, can do it.

Angela: Dan, there may be, like, a useful tip to give here to our listeners, which is, when summarizing student evaluations, it’s always useful to ask for how many people have made comments about a specific theme, because this helps you weight how much the themes that are being brought up are widespread across the students or just a one-off comment for somebody, and that may be something that ChatGPT may not organically. And then the second one is to also split themes between positive and negative, because there are some times that you have a theme, say, like, problem sets are very long, maybe some student makes a positive comment about it because they think that they allow them plenty of time to practice topics in depth, while some other complaints about the workload, right? And so here we comment how interrogating a document may also be a back and forth process where the better questions that we ask, the better the quality of the output that we are going to get from ChatGPT is going to be.

Dan: Absolutely.

Rebecca: One of the other examples that you provide about getting some feedback about teaching is having faculty members record their class audio and then have it transcribe and then ask ChatGPT to evaluate and provide suggestions about their teaching, and then have ChatGPT analyze those recordings and assist with peer evaluation of teaching. Can you talk a little bit about how that feedback might be helpful and maybe perhaps less biased than human feedback on teaching?

Dan: I want to first preface this by saying that this was done by one educator, as we highlight in the book. There are obviously issues of privacy and permissioning that need to be resolved before scaling this, but I think the potential of having an AI tool be able to analyze a class that you teach and provide you with detailed feedback customized to your own teaching philosophy, you can sort of say, here are the teaching principles that I have and that I bring, and I want you to assess what I did relative to those principles, as the educator that we highlighted in the book did, I think is wonderful. Again, comparing with a counterfactual, what would you do if you didn’t have the AI tool? If you had a teaching coach or someone who specializes in teaching and learning, they might be able to see things that the AI tool won’t be able to see. Because they know you, they might be able to give you advice that’s more sort of adjusted to your own needs. But there’s no question in my mind that most of us don’t have access to such a person on a literally daily basis, and so the AI could help scale that feedback, could help complement… Angela and I in the book, talk a lot about how AI can complement things that we tend to do with other human beings, and I think this is one of them. In terms of being less biased, obviously, there’s a lot written about how AI has some built-in biases that goes, I think, beyond the scope of this podcast, but I do think that an AI tool can, if you give it: “Here’s the criteria by which I want you to judge the class that I just gave,” might be able to help you see things that you’re not seeing.” So is it ready for prime time right now? I mean, our educator was super, super innovative in the way that they did it. My sense is that the tools will come so that it’s much easier to do it. And one of the areas that we’re most enthusiastic about is to think about the use of AI inside the classroom, not just to provide feedback after class to a teacher, but how can AI be used inside the classroom to do lots of things? I think that’s a chapter in the book that is a little bit of reflecting experiences from educators, but mostly speculating about future directions. My hope is over the next year or two, we’ll see lots of super interesting uses of AI in the classroom that could help advance our teaching goals.

John: One of the things that people who run teaching centers sometimes note is that faculty will come to them and talk about what they’re doing in classes, and then the feedback they get from more senior people in the department may not always be consistent with evidence-based teaching practice. [LAUGHTER] Of course, that would not happen at our institution, [LAUGHTER] but I think that would be one way in which, while there are biases in AI because it’s trained on human data, it’s at least trained on a very large base of human data, rather than just the perceptions of the one person who’s sitting in observing the class that semester. So it might actually provide somewhat more neutral feedback, again, as long as the prompts are well designed.

Dan: Yeah. And one thing I want to add is that you can literally tell the AI tool, let’s say there’s a book about teaching and learning that you think captures particularly well the evidence and how you would like to teach. You can sort of give that material to the AI tool and say, I want you to use this as a basis for providing the feedback. As a person who teaches and tries to be as helpful as I can to my colleagues when they ask me to observe them teach, one of the challenges that you face when you observe someone else teach is that instead of thinking about, how can I be helpful to this person where they are, it’s almost inevitable you start thinking how you would have done it differently, and that’s probably not the right approach to help someone get better, because whatever you would have done differently might or might not help the learning of the students in that classroom. So I think trained teaching professionals might not have this issue, but if you’re getting feedback from a colleague, again, I don’t think it has to be a substitute. But if you ask yourself, how many times have I had a colleague observe me teach in the last year? I think most of us would reply, not that many. And AI could help scale that feedback and certainly complement.

John: A related note is that when there’s surveys of faculty in terms of how much they use active learning techniques, generally, most faculty report a relatively high level of use, but when students are asked about that, the reported level of use is a bit lower. And perhaps this, again, might be a nice way of getting at some assessment of that that’s independent of the faculty’s self-perceptions.

Dan: Absolutely. Again, I’m not saying that this is happening now, but one very raw measure of how much active learning is happening in your classroom is what percent of the time in your classroom you are speaking. I have had someone measure that in my own classrooms, and I severely underestimate that percentage, I think I speak much less than I do, and so it’s a reality check. But now that plus 100 other statistics about your class, average wait time, how many open-ended questions have you asked? How many times do you say “hmm” when you’re teaching? How much back and forth there’s between students, like a volleyball kind of dynamic rather than ping pong, where they’re always going through you? All of those things which you might have imperfect perceptions of what’s going on. You could imagine an AI tool being able to capture and give you a more realistic sense of what’s happening in your classroom.

John: I believe that there is a tool out there that will record your classes and use AI to give you some feedback. What’s the name of that application?

Dan: The people that I actually know and highly recommend the tool is called TeachFX, and they’re able to do that, and maybe some of the things that I’m describing now they can already do, but if not for sure, they will be close to being able to do. It’s almost like you have a full diagnostic of what happened in your classroom in a way that would be very hard to do otherwise.

Rebecca: So shifting away from teachers thinking about their own teaching and back into the hands of students being in control of some of their own learning, what are some ways that students can use ChatGPT to assist their own learning?

Angela: I love this question, Rebecca, because it’s a big part of the book, and it’s something that was very important for us to reflect. Dan has a very good sentence that he uses when he starts introducing workshop which is, teaching is not the same as learning, and so teaching effectively may not always be the same as learning effectively. So that said, I’d like to start this the answer to this question by highlighting one of the most important principles that we talk about in the book, which is that different uses of AI lead to different degrees of learning. A nd so we strongly believe there’s not such a thing as ChatGPT helps improve learning, or ChatGPT impacts learning negatively. It just depends on how it’s used. So I could give you a range of different uses that students can give to ChatGPT, and we actually talk about six of them in the book. They include things like getting feedback about their answers, which relates to this idea of using ChatGPT as a tutor. They are things like learning about a new topic or finding arguments and counter arguments about a topic. These are basically anything that a student has to do in their schooling process, t here’s a way in which ChatGPT can learn. What I want to give you is this very specific example that we include in the book, and that explains how ChatGPT can be an agent in the active learning process, rather than an agent in making learning passive. And it’s actually an example that I did myself when I was trying to create an essay for one of my philosophy courses. And one way in which I could have gone about this exercise is asking ChatGPT to write the essay for me. And I could have said this topic is about effective altruism, please write an essay of 1000 words with some pros and some cons, and then just submit that essay. That would have, of course, negatively impacted my learning, and it would have been cheating, obviously. But there are many other ways in which I could use ChatGPT for the same task. What I did in particular is I asked ChatGPT to play the role of a philosopher that was very acquainted with Peter Singer, one of the philosophers that was particularly relevant for the topic that I was writing about. And I had a debate, a back and forth debate, with the AI tool while the AI tool was impersonating Peter Singer. And so I threw the AI some arguments, I threw the AI some questions, and then the AI tool would answer me back, and that was incredibly helpful for me. It helped deepen my understanding of the philosophy, it helped me strengthen my arguments, and it overall improved the quality of the essay that I ended up submitting massively. Now it’s very difficult for educators to understand how students are using ChatGPT in the background when they just see the end result, but educators can play a role in encouraging productive use of ChatGPT.

Rebecca: It’s a super fun example. [LAUGHTER]

John: It’s something I’ve encouraged students to do too, to have it play the role of an economist and then ask questions about current policy debates. In particular, I suggested having it play the role of David Ricardo and ask questions about trade barriers in an introductory class. And they enjoyed that, or at least the students who tried that enjoyed it. It wasn’t a required activity, but it was just something I encouraged them to do.

Angela: It’s fascinating. Yes, I agree

John: What are some of your own favorite ways of using ChatGPT.

Angela: I’m using it recently for things related to marketing. I find myself having to create a lot of marketing copy, which is something that falls way out of my comfort zone. And it helps me make my communications more crisp, and it helps me with creativity. I use it a llot to brainstorm ideas where I just say, “Give me 10 different jokes that I could use in starting this email, for example. I find it amazing. It solves my life. It’s my best friend these days. I have been integrated into almost everything I do, and I’ve been finding it particularly helpful for this task.

Dan: Going back to what I said at the beginning, I think of it as a thought partner and as an assistant. As a thought partner, a way of brainstorming lots of things. So it could be teaching related or not teaching related, particularly when thinking about broadening things, I think it can be very helpful. But I want to talk maybe about a use that we haven’t touched on and which I think might be of interest to people, which is that it can be incredibly helpful at doing things that we hate to do with our computers. So for example, the other day, someone sent me a list that, in reality, I would have preferred to have a table, because it really had literally separated by, I’m not even sure, space, double space, sometimes comma, whatever it is. And I just copied and pasted that into ChatGPT and I said, “Give me a table with this.” So anything having to do with formatting, you’re much more likely to be able to do it quickly with ChatGPT or similar AI tool than you would trying to do it manually. So another of those menial things, I know it’s not a big sort of deal to do it, but I basically had to write an expense report for someone who had invited me to do something, and all I had on my computer was a scan of the eight receipts that I had. And one day I said, “Okay, I have to do it.” And so I just dragged the eight receipts to ChatGPT and I said, “Please prepare an expense report with a table in the middle, listing each expense and totaling the expenses” and in three minutes counting the time that I edited that letter afterwards I had the expense report written. I know that’s like a super small sort of thing, but think about a number of things that whenever you find yourself dreading to do something on your computer, you should think of this almost like as an augmented operating system. It’s almost like as an assistant. Normally, when you’re trying to do something in the computer and you need help, you might go to someone and say, “Can you do this for me?” And then you have to wait. Here it’s almost like the assistant is right there in between you and the computer, and able to do lots of things like this, even things like, here’s a printout of a table that I had on a piece of paper, put it on an electronic table, add these two columns. So after you brainstorm if you want things in a different format. The other day I had someone send me a bunch of numbers, and I just copied and pasted and started doing some basic data analysis with ChatGPT. So lots of activities that would normally take you more time are ones that you can do. Those are not very profound uses, but if they save you time so you can go back to your research, your teaching, or anything else, or even fun, that would be great. And then just as a couple of fun uses: one, next time you’re at a restaurant and there’s a dish that looks delicious or that tastes delicious and that you want to have a sense of how it was done, take a picture, upload it to ChatGPT and say, give me a recipe for this dish. I’m sure it’s not going to be as good as the restaurant, but it can literally give you a sense of that. And then nowadays, with audio capability, you can literally have conversations with ChatGPT. You don’t even need to start typing. So all of those things are now possible.

Rebecca: I like the idea of saving dread. That’s good. [LAUGHTER]

Dan: Yes.

Angela: Someone once told me that they took a picture of their fridge in their kitchen and then asked for potential recipes they could cook during that week. So the people really get very creative with what they can do with ChatGPT once they start experimenting

Rebecca: Anything to save a headache, sounds like a good plan.

Angela: Totally.

Dan: Absolutely.

Rebecca: You included a chapter in your book on creating chatbots. Can you talk about some ways that an instructor can use chatbots, either for their own work or for student use?

Dan: I think Rebecca, the distinction you just made now, I think that is a very important one. You can create a chatbot for your own use, or you can create a chatbot for others to use. Others could be your students or any other person that you want. So for your own use, if you think about activities that you’re constantly doing, and you don’t want to write a prompt that has a context every time that you go into ChatGPT, you could have a bot that helps you with this repetitive task. So a simple example would be, suppose that you have a course where you want ChatGPT to help you every class session, you want to upload your slides, and you want ChatGPT to give you feedback. And so you might create a bot where you upload the syllabus of the course, any material of the course that’s relevant for ChatGPT to have as context. And then you have a prompt for the bot that describes the type of feedback that you’d like to receive. Now you could, every time you’re going to teach a class do exactly what I’ve just described, so upload the materials, put the prompt and so on, but without much extra effort you can create a bot that basically allows you to go to it every time, and then instead of having to do all of that from scratch at that moment, you just sort of say, “Alright, here are my slides for tomorrow. Go ahead and give me the feedback,” and so it’s a very efficient way of being able to do tasks that you do often. I would say one important thing, maybe the most important thing that I think Angela and I can say about bots is that they sound like something very hard to do, but they are not. If you already have the basics of how to prompt ChatGPT, creating a bot is not anything extra. There’s no code. It’s English. It basically has two components. You upload files and you write a prompt, and that’s it. You obviously need to test it to make sure that it does what you want and so on. So that’s number one. That’s for the instructor. But you can also create bots for your students or for anyone else. And so there, I think there are interesting opportunities. in the teaching realm. You would upload your syllabus, you will upload a bunch of documents relevant for the course, and then, depending on what you want the bot to do, we have lots of examples in the book about what educators have done. So, for example, our colleagues Sharad Goel and Teddy Svoronos created a bot to answer questions about statistics course and give practice problems for them. We have David Malan from CS 50 created a bot to help answer computer science questions that students often have at different times in the day and at night. So there are many, many different uses. You just have to think about, okay, what are things that I would want my students to have access that maybe I cannot provide that access right now, and it might be a good thing to do. But the bottom line is, this is not rocket science, if you know how to prompt, you know how to build a bot.

John: You also provide a companion site. Could you talk a little bit about what’s on that companion website?

Angela: Many things, but on the topic of creating chatbots, we actually have that same chapter linked for free in our companion site. So as Dan was encouraging educators to just try and give it a go, they can refer to the chapter that is linked in our companion site. A lot of the work that we did with this book was try to demystify AI for people that were not familiar with this technology. So we are not using any particular slang. It’s all conversational English to explain how to use it. In addition to these, we have other resources that, due to the nature of AI and how rapidly it’s changing, we expected them to potentially be out of date soon, and so we have various types of resources linked. One example that we reference on our book and that we feel that people may want to explore in more depth, too, are the same prompts that we included in the book are included in a document that is linked in the site. And we feel this is important, because if anyone is reading the book and they feel that they want to try out something, instead of going to ChatGPT and having to write the prompt from scratch, they can just go to this document, copy and paste and have the prompt ready to go. And then we also include some additional research or additional resources that can be useful for people who are interested in learning more.

Rebecca: One issue that students are definitely concerned about is the effect of generative AI on their job prospects. What types of jobs are most likely to be affected and will the widespread availability of AI assistance reduce the productivity difference between skilled and less skilled workers?

Dan: So, I can see that John was behind this question, because I know he’s an economist, and I’ll be curious to hear your views on this. As part of the job that I do here at the Kennedy School, I started co-teaching a course on generative AI, and we had a session on generative AI and jobs. And so I think the short answer is that I don’t think anyone knows for sure. So people are trying to understand how these tools can be helpful. I find historical evidence about technological breakthroughs to be at least informative, but there are people who say that we’ve never had any technological revolution that was as disruptive as this one. And so those people basically say this time is different. But the one thing I would say is, if you study technological innovation in the past, almost every time, people have said this time is different. [LAUGHTER] So I’m not sure what to make of it. Maybe this time is indeed different, but I think at a super broad level, I think we’re not gonna run out of jobs. I think some jobs will disappear, and some jobs will substantially be augmented by AI, and some new jobs will be created. That’s my best guess as to what will happen. Students who go to college, my sense is that they will probably be in a relatively good position to take advantage of AI. But if I had to give students advice, is to take advantage of the opportunity of being in school to learn how to use AI in a productive manner. I think we tend to sort of think that our students will figure it out quickly, but I do think that there’s ways of using it that can help students learn more effectively and just to work more effectively. Whether it will reduce the gap between skilled and unskilled labor. I think that feels to me that there are different sides of that argument, so I leave that up to you, John, to sort of say in which side you’re in. On the one hand, some people say that it’s substituting cognitive work, and therefore it’s going to reduce the gap; It’s one of the technological innovations that seem to be displacing more of the cognitive work than previous technological innovations. On the other hand, it might be precisely for the skilled labor for which the technological innovation can help increase their productivity.

John: Yeah, it’s really early, as you said, and it’s really hard to guess where this will be heading, but some of the initial studies, I think, have shown that it’s tended to raise the productivity of less skilled workers, or less experienced workers, relative to those of more skilled workers, which could reduce some of those gaps and perhaps narrow some of the pay differentials. And we don’t know in what direction that will go.

Dan: It could.

John: And on a broader level, there’s the whole issue of AI raising productivity of workers, which could result in substantial gains for society. But the question is, how are those gains going to be allocated? Will they show up in the form of higher incomes for everyone? Will it be concentrated among the firms providing AI tools and services, or will these be shared? Could we end up with that 15-hour work week that Keynes predicted way back in the 1930s?

Dan: I’m skeptical that we will end up in the leisure society of 15 hours a week, because one of the people that I follow on this topic is David Autor, a professor at MIT, and he argues, I think convincingly in my view, that we as human beings have an insatiable sort of capacity and demand for new products and services, and I think that’s going to feed the next generation of products and services that we will want to consume. And if you think about occupations that existed at the beginning of the 20th century that no longer exist, and now there are occupations completely new that didn’t exist before. And my sense is that 20 years from now, there will be occupations that we didn’t even think would exist. So my best guess is that it can have a positive effect on productivity and therefore increase in general economic well being, but I don’t think it will increase it to the point where we’re all gonna work less. I suspect that the two of you will work just as hard, even if AI could do half of your job. Maybe you’ll create a second podcast. Maybe you will do some other thing that will keep you occupied. That’s my best guess. But again, this is highly speculative. I don’t really know what’s going to happen.

Rebecca: Dan, I feel like you know John really well. [LAUGHTER]

John: And the work week has been stuck at about 40 hours since the 1940s or so, roughly.

Dan: I do want to sort of at least mention very explicitly that, even though the tenor of this conversation is one that is positive on AI, there are obviously lots of challenges and risks. On the Teaching and Learning realm. Angela already mentioned, but AI can be used in ways that hinder learning, that are not going to help learning if students start using it just to do homework for them instead of with them. So I think there are many, many challenges and risks inside the teaching and learning realm, and there are obviously huge societal level challenges, just to mention a few: misinformation and disinformation, challenges of intellectual property, challenges of jobs that we have just discussed. There are some people who are really worried that, even though, in the long run, we might not lose jobs, the reality is that some jobs will disappear, and so do we have a strong enough safety net to be able to help people transition from one kind of job to the other? And there are obviously existential risk that people talk about, what AI could do if it were out of control… so beyond this podcast. But just to return to the job of an educator. I think if you’re an educator and you’re not using AI, we suggest that you at least experiment to see how it could help you advance your teaching and learning goals, and maybe even try to sort of see how your students are using generative AI, because that would be very instructive in terms of you hinking about how you can use it to stimulate their learning.

Rebecca: So we always wrap up by asking: “What’s next?”

Angela: For us, the hope is to get our work out there to as many educators as possible that could benefit from it and continue learning, because there’s so many changes that are still going to happen, and so many that we don’t know. So staying open to learning and to exploring how to incorporate this technology for better teaching and better learning.

Dan: I agree with everything that Angela just said. I think for me, I’m fascinated with the potential of generative AI helping us learn. Not just help our students learn, but help us as educators learn. And so to me, it’s like incredible to literally have available 24/7, a resource that I can ask questions and have conversation with in ways that wasn’t possible before. So in terms of what’s next as an educator, I’m very curious to continue this journey. As an educator thinking about our students, I’m just literally fascinated just learning how they are using some of these tools. And if you could do one thing as an educator, maybe create space for them to not just use the tools, but share with you how they’re using it. So it could be something as simple as you build into an assignment, answer this question, and then go into an AI tool, get some feedback, and then tell us how useful the AI tool was for you. And I think you’ll start discovering some really, really interesting uses that, as Angela said before, get closer to fulfill the promise of personalized learning that we have been sort of exploring for many years.

Rebecca: Well, thank you so much for joining us. This has been a fun conversation.

Dan: Thank you.

Angela: Thank you so much for having us.

John: Thank you. It’s an exciting topic, and thank you for helping lead people along in learning more about AI and how it can be used.

Dan: Well. Thank you, John and Rebecca, for the work that you do to illuminate us, not just in use of AI, but in so many topics related to teaching and learning, you do a huge service to the teaching and learning community. So thank you very much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

347. CATs and AI

Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, Todd Zakrajsek joins us to discuss how generative AI can enhance these techniques by providing more immediate feedback.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo.

Show Notes

Transcript

John: Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, we explore how generative AI can enhance these techniques by providing more immediate feedback.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Todd Zakrajsek. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo. Welcome back, Todd.

Todd: Thank you, Rebecca.

John: It’s nice talking to you again. It’s been a while. I think the last time was last summer.

Todd: Really? It seems like yesterday. Time goes by. It was last summer. You came to my office and recorded.

John: That’s right.

Todd: It was fun. You left me a really nice set of tea. Wooh.

John: And a mug.

Todd: Yes, I have a mug for the show. [LAUGHTER]

John: Speaking of teas, [LAUGHTER] today’s teas are:… Todd, are you drinking tea?

Todd: Every time I’ve been on this program I’ve been drinking tea and I’ve had some pretty exotic brands and blends but, I’m sorry, today I have water, the lightest of all teas.

Rebecca: It’s a very light tea. I have English Tea Time tea.

John: I have Lady Grey today.

Rebecca: That’s a good choice.

Todd: Alright, I think I came in third place.

John: We’ve invited you here today to discuss classroom assessment techniques. The first two editions to this book were written by Thomas de Angelo and K. Patricia Cross. In last week’s podcast, Tom joined us to discuss the origin of this book, where you joined as a contributor for the current third edition of this classic book. What was your reaction when you were first asked to join in this project?

Todd: Well, this is actually kind of a funny one, because it goes back to Linda Nilson and Teaching at its Best. And Linda was retiring, and the publisher, Jossey-Bass wanted to do a new edition of Teaching at its Best, and Linda says “Why don’t you see if you get Todd to do it and I will work with him on it and do it.” And we did it. And we got the book done a little bit early and on schedule, and it came out very, very well. So there’s a little plug for that book there. And it must have been about two months later, I had called them and said: “If you find another project like this, I love working with people like this.” And a couple of months later, I got a phone call and they said “We have another project we’d like to potentially have you work on” and I said: “Great, what is it?”… and they said: “Have you ever heard of the book Classroom Assessment Techniques?” And I held it together, and I said, “Well, yeah, yes, I have.” And they said, “Well, we thought it’d be helpful given what you do and what Tom does and everything, is maybe the two of you could work together on this book.” And so he talked a little bit about what there was and everything, and I hung up the phone and did a little happy dance because this book was one of the most influential books in my entire career. But the way the universe works is I ended up doing this one because of the work with Linda Nilson’s. And so that’s how I got started on this.

Rebecca: That’s amazing.

John: And, I agree, that is an excellent book as well. So you’ve got two classic books that you’ve worked on recently.

Todd: It’s really surreal, you know, a first-generation college student and every once awhile is kind of like I just can’t believe this is happening. But it was a really good project.

Rebecca: When did you first start using classroom assessment techniques in your classes? You mentioned that this book was influential.

Todd: I taught my very first course in 1987-1988. This book came out, second edition, 1993. And I was using it pretty much the moment it came out. Somebody had pointed it out to me and I didn’t know anything about classroom assessment techniques, and then I started reading through it and picking some out. I thought it was genius. Much like periodically, what you’ll do is you look at something and think this is ridiculous that nobody started doing this before. In some respects, it’s not that hard, right? You ask people questions, and you find out if they’re learning. It just took Tom and Pat to come along and say, “Why don’t we come up with strategies to find out if they’re actually learning.” So I think it was really influential for me because it kind of struck that chord with me that I always believed we should be finding out how much students were learning instead of focusing on teaching. But that’s not what was going on at the time, everybody was focused on teaching-centered educational practices.

John: What do you find most valuable about these classroom assessment techniques?

Todd: It’s interesting, because there’s a couple different levels. The one that’s really valuable, of course, is you find out if students are learning. Secondarily, you can find out if you’re teaching well, and you can change your teaching practices based on what they’re learning and based on what’s going on. But I’ll tell you the real reason I think this is really, really valuable is it addresses equity issues very, very cleanly. We walk into classrooms with all kinds of preconceived notions and stereotypes and implicit biases. And it’s really, really easy to think that a student’s struggling for a certain reason, unless you ask them. And so, for several of these strategies, I think it’s great because I’ve had students that have been practically sleeping through every class and then I started doing the classroom assessment techniques and he had some of the best responses, this one student did. And I would just totally blown away that this person was just processing it to such a level and I had a preconceived notion based on where the person sat and how they dressed and how they acted in class. I didn’t think they knew anything or they didn’t think they were learning and they really were. So I think it’s huge for equity and overcoming some of the biases we have in our classrooms.

Rebecca: That’s a really good example to underscore the importance of that. We had the opportunity to ask Tom what his favorite classroom assessment techniques are. So I’m really looking forward to asking you the same question: what are your favorites?

Todd: I’m one who says over and over again, in education, that we should always be careful about absolutes. Like, there’s never a strategy that always works. There’s never a type of student that’s always the best. Any situation can change. So there’s like 55 CATs in there. And I listened to the program with Tom and I heard him say that he didn’t have his favorites. And then he quickly picked out his favorites, and I’m sure it’s the same with his daughters. [LAUGHTER] So I have three girls, you know, they changed the favorites, I would definitely go with the standards of the muddiest point minute paper, those two are the big ones. Everybody’s done a minute paper, probably at some point. The muddiest point I like, which was the adaptation. And if you haven’t heard this show with Tom, make sure you listen to that, because he explains that very well. I think I like exam wrappers where you go in and kind of have the students talk about the exam experience. But for me, there’s several CATs that are based specifically on assessments. And it’s the thing that we almost never ask students for, and it’s the thing they can give us the best feedback on. So it’s like, when you have group activities, it’s an assessment on how well the students perceived that the activity worked for them. So instead of me just saying, we did a great group project today, you ask the students: “Was this a good group project?” There’s assessments in there about learning interests, what types of things you’re interested in, your study strategies, exam strategy. So you can ask students those things. And I think those make great classroom assessment techniques, to find out the processes that you’re doing, and how the students are receiving those.

Rebecca: Those are good opportunities to respond in the moment in your class, but also probably plan in the future too, to know what to do next time around, perhaps.

Todd: Yeah, I think specifically for exams. And one of the favorite questions I’ve always had on exams, because the students would get to a point where, and I got this, because of one of the assessment techniques I’d use early on, is students always were frustrated that there was something on the test that they studied really, really hard for, and it wasn’t on there. The problem was, it wasn’t always the same. So the students all had different things they studied. So because of collecting that information, and finding out from the students as they were frustrated, I started adding a new question at the end of every exam worth the same as some of the other questions there, was: “Please describe in detail something that you study that I didn’t ask you about on this test.” And students immediately felt better about the exams, they were happier with the test, because they didn’t spend four hours memorizing something that they perceived wasn’t valued on the test, even though the real reason for all of this stuff is life, not the test. But again, that was one from just asking the students: “What do you like and not like about the test?”

John: That’s a really great application of a technique that I hadn’t thought about, I can see how students would feel that their voices are being heard with that type of thing.

Rebecca: …or that their efforts were validated.

Todd: They got their points for their studying, and it just really seriously changed the affect of the class.

John: In the last podcast we did with you last summer, we talked a little bit about how teaching has changed from the time when you and I both started teaching, and this book was written back around the time when you and I were both relatively new at this. What role do you think this book has played in that evolution of instructional practices?

Todd: Well, first of all, I do want to acknowledge and help people just really understand the difference between 1993 and 2023. Some of the things we took out of the book was “display your results on an overhead projector,” there was no mention of the internet anywhere in that book, because the internet wasn’t here yet. Just stop and think about that for a second. So it was a very different time. A lot of heavy lectures. I mean, a lot of times people were lecturing, there were other activities, or other strategies, there was group work and role plays and those types of things. But primarily people are lecturing, because that’s just what people did, primarily. And so the role that this played, I would say, I don’t know if I’ve ever said this before. I think this book had a bigger influence on higher education than any other single book that’s out that I know of. There are some really, really good books out there. But this fundamentally changed the game. Because what it did was it started asking students about their learning, which means it started to really shift the focus from teaching to learning. This did happen to come along, by the way, if you really look at dates, the whole concept of really pushing active learning in higher education, at least in the United States, really took off in the mid- and late-1990s. So 1996 to 1998, we started seeing people like Richard Hake, do some huge studies about this. American Association of Higher Education was changing their themes from taking teaching seriously to taking learning seriously, all of those things happened about three to five years after this book came out. So I think this was the impetus for a huge, huge aspect of the whole push for moving from teaching to learning. I think it had that big of an impact.

Rebecca: In recent months ChatGPT has hit the news everywhere. We’ve heard all kinds of reports in higher education journals and things. accreditation organizations are talking about AI. And many faculty members are also talking about potential harms of AI or other see benefits. What do you think the impact of AI tools will be on higher education practices? …speaking of evolution… [LAUGHTER]

Todd: I don’t have the statistic right in front of me, but there was one I did a presentation just recently on how long it took to get to a million users. And they were talking about some things like Netflix and other things of taking two and a half years, and then six months and stuff. And I believe ChatGPT was five days. So it’s pretty incredible. So Rebecca, I love the question. I think this is a huge issue. The two biggest things that I heard out of the gate was: number one is how are we going to ever have students do their own work again? It was all academic integrity all over the place. And I couldn’t help but talk to a couple of people who were really freaking out about it and I said, “You should really read some of the research about academic integrity already and the proportion of students who are doing things that they shouldn’t be doing, so to speak, there.” And so that was it… academic integrity. And number two was like, “What kind of assignments should we do now? How do we structure assignments to be meaningful?” And I find this fascinating, because they weren’t talking about the learning process. And I still, to this very moment, I’m just starting to hear it now, but I have not heard much of anything about how can we use generative AI to influence the learning process in a positive way. And so I think that’s the huge thing we need to be looking at. So I guess the people who do know me, know me as the individual who writes a lot about helping students to learn and teaching tips in the classrooms kind of things. I think from a cognitive psychologist’s point of view, we can look at some of the foundational things like: we know that repetition is really good, we know that the process of explaining something really helps because it’s a chance for you to practice at retrieving information. We know spaced recall is really good. And there’s all these things out there that can kind of do that. But I’ll tell you, and I’m coming right back to the CATs, but ChatGPT and some of the other generative AI has some phenomenal opportunities. We know that teaching this is good. So I opened up by ChatGPT 4 the other day, and I was practicing and I typed in: I need to learn about metacognition, would it be okay if I taught you metacognition, and then you tell me how good of a teacher I am. And the chatGPT says, “Yeah, this will be fun. So let’s get started.” And I said, “Okay,” and so I explained metacognition. And ChatGPT asked me some questions like, “Well, what does this mean? And what does that mean? …and I acted just like I was teaching somebody. And then when I was all done, I said, “If I were a teacher, right now, how would you grade me?” And I did not get a good grade [LAUGHTER] which was kind of disconcerting there. But I said, “Why did I not get a good grade on this? And then it explained a section that I had completely forgotten, because I was doing this off the top of my head to try it. So I went back and explained that. And then ChatGPT says, “That’s much better.” I think that’s phenomenal. Another one, I will not get into details, but I typed in, or I gave it the prompt… it’s all about the prompts folks… “That I just learned in my introductory psychology class at a medium-sized, midwestern university. I’m a C-level student who’s worked really hard, and what I just learned were about persuasive techniques. Could we pretend we’re on a car lot, and I’m a salesperson, and you’re going to buy a new car, and you think you want to buy it today, but you’re not going to rush into anything.” And ChatGPT says, “This sounds like fun. Tell me about your best selling model.” [LAUGHTER] And I proceeded to explain all these things. And the scariest one is I said, “This Ford van right here has excellent ratings for safety.” And it said, “That’s particularly important for someone with a loved one and two furry friends.” So the learning possibilities are all over the place for this thing. And so for classroom assessment techniques, this is great, because we can actually teach students some strategies, how in the classroom, how to assess whether they’re learning. We can replicate these and have them do them with AI. So we could use AI to do things we couldn’t do before. But we can also do it to model things for students that they can do on their own. So I think there’s some really, really interesting possibilities coming along.

John: And one interesting one is if you ask ChatGPT for your biography, you may very well turn out to be a car dealer in the Midwest.

Todd: You know what, I’m just gonna jump in there and say that is totally true when ChatGPT was about six months old, because I did it. I said, “Write a letter of recommendation for Todd Zakrajsek… and you probably saw this, why you even mentioned it… and I think it had 19 things I counted in there and this term of hallucinations, but 17 of the 19 things were wrong, including where I went to school where I worked and everything else. That was when ChatGPT was a baby. I asked it the same thing just recently, and it was scary, how close it just nailed everything. But yeah, it was for a while.

John: It’s improved quite a bit, but one of the nice things about it is our students come in with really diverse backgrounds and we can provide supportive materials for students when they have some gaps in their prior learning. But, ChatGPT offers the possibility of having something that’s completely personalized to their needs, so that they can ask questions that are specific to the issues that they’re having trouble with, and not what we might have guessed they’d be having trouble with. So, it’s something I’ve encouraged my students to use in my intro economics course, and they appreciated the fact that they were encouraged to use it,

Todd: I think that individualized instruction is going to take on a whole new level. And so I think that’s going to go crazy. But I will say this, and I think it’s extremely important. I’ve been in higher education for 40 years, I have never seen a bigger possibility for an equity gap to be the widest chasm it could ever be. I think we got equity problems that are down the road. And if we don’t pay attention, it’s going to be awful, because I believe that generative AI is going to be a huge learning aid, and it’s going to help us with lots of things, for the people who have it available to them, they will skyrocket and be able to do stuff; the people who don’t have it available… reliable internet, a good computer, or some kind of a device, a safe place to sleep at night… those individuals are not going to be on that tract and they’re gonna get separated. Right now, there’s a couple of grade differences between people who are privileged and not in some classes. That could widen so much. So I think, everybody, we just got to really be careful about how to make sure that this was as equitable as possible for everybody.

Rebecca: Higher ed leadership at various colleges and universities are certainly dancing around the idea of AI and what kinds of policies and procedures to put into place. What would you advocate for? What would you encourage faculty members to advocate for to really make sure that equity is addressed?

Todd: I think one of the very first things we need to do is teach students how to use it properly. Because one thing we know about learning is, the more you know about something, the easier it is to learn something related to that topic, whatever the thing is, and what that means is that learning is not linear, it’s curvilinear. The more you know, the easier it is to learn something, which means you’re going to know more quickly, and as you know more quickly, you’re going to be able to learn more quickly. And this is why people struggle at the beginning with things and then all of a sudden they take off. So if some students know how to use the systems, and some don’t, that’s where we’re going to run into some huge issues. So I think one of the things we can do is teach students how to use it to help you, and when to be careful that it’s not going to help you. The movie Wall-E is a great thing to be keeping in mind. If you haven’t seen the movie Wall-E, it’s a plug for the movie Wall-E, it came out a long time ago, but that concept of the humans that had everything done for them ended up becoming just blobs that laid in chairs all the time, and I think our brains basically do that. If students use AI to do their work, they’re not going to develop critical thinking skills. If students use AI to help them to practice at retrieval and spaced out practice and do those things, they could become very good. So I think schools need to be careful about helping students to understand how this can really help them. That’s the first thing. Then there’s the obvious, helping people to have access to technology, we could have labs open, we could have laptops that we can loan to people, reliable internet, we could have rooms that we keep open 24 hours a day, keep them 24/7 there. Some libraries during exam time, they don’t close for like four or five days. We could have rooms in the library that people can come and have safe spaces to study and work, reliable transportation is going to be an issue, we’re just gonna have to work out how do we address those issues. It’s gonna be a challenge, but we need to be thinking about it.

John: The use of classroom assessment techniques has been growing steadily since they were first introduced. And each time there’s been new educational technology, it seems to have led to an increased use of those. During that time, we’ve seen the introduction of relatively low-cost computers, computer networks, the internet, mobile computers, and so forth. How do you think the availability of generative AI will affect the use and value of classroom assessment techniques.

Todd: I think there’s a couple things that’s going to really, really change. I think for classroom assessment techniques, number one is we’re going to be able to individualize a little bit more. So we can tailor… we don’t have to ask the same question of all the students, we don’t have to look for all the same type of responses. So we can think about it a little more creatively in how we can use it, almost like an individualized instruction toward classroom assessment. Probably the biggest thing, though, overall, is it’s going to allow us to just process data in numbers and levels we’ve never seen before, and particularly free responses. In the past, we’ve often ask closed-ended questions just because if you’ve got a class with 400 students, you can say: “On a scale of one to five, to what extent did today’s lecture help you to understand something?” And the students can pick a number, we could drop that into an excel sheet, or anything, and come up with a number very quickly. We can now say on a muddiest point, “What are you still struggling with?” Take 400 responses, dump them into a generative AI program and have it spit out five things within 30 seconds:” Here are the five things your students are struggling with.” So I think it’s going to allow us to do more qualitative types of things very quickly. And I totally get that this is not hardcore qualitative research with good analysis of the data. But for what I need in the classroom, I think we’re gonna be able to get those responses very quickly and in real time. So what that means is, we’re actually going to be able to ask something like a muddiest point at halfway through the class, and then I could have the students do a quick think-pair-share, while I analyze the information and a matter of three minutes later, they come out of the think-pair-share, to talk a little bit about what they talked about. And then I could say, I see that you’re still struggling with these concepts, so let’s revisit these things before they ever leave the room. That’s never been possible before.

John: When I’ve been teaching classes of two to 400 students, I’ve used the free response option and word cloud. But the word cloud is just highlighting individual words, the ability to do analysis in more detail is going to be incredible. And I think most providers of response software are working on introducing AI, and some of it is expected to be available fairly soon.

Todd: Oh, there’s some great stuff come check GPT and some terribly scary stuff at the same time. By the way before I forget this, because I have to tell you this because I have ADD, that’s one of my favorite phrases.ChatGPT 5,, one of the things I read about that’s going to be coming up fairly soon is the ability for it to launch his own AI’s as needed. That’s the one that we’ve been waiting for, and thinking, “Hmmm, that’ll be interesting.” But yeah, I think that the ability to read through and look through the information is just going to be a game changer. And online synchronous/asynchronous, face-to-face in all environments. We could look at stuff in asynchronous environments in ways we haven’t before as well. I could look at different ways that students are responding, I could ask classroom assessment techniques. In fact, they’re in the book. We have several of them… ways to use CATs in asynchronous environments, and AI is changing how we’re doing it.

Rebecca: Can you give an example of how that’s happening in asynchronous environments?

Todd: In terms of asynchronous environments, everybody likes to go to discussion boards, but quite frankly, discussion boards are boring. We could have students generate something like the script for a commercial. And after they do that, I could have the CAT that comes in to say, “To what extent was this helpful in the learning process for you?” And the student, as they’re developing this thing and as submitting it they could also submit, right along with it, their CAT that comes with it, and then I could be reading the CATs as they come in one by one right behind each one of them. And so the concept, there, would be kind of a real time CAT analysis that’s not waiting until there’s a whole group and then looking at the group. Because typically, when we think about these CATs, it’s like, the 400 students fill out a muddiest point and we analyze the 400 students. We don’t need to do that. In an asynchronous environment, we can have these CATs coming in and we can be analyzing them as they come in. And we could even add them to the previous ones, to group them if we wanted to. But the bigger one is, I can find out how the students are doing as they’re doing. So that’s probably one way to look at it.

Rebecca: One of the things that you mentioned in an in-person or a synchronous setting was being able to analyze results mid class. And that’s in part because a lot of our students have mobile devices and technology that they bring with them that they haven’t had before. Do you see other ways that these mobile devices are also changing the way instructors might implement CATs?

Todd: The way you just mentioned, though, I can’t think of many other than the concept of as an information gathering device. I will say, just because I happened to read this study just a little bit ago here, the devices are very interesting, in a sense that we’ve seen high schoolers tend to have the same amount of time they spent with their peers was very consistent until about 2012. And in 2012, the numbers started dropping off pretty quickly. And then COVID dropped it even faster, but the slope was already there. And the point here is that people are turning their attention to devices so much that they’re not talking to each other exhibited by when you see two people at a restaurant having dinner, and each one of them are on their device. There’s all kinds of psychology about how those things happen. But the overarching thing that’s the issue is that we’re having students at levels we’ve never seen before disengaged in the classroom, even when we’re trying to do CATs and engage. And part of that is because of the devices. So one of the things that’s kind of interesting about this is when you say: “Okay, everybody gets your devices out, and now we’re going to use them for a CAT.” And then we stick with that and how we use that data makes a big difference of whether they stick with it or go back to what they were doing. Strategically, it’s really helpful to use those devices to engage the students.

John: One concern with generative AI is many of the types of assessments we’ve used before can be answered very nicely by generative AI. What might we do to reduce the likelihood that students will use generative AI as a substitute for learning?

Todd: I think it’s really, really important to talk to students about the long-term implications. I know this is not going to be for everybody. I totally get that. But one thing I’d say to students is: “I don’t understand why anybody would run, why would anybody go jogging? You can just get in your car and get there faster.” So if you kind of pitch that to them, that idea is for most of the students, if there are any kind of health-related fields, or if they want to have a cardiac system later in life, exercise is just one of the most important things you can do for your body, by the way, in terms of that getting 150 minutes per week, and just higher respiration. If you take that away, it is bad for the human body. And so the response is the same thing is true of this. Just talking to the students. If you use generative AI to answer your homework problems, to develop the quizzes, to write a poem, and I’ve done all of these in workshops, where I’ve said to people, I can do this real quickly. I don’t care who’s in this room, I can ask you to write a short story. And I will crush you with generative AI. The problem is that, if I do that, I don’t learn how to think for myself. And so I think the biggest thing we can do for students, number one is to build community in the classroom, to number two, to tell them what this all means, and then number three, ask them, ask them, how it’s helping them, ask them how it’s hurting them, how they believe these things are working. And those are classroom assessment techniques, we can use these assessment techniques to find out to the extent that they’re doing this. While I’m on my little diatribe, we will never be at this spot again in our lives. We have just developed something that can totally short circuit cognitive processing and critical thinking. But we did it with people who have critical thinking skills and cognitive processing. So we developed something because earlier in our lives, we had to learn in the ways of joggers, we had to develop our systems. And now we can talk about the automobiles that would move as faster. The students coming along are going to step right into those automobiles, right into the AI, and if they’re never told that it’s important to go out jogging, they’re not going to develop the systems that are going to be needed later. And so again, just think about this for a second, we’ve developed a system using a system that may disappear if we’re not careful. And so I think that’s where we can change it. So it boils down to community, why we’re using it, and then assessment questions of how it can be used, and where it’s good and bad.

Rebecca: I think students really enjoy having conversations about AI and exploring how it’s useful and not. I have had activities in my class where we intentionally used AI to see what it was like and when it would be useful and also analyze where maybe they tried to use it and it was totally not helpful, and why it wasn’t. And they really appreciated those kinds of conversations and learned a lot from those.

Todd: I think the students do. And we could pick different spots in history. You go back to Socrates and his whole belief that if you write things down, it weakens the mind. It makes sense. If you have to learn it, you’re going to be much stronger than if you write it down. But I can’t imagine right now teaching without students writing things down. And Samuel Johnson came along several 100 years ago and said, “There’s ready availability of books. With books all around, why would we really need to teach this stuff, they can just go get a book.” I can’t imagine teaching without books right now. So these things that everybody got scared about, or thought that just going to change just became integrated. And when the internet came about, we talked about this a while back, in the sense that many of us were teaching before the internet actually showed up. But when it showed up the people who were teaching were freaked out. “How am I supposed to teach when the students can go and get anything they want off their computer?” And now I can’t imagine teaching without the internet. I believe five years from now people are going to say I can’t imagine teaching without writing things down, without using some kinds of print format, without using the internet, and without using generative AI. I don’t even know how I would teach without it. What that’s going to need is the same stuff we’ve done in the past, that’s how do you teach well with the Internet? How do you teach well with generative AI?

John: It’s an exciting time to be in the midst of all these changes. And it’s going to be interesting to see how we all answer those questions as we move forward.

Todd: It is, but before we wrap up, because I can tell that tone, [LAUGHTER] I want to put a general call out that there are several people who are saying: “I wish I could retire right now” or “This is a great time to retire.” The statement I really want to make is this is a hideous time to retire if you’re really good at what you’re doing. Because we have never needed humans with really good critical thinking skills as much as we do now, maybe we will later, but to date I don’t believe we ever have. And for some of the people who are saying “I just don’t want to teach anymore. This is awful…” we might need you more than we’ve ever needed you, so I don’t like this concept of the mass exodus of certain people.

Rebecca: It’s an important point.

John: One other thing I will add is Paul Samuelson once said, in describing the way in which models of the economy evolved, that “funeral by funeral, the science makes progress.” So there is the counter argument there too, that people who are very tied into the old ways of teaching do need to either adapt or perhaps they’ll be replaced by people who are more willing to try new alternatives.

Todd: Do I smell a little Kuhn in there? The revolution of science. Ah yes.

Rebecca: And we always wrap up by asking: “What’s next?”

Todd: I’m trying to keep busy. I’m kind of on a roll right now. And I want to tell you, I’ve been writing books and the only reason I’m unhappy about ChatGPT and gen-AI come along is people are going to assume that I’m using them so that I can crank stuff out. But thus far, I have not used any of this generative AI to write any of these last books that I’ve done, these five books in the last five years or something, and enjoyed working with Tom on this one. But I got another one coming out in about four months. I just turned in the final pages for that And it’s the Essentials of the New Science of Learning: the power of learning in harmony with your brain. That book should be out in a couple of months. And so the what’s next after that, I am working on a book right now for helping with neurodivergent learners. We’ve joked around in the past, I’ve got ADHD is about as bad as it gets. If it were a competition, I could come in probably close to first place, and also on the autism spectrum just a little bit too. And when I started sitting down with some of my colleagues who also have ADHD and autism spectrum, it occurred to me, we don’t talk to students enough about these things. And I think the challenge of that is that the students believe that they’re alone, and they’re not sure they can do things. And I was talking to a student fairly recently that said, “I now believe I have a shot, here you are with a PhD, you’ve written books, if you’ve got ADHD, as bad as you claim you do, maybe I can do something.” And then that’s when I really really was getting serious about we need to help folks out. And the topic I’ve been playing around with a little bit in a couple of workshops is The Ones Too Often Left Behind as the title I’m using. And it’s the students that aren’t built for the system that we developed. And so I think that we need to treat some folks out there a little differently. And I think we can really build up the pool of intelligent folks by helping to teach the people who just learn in a different way.

Rebecca: It’s really important work to have models and put models out in front of students because they need to see themselves in whatever discipline, field, etc., that they want to pursue. I’m excited to hear more about that.

John: And when will these books be out? You mentioned the timeframe for one of them.

Todd: The Essentials for the New Science of Learning should be out in September. And then the other one, if everything goes well, is probably looking at a February date that it would probably be available.

Rebecca: Well, it sounds like we’ll be talking to you soon then, Todd.

Todd: We may be. When you got ADHD pretty bad, you just can’t really predict when that books gonna be. But yeah, I’d love to chat with you when it does finally emerge.

John: And what are you going to do next week?

Todd: You know what? You gave me a great idea. I think we should do a book on procrastinating people with ADHD. [LAUGHTER]

John: It’s always great talking to you. And we’re looking forward to each of these projects coming to fruition and we will be talking to you about each of them, we’re hoping.

Todd: I appreciate the opportunity to come and chat with you. I think you two do a phenomenal job with your interviews and the programs that you pull together on really good topics and I just am honored to be one of the guests.

John: Thank you.

Rebecca: Yeah, thanks. We always enjoy talking to you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC] </p

346. Classroom Assessment Techniques

Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, Thomas A. Angelo joins us to discuss the origin of these techniques and evidence concerning their efficacy.

Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993.

Show Notes

  • Cross, K. P., & Angelo, T. A. (1988). Classroom Assessment Techniques. A Handbook for Faculty. Jossey Bass Wiley.
  • Angelo, T. A., & Cross, K. P. (1993). 2nd. Ed. Classroom assessment techniques. Jossey Bass Wiley.
  • Angelo, T.A. and Zakrajsek, T. (2024). 3rd. Ed. Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers. Jossey Bass Wiley.
  • Schwartz, Charles (1991). “An Academic Adventure.” March 4. – A description of the origin of the minute paper by it’s originator.
  • Chizmar, J. F., & Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. Journal of Economic Education, 29(1), 3–10. (the article that John referenced as his first reference to this topic)
  • Mazur, E. (1997). Peer instruction: A user’s manual. Pearson
  • Mazur, E. (2014). “Confessions of a Converted Lecturer” – a recording of a presentation by Eric Mazur at SUNY Oswego, 5/19/14
  • Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses. John Wiley & Sons.
  • Pam Hook and Josie Roberts (2018). “The who, what, when, where, and why of SOLO taxonomy.”
  • Angelo, T.A. (1993). “A Teacher’s Dozen—Fourteen General Research-Based Principles for Improving Higher Learning.”
  • Pre-order for 3rd edition of Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers.

Transcript

John: Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, we discuss the origin of these techniques and evidence concerning their efficacy.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Thomas A. Angelo. Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993. Even if you’ve never heard of these books, you may have included some Classroom Assessment Techniques in your teaching toolkit. Welcome, Tom.

Tom: Good to be here.

John: We’re very happy to talk to you. Today’s teas are:… Tom, are you drinking tea by any chance?

Tom: No, I was drinking black coffee.

Rebecca: It’s one of our most popular forms of tea [LAUGHTER] as regular listeners know. Today have some English tea time, John. How about yourself?

John: …and I have an Earl Grey tea today.

Tom: Excellent.

Rebecca: We’ve invited you here today, Tom, to discuss the third edition of Classroom Assessment Techniques, to be published shortly. These techniques described in earlier editions of this book have been widely adopted by faculty in all disciplines at all different types of institutions. And a lot has changed in higher education since 1993. But this book is still the go to reference on classroom assessment techniques, which is pretty impressive [LAUGHTER] for a book that was published 30 years ago. Can you talk a little bit about the origin of this book?

Tom: Well, I think the origin of the book goes back to the fact that when I was a young teacher, I recognized pretty early that many of the things that I was doing didn’t work and didn’t work for my students, and that I had to accept that things that had worked very well for me as a student weren’t working for many of them. So I began to experiment with asking questions in a variety of different ways and getting students to jot down answers anonymously. And through that, I really learned more about teaching and learning, I think, than from anything else that I did. And one of the things that I learned was not to assume. So, classroom assessment techniques are things that I’m sure almost every teacher has done. But what we did in the 1980s. K. Patricia Cross and I, was to really formalize those. And we did that in order to create a resource for teachers so that they could find simple tools that would help them ask students the kinds of questions that would help students learn better, and teachers teach better, as they went along. So it really began at that time, and it was kind of an accidental start, in many ways. We were doing it as part of a big project, and what was then the assessment movement, and it just kind of took off.

John: Most of our listeners are probably quite familiar with classroom assessment techniques and have seen them at lots of workshops and have probably, as Rebecca noted before, used them in the past. Could you define what a classroom assessment technique is, for those listeners who are new to the concept?

Tom: Well, in a formal way, a classroom assessment technique is a formative assessment technique. It’s a way of gathering data to use to improve and inform practice, rather than to make judgments. So we have too, most of us, to give students tests and exams and assign papers, and grade their work, evaluate their work, for grades and marks. Formative assessment techniques, on the other hand, are tools that we use to gather data along the way, before those moments in which we grade them, or mark them, so that we can help students get ready and improve and succeed when they’re going to be graded and marked and so that we can, at the same time, find out where they are and how they’re doing in order to help them improve along the way. So they’re really ways of diagnosing, in a way, students’ learning.

Rebecca: One of the most popular classroom assessment techniques described in your book is the one minute paper. There have been many studies of the effectiveness of this technique and it’s been widely adopted in many disciplines. Why was this technique, in particular, been used so extensively?

Tom: Because it’s simple and easy to use, probably, and it takes almost no time. But seriously, the one minute paper, the minute paper was invented, to the extent these things are invented by Charles Schwartz, in the 1970s, at UC Berkeley, and he used it as both a way to take attendance (in his case, he was asking students to jot their names down), but more importantly, to find out whether students were learning from his physics lectures, in his huge physics classes, what he hoped they were learning. And that idea spread through UC Berkeley really fast. And so soon, there were dozens and hundreds of people using these two questions. And the two questions, famously, were: what are the two or three most important things you learned from today’s lecture, in his case, and what questions do you still have the end of the lecture? So students would jot these down on those strips of paper, remember, there was no internet and no computers, really, in classrooms in the 1970s. And then he would collect them, and read through a sample of the 1000 or so that he had every lecture. And from that sample, he would gather ideas about what students understood and didn’t from the lecture, and also what questions were most common. And then he would begin his next class meeting, by going over some of those points that students had had trouble with, or not understood, or missed them completely. Sometimes it happened that a 1000 students didn’t get one point that he thought was critical, and also answering one or two of the most important questions. And what he found was, he could do all of that in five minutes at the end of class, and about five minutes at the beginning of class. And I think it’s that efficiency that has made it so popular.

John: My first exposure to this was a study that was published in the Journal of Economic Education way back around the time of your last edition. I did try to find it, but I couldn’t find the citation for it, but what struck me was that the instructor had used it in one section of the course, had not used it in another section, and found that there was a very substantial difference in learning outcomes between the two groups, which he attributed to the one minute paper. And it’s been incredibly commonly used. But one of the things that really struck me is that at the time when you came out with this book, including the one-minute paper, and so many other classroom assessment techniques, most instruction at the time was the chalk and talk type where people were at a blackboard, maybe a whiteboard, if they were a little bit more advanced in the technology, where most assessment was done with high-stakes assessments. So I think your book was a fairly important factor in helping people shift to more formative assessments. Why have these techniques become so widely adopted?

Tom: Well, I think that one of the reasons why people have found classroom assessment techniques and other formative assessment techniques, there are other kinds, so useful and so important, I think there are several reasons. One is that the student body in the United States and across the world has become ever more diverse in the last 30 or 40 years. And it turns out that no matter who we are, as teachers, most of our students are going to be pretty different in terms of their experiences and their interests and their skills than we are. And that level of difference has probably grown over the years, at least our awareness of it has grown. So I think most teachers now realize that it’s very important to check in on students and see how they’re doing, before we get to the midterm or before [LAUGHTER] we get to the end of the course, and to check in regularly and not to assume that students are understanding our brilliantly clear presentations or the material that we give them, or the readings, or the problem sets, and to really check early on to see if it’s making sense to them, and what kind of sense it’s making to them. So I think that there’s much more realization of the diversity of human learning, and of our students, than there was 40 years ago or 30 years ago. And I also think that as teaching has evolved, especially as it’s become online, people have recognized that we need data. We can’t just depend on students volunteering that data, they often don’t know that they don’t understand. And that that data, used well, can help us and can help them.

Rebecca: I think one of the things that you’re talking about in terms of assumptions is reminding me of some of the conversations that we had early on in the pandemic, where instructors we’re talking about not being able to see the sea of faces to know whether or not students are learning because maybe students aren’t nodding along or providing some visual feedback, but classroom assessment techniques are really important because they allow opportunities to gather that data, which is probably far more informative than the head nodding.

Tom: I think there are maybe a couple of other things that are important about this that people discover. One is that by stopping to ask students to reflect on their learning and respond, you give them time to think. And you prompt them to think about their learning. And from my point of view, forever, the most important thing about doing this has really been to give students tools to become more aware, we could say more metacognitively aware, of their own learning, and to better take responsibility and control or at least direction over their own learning. So what I found is I’ve had many, many, many very bright students who had never been taught any kinds of techniques to think, or to ask themselves questions, or to monitor their own attention, and having those sorts of tools, those sort of metacognitive tools, I think empowers students to make decisions about their own learning, and also to improve their own learning in those areas where it matters most to them, which may not be my course.

John: One of the things you mentioned was that the one minute paper was originally developed in a class with 1000 or so students. And one of the nice things about many of these techniques is that they can scale that way. Now, there was a time when the book came out when this was all done on paper. But now we have so many electronic tools that we can use to do this. Instructors could put up a QR code with a link to a Google form or something where students can give their feedback within a minute or so or we could use polling techniques, and so forth. And many polling systems have built in one-minute questions or some other classroom assessment technique as a default strategy. So this impact has been pretty substantial. And it’s really nice for instructors, not just for students, having that time for reflection, but also for instructors to get some feedback on what students have learned. Because if you just call on students, you may happen to get the only student who understands the material responding to a question or you might get the only student who hasn’t understood the material. But with these formative assessments, you can get a good feel for where students are. And they also get some feel for where they are in a way that they might not if they would just passively sitting there listening to a lecture. So these techniques have, I think, become increasingly more common from the time of introduction. So these techniques have held up pretty well. But you now have a new edition of this book, a third edition of this book. Could you describe some of the changes in the new addition?

Tom: Well, obviously, and you’ve touched on this, technology has changed. And that’s had an impact on teaching, and an impact on learning. So one of the things that I did, in going back to do this book after so many years, was to take a look at first what was out there, what had people published on this. And there are literally about 3000 published articles of various kinds and various quality, about classroom assessment techniques. There were about 300 that I thought were of sufficient kind of quality and rigor to be able to use. And in the chapter on the minute paper, I refer to 150 articles and studied them very carefully. There are many, many books that refer to the classroom assessment techniques. And I also read [LAUGHTER] all of those references. So I took a few years to work through this material and new things were coming in all the time. So one thing that differs is that we have an evidence base across many, many disciplines for what seems to work and what seems to work well. And also much useful information on how to use these with various technologies. So how best to use formative assessment with online, fully online technologies on totally asynchronous classes. And so that’s been really valuable. And those are experiences that I couldn’t have all of those experiences. So it’s terrific to draw in those from other teachers. The other thing is people have invented new techniques or developed new techniques over time. And so in the third edition 25%, or about 15 of the 55 techniques, are ones that didn’t exist in the second edition. And that’s about one quarter of all the techniques in the new book. So one of the reasons why we have different techniques in the 3rd edition is that some of the ones in the second edition weren’t much written about, weren’t much commented on, and they seem not to be much used as far as I could tell. So I’ve picked up the ones that were most used and most written about, and most referenced so that people in different disciplines can find those techniques. And I’ve also found another 15 techniques to kind of fill those gaps and extend it a bit.

Rebecca: Your book offers many, many assessment techniques. I’m curious what your favorite few are that you’d like to introduce instructors to?

Tom: Well, just like I can’t say which of my daughters are my favorite, I do have favorites. I’ll be honest. I have always used the minute paper ever since I learned about it all those years ago, and I learned about it really when I was at Berkeley, in the mid 1980s. And I learned about it, as everybody else there did, kind of by chance, and thought, “Wow, why didn’t I do this? Why didn’t I think of this?” So the minute paper, the muddiest point, which is a variant of the minute paper, which is developed by a really famous and eminent statistician at Harvard, named Frederick Mosteller. And Fred Mosteller heard Pat Cross talk about this, when we were both at Harvard, and being the kind of person that he was he immediately experimented with it. What he found was that it was more useful for him, in his courses, to just ask one question, and that question for him was “What was the muddiest point In today’s class? What was least clear to you?” Many other people have used that as well and written about that. That’s another favorite. A third one is one that I’ve used a lot, being a statistics teacher. And that’s predict, observe, explain, which is often called POE. And that’s one that I’ve found useful in many situations. And I’ve had colleagues who’ve used that even in literature courses, and other courses where they’ll say to students, “let’s stop at this point in the book or the film, and predict what’s going to happen next, write that down, then we’ll observe it, and then we’ll try to explain why that either met our predictions or didn’t meet our prediction.” And while it’s obvious how people do that in science, and it’s part of the scientific method, it can be used in all kinds of circumstances. And it’s one that I’ve used with teachers who I’ve worked with, about classroom assessment and about formative assessment. I’ve asked them to predict what they thought would happen, then observe it, and explain what did happen and whether or not that met their expectations. And that’s been a very useful tool for me, I think, and the teachers I’ve worked with. So those are three good ones. I’ll give you one more, if that’s helpful. And that’s something that we call the directed paraphrase. So three of those four were in the first and second versions of the book, predict, observe, explain is in the third edition, but directed paraphrase simply asks the questions that are sometimes called journalistic questions, who, what, when, where, why, who did what to whom, when, where, why, and how? And it asks students to answer those questions about some phenomenon that they’re observing, or to answer them about an essay that they want to write, answer them about any kind of question. So giving students a framework of questions that they can work through, to make sure that they really understood either a book that they’ve been reading or an experiment that they’ve been doing, has been really valuable to my students. And I’ve said to them, “Look, this is something that you can adapt and take with you anywhere to use to make sure that you understand what’s going on fully. And to find out which part of it maybe you really can’t answer, and need to do some more work on.

John: In terms of the predict, observe ,and explain thing. I took a MOOC that was put together by Dan Ariely on behavioral economics. And one of the things he talked about was how he would often present the results of an experiment where the results were not what most people would have expected. But then he’d asked people for their reactions only after he summarized the results of the experiment. And they all nodded and said, “Well, that’s exactly what they would have thought would have happened.” So he started revising his presentations, where he asked people to make a prediction, and then revealed the result. And it got people much more engaged. And the same type of thing is done by Eric Mazur with his polling technique, where he asked people to answer a question where the question is not obvious and where there’s going to be some disputes. And then once they’ve committed to an answer, they work with other people and he goes through it again, with a little bit of peer instruction. But the level of engagement when students have made some type of a prediction of what they think the answer is, and then they have an explanation of what the answer is either given by the instructor or by other students, the level of engagement and interest just seems to go up fairly dramatically. So I can see how that would be particularly effective.

Tom: Yes, and I’ve learned a lot from the work of Eric Mazur, as have many others. And I think that that’s a critical thing. Because one of the things that we need to help students with, and help ourselves with, is managing our inbuilt biases and that issue of when you hear something new, feeling that you’ve always known it, even though you haven’t, is a cognitive bias that’s common, I think, to all human beings, myself included. So doing something like that: predict, observe, explain routine gives us a chance to really check and see if we know the answer. And then to find the answer, or discover an answer. And then to realize, because we have a document, we have a record, we have data about what we knew in the first instance, to recognize that we’ve learned something and what we’ve learned and that we might have been misinformed or unclear in the beginning. And I’ve suggested to my colleagues, one of the reasons that students don’t value their education as much as sometimes we think they should, or value our course as much as they think they should, is because when they get to the end, they may think “I always knew that, that was obvious, I knew those things.” And so we can help students realize that there’s a start point. And then there are many points along the road to learning, and that they may have learned more than they thought they did in a given course or given program. And I think that’s important for us in terms of demonstrating the value of education. And I think it’s important for students in terms of understanding and developing some humility about how it is that we learn and how hard it is and how long it takes to learn anything that’s really valuable.

Rebecca: I think, in my experience, students really appreciate the opportunity to slow down and reflect on their learning, because they’re often not given that opportunity. And when you’re doing that throughout an entire semester, they really do get excited about how much they’re learning and they get energized by that and seeing that they can achieve things over and over and over again.

Tom: I agree, and what we found with most of the studies of classroom assessment techniques, the simple ones is: first, classroom assessment techniques can’t really be separated from everything else that’s going on in teaching, they’re just one little part of the whole teaching and learning spectrum. They, at best, seem to make about a 5- 8% difference in how much students learn, which is about half a grade. And I think that’s important, but I don’t want to overstate it. So they’re not a panacea and they’re not a silver bullet, but they’re something that can lift up, to some degree, students’ performance and their learning. And the last thing I’ll say is nothing works for everybody. And so there is no technique or no series of techniques that every single teacher or every single student that I’ve worked with found valuable. And so one of the points of having 55 techniques is to hope that there’s one or two or five in there that people will actually find that work for them.

Rebecca: So you just reminded us about there’s 55 techniques, how do you imagine an instructor using your book? Is it a read from the beginning to the end? Do you have it organized in ways that instructors can easily dip in and out?

Tom: I can’t imagine reading it from the beginning to the end, and I don’t think anyone ever has. That would surprise me, except maybe the proofreaders [LAUGHTER] and they weren’t worrying about what it meant. So it’s organized so that there are many ways, there are actually eight ways to find a technique that might be useful for a given situation or a given teacher. One of those ways is the simplest and that’s alphabetical. So if someone has mentioned a technique to you, you can easily find it that way. The others are a bit more complicated, but I think the most valuable for many people is an index of all the techniques that have been used by discipline and documented by discipline, and the examples by discipline in the book. And there are a couple of 100 examples in the book of how people use these in different disciplines, kind of from A to Z, from anthropology to zoology, and everything in between. So those are ways that many people use. There are also a couple of inventories in the book. One is something called the teaching goals inventory, which was in the first and second edition, and has been used in lots of ways by many people and written about and research has been done on the teaching goals inventory, and it is exactly what it sounds like, an inventory of what we think we’re trying to teach. For this book, I created what I think is kind of a more contemporary take on that, and that’s something that I call the course learning outcomes inventory, or CLOI, and it’s really flipping that and saying, okay, it’s important what we want to teach, but let’s look at it from the point of view of what do we want students to learn? And so the CLOI, the course learning outcomes inventory, the statements are statements of learning outcomes. There are 57, they’re divided up into kinds of learning outcomes, and people can use it as a self assessment. Those items both in the teaching goals inventory and the CLOI are all linked to classroom assessment techniques in an index. So if you pick number 47 goal or number 47 outcome, you can find techniques that are linked to that. Lastly, we’ve done indexing by Bloom’s Taxonomy, and by Fink’s taxonomy of significant learning outcomes, and also by the big solo taxonomy, which is one that’s much used in the Commonwealth world, in the British and Canadian and Australian and New Zealand systems. So there are eight ways to skin a CAT in this book, or to find a cat at least. And I hope those will be useful to folks. And I think different kinds of indexing work for different kinds of people.

John: You mentioned that when this book first came out these things were done on paper. But one of the things that’s grown quite a bit over the years, as you’ve mentioned, is online instruction. Could you tell us a little bit about how some of the techniques can be used in an asynchronous environment?

Tom: Well, I think there are huge advantages to the technology that we have now compared to having to do all this on paper, having to read all those papers after class, or all those index cards, in my case, after class and then waited until the next class, when I saw students again, the time lag was a real problem. So being able to respond almost immediately, or in relatively short time to students in asynchronous courses, to their questions, and to their confusions, and to also let them know that they’re learning some things really well… they really got that… and that they’ve come up with some ideas that I never would have thought of that are excellent. To do those things in real time, I think, is much more effective in terms of teaching and learning than waiting two or three days or a week. Also, as you mentioned, we have technologies for polling students and for analyzing their responses that are ever more sophisticated and powerful. And so those save teachers a lot of time and work. I was always going through things counting and deciding on the number of times people had mentioned things on paper. This can all be done and is all done for you if you want to do it. And that, again, allows teachers to focus on what really matters. And that is: what’s the meaning of this data? What are students telling me? …and to make decisions about when they’re going to respond and what they’re going to respond to, because you can’t respond to everything in data if you have more than two students. And most of us did, and do. So I think that asynchronous learning has huge advantages, it has some downsides, but it has huge advantages. And one of those probably is that it makes it more difficult to assume that by looking at students, you can tell that they’re learning or not, which I’ve always found an odd idea, but many of my colleagues at least used to seem to think that if students were nodding and looking at them, that they were just tracking with them and understood everything that they were saying. And I used to say to my colleagues, “I’ve known many smilers and nodders that probably were thinking of something else entirely, and classroom assessment proved that to me.

John: I’ve seen that so much. We’ve heard so many faculty during workshops over the years who say, “Well, I can tell whether they’re learning just by looking at the expressions on their faces.” And I’ve never found that. There are a lot of students who will be nodding and smiling. And then when you ask them a question, or they ask you a question, it’s really clear that they don’t have the basic idea. At an economics conference, I was presenting a paper and there was someone sitting in the first row taking furious notes while I was talking. So I went up to the person after and asked them what they were taking notes about, because I was curious. And they were actually preparing for a session later in the day. But they looked really engaged throughout the whole thing.

Tom: They were engaged, just not with your lecture. [LAUGHTER]

John: Yes.

Tom: So I think the converse is also true. And that is their students, and people, who don’t appear to be paying any attention at all, or who are frowning or who have had their head down, who are actually paying total attention and learning a lot. And that’s one of the things I learned as well, early in my career, was not to make assumptions based on those sort of superficial impressions. And it’s hard not to, but when you actually learn that some students who appear not to be engaged are engaged in their quiet way. And that’s a humbling lesson, or it was for me.

Rebecca: One of the things that I really like about your book, and also the way the collection of the CATS all in one place is the flexibility, the flexibility of navigate the book, the flexibility of options for the instructor, but also the flexibility of options for learners as well, in different contexts. So I just wanted to highlight that flexibility piece that we haven’t really talked about quite yet.

Tom: Well, that’s the hope. And if this is created as an ebook, and I expect it will be, then I think that will offer people even more flexibility and more sort of usability. It will be a more user friendly book. And again, there are huge advantages to the kinds of technologies that we have available now that we didn’t then. So I think the real key to this is one of the axioms in the book. And that is it’s really about adapting, not adopting. So there are many, many tools in there that people have adapted in a variety of ways, renamed, and that’s all fine. And that’s all exactly what should happen. You should find them, adapt them to their uses, and if they don’t like the names of the CATs, they can change the name.

John: Would this be a good subject for a faculty reading group where people from different disciplines could get together and talk about how either they’ve applied these things or how they might apply them in their classes?

Tom: Well, it has been from the beginning. And we began this whole process working with groups of teachers, initially in community colleges and state colleges, who were interested in assessing student learning. And, of course, the point being trying to improve student learning. And what we did basically was use the teaching goals inventory at that time, you could use the course learning outcomes inventory, with those teachers to get them to focus on what they really wanted to know. And in the book, I’ve written about, in one of the chapters, four different ways you can approach using classroom assessment techniques and figuring out which one to use. But everybody had ideas and theories and questions or problems that they wanted to try to solve. And so we started with that. And then we said, if that’s your question or problem or theory or outcome or goal, what might be a good tool to try to help you gather a reasonable amount, a manageable amount, of data from students that might help you learn, and might help them learn? And that’s really where we began, and those discussions and the collegiality that came from them, and the interactions that came from that, were always the most valuable part of that professional development. That when we ask people: “What did you take away from this?” The first thing that they said was not a bunch of techniques. The first thing that they said was the interactions with my colleagues, the relationships with my colleagues, learning from my colleagues, and then they talked about the actual content that they had learned and the things that they had done. So I think those interactions and those relationships that people can make, by having a group focused around this. This is an excuse to get together and talk about teaching and learning. I think from our experience, we found any excuse to get folks together to talk about teaching and learning, especially if it’s across disciplines, is always fruitful.

Tom: …particularly if there’s food. [LAUGHTER]

John: Yes. The third edition list you as author with Todd Zakrajsek, could you tell us how this collaboration came about and how it worked?

Tom: I think this was, in part, an insurance policy for the publishers. Given my advanced age, they wanted to make sure that they had someone involved in the project, who could take over if I just kicked the bucket. So, in the last six months of the book project, after I had finished writing the manuscript, Todd Zakrajsek joined with me and was really invaluable in finishing the process, in coming up with great ideas and suggesting great ideas, especially for the use of technology and AI with CATs, and in helping me manage the process with the publisher, which is much more complicated than it was 30 years ago. He also wrote the discussion questions for use by teachers and faculty developers, and I think those are going to be very helpful to folks.

John: We’re recording this in May, and the book is scheduled for release in June. We always end by asking, what’s next?

Tom: Well, I’ll answer that. But I’ll first say to teachers, what I always say, and that is, “Feel free to ask your librarian to order the book, so that you can use it, and your colleagues can use it.” Don’t tell Wiley and Jossey Bass that I said that, but I think that’s important. So what’s next for me is I’ve been working for a while with a group of people in, strangely enough, pharmacy education. I’m not a pharmacist, and I’m doing research with graduate students on what we call core concepts in developing what are the core concepts in pharmacotherapy? And that’s been very interesting for me. I’ve learned a lot from that. And so I’m ongoing with that. And we’re beginning to work on educational materials related to that. So that’s one of the things that I’m doing. And I’ve actually, I don’t want to commit to this too much, but I’m thinking about writing a book based on an article that I wrote that people have found useful, called “A Teacher’s Dozen.” And so I’ve updated that. I’ve been working on that actually for a few years, and something may come of that.

Rebecca: Well, we’ll be excited to find out what that is soon, maybe. But you haven’t committed, but maybe. [LAUGHTER]

John: Well, thank you. It’s been great talking to you. And my copy of the book is on preorder. And I am looking forward to receiving that.

Tom: So am I. [LAUGHTER]

Rebecca: Well, thank you so much for joining us. It was great to hear about the origin story of the book and how it’s evolved over time and I know a whole other generation of teachers will appreciate the new edition as well as teachers who have been practicing for a long time.

Tom: I hope it will be useful. Listen, it’s been a pleasure and a privilege to talk with you both. Thank you for giving me this opportunity

John: We should note that this is the first of two podcast episodes discussing this book. In next week’s podcast we’ll be joined by Todd Zakrajsek, who will be discussing his role in this work.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

344. Failing Our Future

The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, Josh Eyler joins us to discuss research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming  book, Failing Our Future: How Grades Harm Students, and What We Can Do About It.

Show Notes

  • Eyler, Joshua R. (2024, forthcoming). Failing Our FutureL: How Grades Harm Students, and What We Can Do about It. John Hopkins University Press (pre-order link)
  • Eyler, J. R. (2018). How humans learn: The science and stories behind effective college teaching. West Virginia University Press.
  • Brookhart, S. M., Guskey, T. R., Bowers, A. J., McMillan, J. H., Smith, J. K., Smith, L. F., … & Welsh, M. E. (2016). A century of grading research: Meaning and value in the most common educational measure. Review of educational research, 86(4), 803-848.

Transcript

John: The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, we explore research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Josh Eyler. Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming book, Failing Our Future: How Grades Harm Students, and What We Can Do About It. Welcome back, Josh.

Josh: Thank you. Thank you. Great to be here.

John: We’re happy to talk to you. And it’s been a while. Today’s teas are:… Josh, are you drinking tea with us today?

Josh: Well, you know how I roll, you guys, I don’t have tea, but I have some lovely water. [LAUGHTER]

Rebecca: So tasty. I have a Scottish afternoon today.

John: And I’m trying to cut back on caffeine, so I just have a peppermint tea today.

Rebecca: Well, if you wanted to cut back on caffeine, you could join Josh with water. [LAUGHTER]

Josh: That’s true. [LAUGHTER] Come join the club, John. [LAUGHTER]

John: I suppose that probably would not be an unhealthy thing to do. [LAUGHTER] And I have been drinking a lot more water recently.

Rebecca: So, we invited you here today to discuss Failing our Future. Could you tell us a little bit about how this book project came about?

Josh: Sure. Yes. So it has a couple of origin stories, actually. One is that when I was writing How Humans Learn a few years ago, I was working on the final chapter on failure. And I kept coming across research on grades and grading and the obstacles that they set up to learning. And certainly if you’re thinking about failure as a tool for learning, you immediately are confronted with the fact that in systems that prioritize grades, they push back on the natural cycle of learning, where we try something, we fail, we get feedback, we try it again. Grades are set up in the opposite direction, they arrest that process before it can really play out. And so that is what piqued my interest. And I don’t know that I could say honestly that I thought I was going to write a book about grades at that point. But I knew that it was an important element of what I was discussing. So I’d say a few years after that, I think what we were seeing was a lot of discussion about grades, a lot more experimentation with it. That led me to want to write a book for a broader audience, one that certainly included educators, but one that cut across the K-12, higher ed divide one that was also for parents and policymakers and students, one that really took a bird’s eye view of the larger conversation about grades, trying to pull all of what we know together so that we could move forward as a community to try and enact change. So that’s what got us here. And the process was certainly longer than I expected due to the pandemic, but it was also just a very illuminating process. And I heard some heartbreaking stories as I did interviews for this. But overall, I’m really proud of the work.

John: And you mentioned addressing this through the entire educational spectrum. And I think it’s important to address it through the whole system. Because by the time we see students in college, they’ve already been indoctrinated into a system of grades, and there’s a lot of resistance to change, and also a lot of damage, perhaps, may have been done by the use of grading systems early. Could you talk a little bit about how we ended up with this system of grading, which has been so much a part of the educational culture from kindergarten on up?

Josh: Sure, yeah. And you’re right about the fact that students are conditioned for at least 12 years before they get to us to think about grades as being the most important thing about education. So how did we end up here, we could have a podcast episode that would last five hours if we really wanted to dig into that. But I’ll focus really on the more recent history. And that is that the A through F letter grade scale that we have is really about 125 years old. Our first records of a full letter grading scale come from Mount Holyoke in the 1890s. So they were the first to implement it. Back then it was A through E. They later dropped the E because they were afraid that people would think that they were excellent when they were not, you know, God forbid that we allow people to think that they’re excellent. And so they dropped the E and added a letter where there could be no mistake about what it meant, and that was an F, for failure. So that’s when it began. But that system, the letter grade system, was not formalized, or standardized, really until the 1940s. That is when you see a majority, in fact, of school districts and colleges and universities adopt that system. And what I think is most important about that, actually two things. One is that none of this has ever been inevitable, that the 1940s is less than 100 years ago, so this is not something that was cast in stone at the minute the American educational system began; it has changed and can be changed. The second thing, though, is that all of these schools, districts, all of these colleges adopted that system, not because they thought it was the best way to document student learning, but simply because it was the easiest way to communicate across institutions. They felt that if they had a common system that they could communicate more effectively between themselves. So it was not about student learning. It was not about student progress or growth or development. It was simply about communicating. That’s how we got here. And now we’re trying to figure out where do we go next? How do we undo some of the damage that has happened since that time?

Rebecca: Speaking of damage, do you want to underscore what some of that damage is? [LAUGHTER]

Josh: Sure, and I know that there’s room for reasonable debate about the terms that we use here. But I honestly really do think we can use the framing of damage and harm when it comes to grades. And in the book, I have several categories. The first and most obvious category for that damage has to do with learning, that grades set up obstacles to learning. They inhibit curiosity, they inhibit creativity, they inhibit risk taking, they affect intrinsic motivation. This is painfully obvious, but grades are classic extrinsic motivators, which means that they are very good for getting people to do things that they would not otherwise want to do. Extrinsic motivators operate effectively, if your goal is compliance, you want to get people in seats, you want to get people to turn things in on time, you want to get people to say things in class, grades work for that. But if what you’re interested in is learning and quality, that’s what you need intrinsic motivation for. The grades can get people into learning spaces, but they in no way ever guarantee that a student will learn even though they are actually in the environment. So that’s really important. Environments that prioritize grades also incentivize cheating, and they just stand in the way of a lot of different things. But that’s just the academic part of it. Obviously, those are important, but to me, they’re less important than some of the effects of grades on students’ lives and wellbeing. Grades magnify and mirror inequities that have always been a part of the American educational system. For example, students who attend poorly resourced high schools have fewer educational opportunities, fewer textbooks, more teacher turnover, all the things that are tied to less-resourced schools. And what that means often when they get to college, they have experienced what we call opportunity gaps, which show up in their grades, especially in Gen Ed courses. So those grades that you see in the first couple of years of a student’s college experience, one who has experienced opportunity gaps, those are reflections of the past, they’re not indications of the student’s potential. They are indicators of systemic inequity. So that’s just one of many examples that we could rifle through. Grades are surveillance technologies. They’re often used for punishment, especially in high school, but kind of across the board. So that’s another category. inequities. And then the final one, the one that we’re getting more and more research on now, maybe than we’ve had in the past, and one that I personally care a lot about is that we have evidence now that grades are a contributing factor to the mental health crisis in teenagers and young adults, the academic stress caused by grades. We have a number of research studies pointing us toward the significant contribution of grades to that mental health crisis. So lots of areas here, I think, beyond the classroom. And that’s really our central idea of this book that grades have long afterlives, well beyond a semester, well beyond the classroom, that affect people’s lives, sometimes just in the short term, but often in the longer term as well.

Rebecca: Josh, you went from going from grades are no fun to grades are a super downer. [LAUGHTER]

Josh: Yeah, well, that’s the message. It’s a happy book, Rebecca. [LAUGHTER] It’s not going to win any awards for a happy smiley story, that’s for sure. [LAUGHTER]

John: One concern that some faculty have about moving to alternative grading systems is that, while intrinsic motivation is something we’d all really love to see in our students, there may be some tasks in our disciplines that students have to master which may not be as intrinsically rewarding as others, but they might still be foundational for students to be successful. If we move away from grading systems, what type of motivation can be provided to help students master those tasks that may seem tedious and may not be quite as intrinsically rewarding as other areas that students just find much more interesting?

Josh: Well, that’s a great question. There are a couple of things I’d like to address about this. So first of all, it’s not that we’re talking about either having traditional grades or having no grades, and nothing in between. What a lot of people are doing in the area of alternative grading practices is experimenting with a whole range of models. And so particularly for the types of courses and disciplines that you’re talking about in that question, John, I have found that many faculty gravitate toward a model called standards-based grading, where you can identify the skills, the content knowledge, the disciplinary norms and necessities that are important for students to develop. And the grade for the course is based, not on individual performance on exams, but how many standards they meet, over the course of the semester. And as a feature of that particular model, often students are given multiple attempts and multiple ways to meet the standards. So that’s part of releasing the pressure valve a little bit for students that gives them room to grow and honors the fact that individuals learn at different paces. So it doesn’t matter when in the semester that they hit the number of standards in order to get their A or B, because the process is a fundamental part of that grading model. So I think that’s really important. Another thing that often comes up in these conversations, though, is the fact that we do in fact, have gradeless colleges in America. The one I feature in the book is Evergreen State College in Washington. And they don’t have any grades at all. They have fully narrative transcripts. They give only feedback over the course of the semester. And they teach all the disciplines. So they have found ways to heighten the intrinsic motivation, and to use feedback to really help students navigate the path toward the goals that they have for themselves and for the class. So there are ways to do this. You don’t need grades to keep people in the seats or to have them do things that might draw on less intrinsic motivation. I think there are ways to structure our learning environments that allow us to do similar work without the pressure of traditional grades. And I think that’s the ultimate point here. I think what a lot of folks in this area who are interested in grading reform are trying to promote is not necessarily just ditching grades outright, although we could talk about that if you want, but reorienting students relationship with grades, the messages that grades send to students, so that it’s not necessarily an evaluation, but a tool helping them develop their skills and to push their learning forward.

Rebecca: It’s always been interesting to me that the A through F system was meant to provide some standardized ways of communicating between institutions yet, if you look at any single institutions’ body of syllabi, you can see that the grades mean entirely different things [LAUGHTER] depending on a class, like some might be about attendance, some might be about success on a test, some might be about achieving learning objectives. So it seems like if it was meant to communicate anything [LAUGHTER] across institutions, it certainly isn’t meeting that objective. I don’t know what my question is. [LAUGHTER]

Josh: I’m so glad this up, though.

Rebecca: I had a question, but I think I lost it. [LAUGHTER]

Josh: No, I’ll answer it. There’s a question in there, Rebecca.

Rebecca: Thanks for helping.

Josh: What you’re talking about there is what I call the measurement fallacy. And so there has been more and more work over the last few years, including one that I absolutely love, called A Century of Grading Research, where a team of experts dives into a massive amount of material. And the conclusion that they come up with is that a grade is nothing more than a subjective indicator by an individual instructor on a student’s progress toward that individual’s goals for a particular course. So I set the goals for my course, a grade that I would give is nothing more than an indicator of how much progress a student has made on my goals for my course. They call them learning intentions in this paper. It is not in any way, a kind of universal certification of learning in a particular course or discipline. Nor does it mean that the student who gets a certain grade in my course would get the same grade from my colleague down the hall teaching the same course. It is a truly subjective indicator of progress. And because of that, we cannot really say that grades measure what we have been told that they measure and what our educational systems have assumed that they measure, which is learning. They’re not universal measurements of learning, they’re subjective indicators of progress in the course.

Rebecca: And yet they’re used to make all kinds of decisions about students.

Josh: They are, and you’re absolutely right.

Rebecca: It’s very interesting. [LAUGHTER]

Josh: So I think there has been a lot of ill placed weight on grades without a subsequent amount of investigation into what grades have meant, and continue to mean. This is also what fuels the furor over supposed grade inflation. And there are lots of other tendrils of problems, when you begin to put so much emphasis on grades to do so much work in our educational system.

John: You mentioned Evergreen State College, but another college that had gone in that direction, very successfully was New College, and I teach in a program at Duke in the summer, and I had two TAs from New College, and they were two of the brightest people I had ever worked with, and their mastery of the subject matter was very much equivalent to that of other students I had who came from institutions with very traditional grading systems. So they certainly didn’t seem to be damaged by that. And yet, there seemed to have been a bit of a hostile takeover of New College. [LAUGHTER]

Josh: Yes, there has been but there is not evidence yet that the hostile takeover has affected that gradeless model. There was some discussion that I read in transcripts of board meetings about that grading system, but it has not changed yet. And I’m glad you brought up New College, John, because, and they have this on their website, as recently as last year, they were able to boast that they were producing students who went on to get the highest percentage of STEM PhDs in the country, relative to their size as an institution, which says a lot. If you are a gradeless institution, and your students are succeeding at that rate in STEM PhDs, it means that they’re learning not just content, but also the skills that are necessary to do that level of advanced work. So I think that that says a lot for what is possible, even within systems that do not have anything close to a traditional grading system.

John: And I should note that one of those students did go on to a PhD in economics. And the other student went on and picked up a master’s degree in data analytics. And they’ve succeeded very nicely, despite the absence of grades [LAUGHTER] in their college career. [LAUGHTER]

Josh: Imagine that.

John: When we began this discussion, one of the things you noted was that under traditional grading systems, the grades at the end of courses often reflect differences in the opportunities that students face before they enter our classes. Can you talk a little bit more about that?

Josh: Absolutely. So you have the reinforcing of those gaps, you have extensions of those gaps. So if you think about what happens to a student who comes into a college course with those opportunity gaps, and then is in a system, for example, where grades are curved, so not only then are they trying to bridge over those equity gaps, but their grades are really reflective of those gaps. But now they are also competing in a curved grade system with their other fellow students, many of whom have had more practice at the kinds of exams and the kinds of questions and the kinds of teaching strategies that they’ll see in those courses. So it just starts to pile on top of each other. And I think that this is something that we just don’t talk enough about, it’s just assumed, and many of these students are labeled unprepared or not ready for college, which places all the burden on them rather than this more systemic problem that I think we really need to face. Because we don’t talk enough about it, we assume that grades have to be a part of the landscape of education. And because of that, we, and I say we as the community in education, we’ve just been willing to tolerate all of the issues that grades have brought with them without really pulling back the curtain and looking at what the issues really are.

John: And we lose so many students in the first couple of years of college and many of the students we lose are those who come from low-income school districts, whose parents had less education, and who just come in with less preparation. And that’s not a very equitable way of providing an education.

Josh: Right, absolutely. And so this shows up on all kinds of metrics that colleges care about: retention, progress, graduation, all of which, by the way, are tied to grades and GPAs. That grades are the gatekeepers for keeping those students enrolled and moving through to graduation. So yes, that’s a major problem to address with respect to those students who are coming to us with fewer opportunities and from school districts with fewer resources. I think we need to really investigate the traditional ways of being In higher education, really look closely at them, reimagine them in order to create systems that have equity at the center and where we are actually, I think, invested in student success and helping those students from those different backgrounds to succeed.

Rebecca: I think the other thing to underscore related to equity issues is not only does an institution maybe have policies about grades and decisions are made at an institutional level, but at the federal level grades are used for financial aid. And so if you don’t hit a certain grade point average, and if you don’t have a certain number of classes that you complete successfully, so you’re not withdrawing from them, or having incompletes, etc, then you might not be eligible, you could lose financial aid. So whether or not a student has demonstrated growth in any kind of way doesn’t matter. It’s down to this idea of grades. And depending on the classes they’re enrolled in and the different things that grades mean, we’re essentially not allowing students to proceed because they can’t afford to.

Josh: That’s absolutely right. The more you really take the magnifying glass to this, the more you see how implanted grades are in all of these decisions, and all these key points in a student’s academic career, bringing in financial aid, I think, is a wonderful example of their gatekeeping function. And the idea that you’re raising there, that it doesn’t even give students a chance to try and bridge those equity gaps. It doesn’t honor their growth or development at all, it says you got less than a C average in your first year, therefore, we are not giving you any more money to move through the university, and so many leave, many dropout. And I think that there is a better way to do things.

John: And many of those students who leave end up with a fairly significant burden of student debt that they still have to pay, further increasing the inequity. What types of alternative grading systems do you discuss in your book, you mentioned standards based grading, but what are some of the other areas that you encourage faculty to consider?

Josh: Well, the book is divided into two parts. And the first half is all the problems and the second half is possible solutions. And so there’s a chapter for parents on what you can do in the home to help your children reorient their own approach to grades. And then the last chapter is about systemic change. But the one in between that is focused on what teachers in K-12 and faculty in higher ed are doing in their own individual classrooms to help with this. And so, there’s a whole range. There are the models that are related to standards-based grading, I call them the cousins: they’re standards-based grading, specifications grading, competency-based, mastery-based, proficiency-based, they’re all not very different from each other, they just have slightly different names, and maybe a few modestly different features to them. So that’s one bucket. Portfolio grading is one that I recommend for folks who just want to dip their toe in the world of alternative grading, because a portfolio model, you’re giving a lot of feedback throughout the semester, a lot of opportunity to revise and redo assignments. But ultimately, the portfolio that is turned in at the end of the semester gets a fairly traditional grade associated with it, it’s just that that grade honors the whole arc of the work for the semester. So that’s another one. Contract grading, and its various branches that we have recently seen. So classic contract model was developed in the middle of the 20th century. And it looks something like this: here’s a list of things to do for the course, if you do 15 of them at a satisfactory level… I’m just making up that number…if you do 15 at a satisfactory level, you get an A for the course, you do 12 you get a B, etc. That’s a contract model. More recently, there’s been a version of it called labor-based contract grading,…very prominent in writing studies… that tweaks it a little bit to emphasize more the amount of work, the amount of writing that students do over the course of the semester, and that is tied to fulfilling the particular contract. And then if we’re looking at a spectrum of grading models, with portfolio grading being one of the more conservative of the alternatives, on the other end of the spectrum you have what some people call ungrading and other people call collaborative grading where there are no grades throughout the semester, only feedback, lots of student self assessment, self evaluation, and then at the end of the semester, the students proposed their grade and the final grade is determined through a conference between the instructor and the student about that proposed grade. So there are lots of things that people are trying. I tried to send two messages about this, one is that my goal is hopefully to help faculty find models that will work best for them and their students, that it’s not a one-size-fits-all model. The same thing that works for me may not work for my colleague in the next classroom over. So that’s important. The other thing is that you do not need absolute fidelity to any one of these models. It’s not like Moses came down from the mountain, with tablets that you had to follow to the letter, lots of people are mixing and matching and taking elements from a host of different systems that they find works for them. So I’ve yet to find someone marching down the checklist for engraving or for contract grading, everyone’s kind of putting their own spin on it, and that’s good. Well, it’s actually not only good, it’s important since our contexts vary so significantly across higher ed.

Rebecca: As a lot of faculty start considering these different alternative models, collectively we can start to initiate culture change, kind of with a grassroots model. But systemic change requires policy change and other things as well. Can you talk a little bit about what role we might have in pushing forward some of those agendas as well?

Josh: Like I mentioned before, a major reason for writing this book was because my observation was that a lot of people are talking about grading reform, but they’re all coming at it, understandably, from their own perspective, the faculty perspective, the administrative perspective, the policymaker perspective, and the parent’s perspective. And I think that if we’re thinking seriously about systemic change, we need to have everyone on the same page collaborating to move change forward. All that is to say that I think about this question quite a lot. There’s certainly examples that we have of recent systemic change, some of the University of California schools have changed their grading systems in significant ways. Western Oregon just dropped the lowest grades on the scale, they’re no longer available to give. I just met someone a couple of weeks ago from Bryant University, they’re moving to collaborative grading at scale for their first-year seminar courses. So it is happening, what it requires is a network of people across an institution, all of whom believe that this is not only the right thing, but they understand how to use policy to make the reform happen, but also how to tell the story in a way that all the different constituencies can see themselves in it, and can agree that this is the direction that they need to head. So the networking is important, understanding how policy shapes grading habits and processes at a university. If you’re part of a public system, which I know you and I both are, you need to understand how the system policy and the state policy plays into this as well. So that requires having that network of different people at different positions in a university to really move that work forward. But what I really want folks to know is that you’re not alone in this, that there are lots of people thinking about this, and lots of people who are doing this work, sometimes under the radar, sometimes above the radar at institutions, and that one person taking a step is a step toward change. But multiple people taking the same step together begins to create a movement toward change. And so I want people to know that. I also want people to know that there are blueprints out there for how to do this work, that there are people who study grading reform, there are models out there for successful institutional reform. Given my work for the book, I think we have a lot to learn from K-12 school districts. There are so many of those each year that are transitioning to standards-based grading across the district. Last year, for example, the entire city of Santa Fe, New Mexico, all their school districts moved to standards-based grading. So this is happening. And the more we learn about process, the more we draw on our colleagues’ network and expertise in these areas, the more we work kind of as a collective in higher ed to push this forward, the more change you’re going to see.

John: We give a lot of workshops on campus and some of them are focused on alternative grading systems. We haven’t seen quite as much movement with that among faculty as I thought we perhaps might, because during the pandemic, a lot of people were open to trying some very different things. But there’s been some resistance. We’ve had a number of people trying it, but it hasn’t caught on quite as much as I had expected it to. Why are so many faculty resistant to moving away from traditional grading systems?

Josh: Well,I think this varies from place to place. But I will say that I think that a major thing that we all face as faculty is the pressures of time. Everyone wants to do their best for their students, I really do believe that, but when you’re faced with such a limited amount of time in which you can change elements of your course, you’re probably going to gravitate immediately toward the kinds of content that you’re teaching in the course in the assignments, the course design pieces of it, or you’re going to think about active learning strategy that John and Rebecca talked about a few months ago, rather than addressing the thing that is at the very foundation of our educational systems in America. That’s not going to be the place that you go first, [LAUGHTER] kind of an upheaval of everything that you have known about education. So I think some of it is just, in a system where we all have very limited time, and very few resources, where do you put your efforts, and I think that is driving some of that. I also think we’re kind of seeing this interesting bounce back that followed that period of innovation at the beginning of the pandemic. I think what you see is almost a defensive response to that period of innovation, that what we really need to do to bring students back and re-engage them in the coursework is to try and somehow recapture what our image was a traditional education pre-2020, that we need to move backward toward that rather than capitalize and extend the innovations that we were doing in the early stages of the pandemic. I don’t want to paint with too broad a brush on that, but I have seen some of that elasticity, that rubberband effect, in talking with folks across the country about some of these issues. So I think those two things are at the forefront of why some folks might resist that. And none of those really has to do yet with really trying to unpack people’s own philosophies of education, or how their grades and their grading models tie into their values that they have with respect to education. So that’s even kind of a deeper level that you can’t even get to until you address some of these other concerns.

Rebecca: And when we’re crunched on time, things that we’ve done before are easier to implement. And it’s also hard to find time to reflect on our practices and to do some of the things that you’re talking about, in thinking about how our values align with what we’re doing. We’re really good as institutions of talking about equity, but we don’t always fully analyze how our practices impact actual equity.

John: And we were all the products of a graded system. And we were somehow successful in that, or we wouldn’t be in these positions. And so I think there may be a little bit of psychological resistance to changing something that has been so fundamental to our own experience.

Josh: Yes. I think a huge capital Y-E-S [LAUGHTER] to both of what you’re saying, that Rebecca, your point, we are not very good at really interrogating how our practices may or may not be aiding our equity goals or advancing our equity goals. And John, yes, we have succeeded in graded systems. And I do think the psychological is a part of it. If we were able to do it, why shouldn’t other people be able to manage this? I think that is what some people think. And more than that, I often see a kind of defensive fallback. Part of this is natural, in that we love our disciplines and devoted our lives to our disciplines, but a fallback toward if I get rid of grades or if I change my grading system, what does that mean for the standards of my discipline, for how people will engage with my discipline? And so I think that’s a psychological element of this as well. And it comes out in arguments about rigor and standards, but what it’s really about is the individual psychology, I think, of the person making the argument.

John: Are there other topics from your book that you’d like to discuss that we haven’t?

Josh: I think the reform efforts that I have seen succeed, and those that have failed, always have hinged on communication, at every level. Why are we doing this? What is the purpose of it? How do we bring people into the process very early on at all levels. At the K-12. level, that means parents, teachers and administrators working together. I think at higher ed, it means utilizing the expertise of faculty right out of the gate, allowing them to shape the narrative and the conversation and the direction and draw on administrative and staff resources to help enact that vision, rather than having a top-down mandate that you had to change your grading. That’s never gonna work in higher ed. But I will say that the communication is what drives all of that work. And it is the hinge that either allows one of these efforts to succeed or fail.

Rebecca: Well, we have a lot to think about, but what’s next, Josh?

Josh: So, what’s next? Well, the book comes out at the end of August and so I’m really excited to launch that and to finally have it in people’s hands and have lots of conversations about what’s in there, hopefully, So, that’s what’s immediately next. I am thinking about the next project that has something to do with the students who are coming to college now are the students who their entire K-12 experience has been shaped by the Common Core, and the obsession with standardization in America, and so what does that mean for higher ed? And what does that mean for those students and their learning? So, I’m in the very early stages of trying to think about that a little bit, but that might be what the next project is.

Rebecca: You always have great projects, Josh, and get us thinking about really important topics. So thanks for your work. I’m looking forward to sharing your new book.

Josh: Well, thank you both very much. It’s always a pleasure to be on this podcast, one of my favorites, and I’d love talking to you both about all these issues, so hanks very much.

John: It’s great talking to you again, and I’m looking forward to receiving my copy of this book in the summer. I ordered it as soon as I saw the notification that pre-orders were available.

Josh: I appreciate that, John, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]