458. A Neurodiversity Framework

Institutions are increasing their focus on creating inclusive environments. In this episode, Megan Kohler joins us to discuss a course design framework she developed to support neurodiverse learners. Megan is an Assistant Teaching Professor and Learning Designer in the John A. Dutton e-Education Institute at Penn State. She is the recipient of the Marion G. Mitchell Award for Innovative Teaching and is conducting research, funded by the Schreyer Institute Scholarship, on supporting neurodivergent learners in higher education. Megan is one of the co-editors of The Multi-Disciplinary Instructional Designer: Integrating Specialized Skills into Design Toolkits.

Show Notes

Transcript

Rebecca: Institutions are increasing their focus on creating inclusive environments. In this episode, we discuss a course design framework to support neurodiverse learners.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Megan Kohler. Megan is an Assistant Teaching Professor and Learning Designer in the John A. Dutton e-Education Institute at Penn State. She is the recipient of the Marion G. Mitchell Award for Innovative Teaching and is conducting research, funded by the Schreyer Institute Scholarship, on supporting neurodivergent learners in higher education. Megan is one of the co-editors of The Multi-Disciplinary Instructional Designer: Integrating Specialized Skills into Design Toolkits. Welcome back, Megan.

Megan: Hi. Thank you so much for having me here today. It’s great to be talking with both of you again.

John: We’re very glad to have you back again. It’s been a while.

Megan: It has.

John: Today’s teas are:… Megan, are you drinking any tea?

Megan: Oh, I’m always drinking tea. [LAUGHTER] I have a lovely new tea, and if you’re willing to indulge me in a short story, I’d love to share that with you. So the tea that I’m drinking today is called Paris, and it’s by Harney and Sons. And I discovered this tea completely by accident. It was Mother’s Day, and my husband decided to take me to this lovely little creperie. It’s like 45 minutes away, and I only drink tea, not a coffee drinker. And we get there, and they don’t have my tea, and so he grabbed this one, [LAUGHTER] and I tried it, and absolutely fell in love, and this one has become my go-to tea. So I highly encourage you all to try it. because it’s absolutely delicious.

Rebecca: Such a great story! I had a tea emergency that John solved for me today, so I’m drinking a decaf Irish breakfast tea.

Megan: Oh, I love Irish breakfast! It’s so good. It’s got a lot of great flavor to it.

Rebecca: Yeah, nice and bold, nice and bold.

John: And we’re in the midst of getting this cloud of smoke from Canada from the fires that are burning there. In honor of that, I have some lapsang souchong tea, which is a smoked tea. So it just adds to the ambiance of the day here.

Rebecca: So it’s smoky inside and out. [LAUGHTER]

John: Equally.

Rebecca: So we’ve invited you here today, Megan, to discuss your framework for supporting neurodiverse learners. First, though, can you tell us how to define neurodiversity?

Megan: That’s a great question, and it’s also somewhat of a difficult question, and I’ll explain why. So first and foremost, I try to keep things very simple. I think the more simplistic we can keep things, the more approachable they become for people, and that’s something that I really want to keep in mind in doing this work. The more approachable something is, the easier it is for people to buy into it or to understand it. And so I think of neurodiversity as largely just differences in the way that our brains function, most often because of some form of condition such as autism or ADHD. Those are the two most common ones that people think of, but it goes so far beyond that. It also includes things like post-traumatic stress disorder, and a lot of people typically think of like military veterans who have had some type of traumatic experience. But it’s more than that too. You can have trauma from a wide variety of things in your life. So there’s cancer survivors who you talk about their experiences having PTSD from cancer treatments that they’ve undergone. It also includes other conditions such as dyslexia. Dyslexia is actually the most commonly diagnosed condition over autism and ADHD, but yet, autism and ADHD are the ones that people typically go to. And so, the reason why I think it’s a little challenging to come up with or to have a definition of neurodiversity is because there are so many different perspectives that end up tying into this. So we have the medicalized version of neurodiversity and what that means, and then we have the social movement and what that means, and then in between those two sort of opposite ends, we have the lived experience that fits between those, and that lived experience is something that people hold very dear and very close to them, and that can impact the way that they see and understand and perceive certain things about neurodiversity and the many conditions associated with it, so it’s not as simple and as straightforward as it seems.

John: So there’s quite a bit of diversity in terms of neurodiversity, which makes some sense, I think. So how did you first become interested in focusing on support for neurodiverse students?

Megan: It was about eight years ago. My daughter was first diagnosed, and it’s a very long story and heart wrenching for me. I’ll try to summarize it briefly, but essentially we had had her in a daycare. We had her in a Montessori school, and that was after having her in a different daycare, and they kept telling us your daughter’s really, really smart. And we were like, “Oh well, thank you.” And they’re like, “No, no, no, no, she’s really smart.” And it was actually their encouragement that we sought out a more academically focused program, such as Montessori, and that was kind of always in the plan to begin with because my husband and I both work at Penn State University and education is something we’re very passionate about. So we had had her in this Montessori school, and we went there for an open house, and they started telling us all of these wonderful things about her, and they had her demonstrating all of these things that she could do for us. And then the teacher at the end of the night said, “I would really love if you’d come back in and we could have a more detailed conversation about your daughter.” And so I got really excited. I was like, “They’ve just told me how brilliant she is and showed me all these wonderful things that she can do.” And so I went in for the conversation, and as it went on, they just started sharing more and more of these things, and they were like, “We notice that your daughter isn’t engaging in cooperative play, and that’s something that we typically see at this age and stage of development.” And so they just started talking about all of these different things that they had observed with her, and then towards the end of that conversation, they were like, “We think that she might have autism, and we think that you should get her evaluated to see if this is the case.” And so it turns out that she doesn’t have autism; she has ADHD. But it was through that very long journey of trying to get her diagnosed, and then eventually getting the diagnosis that, as a mother, you want to do everything you can to try and support your children, and so I dove into the research and started to recognize two things: one, that I also had undiagnosed ADHD my entire life, and the light bulb went off, and I was like, “Oh my gosh, this explains so much,” and that’s very common in terms of how people are starting to recognize their neurodivergence, but then also something else that occurred to me at that point in time was that there’s a lot of support in K through 12, but then once students were going to college, that support largely dropped away. Now again, this was eight years ago, and we’ve seen some tremendous changes occurring in higher education, especially around the amount of support that’s being offered, but there’s still so much work that needs to be done, and that’s actually one of the reasons why I designed the framework that we’re going to be talking a little bit about today, is because everything I saw was very fragmented, and there was no comprehensive, clear pathway that spoke not just to how do we support our learners from a learning perspective, but how do we also address some of these more personal belonging needs?

Rebecca: Can you talk a little bit about why the framework needed to be created? In other words, why don’t traditional methods work for neurodiverse students? What challenges do neurodiverse learners face?

Megan: So, first of all, I think that there are actually a lot of strategies that do work. It’s just when you’re looking at the ocean that is learning, how do you whittle it down to find the strategies that are most supportive of neurodiversity, and that’s essentially what I’ve tried to do for educators, is to dive into that research and to figure out, okay, well, what are their needs? Where do we start? How do we even go about this process? And that was largely driven by my daughter because I wanted to make sure that once she gets to college, she’s been supported too. So to sort of cycle back to your question, there’s a lot of strategies that do work. Challenges that neurodiverse learners face in the more typical or more traditional classroom consist of a number of different things. First and foremost, there is a lack of understanding and awareness about what neurodiversity actually is, and there’s a lot of stereotyping, and we need to be very careful with that. And we’re also starting to see, because it has become more prominent, there’s a lot of labeling that is going on. I’ve heard people in my different social circles, and they will be like, “Oh, you know that person? Oh, they are so on the spectrum,” and that’s exactly the type of conversations or statements that we want to avoid making. You don’t ever want to do that to somebody, and you don’t know what their circumstance is. You don’t know whether or not they have a diagnosis, or they could simply be a person who is just a little quirky. That doesn’t mean that they are neurodivergent, so we need to be really, really careful about saying things like that and putting labels where labels might not apply. But some of those challenges, I think the most prominent one from my perspective is really sort of a social component. It’s really making sure that our diverse learners a re supported and made to feel like they belong. Belonging is such a huge piece of the experience that we’re offering nowadays, and it’s really interesting because educators and institutions as a whole are starting to recognize how much the belonging piece impacts whether or not a student chooses to stay at an institution, and not only that, there’s a lot of other research that shows us that when we start to engage students in things like social emotional learning, they’re building those skills socially, but they’re also ending up developing and reinforcing other skills that are critical to their academic experience.

John: What are some strategies that we can use to create the sense of belonging, so that students do not feel like they have some sort of a deficit?

Megan: Yeah, that’s a great question. [LAUGHTER] This is one of the reasons why I love talking with you both, you both think so deeply about topics. So some strategies that we can think about, and this is tapping into the model a bit, things like co-designing. So when you sit down and you work with a student, or even if you share an assignment, for example, with the entire classroom, allowing those students to come in and give their thoughts and ideas on the assignment, see how it might need to be altered in order to make them feel as though it’s something that’s achievable and attainable, and to revise instructions that maybe don’t make sense to them from their perspective. Allowing that exchange to occur goes a long way to making a person feel as though they are accepted. We can also talk about things like normalizing if somebody does have some type of a learning difference or a disability in which it can be supported. When I give my keynote talk about this, there’s an example that I give about how using transcripts from videos or even creating audio recordings of the lectures, so having the ability for every student to be able to not just read the text but also to download an audio recording. I had a faculty member who did that several years ago, and the students loved it. They just kept writing her all of these notes saying “this was so fantastic, so wonderful. I was able to download this, and I could listen to it at times that worked for me.” And when you do that, you take some of the stigma away from people who need that different format, and now you make it something that is really cool because now it’s just this different format. It’s this different modality. There’s almost like a novelty to it to some degree that I think students really enjoy and appreciate. So those are a few quick examples.

Rebecca: So we started to talk a little bit about the valued minds model that you developed, but we haven’t really dug in yet. [LAUGHTER] So, can we talk a little bit about the system that you’ve developed and the framework and what that looks like?

Megan: Yeah, absolutely. And you probably noticed I’ve been like holding back a little bit [LAUGHTER] because I’m like, this is in the framework. [LAUGHTER] It’s all in there. [LAUGHTER] Absolutely. So the framework really has three core components to it. So the first is learner agency. The second is system design, and this is where we tend to address a lot of our systemic barriers that occur. And then the last piece is belonging. Now within that, there’s several different layers or different components that should be taken into consideration. So within learner agency, the first thing that we need to do when we’re designing for neurodiverse learners is to start with some foundational components, and that is really about integrating strategies that support the executive functioning skills. So there is plenty of research that exists that indicates that if you have a neurodiverse condition or a neurodivergent condition, a lot of times there will be differences in the executive functions. And so, when we design for those, what we’re doing is sort of leveling the playing field. So, if somebody has a known issue with working memory, and we’re bringing in different types of supports to help balance that out, so that there’s not as much cognitive load being expended on their part. Well, now you’re instantaneously putting them in a place where it’s going to be easier for them to engage in the learning process, rather than for them simply trying to decode or to follow along in the way that you had originally intended them to. And then we bring in this accessibility piece after those foundational components. So making sure that if somebody has some type of a difference, and I think that this is really critical for those more applied conditions such as like dyslexia and dyscalculia. So bringing in those accessibility pieces to make sure that there are screen readers that can communicate that information to them or help support them in the exploration of that content. That’s really the next level or layer. And then when we talk about sort of the custom design, that’s a piece that fits in the top. And this is where I see a lot of people approaching it in the inverse, a lot of people try to start with those custom strategies and work sort of at that level. But if you do the foundational components and the accessibility piece first, you’re setting everyone up for a much better success. So then, when we get to that custom piece, this is really about sort of meeting people where they are and taking a look at who they are and what their skills and abilities are, and very specifically what their strengths are, and so one of the things that we need to be really aware of is the fact that a lot of neurodiverse individuals have received a lot of negative comments and a lot of negative feedback throughout their entire lives. In fact, it’s estimated that by the age of 10, a child who has ADHD – and these are metrics just for ADHD – it’s anticipated that that child has received 20,000 more negative messages than their neurotypical counterparts. So that’s something pretty significant. And so then when we think about the language that we use and the way that we engage with those learners, we need to be helping them recognize what their strengths are, and we need to be speaking in a way, whether it’s person-to-person conversation or even through our text. And a great example that I love to use is when we’re designing our rubrics using strength-based language in our rubrics. So, for example, you’ve probably seen the rubrics that use categories that range from excellent to poor. But what happens when you put the label poor on someone’s work? That’s a label that indicates that they’re lesser than, that they’re not as good, and it’s kind of an endpoint when somebody tells you that your work is poor. So we need to be shifting that to using language that’s more along the lines of expert to novice, because if somebody tells you you’re a novice, your work is at the novice level, what does that immediately make you think of? It makes you think of a trajectory or a journey; that this is a starting point, but that you can grow beyond what you’re currently facing. So, strengths-based language is absolutely a critical component of that. So, those are the three layers of learner agency.

Then we talk about system design, and this is where we get into things such as accommodations, training for faculty, things like the hidden curriculum. And I don’t know how many people necessarily think of hidden curriculum as being something that we need to think about when we’re talking about neurodiversity. It tends to be something that we think about a lot with first-year students, but it’s absolutely a component of neurodiversity, and the reason for that is because a lot of it stems from interpersonal relationships. That’s how you become aware of a lot of those components of hidden curriculum is through your interactions and those relationships. So, making sure that we’re addressing that as part of this process as well, and then we also have other things in there too, such as stigmatization, making sure that we are trying to reduce stigmatization of our neurodiverse learners as much as possible. So that’s all part of that system design component. And the challenging piece with that is that some of this that you can do in your classroom, and some of it occurs at the institution level, so there’s a wide range of approaches that can be integrated within that. And then this last piece is belonging, and where we start with that is with the concept of validation. So making sure that we are bringing in inclusive strategies that, as I mentioned before, not just help the students feel a greater sense of belonging, but can also help them develop those executive functioning skills. And the great example of that is social emotional learning. But there’s other things that we can do too in terms of making students feel validated and supported, such as peer-to-peer confirmation. You’re making sure that you’re enabling students to give each other positive feedback and say, “No, you did really great on that work. I really valued what you did here, and the way that you did it was really great.” So integrating strategies such as that, and then we get to recognition, which is the next piece of the framework. And so, recognition and accessibility kind of parallel one another as those foundational elements, the executive functions, and validation parallel one another. And we talk about recognition. We’re talking about that normalization of the different practices. So as I referenced that strategy before about the audio recordings and just normalizing it, making it feel like diversity is a part of everyday life, and it just belongs here, and it should be recognized and it should be valued. And then the last piece of belonging is this sort of modeling, if you will, of acceptance and what that can look like in the classroom. So we talked about co-designing information previously. So that’s a great example, but also making sure that we’re making students feel like your classroom is a place where it’s okay to have those sort of fast-fail experiences. So nothing’s ever a mistake, nothing’s ever a failure if you’re learning from it, if you’re growing from it. So making sure that we are creating dynamics such as that, and if we do that, making sure that we’re also tapping into things like a student’s motivation, helping them integrate things that are really very important to them, and I think if we integrate all of these pieces together, I think we can create our classrooms to be those more inclusive environments for neurodiverse learners. Because we know that some of the more traditional inclusive practices don’t necessarily apply. And one of the things that I always get sort of stuck on a little bit when it comes to inclusive strategies is that one of integrating research papers or using articles published by authors who have the same types of differences, whether it ends up being a disability or if there’s a different type of ethnicity. But how do you do that when the difference is something that’s invisible? So how do you know whether or not the author of a paper has autism or ADHD or dyslexia or dyscalculia or one of the many other conditions? And so that’s why I think a framework like this is something that’s needed because there’s shifts, there’s nuances that we need to address in order to create these dynamics that are neuroinclusive.

John: Now you mentioned that some inclusive practices don’t work as well for all students. Could you elaborate on that just a little bit in terms of some things that people routinely do that may not work as well for students with various forms of neurodiversity? You gave the example of papers as one case, but could you give us some other examples?

Megan: I think where there’s a lot of confusion is actually… this is another framework that I’ve been researching and working on… but one of the things that I have seen occurring is faculty not necessarily understanding what is inclusive support versus classroom management versus learning support and so on. So I was attending a conference not long ago, and there was two very lovely women who were leading this session, and at one point they pulled up this Padlet and they were like, “What strategies do you think are going to be great instructional strategies for the students in your classroom?” And I saw people just contributing things to the board, and they were like, “Yes, all of these are wonderful.” And I just sat there going, “No, that doesn’t align with this. And hmm hmm…” and I saw a lot of that confusion. So the paper example that we gave just a second ago – that’s typically an inclusive strategy, and that was one of the things that was mentioned in this conversation. And I was like sitting there thinking the same thing that I just communicated to you, like, “Okay, but how do we know? [LAUGHTER] How do we know whether or not this person has a neurodiverse condition?” But yet it was listed in one of those strategies. So a lot of times we’re relying on information that we know from other circumstances, and we’re not necessarily creating those subtle shifts that need to occur. And so another great example is classroom management. So I see this a lot. One of the people in this session listed brain breaks as a great strategy, and the learning designer in me sat there going, “Oh, wait a second. Are we talking about breaks for the sake of taking breaks?” Because we know that there’s no research to back that up, and if we’re talking about breaks for the purposes of taking time to reflect on information, then yes, that does work. But what they were talking about were brain breaks in terms of getting up and moving around the room if you felt like you couldn’t focus on the information that was being shared at that moment, and apart from the fact that there’s research that says that that’s not actually how our brains work, and that’s not the best way to help us develop the ability to stay focused on information for a long time, that also tends to align more so with the classroom management side of things rather than the instruction side of things, so there’s a lot of misunderstandings and a lot of confusion around what strategies actually align with what sort of categories of support, if that makes sense. So yeah, it can be very confusing unless you spend a lot of time researching this, and I think one of the things that made me very concerned at the end of the session was that the two women who were leading this were like, “And we’re going to be writing a book chapter about this,” and so [LAUGHTER] that goes back to kind of the concept of right now neurodiversity is kind of the wild west. There’s a lot of really good advice being put out there, and there’s a lot of advice that we need to evaluate a little bit more deeply. But it’s being put out there and could actually be harmful in some ways, and so we need to have more conversations, deeper conversations around that.

John: And that’s where that co-creation that you talked about before can help to address the needs of the specific students in the room.

Rebecca: Well, one thing that I’m observing or thinking about as you’re introducing the framework is that it has components from a lot of other existing frameworks, and you kind of hinted at this a little bit. Things from universal design for learning, TILT – transparency in learning and teaching, just good learning practices to make sure we stay motivated around agency and things like that that are certainly in UDL or Universal Design for Learning, accessibility practices, relationship-rich practices, but it’s not inclusive of everything in those frameworks. Can you talk a little bit about the process you use to pull what would be most important for neurodiverse students from those existing frameworks, you kind of integrated them all together and put them in one place. Can you just talk a little bit about the process of doing that?

Megan: Yeah, it was a lot of research, [LAUGHTER] a lot of research and reflection and evaluating this. I think the thing is, is that there’s so much amazing work that has already been done around the process of learning. We have all this great research about the brain and how the brain functions and what it needs in terms of learning and how we can best support the learning process for our students, and it was really just about diving in and just trying to understand what the unique needs are for neurodiverse learners, and then pulling the pieces that best supported those needs, and the reality is, there’s no one size fits all. There never is, and so I don’t see this as something that should ever be thought of as trying to replace UDL or trying to replace TILT . That’s not what anything we do in education is ever about. It’s a tool next to several other tools that, when it makes sense for you to pull this in, pull it in. Maybe there’s a faculty member who is tied very strongly to UDL, and they know their students very well, and they know that UDL principles are going to work for them. Then they should use UDL. This is a tool in the toolbox of support, and at the end of the day, there is never one perfect solution for everything, for every learner, for every need. But that’s hopefully one of the things that I would really love for people that when they look at this framework, they see that there is space for those unique needs, and I think that that’s something that maybe is a little more challenging to find in some other models. But when I talk about that custom piece, one of the things that having undiagnosed ADHD my entire life, there were so many times when I felt like I wasn’t measuring up and I wasn’t good enough. And in some circumstances, I had people who were very supportive, and they would say, “Megan, you can do this. You can do this. Just let’s figure this out. I’m going to help you, however you need help, and we’re going to figure it out.” But that was not always the case. I had a lot of other people who saw my differences as shortcomings, and they took that as an opportunity to berate me and to tear me down, and it’s those unique instances where you have somebody that steps forward and says, “I see something amazing in you, and I want to help bring that out. I want to help you shine, because I know that you’re capable of this.” That’s that custom piece of my framework. That is where I think we have the biggest ability to really make students recognize their own strengths, not just as students, but as individuals with something incredible to contribute to the world, we need to be shifting this paradigm from a deficit-based mindset of difference as lesser to a place of different is unique with incredible opportunity behind it. That is the goal of this framework. Sorry, I feel like that was very deep, but you want to know why I do this work. That’s why.

Rebecca: It sounds like the custom layer that you’re talking about is really important for what can really sometimes be competing needs, like very different kinds of needs that, in some ways, like on the surface, can look like they’re directly in competition for one student versus another because this student needs X and this student needs Y. But if you had provided the opposite to the other, [LAUGHTER] it wouldn’t help.

Megan: Absolutely, and I’m so glad that you mentioned that because that’s one of the other things that I’ve noticed. So when I talked earlier about the fragmentation that’s occurring around neurodiversity, that’s a large part of it. So you have people saying these are the 10 strategies that work for individuals with autism, and these are the 15 strategies that work for people with ADHD, and then a large part of the other neurodiverse conditions are just completely overlooked. And so in creating this framework, I wanted to say that’s not the way that we should be going about things. We need to be looking at the common threads across all of these conditions, and that ends up largely being the executive functions, which is why that’s the starting point. But then, how do we look for those common threads, those common building blocks across these different conditions, while at the same time allowing there to be this degree of flexibility, so that we can meet students where they are. A great example of those differences that you just mentioned… So my daughter has ADHD. I have ADHD. We have completely different needs, and yet we share the same genetics. [LAUGHTER] And yet our two conditions are very, very different. She’s brilliant. She is absolutely brilliant. She’s in the gifted program at school, and she’s working at grade levels that are two grade levels higher than where she’s currently enrolled. She’s looking at going into fifth grade, but last year when she was in fourth, they had her working on math problems and reading books that were at the sixth grade reading level and the sixth grade math level. I run away from math. [LAUGHTER] I run away from math as far as I can possibly get, because I just don’t have the working memory to be able to hold the numbers in the right places. But yet she does it, and she does it in her brain. And it’s so fun because my husband’s an engineer, and we’ll be driving in the car, and he’ll just randomly shout out a math problem, and they’ll see who can answer it first. And both of my children already have the answer, and I’m like sitting there with my calculator [LAUGHTER] like hidden down beside me because my brain’s just not structured that way, but hers is. So even when you have a specific condition like ADHD, not all of those strategies are going to work for all of your students with ADHD, so it’s just so complex. It’s so complex, which is why we need to have that flexibility.

Rebecca: I just want answers, Megan. [LAUGHTER]

John: But the answers are different for each student.

Megan: [LAUGHTER] They are. They absolutely are.

Rebecca: It’s really like the underlying subtext always is like “Ah, why is this so complicated?” [LAUGHTER]

Megan: Because humans are complicated. That’s the bottom line: is that we are all unique, and within that uniqueness, there is a beauty, but there’s also complexity, and it gets really hard when you’re trying to be an educator who’s working towards addressing all of that complexity. I don’t know that we give our educators enough credit for how challenging it is to take a single message and ensure that 30, 60, 120, 200, 350 people in the same room all get an accurate understanding of what that message is or what that concept is.

Rebecca: You’re kind of asking for miracles. [LAUGHTER]

Megan: Yeah, it is like asking for a miracle. But that’s also where we, as human beings, that’s where our neuroplasticity or our cognitive functions really come into play because we can have that flexibility of thinking. We can learn new ways of communicating, and once you learn how a specific faculty member communicates things or what their expectations are, you, as a learner, you can be quite successful, but the learning curve to that is not going to be the same for everybody. And so, the more that we can do to address some of those differences, we’re only going to make that process easier on everybody. So, the days of I’m just going to sit down, and I’m gonna create my course and write this lecture, that’s not sustainable, not for the way that we need to be thinking about our students. And a lot of these conditions, the diagnoses are increasing. When I first started, even just a few years ago, and so much has changed. So much has changed. But even just over the past few years, we’ve seen the diagnosis of autism in the general population go from approximately one in 100 down to one in 89, and now the census currently is that I believe it’s children under the age of eight, it’s one in 63 are diagnosed with autism. So it’s going to continue to change and to flux, and the hope is that we also continue to have the compassion and the desire to want to change and flux with it.

John: And I think an important takeaway from what you’ve said is that when we give students encouragement that can make a really big difference, but when we say anything that discourages students, that can also have an extremely large impact. And we have to be really careful in how we phrase things and how we communicate to students, so that we do focus on their capabilities and their possibilities.

Megan: Absolutely, and I think there’s one important thing to keep in mind with that is that it’s not about walking on eggshells. It’s not about “oh, I can’t say this or I’ll upset somebody.” It’s about learning to think about how we communicate differently. It’s learning to think about taking our words from a place where they communicate as though somebody isn’t measuring up, to a place where we are highlighting those strengths and making them see the value in themselves and their future in what they can achieve. And so it’s a subtle shift, but it’s an important one, and one of the reasons why I think it’s so important. I don’t know if either of you have ever heard of rejection-sensitive dysphoria, more commonly known as RSD. It’s a significant condition in which a person really perceives as though the interactions that they have with people, it always has a negative sort of perspective or a negative spin on it. You know, somebody says, “Hey, everybody, we’re going out for coffee,” or “Hey, everybody, we’re going to a party,” and the person sits there and thinks, “Well, of course, they’re inviting everybody else, but they don’t mean to invite me. They must have forgotten that I was here. They didn’t see me or didn’t recognize that I was in the room.” So there’s very much a sense of separation that can occur around that too, and there’s some instances of RSD that are so severe that it actually causes physical pain for the individual. So it really is about shifting our language so that we are raising people up rather than tearing people down, and this is something that I have conversations with my kids all the time. Do you have children?

Rebecca: Yep. [LAUGHTER]

John: Yes.

Rebecca: Earlier, you were describing my car. [LAUGHTER]

Megan: Yeah. So we have these conversations whenever there’s a fight. The kids are in the back. “But he did this to me,” and “but she did that to me,” and it’s “Okay, okay, I hear you. Instead of accusing one another and saying what they did was wrong, how can you change this? How can you rephrase what it is that you’re thinking and feeling in a way right now that communicates what is happening or what you feel is happening, but that you’re not tearing the other person down as a part of that process.” And we do a lot of reflection in our car rides. [LAUGHTER] My kids will say one thing, and I’ll say, “Okay, let’s pause. How could you rephrase that?” [LAUGHTER] Because the rest of the world isn’t going to teach them how to do that. That’s my responsibility as a parent, and so we engage in those conversations a lot to be more aware of the impact of our words on other people.

Rebecca: We talked a little bit earlier about things that we can do in the classroom, but also kind of a more institutional or global approach or systematic approach that needs to happen. And the conversation that we’re having right now makes me think about one of the ways that these things can be modeled is also at like the employee level at an institution, which trickles down also to students. That this kind of a framework can be used in other contexts outside of a classroom setting that enhance the learning environment overall.

Megan: Absolutely, yeah, and I need to do a little bit more research because there are definitely nuances between the classroom versus the work environment. But I think a really quick piece that could be swapped out is take a look at hidden curriculum and replace it with office politics.

Rebecca: It’s all the same. [LAUGHTER]

Megan: To some degree, to some degree.

Rebecca: Yeah, yeah.

Megan: It is. [LAUGHTER] But yeah, and that is the thing that I hope to be able to do with this work is to carry it forward because it’s so interesting when I have these conversations. Practically every single time I’ve had a conversation about this framework, that question has arisen: “Can we use this for the office environment too?” I’m like “Yeah, with some modifications, but yeah, absolutely.” So you have a coworker who continually struggles with deadlines. There’s your executive functioning skills. It can absolutely be applied. I just need to sit down and make some of those modifications and some adjustments with that. So stay tuned. [LAUGHTER] It’s coming. [LAUGHTER]

Rebecca: Seems pretty workable at the moment. [LAUGHTER]

Megan: Thank you, thank you. [LAUGHTER]

John: We always end with the question: What’s next?

Megan: What is next? I’m currently working on a co-edited book with the amazing Adam Davi, who’s a good friend from our days working with the Online Learning Consortium, volunteering with them, and this book is titled Advancing Learning Design in the Age of AI. And what we are hoping to capture within this book is the conversation around now that AI is essentially commonplace and everybody has access to it, and we’re seeing more and more people leverage it for designing their courses, what does that relationship look like as a learning designer? Where do we shift? How do we shift? What do we need to be realigning in terms of our workflows and our responsibilities so that we can leverage AI to better support the work that we do, but also helping understand that AI is not a replacement for a learning designer, and it never will be. And if you know and understand anything about simply writing learning objectives, there’s a great example just within that. So I don’t know if you’ve asked any AI system to write learning objectives for you. They are lacking. [LAUGHTER] They do not follow a very traditional learning objective format, and it can take you quite a bit of back and forth to even get that to a point where AI is getting something even close. So there’s a lot of misunderstandings. People are putting information in or asking ChatGPT to come up with, “Oh, give me a course outline” or “write this assessment for me,” and it does it. But is it what you actually intended? Is it what the students actually need? Is there alignment with your instructional goals? Is there alignment with the overall goals of the course at the programmatic level, like there’s so many things that AI can’t fully understand because it just gives you that response to that one prompt. So there’s a lot that AI can’t do. We still need humans in the process. So what does that look like now? So yes, that’s the thing that’s coming down the pipeline. [LAUGHTER]

Rebecca: Sounds like an exciting adventure.

Megan: I think it will be. [LAUGHTER]

John: Thanks for joining us. It’s always great talking to you, and we’re looking forward to hearing more about these future projects.

Megan: Excellent. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

456. Improve Critical Thinking

Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, Michael Bishop joins us to discuss an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.

Michael is a professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases.

Show Notes

Transcript

John: Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, we explore an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Michael Bishop. He is professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases. Welcome, Michael.

Michael: Thank you. Glad to be here.

John: Thank you for joining us. Today’s teas are:… Michael, do you have tea by any chance?

Michael: No, not today. I usually drink tea in my office, and when I do, it’s usually a Chinese black tea.

Rebecca: Always a nice choice. Today I have an English breakfast tea, nothing that exciting,

John: And I just have nothing that exciting either, a Lady Grey today.

Rebecca: We got to step up our game, John. So, Michael, we’ve invited here today to discuss the field experiment that is described in your co-authored paper entitled “Critical Thinking Classes Can Reduce Common Biases: Results from a Field Experiment.” But before we discuss the study, can we start a little bit with the concept of critical thinking? You teach critical thinking. What are you trying to teach? And what makes someone a good critical thinker?

Michael: Yeah, I think that’s a great question. One of my favorite sayings is the philosopher George Santayana said that a fanatic is someone who, having lost sight of their goals, redoubles their efforts. And so, in teaching critical thinking, I think it’s important to start with an idea of what you want to teach. And so there’s a lot of possible answers. Some people think that a good critical thinker is someone who uses certain rational methods or rules like logic or probability for the scientific method. Other people think that a rational thinker would be an expert debater, someone who can detect fallacies and build a coherent case for their views. Other people view good critical thinkers as virtuous thinkers; someone who is careful, open-minded, curious, intellectually honest, intellectually humble. And I actually think these views are either wrong… so I think that good debaters are often poor reasoners… or incomplete. They capture part of good reasoning, but not everything. So basically, what I am trying to teach in my classes is teaching people to ask good questions. Ask what I call fruitful questions, questions that help us get better answers, not killer questions that end debates, but questions that, if you’re trying to answer them, help get us better answers. So when you view that as the goal, how you go about teaching critical thinking becomes much easier.

John: And as you suggested, people have many different descriptions of what critical thinking is. I know whenever we talked about the importance of this in our Gen Ed program, everyone agrees that critical thinking is extremely important, but they all disagree as to what exactly it is. So it’s helpful, I think, to start with that statement. In this experiment, you’re focused on five common biases. Can you describe the biases that were selected for this study and provide some brief examples of those biases for our listeners?

Michael: Sure. The way psychologists think about these is in terms of biases. The way I think about them is in terms of cheat codes. So there are certain cheat codes people use in debating or in arguments, and cheat code arguments are arguments that you can use to defend pretty much any view you want. And so cheat codes are really tools of manipulation, more than tools of trying to figure out what the truth is or what a good decision is. And so when you think about good critical thinkers as thinkers who can ask good questions, the way I think about it is when you detect a cheat code, when you detect somebody using a cheat code on you, you should be able to ask them a question that diffuses the cheat code and points them in better directions. So let me tell you some of these cheat codes. One I call the focalizing cheat code, and it might go something like this: We should go to war because it has certain benefits, so, if an argument like that, or we should not go to war because it has certain costs. If you’re convinced by an argument like that, I think that you’ve fallen for a cheat code, and the best way to see this is to consider an example where where it’s clear that it’s a cheat code. So, suppose I gave you two envelopes, and I said you can look in these envelopes, and you can take all the money that’s in the envelope that you choose, but you can only choose one envelope. Suppose I open one envelope, see that there’s $5 and I take it without looking in the second envelope. In that case, you’d think that I’ve made a mistake, that I haven’t reasoned wisely about this particular decision because I didn’t look in the second envelope. If I say, “Look, the reason I chose the first envelope is that having $5 is better than not having $5.” It’s true that having $5 is better than not having $5, so there is an advantage to choosing envelope one. But when you think about it, what makes a choice a good one isn’t that it makes you better off than you are now. What makes a choice a good one is that it makes you better off than what else you might have done. Now, economists call this opportunity cost, as I suspect you guys know. And so, one thing I’m trying to do is get my students to, when they see someone argue for, “Hey, we should go to war because it’s going to bring us benefits.” I want them to at least stop and be able to ask the question, wait a minute, what are some alternatives to going to war? Because even if it’s true that going to war brings us benefits, and we can argue about that, but even if it’s true, it might not be the best thing to do, in the same way that choosing envelope one, even though it brings us benefits, might not be the smart thing to do. So I call that the focalizing cheat code. Another cheat code that I teach is what you might call the correlation cheat code. So I say, “Hey, a new healthcare plan is passed, and healthcare costs have gone up 7% a year since this plan is passed. Boy, this plan really stinks. It’s caused our healthcare costs to go up a lot.” This again, the question isn’t is the conclusion right in the war case or in this case, the question is: Is this a cheat code argument? And it is. And the reason is, what you don’t know, and what you really need to know, is how high would have costs been had this healthcare plan not been in place. So again, I say this healthcare plan is great or it’s bad, and I point to something that happened afterwards that. That’s what I call the correlation cheat code because you’re taking this correlation and you’re taking it as evidence for thinking that this policy has caused this particular result. And so again, what I want people to do is just stop and ask the question: “Well, wait a minute. Maybe without the health care plan, costs would have gone up even more, or maybe they would have gone up just the same, or maybe they’d have gone up less.” But unless you ask the question, it’s easy to get manipulated into accepting views that aren’t true. Another cheat code is the regression cheat code. This is a very common one. So lots of coaches and parents think that punishment is effective at molding behavior. I know that when I played sports as a youth, my coaches thought that after a poor performance, if you punish the players by making them work really hard in practice, they’ll do better next time. And sure enough, after a poor performance and after we were punished, we did tend to perform better. But again, the question is: “Did the punishment do that, or was it just a return to our normal level of performance?” So that’s the regression cheat code, rather than just saying, “Hey, maybe you’re returning to your normal level of performance,” saying that “it’s the result of a punishment or the result of a reward.” The third one is sunk cost cheat code, another one that’s a favorite of economists, defending a decision because of the resources already spent on it, so let’s stick with this policy or this war because of sunk costs. Those sunk costs are gone; they’re gone no matter what you do. So you might as well do what’s best for your future. And the last one that we looked at was the gambler’s fallacy, and that is basically treating statistically independent events, as if they were statistically dependent. So, if a coin comes up five times heads, and it’s really a fair coin, thinking that it’s more likely to come up tails next time. So, those were the five cheat codes we looked at.

Rebecca: I think the soundtrack that we have going on with the thunder is really emphasizing that these are cheat codes. [LAUGHTER]

Michael: Indeed, indeed. [LAUGHTER]

Rebecca: It’s very ominous. [LAUGHTER] Can you talk a little bit about how you chose these particular cheat codes?

Michael: Sure. So this goes back to my view about good reasoning. So 20 years ago or so, me and a friend defended a theory of what rational reasoning was, and what we said was that the good reasoning is reasoning that’s cheap, it’s accurate, and it’s focused on significant problems. And so the way I chose these cheat codes is because I thought that they were significant. That is, people use them to manipulate you when it comes to really important issues. So I see these cheat codes a lot out there being used in the world for really important issues. And the cheap part came in because I think that in thinking about how to teach critical thinking, I think one thing you want to do as a critical thinker is to diagnose the problem. Why are my students thinking badly when they do think badly? And I think there’s three general answers that people tend to think about, although often this doesn’t get talked about enough. So one reason students might be thinking badly is that they don’t know these rules, so it might be an ignorance problem. And if it’s an ignorance problem, then it’s going to be really hard to teach them to use these rules in a way that they keep using the rules after they leave the class. Because not only do they have to learn the rules and learn to apply the rules, but when they leave your class, they need to stop thinking in their old ways and use these rules, which tend to be hard to remember, hard to apply in the real world. And so, one reason that I think studies have not been successful or have had limited success in teaching people to reason well in a way that sticks once they leave the classroom, is that too often we teach critical thinking as if the problem is an ignorance problem. So we need to teach our students new rules. So one reason I chose these rules is that I speculated that they already knew these rules and they sometimes applied them correctly. And if that’s right, if they sometimes apply these rules correctly, then I don’t need to teach them a new rule, and I don’t need to teach them how to apply the rule. They’re already doing that. All I need to do is get them to see that it’s the right rule, and then ask them, “Hey, why aren’t you using it in these cases over here, where it seems like you should still be using the rules.” So, for example, think about the two envelope problem that I talked about before. None of my students think, “Oh, yeah, choose the $5, that’s the way to go. None of them think that. They know perfectly well that the way you evaluate that choice is not in terms of “Hey I’m $5 better off than I am now,” it’s “Hey if there’s $1,000 in the other envelope, then I’m 995 dollars worse off than if I’d have chosen envelope number two.” They all see that, so I don’t have to teach them any new rules. All I have to do is when I say “Hey let’s go to war, it has this advantage,” and if they say, “Yeah, that sounds good,” I can say, “Wait a minute. How is this different from the two envelope problem?” So the main reason I chose these, and these aren’t the only rules I teach in my critical thinking class, but the main reason I choose the rules that I do is that I think that my students already know these rules and at least sometimes apply them, and so when I introduce them to these rules in cases to which they are inclined to apply them, they give me what I call the old bovine stare. They look at me like, “Well, that’s obvious. You’re not teaching me anything I didn’t already know.” And that’s what I want. I want them when I introduce these rules to say, “I already knew that. Why are you teaching me something so obvious?” Because then the next class period, or a week from now, or a month from now, when I introduce an argument where they ought to be using this rule that they said was so obvious and they’re not, I don’t have to say, “Hey, you guys have made a mistake.” I can just ask them, “Okay, wait a minute, you said this rule was obvious, why aren’t you applying it here? Either you were wrong on day one about that rule being the right rule, or you’re wrong right now in not applying that rule.” And so there’s a real sense in which the reason I chose these rules is because I think I’m really not teaching my students anything new. So here’s one way to think about it: rather than teaching them like calculus, like a chain rule, like a new formula that they don’t know, what I’m really trying to do is get them to recognize cardinals. They already know what a cardinal looks like, so I’m just trying to get them to recognize cardinals, and so that’s why I think that the way I teach critical thinking tends to work and is way easier than starting off with the assumption that “Hey, my students don’t know anything, and I need to teach them a whole bunch of new rules.” So that’s why I chose these rules, and that’s why I choose the rules that I use in my critical thinking class.

John: And these are rules that, when you think about them as you framed it, they’re pretty obvious. But in practice, they’re rules that people violate all of the time in their decision making. What really impressed me about the study was that you did get some [LAUGHTER] results out of this because so many studies of behavior modification have been found to be ineffective beyond a very short period of time, often not lasting through the end of the class.

Michael: That’s absolutely right. In fact, one of my old professors, he said this about logic classes, but I think it unfortunately is often true of critical thinking classes, but he said that logic classes teach you to do one and only one thing, and that’s take logic tests. And I sometimes worry that that’s the case with a lot of our attempts to teach people to think better, is that what we’re really doing is teaching them to take tests because they will use the rules on the test. But if you want them to think about this afterwards, I really do think that the main thing is to get them to commit to the rule, and then get them to recognize, “Hey, wait a minute,” to be sensitive to when they see it in the real world because it’s all over the place. Once you get sensitive to it, you start seeing it everywhere. And indeed, one danger of my class, of my critical thinking class that I talk to my students is what I call “the turd in the punch bowl problem,” and that is being a good critical thinker isn’t necessarily going to make you popular, and so you need to be careful about when you point out to people that their arguments aren’t very good, because if you do it too much, you will end up being like the turd in the punch bowl at a party. People will look at you and say, “Oh, I’m going to stay away from that person.” And so, at the end of the semester, I will ask them, “Have you had any turd in the punch bowl experiences as a result of this class? And at the end of the semester, a lot of the students – and I don’t know if this is good or bad – but about half of the cases involve the sunk cost principle, and usually it involves the destruction of a relationship. So people will end a relationship because the only reason they’ve been sticking with the relationship is all the pain and effort and heartache they’ve put into the relationship.

John: And I know behavioral economists have noted that quite a bit. That once they really get this message through, it leads to a breakup of so many relationships of their students. But even people who know these things sometimes fall into these traps. And one example I think I mentioned on the podcast before, and I know I’ve mentioned it to Rebecca, is one time I was at a conference with a few other people, and we went over to Universal park, and it was just really hot that day, it was like 97 degrees, and we had spent quite a bit for the tickets for the day, and we were just kind of miserable. And one of the other people said we should just go. And my response to a historian was that we spent $230 and we’ve gone on two rides. We really should stay and get our money’s worth. And her response was, “But that’s a sunk cost. It shouldn’t matter,” and it’s easy for us to fall into that.

Michael: Absolutely, and indeed, this is one of the reasons I choose the rules that I do, is that I think many of them are significant. They really can make a difference to the quality of your decisions and the quality of your judgments.

Rebecca: Can you talk a little bit about the two-stage training program that was used in class?

Michael: Sure. So I gave you a little bit of an example of it already, but let me give you an example with another case with the correlation cheat code. So my go-to example here is I’ll tell the students, “Well, you know, yesterday I ate chocolate and today I broke out with zits. Boy, that means that eating chocolate causes zits,” and they immediately say, “No, that’s terrible.” So the first stage, what I’m doing is I’m introducing examples where they are disposed to use the rule. So nobody says, “Yeah, that’s great evidence for thinking that eating chocolate causes zits.” And then we talk a little bit about, “Okay, well, what would count as good evidence?” And they come up pretty quickly with maybe just a little bit of help, “Well, you want to know what happens when you eat chocolate and when you don’t eat chocolate, because what you really want to know is what difference does eating chocolate make to your complexion, and so what you want to know is what happens when you eat chocolate and you don’t eat chocolate, and everything else relevant to your complexion is the same, and what difference is there in these two cases,” and so that’s the first stage. The first stage is just introducing them to these rules, these cheat codes. So I give them the cheat code, but really a cheat code which they detect right away as a cheat code, and that they know as a cheat code. And then I spend the rest of the semester giving them arguments that use that cheat code, but are not as easy to detect. And often these are political arguments, and one reason I don’t get into trouble – at least I haven’t so far – in giving political arguments is that these cheat codes you can use them to defend views on the left, on the right, in the middle. They’re tools of manipulation. It doesn’t matter what the truth is. You can use these cheat codes to defend the true view, the false view, smart decisions, dumb decisions. And so, what I’m really interested in doing is getting them to be able to detect these cheat codes. So a first step is just introduce them to these cheat codes in a way that they will be disposed to accept and think is obvious. And then after that, I’ll give them lots of examples in which they’re, at least many of them, are not disposed to think is obvious. And then rather than me sort of coming from on high and saying, “Well, you know, you made a mistake there, dummy.” I just step back and I say, “Wait a minute. We seem to have an inconsistency here. You insisted that this rule was right on Monday, and on Friday you’re not using the rule in this case. What’s going on?” And I have yet to have a student, and I’ve taught 1000s of students, I’ve yet to have a student say, “Okay, I’m going to rethink that rule.” They all say, “Oh, well, I made a mistake here. I reasoned badly,” and by the end of the semester, they’re not reasoning badly.

Rebecca: Sounds like you spend a very small amount of time on stage one and significant time on stage two. Do you do all of the cheat codes at the beginning of the semester and then work on utilizing them throughout the semester, or do you kind of do units on different cheat codes? How do you organize?

Michael: So I will introduce a cheat code and then I’ll give them examples using that cheat code and all the cheat codes we’ve covered. So once a cheat code is introduced, they might see it at any point in the future, and I think it’s really important they keep practicing it and keep being able to recognize it, and not just think, “Hey, I know this is the cheat code we’re working on today, so I just need to be on the lookout for that one.” Another thing I do, and I think this is really important, is I will sometimes give them arguments that are promising, that if you grant the truth of the premises, they at least give you some reason to accept the conclusion. Because I don’t want the students just to be skeptical of all arguments. I want them to be able to distinguish between “Okay, that’s a cheat code. That’s definitely a bad argument. The conclusion might be true, the conclusion might be false, but that’s not a good reason to accept anything” and then “Okay, well, what would count as good evidence for thinking that this is a smart decision or that this policy has led to increased costs or decreased costs?” And I want them to be able to say, “Okay, look, that argument right there, that’s pretty promising. This argument over here definitely uses a cheat code.”

John: So you start off with a little bit of retrieval practice with some applications of the same concept, and then you move into some interleaved practice, which raises the cognitive difficulty just a little bit. But it’s one of those desirable difficulties where the more cognitive effort that students have to place in identifying what is at issue, the more deeply they tend to learn it, and the more likely it is that those concepts will stick and also transfer into other applications, which is what you’re building to in this program.

Michael: Absolutely, absolutely, the interleaved practice is really important, and I wouldn’t go back to sort of the blocking practice.

Rebecca: So we’ve talked a little bit about what the intervention is. Can you talk a little bit about how the study was conducted, like the size of the treatment group, the control group, etc.

Michael: Sure. So in the paper, we really describe just the second study we did, we actually did two studies, and in both studies we had almost 400 students. In the first study we had 395 students, and the second study we had 397 students. And some of the students were taking a critical thinking class from me, and the rest of the students were taking other philosophy classes. So in the study that we report on, we had 113 critical thinking students and 284 control students. Then what we did is, in the first week of class, we gave everyone a pretest, and this involved the five cheat codes, and then two good arguments, or at least promising arguments. They weren’t necessarily great, but they didn’t use any cheat codes. And the reason we included these, what we called skepticism checks, was just to check if all we were doing was creating skeptical students rather than students who could distinguish between promising arguments and cheat code arguments. Then, at the end of the semester, we gave them the same survey, same seven questions, and not surprisingly, in both of the studies, the training group students did much better than the control students. In the first study, we waited three months. So basically, the way to think about it is that these classes all happened in the fall. We waited till the end of the spring semester. It might have been four months, and gave them same questions again in the first study. We gave them the same questions again. The trained students did much better, and then we gave them different questions, and they still did much better. So in the second study we did, which is really the only study that we report on in the article, we gave them pre-test at the beginning of the semester, post-test. Then we waited a full academic year plus the spring, so we waited 16 months and only asked them different questions. So the pretest and posttest were the same questions, and again my students did much much better. But then we waited 16 months and asked them seven different questions, and the trained students were still doing much better, detecting the cheat codes and recognizing that the good arguments were still pretty good arguments.

John: DId the trained students do significantly better with all 5 of these cheat codes?

Michael: No, they did better on 4 of the 5 cheat codes. They did better on the correlation, regression, sunk cost, and opportunity cost cheat codes, but in all the studies that we did, on all the surveys that we asked, we just couldn’t get the trained students to do better on the gambler’s fallacy questions.

John: But still, having students do better on 4 out of those 5 cheat codes so long after the initial intervention is a really impressive result. One of the things you talk about near the end of the paper, though, is that you note that there are some possible sample selection issues. Could you just talk a little bit about what source of sample selection might have affected the magnitude of those results?

Michael: So one worry that one might have is that the students were not distributed randomly. So the students chose to take the critical thinking class, the students chose to take other philosophy classes, and so there might be something about the students who chose critical thinking classes that would incline them to be better critical thinkers, despite the fact that on the pre-tests, all the students in both groups did equally badly. So that’s one worry one might have, and the thing to do there is, although it’s not really easy given the institutions of academia, is to randomly distribute students into critical thinking classes and other classes. So that’s the main selection worry. Another reason why we might have gotten good results is that I had those students for 15 weeks. Some interventions talk about full- semester interventions, but often they’re fairly short interventions. I was able to have the students for 15 weeks and made them practice a lot.

Rebecca: It sounds like in the design of the experiment in your class, just generally, there’s lots of opportunities for feedback and correction throughout the semester.

Michael: Yes, absolutely. There’s a lot of feedback, a lot of correction, and I have a big bank of good and bad arguments that I can give them over the course of the semester that use these cheat codes or that don’t use the cheat codes. And one of the things I do for an essay in this class is ask the students to put themselves in the shoes of someone who’s trying to come up with a cheat code argument and ask them to give me bad arguments that use these cheat codes, and they actually turn out to be pretty good at it. And many of the examples that I use on later students are examples that prior students have come up with.

Rebecca: Sounds like it makes it a little bit fun too.

Michael: Yes, I do ask them for any humorous examples or examples that us boomers would have a hard time understanding. [LAUGHTER]

John: One of the things we appreciated about this study is that the data, the results, and the analyses that were done are all reported in more detail on the Open Science Framework website. We’ll include a link to that website in the show notes for this episode.

Rebecca: So we always wrap up by asking, “What’s next?

Michael: A few things are next. One is I’m writing a textbook that uses the materials that I teach. The way I teach critical thinking is different from the way other people teach it. I haven’t found a good textbook that teaches critical thinking the way I like to teach it. So I’m currently writing a textbook that hopefully will help other people get good results. I’m also planning to build, if I can, a game that anybody can play to see whether they can avoid being bamboozled [LAUGHTER] by cheat code arguments, this is sort of a longer term project because writing a book is going to be useful, hopefully, for people to teach this. But what’s really important, in my view, what makes this stick is practicing on lots of examples. And so, in a game where people see an argument and you say, “Well, is this promising argument, or is somebody trying to manipulate you?” we’ll hopefully give people practice at being able to detect those cheat code arguments. So those are two of the projects that I’m working on.

John: On the latter project, will it be a board game? Will it be an electronic game? Will it be an online game, or is that something that’ll be decided further in the future?

Michael: No, it’ll be an online game. Is what I’m hoping to do. I want to do something that can be used not just by my students and that’s easily accessible.

Rebecca: Sounds like some great projects. Thanks for joining us today.

Michael: Oh, you’re welcome. It’s been a pleasure.

John: Thank you. I was really impressed by this study [LAUGHTER] because, as I mentioned before, it’s so rare to see effects that are so significant and persistent. It’s, I think, something that people can learn from in terms of other types of issues where it’s hard to convey this sort of long-term impact on students.

Michael: Well, thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

455. Transformative Coaching

Higher ed has become an increasingly complex and challenging environment to work in. In this episode, Susan Hrach and Kitty Maynard join us to discuss the role trained coaches can play in supporting agency, resilience, and career growth for college employees.

Susan is the author of Minding Bodies: How Physical Space, Sensation, and Movement Affect Learning. She’s served as a Fulbright Canada Distinguished Research Chair, and recently appeared on the TEDx stage in Columbus, Georgia. Kitty is the Director of the Teaching and Scholarship Hub at the University of Richmond. Her higher ed career began as a Professor of French Studies, and she has taken on leadership roles in faculty development at Washington College and Virginia Commonwealth University before moving to the University of Richmond. She is the author of Reveries of Community: Epic in the Age of Henri IV and co-editor of Polemic and Literature Surrounding the French Wars of Religion. Susan and Kitty have both received training from, and are members of, the International Coaching Federation. They are also two of the editors of Transformative Coaching for Faculty and Staff in Higher Education.

Kristin Croyle, a psychologist and the Dean of the College of Liberal Arts, Sciences, and Engineering, fills in for Rebecca in this episode.

Show Notes

  • Hrach, S. (2021). Minding bodies: How physical space, sensation, and movement affect learning. West Virginia University Press.
  • Maynard, K. (2017). Reveries of Community: French Epic in the Age of Henri IV, 1572–1616. Northwestern University Press.
  • Kendrick, J., & Maynard, K. S. (Eds.). (2019). Polemic and literature surrounding the French Wars of Religion (Vol. 68). Walter de Gruyter GmbH & Co KG.
  • Rice, K. G., Hrach, S., Linder, K. E., & Maynard, K. S. (Eds.). (2025). Transformative Coaching for Faculty and Staff in Higher Education: Powerful Tools to Address Institutional Challenges. Taylor & Francis.
  • Little, D., & Palmer, M. S. (2011). Coaching-Based Framework for Individual Consultations. To Improve the Academy, 29(1), 102-115.
  • Cannon-Bowers, J. A., Bowers, C. A., Carlson, C. E., Doherty, S. L., Evans, J., & Hall, J. (2023). Workplace coaching: A meta-analysis and recommendations for advancing the science of coaching. Frontiers in Psychology, 14.
  • Jones, R. J., Woods, S. A., & Guillaume, Y. R. F. (2016). The effectiveness of workplace coaching: A meta-analysis of learning and performance outcomes from coaching. Journal of Occupational and Organizational Psychology, 89(2), 249–277.
  • Reynolds, M. (2026). Coach the person, not the problem: A guide to using reflective inquiry. Berrett-Koehler Publishers.
  • Pope-Ruark, R. (2022). Unraveling faculty burnout: Pathways to reckoning and renewal. JHU Press.
  • International Coaching Federation
  • ICF coach finding tool

Transcript

John: Higher ed has become an increasingly complex and challenging environment to work in. In this episode, we explore the role trained coaches can play in supporting agency, resilience, and career growth for college employees.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Susan Hrach and Kitty Maynard. Susan is the author of Minding Bodies: How Physical Space, Sensation, and Movement Affect Learning. She’s served as a Fulbright Canada Distinguished Research Chair, and recently appeared on the TEDx stage in Columbus, Georgia. Kitty is the Director of the Teaching and Scholarship Hub at the University of Richmond. Her higher ed career began as a Professor of French Studies, and she has taken on leadership roles in faculty development at Washington College and Virginia Commonwealth University before moving to the University of Richmond. She is the author of Reveries of Community: Epic in the Age of Henri IV and co-editor of Polemic and Literature Surrounding the French Wars of Religion. Susan and Kitty have both received training from, and are members of, the International Coaching Federation. They are also two of the editors of Transformative Coaching for Faculty and Staff in Higher Education. Welcome back, Susan, and welcome, Kitty.

Kitty: Thank you, thank you so much.

Susan: Thank you,

John: Rebecca cannot be here today. So, once again, our Dean of the College of Liberal Arts, Sciences, and Engineering here at SUNY Oswego will be filling in, and that’s Kristin Croyle.

Kristin: Thank you. I’m delighted to be here. So, today’s teas are, John, what are you drinking today?

John: I am drinking a spring cherry green tea.

Kristin: Oh ho ho. and for our guests, what are you drinking today?

Kitty: Well, I’m drinking a tea that is called Richmond Blend. I’m here in Richmond, Virginia, and the tea is from Boscovs, which is a sort of an iconic coffee and tea purveyor here in Richmond, Virginia, and it’s sort of a spicy cinnamony clovey tea. It’s probably better for Christmas time, even though we’re in July, but I definitely wanted to play along, because I’ve noticed, anyway, because I listened to your podcast quite a bit, that a lot of your guests have not been following and really respecting the conceit of the show, [LAUGHTER] so I wanted to make sure to do that today with my tea.

Susan: Oh no, and I’m a violator [LAUGHTER] because it’s going to be a swelter today here in Columbus, Georgia, and so I am having a Limoncello LaCroix, or La Croix, [LAUGHTER] as fancier people might want to pronounce it now. [LAUGHTER]

Kristin: Now, if this was afternoon here… we’re recording in the morning… I would be drinking a cold Earl Grey, but morning… just water today.

John: And going back to your earlier comment, one of our serial offenders has been Derek Bruff, who brings his tea for teaching mug, which he refers to as his coffee mug. [LAUGHTER]

Susan: Oh no. [LAUGHTER]

Kitty: It’s heresy. [LAUGHTER]

John: It’s always great to have him on, though, when we do get a chance to talk to him. [LAUGHTER] So, we’ve invited here today to discuss transformative coaching for faculty and staff in higher ed. Could you tell us a little bit about how this book project came about.

Susan: So, this is a book that was born at the Pod Network Conference in 2023 in Pittsburgh, and Karen Gonzalez Rice and I decided to pitch a birds of a feather session, which POD attendees may know more about. They’re very informal conversations. They usually take place first thing in the morning on one of the conference days, and they’re opportunities for people to informally discuss something that’s of sort of emerging interest. And so we pitched a birds of feather session on coaching. Kitty was in the group of attendees that day, and she was already working with Katie Linder and her coaching program for academics, and so within a couple of months after that November conference, the four of us had kind of connected and expressed interest in doing an edited collection and put together a call for proposals, and we were off to the races.

Kristin: Excellent. And who’s the intended audience for the book?

Kitty: Well, it’s really interesting, because we started out thinking that this book was going to be mainly for other coaches working in higher ed, or possibly people who we call coaching curious, just those who are wondering what coaching is about, and maybe if they want to become a coach. But it was really interesting as we started to talk to potential collaborators and contributors, our vision of what this book could be really expanded, and we realized that coaching has applications that really go beyond what we had initially proposed, so while the most obvious audience would be those who support faculty and staff in higher ed, that’s to say, people like educational developers, or maybe even folks who work in HR, several pieces relate directly to, for example, faculty roles like chairing a department, or even teaching, and so we ended up adding some other chapters to have a more capacious vision of what coaching could do in higher ed, so we have chapters from mid-level administrators, we have some other chapters for people working in different parts of higher ed, like instructional design, and I guess I would say our real wish for an audience, our sort of shadow audience, is upper-level administrators who hold the purse strings and can really hopefully have an opportunity to see how coaching might benefit their institutions and those working in them, and how it might be deployed throughout the university, but I think our overall takeaway here is that coaching is a really nimble way to support individuals at a complex moment in higher ed and the world.

Kristin: Excellent.

John: How did you select the authors of the chapters of this book?

Susan: We circulated our call for proposals widely among all of the coaching networks that the four of us were already a part of, and encouraged them to share freely. One reality that we recognized in seeing the need for this book was that there aren’t that many… there really aren’t any… formal organizations for those who work in coaching in higher ed, and so it had to be sort of a grassroots solicitation, and we got a really generous, lovely set of submissions that then we got together, talked about where there might be some topics that we wanted to see, but no one had submitted on, recruited a few more people who we knew had expertise in those areas, and overall we’re just super excited to have almost 30 contributors, or maybe a little more than 30 contributors to the book.

Kristin: Yeah, I was struck in looking through the book at both the unified voice, but also the really personal voices in it, and having that many contributors gives you an opportunity to have a lot of different voices in a kind of personal way. The book is divided into six sections. You were talking previously about all the different aspects that you wanted to bring in. Can you give us an overview of what kind of thing is addressed in each section?

Kitty: Sure, I’d be happy to. We start with a section on the foundations of coaching, which is really an introduction to what coaching actually is and is not. Susan just referred to the fact that sometimes coaching isn’t even really well defined, and can even have a questionable reputation, depending on who you’re talking to. So, this book, in the intro, we really established the norms of the International Coaching Federation, which is where most of us have been trained, or we’ve been trained in with organizations that are adjacent and use some of those same principles, and that is defining coaching as a collaborative process between a client and a coach to support the client in reaching goals. And then the second part is about growing coaching skills, and it really explores how to develop some of the essential skills and competencies that we associate with coaching, and how those can support folks in higher ed. So we talk about cultural awareness, we talk about knowledge of the profession, and we also talk about ways we can support the entire spectrum of neurodiversity with our clients. The next section is about applying coaching skills to specific scenarios, and so we really explore how coaching can be used in teaching, research, and leadership positions, and a subsequent section talks more about coaching for specific audiences, so we look at coaching for folks who are in roles like senior administration or student affairs, instructional technology, and also people who may just be having certain experiences in higher ed, for example, those experiencing burnout. And then we have a section on coaching the career cycle, where we focus on how coaches can help develop or help others develop vitality in the different stages of their careers. So that one’s a little more faculty focused, although not exclusively so, and then our last section is about group, team, and peer coaching materials, and that section we think about how we can scale coaching to reach more people than just a typical one-on-one individual interaction, so we think about how coaching can play a role in organizational transformation, or how we can use coaching to develop community.

Kristin: I appreciated your use of the term coaching curious, because I think having that variety of topics really gives people a chance to see the range with which coaching can be used in higher ed.

John: What led you to see coaching as a useful framework for faculty and staff development in higher ed?

Susan: Well, I had begun my academic career working in writing centers, and in that field the writer is always given the agency and autonomy to be in charge of their own work, and you work with the peer tutors, or the peer consultants, as they’re called sometimes, to respect the control of the output to the student, and so when I first learned about coaching, I immediately recognized a similar kind of ethos, which is different than mentoring and different than consulting, and different from therapy in an important way that maybe we can address later, but mostly in that a mentor or a consultant is someone who has the knowledge that they’re planning to impart to the person being mentored, whereas coaching respects the client or the coachee as the expert in their own situation, and the person who the answers really need to be drawn out from. And so I also connected with it as an educational developer, and thinking about active learning, because telling people what to do just doesn’t work, [LAUGHTER] as wise as it may be, and telling them about the knowledge or skills you want them to develop is just not as effective as allowing them to come to those realizations for themselves, and so I saw the value in it. I was already committed to its usefulness in learning, and coaching, in the end, is a lot about the learning process of the client.

Kristin: And John didn’t mention, but I’ll say, I’m a clinical psychologist, that’s my background, so I’ll just double down on what you said there. [LAUGHTER]

Susan: Yes, thank you.

Kitty: And I would also say that coaching for educational developers, coaching is kind of in the air anyway. There’s a really canonical article from 2011 by Michael Palmer and Deandra Little, which is called “A coaching-based framework for individual consultations,” and a lot of us use that, and I used it for years to help me be a better consultant when folks come in for specific pedagogical reasons, or even just to talk about career questions, and so I was already using some of those principles, and so it was a really exciting opportunity to develop those coaching skills and really take them further.

Kristin: Beyond some of the iconic articles in the field, your book cites several studies concerning the impact of coaching in workplace settings. Tell us a little about the research foundation here.

Kitty: Absolutely, so we have a very short literature review in the book that will point you to some helpful resources that talk about what coaching is and how it works. In particular, there are two meta analyses, one that came out in 2016 and one that came out in 2023 that have both looked at the literature to demonstrate how coaching is an effective workplace intervention, and more specifically, what that could mean is that coaching has been linked in the research to things like greater optimism, work-life satisfaction, engagement in work, productivity for workers, and it’s also been studied in terms of how it can improve management skills for those who are in supervisory roles. So, there’s a recognition there that more research is needed, that it’s really an evolving field and has become more and more common in the workplace over the past 20 years, but there’s also a lot that’s very promising there, and I would just say that what it really does for me is confirm the positive results that we tend to see firsthand as coaches in our sessions, or even those of us who’ve been coached ourselves, so it sort of confirms that anecdotal evidence that we already had, that it’s a very powerful practice.

Kristin: Thank you.

John: Your book title uses the word “transformative.” What makes coaching transformative rather than simply supportive or developmental?

Susan: Yeah, we address in the introduction just briefly the fact that the word transformative is more than just an adjective for a number of theories of leadership, and we want to signal with that word that we are taking a particularly open approach to multiple different ways of thinking about transformation, but I can say about the way that coaching, in particular, signals transformation to me comes from one of the leaders in the field, Marcia Reynolds. She wrote a book, I think, called Coach the Person, Not the Problem, [LAUGHTER] in that one of the most important questions and transformational questions in coaching sessions can be what makes this important, and so you’re drawing out from the client not just a solution to their immediate problem, but something deeper, something that would signal a shift that is a new perspective for them, a new insight that might change their behaviors or even beliefs, and so in that respect it’s more than a surface- level change.

Kristin: I know that none of us are unaware that higher ed is going through a lot of change right now, and that we have many faculty and staff colleagues who are experiencing burnout, who are feeling particularly uncertain, who may have difficulty seeing what their future is in higher ed and aligning their own values to some of the changes in their regions of the country. Tell us a little bit about how coaching can be useful in a setting like this.

Kitty: So I think to go back to our conversation about that word transformational, I would say that coaching is really particularly useful for people navigating transitions and change, and frankly, that’s when most people start to seek out coaching support is when they’re going through something where they’re really not sure what to do, [LAUGHTER] they’re really not sure where to go next, and so anytime there is a lot of uncertainty and change in someone’s life, coaching can be a really helpful response. It kind of gives people back that sense of agency in some way or another. In terms of burnout, in particular, we have a really great chapter in our book by Rebecca Pope-Ruark, who wrote the book Unraveling Faculty Burnout, where she really explores this in terms of faculty, and in that chapter she offers some tools for coaches to help people rediscover purpose, compassion, connection, and balance, and those are the four things that she notes that are pillars of burnout resistance, the things that help us find ourselves when we’re facing those challenging moments in our careers. And I would say more globally, as we’ve started to think about coaching as an ecosystem that could exist within higher ed, I see it as a hopeful space where we can actually build more humane and inclusive higher education workplaces that really respect the individual needs of the people who work here.

Kristin: Thank you.

John: Teaching centers and faculty developers often serve a role where they answer questions for people, they provide models of effective teaching practices. They do workshops and so forth, but the coaching approach is a bit different than that type of interaction. Can you talk a little bit about how a coaching approach might change the nature of interactions with teaching centers and what a typical coaching session might look like?

Susan: So, the most conventional kind of coaching is one on one, and so it’s not a workshop typically, not that a workshop can’t employ coaching strategies, but as we were discussing earlier, it’s about drawing out the expertise of the person that they already possess, so a coach would ask: “Where are you now? What would be different if you achieve this change that you want to make? What are your options for getting there? What obstacles can you anticipate, and how will you respond? What’s the first step?” And so those are each pieces of a coaching framework that the client has to answer for themselves, and so if you think about it in terms of a client who might be also interested in changing their teaching strategies, let’s say it’s slower but might be ultimately more successful route to making that change. You’d have to acknowledge that it’s going to take significant time and resources to adopt new teaching strategies and refine them, or to envision a different career path, or a purposeful retirement. And coaching is allowing you to get clear on why making that change is important enough to invest that time and energy. And so if you don’t know what strategies you might want to employ, then the coaching conversation might include, “Let’s talk about what your resources are. Who do you know? Where could you go? What actions can you take in order to find those things.” So again, you’re not providing the answers, you’re sort of helping the person think about where they want to pursue those.

Kristin: And I know many faculty are well versed with what an individual teaching consultation might feel like, but how would that differ from how a typical coaching session might feel or look like?

Susan: It’s different in that, and this is very important for clients to understand, it’s usually part of the initial meeting in which we take time to talk to them about their expectations and how coaching is different from mentoring or consulting, which is that the coach is not going to hand you the answers. [LAUGHTER] If that’s your expectation, then this is not, maybe the interaction, or the…

Kristin: Oh, but that would be so much easier., wouldn’t it? [LAUGHTER]

Susan: It would…

Kristin: You’d just have the magic answers on hand.

Susan: It would and yet, it wouldn’t be as successful in leading to a lasting change, or growth, or learning, which is hard and takes effort on the part of the person who’s seeking out that growth.

Kitty: And I would just add to what Susan said by saying there are times when coaching isn’t the solution. Sometimes, in a pedagogical consultation, your expertise does come into play, that people really do want to know, like, what should I do in my classroom, what kind of warm-up activities should I do in my classroom, and do you have any ideas for me that I could use? And that’s not a coaching conversation, that’s a consulting conversation, and that’s okay. It doesn’t all have to be about coaching, but I think that coaching skills are valuable, and I think to get back to Susan’s point about her writing center work, the idea that you are seeing the person who’s coming to you for a consultation or a coaching session, you’re seeing them as someone who has more answers than maybe they even realize, and helping draw that out for them with powerful questions, and sometimes coaching tools and deep listening, that is valuable in a consultation too, even if it is a moment when you maybe are going to give somebody a more direct answer, or you’re going to be like, “Here’s a book you could take a look at, and it has all sorts of great ideas in it.” So, they both can be very helpful in this work.

Susan: That’s right.

Kristin: Yeah, I like the writing center analogy, because the best writing teachers are helping people find their voice and express their own voice, they’re not telling them what to write.

John: So, part of coaching involves having the clients identify their own goals, but one of the problems that we all have sometimes is having the motivation to achieve those goals. How can coaches help their clients remain motivated to achieve their goals?

Kitty: Well, this is a foundational piece of what coaching does, and I think the first thing is that it helps people identify their values as well as their true priorities, which sometimes can get lost in all the doing that consumes so much of our time, so it taps into intrinsic motivation, which means that if we are able to set goals that matter to us, that helps us make meaningful progress on those goals. I think sometimes we have goals in mind and they’re actually not our goals, their someone else’s goals. [LAUGHTER] I mean, sometimes that has to be the way it is in our jobs and in our lives, but a lot of times there are ways that we can find to tweak the goals, so that we are actually more naturally motivated to accomplish them. But coaching is also really helpful with project management. It’s helpful in how to plan to find mentors and to find the help that you need to accomplish a goal, and what kind of resources are needed, and then it also can provide accountability structure, which is another thing I think, in particular for academics who do a lot of independent work, where deadlines are a little bit more fluid, or sometimes incredibly fluid, depending on where you are. It helps kind of stay on track. Also, sometimes, for example, in coaching, I actually will write follow-up emails to people, so that’s part of my support is checking in to say, “Hey, last week you said you were going to do this, how’s it going for you, how is that going? And sometimes that’s enough to help people kind of loop into the goals that they want to achieve.

Kristin: And I know that faculty who have worked in a more traditional faculty role versus ones who are working in an administrative role, where there’s more of a team environment, will sometimes find that different structure also to be helpful. So, yeah, it’s finding ways to build your own structures in, yes?

Kitty: Absolutely. And that’s one of the other types of conversations we would have with a client, is: “What’s going to help you keep on track. What do you need to keep on track?” and “How are you going to build that into your life?”

Kristin: What role can coaching play in helping faculty and staff make more sustainable decisions about their time, their energy, their professional identity? How they can make decisions that will pay off for them in their own satisfaction for years to come.

Susan: Yeah, thank you for that question. I think it’s been really interesting to me over the past few years of working with colleagues how much it matters to have someone give you their undivided attention and presence for a number of hours as your coaching series evolves. Even when I was feeling very sensitive about my own development coming out of training as a coach, I would be very conscious of where I had not asked the best question or had fallen through the cracks of where the conversation might have gone and yet there was no question by the end of the conversation that the person was just so pleased to have had the chance to talk and have someone really listen to them carefully, and how desperately people needed that, [LAUGHTER] and so I would say that the sustainable decision making is partly an outcome of having made the choice to invest time and energy in a coaching relationship, like you’ve already taken the first most important step, which is to say, I am going to carve out some time, which, as we know, time is maybe the most precious resource that most academics have, we’re not in it for the money, but we sure do spend time. So, if you’ve decided I’m going to intentionally dedicate an hour on several occasions to talk through an important issue for me, then you’ve already kind of signalled to yourself that this is something you’re prepared to sustain and be committed to, and as Kitty was saying earlier, the accountability that coaching provides is one of the really important things that we practice in between each session, and that ultimately by the time we reach the point where we’re ready to at least temporarily bring that series to a close, t he person needs to have identified what their own accountability practice is going to involve, and so there’s a sustainability priority that’s kind of built into the coaching conversation.

Kristin: I can imagine that some people who have not experienced coaching may listen to this and may think, but I have a very supportive colleague or partner or friend that I have these kinds of conversations with. What do you see as different in the coaching relationship than a great supportive personal relationship?

Susan: That’s a great question, and it’s a question that I asked myself when I was starting the training, because I thought, “Well, you know, I have decent conversational skills. How hard can this be?” And then I realized that it’s actually incredibly hard, because coaching conversations do not follow normal social turn taking. You, the coach, are not an equal, like, interlocutor in that relationship. The client should be doing way more talking and way more thinking, and you are posing powerful questions for them. And so it is not a normal conversation, [LAUGHTER] and it’s hard to develop the skill to sort of be quiet and make space for their thinking to emerge. Sometimes there’s long silences while you’re allowing people to process their thinking, and that’s not the typical friend or colleague kind of conversation.

Kitty: And coaching sessions are actually very highly structured. We follow a pretty standard structure of every coaching conversation, so in that sense, as well, it’s different than having a good supportive friend or colleague, although we all need that too. So I certainly don’t want to say that those conversations are not incredibly valuable, but I would say in my own experience, when I’m having a conversation with a colleague or friend, a lot of that is about getting advice, specific advice from people, which is also incredibly helpful, but coaches aren’t going to give you that advice, they’re going to draw out your own advice for yourself. One of my favorite coaching questions is something that Katie Linder taught us, which is a lot of times clients come to you and they actually expect to be getting advice from you, and so that’s something that we sort of have to prime with our clients, is that we’re not going to be giving you specific advice, because that’s something we’re going to trust you to find yourself, but when somebody does come to you and they really want advice, Katie’s question is, “What would you like me to give you permission to do?” because it turns it back on them again to say, “Hey, what do you really want to do, what are you looking for me to say to you that you want to do yourself?” which I think is a very clever question, and it’s really fun to ask it, [LAUGHTER] it’s really fun to see what kind of answers you get.

Kristin: And I’ll add, too, that having a separate relationship, a coaching relationship, you don’t have to worry if I say this thing, how is my friend going to judge me, or if I say I want to do this, am I ready to talk about this piece with my partner? There’s other complexities in our long-term personal relationships that in a coaching relationship you don’t have to worry about how are they going to change how they see me or what is the impact of this going to be, where there’s a reciprocal kind of emotional care where you’re trying to take care of the other person. Coaching gives people an opportunity to not have to care about [LAUGHTER] not in the same way, to not to take care of the other person, that that’s the time for them to think about how to care for themselves.

Kitty: That’s an incredible point. I think it’s really, really important, and gets us back to some of the principles of coaching, where we are confidential. It’s really our relationship with our clients, it is 100% between us, it never leaves that conversation unless it’s some professional obligation, because someone is expressing harm or something like that, but otherwise it is absolutely confidential, and so people can try on things that they might not feel comfortable trying on with a colleague or a partner.

John: It sounds a lot like psychological counseling in many ways, doesn’t it?

Kristin: In just certainly non-directive approaches, there are similarities, and many of the roots come from the same roots.

John: One of the things you address in the book is how coaching needs will be different for different people at different stages of the career and in different roles. How might coaching needs differ for early career faculty, mid-career faculty, senior faculty, chairs, and staff leaders?

Kitty: I would say in many ways the coaching approach doesn’t change much with all of these audiences, because our practices are similar. We are going to ask powerful questions, we’re going to be listening deeply in a way that empowers the individuals we work with, but coaches often develop expertise in a particular area, depending on where they’re working and the spaces they’re working in, and when that happens, coaches tend to see similar issues come up, so they become more skilled at asking the right question at the right time, or understanding a particular tool is going to be really useful for somebody in a particular career phase. So to your question, John, it’s not unusual, for example, when I encounter people in their early career to be struggling with finding mentors or to be trying to figure out what kind of workflows are going to actually be effective for them as they’re juggling all of the new things that they may have not had to juggle in graduate school. On the other hand, someone who’s a little bit more advanced in their career might be thinking about legacy or thinking about retirement. Someone in mid-career might be just getting tenure and trying to figure out what’s next for them after they’ve jumped all these hurdles and they finally come to the end of the hurdles and are looking ahead and saying, “Well, what do I do now?” So, really, I think a general coaching approach can help with any of those concerns, but a coach with a background in those areas is likely to have the tools to address a particular need. And so, one of the nice things about our volume is it has some great chapters to help both coaches and those who are kind of going through those different career phases.

Kristin: How can coaches help faculty and staff navigate conflicts between their own goals, values, and commitments and institutional pressures, particularly in settings where coaching may be available within the institution.

Susan: This is a really important question, and it’s one that touches on some things we’ve already mentioned having to do with ethical commitments. The International Coaching Federation, in fact, requires an ethics training as part of the credentialing process and re-credentialing process, and so for internal coaches this is particularly important, and as Kitty mentioned, the confidentiality piece is primary, but the other thing that kind of surprised me when I first started the training and remains really important to my sense of the value of doing this in higher ed is that the coach’s primary responsibility is with the client, not the institution. And so, I was really kind of shocked when I asked the question in my training: “Well, what if the person wants to quit? Then what do you do?” And the response was, “Well, then you need to explore that with them.” This is not a place where your job is to make sure that they stay with the institution, no matter what. You’re helping them think through a very difficult professional crossroads, and so if maintaining their affiliation with the institution is really a valuable and meaningful priority for that client, then okay, we know that that’s one of our anchors. We’re going to have to figure out how to manage this conflict in other ways, but if the conflict turns out to be more significant than staying in that job or role, then the job of the coaching partnership is to think about how to manage this transition. What would be a better fit? What does that look like, how are you thinking about that, and so there’s a great chapter in the book, in particular by Chinasa Elue, that is about coaching. Well, let’s see, what does she call it?

Kitty: Walking the tightrope…

Susan: Walking the tightrope: Coaching faculty clients who decide to leave the university.

Kristin: And as a Dean, I will say that if someone talks with me, and there are faculty and staff who talk with me about their quandaries about whether they should stay or go, I actually see my role in a very similar way. I want them to be thriving. If they’re thriving here, it’s good for the institution, but if they’re not thriving, it’s not good for them, and it’s not good for the institution,

Susan: That’s right,

Kristin: But that’s a very different conversation, certainly, than it would be with a coach.

Susan: It is, and I have noticed that it’s particularly difficult and regrettable that the system of tenure makes many people feel that there is no possible way for them to walk away from this. They grabbed the gold ring, and so, boy, they better stay. And so it’s a life crisis. [LAUGHTER] And so coaching is a really wonderful tool to have to be able to work through that kind of crisis.

John: So, who should be trained in coaching skills? Clearly, from the main theme of the book, teaching center staff should be, but might this also be appropriate for chairs and deans and student affairs professionals and supervisors and peer mentors, and might it even be helpful for faculty, because our relationship with students is also a coaching relationship in many ways.

Kitty: Yeah, I would say absolutely to all of those things. I think coaching skills are really helpful for anyone, and in particular, anyone in a type of role where you’re being asked to mentor or support or guide others in some way. I think one of the biggest revelations for me during my coach training was the breadth of occasions where I found deep listening and powerful questions to come in handy, not just professionally, but even personally, in some moments. So, I think when I think of faculty members, John, as you noted, working with students, coaching skills can be so helpful, or if you’re doing advising with students in any way, but I’d also say with colleagues, I think one of the things that I have experienced myself as a former faculty member is we’re not always good at talking to each other, we’re not always good at listening to each other, we’re not always good at asking the right questions of each other, and so those skills can be applied in all sorts of contexts.

Kristin: What mistakes should campuses avoid when they first try to introduce coaching?

Susan: In the beginning of the book, we have a couple of chapters. These are Katie’s on how to seek out certified training programs, what questions to ask, what are all of the options you might consider, and for what reasons. But training is key. So I hope that our book is an important contribution to people who are at the coaching curious stage, but I wouldn’t want someone to read our book and then decide, okay, we’re ready, [LAUGHTER] let’s do it. There’s some very important standards that we want coaching professionals to be able to meet and uphold, and institutions can also look into outsourcing, just like with any other professional development. The imperative to develop your own internal coaches doesn’t have to be where you start, and in the corporate sector, executive coaching has been a thing for a long time, and I think upper-level administrative leaders get a little bit of that because they fall into that kind of executive identity, but what we’re really hoping to do is democratize this resource, and so where deans or provosts or others who have control of institutional budgets might be interested in whether coaching could be a useful way to help people who could benefit. I think there are other places to begin.

Kristin: Yeah, I think the influence of kind of corporate leadership coaching has been a positive in many ways, because people see it, they see the value, they think, “Well, if that’s something that is a given in those settings, and people will pay a lot of money for it in those settings, that it must have actual value.” So that people who are not responsive to the research foundation may be responsive to that argument. Instead, I think the first time I hired a coach for someone was 15 years ago, saying, “Look, you have great leadership capacity and skill, but it’s not something that you’re able to bring out. Let’s see if a coach can help bring that out.”

Susan: Oh, thank you for saying that, Kristin, and if I could just add something else about ways we would prefer not to see coaching implemented is as remediation. Your impulse was to see someone who has great leadership potential, because coaching is about where are you now and where do you want to go, and I would love to h ear more from you about how you see therapy and counseling as distinct from that kind of orientation, because my understanding, or the way that I explain it to my clients, is that if we get to a place where you seem unable to move past something that happened to you in the past or some conflict you had with someone in the past, and we just keep being drawn back to this issue that clearly needs to be resolved or healed in some way, then I’m going to make a referral to you to a professional counselor, a professional therapist. A coach should be trained to recognize when referrals are the appropriate action.

Kristin: Absolutely, absolutely, and of course it depends on the therapeutic approach. Some therapeutic approaches focus more on skill building as well, or someone may not have developed the skills that they need to really thrive in interpersonal relationships, for example, or they may have significant substance abuse backgrounds, or there may be all kinds of things going on, and a great therapist is tooled to be able to handle complexity and to be able to recognize how those complex pieces act together and help somebody move forward in their lives and their goals. I certainly would think that the ultimate goal is very similar. We’re all trying to help people in education as well identify the things that they see as really valuable in their lives and the capacities that they can see that they have to live a meaningful and joyful life, that is our goal across the board. I’m sure that is the goal of many of our faculty as well, with the students that they work with. We just take a somewhat different approach in doing so. I’m also thinking that listeners may have one other question. You may have gotten them really excited, so can I ask the question, if someone is looking for a good coach, what might you suggest in how they could find one.

Kitty: Well, I would say one thing to know about coaches is that most of them do a free half an hour consult with their coaching practice, so there’s nothing lost if you find someone who looks interesting to you, and you just want to have a conversation to see if you vibe with that person and their approach. In terms of finding a coach. I would look for someone who has an ICF credential personally, just because having gone through that training, I know that it’s quite rigorous and quite demanding, and the people there have been trained quite a bit with ethics, and they’re trained to recognize the difference between therapy and coaching, and know where their limits are. So, I would definitely look for that as well. And I might just ask around a little bit to see if anybody you know has had a great experience. I know that’s how I found my career coach that I had back in the day, that I had a really excellent experience with, which was one of the things that got me into coaching in the first place. So, Susan may have some better advice than I do on that one.

Susan: No, I think you answered that really nicely. There is, I believe, a coach finder tool on the ICF website that would be the whole breadth of coaches all over the globe, so that may be a deeper dive [LAUGHTER] than you’d want to do right at the beginning.

Kristin: And I’ll also double down on a short consult session is a great idea. Sometimes people just don’t jive, and it’s no reflection. The coach may be a fabulous coach, it doesn’t mean that the person requesting coaching is all screwed up because they didn’t jive with this coach. Sometimes there’s just not a connection, and it’s not a reflection on either person, but it’s worth that first conversation to say, “You know, this just didn’t seem to go. I’m going to just try again,” because having a good connection can provide a really good foundation.

Susan: That’s right. And coaches are typically very generous. I have also had other coaches make referrals to people who might be better suited, for example, when I was seeking out mentor coaches for my re-accreditation, and I appreciated that generosity.

Kristin: Tell us about your experience of writing the book.

Susan: I would really like to praise our contributors to this volume, not least of which for being the most organized, well-managed time writers that I think have ever existed in a co-edited collection of this size. It was such an interesting experience to work with people who all sort of work with other people about time management issues and writing, because they were on it. No one had to be repeatedly nudged into getting their submission in. It was a really smooth process, and it was such an enjoyable experience to work with Kitty and Katie and Karen, because four people is a nice group as a editing team. When one of us had a life got in the way of being able to keep our balls rolling, the other three people would always pick up the slack, and so I think we were able to make incredibly smooth progress on getting this book to the press.

John: We always end with a question, what’s next?

Susan: I am actually heading full force into this coaching world. As of this summer, I’ll be leaving my role as a director of Center for Teaching and Learning to focus on coaching for individuals and for organizations, in addition to the speaking engagements and workshop facilitating that I have been so honored to do with a number of universities, my focus on active and embodied learning and my coaching practice specifically offers an embodied approach, because I like to conduct outdoor audio only sessions, in which clients can be walking or enjoying some fresh air to develop new perspectives and spending time outside.

Kristin: Excellent.

Kitty: And for me, I’m sort of in the midst of a lot of reflection and planning for the academic year here at the University of Richmond. So, I’ve been particularly busy thinking about ways to support folks in our mid-career. We revamped our early career support programs with the Provost Office last year, and this year we’re starting to focus on mid-career folks, so really post-tenure, or for those not in a tenure-line position, more advanced teaching associates, and so I last year did a group coaching with the Associated Colleges of the South across all the schools of that consortium, and we’re going to do one here at UR next year. So, I’ve been working a lot on that and thinking about group coaching and how we can apply that more effectively in my institution.

Kristin: Sounds so interesting.

John: Thank you both. It’s great talking to you again, Susan, and it’s nice meeting you, Kitty. I really enjoyed reading through your book. It’s a really nice collection of resources.

Kitty: It’s been a pleasure.

Susan: Well, thank you. Been great to be featured on Tea for Teaching, because it’s a little outside of the normal topics, and so we’re so happy to spread the word.

Kitty: And I’ve been a fan of Tea for Teaching forever, so it’s so exciting to finally make it on the show. [LAUGHTER]

Kristin: Well, I would love to hear about the group coaching and how that goes, [LAUGHTER] but that’s just putting in a listener request. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

454. Designing Impactful College Courses

Student motivation is enhanced when the framework of self-determination theory is used to design courses that support student autonomy. In this episode, Daniel Guberman and Chantal Levesque-Bristol join us to discuss a new resource designed to assist in the development of such courses.

Daniel and Chantal have leadership roles at the Center for Instructional Excellence at Purdue University. Chantal is the Executive Director and Daniel is the Assistant Director for Student-Centered Pedagogy.  They are the authors, with Emily Bonem, of Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy Supportive Learning Environments, which was recently released by Routledge.

Show Notes

  • Bonem, E. M., Guberman, D., & Levesque-Bristol, C. (2026). Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy-Supportive Learning Environments. Taylor & Francis.
  • Kuh, G. D., & Levesque-Bristol, C. (2023). Student-centered pedagogy and course transformation at scale: Facilitating faculty agency to IMPACT institutional change. Routledge.
  • Purdue University’s Instruction Matters: Purdue Academic Course Transformation (IMPACT) program
  • Lang, J. M. (2021). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • The Grading Conference
  • Mentzer, N., Asad, A. M., Koehler, A. A., Mohandas, L., & Mammadova, E. (2025). Engaging Learners Everywhere: The Interactive Synchronous HyFlex Method. Purdue University Press.
  • Student Pedagogy Advocate Program at Purdue
  • Center for Self-Determination Theory

Transcript

John: Student motivation is enhanced when the framework of self-determination theory is used to design courses that support student autonomy. In this episode, we discuss a new resource designed to assist in the development of such courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Daniel Guberman and Chantal Levesque-Bristol. Daniel and Chantal have leadership roles at the Center for Instructional Excellence at Purdue University. Chantal is the Executive Director and Daniel is the Assistant Director for Student-Centered Pedagogy. They are the authors, with Emily Bonem, of Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy Supportive Learning Environments, which was recently released by Routledge. Welcome, Daniel and Chantelle.

Daniel: Thank you. It’s great to be here.

Chantal: Thank you. Thank you so much.

Rebecca: Today’s teas are:… Daniel, are you drinking tea with us today?

Daniel: Yes, I have a Bengal spice tea, which is basically like chai without the caffeine, as I understand it.

Rebecca: That sounds nice. How about you, Chantal?

Chantal: Yeah, I just made some tea. I have some green tea with some notes of apple spices and oranges.

Rebecca: That sounds nice.

John: Very nice.

Rebecca: How about you, John?

John: And the Bengal spice, I think, is the first time we had that on this podcast.

Rebecca: I think yeah, it might be a first time. It’s special. [LAUGHTER]

John: And I have an English breakfast tea today.

Rebecca: Okay, and I think I have a Scottish breakfast that’s mixed at this point. I think with an English afternoon, it’s all mixed together. [LAUGHTER]

John: Well, as we approach late morning, it sounds like a good brunch time. So, we’ve invited you here today to discuss Designing Impactful College Courses. Can you tell us a bit about how this book project got started?

Chantal: Yeah, so really I’m going to go back to a first book that I wrote on the Impact program, was published in 2021 by Stylus, which is now Routledge as well, that was really talking more about the structure of the Impact program and a lot more for faculty developers, and this book is sort of the sequel to that, but really looking a lot more about how you would design specifically a course, and this one is a lot more for faculty themselves or instructors, people that are actually teaching that are really interested in understanding how to use the principles of self-determination theory to design courses that are autonomy supportive and self-determined.

Daniel: And maybe I’ll add a little bit about the Impact program, for those who don’t know, which is Purdue’s course design program, which has been going on for over 15 years now. I’m a few weeks away from my 10-year anniversary at Purdue, and Chantal is a few years longer than that. It predates both of us, and it’s a semester-long course design program based on a faculty learning community, and it’s done in coordination with our teaching center, libraries, and teaching technologies. So, we get groups of faculty, we break into groups of three to five, and they each work with members of those three teams, and it varies every year on what our capacity for support is, but we’ve worked with over 600 faculty across this time, and so we’ve had semesters where over 90% of students at Purdue are taking a course that went through this program, so it’s been a really impactful program, pardon the pun. It’s a good name to just like constantly have puns, although unfortunately, whenever I write the word impact out, I write it in all caps, so that I need to go back and edit all my documents. So it’s been this really long and successful program, and we presented on it, and we recognize that a lot of faculty, instructors, graduate students don’t have access to this type of program, and even instructors at Purdue, for various reasons, can’t commit a whole semester; their schedule doesn’t work. So, we’re really excited to bring a lot of these lessons and ideas to a broader audience.

Rebecca: It’s incredible to have such an impactful program, certainly reach so many people, and to build on that in your book. One of the things that we observed as we were just even skimming the table of contents is that the organization seems to have something in common with a philosophy or a strategy that instructional designers often recommend, which is backwards design. Can you talk a little bit about the organization of the book, what it does have in common with that strategy?

Daniel: So, backwards design was at the core of the book, and the program, that program followed a backwards-design structure even before Chantal arrived and brought self-determination theory, and we’ve kept that. Probably the biggest change is that we brought this theory as the foundation, which in a lot of backwards design work does not have a sort of core theory that is structured to follow throughout everything that you’re doing. So we start with the theory, and then following traditional backwards design, for those who don’t know, we talk about the outcomes and objectives of a course, then the assessment, which we frame around grading specifically, because we found a lot of traditional course design approaches talk about assessment, but don’t tie it to grades, and then at the end they just sort of throw some numbers on all of it, and it feels like that can undermine the connection between the assessment process and what students are experiencing. And then we go to assessment types, which we have chapters on a variety of different assessment types, and also activities and core structures that are basically from a backwards-design perspective, going from what will students know or achieve by the end of the class, how will you know, which is the assessment part, and then how will you get students to the point where they can demonstrate that effectively. So that’s sort of our broad picture overview of the book, and we think it makes a lot of sense for someone who’s designing a course, which isn’t to say we view it as a linear progression, but rather we’ve tried to structure it in a way that you can skip around and you can go back, so you might get to thinking about, “Oh, I really want to bring experiential learning into my class, and that means a group project, which means I need to go back to assessment, “so we don’t think of backwards design as a linear process, but more sort of like a spiral that you move through.

John: And it may nudge people to think about it, because faculty don’t always think about course design using a backwards-design perspective. In fact, it’s often the opposite, but in addition to having this backwards- b design structure through the book, after going over the basic theory, you also have a consistent structure for each of the chapters. Can you describe this structure and talk a little bit about why you chose this particular organization?

Daniel: So each chapter we start with a vignette that comes from a student perspective, because it was really important to us to think about how students are really a central part of the learning environment, and they’re often omitted from the course- design process, and this has been something we’ve been doing throughout Impact for a number of years, is better integrating students into that process. The vignettes are based on true stories from students, but adapted a little, actually, a lot of the students, they came up with overly positive stories all the time [LAUGHTER], we’re like, “Tell us some of the negative ones too.” So then, after the vignette, we have an intro to what the topic is. So some people might not be familiar with what we mean by the experiential learning, or even when we say, like, this is a chapter on lecturing, like, what does it mean to write a chapter on lecturing or class discussion, then we get into the basic needs, and we talk about how the basic needs can be met or hindered in each of these contexts. Then we always follow the same order from competence, autonomy, and relatedness. So I’m sure I’ll talk more about what those mean in a bit. And then after that, we go into case studies. So one of the really fun parts about writing this book, is we got to go back to some of these 600 faculty members who have worked with this program, and we interviewed them and had really meaningful discussions to understand the ways that they’ve actually applied it, because as anyone who’s worked in a course- design program will know, there’s a lot of work that gets done in the design process, and then students come into the picture, and things change, and that’s important. We don’t want someone to design a course, and then students arrive, and they’re like, “Well, you are not viewing the course in the way I do.” So, we want courses to adapt. So, it was really cool to hear about how these courses adapted, and how approaching course design through a self-determination theory lens helped them make those adaptations. After the case studies, we have a series of guiding questions to help someone get started, and we have our short reflections about each of those, inspired by James Lang’s Small Teaching, which I feel like everyone knows by this point. We have a section called Starting Small, which are designed to be small things you can do, because while it’s a course design book, we realize not everyone is going to approach it thinking I’m going to redesign my entire course from the ground up, and again we have some sort of reflections and ideas about what that might look like, and then we just end with some short takeaway bullet points.

Rebecca: You’ve hinted at this already a little bit, but can you talk a bit about who your intended audience is for this book.

Chantal: This one is really for anyone who’s teaching a college course, or anyone who’s even interested in teaching. So we could be thinking about graduate students, for example, that may be asked to teach a part of a course or an entire course with little preparation. This book would really help them understand how it is that they can really approach teaching from a really a deep perspective in really understanding students, which I think is very important, as Dan mentioned, that we start with a vignette, which brings in the student voice first and foremost to really think about them as part of the learning and teaching endeavor. So also it could be for someone who is working at an institution where they don’t have their own faculty development program, or it could be for some faculty developers that do not have a very established structure at their institution. How can they talk about teaching and learning in a different way to their administration for example, and also I would say anyone who’s interested in self-determination theory and the teaching and learning principles of self-determination theory and autonomy-supportive environments, and how would that look like in a course-design environment?

John: So, you mentioned self-determination theory, and we talked about this a little bit on earlier podcasts, but it’s been a while, and many of our listeners may not have caught that, so could you talk a bit about what self-determination theory is? Because it does serve as an organizing framework throughout this book.

Chantal: It does stop me if I start talking about it too long. That’s definitely my passion. I say this is how I’ve been raised in terms of an academic, so my research area is also in motivation, and this is as Dan mentioned. When I came to Purdue University in really the impact program was just starting, it was really something that I brought in terms of thinking about how we design courses. So to summarize really quickly, the core principle of self-determination theory is the existence of three basic psychological needs, right, that all humans have. So, it’s a humanistic theory of motivation, and it’s about competence, autonomy, and relatedness, which is how we structured the book. And they’re really baked into every chapter. And competence is something that we understand very easily in higher education. It’s this idea of developing your skills, mastering, becoming proficient, but often we focus very much on competence to the exclusion of the other two needs, and they’re also very, very important. And when we talk about autonomy, which is probably one that is a little bit harder to understand, autonomy does not mean independence or doing whatever you want to do, right? That’s not what it is. It’s not about that independence, it’s about choice and volition, and really empowerment, and really feel like you are sort of the agent of your own development, and that applies for students and for instructors as well. And then relatedness, which is very important, which is the sense of connection with other human beings, with the instructor, with other students, it’s very relational, but also we think of it as relationship with the content that we’re covering as well, which also is highlighted in many of the chapters, and when we are creating environments where the satisfaction of the needs is met, where those basic needs are satisfied, then we see a progression toward more self-determined types of motivation, so from extrinsic type of motivation to more self-determined or intrinsic type of motivation, and this is really one of the very important contributions of self-determination theory, is to think about extrinsic motivation in four broad categories of extrinsic motivation. So, you may be familiar with extrinsic motivation as doing something for the rewards or to avoid the punishment, which is only one type of extrinsic motivation. So, extrinsic motivations are about means to an end, you’re doing something to achieve an outcome, but there’s different levels, so after external regulation, which is really doing something to attain a reward or to avoid punishment. So the typical understanding of grades would really fit within that type of extrinsic motivation. You’re trying to get an A because that’s how you get the rewards, and you go to class because you want to avoid losing points, that’s really the type of extrinsic motivation that we’re talking about in external regulation. We move towards still something that is not self-determined, but we think about introjection, which would be doing something because you feel guilt or shame, so you’re trying to avoid feeling those sorts of feelings, so it’s more internalized, but it’s still not self-determined, right? It’s an internal pressure, it’s perceived as an internal pressure. And then the next two types of extrinsic motivation are more self-determined. Identification, for us, and we talk a lot about this in the book, is sort of the sweet spot for us in academic endeavor. It’s really when you start to understand the value of your behavior and how it makes sense, how it’s meaningful, how it’s important to you, so that’s really how identification is formulated. And although you’re trying to do a behavior or you’re trying to still achieve an outcome, the outcome is perceived to be important and valuable, and then the last form is integration, which is about an alignment with the self, so it’s something that is so in line with yourself that you’re doing it because it’s part of who you are. So, I think this is important to keep in mind in terms of the different types of extrinsic motivation. I’ll stop here, see if Dan has anything to add.

Daniel: And what I’ll share is my background. Emily and Chantal are both psychologists by training. My background is as a music historian, so very different. But when I came to Purdue and was introduced to this theory, I just found it really compelling. So, I think this is what we’re hoping readers from a variety of disciplines will find, is that they read about the theory, they hear about the theory in places like this, and they find that same connection when you think about, “oh yeah, I can see examples of myself in each of these types of extrinsic motivation, and I can see how in environments where my basic needs are met, I feel more internally motivated to do something.” If you start to think about it, you really start to feel that, like,”oh, this resonates, it works, it’s not this abstract theory that, like, we hope will apply in our class, that we can feel it throughout our everyday lives,” and then it’s a matter of how can we apply this meaningfully in our classes as well.

Rebecca: I think many faculty experience moments in their classes where maybe particular material or concepts are not overly exciting or interesting, but maybe very foundational, or maybe students are in a general education class and it’s hard to get motivated around that. What are some strategies for helping students develop motivation to develop their knowledge and skills in these areas that maybe we deem as important, but maybe initially they don’t.

Daniel: Yeah, I think that’s such a great question that gets at the core of the book, and one thing that I think is really important, and one of the key misunderstandings or like the threshold concepts, if I may, when it comes to self-determination theory is the recognition that when we talk about these basic needs for autonomy, confidence, and relatedness, it’s about the individual’s perception of those needs for themselves. I think a lot of people come and we do an introduction the first week about, like, what are the basic needs, and they’re like, “Oh, that’s great, I’m giving my students choice, so I do unessays, so I give them choice that they can choose what format their project is going to be, and therefore I’m meeting their need for autonomy,” but and I’ve experienced this myself, is that that doesn’t necessarily work for all of my students, that some students will see, like, “oh, I can do anything,” and they’re like, “well, that’s scary, I don’t know, I guess I’ll just write a paper, and it’s not because I really want to write a paper, it’s because that’s the thing I’m used to doing, and I know that’s a safe thing to do.” So that student has not perceived that they’ve had autonomy in this assignment, even though I’ve told them, like, “oh, you can do a poem, you can make a video, you can bake a cake,” but at the same time they haven’t seen paths for them to do that, and I haven’t necessarily provided… at least when I started doing this, I really did not provide guidance to make a meaningful choice, and that’s the key when we think about autonomy, is how do I create a structure where a student is going to see this choice in my class and feel like that choice is meaningful to them, so when I give them the unessay and say you can write a poem, I need to then say, like, well, what does it look like to write a poem for this, and what resources, because I am not someone who could help them write a poem, but what resources are available on campus if this is something they want to explore that can help them do that in a meaningful way, and then I can also provide the feedback that I’m capable of providing, in thinking about how to tie the content of the class with the poem, and really help them then also feel that sense of confidence that all these needs get intertwined all the time, because that’s also then about the relationship I’m able to have with students to help them see that, like, oh, when I give you a choice, I mean it, and I’m also here to help you make a meaningful decision for you. So, I think the key is always to try to think about the students’ perception and experience. When we start doing that, we start to help overcome these motivational boundaries. And I say this, coming from a background in music history, where even when I taught music majors, a lot of my colleagues on the performance side of that school would, like, tell them, “Well, the music history and theory classes aren’t that important, you should spend your time practicing.” So at that time, I felt like it was always the struggle of, like, I need to motivate students for this class in their major, and I didn’t know about self-determination theory at the time, but now, like, oh, I missed all these opportunities to help them really understand why these things were important, and to connect with them in doing so.

Chantal: Yeah, and I want to add that it’s really important to think, like, it’s not a recipe, it’s not a recipe book. You’re not going to find in this book that, like, this is exactly how you need to do it, and here’s step one, two, and three, and that’s why the student voice is very important, that student perspective and the perception, as Dan mentioned, because it takes getting to know your students to really understand what meeting the basic psychological needs is going to mean in your context, and also what you’re able and willing to do as an instructor, so you could implement a technique that would be very effective at meeting the basic psychological needs, but if you do it poorly, you’re going to have the opposite effects. So, if team-based learning is not your thing, and you don’t think you can implement it well, that is not why it works, if that makes sense. It’s really important to understand that it’s not the what that is as important as the how you implement that pedagogical strategy.

John: One of the things you note in this book is that grading is one of the things that faculty tend to really not enjoy particularly much, and it’s also something which can be somewhat demotivating for students. What suggestions do you have on trying to shift the focus on grading, so that it can be a bit more motivating, and perhaps not quite as unpleasant for faculty too.

Chantal: That is so Dan’s question. [LAUGHTER]

Daniel: Yeah, [LAUGHTER] and I’ll say, as we’re recording this, just before the Grading Conference begins this summer, and I’m a co-organizer of the Grading Conference. If you’re familiar with the Grading Conference, you might understand where my perspective on grades is coming from, but with that said, I don’t think that means we need to, like, do away with all grades, which is some answers, but I think the key is getting away from the view of grades as a mechanism to control students’ behavior, because ultimately, what self-determination theory is, in contrast to more traditional theories of motivation, and self-determination theory is like 50 years old now, so I guess it’s also traditional, but theories before that were really about mechanisms of control. I’ve just got a new dog a few weeks ago, so like how you train the dog is like, dog does good thing, I give treats, dog does bad thing, I don’t give treats, [LAUGHTER] and I think a lot of early motivational theories were rooted in that, of like, “oh, student does good thing, I give grade, student does bad thing, they lose points towards grade,” and that still happens all over our campuses. So, I said, my background is in music, so, like, one of the things we do in music departments is we want students to go to our concert, so what do we do? We give them extra credit for going to a concert, and what we don’t do in doing that is explain to them the value of going to that concert and give them opportunities to engage with that concert in ways that they might not otherwise, and that connect that concert to the experience of what they’re learning in our class, and I think that’s what we really would advocate for in self-determination theory, is to really think critically about grades and what grades mean, because ultimately grades, at their best, are a form of information, they’re a communication form between instructors and students, that is unlike basically any other form of communication, and that can be used effectively, [LAUGHTER] or it can be used for harm. And I think too often we view grades as a mechanism to control. So students after COVID, my class attendance has dropped, so I’m going to give more points for students coming to class, or more penalties for not coming to class, and instead we can think, well, what is it that’s happening in class that we can make more valuable for students, and then that doesn’t mean you can’t have grades tied to it, but like, how do we then make the grade tied to the learning that they’re doing in class that’s tied to the learning outcomes and objectives that will help them in their long-term career personal goals, whatever those may be, and trying to really build that connection, so students don’t see their grade as well. This instructor liked me or disliked me, and therefore they gave me this grade at the end, but really that this grade reflects what I’ve been able to achieve, and my growth, and then I can go talk about that, and like what it meant to earn an A or a B in this class, and what I gained from it, and that grade is one piece of information that is sort of a tightly packaged representation of that.

Rebecca: This conversation’s making me think a little bit about self-determination theory related to instructors, too. We have needs, especially around grading, and some of these other things, too. I’m wondering if you have any thoughts about that side of the coin in thinking about some of the basic needs of instructors and how some of the techniques that you’re recommending actually [LAUGHTER] serve in both directions.

Chantal: Yeah, absolutely. I’ll just kind of get started, and then Dan can jump in, but when we go through our Impact program, which Dan described, our 13-session semester long program, we really emphasize that to faculty, and that’s the beauty of self-determination theory. As you mentioned, it’s a theory of human motivation, so it’s about humans, so it’s really the humans that are doing the teaching and learning endeavor, and that’s really the most important thing, and what we really need to focus on. So, we tell faculty, you also need to think about the satisfaction of your basic psychological needs, and that will revitalize your teaching, how you approach your teaching, how you think of your teaching. So, again, if you encounter a technique that you think is the next sexy, flashy thing, but you don’t think it’s going to work for your class, or it’s not going to be something that you can do that you can implement effectively, you’re not going to be in a position to satisfy your basic psychological needs, so that will translate into something that will hinder the needs of your students, because you’ll feel forced, controlled, and not self-determined in applying those techniques. So we really engage in conversations with faculty around that, and to find, through asking questions of the faculty, to really find the thing that’s going to create the best possible autonomy-supportive environment for them and their students.

Daniel: Yeah, and I’ll just add this is what, to me, is really fun about being on the I mpact team and working with faculty every semester in this way because I might be a person who thinks ungrading is the greatest thing for my classes, but I’m not there to convince them of that, I’m there to try to understand what they’re doing and what they’re achieving and what will be meaningful to them, and to me that’s a really fun thing because it makes every semester unique and different, and our hope is in having all these different case studies, readers will be able to see how people have gone through this same program, but come out with very different classes, but all classes that are at their core achieving the same basic goals of creating a learning environment that’s going to support students to internalize their motivation and to succeed and to value the things that they’re learning.

John: We talked a little bit about the structure of the book, but we should note that you have chapters on pretty much all aspects of teaching that should apply to pretty much any faculty member out there. You have chapters associated with creating tests, writing assignments, group projects, lectures, labs, and experiential education, and in each of those cases, you provide some really nice suggestions on creating an autonomy-supportive environment. Could you each highlight one or two ways in which the approaches that you advocate in either one of these chapters, or generally across the chapters, might differ from what we see in many classrooms today?

Chantal: I love how you set up that question, and I want to kind of not push back on it a little bit, but “different” is not how I would really think about it. It’s not like many of the techniques that we have in there, or suggestions are necessarily different. There’s a lot of the things that are the same things, but sometimes it’s just to think about them in a context that is really based into self-determination theory. So I would say not necessarily different in kind, but it’s really about using the self-determination theory lens and the lens of the satisfaction of the basic needs to make meaningful choices and decisions about your class and your content, and maybe how you’re delivering it, so that’s I think is important to understand that you’re not going to read this book and say, “Oh, I’ve never heard about this before, this is amazing.” That’s not really at all what it is, and again, the suggestions are suggestions, we don’t intend them to be recipes. But if I give a couple of examples that are in the case studies that I really find appealing is that Nathan Mentzer, for example, is a faculty who we talk about in our technology chapter, and really he used HyFlex during the pandemic as a way to really survive as we were re-engaging and Purdue came back to in-person instruction very quickly, and he needed to really continue to use his HyFlex model. He actually wrote a book about it, so that’s also another very interesting piece of it. But he says he will never go back. The HyFlex and what he learned about it, and actually he did research on it using self-determination theory as a framework, allowed him to provide choices and options to students and build connection and relationships with the students and amongst the students that he was not able to do before, so he will never go back. He will always do HyFlex and he talks about how he really talked to his students about how important it was to attend in person if they could, but he was not forcing it. He was not taking attendance in person. He was not making the students feel guilty if they could not attend in person. And then the virtual option was something that everybody could connect to, everybody, even in person. Every student could log in with their teammates and be in this in-person and virtual environment all at the same time. It allowed students to have flexibility to have choices of whether they could show up in person that day. If they were sick, they didn’t have to explain why they were joining in virtually, and they didn’t miss anything. They had all of the content, they could continue to be engaged with their group. He really also focused on the autonomy piece by providing a rationale. He would say, I think it’s important and valuable if you can come in in person, but if you can’t, that’s okay. And here is the content. He also talked a lot about whether you should have your cameras on or off, but again, he wouldn’t force the students to behave or do certain things in a certain way, but he would really spend time explaining the rationale. This one is a great example. I’ll let Dan jump in with a few, and we can go back.

Daniel: So, I’ll share one. It’s actually not the story in the book about this person, but by Lindsay Hamm, who is in the book, I think also in the technology chapter about what’s called Charlie, which is her AI writing feedback tool, and she teaches 500 students sociology class, so it’s like functionally impossible without something like this to give feedback to students, which is very important to her. But what I want to share is actually she is someone who is very interested in the development of AI from the very beginning, and she’s become a campus leader on this, not just in her use of this AI-based tool for feedback, but she took her writing assignment, and in the middle of one semester she was confident that AI at the time was not like providing results that had to make her concerned about students using it, and then in the middle of one semester, an update came out, and she was like, “Oh no, [LAUGHTER] it’s time, it crossed the threshold,” and the question then was, “Well, what do I do now?” And this is where I think self-determination theory is so useful, because it really gives you a framework to think, “Well, now I’m in a situation I didn’t expect, and what do I do?” So, what she did was she first met with some students, because this is 500 students in this class, so she didn’t want to just go in front of all of them first. So, she met with some students, because she trusted students who had taken the class already, and she talked to them and said, “This is what’s going on, and I want you to talk through some options with me.” So then she came to the class and had three different options for like what we could do going forward, which some were just like rebalancing other assignments, some were creating something new, but all with the idea that okay, there are different paths forward, I’ve vetted these and now we’re going to give students in the class, different options, and I think students could actually pick which they wanted, so it wasn’t like, okay, 51% voted for this thing, and it stinks for the rest of you. So she was able to make it so that every student could have a meaningful choice, and she could progress the class in a way that she still felt good about meeting the learning outcomes and objectives for the class, and then over the summer, she could do [LAUGHTER] a much bigger, like, “okay, what does this mean now, and how can I adjust and adapt in a more meaningful way, that’s not just like throwing a brand new syllabus at this 500 student class mid semester.” So to me that’s like a really fascinating example of how you can take these principles and respond to the unexpected, which it feels like now every semester we’re always responding to the unexpected in a way that at least I didn’t feel 15 years ago [LAUGHTER] when I started teaching in higher education. So to me that’s one of the real appeals of this kind of framework for thinking about course design.

Rebecca: Stories for responding are always very valuable. [LAUGHTER] One of the things that you’re just hinting at in this instructor’s approach is kind of checking in with multiple students to develop some choices, and kind of at the start of our conversation, you mentioned the importance of bringing students into the conversation of course design, but of course, students have competing interests and values in what motivates them? Can you talk a little bit about kind of balancing the very different needs or perceptions of needs of different students in a class?

Daniel: I’ll start. So, part of what has enabled this is we have a Students as Partners program at Purdue called the Student Pedagogy Advocate, so that has provided a structure with students who are not like heavily trained but are introduced to self-determination theory and are really interested in working with faculty to make meaning of their classes and really centered around achieving the instructor goal, so we have some students who are prepared to do this, which certainly helps when we bring them in, but I think that’s to me part of the exciting part of teaching is when you are teaching the same class sometimes for decades, for some of us, what makes it exciting is that the students are different. And to me that’s at the core of what it means to teach, is to really think about the connection with the students that you can make and the students who are in the room, and how you can be responsive to them, and I think again thinking through self-determination theory gives us a way to integrate students into that thought, because it gives us a way to understand students and a set of goals for what does it mean for a student to feel a sense of competence, and whether that’s that they aced the test, but maybe guessed on half of it, but we might not know that, or they might have struggled on the test, but they felt good going into that, so then what does that mean for our discussion, or they struggled on the test, but they say, “well, I know I didn’t have time to study, so it’ll be fine next time,” but like what does that mean to help us, for us to help them think about what strategies they’ll take. So yeah, I think to me it really helps us think about how we engage with students, both to build a sense of relationship, but also to really understand what will be meaningful to them.

Chantal: Yeah, and I would add to that, that in some of those classes it’s not like instructors all have to do this, but some have used this very effectively, is to really work with students as co-creator of this class and this content, and even how it’s delivered, and sometimes it’s not possible; in certain situations or certain courses it’s not possible. But to the extent that you can think about students as partners, as Dan mentioned, not just the program that we have at Purdue, but also in terms of its philosophy of how can you really engage with students as guides and partners, as opposed to in a hierarchical sort of relationship, that’s really helpful when we’re thinking about our students as coming to this learning endeavor as co-creator of their pathway, and that’s really the essence of what we mean by autonomy and supporting autonomy. It’s not doing whatever we want and letting the students do whatever they want, but it’s providing a structure and then guiding within that structure. One word, one sentence that I really like to use to think about this is choices within a structure, so structure is also very important within that, so it’s not like there’s no boundaries, there’s no structure, there’s the course objectives, but the alignment, as Dan talked about, and we talk about in the book, the alignment between your learning outcomes, your objectives for the course, and how you assess the kinds of activities that you engage your students in, it’s very important to think about.

Daniel: And one other thing that I’ll add, we’ve been collecting student data on this program through a self-determination theory lens for over a decade, and one of the great things about self-determination theory being a theory that’s been put into practice around the world for 50 plus years is there’s a lot of great existing survey tools, and we can probably, in the show notes, have a link to these, but you go to the Self-Determination Theory Center website, and you click on metrics, you can download all sorts of them. One we really like is called the Learning Climate Questionnaire, and there’s a short version that’s, I think, eight questions. Chantal can correct me, but it’s eight questions. [LAUGHTER]

Chantal: That one is actually published in the first book, so the student-centered impact book that I talked about earlier, and we have some of those scales published in there.

Daniel: To me having eight questions that I can give to my students and get a clear sense of how are they perceiving the learning climate, and we tried to get these into the end-of-semester surveys with some luck, [LAUGHTER] but not as much luck as we’d want, and also you can ask these earlier, you can ask these in the middle of the semester and get really valuable information to understand how students’ basic needs are being met or not being met, and then you can take action from there to address those gaps. So I think that’s one of the other benefits we don’t talk about enough is that having these set surveys, because making class surveys is really, at least for those of us who are not psychologists by training, really hard, and also then we make them, and we’re like, well, did students interpret this question how I thought, [LAUGHTER] and the general answer is probably not, so having these set surveys that have been tested in hundreds, if not 1000s, of classes can be really valuable to collect some quick feedback and understand, like, okay, is this achieving what I wanted when I designed the class in this way.

John: We’ll include a link to the survey site in the show notes. Now, we wouldn’t be having a podcast in the last three years without some mention of [LAUGHTER] artificial intelligence, we keep trying, but… [LAUGHTER]

Rebecca: I keep trying to run away but it’s not helping. [LAUGHTER]

John: However, that is certainly an issue that does relate to educational technology, and you do address this in your book a bit. Can you talk a little bit about ways more generally in which educational technology can support student learning and autonomy, and also perhaps some ways in which we can redirect efforts, so that we don’t see students engaging in excessive cognitive offloading that can be harmful to student learning and skill development.

Chantal: I’ll start with a general comment, and then I’ll let Dan answer the tough part of that question [LAUGHTER] …so I’ll run and avoid it. To me, the general message that I really want the listeners to think about is that technology, including AI and others that will come in the future, are simply tools, and I don’t mean this in a sort of dismissive way, but tools are there to help us create environments that will foster learning, just the same way that our computer and typewriters and smartphones were tools, and some when those came out, were really afraid of those particular tools, but the challenge is to really think about how to integrate them effectively into your teaching, into the learning of the students, and what does it mean to support the basic needs with those tools? So, again, they’re tools. Your voice is a tool, like we’re doing a podcast now. So, your voice is a tool. How do you use those tools to support the learning is important. So, I’ll just put AI in that same category. We just need to think a lot about it. So, now Dan is going to answer the hard part of the question. [LAUGHTER]

Daniel: So I guess in thinking about, like, how do we prevent students from using this as not just a tool, basically to avoid the class or the learning process, because ultimately meaningful learning is hard. That if we haven’t challenged ourselves and haven’t struggled, we probably haven’t learned that much, and our society, our human nature is we try to make things easier. So it’s human nature to look for ways to skip hard things and make them easier, but what that means is we really need to think about motivation, and we need to think about how we internalize motivation, and this goes back to when Chantal described the different types of extrinsic motivation, is we need to really think about how can we help students identify the value of what they’re learning and the skills they’re developing, or ideally even integrated, and feel like it’s a sense of who they are. And often this means really focusing on the basic skills that get before the stuff in their major, especially like the stuff that they perceive as fun. So when I taught the intro music theory class, not many students were very excited to do like the basic lessons and learning how to read music in multiple clefs that they didn’t necessarily deem apply to them, but you need to have those skills really solid before you can do much else, so how do we help students either internalize the importance of developing those skills at a really high level, or to integrate it and feel like, well, I’m a musician, a musician, this is who we are, and we need to have these skills. And this is where the idea that when we create environments that meet those basic needs, students are more likely to have that sense of internalization, and we’ve talked about lots of ways to do that. But really I think that at the core for us is to think about that relationship element, because ultimately students need to trust us when we say this skill is important, and we also need to believe it, and I think this is something we sometimes forget, because most of us now use AI for some of these things, and there’s a difference. I would say using AI for something that I can do and it will save me time, but then I also know if the output is good or not, because I’ve done this enough times, and doing something where, like, I really don’t know if this is good, it will save me time, but it’s not something I could do otherwise and I think those are two very different places, and a lot of students are in the “it’s not something they could do otherwise,” and we really need to get them to the point of recognizing the value of those basic skills and not skipping them, because when they skip them now, they’ll run into challenges later, and we can say that all the time, but unless we really develop a sense of trust and meaning, and have students really buy into the core elements of the class, they’re not going to believe us, when we say it. So that’s where we need to create this environment that’s internally motivating, and an environment where students trust that we have their best interests at heart, and I think most of us do, but I think also there’s, at least when I started teaching, I felt like I wasn’t supposed to say that, I wasn’t supposed to tell students I cared, and I think we’ve turned a corner on that. I think now it’s more acceptable for students to believe we care and us to articulate that we care, and I think now it’s more a matter of like really creating an environment where students internalize that we care, we want them to succeed, and what we’re sharing with them are strategies that we know will help them succeed in the long run.

Chantal: I’d just like to add something, because Dan, you made me think about this, is that with regard to the offloading, yes, humans will tend to find things that they can do that tend to be easier, but humans are not lazy by design. If you really think about self-determination theory, and you really go to the origin of the theory, it’s really about humans are really wanting to grow and develop and become self-determined… self-actualized is another word that’s been used in psychology. It’s like one place in the right environment where the needs are supported, humans will do a lot, and if you think about humans that are alive and vital, they’re doing things, they’re just finding ways to write, to read, to consume, like students read and consume a lot of information, maybe not the textbook, but that’s a question that we can think about, like why not, why is it, but humans are not inactive or lazy, like we really engage in things that we care about, that we value, and that we want to be intentional about. I think that’s kind of the way to think about this in a broader perspective.

Rebecca: So, Chantal, you mentioned the last question is a little challenging, but some folks find our last question the most challenging, which is [LAUGHTER] we always end by asking what’s next.

Chantal: I’ll answer with something that’s kind of on my mind right now, and in things that we’re really thinking about at Purdue is, and you mentioned technology, so it has to do with AI, but really thinking about how critical thinking gets developed and is fostered, including how we think of AI using critical thinking, and this is a program that actually Dan is part of. We’ve been sort of pioneering this, piloting this for the past couple semesters, but in our conversations with faculty, we’re really thinking about how do we develop this sense of problem solving and critical thinking, because when you talk to faculty, right, it’s often something that they’ll say, “Oh yeah, I want my students to become better critical thinkers, but it’s just going to happen, it’s not something that I’m going to build into my class,” and we disagree with that, we say no, we really need to build in how you define critical thinking, how you think about it, how it evolves throughout the curriculum, and how is it that students progress from an emerging skill in critical thinking to really proficiency, and what does that mean, and what does that even look like. So that to me is what I’m rethinking about right now, and what is sort of our next steps in terms of applications.

Daniel: And I’ll just add, because I’m also involved, I’ve really loved this program, that one thing that’s been interesting is we’ve approached this through the same Impact framework, so it’s still based on self-determination theory, but the idea that you need to create a self-determined environment to get students to engage critically with material, but we’ve changed it, as we mentioned earlier. Traditionally, the program we’ve explicitly brought together faculty from different departments, and that was one of the core aspects, was this is a time to go outside your discipline and to realize that teaching has significant differences, but also commonalities, and for this program we started bringing together groups from the same department, and this has been like really interesting in different ways, and part of that is the recognition that a lot of faculty don’t actually talk about their teaching with their colleagues in their department, so this has been one of those opportunities for them to get together and think about what does critical thinking look like in our context, and how can we actually connect our classes in meaningful ways that help a student, whether it’s the 100 level class, start to develop certain skills that will be applicable in that context, and then how can those be reinforced, and how can we use similar language and structures, or one department we’ve been working with that I’m really excited they built a whole curricular long rubric with like here are key critical thinking skills and here are where we think students will develop and actually the department is Speech Language and Hearing Sciences, so the vast majority of students go on to get a master’s degree because you need that to be a practicing professional. So they actually said, well, at the top level we don’t even expect you to be here until you’re in graduate school, and the idea is to share this with students from the beginning, and obviously figuring out how you share it with students at your 100- level class and say we expect you to end this class at level one, maybe two, right, [LAUGHTER] but then show that that progression, and then when they’re applying to graduate school, give them the structure and language to talk about how they’ve developed and what they still plan to learn, because ultimately you go to grad school to continue your learning and development to become a professional in this field, so that to me is also really exciting, and like a twist on this program, but our early success shows how this basis in self-determination theory can really apply in lots of different contexts.

Rebecca: Sounds like a really great initiative. Well, thanks so much for joining us.

Chantal: Thank you for having us. This was a lot of fun.

John: Thank you. It’s a really good book, and it’s one I’d recommend to anyone who is beginning to develop or redesign an existing course.

Daniel: Yeah, thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

453. Prequestions and AI

A growing body of research indicates that asking questions about concepts students have not yet studied increases learning. In this episode, Steven Pan and Judith Schweppe join us to discuss how generative AI can facilitate this technique by generating prequestions.

Steven is an Assistant Professor of Psychology and the Director of the Learning Sciences Lab at the National University of Singapore whose research focuses on evidence-based teaching approaches. Judith is a Professor of Educational Science with a focus on teaching with digital media at the University of Erfurt. Her research focuses on working memory, long-term memory, and learning.

Show Notes

Transcript

John: A growing body of research indicates that asking questions about concepts students have not yet studied increases learning. In this episode, we discuss how generative AI can facilitate this technique by generating prequestions.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Steven Pan and Judith Schweppe. Steven is an Assistant Professor of Psychology and the Director of the Learning Sciences Lab at the National University of Singapore whose research focuses on evidence-based teaching approaches. Judith is a Professor of Educational Science with a focus on teaching with digital media at the University of Erfurt. Her research focuses on working memory, long-term memory, and learning. Welcome, Judith, and welcome back, Steven.

Steven: Thank you.

Judith: Thank you, thank you for having us,

John: And thank you for agreeing to come on at times that are a little bit challenging. Could you mention for our listeners what time it is where you are?

Steven: I’ll go first. It’s a bit after 4:30 in the morning [LAUGHTER] here in Singapore.

Rebecca: Starting your day right.

Judith: And for me it’s 10:30 pm, so I’m about to go to bed after this recording.

Rebecca: Best way to end the day. [LAUGHTER]

John: Today’s teas are:… Judith, are you drinking any tea?

Judith: Yeah, in fact, I’ve prepared a ginger and lemon tea.

Rebecca: A good one for the evening.

Judith: Yeah, exactly, that was the idea.

Steven: Well, I sometimes drink tea. Actually, I should drink more right now. I just have water.

John: And Rebecca,

Rebecca: I have Golden Monkey today.

John: That’s been a while.

Rebecca: Yeah, it’s been a while.

John: And I have an Irish breakfast tea today.

Rebecca: Awesome. So we invited you both here today to discuss an article in the Journal of Applied Research in Memory and Cognition that you both co-authored with three additional co-authors, entitled “Using ChatGPT Generated Pre-Questions to Improve Memory and Text Comprehension.” But before we discuss your study, it might be helpful to talk a little bit about what is meant by pre-questioning, and what past research has suggested about the impact.

Steven: Well, pre-questioning is a learning technique that involves attempting to guess the answers to questions about information you haven’t formally studied or know much about, and then there is, after you’ve attempted to answer those questions, the opportunity to learn the correct answers from some form of instruction, such as a lecture, watching a video, or, for example, reading a text passage. So you guess first, then you discover the answers afterwards. That’s pre-questioning. It’s similar to, it could you even be considered a part of a class of strategies that sort of involve what you might call pre-instruction learning activities: do something in advance of formal instruction, and pre-questioning, along with those other strategies or techniques, can improve learning in a variety of circumstances, improving, as I mentioned, learning from videos, learning from lectures, learning from text. There are a growing number of studies showing that it has these kinds of benefits.

Judith: What’s remarkable about this is that learners typically don’t know the answers to these questions, because they only learn afterwards, so they guess wrongly or can’t come up with an answer, they don’t get any feedback, but still they benefit on the criteria test questions after learning.

John: What are some of the possible explanations for why pre-questioning is so effective in enhancing student learning?

Steven: So this is an active area of research, and there are many possible explanations. Just to name some of them, for example, one explanation is that pre-questioning sort of activates attentional processes that either make you aware of your gaps in knowledge and want to focus on finding the answer, or when the answer is presented, you notice it better, or simply when the instructional opportunity happens afterwards, you are more focused, and you are more primed to learn from it. So that’s a sort of an attentional class of explanations. Then there is, for instance, an error signal type of explanation, where you make a guess, and usually it is incorrect because of the lack of prior knowledge or instruction, and your awareness of that error, particularly when the correct answer is presented, there’s a discrepancy. You guessed this, and then the answer is something else. And there are potential mechanisms in the brain that are responsible for learning from those kinds of situations, and the pre-questioning triggers those mechanisms. So that’s a second possibility. There are other ones that involve, for example, your memory of making the error. Okay, I guessed this, and now this is the correct answer, and you sort of integrate this, and that becomes a more powerful learning event, because it’s not just you learn something, it’s you made a guess first, it was usually wrong, and then you have the correct answer, and that becomes a richer, more integrated representation that helps you better remember that information. So, those are three candidate explanations for how pre-questioning can improve learning. There are more beyond that, but those are some of the ones that come to mind.

John: One of the other ones that was mentioned in your paper was that it also activates prior knowledge to help prepare you to integrate new knowledge into that framework.

Steven: Yes, that’s also a potentially useful way to consider how pre-questioning may enhance learning, and this idea that it may activate that prior knowledge, so you are then better able to integrate the new information, you may be primed to learn that information, and so on.

Judith: And that explanation might even be more important in real life in class, because even when you ask students questions for a new topic, they will have some prior knowledge that they can activate, whereas in the lab we usually use texts that they know hardly anything of, so that the importance of prior knowledge is even stronger, maybe in class.

John: And however it works, it does seem to work. [LAUGHTER] So it’s an effective technique, whatever the specific mechanism.

Rebecca: And despite the evidence on the value of pre-questioning, you know that the practice is rarely used in the classroom. So why might this teaching strategy be so rarely used?

Judith: I think there can be several reasons. For one, the research field is still very young, so there was not that much time for the evidence to distribute to the public, as compared to retrieval practice, for instance, which has been far better known in the past years. And another reason may be that it may also feel weird for a teacher, for an instructor, to ask their students questions that they know they won’t be able to answer, and that might refrain some people from using pre-questioning.

Steven: Yeah, those are, I think, very good reasons for why pre-questioning may not be used that widely. I’ll add another one that’s relevant to the paper that we’re going to talk about, which is this idea that you need the materials to engage in pre-questioning, and particularly for students who know nothing or very little about the topic, which is sort of definitional requirement of pre-questioning, it would be very difficult for them to come up with questions on that topic to practice with, and also even getting them from instructors, instructors commonly, for example, in surveys, they’ll say that they’ll provide practice questions on occasion, but typically after instruction has occurred, rather than beforehand. So all of this points as sort of a materials or implementation challenge. Not only do learners and instructors not necessarily know much about pre-questioning, and it also does feel weird to use it, but even if they wanted to use it, they might not have the materials, either the time to come up with the questions or access to questions that are appropriate for pre-questioning. So there are a host of obstacles that make it difficult, at least in many circumstances, to use this particular technique.

Judith: That was the main starting point for our study, because it is not only recommended as a teaching approach, but also as a learning strategy. But for learners to use it on their own, it’s a lot harder than retrieval practice, because, as Steven said, you cannot come up with new questions before you know about the topic, you’re totally reliant on somebody else providing these questions.

Steven: Yeah, exactly. And with retrieval practice, sometimes it can be as simple as: “take out a sheet of paper and write down as much as you can remember about whatever you’re trying to learn.” You just retrieve it. You don’t even necessarily need to have a database of questions to practice, but with pre-questioning you really do need those questions, and if you don’t have those easily available, then it’s very difficult to engage in the strategy or technique.

John: Which brings us to your paper. What are the primary questions you’re investigating in the study?

Steven: Well, there are sort of a sequence, we sort of went from very simple and basic, and we got more and more specific. The first question was simply, if you use generative AI to come up with these questions, and you use them for pre-questioning, do they actually work? That was sort of the most simple question, and that was the first experiment focus. We then advanced to, okay, how do those questions generated by AI compare to questions that a relatively expert human being creates? Are they similar or different in efficacy if you use both human-generated and AI-generated questions for pre-questioning? And that was the second experiment. And then, for the remaining two experiments, we went into more fine-grained issues, specifically comparing AI-generated questions made using what we called a sophisticated or elaborate prompt that was very carefully engineered to generate the questions we wanted to see, versus a simplified or more basic prompt that any person could use without much training, so different prompts to generate these questions, and we compared that to a condition that didn’t use pre-questioning at all, but did complete an alternative activity. In this case, it was studying an AI-generated outline, so that was sort of a more rigorous test of the effects of pre-questioning to see whether using pre-questioning compares favorably to spending the same amount of time studying something else.

Judith: It’s an additional question concerned whether answering pre-questions generated by either an AI or human would generate to a different source, because if you think of a regular learning situation, you might generate as a learner your question with an AI. Well, meanwhile, the instructor might also generate the exam questions with an AI, but they might also be generated by the teachers. So, it might be worth knowing whether there is any decline in the transfer from one source to the other, and the same with the more elaborate prompt, compared to a basic prompt. If maybe a student would use a more basic prompt than a teacher.

Steven: Right, and that issue speaks to this idea of, shall we say, transfer or generalizability. So you ask someone in pre-questioning a particular question, and then you assess learning using that same exact question, that’s one way to measure learning, but as Judith just said, it could be different questions that the instructor generates, either by themselves or using an AI, or using different prompts, or whatever, and so a critical question really is, “What happens when the assessment, the questions used on that assessment differ in some way from the questions used earlier for pre-questions?”

Rebecca: So we’ve talked a little bit about the research questions. Can you talk about the design of the study itself?

Steven: So each experiment, these were online studies with online participants, adult learners, and in each experiment the learners would be assigned to either a pre-questioning condition of some type, or in some experiments a control condition where they didn’t have pre-questioning, or they studied an outline, or some other activity happened. So that happens at the beginning, they’re randomly assigned to these different conditions, and then they would engage in some learning activity, for example, pre-questioning, attempt at least two pre-questions one at a time. All participants would then, and to be clear, all of these experiments involve learning from an encyclopedic text, in this case, I think it was about brakes, the function of brakes, different types of brakes, so it’s sort of a great proxy for certain kinds of information that you would find, for example, in an engineering course, something like that. And so they engage in the initial learning activity, such as pre-questioning. Then everyone reads this passage on brakes, which contains the answers to the pre-questions, and then after that, after a short delay, then they take an assessment of learning, which is a series of questions about brakes, which include questions the same as used for pre-questioning, but also other questions that were not necessarily exactly the same. That’s the broad design. Of course, across experiments, there are variations, so to get more specific, experiment one, the one condition was pre-questioning, where they attempted AI-generated pre-questions before reading the passage and taking a test. The other condition had no pre-questioning, so they just read the passage and took the test. In experiment two, it was AI-generated pre-questioning versus human generated pre-questioning. So one group got the AI-generated questions, one group got the human generated questions. They all read the passage, then they took the test. And then experiments three and four were either what we call elaborate generated pre-questioning, the prompt that was used, or basic generated pre-questioning, again referring to the prompt that was used, or studying an AI-generated outline. So, depending on the group, they would have different activities before reading the passage, and then taking the test.

Judith: And in any of the experiments in which we contrasted different questions, both types of questions were also used on the criteria test, so that we could see differences between the tested questions and the untested questions, and in some cases even between new questions generated by the same source, either both AI-generated or both human-generated, and transferred to new questions generated by the other source.

John: A very well designed experiment, and one of the things that I appreciated about this was that it was an experiment that was pre-registered. That’s not something that happens in all cases, in all disciplines. Could you tell us a little bit about this process? It’s something I think that came up because of people attempting to replicate some past studies and not always finding that results were replicable. So, could you just give our listeners a little bit of background on pre-registration of studies?

Judith: We try to do it in a brief manner. You’re right, this originated in what we call the replication crisis in psychology and social sciences overall, in which several well-published studies did not replicate, and so there’s a counter movement right now, the open science movement, and the idea is to restore trust in science and psychological research. We want to be as transparent, as open, about what we are doing as possible, and there are several reasons for why some research findings are not that reliable. There’s a tendency for significant findings to be published more often than non-significant findings, so that we are rewarded for getting significant results, and even more for getting our predictions right. There’s a study that indicates that more than 90% of hypotheses in psychological studies turn out to be right, and that’s not because we’re so clever, that might be due to some questionable research practices. So pre-registration is one way to make sure that we write down in advance what we predict, how we conduct our study, how we analyze our data, how we decide whether a certain data point is an outlier, how we decide whether to include any covariate. So to write that down in advance, so that we’re not subject to our own biases, so that may be the one analysis that brings about the best result that we’re convinced that that was the one we wanted to do right from the start. No, we can look that up because we’ve written that down, and to make it transparent to everyone, you pre-register publicly, so you are the one who decides whether others can see it, but when you submit a paper, you can include this pre-registration, and the reviewers can see that the date of the pre-registration is really before you collected your data.

John: You mentioned that so many studies, and this is true, it’s not just in psychology, it’s also true in pretty much all social science disciplines. And two of the explanations for that are publication bias and something called p-hacking. Could you just explain a little bit of those two things for our listeners?

Judith: Yeah, the publication bias, so that indicates, as I just said briefly, that of the data that are published, most of them are significant, and that’s not realistic, because a lot of studies we conduct, they yield non-significant results, and the idea is that this is due to several biases in the publication processes. So reviewers and peer-reviewed journals are more enthusiastic for significant findings, researchers don’t have that much time to write up their findings, so they tend to focus on those that are more publishable and leave the others in their drawers. So this is also called the file drawer problem, that there are a lot of non-significant findings in the file drawers of lots of researchers in all the ivory towers, and this leads to us overestimating the size of published effects, because we don’t know about this heap of unpublished studies, and that’s a problem for evaluating how important certain effects are for us. And p-hacking is one way where one type of questionable research practice, because p is the value that indicates statistical significance, and the field has agreed that p that is lower than .05 is the threshold for importance, and this statistical significance is often mistaken for importance. And there are several ways to analyze your data that can influence whether your findings yield a p value below this threshold, and all these ways to treat your data are called p-hacking. So, you can decide to include gender as a variable that was never part of your hypothesis, but you can always see whether it makes a difference what gender the participants are, and that might bind some variance, so that your main effect is significant. You can first analyze your data, and the p-value is just above .05 and you think, well, just a few more participants, and they might be below the threshold, or you might notice that there’s this weird participant, which must be an outlier, and that often happens after your first analysis, and helps you get below this very important threshold. This often is not outright fraud, it’s just us being human beings and being subject to our biases, and that’s why the open science movement thinks research science needs the self-control mechanisms to be open with our data, with our materials, and with what we predict. And we are both committed to that, so it was important for us to pre-register the study and also to publish the data and the material.

John: Another way of expressing the issue of p-hacking is that if you torture the data enough, it eventually will confess. [LAUGHTER]

Judith: That’s nice.

Rebecca: So gruesome. [LAUGHTER]

Judith: Something else that’s related to this is also HARKing, that’s hypothesizing after the results are known, so that you don’t actually make predictions, but rather post-dictions, and then there’s this analogy to the Texas sharpshooter who draws the target after his shots, so that that is easy to hit bull’s eye, and that’s also often used to compare this research.

John: But your results are not subject to any of those issues, because you’ve done all these things to prevent that with pre-registering and sharing the data.

Rebecca: Seems like a good transition to results, huh? [LAUGHTER] Can you talk a little bit about the impact of providing the AI-generated pre-questions? I imagine that folks are really interested in this particular piece because it might save time.

Steven: So we were very excited to see that the AI-generated pre-questions were effective at enhancing learning, or producing meaningful learning, shall we say, across all the experiments. So, for example, in experiment one, just presenting two, I think, AI-generated pre-questions before reading a text passage, that was enough to enhance learning from that passage, and that enhancement extended not only to later questions that were exactly the same as the questions presented in the beginning, but also new, never before seen, questions, so there was a generalizable benefit to some extent. In the second experiment, we found that the AI-generated pre-questions yielded similar levels of learning as questions generated by competent human beings, so that suggests that you don’t necessarily need a human being to invest the time and effort to come up with the questions. An AI can potentially, an AI chatbot can come up with similar questions of comparable efficacy, and then in the third and fourth experiments, again, they were similar designs, those two experiments, both AI-generated pre-questions that were generated using a more sophisticated, detailed prompt, as well as those generated using a very simple prompt, they both enhanced learning relative to studying an AI-generated outline, so that shows the power of pre-questioning extends to not only using AI to generate the pre-questions, but even compared to using AI to generate some alternative materials that are used for some alternative activity. So those are the broad outline of the key results. Now there’s, of course, that other finding about, or other issue about what happens when the questions on the test are not the same as those that you were practiced on, or they were generated by a different source, and whether, for example, you practice with AI-generated questions, but then some of the questions on the test are generated by a human being or generated by a different AI prompt, or things like that, and I think one general way to describe those results, because they varied across experiments, was that we could still see across experiments some evidence that learning still benefits in those cases, so that it’s not essential that the questions that are used on assessment have to be exactly the same as the ones that were seen earlier. There is some degree of generalizability, although there was sort of a trend, I would say that as the questions get more and more different in source for the content targeted, the learning benefits may not be necessarily as strong, at least numerically, but there’s still evidence that the learning is not restricted to the exact same questions that you saw earlier.

Judith: And we were very happy about that, because that’s not always the case in pre-questioning studies, because often these effects are restricted to post-test questions that target the very same information, so that the transfer to these untested questions really excited us about these findings, and also the case that this very, very basic prompt, which just said, “Can you generate two questions for this passage?” worked, that really sounded feasible.

Steven: One thing I would add to that, we were really very excited by that finding, because literally that means that anyone practically could just use a prompt that is one line and come up with questions that are useful for pre-questioning or potentially other purposes. One thing I did want to note, though, is that when we were conducting the sequence of experiments that we were using ChatGPT, the version of ChatGPT was getting more and more advanced as the experiments progressed. So, by the time we got to the latter experiments and we were comparing different types of prompts, we were using a more advanced GPT version, and in that case, the questions made using, again a very elaborate prompt versus a simple prompt, they seem to be equally effective for the most part, which is again I think in part a reflection of how advanced the technology has become, and also how just generally how easy it is now to use.

John: Given that you’ve got these results, and it doesn’t require a very elaborate, detailed prompt, or even much knowledge of the subject, might you suggest that this would be a useful way of students learning things more deeply as part of their studying? Because students often use it to generate an outline, which was one of your control conditions there, and that was not as effective as having these pre-questions. Might this study suggest that this would be an effective study technique for students to use on their own when they’re learning new things?

Judith: I guess we would say yes. That was the idea, yeah. That was the hope to find evidence that it can work if they generate the questions on their own. We might also add a cautionary note to that, because we found transfer to untested questions, but that may not necessarily be the case for other questions. We noticed that there was some overlap between the questions generated, because all of them were constructed in a way that they were integrative, so they targeted more than one piece of information, and that resulted in somewhat overlapping questions, so the suggestion is a bit different compared to other studies, in which the new questions are totally different, and it might still be more helpful to get the pre-questions from your instructor, who will also post the questions for the exam, because they also tell you what this person thinks is most important, and that may be a valuable addition, but if your instructor doesn’t want to give you any pre-questions, because they don’t believe it’s effective, or because they don’t want to waste their good exam questions on students preparing for learning, that might still be good alternative to ask a generative AI to be a question generator.

Rebecca: I’m imagining that faculty hearing this are excited about the possibility of integrating pre-questions into their teaching methodologies. Do you have some advice about how they might do this, how they generate effective questions, or even like how many questions would be useful?

Steven: So, there are some tentative recommendations that we could offer, but again, these are tentative because we have barely scratched the surface of implementing pre-questions. We are still sort of exploring the efficacy and mechanism stage. Nonetheless, in terms of possible ways that the pre-questions could be generated in an effective manner, well, firstly, one thing I would note was you mentioned number of questions, so it seems at least in the studies that have been conducted thus far, the number of questions is not that many. For example, I don’t think we went more than four or so pre-questions, depending on experiments, and sometimes there’s just two. Why is that? Well, in part, because if you ask people many, many, many questions on something they know very little about, and then they have to try to find the answers afterwards, trying to keep in mind, say, I don’t know, 10, 20 questions is difficult, and so a limited number of targeted pre-questions on perhaps the topics that are most crucial is probably a good way to go in terms of number. Now, in terms of type of questions, you just mentioned the questions that we used were integrative questions, and one of the reasons why, besides what was mentioned, is that there’s some prior work on different types of pre-questions. Some pre-questions that have been investigated are so-called detail-oriented, that might just target a little fact or a number or a little detail, and then there are these integrative questions that might connect two concepts or different parts of a text passage, what have you. Those seem to be more effective at improving learning. So, in addition to the number of pre-questions, the type of pre-question probably I would recommend to instructors to think more about integrative pre-questions. Even if you want a student to learn one particular small detail, maybe couching it within an integrated pre-question might actually be better. And then there is a third sort of practical point that I just wanted to raise, which is that definitely keeping in mind that pre-questions, in a way, serve as almost like a introduction to the topic, so these are not the kind of questions used necessarily to really assess mastery in a retrieval practice sense: “Explain this concept or this theory that you’ve just learned.” If you ask such a question to someone who has not learned that concept, that theory, that question may be so difficult they don’t even know how to begin to answer such a question. Then it’s probably not the best type of question… more like a question that is in some sort of currently perhaps nebulous sweet spot of they can come up with some sort of guess, it’s probably not correct, but it’s also not so impossibly foreign that they look at this question, and think, “Well, I don’t know where to begin.” So, those are at least three tentative recommendations that I have about how pre-questions could be developed by instructors for effective use.

Judith: Something else I would like to add is that you can also first think about what do you want your students to learn the most. So, what are the most important aspects? Knowing that these pre-questions work best for what they ask for, you could make use of that and target the most important aspects with the pre-question. That’s also something related to our study, to think of the cognitive levels associated with your learning objectives, and to see that the questions are generated in a similar way, so that also guides the learners. We also looked at this in our pre-questions, and noticed that for the first experiments, the AI-generated questions were mostly on the lower cognitive levels of Bloom’s taxonomy, so mostly remember and understand, whereas the human-generated questions we used that were used in an earlier study, use comparisons, and that was not the case with our AI questions. So that was the focus of the fourth experiment, where we engineered a prompt that asked for comparison questions, and then the AI generated these questions, but only then. So that might also be something to take from this study that the Gen AI might need an additional prompt to generate this kind of question, but that might also be the case for a human being that thinks of questions that maybe we also need an additional prompt to not just ask remember and understand questions, but also target these higher levels that we name in our learning objectives, and to kind of reverse engineer our questions from what we want learners to take away from the learning material.

Steven: Yeah, and if I may add that Judith raises a very important point, which actually qualifies the finding about basic versus more elaborate prompts, even if you use a basic prompt, you know, generate questions, you can still add in there, generate questions that involve comparison. You can add some details that’s not necessarily required multiple premises in a very complex prompt, but some of those things actually might be necessary through even a process of a little trial and error to come up ultimately with questions that meet certain characteristics that are desirable for learning, including higher order type questions, questions that tap different cognitive levels, that promote the types of learning objectives you want to promote.

John: One of the things you were doing in the study is comparing AI questions with human-generated questions. Was there any apparent differences between the types of questions that humans generated and the types of questions that AI generated?

Steven: We did have sort of as an additional measure in the study… we didn’t tell them these were AI-generated questions, we asked them to guess who made these questions, and we would just give them all the questions that we had in a particular set and some were human-generated, some were AI generated. We just asked them, and they just could not tell, and so this is, I think, part of a larger challenge now, which is differentiating between AI-generated and non-AI-generated materials. Now, not only are the questions as effective as those generated by human beings, but also human beings cannot tell that they were AI generated, but I thought that was kind of a fun little thing, and it speaks to this larger challenge that we have, whether it’s trying to detect AI use in our students’ essays, other things like that.

Rebecca: We always wrap up by asking, what’s next?

Steven: Well, in my research group, and also in collaboration with Judith, we’re really interested in exploring a variety of aspects of pre-questioning, in terms of, for example, ways to implement it in authentic educational circumstances more successfully. So, for example, studies in the classroom, or studies exploring how you might integrate pre-questions into learning activities that are very common, like reading a textbook chapter, things like that. So we’ve explored AI-generated pre-questions, we’re very interested in implementation that could be easily translated to real-world practice, so that is one area of very active interest related to pre-questioning.

Judith: And then a related question in our labs… Well, I thought of it because it’s somewhat related to the current idea, because I’m currently working on a study with a PhD student where we want to investigate whether spacing, distributed practice, which is recommended both as a teaching strategy, but also as a learning strategy, but it’s investigated almost always so that the entire schedule, sticking to the schedule, that’s decided by the experimenter, and our idea was we really don’t know whether that works as a learning strategy, so we implemented it in a way that gives some responsibility for the learning, for sticking to the schedule, to the learner, and see whether it’s still an effective strategy, and also to see whether something like gamification might help learners become better spacers.

John: You’re doing some great work here, and actually that reminds me a little bit of the last interview with Steven and his co-author on another study that looked at classroom implementations of interleaved practice, because a lot of the interleave practice studies had been done in the lab and it was nice to see what happens when it was actually introduced in the classroom environment. You’re both doing some really good research, and we’re curious to see what you’re going to find in those new studies. Thank you.

Judith: Thank you.

Steven: Thank you.

Rebecca: Yeah, thank you so much for joining us.

Judith: Yeah, thanks for the great conversation.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

452. Revisiting A Curated AI Framework

Shortly before the start of the spring semester, Tim Curry joined us for an episode discussing his ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology.

Show Notes

Transcript

John: Shortly before the start of the spring semester, we had an episode discussing an ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Tim Curry. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology. Welcome back, Tim.

Tim: Hi, it’s great to be back here.

Rebecca: Today’s teas are:… Tim. Are you drinking tea with us today?

Tim: I’m not right now, but I was, and the tea I had this morning was a spice blood orange tea. It was quite tasty.

Rebecca: Looked like it was in a beautiful mug.

Tim: Oh, yeah, yeah.

Rebecca: turquoise… ribbed…

Tim: Yeah, with a little bit of a, almost a goldish highlight on the ribbings.

John: A bit iridescent.

Tim: Yeah.

Rebecca: How about you, John?

John: I am drinking a Lady Grey today.

Rebecca: Oh, I have its companion, a British breakfast.

John: Very good. Earlier this year, Tim, right before the start of the spring semester, we discussed a plan that you had developed in which students would engage in a structured sequence of interactions with a chatbot tutor that you created in each of the course modules in your health science course. The course was scheduled to begin actually the afternoon of the day when we last talked. We’re following up today to see how this plan has worked out once it hit the actual classroom. First, though, could you provide our listeners with an overview of the course, because not everyone may have listened to that, or it’s been a little while, and just remind us of what the course is and how the AI activities were structured.

Tim: So, as a rewind here, so I actually use this in two of my classes. The first one being our HS 300, which is a class focused around the diseases we get as people and how we work in that space, in more of a public health focus. And then the second class was a fitness wellness class that’s focused on how do we do physical activity more of a clinical population focus, and it had that as the topic, and then it’s also our writing-intensive higher-level course. So, in both of the classes, we had a tutor bot. The way that it functioned was I gave them a file that I built out that was a learning file that the bot would read and would have an idea about what are the topics that the student needs to learn and then would work with them in a very Socratic style to help them learn the content and the depth that was required as kind of the minimum foundation points for them walking into class in the first day and they did actually fill in the questions I had in that file, and it was required to hand that in, as we could call it proof of learning.

Rebecca: So I’m picturing the first day. How did students react to your plan?

Tim: About what I expected, honestly, which was tentative. And that first day, and I’ll actually expand that, and say that first week, we had a lot of discussion in class about what does this look like, what’s the purpose of it, as we had, I believe, discussed about six months ago. One component of that first week was about roughly half of one of the first two days of class in that first week, they had built-in time in class for them to actually dive in, learn about this GEM that was the host of our bot, and actually get to start working with the first file for that first week, and get their hands on. And I’ll say that first day it was very interesting to hear about the reactions, and some students, within the first five or 10 were like, “this is not for me,” which is fine, because that was how it was built, was that they did have an option B that they could do without having to engage with the bot. Quite a lot of them had said, “I’ve never ever engaged with an AI like this before, and I’m a little uncertain, but I’m willing to go and give it a shot.”

John: Roughly what proportion of students chose to continue with it, and what proportion chose to opt for plan B?

Tim: Yeah, that’s a great one. And in fact, I’ll say that was one I was watching throughout the entire term, because they have to hand in this like proof of learning, which allowed me to kind of track what was the proportion. Roughly, the proportion was like 70% continued with AI, and 30% chose to go the more traditional text-heavy route. I’ll go and say this. I would argue in a 16-week term, I would say just off the top of my head that by about week six the students had settled into the way that they were going to engage.

Rebecca: That’s not too bad, about a third in.

Tim: Yeah, it’s not too bad.

Rebecca: How did it work?

Tim: Really well, I would say. Overall, there were definitely things, that within the first two weeks I hit the fun point that I’m sure all three of us have at some point, which is Gemini pushed a major change in their models and updated, and suddenly the AI tutor [LAUGHTER] wasn’t working the way I was expecting it to work. When I had trained it and done the last round of kind of making sure it was working, that was end of December, start of January. By the time we hit like week two of the class, the model had been adjusted and suddenly it just kind of wanted to go off in its own in right field and treat the student as if they were like a medical student and go in way too much depth. So after that, I realized very quickly in that week, too, like, “oh, we have a problem,” so very quickly went back, adjusted the system prompt, put up some more forms of constraints in a guardrail to rein it back in, bring it back in, and effectively the way I did it was to get the bot to do less explaining and focus more on explicitly what the student needed to learn and allow the student to drive where they wanted to engage further. And after that adjustment it seemed like that solved the issue and the students were able to better engage with the bot overall and I’d say overall, for both the AI bot and the textbook route, the big change of having these effectively notes that they were expected to hand in dramatically improved their readiness to engage in a flipped classroom, which is the way I teach, and so I would count that as a win period for the AI bot, the fact that at the end of the day the grades were effectively the same regardless of the way that they engaged. So I would go and put that up as a positive sign overall that the AI at least performed as good as the traditional way.

John: And in a first iteration that’s probably a very positive outcome, because often new experiments have some rough edges on them. I was especially curious about the performance of the Gemini Gem, because I was working with a group of 18 faculty in SUNY from various campuses who were experimenting with chatbots, and many of them started with ChatGPT, but then moved to Gemini, and they had very similar experience, that the GPTs in ChatGPT seemed to pay much more attention to the guardrails that were put in, while Gemini Gems seemed to go off in directions that were different from what the instructions were, and that caused a bit of frustration for several of the faculty, because one of the reasons they moved to Gemini Gems was because their campuses had Gemini, and they thought that would allow all students to have more equitable access, but they had begun in ChatGPT because the grant had funded Chat GPT accounts for them, but again students were complaining that the free accounts didn’t allow that much interaction with it, while those who had paid accounts had much better quality of interactions, so it was a trade off, and ultimately pretty much everyone who had that choice did stay with the Gems, but it did require quite a bit more training. So, I think your experience there was not unique.

Tim: And I’ll just go and say this as well. I did also one more adjustment I did in week three which further helped was, in Gems, you can have certain variations of it, right? Just go and turn on as the default that the student can go and turn off if they want, but generally they don’t, and I defaulted on the guided learning mode, and that further enhanced that Gem as well. 8:11

Rebecca: So it’s always interesting when you try something new and something goes awry, and obviously it sounds like you adapted well. How did the students adapt to this minor interruption, especially because it happened early on.

Tim: I think the biggest item in terms of the feedback was that I was very open that first week of “this is an experiment, we don’t know what’s about to happen. I don’t know, I know what I’ve done for my testing, no idea what’s about to occur here, and I want to know your feedback, so let me know. We’ll go and talk about this.” And opening up that window of this is an experiment, you are my group. If this doesn’t work, we’ll adjust and change. I believe that it allowed that space for a dialog to happen, and that first week we did half of one class where they had a chance to go and get in there, work on it. I got a lot of feedback from them in that class, then once we got in to week two, that feedback of like, “Hey, this took a really long time because it kept like wandering off over here, and I had to bring it back, and it was really annoying,” I think was the biggest response I had, and then after we got it tuned in, I began to hear more feedback about, “Okay, I kind of like this, because if I’m totally lost on what is this problem with the heart, why do I care about it?” It really allowed them to explore if there was an area or a disease that they really found fascinating, and it just gave them more freedom to go and explore that. So, overall, the response that I had from the students was, I’d say broadly, it was kind of two broad groups. One was: this was very intriguing, and it was about the same as what I’d expect if I read the book. I’d say out of the 70%, like roughly I’d say like maybe 10 to 20 of that was that group of like it was nice I didn’t have to pay for the book, and then the rest of the group was generally that they liked the ability to have that freedom to explore or to ask the bot for more help in a space as they felt need.

John: Going back just a little bit, you mentioned you have these chatbots that are trained as Socratic tutors on that module, and then students complete an assignment at the end. Was the assignment built into the chatbot or was it something that was a standalone, and how did you receive all that? Did you get a transcript of the interactions along with some responses or something else?

Tim: So what I did was I gave them a file that was in Canvas, and the chat bot understands how the file is laid out, and it has a set of questions and tables and cases and all of that stuff without the correct responses put in there, and so the student’s job was to start a chat with this bot and to give it the file, and so that the bot knew how to drive the engagement, and then what they did was, as they engaged with the bot, they would go through and fill out each of the components of this file, and then they would just go, and it was a hand in before the first day of class that week that we did through the Canvas site. Now, a big part of not only these classes, but just the classes I go and teach across the board is that I do a contract grade, so for these items, because the focus is on “I want you to engage with the content, I want you to learn it, so in class we can work on like going further and building out skills,” so they knew that as long as they completed all the work, it got marked as done. I wasn’t about to go through with a comb and dock them points for everything that wasn’t exactly right, but once they had that handed in, it unlocked the key for them, and then they were in charge of ensuring that they went through and reviewed the key to make sure that they were in the right space.

Rebecca: Seems like a good plan, when otherwise that would sound like a lot of grading.

Tim: It did work well.

Rebecca: And you said that the students who chose the textbook method essentially did the same assignment, right?

Tim: Yeah, so it was adapted for the textbook, but exact same idea. The plan, as well, was to keep the overall load on the students the same. It seems like, based on what they reported in evaluations, that was the case, that regardless of path, it was about the same amount of time to complete each one.

Rebecca: Besides the tech surprise, were there other surprises, things that you didn’t expect? Anytime you go into an experiment, things come up that you don’t expect.

Tim: I think the one that was unexpected and interesting, and I suspect both of you, being faculty, you’ll hopefully laugh at this, was HS 300, in particular, is a content-rich class. I mean, not only is it a 300 class, but human diseases, there’s a lot of them, [LAUGHTER] like there’s far more than we can fit in one class, so we only focused on functionally the major diseases for each of the body systems, and still there’s just a lot of content that you need to know, and the feedback that I had at the end of the day kind of made me laugh, which was regardless of the path that they picked, because now we were requiring them to show at least foundational knowledge prior to walking into class as a core component of this class was that they were spending three to four hours a week outside of class doing all this work and going and like preparing for class and that it was more load than what they were used to or expecting and a lot of the feedback was “and I feel like I learned so much more because of this.”

John: That’s a nice positive outcome. We don’t always get that. We often get the “this class is so much more work than I expected,” but it’s not always combined with “and it was worth it.” That’s a really positive outcome.

Tim: Yeah.

John: You had mentioned that you were using a flipped classroom approach, and earlier when we talked to you, you mentioned that you were using active learning activities. Could you talk a little bit about what a typical class would look like in terms of a typical day in the classroom?

Tim: I’ll actually expand, I’d say a typical week, because usually things span like day one to day two. So, a typical week, the first day of class, we go and start off, and the first thing that we do… well, after the “hey, this week in class… , and don’t forget about this,” all of that fun at the start of class was we would do a tRAT, and for those who don’t know, a tRAT is a team readiness assessment test, and it is classically done with these cards that look like a card I’d buy for a lottery game, where we would code effectively a multiple choice quiz to the card, and then they work as a team to go and figure out what do they think is right. If they’re not right, they have to keep working on it until they go and get it right. I’ve created a digital version of that, so we begin with that, not only to kind of like warm their brains up, because I would get them Monday evening at 5:30.So, a lot has happened. [LAUGHTER] So that helped them kind of get their brains back into our content for the week and help them understand, like, “What do I recall now? What do I need to go work on?” and then we would dive into one of two things, if we were having a bigger application week, which was a lot of the more impactful work, we’d walk through, here’s our work for day one, day two, here’s our task for the end of the day, and then they would just dive right into it, and in those cases, day one was like they were working with a small, and by that I mean no more than four of them in a team, a small team of their peers, and they would work on doing more of the engagement part, and then day two would be wrapping up that, and then doing usually a presentation to go and share out. So, an example of a week, this one was a hit. It was new in this format, and it seemed like it went over well. When we were talking about diseases of the lung, the pulmonary system has a lot of diseases that are infectious, a lot of ones that we would have to report to the state if you happen to get it, and even if it isn’t directly involving the lung, that might be the way it’s spread. So we wanted to introduce them to the fact that there are these things called an investigator that works for the state, and all they do is investigate potential forms of these that might spread that we might need to do some public health work on. So in the first day, they were handed their initial case that they had to go and talk with this AI bot to figure out what had the patient done, where were they at? They knew that they had TB, but they didn’t know anything else. And then they were allowed to expand their investigation to potentially up to three additional bots, and they were told you might have to talk to all three of them, you might only have to talk to one of them. You have to figure this out on your own. You have to do the investigation. So they went through the path and the process to go through and investigate. As I’ll just go and say they did have to talk to all three. The first one went and led them to two of the three bots, and then one of those two went and led them into the last bot. And then they had to go and figure out what do we think happened, and then our day two of class, they wrapped up that work, and we had a chat as a whole class about what did you find out, how do you think this spread, what do we need to do next, and it was actually quite a complicated case, but that’s the way that it spread throughout that week was to go and show them, like, hey, this is an airborne thing, it’s a dangerous thing, and this is a real job you might be able to actually get, and by being able to engage with the AIs as effectively a patient, they had to go through the process of the fact that one of the bots was extremely not helpful, in fact, actually almost like was very rude, and the reason for it was because of the circumstances built into this bot if they were found out to have a TB test that was positive, they had significant potential losses that would be a major problem for them. So that allowed them to actually engage and attempt to work through real-life experiences that they might actually have to go and deal with.

Rebecca: That sounds like a fun activity, that of course leads me to want to know how did you build those bots? … and a little bit of detail about using AI in that context.

Tim: Yeah, so using AI in this format is, I’ll say for myself, has not been a new thing, it’s actually been something I’ve been doing for a while. It started for me out pre- 2022 and the world exploding and changing on us, but it started before that, because in my exercise physiology course I used to teach, engaging with a client is really important, and learning how to do that, so back in the pre-AI time [LAUGHTER] I had used IBM Watson bots that you had to build very crudely when we go and look at then and now, where they had to work with this fake form of a client to pull out all the risk factors and try to understand what were their goals and how do I work with that, and it worked okay. It didn’t work great. It worked okay, because of just the inherent limitations. After we got into our current era, the last four years, roughly, that expanded greatly, and what I would do is I’d build out a prompt for the GPTs first, and now for our gems that included all of the different things I wanted, as like building, so it did include things like here is the background of you as a person. Here is a personality. Here are the, for example, in this case, if they had had any type of testing done, what are the tests? What are the results? What are the tests that we don’t go and know about within all of those? What are the ones that you immediately share? And then there might be additional forms of like a test, like maybe an X-ray of the chest, that unless the student asks, the bot won’t just offer it up. So, what are you allowed to just go and bring up? What are you allowed to bring up only after they go and ask about this thing? And then I did find that when we get into the disease space, where I want them to model the signs and the symptoms of a certain form of a problem. I need to give them exactly what are the symptoms you’re feeling, and I never give them strictly the textbook, because that’s usually not the reality. They’ll have some, but not all. And then I found that it was also very important that I put a very strict box around the bot, this is what you have, so you are a person who has a problem A, it occurred because of B, C, D, E, F, G. You’re not allowed to share any of this. It’s only for you to use to better drive your responses to the students, and so that as they ask about a thing like, have you had a fever, if I hadn’t defined that, it at least knows which way it should kind of push.

John: One of the things you had mentioned when we talked last time is that you had students engage in some self-reflection, and you referred to this a little bit earlier. What did they reflect on? Was it a metacognitive type of reflection, or was it reflecting just on the material? Could you tell us a little bit more about that?

Tim: So this actually, I won’t say that I invented it, because it’s not mine. I actually borrowed a resource from the American College of Sports Medicine annual conference three years ago in Boston, where there’s a really good discussion about flipped classroom approaches within the exercise physiology space, and one of the items that was brought up that I borrowed because it had been used in some research work was this weekly graded survey. So, in the Canvas shell, it’s arranged as a quiz, but it’s marked as a survey that is graded, and all that that means is once they go and do it, it’s just marked as check done, and it runs through a range of different things, and it’s the very last thing that they’re in charge of doing every single week of the term. So it begins with exploring how was the week overall, and I don’t say just for this class, I just say how was the week overall for everything, and that’s a multiple choice, and then we go and walk through, how well did you feel prepared for this week in class? How was the week for you overall in your life? Because we all know that we teach folks who are increasingly more stressed out right now, like I don’t know about both of you, I might have brought this up before, but I have students who, 50% plus of my class, is in six to seven classes and working one or two part-time jobs. They are incredibly busy, and then there’s other life that happens on top of that. So, having a feeling of like maybe they were off this week and maybe their life just took a bad turn in this week, so that makes sense. And then we went and moved into some more open-ended stuff, which included: What did you learn this week? What do you have in terms of a question? Were there any types of like a moment that was an aha for you (it’s fine if there weren’t)? And then we do a bit of goal setting at the very end, which is: “What do you want to do next week to improve on what happened this week? And then when you look back at this week: Did you accomplish the goal that was set for the current week? And then finally, just kind of a broad: “Anything else you want me to know, like did your car break down on the drive up from Phoenix, and you were stuck on the side of the highway for a whole day? And it’s been a thing that I’ve used for a couple semesters now, and overall it seems like it works very well, and the feedback has been very positive, in that a lot of the students are like, “I’ve never had a class require me to, and I’ve never thought about, stopping at the end of the week and reflecting over how things went,” and it’s helped them with a thing like, especially my students, who are those “got to have an A” students, which was me, in fairness of, they might have a bad week, and then when they have to stop and reflect, they’re like, “Oh yeah, I had to randomly work more hours than I thought, and I found out that my dad has to go in for a cardiac stent, of course it was a hard week.” And so it helps them kind of put the week in context better.

Rebecca: In the spirit of reflection, now that you’ve had [LAUGHTER] this experience over a semester, would you continue the approach? What are some things that you might do differently next time? This is your moment to reflect. [LAUGHTER]

Tim: Yeah, so reflection for me would be this, I think I wouldn’t, and I’m not planning on moving forward in exactly the way that we did. Learning that just the fact of Google will push adjustments to their models regularly, and it’s not always easy to go and track it down, that makes it a little bit unpredictable, and I think there definitely is time that could be clawed back because there are times when AI was just inherently not efficient. So in the current model I think there are improvements that could be made, and how I would improve this is I am very much a fan of OER, especially at the current time when things are just expensive, so I would want to create and provide content that provides that foundation that is a little bit more controllable, so I have a better idea of like this is the time that they’ll spend here, so I don’t say like it’s going to be an hour or two, and then bringing in that AI in a more limited way, but in a higher impact way, where instead of using the AI to learn, they’re going and using it as a way to apply and test their own knowledge in a manner that’s going to give them immediate feedback and allow a challenge to the way that they actually approach the content for that week.

John: One of the big concerns with the use of AI models is personally identifiable information. Were you concerned about that issue? I know the contract that Google provides with campuses is that they don’t use the data for training, but were there any residual concerns about that issue?

Tim: This is a great point, because this is a big thing that I work with our faculty on a lot, because with AI’s advancement so fast, it’s so easy to ignore the legal terms of use, which I know everybody reads in fine print. [LAUGHTER] And so for this spring semester I was not worried about that, number one, because as you’ve alluded to, we have a Google ed license, which explicitly prohibits them from training on our chats that are through our accounts, and as a part of that, I made sure that we had a pairing up with that, with ensuring to be very clear, I want you to log in through your college accounts onto this, that’s the only way that it’s like a guaranteed private, and part two, even with that, I don’t want you to put in anything that you wouldn’t mind having as like a TikTok post, as like a public thing. It’s here to help you go and learn the content. It’s not here to help you troubleshoot why you’re not [LAUGHTER] feeling great today. So I did go and build in a bit just to help our students better understand, like, what are the safe and unsafe ways to engage with AI in this format, because I don’t think that’s talked about enough in the world. Now, with that said, I will say that there’s a very interesting change on the horizon, and that’s because our institution has been… for quite a number of years now. We have both Google and Microsoft. Don’t ask me why, I don’t know why [LAUGHTER]… but moving into this fall we’re moving to purely a Microsoft license, so we no longer have the Google license, which has been an interesting change to find out about, because it’s been very recent that we’ve found out about it, that really affects this work. So, moving forward, we, we being myself and the faculty who go and teach these classes, we’re going to have a conversation over the summer about how do we manage this, because the Microsoft format is through the Copilot, and unfortunately the way that it works is I can make these – they’re called an agent in place of a Gem, however, once I go and push them for a student, at least right now, from what we’ve been told, it costs us tokens, which means that there’s an ongoing cost that’s not like a Gem, where it doesn’t matter how much they engage, it’s all wrapped up in that fee. This is a big change, because, as you know, as I suspect, both of you are feeling like finances are a worry for a lot of institutions, so we want to be very thoughtful. So we’re having to go and talk about this, about how do we move forward in the future, is this a thing where we keep it as a G em, and we’re just explicitly clear about you should put no private information in here, because even with a free account, there’s a fair amount of use you can do with these, so that’s a big change we have to go and figure out as we’ve moved away from the Google group.

John: That’s really unfortunate, because I know in campuses where people have had both the choice of faculty is to use Google tools and to use Gemini and Gems, both because of the cost and because Copilots development is behind the others. Even though it’s based on the Open AI model, the training and everything else is unique to Microsoft, and it’s designed for a business environment to keep information locked down and also to build based on the use of the chatbots created by the businesses that are running Microsoft, because it’s designed more for commercial enterprises than for academic enterprises.

Tim: Yeah, well, and I’ll just say, as an expansion, too, because depending on the institution, I mean, one of the things I want to look into as well is there are increasingly more open source models that we can use, and there’s plenty of techniques that, if you have the compute capacity, you could potentially train a larger model, just going and thinking, like maybe you train a Llama 4 and use that as a teacher to train a student that’s smaller that then we can host locally. So there’s plenty of alternates, but it requires you to have the proper not only compute and server capacities, but also the fact that you understand how to make that work. So that is a thing I want to look into, because I do agree with you, Copilot is much more on the business versus straight up education side. So it’s going to be interesting to see how this changes, and perhaps that’s a point to go and have a chat about in like a year after we’ve had a year to explore how have we adapted to the Microsoft model.

John: And that is probably going to be a much more of a transition than adjusting to the change in Gemini.

Tim: Yes, it will be. [LAUGHTER]

Rebecca: It’s a good reminder how quickly this landscape is shifting in the role that the changes in technology have, the role that these companies play in what we’re doing, the finances around it all, they all have an impact on what we do in the classroom. We always wrap up by asking: “What’s next?”

Tim: Yeah, [LAUGHTER]I think I might have alluded there. So, what’s next is for this fall, for both of the classes where we’ve used this in. First of all, one have to figure out what to do with the fact that we will no longer have G ems in an institutional license, so have to figure that out. And then two, how are we going to do adjustments with the classes to provide content in more than just text, because that gets a little bit boring. So, how do we provide the content foundation, and then back it up with applied learning through the use of an AI to further enhance learning? So, we want to keep moving forward with this idea. We just have some hurdles we have to go and jump over first.

John: It’s going to be a challenging experience moving forward, but I hope things go well.

Tim: Me too. [LAUGHTER]

Rebecca: I hope you’re a hurdle jumper, because they’re going to keep coming. [LAUGHTER]

Tim: Yeah, I think I might be sore by the end of this fall. [LAUGHTER] It’s going to be interesting to see what happens.

John: Going back to the multimodal thing, one of the people I worked with this year from one of the other institutions in the health sciences field had put a microphone in a mannequin and a speaker there, and they had voice interactions with a patient who was basically just a plain mannequin, but they were interviewing it and asking questions. They enjoyed the multimodal interactions, probably a bit more than just text interactions.

Tim: That’s a really interesting idea. I might have to follow up with you more about that, because that’s a really clever idea.

John: It wasn’t mine.

Rebecca: I immediately saw your gears turning. [LAUGHTER]

Tim: Yeah.

Rebecca: Well, it’s always a pleasure to talk to you, Tim. Thanks for joining us again.

Tim: Yeah, thank you. I’m sure we’ll follow up again in the future about how did the transition go. [LAUGHTER]

John: You’re not alone, because institutions are cutting costs, and many institutions, including us, have access both to Microsoft and to the Google tools, so we’ll see.

Tim: Yeah, we have some strong feelings as faculty.

Rebecca: Yes, we do. [LAUGHTER] All the big feelings. [LAUGHTER]

Tim: Yes.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

451. All Day TA

 Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, Joshua Gans joins us to discuss an attempt at creating an AI platform that accomplishes this.

Joshua is a Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.  Joshua has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours.

Show Notes

  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Prediction machines, updated and expanded: The simple economics of artificial intelligence. Harvard Business Press.
  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Power and prediction: The disruptive economics of artificial intelligence. Harvard Business Press.
  • All Day TA

Transcript

John: Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, we explore an attempt at creating an AI platform that accomplishes this.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is

Joshua: Gans, Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.

Joshua: has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours. Welcome,

Joshua:.

Joshua:: I’m happy to be here.

John: We’re very glad you are joining us. Today’s teas are:…

Joshua:, are you drinking tea by any chance?

Joshua:: I am not. I’m sorry. I see it in the title. I don’t see it in the instructions for the podcast, and I would have followed them. [LAUGHTER]

John: Most of our recent guests have not.

Joshua:: I guess the listeners out there should pause this now and go get their tea.

Rebecca: Sounds like a good plan. I have English afternoon today, John.

John: and I have an Irish breakfast tea today.

Rebecca: Going all imperial. [LAUGHTER] At the initial release of ChatGPT in late November 2022 a primary concern of faculty was really on the impact of generative AI platforms on academic integrity, but now that we’ve had more experience with generative AI tools, what have faculty and institutions learned about the risks and opportunities of generative AI?

Joshua:: Well, those initial concerns that people were worried about have only got worse [LAUGHTER] in many respects. Nothing in the development of AI has brought any comfort to those concerns, at least insofar as the foundation models are concerned. I think this was always on the cards, which is one of the reasons we started working on this problem back in November 2022 when ChatGPT came out. Put simply, it’s not just a student issue. ChatGPT helps you do stuff, and it helps you do stuff that you don’t like doing. It just so happens that we have set ourselves up for a challenge there with regard to students, because we give them a whole lot of things to do, for most of students, most of the time they don’t like doing it, [LAUGHTER] relative to other things they could do. And I think they very much understand why they should be doing it, but unfortunately, our students, it turns out, are quite human, and so a temptation is right there that they will take it. That said, there are actually things we’ve learned from this experience. I think actually students have been learning faster than faculty about AI. One of the challenges we have is, well, how do we prepare the students for the world that they’re going to be working in, where they’ll have AI tools, etc. when we, as faculty, and everybody else doesn’t know anything about that, [LAUGHTER] and really it’s very difficult to even anticipate. Well, one of the fortunate answers is that, amongst all groups in society, students by this very mechanism have actually spent more time with AI than most people, [LAUGHTER] and so they are more keenly aware of where it works and where it does not work. That may still pose a challenge for assessment, but I think students I’ve noticed have got better at using AI, even in their assessment, which has got to bode well for something.

John: We give a lot of workshops here, and there’s two sort of responses we tend to get from faculty. Faculty tend to be really excited about all the things that AI can do for them when they learn about them, but they’re really troubled about all the things that students can use AI for. What sort of changes should we be making in teaching practices to deal with an environment where student access to AI tools is pretty much ubiquitous?

Joshua:: I think, when it comes down to it, the problem we have is the problem we’ve always had, [LAUGHTER] which is how do we engage students and get them to learn for their own good. Now our tried and true method is to basically hold a set of carrots and sticks, or some mix between the two, and then come up with surrogates for learning, which is essentially not proof that you’ve actually learned or deeply understood, the things that are correlated with it, and they may be more correlated with it, or more appropriate for certain types of students, and certain types of learning patterns, and neurodiversity, we call it, and things like that, but that’s always been the case, and so basically our approach has been: you should learn this or else, [LAUGHTER] and the “or else” is also done with a caveat as well. It basically says “I’m a professor, I can come and I can give you lectures and talk with you. I can spend a little bit of time in office hours if other people have additional questions, but by golly, I hate grading. I hate setting assessment. I hate grading assessment. So I’m going to give you assessment a couple of times a semester, and we’re going to be all in on that, you either do well there or you get whacked with a stick.” Now, that creates a very high-stakes, high-pressure environment, but I don’t believe there is any scientific merit that that’s a good way to learn. The way to learn is the way we actually assume students learn, but none of them do, which is every week you prepare for the lecture, you do the readings, you come in and you pay 100% rapt attention in it, and then you repeat. That would be the nice way to do things, [LAUGHTER] but we haven’t matched the assessment with that at all, we’ve made the assessment to do with cramming for final exams and things like that. Now, to be clear, if you do do what the professors want you to do and pay attention in a methodical way, you do quite well. And you know which group of people actually did that? The professors themselves, when they were studying, they generally tended to do that. That’s how we select professors. So we basically created a mythical system with practical implications that are very predictable, and I don’t like to condone cheating, but if you tell people their livelihood depends on something, which is a critical moment, and they haven’t learned all this stuff, they’re going to do things. [LAUGHTER] I’m an economist, and that’s what we’d expect, and we’ve set up everything about assessment, actually, on the presumption that they will try and do those things as well. So, throw in ChatGPT, [LAUGHTER] you’ll get more of that. The sort of good news is that ChatGPT, and AI in general, is so powerful. We’ve got a whole group of professors saying, well, I guess I’m going to have to spend more time creating exams and proctoring it, and some people are talking about oral exams, and a whole lot of things. I guess I’m going to have to do all that, but the alternative is, or maybe not. Is there a way we can actually use the AI in a way that will actually alleviate the fundamental problem, which is high-stakes assessment, and so that was our motivation for creating All Day TA. We haven’t got to that aspiration, but that’s where we were starting from. Is there a way that we can reduce the incentives of students to go to ChatGPT and instead pay attention to the things that the professor wants them to pay attention to.

Rebecca: What I’m hearing you describe are faculty that sound a lot like students and students that sound a lot like faculty.

Joshua:: They have a lot of commonalities. [LAUGHTER] If you are honest with yourselves, we are all balancing all sorts of things, and in many respects, depending on your perspective, they can look the same way, which is one of the reasons why I don’t jump up and down with the moral force and outrage of some [LAUGHTER] of my colleagues, because we’re not exactly pure in this endeavor. So you can think about it, what if we were so pure and given that as professors we actually wanted to do the right thing by learning, what would we do? We’ll spend all our time on it. Then we might come up with something closer to what they do in Oxford and Cambridge, which is they have two or three students a classroom sessions. They teach all the time. They are able to monitor the students’ progress every week. It’s in an intimate setting, so everybody’s got an incentive to study and learn as they go along, and then who cares about the final exams? [LAUGHTER] They probably don’t mean anything whatsoever, and who cares about AI, because you’re sitting right there, you can talk to people, you can find out if they know things. So we know that if we alleviate the chief cost for faculty, that is their own time, then we can design the right system. So then there comes the question is: “Well, can we use AI to do that same thing?” And our answer is yes. [LAUGHTER] Yes, I think you can, because it is possible to take your own material, the things in the way you like to teach, transcription of your own lectures, add other stuff to it, and I know it’s not a popular phrase, but you sort of create a digital twin of yourself, and it’s a twin that can interact, and 95% of the time will give exactly the same answer to the students that you would if you were spending time there. But there’s a big difference, even in this ideal Oxford Cambridge world, I only have 24 hours a day, seven days a week, even in that ideal world. When we first did a very crude version of our chatbot for our own courses, trained on our own material, back in 2023, we had 200 students in the class, and they asked, over the course of semester, this chatbot, which is ChatGPT 3.5 which is like the very first version, 10,000 questions. There is no professor at Oxford and Cambridge who is answering 10,000 questions, and so you can see that. And now most of those questions came near the midterm and near the final… what a surprise… because it’s actually a very useful tool, given the current system, but at the same time, when we’ve done things since then, you can now use these same materials to generate questions to ask the students, and they use it for self-testing, and it turns out they like that even more, and that’s not even assessment, that’s just practicing for midterms and finals, they like that even more. And we got up in one class in Texas was 80,000 of these, they were using it all the time, and they’re very high quality because they’re trained on your data and with your learning objectives in mind, so it’s only a small step away from that to saying, “Why don’t the students do something a bit more like the Khan Academy does for elementary, middle, and high school, where it does little tests, and you don’t get to learn the next bit or do the hardest stuff until you’ve mastered the other thing.” What if we had these multiple choice short answers, or whatever things we might evolve to, if we put our minds to it every week, and then the students will be sitting there: “Well, I could go to ChatGPT for these questions, but the problem is by the end of the semester, ChatGPT won’t be able to answer them, and so I’ll need to answer them. So it may be just easier for me to just answer them now when they’re easy [LAUGHTER] and spread out my learning of it.” I don’t know if that’s possible. We have another thing we deal with in this, which is IT departments and integrations in universities. We haven’t had that sort of access to really put in the tests, but that’s the goal. The goal is, if we could alleviate the professor constraint, we can have low- stakes assessment at regular intervals, paced at the student’s own pace, that allows us to do things, for instance, like making it not by pass and fail, keep doing it until you know it, which is very low-stakes indeed. So that’s why I think there’s a lot of promise, but we’ve had a wake-up call. But the wake-up call reaction is not to run away from AI. I think it’s to run directly towards it.

John: Now, does All Day TA track student progress and provide reports to the instructor, or is it just something used by the students themselves without that sort of information. You’re getting some information on the number of uses, but do you have other information about whether people have attained mastery?

Joshua:: Oh, yes, we definitely do have other. So we don’t identify students by their questions and their performance, and you could imagine a world where some, for actual assessment, they give permission, and of course those get recorded, but there’s a reason for that. Is because we saw the sorts of questions students were asking, even if we didn’t know who they were, and like 40% of them were ones that they would be embarrassed to ask a professor, but it’s so much easier to ask the chatbot than to ferret around in their notes and textbook for the answer. So it’s really a smoothing tool, but if they knew the professor was looking over them, they probably wouldn’t want to do that. You want to encourage the students not to fear this thing and not to worry about, but at the same time, look at all this information you’re getting as a professor about what the students are asking about, where they’re struggling, and there could be also things that you expressed incorrectly. If the chatbots answering something incorrectly, chances are we know it’s because you failed to describe it properly, [LAUGHTER] and not because of the chatbot making stuff up. That’s not the way we’ve designed it. So, we provide basically, like you would with a really good TA, you’d sort of say, “Oh, where are the students struggling?” So, every week you get a summary of what they’ve been looking for, what they’ve been concentrating on, or where the difficulties are. You get access to all the questions, even though you don’t know who’s asked them, and you can look through all that. You can adjust your lectures next week because of the information you receive. You can also adjust the answers right on the spot, and say, “Oh, that’s not the answer I want to give here,” and I can adjust that. It all depends on how much time, of course, you want to spend with it, but all that information is there, and our experience is that professors love getting real-time feedback on what students understand. It turns out, for all our wanting to save time on things, we are interested in using our time well.

Rebecca: You’ve indicated a little bit about how many questions, the sheer quantity [LAUGHTER] of questions that students have asked. Do you have more feedback on how students are responding to the platform, and maybe what profiles of students are benefiting the most from All Day TA?

Joshua:: So, it’s varied on things. We’ve rolled out this platform that can be used in any course, regardless of the discipline, regardless of the subject, where it really has a nice fit, especially. And basically, the thing that gives them a nice fit is where those assessments are very high stakes, ironically, where you’ve got midterms, finals, and increasingly so at the moment, or where you’re in a profession like nursing, where not only do you have to do an exam, you’ve got to score high on it, very high usage of All Day TA for those circumstances. Where people have a PhD course, where they’ve got the assignments getting done, and other courses, which are a bit more discussion-oriented, and things like that, not so. It all depends. Now, we haven’t done nearly enough to know all those gradations, but we do know that as people currently move back towards in-person exams because of the AI issues, at least until we sort out something else. All Day TA is a huge component for that, because it allows you to do that without getting simultaneously overwhelmed with students in those exam periods, panics, and what have you. And we haven’t had a study of this, but I imagine also it alleviates student anxiety. I mean, we notice that most of these questions are coming out outside of work time, and that’s why we called it All Day TA, [LAUGHTER] because it really is around, and it’s there when the students actually need it, and so I think it can reduce anxiety just by its mere existence. Even though I would like it all to evolve into another system.

John: You mentioned this evolving into another system. Where would you like to see this go in the future?

Joshua:: Well, our business [LAUGHTER] is convincing universities to adopt this. I don’t know what your impression of university is, but fast to adopt technology is not their characteristic. The students are quick, [LAUGHTER] but the universities tend to be slow. Imagine you could integrate this into the LMS, and the students, you had all their courses and stuff like that. Imagine they’re taking a third-year course and they’re studying for it, and the previous prerequisite was in first year. Can you imagine that they have access to their first-year TA still in third year? That’s got to be a good thing. And then when you start thinking in those terms, you say, “Hey, why would we stop when the student leaves the university?” So the possibilities are quite enormous in terms of being able to provide something better for the students to learn and to work, taking into the very real account that they’re human beings [LAUGHTER] and have those sorts of issues with them. And so the immediate thing that I want to do is, I would like to, and we’ve had a couple of successes in this, being able to integrate right into the LMS at an institutional level. Fortunately, AI is actually not that expensive in terms of it requires authority, because doing things with LMS should require authority, but it’s not like a big budget item. [LAUGHTER] It’s actually quite small for what it is doing, and so I would like to do that, because I think that’s the path to providing a better student experience, but also providing the information we need to design whole curricula and other things. What is being remembered? Wouldn’t it be great to know in your third-year class what people are asking about on day one about the first-year class? Wouldn’t that be really helpful to know what they have retained and what they haven’t, and just be aware of it? We currently do all this kind of blind, and then once we’ve done that, working out how to put the assessment in, and the low stakes-assessments, and other things, is all possible, but at the moment, access to student records, and student assistance, and student privacy is extremely important, and we think it is the hallmark of this business, and these sorts of products, so it needs universities to be very proactive in wanting to do that.

Rebecca: One of the things that you just hinted at a bit is the idea of access to AI beyond the classroom and in jobs and beyond. So many college students are constantly expressing concern about the impact of AI on their future jobs, and what occupations might AI tools serve to complement, and where might they serve as a substitute for highly educated workers?

Joshua:: Oh, well, that is a very big and important question that we haven’t got a resolution to. We could have an entire other podcast about that issue, but the way I sort of thought about it in our modest example was, you’re a student, you’ve started your new job, you’ve had some on-the-job training and other things like that. Then something comes up, and they’re assuming you remember whatever, and you don’t. [LAUGHTER] So you can go to the internet, but it may be something that you recall your professor having talked about, and so you could then basically ask your professor again without emailing them for those things, and it could be brought back up, and it triggers the memories, and it makes those connections better, and it might just be much easier to take the knowledge that we’ve given the student, pour it into their head, force them to cram, but maybe they’ll just remember a bit of it, and they can pull that straight into the workplace and make them more productive on day one. That’s the kind of thing that I just feel we seem to say, “Well, you’re on your own,” [LAUGHTER] and everybody’s talked about lifelong learning and other ways of staying connected. Well, here’s another potential opportunity here that I can see how it might work.

Rebecca: Or an opportunity for quality control over the sources,

Joshua:: Yeah. Well, what if there are updates as well? It could be that, yeah, you learned it this way, but now there’s some additional things. I don’t know, the possibilities are endless. And, of course, this is not something any centralized company is going to be able to provide. It’s going to be the people who taught you in the past who are updating their own systems, and so long as you can push that out. It’s interesting, though. I think there’s a lot of faculty members who are concerned about such things. [LAUGHTER] I can imagine dire scenarios in which that occurs, but I have to admit I still have more than enough faith in what we add, in both developing content and also structuring it in nuance and thinking about learning and stuff that the replacement option isn’t there. I’m all about replacing the stuff that we don’t like doing, so we can concentrate on the stuff that we do like doing.

John: And colleges in general haven’t always done a very good job of encouraging that lifelong learning, because, for example, they set up systems to buy back student books, so that students don’t get to keep their books, and LMS systems, almost every institution will lock them down shortly after the semester ends, so all the material that they had and resources disappear. The practical reason for that is, if students have to retake a course, they don’t want them to have access to similar assessments, but for most students, having access to those resources would be really helpful, especially if that could continue after they left college, as you mentioned.

Joshua:: I mean, again, think about what having access to recent assessment is. Why is that the case? That’s the case because we kind of like to reuse things that work, but with AI, you can tweak it up enough that that wouldn’t be an issue either. And also that presumes that these are high-stakes assessments that people want access to, as opposed to low- stakes ones that they don’t. So, really, from my perspective, you can really see how we could push to a new system given the right set of circumstances. I’m hopeful one day one university will sort of like realize this and we’ll be able to evolve to it and really experiment with it. We’re doing a lot of that here at the Rotman School, but even here, where it’s been in our classrooms for three years, and a larger number of classrooms, we’re still not integrated with the LMS, because that’s controlled by the University of Toronto, which is one of the largest universities in the world, [LAUGHTER] and I must admit, with the current security issues that hit Canvas, no one’s jumping up and down to do things, so it’s a challenging environment.

Rebecca: You spoke a bit earlier about the concerns of faculty not necessarily having a lot of experience with AI, but still needing to help students prepare for a workforce that does use AI. Can you talk a little bit about additional ways that we can better prepare students for that work environment?

Joshua:: That is a very difficult question. One of the problems is that things are changing so quickly, so even when we get some research on what might be working for students or what they might need in the workplace, that was from a model that was a year ago, and now is it relevant? Now that will settle down eventually for that stuff, but I don’t see any short-term thing there. I think it’s more we have to understand that we’re running big experiments all the time now, and not only the students learning, but we are learning as well as we do them. A lot of faculty I’ve encountered have chosen: Well, what if we just pretend that the thing doesn’t exist?” [LAUGHTER] And the problem with that is you will not learn, you cannot learn, you cannot serve your students if you pretend the thing doesn’t exist. Now, there could be great rationales for trying to work out what you want the students to learn without AI and what you want them to learn with AI, and we’ve had this challenge before. We had the same challenge when we had calculators, and we had the same challenge when I was told, when I was going through high school, I wasn’t allowed to type up my assignments at all, because computers started to have spell checkers, and that boy did they not want me to use computers to spell correctly. I should just somehow do it myself, even though let’s face it, the second I was out of there, I never had to worry about that again. So [LAUGHTER] that’s the kind of things that we’re dealing with, but there are things you have to learn, and any new technology really sort of forces you to think about what is it that I need to get into a student’s head, and will AI get in the way of that, which it could.

Rebecca: Probably like the need to prioritize limited resources, limited time with students. You have to decide what’s the most important to focus on, and what maybe we can offset to technology.

Joshua:: Indeed.

Rebecca: It’s always about resource management, isn’t it? These economists….

Joshua:: Ah yeah, exactly, [LAUGHTER] that’s correct. [LAUGHTER]

John: Going back, though, to the labor force, as Rebecca had mentioned before, there’s a lot of students who are really anxious about what their job prospects are like, and we’ve seen in the last couple of years that the job prospects in the U.S., at least, for a new college graduate have deteriorated quite a bit, and it’s not clear how much of that is due to AI, how much, because of the random policy shifts that are occurring at the federal level, or whatever else might be causing these things. One concern that’s coming up a lot in discussions is whether college is still going to be relevant in a world in which people have access to AI tools, and can use them to substitute for the types of learning that occurs in college. Do you think college is going to disappear or change fundamentally in some way?

Joshua:: Well, that is a, of course, a great question, but it’s not a question that just exists because of AI. I think, actually, with proper introspection, we should wonder about whether we have been, for the past few decades, serving students well. Now the evidence is that they earn more, they can get more jobs and other things for a college degree than with not. In the U.S., at least they’re quite expensive. It’s not clear what the value of that is, since at other places it’s much cheaper and students seem to get similar benefits. So we could discuss that. So there’s always a constant problem of what is relevant, what is out of college, and also half of what you get out of college is not related to the formal education, and it’s related to other things, growing up and socializing, and becoming an adult, and all those other things we list [LAUGHTER] to justify our existence. But the student anxiety and stuff, I’m not surprised, students are anxious about AI, because all of the big proponents of AI are going running around saying they’re going to wipe out all the jobs, and I have never bought into that. I’m, amongst my colleagues, a very prolific user of AI. I use it for all manner of things, and some of those things are quite excellent. I’m a head of department, and I use AI now to get rid of all of those reports and things that I have to write for the university that might have taken a day. They now get done in seconds, and it’s great because no one was ever going to read those reports. They just wanted to see that they existed, and now that I can use AI, they exist, much like students producing assignments, and so I can’t blame them. Students are producing assignments, and there’s producing for professors who, for the most part, don’t want to actually read them. Anyway, nonetheless, [LAUGHTER] the thing that I’ve seen from that is that anything that is enjoyable and similarly meaningful to you, and also involves judgment and taste, these are things that you can’t get from the AI. These AI models, as we wrote in our first book, are prediction machines. They’re good at certain aspects of intelligence, but not others. They’re quite miraculous in the way they’ve evolved over the time I’ve been studying them. No doubt about that. They’re amazing things, but in all of it, I haven’t had any reason to doubt the value and usefulness of what humans do, even though it’s sometimes very difficult to define. It’s come in my own experience, it’s come in all of the data that is generated. As I said, we can’t identify whether people have been losing jobs from AI. Will some people lose jobs from AI? No doubt, because it’s going to reorganize things, that happens, but is there going to be the mass 50% unemployment that somehow engineers in Silicon Valley can imagine, but no one else can. That’s not going to happen. And fundamentally, it’s this: if you believe enough that humans matter, that even with the best AI, humans provide something of value. If you believe that, you can simultaneously believe that AI is going to be great, and AI is going to be great for people. It’s only where you are in a mindset that I think is what drove the Silicon Valley types, is that humans are crappy, stupid, and should be replaced, that you get big use of AI and big displacement of humans. And so, fundamentally, that’s the different assumptions, people who are anxious have to say: “Is there something… what is it that I can do?” Now, I mean, the problem is that, being older folk, we sort of have more opinions about our value [LAUGHTER] as we’ve learned it over time, and university graduates, all they’ve been doing is not getting that, [LAUGHTER] not yet. And so it’s sort of like you would have to take that on faith, but I must admit I’ve seen no evidence to the contrary. That’s my resident belief at the moment, and so I’m on team human.

Rebecca: So pro human, I hear.

Joshua:: And I was perfectly open to the idea that we weren’t. I mean, when it comes down to it, my use of AI has been a whole lot of things that I’ve decided the AI is better than I am. I’ve decided that literally answering student questions, I think it’s better than I am… [LAUGHTER] doing those sorts of things. But ther’re other things, there’s enough of the other stuff, enough of the other stuff that I think it works, and I can spend more time doing that, so even better.

John: And at two in the morning, AI tends to be much better at answering student questions than we do.

Joshua:: Exactly, and the students know that too. The students don’t like going… there’s some students who like seeing professors, students who are really at the top of the class want to see it, but not the seventy on the test. And then the students who are struggling also like to go because they need that extra help. And the beauty of it is, is if you use All Day TA, you don’t have the mass in the middle coming and bothering you, and you can concentrate on the parts where you can actually provide the most value. It’s absolutely true. What do we get most value out of a professor? It’s engaging with really smart students who we think are going to go somewhere, or a struggling student who really was close to giving up, that you sort of explain things to, and they’ve been able to come through. Those are our rewarding parts. That’s what we want to do. We don’t like the students to come in the middle saying, “Will this be on the test? They can ask the AI.

John: And there is a fair amount of research that suggests that first-generation students and students with imposter syndrome are very unlikely to go to their professor to ask questions, because that would reveal that they don’t understand things that well, and they’d rather keep that hidden, and many of them would just give up. So having access to free tutoring service without any judgment 24 hours a day is a valuable service.

Joshua:: I would go beyond that. Well, I’m in one of the elite universities, University of Toronto. The vast majority of tertiary institutions are not that, they are ones where a lot of students struggling financially, they have other jobs, they’re working part time, they’re studying part time, and things like that, and those are the students who drop out. And why do you imagine they drop out? Because they’ve got these assessments, because they have under-resourced teachers as well. So, they have these high-stakes assessments, and it’s basically on the night before, they haven’t been able to learn it, and they can’t get any help. And they said, “Well, I can’t do this anymore.” Now, if just a few of those students, [LAUGHTER] because of All Day TA, can get the answers they need and be able to hump over those moments, they can complete their degree. And so, if we can just move a few percentage points on dropout rates, I will be more than thrilled.

Rebecca: Well, we always wrap up by asking, what’s next?

Joshua:: Well, what’s next? It’s very interesting to think about what’s next, because normally when we talk about what’s next, we’re thinking like in five or 10 years. Now I’m talking about what’s next is maybe later this year… [LAUGHTER] would be the sorts of things we’re thinking of, and I actually no longer have a very clear view of it. I kind of feel, “Oh, it’ll be exciting and new,” and I get excited anytime some new thing comes forward. I think what we’re going to see, what’s next, more is that the AI is going to start coming into the fore in the other parts of our jobs. As I already said, it can help administration, it can help teaching. We kinda know those. I think in research it’s going to be a very interesting development, where we will be confronted with our desire to know what the answer is versus what used to be a complimentary desire to be the one who found it, [LAUGHTER] and so we will have to reckon with that in regard, and everybody will have sort of views to it on that, but it’s not until it actually happens, or it happens to you that you really know how you feel about.

Rebecca: Much to look forward to, that’s for sure.

Joshua:: It’s going to be exciting. [LAUGHTER]

John: It is, and the rate of evolution of AI tools has been phenomenal. I don’t recall seeing anything else develop as rapidly as this.

Joshua:: No, it’s crazy. It is crazy, and I was studying this seven years before it occurred, [LAUGHTER] all the latest stuff, and it was like a surprise, how quickly it happened.

John: And just a few years ago, there were all these reports about people saying AI will be there in the next five years, and that kept getting extended, and all of a sudden it was there.

Joshua:: There it is. We arrived through the Turing test, the whole thing. Amazing.

John: Well, thank you. This was a really interesting conversation, and it’s a really interesting time to experience these things.

Joshua:: Thanks for having me. I’ve enjoyed it greatly.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

450. Collaborative AI Guidance

With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, Kathleen Landy and Joshua Foster join us to discuss a collaborative approach involving student government, administration, and faculty.

Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College.

Show Notes

Transcript

John: With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, we discuss a collaborative approach involving student government, administration, and faculty.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Kathleen Landy and Joshua Foster. Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College. Welcome, Joshua, and welcome back, Kathleen.

Kathleen: Thank you so much. Great to see you again.

Joshua: Thank you for having me here.

John: Thanks to both of you for joining.

Rebecca: Today’s teas are:… Joshua, are you drinking tea with us today?

Joshua: I am. I am having some Japanese Sencha green tea here in my GCC mug.

Rebecca: Amazing. And how about you, Kathleen?

Kathleen: Well, I don’t know if this is like wearing the concert T-shirt to the concert, like the band T-shirt to the concert, but I have a Tea for Teaching mug, and I’m defaulting to Irish breakfast tea.

Rebecca: Clearly one of the cool kids. [LAUGHTER]

Kathleen: That’s right, that’s right. [LAUGHTER]

Rebecca: How about you, John?

John: I have a Twinings English Breakfast tea today.

Rebecca: Oh, nice.

Kathleen: Very lovely.

Rebecca: I think I have… maybe I should verify…

John: Another imperial tea?

Rebecca: I do, [LAUGHTER] I have a Scottish breakfast tea today.

Kathleen: That sounds lovely. I don’t think I’ve ever had a Scottish breakfast tea.

Rebecca: It’s got a little more oomph to it. [LAUGHTER]

Kathleen: Okay.

Rebecca: A little bolder, I feel like.

Kathleen: There you go. Okay, I’ll have to check it out.

John: This is a follow-up to our earlier podcast, where we talked a little bit about program- level AI policies, and you mentioned that you had been working with a member of the Student Government Association to come up with some recommendations for faculty and students. Could you tell us a little bit about the origin of this collaboration on the policy?

Joshua: Sure, I would be happy to do that. As a liaison between students and faculty on campus, students often approach me with various concerns, and one concern that came up multiple times during the fall semester was students having their legitimate work flagged as partially or even completely AI-generated. This initially prompted a discussion in one of my classes, at which time I learned that many students actually avoid the use of, or even the mention of, AI in order to avoid the potential of a suspected academic integrity violation, but I also learned how frequently academic integrity violations related to AI do occur. So I realized this was a topic that necessitated a more direct and open conversation involving both the faculty and the student body, and that is what led Kathleen and I to develop these guidelines as a way to guide those conversations,

Kathleen: I’m happy to add to that, Joshua, because this came up organically in one of our routine meetings. So, Joshua and I, as part of his role at the SGA (Student Government Association), and my role in Academic Affairs, we meet every three weeks, and so this came up in one of our conversations, and it was so timely, and it was totally a function of happenstance that at the time I had been having conversations with academic program directors and multiple faculty about the necessity of having course-level AI policies, and so it was very serendipitous, almost, that there was this student concern that Joshua articulated, and I had been exploring this question with faculty, and it seemed like a really lovely opportunity to bring everyone together and explore this topic.,

Joshua: And I had actually heard at the time members of the faculty discussing that they did have concerns as well, and so it sounds like the people who we serve both had concerns, and so we could kind of bring those concerns together, merge them together in the development of these guidelines.

Rebecca: Can you talk a little bit about how you facilitated the creation of the guidelines? Who participated in developing this initial round and this initial draft?

Kathleen: So, Joshua and I, we opened a shared Word document, and we just got some thoughts together, and we thought initially it was a no-brainer to have this set of guidelines have two parts, one that was faculty facing and one that was student facing, and we knew that they would kind of run parallel to each other and reinforce each other, but we started with a little bit of a brain dump, and we just kind of mapped what we thought were good ideas and good starting points and points of entry for the conversations about AI and academic integrity, and we landed on, I guess, we would call a sacrificial draft that we thought we could both bring to our respective colleagues and get more input. So it was with real humility and curiosity that we did a little road tour. We hit the ground running with this document that we’ve shared with you as well, and I personally brought it to colleagues in Academic Senate. We have a standards committee, and so I brought it to the chairs of that committee and shared it with our academic program directors, and we went from there. We solicited feedback, and it was great feedback that we got. And Joshua, I don’t know if you want to speak to folks that you brought it to. I don’t know if it was the SGA cabinet.

Joshua: Yes, so I initially didn’t bring it to too many students. What I did rather was I sort of took what the guidelines were without actually taking the draft, and I took that and I talked to different stakeholders about their concerns, and I brought up where different points that were covered in those draft guidelines and got their feedback on each point that I thought was relevant to them, and then I would take that and contribute it back in, and then I would also look at what some of the different faculty members are who I talk with and get their feedback and kind of combine that feedback with the feedback from the students and bring that back to Kathleen and to her colleagues, and I think that was really a good collaborative way that we took this, to include both levels of feedback, but not have the students edit the guidelines, but have them indirectly contribute through my contribution, I think, allowed them to really be able to express their thoughts and their concerns more clearly.

John: We have a student committee on learning and teaching here, and this came up in all of our discussions with that group, and the same sort of concerns that you mentioned, Joshua, appeared, that some students were really upset that they were accused falsely of engaging in academic integrity violations, and also others were concerned about whether it was safe to use it, because they’d like to be able to develop skills in this, because it may be relevant for their careers. It’s a topic that I think applies to a lot of people. Could you give us an overview of the guidelines, both for faculty and for students in this draft document?

Kathleen: The guidelines are surprisingly simple, but we strove for them to be very clear. So, the guidelines include having a clear AI policy that’s articulated in every syllabus, even though that’s a soft expectation, we wanted to make it a little bit more concrete and a little bit more pronounced, and so we wanted to encourage faculty to have a clear policy that includes specific examples of acceptable and less acceptable uses of generative AI, and a disclaimer or an explanation about any possible AI detection platforms that might be in use for the course. So that’s kind of an equity issue. It’s a social justice, it’s a transparency related issue that if there are detection platforms in play, students should be aware of that. That should be a known element in any learning environment, and importantly, this same AI policy should have a clear list of consequences for initial and possibly for subsequent violations of the class policy. So we wanted to start with having something concrete in place, something that’s published for students and faculty alike to refer to. And then the second part of the guidelines for faculty involve posting open and candid conversations at the beginning of any course or assignment, something that came up in the previous discussion that we had was this idea that in sympathy for the schedules that students have and the potential for them having five different course level policies to keep track of, it’s really not realistic for any given course level policy to be expressed once and then never mentioned again, or to be published on page eight of the syllabus, and then never revisited again. So, the idea of not only having a clear AI policy, but revisiting that and explaining it and communicating it frequently and clearly, that’s a kindness that we can show to students. It’s a good practice. So, this idea of hosting ongoing and candid conversations within the class, especially at the beginning of a course or the beginning of an assignment, that’s really critical. And then finally, the guidelines involve one or two strategies for fostering greater student buy-in, for lack of a better term, or a greater sense of accountability among students for the policy, so things like having students sign an integrity statement, a commitment to acting within the confines of an academic integrity policy, having students sign an integrity statement, or including a question about AI use and implications within the context of a syllabus quiz, or something that’s a little bit more interactive. We thought that would allow for a structuring of engagement with the policy, to not just have it live somewhere, but to have students actually engage with the policy.

Rebecca: It seems really important not to just have something exist on paper or some virtual ether, right? [LAUGHTER] It’s like have people actually interact with it.

Kathleen: That’s right, especially because for many students, and this will change over time, of course, but they might not be accustomed to having to check with the course-level policy. These are relatively new and emerging caveats or elements of a syllabus. So, how could we not redirect students to say, “Hey, remember we have this policy, we have this expectation?” It’s really a kindness, I think, to students. And then Joshua, I’m happy to turn it to you for the student side.

Joshua: Yeah, thank you, Kathleen. So, the guidelines for students really do mirror and complement the guidelines for faculty, and I want to start by pointing out that, similarly to the guidelines for faculty, the guidelines for students do not dictate when the use of AI is appropriate or not appropriate. Instead, the goal of the guidelines is to help students learn how to communicate with their instructors to be sure that the course policies are clearly understood prior to any misunderstandings happening. So, with that in mind, the guidelines mentioned topics like requesting a course policy on AI, if one is not provided, and asking questions of the professor or the instructor to understand that policy. They encourage attending office hours, sending a professor an email with questions, or even asking a question in class, so that other students can benefit. Maybe other students have that same question, and maybe they don’t want to ask for the reasons that we discussed earlier. So, the guidelines, in addition to this, include steps for how a student could start a conversation with a faculty member if that student’s work has been flagged as having been AI generated, regardless of whether or not that work was correctly flagged as having been AI generated. The guidelines stress being respectful and aware that misunderstandings do occur, but they also encourage students to advocate for themselves when necessary, and I personally believe that student self-advocacy is a very important skill for students to learn early on during their college experience. The last major focus of the guidelines for students is the importance of building strong and collegial relationships with faculty members, which can be difficult or is something that is even avoided by some students, especially younger or first- nyear college students who are still adjusting to leaving a high school environment, and I believe that this is actually one of the most important parts of the guidelines.

Rebecca: I think I hear some really common themes around transparency and relationship building.

Kathleen: There’s no doubt. I was speaking with Joshua. We had a debrief of an event that we co-facilitated this past Friday, and I was struck by the extent to which the following statement is true. Almost all of my recommendations or strategies or suggestions for how to handle issues involving academic integrity and the use of AI, in particular, almost all of my strategies involve the human elements in a learning environment, and they’re all relational. It all comes down to either communicate… not all, but it often comes down to communication and relationships rather than the tech itself, and so I think that’s a really interesting relationship: that the tech challenges that we encounter often have very human or relationship-centered solutions. That’s been my experience to date. So I agree with you, Rebecca. There are trends in the solutions that we keep coming to.

John: I believe you also did some sessions with faculty, staff, and students about this. Could you tell us a little bit about that, and what sort of reactions were received?

Kathleen: So, we’ve had one session, and we hope it will be the first of many, and I have to tell you, it was awesome. We were delighted, we had over 50 participants on a Friday afternoon, no less, which is no small feat. We were very excited by the turnout, and we tried to be very mindful of how we engaged folks from the beginning. We wanted to set the right tone, and so we started with a little bit of a series of short interactive polls to let folks know that we’re all coming at this from a point of intellectual curiosity, and we wanted to build a little bit of a sense of community around this topic, and so the very simple question that we started with, it was a simple qualitative poll everywhere to just text your answer in, and we asked a very simple question. We asked what word best captures how you feel about generative AI in your classes, very simple, and we knew we could get anything, and so we gave everyone simple directions. Everyone had the option to pass if they wanted to, but everyone had the option to text a single word, and we generated a word cloud right there. I can send you, if you like, a picture of this word cloud. I’m looking right at it. I have to say, it was really lovely to see the extent to which students could see the faculty’s human responses to this question, and faculty could see students’ human responses to this question, and so the responses were anonymous, but the word cloud was really thought-provoking, and it was populated with words like conflicted, concerned, curious, complicated, stressful, uneasy, overwhelmed. I think it set a tone where everyone could kind of check their egos at the door, or check their personas of being an academic, or a scholar, or an honor student, what have you, fill in the blank. Everyone could check that at the door, and we all kind of became students of this together, and it was, I thought, surprisingly effective at just setting a nice tone of inquiry for the remainder of the session. The session was about an hour long, and so we had interactive polls and a bit of an overview of the guidelines. We distributed a physical copy, like a paper copy of these brief guidelines in the room, and we posted them for the hybrid. It was a hybrid session, that’s worth noting. We posted them for the online participants, and it generated rich discussion. We got some great suggestions from both instructors and staff alike. We had students there who weighed in. We had folks, both students and instructors, share experiences, times where they wish they had these guidelines. Some expressed a concern about having or desiring more robust guidelines, like they wanted more of the same. And I’ll maybe stop there, because Joshua, I’m sure you want to add to that, but I was surprised at how eager folks were, and I was gratified by the enthusiasm that the group expressed.

Joshua: I was very excited as well, after having been at that session, and I joked with Kathleen a little bit at our follow-up that I wish we had had two or three more hours to go on, because I think that we certainly could have, but I think that the thing I was most pleased about was that we had a good amount of both faculty and students in attendance, and that they could see that, in most cases, they are in agreement with each other on these issues, or at least there is some mutual common ground between our students and our faculty, and I think that was the first thing that was really important for both of those groups to see, and then the second thing was that a lot of students came to me afterwards and said that they were really excited to have learned about how AI is becoming a common issue and how interested people are in it. As an accounting student, I have found that within my classes I’ve had a lot of people who are both interested in AI and who are both very hesitant about it, and ironically, if I had to make a really, really wide swoop with a brush to say who is more pro the implementation of AI, at least in the specific classes that I have been in, it’s the faculty, not the students, and so I think that the students being able to see that was really helpful, that that may be what we students imagine that the faculty are thinking about, and what the faculty maybe imagine that the students are thinking about maybe those are being too heavily stereotyped, maybe those aren’t really, really how we are seeing this, and so I think that was really good to have experienced, and I just think it was the meeting was very well received. There was obviously a lot of desire for follow up to it, for more conversations, but I think that the conversation that we did have was a very good starting point, and it helped to really demonstrate that where the direction that we’re going is really the direction that we should be going.

Kathleen: Thanks, Joshua. The only thing I would add, because I don’t want to paint this picture that it’s all rainbows, and we all sang together at the end of this session. I did have a couple of faculty members, and I was super grateful that they attended. They didn’t express frustration, per se, but their response was more like, “yeah, and…” like they were just ready, and I’m loosely paraphrasing, but this is close to the exact language. They said, “this is good, this is fine, Kathleen, but this isn’t the real work. The real work has to come in the articulation of departmental policies, and what’s that look like?” And I guess my response to that was, “well, yeah, of course, that’s the departmental next step, and that’s not for Academic Affairs or SGA, for that matter, to dictate to departments what their departmental or course level policy should be,” and so I think the workshop work is what should follow. So, there was a little bit of maybe frustration, that might be too charged of a word, but there was a little bit of feedback that was, “yeah, and okay, like, get to it, get to the next step,” which indirectly I interpret as affirming feedback that we’re on the right track, that folks are eager for guidance or at least engagement on this topic.

Joshua: And I believe that the same is true for students. I think that students really want that greater level of course-level policy or program-level policy, but I think that we’re not to that point yet, and so at this point a student came to me today and asked me. My response would be, “think about communication, because that’s what we are able to do now. That’s what we can do now.” Those other more clear things will come, but now we have to focus on what we can realistically do together.

Rebecca: It seems like a really valuable opportunity to demystify what people are thinking about, and that’s a really important first step into moving towards policy, or something that’s more formalized. You’ve certainly presented this as a working draft, and some revisions, and things like that. What do you see as the next steps to maybe formalizing this work at an institutional level, and then obviously there’s some work that would need to follow at the departmental level.

Kathleen:I think there’s a few next steps, and one has to deal with instructional practice and faculty development, and the other has to deal with steps to take leading towards policy articulation. I think the immediate next step involves endorsement, to take the feedback, the great feedback we’ve been given, and refine these guidelines, share them again, probably have a few more rounds to get some greater input, but then go through the formal channels of shared governance to have endorsements, formal endorsements, so that we can say “here are some endorsed guidelines, and those endorsements are coming from Academic Senate, from our local union, from Cabinet, from SGA, from program chairs.” We can get a lot of structures on board to kind of promote these and broadly share them. On the instructional side of things, and this is connected closely, of course, to faculty development, I think we can generate conversations about all the things that we’ve been discussing and all the things that are focused on in these guidelines. One of the polls that we asked, one of the questions we asked this group, and we had some questions restricted to students and some restricted to faculty in this one- hour session that we had, and one of the faculty questions was very simple, yes or no question: “Do you presently have a clear published AI policy for your courses?” And 57% of faculty said yes, and clearly 43%, my follow-up question is, “why not?” Let’s get that number a little higher of those that do have a clear policy, and I think that’s really good for that 43% of faculty to see, like, “oh, wow, 57% of my colleagues present at this session have something in place. Let’s talk about what that is, and maybe I should have that,” and I think that those collegial conversations, and then doing the work of adopting, crafting your own AI policy, and publishing that in the context of whether it’s a Brightspace shell or a syllabus, likely multiple places, that’s part of what’s involved in the next steps from my perspective.

John:I know in our campus there’s been a fairly wide range of perceptions concerning AI detection software. Some faculty trust them very strongly, even when the evidence suggests otherwise. Other faculty very reluctant to rely on them at all, except maybe as a signal that they may want to contact a student to talk about the issue. I was really pleased to see that your guidelines explicitly stated that AI detection software is fallible. Could you talk a little bit about that, and whether people in general have accepted that message there?

Kathleen: I’d like to think they have accepted it, I think, and I also think it’s a case that folks are still exploring what detection platforms do and how they operate, and how they might be of use to them in their own classes. So, I do think it’s variable in terms of comfort with and knowledge of these platforms, but I think there’s a lot of false positives, there’s a lot of false detections, and I think disclaiming that and modeling how to have the conversations about that, that’s something that we hoped the guidelines would get into. I think dragging that issue into the light is part of the intent of these sessions. I think folks are pretty receptive to it, because there might even be a self-consciousness about what we don’t yet know. So once again, I do think there’s a dotted line to faculty development there, and I think we have to model the questions that we ask as we analyze the efficacy of these platforms. So I think it’s time for candor, and I think it’s a time for inquiry, and just the humble recognition that we have to talk about what we need, what we want, and how good these platforms are. How efficacious are they?

Joshua: And something that one of the faculty members, who was present at our session, brought up, and something that I really appreciated from the student perspective is the acknowledgement that because these AI detection platforms are not perfect, they can actually serve to erode the level of trust that has been built between students and faculty members. And so it’s unfortunate that they do that, but that’s one thing that will have to be accepted when using an AI detection platform, and so, as a student, I appreciate that faculty are acknowledging that reality, and so that means to me that they will be more sympathetic and more understanding, because obviously in some cases those tools, those positives, are accurate, but in other cases they may be a false positive. And so just having that awareness and the fact that faculty members are acknowledging that level of student trust that may be called into question, I think is really good for them to be able to recognize.

Kathleen: Well, it’s great that you say that, Joshua, and I do want to respond to it. It prompts me to underscore this. I think the disclaimers regarding the effectiveness of AI detection platforms, it drives the conversation to prevention rather than detection, and I think that’s a huge part of not only instructional and assessment practices, but also the learning environment, and it goes back to the relationships and the trust that you mentioned, Joshua. So, I think that’s a huge part of it, the shift from detection to prevention, that’s a nice point of entry for me, and the conversations with faculty.

Joshua: And I think that that level of trust between students and faculty may be the first step. probably is the first step, in that prevention.

Kathleen: That’s right. Agreed.

Rebecca: So, you’ve talked a little bit about wanting, or even departments, maybe, being eager for like the next steps. Can you talk a little bit about what movement there has been in terms of the course-level or program-level policies on AI, or what you maybe expect to see as we enter into the fall?

Kathleen: We’ve had a few departments, a few programs, or program clusters, I should say. We’ve had a few folks self-organize, and they’ve proven to be self-starters in this domain, and they’re really gathering and just meeting routinely to articulate policies and to explore what other institutions have to craft that together, and that’s inspiring. That’s really great to see. Our business department comes to mind immediately. They’re doing great work, and there are others that I think are curious, and folks are kind of coming to those realizations of, hey, we need to huddle up and figure this out. I think folks are coming to that conclusion on slightly different yet overlapping timelines. So, I think there’s a role for my role and Academic Affairs Administration to prompt that collaboration a bit more intentionally, whether it’s the articulation of a strategic objective relative to having a target, the number of academic programs to articulate as policy, regardless of what it is, I think we can prompt that a bit with the strategic planning that we do, with the faculty development planning that we do, and I think to a certain extent we’ve done that, but I think we can turn that up a little bit. I think we can be a bit more intentional about it, but it’s happening just at different degrees and varying by program.

John: We talked a little bit about AI detection and how perhaps that could be used to nudge people in ways to use AI within the guidelines of the course, but are there other strategies that faculty are using to encourage students to use AI ethically rather than to shortcut their learning?

Kathleen: It’s a great question. I think, and this will vary by discipline, so I don’t want to speak in too broad of terms here, but I might end up doing so. I think to the extent that AI can be leveraged to enhance or scaffold or facilitate a process, I think that’s always a good thing. And I think if the use of AI can underscore the importance of the integrity or ethics-related outcomes for a course or a discipline, that’s great. I think there’s incredible potential for use cases, in terms of here’s how to use, here’s how not to use, here’s how this could go really well, or here’s the quickest way to work yourself out of a position, a passion, a process, whatever is appropriate there. So, I think there are examples, but they’re highly discipline-specific, and I know that that’s not the most robust answer, but I do see evidence across disciplines that folks are incorporating AI in really transparent ways that scaffold learning rather than inhibit it.

John: Joshua, do you want to take a stab at that in terms of things that you’ve seen where faculty are encouraging students to use AI in productive ways,

Rebecca: …or even how students are doing that on their own. [LAUGHTER]

Kathleen: Yeah, yeah.

Joshua: True. Yes, I have seen it. So, Kathleen shared briefly earlier that one of our questions was how many faculty members have an AI course policy, but another one of our questions that was restricted to students was, “How many of you, how many students used AI to complete their coursework?” …and that’s not meant to be AI that’s sanctioned, that is meant to be using AI in appropriate ways, assumingly, and that answer was 89%. 89% answered yes. And this is an interesting thing, because in my personal experience, the people I’ve talked to, I would say that that percentage is probably lower, maybe more around 60% but I found that answer quite interesting, and from the students that I’ve talked to, even the faculty that I’ve talked to, or other tutors, people who work with students, a lot of students are using AI in ways, for example, they may ask for help coming up with a topic idea, or maybe to brainstorm a topic, or maybe they’re brainstorming a first paragraph, or a direction that they could go within an assignment, or maybe they’re comparing scenario, and I think that those things, those getting started is one way that I’ve seen a lot of my peers using AI. Another thing that I have seen is where they use it for research. Now that will get into another topic, which is the reliability of that research, and so that’s something that would be important to talk about, but I do see students using it. I also see a lot of students using it, and they have very good reasons for that. But I think that it is something that is becoming more in use. I’ve also seen faculty members, and not in particular any faculty members that I have personally had a class with, but who I have talked to, or have actually had assignments in which they ask students to use an AI tool to begin an assignment. For example, there was an assignment in one class that was used for the creation of a resume, and one of the parts of the assignment was to take the resume and run it through an AI tool and have it give you suggestions and feedback on where you could make some words stronger, and so I think that those are ways that AI could be used in ways that are actually helpful or productive for learning.

Kathleen: If I can add one additional comment, and Josh, your comments prompted this thought. One of the more valuable pieces of feedback that we received on the guidelines was our oversight or negligence in not mentioning the library and the learning center in the guidelines, and so to your question, John, I think it’s a really good opportunity, a great idea, in fact, to leverage the use of AI to facilitate engagement with the other support services and the library on our respective campuses. So I think to build that kind of analytical info literacy related analysis of AI generated content as part of a process for a course-based assignment, I think that kind of a no brainer that we have this fodder, this generated content that we can use to engage with tutors and librarians, and of course, faculty, TAs, fellow students to kind of engage in the analysis, the fact finding, the verification of information, all of the info lit and AI enhanced info lit competencies that we’re looking to develop. I think there’s opportunities there as well.

Joshua: The other thing that I have had some students tell me is that they turn to AI because they genuinely need help thinking through something, or they struggle with a certain topic. So, for example, I know a couple of students who use grammarly because they struggle with their grammar, and I think that one thing that we are encouraging faculty to do with these guidelines, and thus also encouraging students to do these guidelines, is to have faculty make clear when these tools are allowed, and have students understand when they’re not allowed, because sometimes there’s a fine line between when is something like Grammarly, when is that allowed, or when is it allowed to use Chat GPT to think through your thought versus when is that prohibited, and I think that having those guidelines is essential, including those in your course policy is essential. The other thing that Kathleen was mentioning with the library and with the tutoring, is those are alternatives, so when AI is not permitted, there are other things that can help students. They aren’t just left on their own. There are tutors. Tutors can help you think through your ideas. There are librarians. Librarians can help find information. They’re very good at information literacy and finding those information. They can also help to verify research, and maybe if research was done via an AI tool, maybe that information needs to be verified from another more academic source. And so librarians are very helpful with that as well.

Kathleen: Now, Joshua, I kind of want to jump in, and I want to challenge one phrase that you mentioned, and I’m doing so as your thought partner in this, so you’ll appreciate the sentiments and the collegial approach here, but I would not want to take away from your comments to be that librarians and the Learning Center are alternatives to generative AI. I think that’s a dangerous framing, and I know that you don’t mean to say that they’re like alternatives to AI, but I think that distinguishing between the tool and the resource that many AI platforms offer, distinguishing between that and the human-centered expert librarian and the qualified and well-situated, well-resourced tutor, they can help us use this tool to our advantage and help us navigate the best uses, for that’s often a distinction. I wouldn’t want any of our librarian and tutor colleagues to say, “Wait a minute,” of course, if you don’t mind me clarifying what I think I know you mean by that. I think that’s important.

Joshua: I don’t mind, and I think that there are going to be times, though, when students choose not to use AI, and when they choose that they would rather have that more experienced human they can go and talk to, and I think that’s what I was trying to say, is that you don’t have to use AI to succeed in your learning. There was so long that this wasn’t available, and so I think that it’s coming available, and it’s a great tool, but it’s not the be all end all. It’s becoming a great tool, but you still need to be able to learn as a student and be able to talk to people, and I think that is really, really important.

Kathleen: Yeah, and I think going back to the spirit of our session, we have to figure out what this means together, what AI use means in our respective disciplines and our respective classes, which is absolutely how we approached this set of conversations. So, I think it may be the be all end all, it may not be. I think we got to figure that out together, which is the spirit of these guidelines, certainly, and the conversations around them. It’s important.

Rebecca: And pointing students and faculty to all the available resources and the roles that they can play in learning is always really important.

Kathleen: Yes.

Rebecca: So we always wrap up by asking: “what’s next?”

Kathleen: In terms of what’s next. I think it’s a clear next step for us to continue bringing faculty and students together, that was so elucidating. It was really great for that exchange of perspectives and for that shared exploration of this really important and impactful topic. But I think we got to keep at it. We got to keep bringing people together in thoughtful ways. That’s what’s next from my perspective.

Joshua: Yeah, I agree with that, and I think whether it’s bringing students and faculty together again or talking among students, talking among faculty, and then coming back together, regardless of how it’s done, just continuing that conversation is really the next step. You know, I agree, Kathleen, having that continued conversation is just really important.

John: We know AI has been somewhat disruptive for students and for faculty, and it’s nice to see this sort of collaboration, and it’s also something that can take place really nicely at the class level as well as at the institutional level. And it sounds like you are making some really good progress there at GCC.

Kathleen: We’re really trying. Thanks so much to you both. Thank you.

Joshua: Thank you. It’s been wonderful to be here to talk with him today.

Rebecca: Yeah, it’s great to have both of your perspectives, because I think the model you’re sharing around the topic of AI could be applied to many other challenging topics that challenge both faculty and students, and bringing those two groups together to work through those challenges and establish guidelines and other strategies is really helpful.

Kathleen: Thank you. I think so too. One of the goals is to model. We’re trying to model as a campus community for each other, whether you’re a faculty, staff, student. We’re trying to model how to problem solve and how to figure things out together and do so in a way that’s ethically sound and mission-centered, and focused on learning. So, we’re trying, we’re hopeful, but we’ll get through it together.

John: And when both students and faculty are involved in making these policies and have a say, there’s going to be a lot more buy-in to the policies than if they were just imposed.

Kathleen: That’s right. We agree.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

448. AI-Aware Teaching

While there is substantial debate over the appropriate role of generative AI in higher education, one area of agreement is that AI cannot be ignored. In this episode, Annette Vee, Marc Watkins, and Derek Bruff join us to discuss what faculty need to know to be AI-aware in their teaching.

Annette is an Associate Professor of English at the University of Pittsburgh where she has been actively involved with AI initiatives. She is the author of Coding Literacy: How Computer Programming is Changing Writing and is co-editor of TextGenEd: Teaching with Text Generation Technologies. Marc is a Lecturer in Writing and Rhetoric and an Assistant Director of Academic Innovation at the University of Mississippi, where he directs the AI Institute for Teachers. Derek Bruff is an Associate Director at the University of Virginia’s Center for Teaching Excellence, where he supports faculty in integrating generative AI in their teaching. He is the author of Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching and Teaching with Classroom Response Systems: Creating Active Learning Environments. He is the host and producer of the Intentional Teaching podcast. Annette, Marc, and Derek frequently serve as keynote speakers at academic conferences and write frequently about AI and higher education on their blogs. Annette, Marc, and Derek are the co-authors of The Norton Guide to AI-Aware Teaching.

Show Notes

Transcript

John: While there is substantial debate over the appropriate role of generative AI in higher education, one area of agreement is that AI cannot be ignored. In this episode, we discuss what faculty need to know to be AI-aware in their teaching.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Annette Vee, Marc Watkins, and Derek Bruff. Annette is an Associate Professor of English at the University of Pittsburgh where she has been actively involved with AI initiatives. She is the author of Coding Literacy: How Computer Programming is Changing Writing and is co-editor of TextGenEd: Teaching with Text Generation Technologies. Marc is a Lecturer in Writing and Rhetoric and an Assistant Director of Academic Innovation at the University of Mississippi, where he directs the AI Institute for Teachers. Derek Bruff is an Associate Director at the University of Virginia’s Center for Teaching Excellence, where he supports faculty in integrating generative AI in their teaching. He is the author of Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching and Teaching with Classroom Response Systems: Creating Active Learning Environments. He is the host and producer of the Intentional Teaching podcast. Annette, Marc, and Derek frequently serve as keynote speakers at academic conferences and write frequently about AI and higher education on their blogs (which we’ll link to in the show notes). Our guests are the co-authors of The Norton Guide to AI-Aware Teaching. Welcome, Annette, and welcome back, Marc and Derek.

Annette: Thank you.

Derek: Thanks for having us.

Marc: Yeah, thank you, guys.

John: Thank you for joining us. It’s great to talk to all of you today. Today’s teas are:… Annette, what is your tea? [LAUGHTER]

Annette: I have to confess that we are out of tea season. For me, it is warm, and I’m on my third floor office, where it is not air conditioned, so I’m just drinking water. But I like tea. I like herbal tea… when it’s cold outside.

John: And Marc, are you drinking tea?

Marc: I wish I was. Our tea kettle in the office is broken today, so I have a very anti-tea Celsius that is a Kiwi Guava flavor, so it has caffeine and its liquid, [LAUGHTER] that’s the sort of the most related thing you can have with tea. But yeah, unfortunately, I’m like Annette, I don’t have access to good tea right now.

John: And Derek?

Derek: Well, I’m 0 for three. I am drinking coffee, [LAUGHTER] but I am drinking it out of my tea for teaching coffee mug.[LAUGHTER]

Rebecca: I think that might be blasphemous. [LAUGHTER]

Derek: I think I should get some points for having the on-brain coffee mug today.

John: Yes.

Rebecca: I think it’s good. It’s good.

Derek: Thank you, John, for gifting me a coffee mug when I saw you a couple of years ago.

Rebecca: It’s a tea mug. [LAUGHTER]

Derek: Not right now.

John: it’s a tea for teaching mug, and I think it was a coffee mug last time too.[LAUGHTER]

Rebecca: Yeah.

Annette: Get into some Wickenstinian conversations about definitions here. [LAUGHTER]

John: Well, I went to the POD conference with, I think, 48 tea for teaching mugs, and I think I came back with about two or three of them, because the whole place was filled with former guests from the podcast.

Marc: That’s great.

Annette: That’s lovely.

John: It was nice to be able to distribute them. And Rebecca, your tea?

Rebecca: I have British breakfast, nice and piping hot.

John: I have darjeeling tea today, which I picked up at the tea caddy at Epcot at the OLC conference last fall.

Rebecca: So we’ve invited me here today to discuss the Norton guide to AI-aware teaching, apparently, and not tea. [LAUGHTER]

Annette: It’s good because the conversation’s over otherwise. [LAUGHTER]

Rebecca: So, can you tell us a little bit about how this project came about?

Annette: Sure, I can start with that. So, this project was kind of a brainchild of Norton themselves, and I had been working with Erica Wnek, who is one of their fabulous editors there, and who works with composition, in particular. I’d been doing the Everyone Is an Author book. They wanted to add a chapter on AI for their next edition, which just came out, I think, in the fall, and from there we started talking about what else Norton might be able to do to support teachers with AI, and I started writing for a Substack, a newsletter that they launched, and worked with Erica primarily on that. It was the first time that Norton had done a Substack like that, and it ended up being reasonably popular, and then there seemed to be some demand for further guidance about AI teaching, but in particular teaching in the midst of AI that wasn’t just, “Hey, let’s lean into AI or here’s how we kind of route around AI,” but something a little bit more measured that was kind of the voice that I had been using in that blog, and so Erica asked me, “What would it look like if we did some sort of guide like that?” At first, I thought, “Well, there’s so many guides out there,” but ultimately I was interested, and even better was that they were really interested in recruiting some excellent co-authors, and so from there they brought on Marc and Derek, and so I can pass that off to them, which I think has really been wonderful for the project.

Marc: It has indeed. I really have learned a lot from both of you writing this, and it was kind of a sprint to go through this, and I think as we were writing it, we were both going through our own sort of basic just feelings about AI and seeing how other people were dealing with it too. We’ve been there for the. Last three and a half, four years, and asking ourselves some very deep questions about our own work with this technology, too, where this is coming to intersect, and trying to help faculty tred a path forward.

Derek: I’ll add that, as Rebecca mentioned, I’ve written a couple of other books on teaching with technology, and I think I have a writing voice that works well with Annette and with Marc, and with the goal of this project, a mix of what does the research tell us, to the extent that it tells us anything, what are the big ideas we need to be grappling with, and what does this look like in practice? I think that’s one of the things I really like about this book, is there’s lots of examples of how faculty are adapting and responding to AI in very concrete ways. That’s what I love to do, is to talk to faculty about what other faculty are doing, and help them see how they can adapt that to their own teaching. Yeah, so happy to be a part. And also, I would say I haven’t co-written before this project.

Rebecca: I didn’t know that, Derek. You reveal that now. [LAUGHTER]

Derek: I was a little skeptical. I kind of like to be in my own space when I’m writing, but this has been a really, really enjoyable process of drafting and getting feedback and revising and commenting and figuring out the structure together. I like the co-writing a lot more than I expected I would.

Annette: That’s really nice to hear. I want to say, too, that Norton brought us together, which I think is wonderful, because they just had like great vision of how we could each bring different expertise, which I think was like absolutely true. But the other thing is that I knew Derek and Marc’s work before this, and so I was really, really glad to be able to collaborate with them, and I do think, like BTS or other bands that the studio brings together, sometimes there’s just synergy, and it works. [LAUGHTER]

John: I really do like the title of AI-aware teaching, which I think fits nicely with Annette’s comment that it was somewhat in the middle, not at one of the poles, as many of the other books and approaches tend to be. Before we start talking about AI and teaching with AI, could you explain what you meant by AI-aware teaching?

Annette: So, I can start with that one. As Marc said, we were all in the process of writing this book, confronting our own feelings about AI in complex ways, and I think all of us had been wrestling with this, of course, before the book, because we’d all been working in this space for years, but the fact that I am absolutely an AI user, I find it really interesting. My work is in writing technologies, it’s the most exciting writing technology, certainly in my lifetime. So, there’s all these really interesting feelings, and then as a teacher, there’s these other feelings about students and shortcutting and the future of higher ed, and all these things, but the main thing is, as a technology, I think it’s just really important to be aware of what it is and what’s out there, and what that means is, for AI-aware teaching, it means understanding what the technology is and does, and not, you know, what it did three years ago, but what it’s doing now, and what are the possible future trajectories for it, and that includes all sorts of things about environmental impacts and politics, as well as the technology, and then also what students are actually doing with it, not what we would like them to be doing with it, or what they say they’re doing with it, but what they’re actually doing with it, so that involves a little bit of research and listening, and then figuring out what it is that we want students to get out of our classes, rethinking our learning goals, things like that, and then kind of recalibrating, given the current landscape, how we get to those learning goals that we have had, or revisiting those learning goals, maybe they need to be revised, and that is, I guess, my kind of sense of AI-aware teaching.

Derek: And I’ll add that I feel like there is a lot of discourse in higher ed that seems to indicate there are two camps: either you are refusing to use AI or resisting it, or you’re all in and you’re doing all these things with it, and I like the AI-aware framing, which is to say those could both be reasonable choices, but we need to understand what we’re dealing with, both the technologies and our students and our own teaching goals, and then make informed intentional choices. And I would argue that most faculty, once they’ve gone through that process, land on some mix of resisting AI in some parts of their teaching and engaging with AI in other parts of their teaching, and I think that’s okay, and I think we have a lot to offer in the book for faculty who are any mix of those two poles in how they actually practice AI-aware teaching.

Marc: Absolutely. I find myself sometimes in the middle of an assignment, designing assignment, being drawn into, let’s use AI, this aspect, let’s absolutely resist, refuse using AI in these different aspects, and then thinking about, from the student experiences, how they’re going to go through those same types of things to confront it. I think Annette’s point too is very apt for a moment too. The interface behind ChatGPT has not changed, but the models have underneath it, and the expectations I have from a lot of faculty is that they’ve tested these systems back in 2022/3, or 24 that they have a singular experience using them, and it was not very good or productive. Now things have changed pretty radically. These are not just language models, they’re, in theory, world models gainng the simulations of these different types of experiences, and so we have to be aware of what that is going to do for ourselves and for our students, and there are some, I think, very good benefits that can come out of this, but also be very cautious about it too, just trying to find that balanced pathway.

Rebecca: Can you talk a little bit about the structure of the book? There’s a couple parts to it. Can you reveal the underlying structure?

Derek: Yeah, I’ll take a stab at that. The book has two parts, and like these books, you can read any chapter in any order you want. I never do that. That’s not who I am. I read things from front to back. [LAUGHTER]

Annette: You’re revealing a lot of things today about yourself, Derek. [LAUGHTER]

Derek: People tell me they read books in other ways. [LAUGHTER]

Rebecca: All these exclusives…

Derek: But, I do think part one is meant to be moved through somewhat linearly, and that it walks you through the AI-aware teaching idea, so that we’ve have a chapter on what AI is and what it does. We have a chapter on how students are using AI, as best we can tell from the research and the surveys and the focus groups that have been done. And so the goal of part one is to help you do those aspects of AI-aware teaching, and then hopefully land on an approach to AI that really fits your course and your goals, and to start to shape how you might communicate that approach to your students through policies or guidance. Part two is very much more a random access part. We have a chapter on in-class activities that are AI aware and out of class activities that are AI aware. We have a chapter on AI and reading, a chapter on AI and writing, a chapter on AI for tutoring, and so it’s much more about where do you see AI showing up in the particular pieces of your courses and your teaching, and what ideas can you get to help you make again more informed intentional choices in those particular aspects. We also have a chapter on AI and academic integrity, because that’s kind of the elephant in the room. [LAUGHTER]

Annette: …that’s the first thing everybody asks us at workshops.

Derek: So we put it last.

Marc: It is, yeah. yeah, yeah. You got to get through the book if you want to get to the last chapter.

Derek: Well, I do at least.

Annette: Yeah, yeah.

Rebecca: I like how it was almost an afterthought there, In the way that you’re presenting it, [LAUGHTER]like it really was an elephant in the room.

Derek: Yeah, because I feel like it’s where so many conversations start, but I don’t know that those conversations are always super productive. I think it’s much more productive to get more grounding in who our students are, what the tools are doing, what our goals are, and then once we’ve got some of that infrastructure, that kind of mental infrastructure in place, we’re in a much better position to make good choices about how we respond to academic integrity concerns.

Marc: Imagine that, approaching this from not an emotional standpoint, but one that’s actually based on evidence of those types of things. I kid, of course. I mean, look, a lot of people are at that really gut-wrenching situation three, almost four, years since ChatGPT was launched. I do think one of the goals that I had personally writing this book, too, was to figure out ways to help faculty move past that initial gut reaction of just thinking about ChatGPT, generative AIs, as a cheating tool, and to start thinking about more broadly about when is it acceptable to use this, when it is not. How can we articulate that to our students through our learning outcomes? And what we did, I think, a really good job in the book, too, is looking out at the world of academia, too, and finding wonderful examples from faculty in how they’re actually incorporating a lot of these different types of experiences for their students, and how they’re sort of wrestling with these questions too.

Annette: Yeah, absolutely. I do think that that’s really the most exciting part about this book, is that we feature so many really concrete, tested examples from teachers at community colleges, at research universities, teaching colleges, just like everywhere, and for a long time I ran our graduate training in the English department for teachers, and now I run faculty development for AI, and this book sort of answers a lot of questions of sources that I would like when I’m teaching those kinds of things, like what do I want to assign that gives a kind of theoretical background, and also that kind of trust students, acknowledges student agency, trusts teachers and their expertise in whatever disciplines they’re teaching in, because it is meant to be something that moves across disciplines, and then gives them a whole bunch of examples, and there’s not answers in there, but there’s a whole lot of things to adapt, and so hopefully that’s useful for working teachers.

John: As you’ve noted, one of the concerns that many faculty have is that students may use it in ways that are not conducive or supportive of their learning. And actually, Derek had a recent podcast where he was talking with some people in math, how earlier on it was easy to detect the use of AI, because AI tools often got things wrong, but that’s much less true today, especially for the paid versions and the deep thinking versions of these tools. So, what are some strategies that faculty can use to assess student learning and their skill development in a world where they have access to these increasingly capable AI tools?

[AWKWARD PAUSE]

Marc: Let’s keep the awkward pause in here… [LAUGHTER]

Annette: I do feel that awkward pause. [LAUGHTER]

Marc: I do. Well, I can take the lead here on this one, if you guys don’t mind, then I’ll hand it off. I think strategies are evolving. I think that how we are seeing these tools being deployed and marketed to students, too, is part of our awareness to see this sort of version of it too. AI detection is also changing alongside all these other tools as well. I’ve really leaned in personally for myself, and also included examples too, about emphasizing disclosure when AI is being used, both from the student perspectives, but I think it’s also important for faculty to start having these conversations themselves, too. We are not only seeing students use AI, we’re starting to see faculty now really adopt these in all parts of the course, including in some cases grading. So, if you’re going to use AI in that sort of manner, too, you probably want to think about disclosing it as just sort of a baseline expectation and giving some great examples about what, as I said, in a recent talk, too, there’s performative disclosure, there’s exhaustive disclosure, we give them everything, but then there’s authentic disclosure, where you want to have a conversation, we want to have a dialog about how AI is being used, because in some ways you’re genuinely curious about how this is actually being used by students, and if it can be able to support them, and there’s lots of analytics you can use with the course management system, too, to track student behavior. That’s one thing to kind of be there to just make it visible, but the main strategy there is that you really want to have a situation where you can have conversations with them in some way, shape, or form. That’s very difficult for a high enrollment class, or if you’re teaching six sections, which some of our faculty are doing, and can’t actually have those types of relationships,

Derek: I’ll add that what I hear fairly consistently from students through surveys and panels that I’ve been a part of is that they want some clarity on what the boundary lines are around what’s appropriate use of AI and what’s not, and there’s lots of different ways to try to communicate those boundary lines. There’s the AI assessment scale, which says for this assessment, these are the types of uses that are okay. You can do it at a course level and say it’s okay to use AI for figuring out a hard concept or brainstorming ideas, but you can’t use it for drafting, or you can use it for editing. We’ve seen lots of different examples of how faculty are helping to define those boundary lines for students. In some cases, to Marc’s point about good conversations, they are figuring out those boundary lines with their students through conversation. They’re kind of coming together to come up with good policies around this kind of thing. And I think, though, an important piece of this is that most students want to know kind of what’s allowed and what’s not allowed, right? That’s a big piece of it. But I think students are also concerned about what’s going to help my learning and what’s not going to help my learning, and they are not always in a good position to know what uses of AI are going to be conducive to their best learning. They’re novices, they’re figuring out how to learn. We may have a better idea of that. We may be figuring it out too, in many cases, because AI is so new, but it’s not just about how can we secure our assessments. And from the student’s perspective, it’s not just about how can I stay out of trouble, but there’s a strong element of how can we actually help learning happen in the ways that are meaningful, and figuring out those boundary lines together is one way to get at that shared goal that we and our students both have.

Annette: I think that’s important, that shared goal, because one of the things that we know from what students tell us about their experiences of AI in classes is that they are really frustrated about a lot of what’s happening in that there is lack of clarity in policies, most of what they see in policies is do not do this, rather than here’s some good ways to use AI, and they find that sometimes infantilizing, I think, and unrealistic, and then they also don’t like it when their peers are overusing AI, and that kind of cheapens their own experience and degree and their effort, and so they have a lot of feelings about it. Marc started out by saying, talking to students, I think that’s really the crucial thing, and I think there’s ways you can do that and certainly we talk about that in the book, you can do that in large lecture classes and through asynchronous surveys, all sorts of ways. It’s possible to hear from students in a lot of different ways, but I think that’s super important, and also one of the reasons why I said in the answer to the last question about student agency is that we need to make sure that students are getting what their goals are out of our classes, we have external goals and things that we want them to get out of classes, but they have their own goals, and they have their own interests in learning, as Derek mentioned. So, can we help them align those goals that they have to what we’re doing in the class, and help them articulate how this class is going to help them in the future, because our students are also nervous about job prospects and all sorts of things, and I think the more that we can lean into the fact that we’re there to support their learning, we can help all of those conceptual things. The other thing that we talk about in the book some is the University of Sydney two-lane assessment, which I think is a helpful framework, thinking about, on the one hand, secured assessment, and that is like proctored exams or things where you have a really clear idea that the person who is represented here in this space is the same person you’re assessing learning of this particular person, and then you’ve got a lot of spaces of open assessment where students are doing homework, low key, et cetera, they may or may not be able to use AI. And so, I think that one of the reasons why we get that question all the time, about like, how do you stop students from cheating, or whatever, in workshops, is that the labor that teachers are going through is so significant right now, and nobody wants to do the surveillance and policing, zero teachers want to do that, and trying to figure out how do you actually assess what a student is doing is so much work, and so if we can narrow that space down, and then let students play around a little bit more with here’s possible positive ways of using AI to enhance your learning, in ways that they may not necessarily know already. As Derek said, they want to work on their learning, but it’s not always clear what that looks like, and that’s part of why we have teaching experts in the room, that is us, right? So we’re giving all very complicated answers to this, and that’s because there is no silver bullet. We keep coming back to that in the book. There are no easy answers, and we do not give easy answers in this book, because they’re just aren’t present.

Marc: That’s absolutely right. And she also hit on something that I think we were all really attuned to, is the labor of this going through the sort of response to AI. Doesn’t matter if you want to engage AI to or refuse it to, you’re still going to have to really focus on work, and part of what we’ve been doing too is trying to find different strategies to support faculty, so that’s not so overwhelming, trying to think about even taking some steps from, like, Jim Lang’s small teaching moments to bring up these types of conversations, because we realize that not all faculty are going to have support from wonderful teaching and learning centers. Some faculty are going to be contingent and sometimes not have much support at all, too, and they still are going to want to practice some type of AI awareness with their class and their actual day to day lives, so labor, I think, is a really big one we’ve been talking about as we go through it.

Rebecca: I think one of the spaces we get a lot of questions is on the asynchronous online environment, in particular, because it’s extra challenging to figure out ways to do assessments or to do quick checks for learning in that environment where it’s completely vulnerable to AI in the ways that maybe we have historically done these things. Those that teach in in- class settings have moved to using blue books and some other things occasionally. What advice do you have for some faculty that are specifically in this asynchronous environment that are hitting some slightly different challenges than you might encounter in a face-to-face class.

Marc: I think the best piece of advice I could offer, because I drafted, I think, a chapter that we try to talk about this, is to breathe and not take this personally. It’s not against you, and there’s certain things you can control. Sometimes you have options with your modality you pick as a teacher, sometimes that’s picked for you, but you can also really search for ways that you are more present inside of your online class that makes it more embodied for yourself and for your students. I’m doing that myself now too, with my hybrid and online classes too, trying to be more there with them, having more conversations, and just also making myself more available to them too, because one thing that I think we’ve all noticed is that if you are doing an asynchronous course on autopilot, not only is that very prone to AI misuse, but also there’s a debate too, if you’re not there as the teaching expert, that you’re not there as the person in the room, too, what is necessarily being lost if a student is going to use AI in that context? So that does really reveal a lot of our practice that we’ve gone through to start thinking about it. For the security aspect, too, there are layers of things that you can do in these courses. You can, if you’d like to engage a lockdown browser for some assessment types, secure it. That’s not going to work for everything, but you can also switch over and think about doing a mix of video assignments. I do a lot of work instead of the discussion board in my online classes with videos. I don’t do the talking head thing anymore, too, where they just read a script, which, because that script is generally generated by AI, I have them actually write something or do something else there with, if they’re doing a multimodal project, share their screen and work in real time on the screen too, or they do a recording of that asynchronously, so I want them actually working in practice, showing me that process, because even the AI is involved in that too. They’re still working through those decisions, they’re still thinking about those things. So, there are some strategies we can use. They are going to be complicated. I do think we need a little bit more support from our course management systems to work with us, especially now that agentic AI is here. How many students know about agentic AI, I think is a very good question. I think most of them will associate generative AI with ChatGPT, just like Google is synonymous with search. I don’t think they’ve risen to the point, too, of they’re all going out to use agents to go through and crash their courses. I would actually say that faculty are maybe more aware of that than students are, which is a good thing to start thinking about where this is headed, but it is something to be thoughtful about as we go forward too, because as these systems become more and more advanced and also more autonomous in what they can do with the different workflows, it’s not just gonna be students that are going to be adopting them, we have news that Canvas is going to have an AI teaching agent there as well, with certain limitations and guardrails, but this is just sort of the start of this new version of working with these tools, and we need to start thinking about what sort of boundaries we want to put in place, both for students and for ourselves with them.

Derek: I think for the online space, as you say, Rebecca, the online asynchronous is kind of the perfect storm of AI problems. I’ve been thinking that there’s an impulse to try to solve that problem through better course design. If we can have more transparent learning objectives, if our assignments are more authentic, if we can combine those, so students know here’s why I’m doing this project, and here’s what I’m going to get out of it, and it aligns with my personal and professional goals. Those are all things that are going to help students resist an unhelpful use of AI and lean into the hard work of learning. I don’t think that’s sufficient. [LAUGHTER] I do think that’s necessary. I think that’s good teaching to begin with. I do think we also need to have a cybersecurity mindset that says there’s going to be bad actors who are using tools that are problematic, and we’re going to need help from our IT infrastructure to figure out how we can block things and prevent things. I think there’s sometimes an impulse just to do that, right? Like somehow we can technology our way out of this problem. I really do think we need both. I think we need better curricula that better align with students’ goals in more transparent ways, and we also need to think about the technology tools that we’re going to use. So, one of the metaphors I think I shared with our author group was the notion of a locked door, so I have a locked door in the front of my house. It’s my understanding I haven’t tried this, it’s my understanding that with about an hour and a few simple tools and YouTube, any one of us could figure out how to pick the lock on the front of my door. The technology itself is not super secure, but it communicates to anyone coming by, you’re not supposed to just walk in my house, you need to ask first, and so sometimes it’s tempting to think about the kind of anti-cheating technologies as like a secure barrier, like we’re going to keep out all of the cheating, we’re not going to keep out all the cheating, we’re going to keep out some of the cheating, but we’re also going to communicate what those boundary lines to students are a lot more clearly to say these are signals we give to students to say don’t do this and please do this instead. The research on academic integrity says most students don’t cheat because they’re evil people, [LAUGHTER] mostly it’s because of the current environmental situation that they’re facing. And so if we can create those signals and boundary lines that help students make a better choice in the moment, I think that’s a lot of the good that we can do in this case.

Annette: I love that metaphor of Derek’s. I think it’s really helpful, but one of the things that signals is not only to the students who would be cheaters, who might, if it were just a little bit easier, make a different decision, but it’s also indicating to students that their fellow students are getting the same message, and so there is a “here in this space we respect academic integrity, and that means you should also feel comfortable doing the real work and leaning into the hard work of learning,” as Derek said, because they do feel like suckers if they’re doing that hard work and their peers are not, even if it’s not on a curve, you know, whatever, there’s like that deep inherent feeling of unfairness, so I think that’s really crucial. Marc brought up the point of agentic AI, and how that might be more of an issue in online teaching, and that maybe faculty are more aware of it now than students, which is maybe a good space for us to be thinking about that. One of the reasons why I feel like I’m not as worried about agentic AI is that it is so clearly an academic integrity violation. Something else is impersonating you and doing the work of you on a platform, which isn’t to say that it won’t happen, of course it’s happening even now, but that casual, like, is this or is this not against the policy? It’s really against the policy you survey students, and they’re like, “No, flat out, that is just absolutely wrong,” versus, “well, this policy says no AI, but it doesn’t go into details, and I did some brainstorming, but I didn’t use it for writing,” like that is “who knows, like maybe,” they weren’t really specific. So I think that’s more the space that when we talk to students, we can help them think through, “Okay, this is why I really don’t want you to use AI for brainstorming… for these specific reasons here, and maybe it’s good in other contexts, but not here.” And so this is why I think agentic AI, certainly we should be paying attention to it, but I just do not envision a future in which all college students are absolutely botting their way through their degrees.

John: You talk about a number of strategies that can be used in terms of getting student buy-in to reduce the excessive reliance on AI, but are there any other strategies that you can think of that might be able to deter students from using AI in ways that are harmful?

Derek: So, I can think of a couple more. We’ve talked about the big ones around having transparency with students about learning objectives and what they’re going to gain out of this learning experience, we’ve talked about more authentic assignments that kind of connect more with students’ interests. I think a lot of faculty are experimenting with alternative grading schemes that are changing the motivational landscape for students and giving them more opportunities to revise and resubmit and try again. That’s going to take some of the high stakes away that might lead students to take shortcuts. Again, not a silver bullet, but it’s part of those environmental factors. We talked about online courses, and it’s easy sometimes to forget that even in online courses our students are humans with bodies in real physical environments, and so I’m seeing some faculty start to think creatively, like Marc was saying, about how could we do some process tracking where they are doing some work, and it’s recorded, or are there assignments that are going to get them out of their laptop zone and into the real world to do something that’s connected to your course? Again, not a silver bullet, but adding that to our mix of engagements, I think, can help students shy away from shortcuts. And the other thing I would say is, I referred to this earlier, students are learning how to learn when we encounter them. We are probably better learners than they are, maybe not in all cases, but we’re experienced academics, we’re pretty good at learning. AI is changing how we approach learning in our own disciplines, but what I would argue for is giving students more opportunities to do some metacognitive work, reflecting on their own learning in our courses, and thinking about how they used AI. Was it helpful? Was it not helpful? They had to read a hard chapter this week. What were the strategies they used to make it through that hard chapter? Which strategies were more or less helpful for that? And some of those strategies may involve AI, and some may not. Our students are going to go, and they’re going to have access to AI tools when they leave us. Hopefully, we can equip them to use those tools with some level of discernment, and that involves some self knowledge and some metacognitive thinking, and so I think we should start to build those opportunities in our courses for students.

Rebecca: Well, you’ve teed this up nicely, Derek. Thinking about the workplace and how AI is used in the workplace, are there ways that we should be rethinking or adjusting our learning objectives for our courses to either cycle out some skills or particular things that maybe were in there before, and put in some new things. I think at the top, Annette, you talked a little bit about a moment to reflect on [LAUGHTER] learning objectives and potentially take it as an opportunity to revise.

Annette: Yeah, absolutely. Derek used the word discernment, and that’s a word we really hooked into in the book, is thinking about how students can exercise careful judgment about when to use AI, when not to use AI. We all know a dozen examples of contexts in which people have used AI in places that are absolutely inappropriate, either for human means, either for signing off on something that shouldn’t have been signed off on, hallucinations, inaccuracies, all sorts of things, and I feel like my goal is to have students [LAUGHTER] not do that, like don’t put us in the headlines, you know,[LAUGHTER] which I think is something that goes over well with administrators at any university. But, really it’s like, how do you make good judgment on what AI looks like when you go into the workplace, and there are so many workplaces that really are leaning in, and it is kind of unregulated in a lot of ways. It’s surreptitious, it’s tacit, it’s the secret cyborg that Ethan Mollick talks about, and so I think it is really important for students to not just like, “here’s how you do prompt engineering,” but something that’s much more sophisticated about how are you supplementing the knowledge that you have when you’re using these tools.

Marc: Yeah, the goal with discernment, too. I mean, I’ll just give you an example, too. I just spoke with our law school, and they are getting a lot of interesting signals from the legal industry about how they want AI to be used and not used. You can actually go on to the AI hallucination case database. Someone’s actually set this up. There’s 1300 hallucinated cases, along with the sanction that the judge, who, of course, was very upset, found out AI was used in a way that created fictitious case law. So there is that issue too, of using AI inappropriate.

Derek: And just to clarify, these are real lawyers who have submitted briefs to courts where there were hallucinated content in those briefs, and they were caught on it and called out and punishment. And there’s a website that tracks all this, and I think it’s just a weird, strange world we’re living in, where that continues to happen, and someone is tracking all of that.

Marc: Indeed, yeah. And to make it even stranger now, the opposite end of the spectrum is they’re hearing from a lot of corporate interests too, is that when they go to bill them for their hours, they want to have AI listed there, not as disclosure for hallucinations, but how much money did you save us by using AI, or did you use enough AI. So there are some really wild mixed signals, depending on what your discipline is, about what your industry is, as well, too. And helping students navigate that is a big challenge. Helping faculty now navigate those questions, too. It’s like, okay, do I need my students to learn about AI to avoid hallucinations? Oh, wait, you have to learn about AI because you’re being billed for it if you don’t use it enough. So, it’s really fascinating just to see this play out. That’s just one example. We have this in different industries too. I’ve talked with folks in hospitality management. AI is a massive tool in that industry right now, and they’re having some issues that they’re doing this with too much AI for the audience and upsetting them too, but also, having that same sort of situation too, where your company you’re working with, or you’re supposed to be working with, is demanding you use AI to cut overhead. So it is going to be a challenge to navigate, for sure.

Rebecca: So, are we revising our learning objectives?

Derek: I think, to Marc’s point, I think it depends a lot on the discipline that you’re in and how things are shaking out. So, like in computer science, the goal of learning how to write basic code is less important now. The goal of figuring out how to evaluate the quality of some other code that you’ve been given, whether it was written by a human or an AI or not, the kind of reading and evaluating code that’s more important now. Over in a writing class, though, writing the first draft is kind of the equivalent to drafting that code, right? Is writing the first draft not important because AI can do it? I don’t think so. I think the ideation and brainstorming that happens during that first draft is super important, and so similar moves with AI in different disciplines may be thought of very differently in those disciplines because of kind of what’s under the surface there, and so that’s why I think a lot of these conversations, around learning objectives in particular, need to happen within the disciplines and within the fields, particularly if those disciplines are having some conversations with how the workforce is changing.

Annette: One of the things that we talk about in the writing chapter, to kind of extend a little bit what Derek was talking about, is that there are different important aspects of the intellectual process across different aspects of writing, so I teach in an English department, and that means that that ideation, the brainstorming, the writing is kind of the argument, and that’s the work. I also spent eight years working in a neuroscience lab, and in that lab I did some of the writing. That was inconsequential, because, like, I didn’t design the experiments, I didn’t, I mean, sometimes I actually did sleep deprive [LAUGHTER] the rats, but I didn’t really do the work, the intellectual work of the lab, and so I didn’t get an authorship credit, rightly so. And so when we look across disciplines, it is really important. This is why we keep saying in the book, we trust you to figure this out, is like you have to think about like what is the intellectual work that’s happening in your class and in your discipline and for your students, so that you can pinpoint that intellectual work and make sure that it’s enhanced or at least not undermined by AI. And then maybe, such as what Marc is talking about with the legal profession, you need to integrate AI or help students think about how they might thoughtfully do that, because they’ll need to have that skill moving forward,

John: And by connecting to what people in the fields where students are likely to go into when they graduate, it can create more authentic learning opportunities where students are using AI in the same way they might be expecting to use it later, I think. We’ve talked a little bit about all the concerns that faculty have, but there’s lots of ways in which AI might be useful to assist with student learning. Can you talk a little bit about some ways in which AI tools can be helpful in increasing student learning?

Derek: So, I’ll jump in. I’ve had a journey on this, because I feel like when I started writing with these two, I was the like all in on AI, and let’s use it to augment our teaching and then create new learning experiences for students. The more I went through the writing process, the more I’ve talked with faculty this past year, the more mixed I am, and the more kind of avoiding AI piece has been a much more important to think about, like we do want to protect the space for students to learn in meaningful ways. That said, here’s my current metaphor for thinking about this. [LAUGHTER] You’ve probably heard of the forklift in the gym metaphor. If you want to move weights around in the gym, you don’t bring a forklift in and move them, you go in and you lift your weights. That’s the point of the exercise, is for you to do it and to let your muscles get that strength training. Yes, the forklift will move the weights, but that’s not the point. Some uses of AI are like that, they are doing the work for students, they are moving the weights for students, and the students aren’t getting anything out of that. But I would argue that there’s another machine that actually you do find in gyms, which is the treadmill. I’m a runner, when I want to run and it’s 35 degrees outside and raining, I don’t want to run outside, I want to go to the gym and run on a treadmill. The treadmill is not doing the work for me, but it is creating conditions under which I can do the work, and so those are the uses of AI that I was initially drawn to, and I continue to be drawn to. How can we use AI in ways that are going to create new learning opportunities for students without robbing them of the exercise themselves? And so that’s things like having students critique the output of some AI chatbot that’s doing some task associated with your course. It gives them a chance to practice the knowledge and skills they’re building. This can be a tricky assignment if the AI does too good of a job in whatever you’ve asked it to do. There may not be enough to critique, but lots of faculty are finding places or particular AI tools that still have kind of big flaws in them. We have one example in our book of a math faculty member who had an AI generate flawed proofs to mathematical statements, and the student’s job was to evaluate those proofs. That’s a great opportunity for practice. Other faculty are having AI give feedback to the students on their work, and I think you have to do that in a thoughtful way, so that students aren’t just kind of taking the AI feedback. As gospel, that they are in a position where they can question that, maybe compare that to feedback they’re getting from their peers on a particular assignment. You also don’t want to kind of abdicate your own role of giving students feedback, so thinking of AI-generated feedback as one step in a process that later involves you giving feedback to your students. But well-designed prompts to AI can generate really useful and actionable feedback for students, and so that’s a treadmill kind of thing. And I guess the third big category I would mention is custom AI chatbots that are designed to simulate certain types of interactions or conversations for students, so that they can practice talking to a client or talking to a patient, or what have you. That’s a treadmill kind of use of AI, and a lot of faculty are having a lot of success with that, in part, because the fuzziness and unpredictability of an AI chatbot becomes a feature, not a bug, of that type of activity. The fact that it’s unreliable and contradicts itself is okay, because that often happens in the types of professional settings that we’re preparing students to enter.

Annette: One additional thing I think teachers are doing, which is super exciting, and I saw evidence of this in the faculty development seminar that I teach at University of Pittsburgh, is teachers are starting to basically vibe code their own apps that are supporting learning, which is just amazing to see, actually, because here is this opportunity to create this nichey little thing that is not going to be served by some big platform and subscription blah blah blah, it’s like I teach this particular really difficult concept. It draws on these different databases and government websites, and blah blah blah. We always have trouble with it. Students struggle to put this all together. It’s really hard to visualize, etc. Or you can ask Claude to make up a cool interface where students can play around with it, and then actually really get a good sense of this complex thing that I try to teach. Or here’s this visualization thing in a science class of we’re looking at a sample on their microscope, and you can change it into a diagram, and that can make it a little more easy for students to tell what it is, label the parts, things like that, and those kinds of real-time transitions across modes, say from image to diagram, or the interfaces. I hope that we see an explosion of those kinds of teaching tools that teachers are able to innovate with using AI on the back end, then, so, like, students aren’t directly using AI, but they’re interacting with the results of AI.

Marc: Yeah, I think for me the easiest way to start is to define AI’s role in the sort of relationship you have with your own teaching, and also set that up for your students to give it a specific role, develop rapport with your students, that level of trust as you go through this to see if this is effective in their learning outcomes. I am a little bit concerned about the fact too that some students have very expensive subscriptions to certain AI tools and services, while others are using the free plan, and what that’s going to do for assessment, and also equity and accessibility issues too, going forward. But you only get to find that out if they’re talking to you about this, if they’re doing this openly, and so when you have that rapport set up there too, like I’m doing right now, my advanced class about AI and society, my students are now given free rein to design things that they like with AI, after knowing about it too, and showing me and teaching me how they’re using these tools to learn, which I think is really cool, and as Annette mentioned, too, a lot of them are vibe coding now with Claude, which is just wild. When we were doing this six months ago, too, Claude really wouldn’t do a very good job. You do some very simple sort of things. Now we have some students vibe coding entire simulations of video games that are pretty impressive, and actual apps too. And teachers can do the same. So, once you’re comfortable with it, once you have that rapport there too, you can really start going into the creative possibilities about what AI can do to help. The challenge there is that there isn’t kind of a limit to it. It’s very hard to basically say where this learning stops with the technology when you can design so many different things based off of your imagination, and that is something to just kind of be thoughtful about, too, because it is a lot to just kind of press faculty on as well, especially if there are just one or two different use cases. Well, it’s this technology’s like clay, throw it on a pot, you can really turn it into anything that you want. And my response from some people have been like, “that’s a lot to take in,” like, “yes, it is.”

Rebecca: Just don’t forget to ask for accessibility features when you’re vibe coding,

Marc: Exactly, exactly. It’s a big thing.

Rebecca: So, we’ve talked a little bit about, well we’ve talked a lot about ways that generative AI might be able to be used to enhance student learning. In some of those discussions, we’ve talked a little bit about how it might also support teaching. Do you have any additional examples on how AI might be used to enhance our teaching?

Marc: Yeah, I’ll give you one that I’ve been using personally, that, like many faculty, I think I was in a bit of a secondary sprint here to try to bring my courses up to Title II compliance, and we got a reprieve that was sort of last minute, but it’s only a year-long rep, so we are still going to have to make sure our additional materials are fully accessible to be compliant with federal law. And I actually used a version of Claude to go through some of my course materials, and set it up to look at the different accessibility guidelines. I thought I’d done a very good job six months ago, and it turns out that no, I missed quite a few things, and it was actually very good to have that sort of experience for an AI tool to go through and sort of check your work. So, I think having AI as a tool that could sort of audit certain things – accessibility is one of them too – but also going through some of my course learning outcomes and seeing the ways that an AI system might look at that and compare it to a different university’s course learning outcomes. How up to speed am I and my peer institutions, and give me some options. I really do think if we were using AI iteratively, we can kind of make sure that we are the ones that are staying in the driver seat, if you will, and really center both not only student agency but in this case, teacher agency, to make sure that they’re making the right choices, and giving yourself some good options with AI to go through that process can be very helpful, and in my case, very eye-opening to me about these types of situations. I will say, my course, I had really done is one of my summer courses that I had not run in about two years. [LAUGHTER] I was just like, “oh yeah, I did that,” and no, I did not actually revisit that for accessibility concerns from two years ago. So AI can do quite a bit.

Annette: Yeah, like Marc, I’m very excited about the possibilities of AI to augment all of the details of teaching that we do, not the kind of human connection and the grading and the assessment and the reading and that kind of thing, but I think Marc’s example about how to make a course more accessible is excellent. The other thing I’ve been using right now is we have an institutional subscription to Claude, and I’ve used the MS 365 connector now, which basically hooks Claude up to my Outlook on my email, and what I found is having semantic search on my email is like a game changer. It’s just like I can ask questions of 15 years of emails and folders and all sorts of things, but I can ask questions of it that help me summarize conversations that are revived. It helps me right now think about my year-end report, like reporting that we have to do. How much time did I spend and what did I forget about the things that I did? Inevitably, you always forget things, and Claude is helping me locate those things right now. I have colleagues who have used Claude to design things that help fix the somewhat broken scheduling system that they have to deal with. It’s all these things that are super essential to the infrastructure of how a university runs, but there’s a lot of technologies that are either too big, clunky, they aren’t specific to whatever things that we need to do, and I think that I absolutely agree with Marc that the more we’re practicing these things as faculty, the better we are at understanding their limitations, their possibilities, and these are things too, like if I had a secretary, then I guess it would be replacing my secretary, but I don’t. This isn’t like undermining the intellectual work that I do as a professor. It’s not putting somebody out of a job, but it is really making my reporting more accurate and better. And so I think it’s important to recognize those possibilities.

Rebecca: I think that gets back to something you guys were talking about earlier, and using it to do some of the tasks, but it’s not the intellectual work of being a faculty member.

Marc: It’s also not the relationship aspect, too. That’s the big thing that, to me, again, part of AI awareness is just where are you going to put some boundaries in place, too. I use these tools all the time, like Annette does, but there are certain things I don’t want to do with my students. I’ve told them I’m not going to do this to write letters of recommendation, I’m not going to use it to answer emails, I’m also not going to use it to grade. Those are my sort of boundaries. Other people are going to be very different what they set up, but to me the best thing to do is just put yourself in conversation with that and start considering where you are going to draw the line and why.

Annette: And the other thing to note, too, that what I love about the way that Marc is talking about this is he’s really open with his students about that, like I’m not going to use it to grade, I’m not going to use it to do this, and I do think that as we ask our students to be transparent with their uses, it’s really important for us to be transparent with them about the uses that we have of AI, and I think they have some suspicions, they are frustrated, they’re interested. The students have all sorts of feelings about faculty uses of AI, and we need to take that into account when we use them. But I don’t think the students would care about me using AI in my email, [LAUGHTER] not for the writing of emails. I don’t use it for the writing of emails, but like for searching, like what are all the things I did for this committee for my department.

Marc: Now they’re not going to care about that, they’re going to care about the writing of emails. I agree, and you know the big thing to think about too, and I know we’re kind of dragging on the question, but this fall it’s going to be four years that high school students have had complete access to ChatGPT to generative AI coming forward. They’re not just going to be Plato exiting the cave, they never were entering our classes. They’re going to have experience using these tools, resisting these tools, and having that. And having this sort of conversation with them too, I think, is really important to think about their own agency when it comes to using or not using AI.

John: And the things you’ve discussed are ways where we can free up some time, so that we can have more time to devote to students, but there might be also some ways in which we could use AI directly to perhaps improve our teaching. One of the things I’ve done with faculty is have them use AI tools to help put their assignments in a TILT framework to make it more transparent to students, because we know what we mean when we ask students to do an assignment, but they don’t always know that, and that can be a nice way of evaluating it, to have an AI tool read it from the perspective of a student and put it in terms that may be meaningful, or perhaps to evaluate a syllabus for inclusive language and things that might get past us without the use of an AI tool, or use it to suggest some active learning exercises if we were mostly doing lectures, for people who want to move away from that, but aren’t quite as comfortable.

Marc: Yeah, absolutely. And there is an element of humility in that, too. You’re asking a system to help you perform your job, and that is something that I’ve talked with a lot of faculty about, that for them, they’re like, “no, I’m not going to do this,” but like, try it. Here’s an exercise, see if it is. And generally, there is that sort of aha moment when we do this in a workshop. They’re like, “oh, wow, so yeah, this is actually kind of helpful for me.” This is something I might take back and show to my colleagues and help them understand what’s going on too.

Rebecca: So we always wrap up by asking, what’s next?

Derek: I’m excited that the book is coming out this summer. That’s a big what’s next for me, I think, to help support that and to continue these conversations. I’ve been reaching out to a lot of the faculty whose work we mentioned in the book to have them on my Intentional Teaching podcast, and so I’m excited to continue some of these conversations in this audio format this summer.

Marc: Like Derek, I’m very excited about the book too, trying to plan what level of support that I can bring to faculty from a lot of these examples that we go out there to do lots of workshops and also lots of different engagements, both internally here at my university and also externally too. So happy to see that basically pop up and be out there in the world too, starting this summer for the ebook, and then the print book will be out at some point in time, early in the fall.

Annette: I’m excited, also like my co-authors, about the book being done. It’s always a big project, as Marc said at the beginning here, it’s a real sprint, it was a really intensive timeline. To take a commercial publisher and get something from conception to publication in less than a year is unbelievable, which is kind of the pace of AI, and it shows Norton’s priority for the book, too, which I was happy about. So, yeah, what’s next is, well, I’m working on an academic monograph about a long history of AI, and so I’m excited to do that. I feel like we learned so much in the process of writing the book, like all of the different worked examples from the teachers. I really am so grateful for the teachers that were happy to share those examples with us and with other teachers through this book. What I’d really like to do is I’ve been integrating some of those examples again with permission and workshops and things like that, and then to do that work at home too at Pitt and think more cutting edge about what we can do with the technologies that we have access to locally, and how I can work with the incredible colleagues that I have here at Pitt, and the students, how we might further student research too, which I’ve been working on here at Pitt too. So I’m excited about that next chapter, and I really am excited to learn more from other teachers, especially with the new innovations with vibe coding and all those kinds of experiments. I think we are on the precipice of some other hopefully interesting pivots in the ways that we approach education.

John: Well, thank you for joining us. We really enjoyed the conversation, and we’re really looking forward to seeing the book when it comes out. In fact, we’ve listed it as a reading group we’ll be running on campus in the fall, so we’re especially looking forward to seeing the final version.

Rebecca: Well, thanks. This was really great.

Annette: Thank you so much.

Marc: Thank you, guys. I appreciate it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]

447. UDL at Scale

The Universal Design for Learning framework is often adopted by individual faculty for particular courses. In this episode, Tom Tobin joins us to discuss the potential benefits associated with an institution-wide adoption of this framework.

Tom is an internationally recognized scholar, author and speaker on technology mediated education, especially copyright, evaluation of teaching practices, academic integrity, accessibility, and universal design for learning, which is a topic we’ll be talking about today. He helped found the University of Wisconsin Madison Center for Teaching, Learning, and Mentoring. Tom is on Ed Tech Magazine’s Influencers Dean’s List, and has been honored with the Wagner Leadership Award in Distance Learning Administration, and he is one of EduFlow’s global top 100 learning influencers. Tom serves on the boards of Advances in Online Education, The Online Journal of Distance Learning Administration, and the Oklahoma University Press: Teaching, engaging and thriving in higher ed series. We’re very glad that he’s made time to be with us with all these activities. His books include Evaluating Online Teaching, The Copyright Ninja, Reach Everyone, Teach Everyone: UDL in Higher Education, Going Alt-Ac: A Guide to Alternative Academic Careers, Implementing UDL in Irish Further Education and Training, and what we’ll be talking about today, UDL at Scale: Whole-Campus Universal Design for Learning, which is coming out this summer.

Show Notes

Transcript

John: The Universal Design for Learning framework is often adopted by individual faculty for particular courses. In this episode, we explore the potential benefits associated with an institution-wide adoption of this framework.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Thomas J Tobin. Tom is an internationally recognized scholar, author and speaker on technology mediated education, especially copyright, evaluation of teaching practices, academic integrity, accessibility, and universal design for learning, which is a topic we’ll be talking about today. He helped found the University of Wisconsin Madison Center for Teaching, Learning, and Mentoring. Tom is on Ed Tech Magazine’s Influencers Dean’s List, and has been honored with the Wagner Leadership Award in Distance Learning Administration, and he is one of EduFlow’s global top 100 learning influencers. Tom serves on the boards of Advances in Online Education, The Online Journal of Distance Learning Administration, and the Oklahoma University Press: Teaching, engaging and thriving in higher ed series. We’re very glad that he’s made time to be with us with all these activities. His books include Evaluating Online Teaching, The Copyright Ninja, Reach Everyone, Teach Everyone: UDL in Higher Education, Going Alt-Ac: A Guide to Alternative Academic Careers, Implementing UDL in Irish Further Education and Training, and what we’ll be talking about today, UDL at Scale: Whole-Campus Universal Design for Learning, which is coming out this summer. Welcome back, Tom.

Tom: Thank you, John and Rebecca, it’s a pleasure to be back on Tea for Teaching. Thank you for having me.

Rebecca: Today’s teas are… Tom, are you drinking tea with us today?

Tom: I am. I have some Pure Leaf Unsweetened Iced Tea. So that sound of ice cubes that you hear that’s my preferred form of my iced tea.

Rebecca: Sounds nice and refreshing.

John: And I have spring cherry green tea today. Given that we are apparently in spring, the snow has stopped at least for a few days, so we’re moving into spring here.

Rebecca: I have Lady Grey today.

John: Very good, one of my favorites. So we’ve invited you here today to discuss UDL at Scale. While most of our listeners are familiar with UDL to a greater or lesser extent, could you at least provide a brief overview of the UDL framework to help set the framework of the discussion?

Tom: Absolutely. Universal Design for Learning, or UDL, is a framework based on how we learn as human beings. So it’s based in neuroscience, fairly loosely, if I’m being honest. That when we learn anything, we have to activate three different chemical pathways in our brains. We have to have a reason to learn something. We have to have the content that we’re learning, and we have to be able to reinforce it, practice with it, tie it to other things that we already know. And so these three phases of the learning process, which are themselves a pretty good simplification of very complex chemical cascade in our neuroanatomy, they correspond to three different things that we can do as designers to help people to get started and stick with us when things get challenging, have the information to be able to learn it, and then to show what they know and practice with it. The neuroscientists at CAST in Boston, way back in the late 1980s and early 1990s, they discovered that if we give people one way to get started or take in information or practice, then we’re leaving out other ways that people could engage with that stuff. So what they said was, make more than one way, make multiple ways for people to get engaged, for people to take in information, and for people to practice, and people will choose which one seems best to them in the moment, and that helps the learning to stick. So Universal Design for Learning asks us, as instructors, developers, designers, to make multiple ways that people can get engaged, take in information, and show what they know.

Rebecca: Can you tell us the origin story of your new book, UDL at Scale, and its relationship with your previous work?

Tom: This is a very different book than my previous two books on Universal Design for Learning. When I did Reach Everyone, Teach Everyone: UDL in Higher Education with Kirsten Behling and then when I did the Universal Design for Learning in Further Education and Training for the Irish government, those were practical application based books. People were hungry to have, how do I do this in my own classroom or my own institutional design? That was at an individual level. People wanted to have the nuts and bolts of, what do I do when I’m designing a lesson or an activity or an assessment? One of the big challenges that you started to see over and over again is that whether it’s by law or by policy or by people adopting good ideas, we’ve been putting the burden of making things accessible on the shoulders only of instructors. And we also have not been giving them any release time or extra money or a graduate assistant, or we’ve not taken any work off their plates in order for them to be able to make their materials and engagements more inclusive and more accessible. So it’s not actually a wonder that only about 10% of instructors, wherever we look, have adopted inclusive design techniques of any kind, let alone a specific one, like the Universal Design for Learning Framework. Because of that, when I would talk with boards of trustees, college presidents, provosts, deans, campus leaders, and I would say, “You know that you have these legal requirements.” In the United States, we have the Americans with Disabilities Act. In Canada, it’s the Accessible Canada Act. Worldwide, there are laws that apply to the design of content and materials, and we’re not even doing that really well. So when I would talk with campus leaders, and I would say, you recognize that there is this requirement there, but there’s also an opportunity there to lower access barriers and keep our students with us at greater rates, helping with student persistence, student retention, their satisfaction as alumni. Everybody said, “yeah, those are great ideals, and you know, they’re part of our mission and our values, and we just don’t have the people or the time or the funds to be able to do that widely.” So I wrote the book UDL at Scale as an advice guide and a how-to-manual for campus leaders who think they don’t have the resources to actually get that stuff done. They do, and it’s actually when we think about lowering access barriers, not just in the classroom, not just putting the burden on instructor shoulders, but when we think about training everybody across all of our service touch points: the library, mental health counseling, academic counseling, IT, the media services team, you name it. Training all of those service touch points, to be a little bit more inclusive, to give a few more options, to lower accessibility barriers. It turns out that’s a really good financial decision. And so that was the genesis of the book.

John: In terms of that financial decision. What does the research tell us about the effect of UDL on student success, completion, retention, etc.?

Tom: Oh, this one’s great, because we actually do have research about the impact and effect of inclusive design changes, and every leader at every campus has been tasked with being a change agent. We want to move our institutions into spaces where we are making changes because they are driven by the needs of our employer bases, the needs of our students, the needs and vision of our institutions. So here’s the data, folks. Students, are five times more likely to earn a credential when they feel that they have a voice in their education, when they feel that they have a choice in how they move through their educational paths, when they feel a sense of belonging, that they are a part of the institution and its culture and its community, and rather than apart from it, when they feel like they have some agency, like they’re not just being told, “do this, do that, do the other.” That they have some say in how they move through their conversations with a college or a university, and when they have a sense of safety, that they are psychologically okay to be wrong, to explore, to get things wrong, to fail at things and come back to them. Those five things: voice, choice, belonging, agency, and safety. When students say “yes, I felt these things” during their educations, those students are five times more likely to persist and earn a credential with us. So when we think about all of the touchy feely aspects of it, that actually translates into money, that translates into tuition dollars, that translates into students not dropping out or stopping out as much as they otherwise might. And a lot of different ways that we can approach that have to do with access barriers, and we’ve talked about this in other books, but instructors aren’t the only teachers on our campus. Your media services people are teaching students how to engage with materials and content. Our librarians are teaching students how to discern good information and reliable and trustworthy information from slop that’s out there on the internet. and on and on and on, right? So our registrar is teaching students how to navigate the systems of the college or the university through the catalog and the various programs that we have on offer. So wherever we look at engagement with students, they are, in many cases, teaching engagements, and so we can design for those engagements in order to give people the kinds of choices that you heard about as we started our conversation. Multiple ways to get started and stick with it, multiple ways to take in the information that they’re learning about, and multiple ways to take action or express themselves once that they’re engaged with our service areas across all of our service touch points.

John: When you talk about people being five times more likely, is the benchmark of that students in high school or middle school and so forth, in terms of the probability of going to college and acquiring a degree? Because we do have over half of the students who enter college leaving with some sort of a credential, at least, on aggregate. So to get that number, it would seem like it would have to go back a bit to an earlier stage.

Tom: Oh, I’m actually not even talking about being that ambitious in our research. What I’m talking about is of all the current students who enroll right now, regardless of whether they are 18 years old and coming out of high school, or whether they’re 50 years old and coming back to re-skill. Out of all of those folks, you are correct. About half of them end up with a credential after 6, 7, 8, years usually. So the myth of the undergraduate who gets the degree in four years, that full-time student, is kind of a unicorn. At the same time, when we think about the 50% of students who drop out temporarily or stop out altogether, the number one reason why people don’t receive a credential is financial, “I can’t afford to continue now, or I can’t afford to continue at all.” And the research is fairly clear on this one, that the majority of people who stop out or drop out by financial reasons, it’s for $500 or less. So a lot of colleges and universities have now implemented micro-forgiveness programs, where if you owe less than $500 we’ll just write that off, and you keep going. And that’s an investment in the student, and it turns people into degree earners at far greater rates. But what’s the number two reason? If the number one reason is financial, what’s the number two reason why people stop out or drop out? It’s time, it’s the clock. We are all trying to put our educations into busy schedules. We have work responsibilities, caregiving responsibilities, military service, we just live far away from campus, whatever it is. If there’s a reason why people are busy and trying to put things together, and they don’t really fit all that well into the 24 hours of the day, those reasons are out there. So universal design for learning helps make processes go a little more smoothly. Helps students not to have to call back three times to get their question answered with IT. Helps for there to be less rework, and that’s actually one of the central arguments of the book, is that we’re actually making things a little bit more efficient for us as providers of services, and we’re also making things smoother and easier for our students in terms of process. We’ll talk about academic rigor later, but your question is an astute one, and the research that I’m citing in the book is fairly narrow, taking our existing pool of people who start, how many of them will finish?

Rebecca: Well, it’s certainly a compelling argument to implement, and we were talking about the time of not implementing. It costs a lot of time if we have inefficient and terrible processes. [LAUGHTER] We also talked about the burden on faculty trying to meet accessibility requirements, or introducing or rolling out UDL on their own. So if we think about institutional rollout, you can think about is there like aggregate benefits by rolling it out, but there’s also, seems like there might be some startup costs there too, in terms of time and effort. Can you talk a little bit about that?

Tom: Oh, Rebecca, that’s an astute question. Because right now, listeners, we’re recording this in late April of 2026 and in the United States, we just had a deadline to implement the Title II updates to the Americans with Disabilities Act. For those of you not familiar with that, by the end of April in 2026 we were all, colleges and universities, supposed to have all of the materials that we host in internet and in mobile formats comply with the Web Content Accessibility Group Standards. So there are 50 different standards for web hosted materials. And absolutely no one, and I mean no one, was going to be compliant as of the deadline. So the United States Department of Justice and Department of Education pushed that deadline back a year, and now it’s April of 2027. And it’s going to be April of 2027 and we’re going to be wringing our hands, going, “Oh my gosh, we’re not compliant still.” I guarantee that. So there’s a prediction for you, listeners. But Rebecca, your question has to do with that kind of situation. So when we, as colleges and universities, think about compliance with the law around accessibility, that means that that compliance usually falls on individuals’ shoulders. You, instructor of this biology course, you created all these materials, now it’s up to you to make them accessible. And when we think at that individual level, every individual has to be perfect in order for an institution to say that they are compliant. So if you’ve got one department that doesn’t get there, then the entire institution fails. So let’s turn that around, and when we think about Universal Design for Learning, when we’re applying it in an individual course, I actually want people to do a little bit less when we’re thinking about institution-level application. And one of the reasons why people are a little panicked now about those WCAG 50 standards is that we are asking communications professors and art historians and chemists to suddenly become experts in web design, and that’s not what the idea is about with that legal compliance issue. So when you think about people engaging in Universal Design for Learning at the individual level, the impact doesn’t have to be perfect. It doesn’t have to mean everyone does UDL to some perfect level where you get green check marks all throughout your learning management system with a particular tool. Imagine, and I’ve used this analogy before, it’s a good one for this conversation, too. Listeners, imagine that there are 20 steps to shining perfect, accessible content in your courses. There aren’t, but just for the sake of argument, let’s say there are 20 steps. Most places will spend some time, money, and effort to train a couple dozen people to get to step 18. And then we say, “You are our champions. Go train everyone else.” That’s where it breaks down, the “go train everyone else.” Other people are not being given extra time or time off, or they don’t get any support to actually focus their attention on these issues. What I wanted to do with Universal Design for Learning at Scale was to offer ways that we not get everyone to step 18, 19, 20, but to get absolutely everyone in the college or the university to step 2. What are some simple things that everyone does that become part of our everyday workflows? And so I’m indebted to our colleague Jordan Cameron at Kennesaw State University. Years ago, she came up with what she called “The Basic Four,” and that was if everybody did these four things, then some people on campus who were specialists could help with the really tough or knotty or detailed nuanced affordances, but everybody would do these four things. They would make alternative text for images that are displayed on websites in the learning management system and elsewhere. They would use semantic structure, so the heading level. putting things as titles in text-based documents. They would use captions or transcripts for video resources. And they would engage in inclusive procurement. They would engage in a process where, if the college or university is buying and supporting technology tools, they would only buy and support tools that were accessible and met those legal requirements. And if everybody did just those four things, it would, One, spread the workload around. Two, it would make sure that the expertise was available for the really challenging cases of accessible design. Three, it would allow everyone to adopt inclusive practices in a low-stakes way across their everyday workload. Meaning it’s not extra to do inclusive work, it’s actually part of our jobs, and it’s not a huge part of our jobs that we would need to fold in new things. So when we think about “the individual has to be perfect, but the collective only has to be good.” That’s what I’m aiming for. So when we think about individual courses with UDL, yes, institution-wide adoption is a little bit different than the sum of everyone just doing UDL in their own space. In fact, I don’t want to turn everyone into experts in the three principles and the nine guidelines and the 36 considerations of the Universal Design for Learning Framework. There should be people on campus who do that and know that. At the same time, everybody should have some real simple principles and practices that they can engage in that are consistent throughout all of our employees in a college or a university. So, excellent question. Thank you.

John: Have any institutions committed to a UDL-at-scale approach?

Tom: There have been, to my counting, 13 institutions who’ve actually gone from, “Hey, that’s a good idea” to “our institution is actually doing it.” I had the privilege and the pleasure of working with five of those. And I actually want to highlight one where I wasn’t the consultant, and they did it all on their own. This is Atlantic Technological University in Sligo, Ireland. And for listeners here in North America, they are a cross between what we would call a technical college, a community college, and a four-year undergraduate college. They do all of those things within their technological university mandate. What they discovered was that they were serving learners who lived all over the place in northwestern Ireland. So folks were living a 45-minute or an hour commute away by car, and a lot of their students were driving very long distances to be able to come to one of their many campuses. So what they did was they adopted an inclusive mindset, and of course, in Europe, they have the United Nations Strategic Development Goals, or SDGs. Inclusion is one of them, lowering access barriers is another one of those. So they used those as their guiding principles, and then they adopted Universal Design for Learning as a framework that allowed them to operationalize those aspirational principles. When we think about how that works, one of the first things that they did was convincing instructors and employees and staff members to adopt UDL. And of course, there were a lot of folks with their arms crossed saying, “I got along without this before. Why do I have to do this now? This seems like extra work.” The one change that they did, and I want to highlight this, their President and their board of directors said, “We want to be the most inclusive college or university in the country, and that’s a goal toward which we are all headed. We are going to put resources toward this in terms of time, people, and money,” but they didn’t do the next step that you might suggest. You might think that the next step was, “Here’s our plan, and now everybody follow it.” Their next step was to move away from “you must,” to move away from mandates and compliance and move into support. They said, “Here’s the goal. We want everybody to achieve these kinds of inclusive practices in your support area, in your department, in your study, and you tell us what it’s going to take to get to that goal, and we’ll support you.” So what happened from a top-down level was that the support that people were asking for was very different in different places. Some places wanted time, some places wanted funding, some places wanted a couple of experts to come in and help them think about things, and the senior leaders at that institution did those things. Did they do them perfectly, exactly as asked? No. At the same time, that listening attitude, that attitude of “this is a service that we’re providing to you,” it was great. When instructors would think about, “Oh, I want to make some of my content more universally designed,” and they would go talk with their teaching center folks or their learning designers, the learning designer would say, “Oh, yeah, we can teach you how to do this, or we can do this along with you.” And so that wasn’t perceived as, “Oh, here’s another law that applies to me, and now I have to do things differently.” It was perceived by instructors as, “Oh, the whole place is moving in this direction, and now this is a service on which I can rely.” Doing that actually spread the workload around and made it a slow and incremental change where people could ask questions and voice concerns and they got listened to. So that was a wonderful way that the folks at ATU actually implemented Universal Design for Learning. So now it’s part of their policy manual. It’s part of their mission and vision, and it’s also part of just the way they do things, because they actually shifted the culture.

Rebecca: What I’m hearing you say, Tom, is that the leadership at the institution modeled this practice by rolling out UDL in a UDL way, and gave folks some agency and some choice.

Tom: Yeah, students are not the only people who are five times more likely to stick with something when they feel they have a voice, choice, belonging, agency and safety, correct? So you’re making a connection, Rebecca, that I think listeners, you can also make. We as employees, whether we are part of the leadership team, whether we’re instructors and faculty members, whether we are service members and staff members. We are also learners, we are also human beings who want to have a clear reason why we’re doing the thing. We want to be able to get access to the information and the content that we’re going to use, and we want to be able to take action in a way that is meaningful for us. So we absolutely can UDL the process of adopting UDL. Yes.

John: Now, on your first book on UDL, you advocated a plus one approach to UDL adoption, and I think that’s something that many people have picked up on as making it a little bit more feasible or easier to manage. Do you make a similar suggestion at the institutional level?

Tom: It’s an unfair question, John, because I didn’t predict when I wrote, Reach Everyone, Teach Everyone with Kirsten Behling, that my idea of approaching Universal Design for Learning, not as here’s three principles that turn into nine guidelines that turn into 36 considerations that we should just approach that as plus one thinking, if there’s one way for an engagement to happen, now make one more way. That was always meant to be a starting point for UDL practice, not the entirety of everyone’s understanding of it. And I’ve been very honored and pleased to see that that plus one idea has really taken root. People know plus one as a way to start Universal Design for Learning, because it’s simple, it’s easy to understand and it’s easy to get your brain around it, and it also is small. Take one action, try one thing, do one experiment, and see what happens. Now you’re asking me to predict which of the ideas from my new book are going to take off because they are so simple or so resonant with people that they’re just going to latch on to it. I don’t know which one of those ideas is going to take off. If I had to put forward two candidates, though, one of them, we just talked about, shift from compliance to support. It seems like a very small and philosophical change. At the same time, changing the messaging around the activities that you’re doing as an institution, changes people’s perception of what those things are. It goes from “Oh, this is required of me,” to “Oh, this is a service on which I can rely.” And shifting from compliance to support narratives, those are the kinds of things that actually shift cultures and change people’s minds. So that’s one. And then a second one, I have a diagram in the book. Listeners, you can’t see it, but I’ll describe it very briefly. Think about a rectangle that is wider than it is tall. Over on the left-hand side, we’ll put “beginner” at the bottom of that. We’ll put “proficient” in the middle, and then we’ll put “approaching expert” on the other side. That’s a timeline of the learning process when people are beginners, when they’ve got a few ideas, when they’re getting it, and they’re almost experts with us. Now draw a line that starts at the very bottom, over by beginner, and goes to the very top, near proficient. We’ve just divided our rectangle into a growing and a diminishing triangle. The diminishing triangle is UDL, Universal Design for Learning, and it’s designed UDL. This is where we, as instructors, designers, developers, facilitators, when people are beginners, we want to design almost all of the boundaries and the possibilities for our learners. We tell them what to read, what to look at, what to do as exercises, so that they will get all of the foundational information. And then, as that triangle gets smaller and smaller over time, we start taking our safeguards and guardrails and left and right boundaries away. And then, by the time people are almost experts, we’re not giving them hardly any guidance or structure, we’re asking them to do. Now, the triangle that’s increasing over time, it starts out almost at nothing when people are beginners. That’s tool use. This is using AI tools, this is using calculators, this is any kinds of tools that people can use to support their learning. When they’re beginners, we don’t want them using any tools. We want them to learn things the hard way, the manual way, the long way, so that they actually struggle with and get the foundational concepts and the practice that we’re doing. Then by the time they’re proficient in the middle there, tool use grows and we start saying, “Okay, now here use tools to help you learn.” This is telling people not to use artificial intelligence to summarize their reading, but to use AI to make study questions about the reading and to use AI to help the learning process itself, but then by the time they get to be proficient, and, after that, when they get to be expert with us, then we say, “Okay, go ahead and use the tools to skip some steps,” because we’re going to use those AI tools, perhaps, to do easier or introductory level kinds of things, so that we can use our human brains to start working on more nuanced or more complex thinking processes. Now, readers, take one more line up at the top of this rectangle, start and draw a rising line from left to right, and we’re going to put one lopsided triangle on top. That is agentic UDL. And it has to go in lock step as our designed UDL diminishes over the learning process, we have to start replacing it with learner agency. We tell our learners not go read this thing and do this activity, but we say, “What kinds of resources do you need or want now? What kinds of activities would help you to learn better on this?” And we become more guides rather than dictators of the things. So when we think about that lopsided house, my hope is that that diagram will take off, because it’s a simple way to think about the entire learning process and what goes into it. And spoiler alert, listeners, it’s fractal. That learning process can happen over the course of an individual lesson, over the course of a unit, over the course of an entire class offering, over the course of an entire credential. So we can go through that process at lots of different scales. So awesome question, and even though it was unfair, I do have two candidates for the things that I hope will take off.

Rebecca: But I think you also reiterated the idea that the plus one can help an institution get started, because it’s iterative, it changes over time, it’s scaffolded, it’s all the things that you’ve been describing.

Tom: And that’s 100% on purpose. I try to offer ideas that are easy to put your brain around, and then get really nuanced and complex when you start implementing them, because you have the agency and the choice to be able to do that design in a way that works well for you and for your learners.

Rebecca: In addition to the support model that you’ve been talking about, as institutions begin to commit to UDL, can you talk a little bit about what kinds of incentive structures they may want to consider to help faculty employees stay aligned with this framework?

Tom: Ooh, can I turn your question inside out a bit, Rebecca? You talked about incentive structures. And incentives are things that we give people when they’re doing things that they don’t normally do. What I want to do is I want to make Universal Design for Learning just part of our everyday workflows. So an incentive would mean “I’m going to pay you extra to do some extra thing” or “I’m going to give you release time, because this is hard and you need some time to do it.” Initially, yes, those things need to be in place because it’s different from the way that we do things now. The goal for me, though, is that we’re not creating incentives that mentally label UDL as extra or on top of the work that we should be doing. So I want to turn the question a little inside out and say that my advice to everyone who’s a campus leader is the very first thing you can do is to put Universal Design for Learning into job descriptions for new hires and into the performance evaluation process. Make Universal Design for Learning expected of people who come into the institution as new employees, and make UDL practices part of the measurable criteria by which people’s jobs are assessed. That’s not exactly an incentive program. And if you just put them into the job descriptions and new hire offerings without actually training people on what that looks like, then you’re setting them up for failure. So having the training available before those changes happen, having an opportunity for people to be assessed on Universal Design for Learning once or twice through the cycle without it counting, so that they get the feedback, but if they haven’t been doing well with it, that they get an opportunity to say, “Oh, wait, I probably need to go attend the workshops or get some coaching or work with a mentor on this.” But I’m a big fan of letting people fail without consequences.

Rebecca: Jeez, that sounds like another UDL principle being highlighted right here.

Tom: It’s not technically in the UDL guidelines, but it does absolutely show up in guideline eight, and we can nerd out about that later. [LAUGHTER] But listeners, what we’re thinking about here is when we think about giving people the opportunity to try things out, work with them in their own fashion, but also holding them accountable, like accountability is a key driver to scaling up any change at any institution, and with Universal Design for Learning, it’s no different. The difference here is that I want to suggest give people an opportunity to have a minute to get in touch with and apply these ideas and then assess them, but don’t ding them if their assessment is poor. Say, “Okay, this didn’t work,” or “You need more time on this,” or “You need this a different way.” Let’s work with what actually is going to help move the needle here. I also want to share, the bar should be incredibly low. We’re not looking for a 100% or a green score in the learning management system on Blackboard Ally, where it’s checking all the accessibility. Universal Design for Learning and accessibility are actually different. Accessibility just means that that’s sort of the floor that’s where everybody should be doing this as a matter of course, and Universal Design for Learning is a framework that allows us to move beyond just what’s compliant. So excellent question, and thank you for letting me play with it.

John: And by making it part of the culture of the institution and part of the regular expectations, then it does build in a little bit of loss aversion, which is a behavioral economics principle, which suggests that the threat of losing a job [LAUGHTER] by not doing it might have much more of an impact than the possibility of getting incentives for doing something. That’s perhaps an overly harsh way of suggesting that.

Tom: Listeners you heard John say that, and not me, right? [LAUGHTER] But I’m also not a behavioral economist, fairly, though.

John: But one, point though, that I think is worth at least discussing is that many faculty become concerned about new initiatives that are initiated by their current administration in the institution, when institutions have a fair amount of turnover at the top. How can institutions build this into the culture of the institution in a way that is going to persist after the administration turns over?

Tom: Ooh, excellent question. There’s a very short phrase in higher education that is meant to be ironic or satiric. It’s called “new provost, new priorities.” So every few years, a new senior leader comes in and they say, “We were doing this, and now we’re going to do that.” And in some institutions, it changes every year or so, right? The campus leaders go to a leadership conference, they hear about the next big new thing, and suddenly the funding and the winds change. It’s a reasonable response if you’ve been there long enough to kind of fold your arms across your chest and say, “I’m just going to wait until the winds change again and change again, and maybe I won’t have to actually do an awful lot with all of these big new priorities.” So there’s an individual and a collective way to respond. Individual way is to play to people’s own interests. And we’ve talked about this in other conversations and books. It’s still true. Ask people where they’re already finding challenges or gaps. Where is it that you give the directions for an activity and you get an email 400 times from your students, all asking the same clarifying question about the direction? Where is it that you give a splendid lecture, and then everybody gets that concept wrong on the exam, you’ve got to reteach. Where are those places where people are already going to do a little work to redesign or think again about how they’re approaching a particular idea, concept or process? Those are always good places to start thinking in a more inclusive way, because you can lower some barriers, not only for the learners, but also for yourself. So when we think about UDL, the selling point of UDL for individuals is that a little bit of effort now actually saves you a lot of effort later. And you know where you’re putting in all of that repeated effort that just seems to be a time sink. So in your own career, think about that. And then when we think about, at an institutional level, if we’re going to have leadership change over, there are three things we can do right away, and this is all chapter five, people. One, make sure that you are enshrining Universal Design for Learning in your policy manual, this takes the longest amount of time. When you’re involved in shared governance, things do not move fast. But if inclusive design is actually part of the identity of the institution and it’s part of the mission that we’re on. Folks at Greensboro College in North Carolina actually did adopt this first into their mission, and worked back from that into practice and into making it part of their everyday work. Two, when we think institutionally, we actually talked about this as well. Putting Universal Design for Learning into those job descriptions and performance evaluations means that you’re changing culture by fiat. And John, you said it in a different way than I would have [LAUGHTER] in terms of risk or loss aversion. But when we think about risk, we can think about risk in negative ways and positive ways. So the negative risk is, “Oh, if I don’t do this, I might lose my job, or I won’t get my promotion.” But the positive risk aspect of it is, if we do this, then we are likely to have more students stick with us. If we do this, we are more likely to have fewer students asking the same question over and over and over again or tripping over the same stumbling blocks intellectually or practically. And then the third thing that we can think about when we’re working at scale is that institutional culture, the people who are there the longest typically aren’t the senior leaders, it’s typically the folks in operations. So the folks who are running the departments and the support units, we’ve got the institutional memory. So in trying Universal Design for Learning in your practices at your unit level, put it in the handbook, put it in the manual, put it in the new employee onboarding that you do. That’s where it’s going to stick, because provosts and presidents come and go, it’s a truism in higher education. And the folks who are here for the long haul, Universal Design for Learning helps us do our jobs more efficiently, with less stress, less anxiety, and it helps our students to be more successful. It does require effort and work, and to the extent that we can make time, put money, assign people to those tasks, the better. But even if we don’t have a lot of any of those three things, we can still shift how people act.

John: And if students are learning more, it tends to be much more enjoyable for faculty.

Tom: Wow, I didn’t put that in the book. Seriously, I didn’t put that in the book because one of the biggest arguments that we have for Universal Design for Learning is that it helps students to learn more effectively, and I guarantee you that that is not an argument that moves anyone to open their budget. So this book, like I was saying at the very beginning, is a very different kind of argument to the one that I have produced in other books. In other books, we’re talking about social justice, we’re talking about efficiency, we’re talking about efficacy for the students. In this book, we’re talking about, “Yeah, this actually makes sense, because it means that we’re spending our money more effectively and wisely, and helps us with reducing rework and making processes more efficient.” This is language that I’m not used to using all the time, and it’s language that most campus leaders are very used to hearing.

Rebecca: It’s important to have campus leadership on board to make this a cultural shift, but it doesn’t have to start there. It can start from anyone at a university who’s committed to this work. Do you have some strategies that individuals might use, collectively or individually, to start to nudge an administration to consider adopting the UDL framework for the entire institution?

Tom: We’re actually heading in that direction in our conversation already, aren’t we?

Rebecca: We are.

Tom: So if you’ve got a grassroots effort where you can show that Universal Design for Learning actually does have an effect on student retention, or more tuition coming in, or fewer people dropping out. Collect those data and share that information with your senior leaders and share it with language that they understand. That is task number one. Talk about its impact on the budget for your unit. Or if it’s not a budget impact, talk about how, instead of putting out little tiny fires over and over and over again, that adopting UDL has meant that you’ve addressed some regular concerns and gaps that happen every day, and that’s actually freed up your staff to go after bigger or more complex challenges and goals. So it actually does help us to shift how we spend our time. Listeners, I don’t know what area you work in, in a college or a university, I guarantee that you have the story of the thing that you spend all your time doing once a year. So for example, I worked with some colleagues at D’Youville University in Buffalo, New York, and the head of their Disability Services Office. She was one of the champions who went through a process of learning more about Universal Design for Learning. She shared that on average, she was chasing 70 plus students every semester because they had to actually actively request their accommodations every semester, it’s not just like you get your accommodations and then you’re good for four more years or six more years. So every semester, she would be sending out email messages to people’s .edu address saying, “Hey, quick reminder, you have to request this every semester.” And she spent literally 80 to 120 hours of her own time, and then 80 to 120 hours of the four people in her office chasing down these requests. And then they made one UDL change. When people initially sought a disability accommodation, they asked the students, “We’re going to send stuff to your campus address. Would you be willing to share your cell phone number with us so we could text you? We promise we won’t spam you.” 90 plus percent of the students on intake shared their cell numbers, and when she sent out the reminder, “Hey, you got to do this every semester,” it went to their email address and it went to their cell phone number, if they had shared it. They literally went from chasing 70 or more students over the course of weeks every semester to two. Two people needed reminders more than once. So making that UDL choice about giving people more than one way that they can get the information helped them to actually reduce a huge time barrier for their staff, and that meant that their disability service staff weren’t playing phone tag and email tag with people, but they were able to actually do the work that they were hired to do, which is helping students to overcome barriers in the classroom environment. So when we think about convincing the administration, show them examples like that. “Hey, we saved you a whole bunch of money, and now we can work on more complex challenges with the same number of staff members.” That’s an argument that a provost is going to listen to. So show that it works and use language that they understand: budget, retention, persistence.

John: We always end by asking, what’s next? So what’s next for you, Tom?

Tom: My focus is always on quality in higher education. So I’ve got two new books coming out after UDL at Scale, which is coming out in August of 2026. Sometime in 2027 I have Evaluating Teaching in the Digital Era with Annie Taylor and Jean Mandernach. Then in 2028, it’ll be Peer Observation Made Practical with Lauren Barbeau and Claudia Cornejo Happel. And sometime in 2026 or ‘27, I worked with Chris Hromalik at the SUNY System, and we have an article coming out on Universal Design for Learning in Project Management. Plus, I am now a full time professional advisor, so I’m enjoying working with colleges and universities on really complex problems. Listeners, if you go to my website, it’s just ThomasJTobin.com, there’s a free UDL book there on further education and training, John mentioned it in our opener, and you can download that. And if you’ve got a wicked problem that you’re trying to figure out that has to do with the quality of our teaching or access to education, I’d love to hear from you. So thank you again for having me on the program. I love Tea for Teaching. I’m a regular listener. I’ve got my Tea for Teaching mug, and I’ll look forward to where this all goes down the line.

Rebecca: Well, thanks so much, Tom, we always enjoy talking to you and brainstorming how to make some good institutional change.

John: Yes, It’s great talking to you again, and I’m looking forward to seeing you at one of the next conferences coming up.

Tom: Yep, and listeners, if we cross paths in three dimensions, just look for the tall man with white hair, glasses and a giant black mustache and say hello, I’ll be glad to talk with you. Thanks everybody.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]