458. A Neurodiversity Framework

Institutions are increasing their focus on creating inclusive environments. In this episode, Megan Kohler joins us to discuss a course design framework she developed to support neurodiverse learners. Megan is an Assistant Teaching Professor and Learning Designer in the John A. Dutton e-Education Institute at Penn State. She is the recipient of the Marion G. Mitchell Award for Innovative Teaching and is conducting research, funded by the Schreyer Institute Scholarship, on supporting neurodivergent learners in higher education. Megan is one of the co-editors of The Multi-Disciplinary Instructional Designer: Integrating Specialized Skills into Design Toolkits.

Show Notes

Transcript

Rebecca: Institutions are increasing their focus on creating inclusive environments. In this episode, we discuss a course design framework to support neurodiverse learners.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Megan Kohler. Megan is an Assistant Teaching Professor and Learning Designer in the John A. Dutton e-Education Institute at Penn State. She is the recipient of the Marion G. Mitchell Award for Innovative Teaching and is conducting research, funded by the Schreyer Institute Scholarship, on supporting neurodivergent learners in higher education. Megan is one of the co-editors of The Multi-Disciplinary Instructional Designer: Integrating Specialized Skills into Design Toolkits. Welcome back, Megan.

Megan: Hi. Thank you so much for having me here today. It’s great to be talking with both of you again.

John: We’re very glad to have you back again. It’s been a while.

Megan: It has.

John: Today’s teas are:… Megan, are you drinking any tea?

Megan: Oh, I’m always drinking tea. [LAUGHTER] I have a lovely new tea, and if you’re willing to indulge me in a short story, I’d love to share that with you. So the tea that I’m drinking today is called Paris, and it’s by Harney and Sons. And I discovered this tea completely by accident. It was Mother’s Day, and my husband decided to take me to this lovely little creperie. It’s like 45 minutes away, and I only drink tea, not a coffee drinker. And we get there, and they don’t have my tea, and so he grabbed this one, [LAUGHTER] and I tried it, and absolutely fell in love, and this one has become my go-to tea. So I highly encourage you all to try it. because it’s absolutely delicious.

Rebecca: Such a great story! I had a tea emergency that John solved for me today, so I’m drinking a decaf Irish breakfast tea.

Megan: Oh, I love Irish breakfast! It’s so good. It’s got a lot of great flavor to it.

Rebecca: Yeah, nice and bold, nice and bold.

John: And we’re in the midst of getting this cloud of smoke from Canada from the fires that are burning there. In honor of that, I have some lapsang souchong tea, which is a smoked tea. So it just adds to the ambiance of the day here.

Rebecca: So it’s smoky inside and out. [LAUGHTER]

John: Equally.

Rebecca: So we’ve invited you here today, Megan, to discuss your framework for supporting neurodiverse learners. First, though, can you tell us how to define neurodiversity?

Megan: That’s a great question, and it’s also somewhat of a difficult question, and I’ll explain why. So first and foremost, I try to keep things very simple. I think the more simplistic we can keep things, the more approachable they become for people, and that’s something that I really want to keep in mind in doing this work. The more approachable something is, the easier it is for people to buy into it or to understand it. And so I think of neurodiversity as largely just differences in the way that our brains function, most often because of some form of condition such as autism or ADHD. Those are the two most common ones that people think of, but it goes so far beyond that. It also includes things like post-traumatic stress disorder, and a lot of people typically think of like military veterans who have had some type of traumatic experience. But it’s more than that too. You can have trauma from a wide variety of things in your life. So there’s cancer survivors who you talk about their experiences having PTSD from cancer treatments that they’ve undergone. It also includes other conditions such as dyslexia. Dyslexia is actually the most commonly diagnosed condition over autism and ADHD, but yet, autism and ADHD are the ones that people typically go to. And so, the reason why I think it’s a little challenging to come up with or to have a definition of neurodiversity is because there are so many different perspectives that end up tying into this. So we have the medicalized version of neurodiversity and what that means, and then we have the social movement and what that means, and then in between those two sort of opposite ends, we have the lived experience that fits between those, and that lived experience is something that people hold very dear and very close to them, and that can impact the way that they see and understand and perceive certain things about neurodiversity and the many conditions associated with it, so it’s not as simple and as straightforward as it seems.

John: So there’s quite a bit of diversity in terms of neurodiversity, which makes some sense, I think. So how did you first become interested in focusing on support for neurodiverse students?

Megan: It was about eight years ago. My daughter was first diagnosed, and it’s a very long story and heart wrenching for me. I’ll try to summarize it briefly, but essentially we had had her in a daycare. We had her in a Montessori school, and that was after having her in a different daycare, and they kept telling us your daughter’s really, really smart. And we were like, “Oh well, thank you.” And they’re like, “No, no, no, no, she’s really smart.” And it was actually their encouragement that we sought out a more academically focused program, such as Montessori, and that was kind of always in the plan to begin with because my husband and I both work at Penn State University and education is something we’re very passionate about. So we had had her in this Montessori school, and we went there for an open house, and they started telling us all of these wonderful things about her, and they had her demonstrating all of these things that she could do for us. And then the teacher at the end of the night said, “I would really love if you’d come back in and we could have a more detailed conversation about your daughter.” And so I got really excited. I was like, “They’ve just told me how brilliant she is and showed me all these wonderful things that she can do.” And so I went in for the conversation, and as it went on, they just started sharing more and more of these things, and they were like, “We notice that your daughter isn’t engaging in cooperative play, and that’s something that we typically see at this age and stage of development.” And so they just started talking about all of these different things that they had observed with her, and then towards the end of that conversation, they were like, “We think that she might have autism, and we think that you should get her evaluated to see if this is the case.” And so it turns out that she doesn’t have autism; she has ADHD. But it was through that very long journey of trying to get her diagnosed, and then eventually getting the diagnosis that, as a mother, you want to do everything you can to try and support your children, and so I dove into the research and started to recognize two things: one, that I also had undiagnosed ADHD my entire life, and the light bulb went off, and I was like, “Oh my gosh, this explains so much,” and that’s very common in terms of how people are starting to recognize their neurodivergence, but then also something else that occurred to me at that point in time was that there’s a lot of support in K through 12, but then once students were going to college, that support largely dropped away. Now again, this was eight years ago, and we’ve seen some tremendous changes occurring in higher education, especially around the amount of support that’s being offered, but there’s still so much work that needs to be done, and that’s actually one of the reasons why I designed the framework that we’re going to be talking a little bit about today, is because everything I saw was very fragmented, and there was no comprehensive, clear pathway that spoke not just to how do we support our learners from a learning perspective, but how do we also address some of these more personal belonging needs?

Rebecca: Can you talk a little bit about why the framework needed to be created? In other words, why don’t traditional methods work for neurodiverse students? What challenges do neurodiverse learners face?

Megan: So, first of all, I think that there are actually a lot of strategies that do work. It’s just when you’re looking at the ocean that is learning, how do you whittle it down to find the strategies that are most supportive of neurodiversity, and that’s essentially what I’ve tried to do for educators, is to dive into that research and to figure out, okay, well, what are their needs? Where do we start? How do we even go about this process? And that was largely driven by my daughter because I wanted to make sure that once she gets to college, she’s been supported too. So to sort of cycle back to your question, there’s a lot of strategies that do work. Challenges that neurodiverse learners face in the more typical or more traditional classroom consist of a number of different things. First and foremost, there is a lack of understanding and awareness about what neurodiversity actually is, and there’s a lot of stereotyping, and we need to be very careful with that. And we’re also starting to see, because it has become more prominent, there’s a lot of labeling that is going on. I’ve heard people in my different social circles, and they will be like, “Oh, you know that person? Oh, they are so on the spectrum,” and that’s exactly the type of conversations or statements that we want to avoid making. You don’t ever want to do that to somebody, and you don’t know what their circumstance is. You don’t know whether or not they have a diagnosis, or they could simply be a person who is just a little quirky. That doesn’t mean that they are neurodivergent, so we need to be really, really careful about saying things like that and putting labels where labels might not apply. But some of those challenges, I think the most prominent one from my perspective is really sort of a social component. It’s really making sure that our diverse learners a re supported and made to feel like they belong. Belonging is such a huge piece of the experience that we’re offering nowadays, and it’s really interesting because educators and institutions as a whole are starting to recognize how much the belonging piece impacts whether or not a student chooses to stay at an institution, and not only that, there’s a lot of other research that shows us that when we start to engage students in things like social emotional learning, they’re building those skills socially, but they’re also ending up developing and reinforcing other skills that are critical to their academic experience.

John: What are some strategies that we can use to create the sense of belonging, so that students do not feel like they have some sort of a deficit?

Megan: Yeah, that’s a great question. [LAUGHTER] This is one of the reasons why I love talking with you both, you both think so deeply about topics. So some strategies that we can think about, and this is tapping into the model a bit, things like co-designing. So when you sit down and you work with a student, or even if you share an assignment, for example, with the entire classroom, allowing those students to come in and give their thoughts and ideas on the assignment, see how it might need to be altered in order to make them feel as though it’s something that’s achievable and attainable, and to revise instructions that maybe don’t make sense to them from their perspective. Allowing that exchange to occur goes a long way to making a person feel as though they are accepted. We can also talk about things like normalizing if somebody does have some type of a learning difference or a disability in which it can be supported. When I give my keynote talk about this, there’s an example that I give about how using transcripts from videos or even creating audio recordings of the lectures, so having the ability for every student to be able to not just read the text but also to download an audio recording. I had a faculty member who did that several years ago, and the students loved it. They just kept writing her all of these notes saying “this was so fantastic, so wonderful. I was able to download this, and I could listen to it at times that worked for me.” And when you do that, you take some of the stigma away from people who need that different format, and now you make it something that is really cool because now it’s just this different format. It’s this different modality. There’s almost like a novelty to it to some degree that I think students really enjoy and appreciate. So those are a few quick examples.

Rebecca: So we started to talk a little bit about the valued minds model that you developed, but we haven’t really dug in yet. [LAUGHTER] So, can we talk a little bit about the system that you’ve developed and the framework and what that looks like?

Megan: Yeah, absolutely. And you probably noticed I’ve been like holding back a little bit [LAUGHTER] because I’m like, this is in the framework. [LAUGHTER] It’s all in there. [LAUGHTER] Absolutely. So the framework really has three core components to it. So the first is learner agency. The second is system design, and this is where we tend to address a lot of our systemic barriers that occur. And then the last piece is belonging. Now within that, there’s several different layers or different components that should be taken into consideration. So within learner agency, the first thing that we need to do when we’re designing for neurodiverse learners is to start with some foundational components, and that is really about integrating strategies that support the executive functioning skills. So there is plenty of research that exists that indicates that if you have a neurodiverse condition or a neurodivergent condition, a lot of times there will be differences in the executive functions. And so, when we design for those, what we’re doing is sort of leveling the playing field. So, if somebody has a known issue with working memory, and we’re bringing in different types of supports to help balance that out, so that there’s not as much cognitive load being expended on their part. Well, now you’re instantaneously putting them in a place where it’s going to be easier for them to engage in the learning process, rather than for them simply trying to decode or to follow along in the way that you had originally intended them to. And then we bring in this accessibility piece after those foundational components. So making sure that if somebody has some type of a difference, and I think that this is really critical for those more applied conditions such as like dyslexia and dyscalculia. So bringing in those accessibility pieces to make sure that there are screen readers that can communicate that information to them or help support them in the exploration of that content. That’s really the next level or layer. And then when we talk about sort of the custom design, that’s a piece that fits in the top. And this is where I see a lot of people approaching it in the inverse, a lot of people try to start with those custom strategies and work sort of at that level. But if you do the foundational components and the accessibility piece first, you’re setting everyone up for a much better success. So then, when we get to that custom piece, this is really about sort of meeting people where they are and taking a look at who they are and what their skills and abilities are, and very specifically what their strengths are, and so one of the things that we need to be really aware of is the fact that a lot of neurodiverse individuals have received a lot of negative comments and a lot of negative feedback throughout their entire lives. In fact, it’s estimated that by the age of 10, a child who has ADHD – and these are metrics just for ADHD – it’s anticipated that that child has received 20,000 more negative messages than their neurotypical counterparts. So that’s something pretty significant. And so then when we think about the language that we use and the way that we engage with those learners, we need to be helping them recognize what their strengths are, and we need to be speaking in a way, whether it’s person-to-person conversation or even through our text. And a great example that I love to use is when we’re designing our rubrics using strength-based language in our rubrics. So, for example, you’ve probably seen the rubrics that use categories that range from excellent to poor. But what happens when you put the label poor on someone’s work? That’s a label that indicates that they’re lesser than, that they’re not as good, and it’s kind of an endpoint when somebody tells you that your work is poor. So we need to be shifting that to using language that’s more along the lines of expert to novice, because if somebody tells you you’re a novice, your work is at the novice level, what does that immediately make you think of? It makes you think of a trajectory or a journey; that this is a starting point, but that you can grow beyond what you’re currently facing. So, strengths-based language is absolutely a critical component of that. So, those are the three layers of learner agency.

Then we talk about system design, and this is where we get into things such as accommodations, training for faculty, things like the hidden curriculum. And I don’t know how many people necessarily think of hidden curriculum as being something that we need to think about when we’re talking about neurodiversity. It tends to be something that we think about a lot with first-year students, but it’s absolutely a component of neurodiversity, and the reason for that is because a lot of it stems from interpersonal relationships. That’s how you become aware of a lot of those components of hidden curriculum is through your interactions and those relationships. So, making sure that we’re addressing that as part of this process as well, and then we also have other things in there too, such as stigmatization, making sure that we are trying to reduce stigmatization of our neurodiverse learners as much as possible. So that’s all part of that system design component. And the challenging piece with that is that some of this that you can do in your classroom, and some of it occurs at the institution level, so there’s a wide range of approaches that can be integrated within that. And then this last piece is belonging, and where we start with that is with the concept of validation. So making sure that we are bringing in inclusive strategies that, as I mentioned before, not just help the students feel a greater sense of belonging, but can also help them develop those executive functioning skills. And the great example of that is social emotional learning. But there’s other things that we can do too in terms of making students feel validated and supported, such as peer-to-peer confirmation. You’re making sure that you’re enabling students to give each other positive feedback and say, “No, you did really great on that work. I really valued what you did here, and the way that you did it was really great.” So integrating strategies such as that, and then we get to recognition, which is the next piece of the framework. And so, recognition and accessibility kind of parallel one another as those foundational elements, the executive functions, and validation parallel one another. And we talk about recognition. We’re talking about that normalization of the different practices. So as I referenced that strategy before about the audio recordings and just normalizing it, making it feel like diversity is a part of everyday life, and it just belongs here, and it should be recognized and it should be valued. And then the last piece of belonging is this sort of modeling, if you will, of acceptance and what that can look like in the classroom. So we talked about co-designing information previously. So that’s a great example, but also making sure that we’re making students feel like your classroom is a place where it’s okay to have those sort of fast-fail experiences. So nothing’s ever a mistake, nothing’s ever a failure if you’re learning from it, if you’re growing from it. So making sure that we are creating dynamics such as that, and if we do that, making sure that we’re also tapping into things like a student’s motivation, helping them integrate things that are really very important to them, and I think if we integrate all of these pieces together, I think we can create our classrooms to be those more inclusive environments for neurodiverse learners. Because we know that some of the more traditional inclusive practices don’t necessarily apply. And one of the things that I always get sort of stuck on a little bit when it comes to inclusive strategies is that one of integrating research papers or using articles published by authors who have the same types of differences, whether it ends up being a disability or if there’s a different type of ethnicity. But how do you do that when the difference is something that’s invisible? So how do you know whether or not the author of a paper has autism or ADHD or dyslexia or dyscalculia or one of the many other conditions? And so that’s why I think a framework like this is something that’s needed because there’s shifts, there’s nuances that we need to address in order to create these dynamics that are neuroinclusive.

John: Now you mentioned that some inclusive practices don’t work as well for all students. Could you elaborate on that just a little bit in terms of some things that people routinely do that may not work as well for students with various forms of neurodiversity? You gave the example of papers as one case, but could you give us some other examples?

Megan: I think where there’s a lot of confusion is actually… this is another framework that I’ve been researching and working on… but one of the things that I have seen occurring is faculty not necessarily understanding what is inclusive support versus classroom management versus learning support and so on. So I was attending a conference not long ago, and there was two very lovely women who were leading this session, and at one point they pulled up this Padlet and they were like, “What strategies do you think are going to be great instructional strategies for the students in your classroom?” And I saw people just contributing things to the board, and they were like, “Yes, all of these are wonderful.” And I just sat there going, “No, that doesn’t align with this. And hmm hmm…” and I saw a lot of that confusion. So the paper example that we gave just a second ago – that’s typically an inclusive strategy, and that was one of the things that was mentioned in this conversation. And I was like sitting there thinking the same thing that I just communicated to you, like, “Okay, but how do we know? [LAUGHTER] How do we know whether or not this person has a neurodiverse condition?” But yet it was listed in one of those strategies. So a lot of times we’re relying on information that we know from other circumstances, and we’re not necessarily creating those subtle shifts that need to occur. And so another great example is classroom management. So I see this a lot. One of the people in this session listed brain breaks as a great strategy, and the learning designer in me sat there going, “Oh, wait a second. Are we talking about breaks for the sake of taking breaks?” Because we know that there’s no research to back that up, and if we’re talking about breaks for the purposes of taking time to reflect on information, then yes, that does work. But what they were talking about were brain breaks in terms of getting up and moving around the room if you felt like you couldn’t focus on the information that was being shared at that moment, and apart from the fact that there’s research that says that that’s not actually how our brains work, and that’s not the best way to help us develop the ability to stay focused on information for a long time, that also tends to align more so with the classroom management side of things rather than the instruction side of things, so there’s a lot of misunderstandings and a lot of confusion around what strategies actually align with what sort of categories of support, if that makes sense. So yeah, it can be very confusing unless you spend a lot of time researching this, and I think one of the things that made me very concerned at the end of the session was that the two women who were leading this were like, “And we’re going to be writing a book chapter about this,” and so [LAUGHTER] that goes back to kind of the concept of right now neurodiversity is kind of the wild west. There’s a lot of really good advice being put out there, and there’s a lot of advice that we need to evaluate a little bit more deeply. But it’s being put out there and could actually be harmful in some ways, and so we need to have more conversations, deeper conversations around that.

John: And that’s where that co-creation that you talked about before can help to address the needs of the specific students in the room.

Rebecca: Well, one thing that I’m observing or thinking about as you’re introducing the framework is that it has components from a lot of other existing frameworks, and you kind of hinted at this a little bit. Things from universal design for learning, TILT – transparency in learning and teaching, just good learning practices to make sure we stay motivated around agency and things like that that are certainly in UDL or Universal Design for Learning, accessibility practices, relationship-rich practices, but it’s not inclusive of everything in those frameworks. Can you talk a little bit about the process you use to pull what would be most important for neurodiverse students from those existing frameworks, you kind of integrated them all together and put them in one place. Can you just talk a little bit about the process of doing that?

Megan: Yeah, it was a lot of research, [LAUGHTER] a lot of research and reflection and evaluating this. I think the thing is, is that there’s so much amazing work that has already been done around the process of learning. We have all this great research about the brain and how the brain functions and what it needs in terms of learning and how we can best support the learning process for our students, and it was really just about diving in and just trying to understand what the unique needs are for neurodiverse learners, and then pulling the pieces that best supported those needs, and the reality is, there’s no one size fits all. There never is, and so I don’t see this as something that should ever be thought of as trying to replace UDL or trying to replace TILT . That’s not what anything we do in education is ever about. It’s a tool next to several other tools that, when it makes sense for you to pull this in, pull it in. Maybe there’s a faculty member who is tied very strongly to UDL, and they know their students very well, and they know that UDL principles are going to work for them. Then they should use UDL. This is a tool in the toolbox of support, and at the end of the day, there is never one perfect solution for everything, for every learner, for every need. But that’s hopefully one of the things that I would really love for people that when they look at this framework, they see that there is space for those unique needs, and I think that that’s something that maybe is a little more challenging to find in some other models. But when I talk about that custom piece, one of the things that having undiagnosed ADHD my entire life, there were so many times when I felt like I wasn’t measuring up and I wasn’t good enough. And in some circumstances, I had people who were very supportive, and they would say, “Megan, you can do this. You can do this. Just let’s figure this out. I’m going to help you, however you need help, and we’re going to figure it out.” But that was not always the case. I had a lot of other people who saw my differences as shortcomings, and they took that as an opportunity to berate me and to tear me down, and it’s those unique instances where you have somebody that steps forward and says, “I see something amazing in you, and I want to help bring that out. I want to help you shine, because I know that you’re capable of this.” That’s that custom piece of my framework. That is where I think we have the biggest ability to really make students recognize their own strengths, not just as students, but as individuals with something incredible to contribute to the world, we need to be shifting this paradigm from a deficit-based mindset of difference as lesser to a place of different is unique with incredible opportunity behind it. That is the goal of this framework. Sorry, I feel like that was very deep, but you want to know why I do this work. That’s why.

Rebecca: It sounds like the custom layer that you’re talking about is really important for what can really sometimes be competing needs, like very different kinds of needs that, in some ways, like on the surface, can look like they’re directly in competition for one student versus another because this student needs X and this student needs Y. But if you had provided the opposite to the other, [LAUGHTER] it wouldn’t help.

Megan: Absolutely, and I’m so glad that you mentioned that because that’s one of the other things that I’ve noticed. So when I talked earlier about the fragmentation that’s occurring around neurodiversity, that’s a large part of it. So you have people saying these are the 10 strategies that work for individuals with autism, and these are the 15 strategies that work for people with ADHD, and then a large part of the other neurodiverse conditions are just completely overlooked. And so in creating this framework, I wanted to say that’s not the way that we should be going about things. We need to be looking at the common threads across all of these conditions, and that ends up largely being the executive functions, which is why that’s the starting point. But then, how do we look for those common threads, those common building blocks across these different conditions, while at the same time allowing there to be this degree of flexibility, so that we can meet students where they are. A great example of those differences that you just mentioned… So my daughter has ADHD. I have ADHD. We have completely different needs, and yet we share the same genetics. [LAUGHTER] And yet our two conditions are very, very different. She’s brilliant. She is absolutely brilliant. She’s in the gifted program at school, and she’s working at grade levels that are two grade levels higher than where she’s currently enrolled. She’s looking at going into fifth grade, but last year when she was in fourth, they had her working on math problems and reading books that were at the sixth grade reading level and the sixth grade math level. I run away from math. [LAUGHTER] I run away from math as far as I can possibly get, because I just don’t have the working memory to be able to hold the numbers in the right places. But yet she does it, and she does it in her brain. And it’s so fun because my husband’s an engineer, and we’ll be driving in the car, and he’ll just randomly shout out a math problem, and they’ll see who can answer it first. And both of my children already have the answer, and I’m like sitting there with my calculator [LAUGHTER] like hidden down beside me because my brain’s just not structured that way, but hers is. So even when you have a specific condition like ADHD, not all of those strategies are going to work for all of your students with ADHD, so it’s just so complex. It’s so complex, which is why we need to have that flexibility.

Rebecca: I just want answers, Megan. [LAUGHTER]

John: But the answers are different for each student.

Megan: [LAUGHTER] They are. They absolutely are.

Rebecca: It’s really like the underlying subtext always is like “Ah, why is this so complicated?” [LAUGHTER]

Megan: Because humans are complicated. That’s the bottom line: is that we are all unique, and within that uniqueness, there is a beauty, but there’s also complexity, and it gets really hard when you’re trying to be an educator who’s working towards addressing all of that complexity. I don’t know that we give our educators enough credit for how challenging it is to take a single message and ensure that 30, 60, 120, 200, 350 people in the same room all get an accurate understanding of what that message is or what that concept is.

Rebecca: You’re kind of asking for miracles. [LAUGHTER]

Megan: Yeah, it is like asking for a miracle. But that’s also where we, as human beings, that’s where our neuroplasticity or our cognitive functions really come into play because we can have that flexibility of thinking. We can learn new ways of communicating, and once you learn how a specific faculty member communicates things or what their expectations are, you, as a learner, you can be quite successful, but the learning curve to that is not going to be the same for everybody. And so, the more that we can do to address some of those differences, we’re only going to make that process easier on everybody. So, the days of I’m just going to sit down, and I’m gonna create my course and write this lecture, that’s not sustainable, not for the way that we need to be thinking about our students. And a lot of these conditions, the diagnoses are increasing. When I first started, even just a few years ago, and so much has changed. So much has changed. But even just over the past few years, we’ve seen the diagnosis of autism in the general population go from approximately one in 100 down to one in 89, and now the census currently is that I believe it’s children under the age of eight, it’s one in 63 are diagnosed with autism. So it’s going to continue to change and to flux, and the hope is that we also continue to have the compassion and the desire to want to change and flux with it.

John: And I think an important takeaway from what you’ve said is that when we give students encouragement that can make a really big difference, but when we say anything that discourages students, that can also have an extremely large impact. And we have to be really careful in how we phrase things and how we communicate to students, so that we do focus on their capabilities and their possibilities.

Megan: Absolutely, and I think there’s one important thing to keep in mind with that is that it’s not about walking on eggshells. It’s not about “oh, I can’t say this or I’ll upset somebody.” It’s about learning to think about how we communicate differently. It’s learning to think about taking our words from a place where they communicate as though somebody isn’t measuring up, to a place where we are highlighting those strengths and making them see the value in themselves and their future in what they can achieve. And so it’s a subtle shift, but it’s an important one, and one of the reasons why I think it’s so important. I don’t know if either of you have ever heard of rejection-sensitive dysphoria, more commonly known as RSD. It’s a significant condition in which a person really perceives as though the interactions that they have with people, it always has a negative sort of perspective or a negative spin on it. You know, somebody says, “Hey, everybody, we’re going out for coffee,” or “Hey, everybody, we’re going to a party,” and the person sits there and thinks, “Well, of course, they’re inviting everybody else, but they don’t mean to invite me. They must have forgotten that I was here. They didn’t see me or didn’t recognize that I was in the room.” So there’s very much a sense of separation that can occur around that too, and there’s some instances of RSD that are so severe that it actually causes physical pain for the individual. So it really is about shifting our language so that we are raising people up rather than tearing people down, and this is something that I have conversations with my kids all the time. Do you have children?

Rebecca: Yep. [LAUGHTER]

John: Yes.

Rebecca: Earlier, you were describing my car. [LAUGHTER]

Megan: Yeah. So we have these conversations whenever there’s a fight. The kids are in the back. “But he did this to me,” and “but she did that to me,” and it’s “Okay, okay, I hear you. Instead of accusing one another and saying what they did was wrong, how can you change this? How can you rephrase what it is that you’re thinking and feeling in a way right now that communicates what is happening or what you feel is happening, but that you’re not tearing the other person down as a part of that process.” And we do a lot of reflection in our car rides. [LAUGHTER] My kids will say one thing, and I’ll say, “Okay, let’s pause. How could you rephrase that?” [LAUGHTER] Because the rest of the world isn’t going to teach them how to do that. That’s my responsibility as a parent, and so we engage in those conversations a lot to be more aware of the impact of our words on other people.

Rebecca: We talked a little bit earlier about things that we can do in the classroom, but also kind of a more institutional or global approach or systematic approach that needs to happen. And the conversation that we’re having right now makes me think about one of the ways that these things can be modeled is also at like the employee level at an institution, which trickles down also to students. That this kind of a framework can be used in other contexts outside of a classroom setting that enhance the learning environment overall.

Megan: Absolutely, yeah, and I need to do a little bit more research because there are definitely nuances between the classroom versus the work environment. But I think a really quick piece that could be swapped out is take a look at hidden curriculum and replace it with office politics.

Rebecca: It’s all the same. [LAUGHTER]

Megan: To some degree, to some degree.

Rebecca: Yeah, yeah.

Megan: It is. [LAUGHTER] But yeah, and that is the thing that I hope to be able to do with this work is to carry it forward because it’s so interesting when I have these conversations. Practically every single time I’ve had a conversation about this framework, that question has arisen: “Can we use this for the office environment too?” I’m like “Yeah, with some modifications, but yeah, absolutely.” So you have a coworker who continually struggles with deadlines. There’s your executive functioning skills. It can absolutely be applied. I just need to sit down and make some of those modifications and some adjustments with that. So stay tuned. [LAUGHTER] It’s coming. [LAUGHTER]

Rebecca: Seems pretty workable at the moment. [LAUGHTER]

Megan: Thank you, thank you. [LAUGHTER]

John: We always end with the question: What’s next?

Megan: What is next? I’m currently working on a co-edited book with the amazing Adam Davi, who’s a good friend from our days working with the Online Learning Consortium, volunteering with them, and this book is titled Advancing Learning Design in the Age of AI. And what we are hoping to capture within this book is the conversation around now that AI is essentially commonplace and everybody has access to it, and we’re seeing more and more people leverage it for designing their courses, what does that relationship look like as a learning designer? Where do we shift? How do we shift? What do we need to be realigning in terms of our workflows and our responsibilities so that we can leverage AI to better support the work that we do, but also helping understand that AI is not a replacement for a learning designer, and it never will be. And if you know and understand anything about simply writing learning objectives, there’s a great example just within that. So I don’t know if you’ve asked any AI system to write learning objectives for you. They are lacking. [LAUGHTER] They do not follow a very traditional learning objective format, and it can take you quite a bit of back and forth to even get that to a point where AI is getting something even close. So there’s a lot of misunderstandings. People are putting information in or asking ChatGPT to come up with, “Oh, give me a course outline” or “write this assessment for me,” and it does it. But is it what you actually intended? Is it what the students actually need? Is there alignment with your instructional goals? Is there alignment with the overall goals of the course at the programmatic level, like there’s so many things that AI can’t fully understand because it just gives you that response to that one prompt. So there’s a lot that AI can’t do. We still need humans in the process. So what does that look like now? So yes, that’s the thing that’s coming down the pipeline. [LAUGHTER]

Rebecca: Sounds like an exciting adventure.

Megan: I think it will be. [LAUGHTER]

John: Thanks for joining us. It’s always great talking to you, and we’re looking forward to hearing more about these future projects.

Megan: Excellent. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

457. Intellectual Property

Faculty create large quantities of content, but aren’t always aware of their intellectual property rights, or lack thereof. In this episode, Kent Kauffman joins us to discuss legal issues associated with copyright law. Kent is a Professor of Business Law and MBA Programs Faculty Liaison in the Doermer School of Business at Purdue University Fort Wayne. He is also the author of Navigating Choppy Waters: Key Legal Issues College Faculty Need to Know.

Show Notes

Transcript

Rebecca: Faculty create large quantities of content, but aren’t always aware of their intellectual property rights, or lack thereof. In this episode, we explore legal issues associated with copyright law.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Kent Kauffman, a Professor of Business Law and MBA Programs Faculty Liaison in the Doermer School of Business at Purdue University Fort Wayne. He is also the author of Navigating Choppy Waters: Key Legal Issues College Faculty Need to Know. Welcome back, Kent.

Kent: Thank you very much for having me. I’m always glad to talk with the two of you, and I appreciate the invitation again to be on your awesome podcast.

Rebecca: Today’s teas are:… Kent, are you drinking tea with us today?

Kent: I am drinking tea with you today. Due to the heat here in Indiana, I don’t know if I’m pronouncing it correctly, but I’ve been drinking the Luzianne iced tea, where the tea bags are turned into an iced tea. So that’s my tea of choice as of late.

Rebecca: Something nice and cool seems like a good option in the heat.

Kent: Absolutely.

John: It’s a hot summer pretty much everywhere. And I am drinking a blueberry green tea.

Rebecca: Green must be the theme in here because I also have green tea today, John. I have a jasmine green.

John: Very good.

Rebecca: Looking for something lighter for the summer. [LAUGHTER]

John: So we’ve invited you here today to discuss intellectual property issues for faculty. One issue that comes up fairly often for faculty is the question of ownership of course materials that are created by faculty. Who owns the copyright to course materials that is stored in a learning management system.

Kent: Well, I think that’s a really needful and urgent question, and to me, it sort of parallels the urgency of academic freedom issues that faculty face right now. Because both of those issues involve finding out well, what what are my rights, whether it’s my right to teach and research the way my discipline respects, or my rights with respect to the work products I create that are attendant to, in this case, my teaching and also my scholarship as well. So if I can just do a quick tutorial, and I’ll try not to take too much time on this. I think of intellectual property like three siblings and a cousin that’s like invited to Thanksgiving dinner, but isn’t really a part of the family, and the siblings are trademarks and patents and copyrights, and that distant cousin is the trade secret. And the difference I see is this: intellectual property traditionally gets its value from being publicly displayed. That’s the point. What is the value of a song if no one’s ever heard you play it or sing it? What’s the value of a script if no one’s ever said it somewhere? And so intellectual property law, federal law, by statute, protects those particular aspects of our creative property. Trade secret law actually is protected by you, and so there’s a completely different way that we get value out of hiding something that we think is valuable, and that’s trade secret law. For us and copyright, for which faculty more than any other intellectual property, notwithstanding some faculty might deal in patent creation, but that’s a whole ‘nother discussion. But as it concerns copyrights, I think the first question to ask is, “What are we creating that is actually worthy of copyright?” And that’s not to be insulting. That’s just to say, by way of background, copyright does not protect our ideas. Don’t tell someone your great idea. Keep it to yourself. Copyright law protects the expression of your ideas. What the statute calls when your ideas are “put into a tangible medium of expression,” or what I might say, a fixed medium of expression. Your podcast is an example. By talking into this recording device, there’s copyright created automatically because this is a recording of the expression of our ideas. So once we understand that copyright protects our work automatically, we don’t need to seek a copyright registration to get that protection. The beautiful thing about it is it protects it automatically once it’s put into a fixed or tangible medium of expression, but the next question is, and this is the perhaps insulting part, is “What are we doing that’s worthy of copyright protection?” Because that requires creativity, and there is a Supreme Court case involving the phone book to which many younger listeners might say, “What’s that?” [LAUGHTER] But there was a lawsuit over the phone book because a competing company was using the white pages out of a phone book, and that went to the Supreme Court of the United States, and they had to answer the question, “What here is worthy of copyright?” And just to cut to the chase, the yellow pages are protected by copyright, but the white pages are not because there’s no creativity in assembling people’s names and addresses. And one feature the court mentioned is that copyright requires that your work have a spark, that was their word or a modicum, their word, of creativity. So if we think about that, the question then becomes: “Am I doing something that is put into a tangible medium of expression that is worthy of copyright that is instructional materials?” The foundational document probably is the syllabus. I think of the syllabus as the starter yeast of academia because it began somewhere else and it keeps getting handed over and over again as new things are added to it. And so, where I work, my syllabi is longer than it’s ever been – not by design, but because new parts must be added to it. Well, I don’t own a copyright in the works of others, so we jump out of that for a second. If I’m creating PowerPoints or tests or any other materials that come from the publisher of the textbook I might use, then I have no copyright in those materials whatsoever. So then we say, okay, is my syllabus copyrightable if I have enough of my own original content in it. That’s a worthy question.

If you think that you have enough originality, creativity, that your compilation of facts, which can qualify for copyright, are original enough, then you might be able to say it belongs to me. There can be a problem with that because one might argue that the syllabus is the department’s document as opposed to your document, since even if you’ve created the class, it’s still under the aegis of your department and your university as well. But in terms of these instructional materials, whether it’s a syllabus or PowerPoints, tests, really interesting essay assignments or projects that students do, I would argue that a lot of those can be worthy of copyright protection because they are first put in a fixed medium of expression because they’re written down digitally or on paper, and I would say that really, for faculty that care about what they’re doing and trying to make a vibrant life out of their classes, absolutely they can be protected by copyright because they meet those modicum standards, those spark standards of creativity. So yes, I would say this, from an initial standpoint, if you believe that your work meets that expectation of creativity, then you could make the argument that your instructional materials could be copyrightable and belong to you, but there’s this however, and the however might lead to the next topic, which is instructional or institutional policy.

Rebecca: I can imagine that if folks were looking at copyright law at face value, they might assume that this would fall under works for hire because it’s part of your job, maybe, to create instructional materials. So, can you talk a little bit more about where this role of institutional policy kind of dovetails with the law here in determining who has the copyright to these materials?

Kent: Yeah, there’s two issues there to think about, and one is policy, and the other is works for hire. And if I can just swing around to the policy issue, and then get to works for hire next. The beauty from an employment standpoint is that universities, and I’m saying this not as an employer, but from their standpoint, the beauty is that they can create institutional policy that also covers intellectual property. And one way that a lot of universities cover our instructional materials is that they will define categories of work we create at a university, and those are usually three types: traditionally administrative, and then scholarly, and then instructional. Now, someone like me who’s really a big believer of scholarly teaching… One could argue that teaching materials or instructional materials could fall into the category of being scholarly, but that’s again another discussion. As it concerns administrative materials. If you at your university are doing a teaching evaluation of a tenure-track faculty member, no matter how well you write that letter, it’s likely not to belong to you because universities categorize those documents as simply being university-owned property. As it concerns scholarly works, peer-reviewed papers, poster presentations, books we write. What I find interesting there is that: 1. universities tend to exempt themselves from ownership of those to start with, which I think is almost a moot point because I have to transfer copyright to those works anyways to the publisher generally to get them published. And there is open-access publications, I realize that, but still we live in a world where lots of our publications are conditioned on our releasing our copyright. So my university can’t claim ownership to my books, for instance, because I don’t have any ownership [LAUGHTER] to my books as well. So then the next thing is the instructional materials, and here we get a situation where universities do something that I think I’m not comfortable with it. But what happens is they often will say you own your instructional materials, but you have granted to us, by condition of employment, a non-exclusive license to use your materials, and the key there is, if you look at your institutional policy, if it has that, is it will often say non-exclusive because an exclusive license, similar to a transfer of copyright, requires both signatures. I must sign, and so must the proper agent of the institution sign as well. But a non-exclusive license doesn’t need signatures. And so, for instance, Purdue’s policy, the university-wide policy, not just where I work at Fort Wayne, but across all of Purdue institutions, is a non-exclusive, royalty-free, perpetual license. To me, that means that as it concerns my instructional materials that are in… and Purdue uses what’s called Brightspace as its LMS… all of that is, in my view, effectively co-owned by Purdue, not owned because they granted me the right to keep my copyright, which is an interesting language phenomena to say we allow you to keep something that you already own as the author of the copyrighted material, but a non-exclusive, perpetual, royalty-free license does mean that they can use my materials, everything in Brightspace or anyone’s LMS for as long as they want, no matter if I work there or not. Some institutions limit that, they might provide a royalty structure if they’re using your materials in a class you don’t teach. Other institutions might say we will cease using your materials if and when you cease working for us. And those are nice, gracious ways of sort of managing that. But to faculty that put their instructional materials on an LMS, as almost all of us do nowadays, I would say yes, you still would own what is worthy of being owned for copyright purposes, but you sort of now have a co-owner, effectively by your employer, who has taken a non-exclusive or maybe even an exclusive license in them. But the work for hire situation is a different angle to this. There’s two ways to create a work for hire, which would mean that the author is not even the copyright holder. They’ve transferred the copyright altogether. The first is the easy one, and that’s where I transfer my work as an independent contractor to the hiring party, and that requires two signatures there.

A couple years ago, I wrote scripts for some educational cartoons for a business law course that was online, and I created the subjects for what they were looking for. They approved them. Then I gave them ideas, and they approved them. Then I wrote scripts for educational cartoons, and they approved those as well. Well, none of those I own because that contract, being a work-for-hire independent contract, meant that they own all my work. The other way is what is known as what creative works that we produce in the scope or in the course of our employment, and that’s another part of the copyright code. And the scope of employment doctrine is an agency law doctrine that says that if I’m hired to do certain types of things that might involve creating intellectual property, my employer gets them automatically, not by needing a signature because I’m creating them in the scope or course of my employment. For faculty, an outsider might say, “Well, everything you do is in the scope of employment, and therefore, wouldn’t your university own them all? Well, yes and no. And to the no part, I’m not a greeting card writer at Hallmark, so I can’t disclaim some of my really pithy statements and say, “Well, I own this one because I’m hired to do that.” But as agency law looks at what faculty do, as it concerns instructional materials, it’s a confusing world. Is that in the scope of my employment? There’s no consensus on that. Some courts will say it is in the scope of your employment, particularly as it concerns curricular materials. There’s a case, I find it kind of sad, where a woman… she was a department chair at a two-year college tasked with creating a whole new program at her college within her discipline… and eventually she was let go and she brought an age discrimination claim and along the way she filed copyright registrations for the curricular documents that she created. One doesn’t have to do that to own them, but one needs to do that if they’re going to sue someone for infringing their copyright. Well, she lost on both counts and, as concerned the copyright issue, the court said part of her job as a department chair included doing things like creating new curricular materials, and those are clearly the work of her employer. But for regular faculty who are creating courses, not new courses by way of a grant, so if you’re hired to create a new course by way of a grant, that’s a work for hire in the first category. You’re going to sign a document and give up your rights to that material. But for the work we create that’s regular instructional work, there isn’t consensus, in my opinion, legally on whether that really is work for hire, and therefore we don’t own it, anyways. But if there’s an institutional policy that actually carves out the whole work-for-hire concept. So for instance, I know that my instruction materials are not work for hire at Purdue because the institutional intellectual property policy already says I own them and grant them a license. Well, therefore, it’s not work for hire because they said I own them; in a work-for-hire situation, I wouldn’t own them to start with.

John: I know in the SUNY system, any work that’s created for online classes, as part of an agreement with the union, copyright is maintained by the faculty members, however, at least a number of institutions have worked with adjuncts to develop courses as works for hire as a way around that. So that we have a mix of both of those in SUNY. I think that Oswego, we allow the creators to own the copyright unless there’s a work-for-hire arrangement that overrides those conditions. So most of the issues that I’ve seen have been in terms of online content that’s created. But what about lectures that are delivered in person? Do you own the rights to those?

Kent: There’s the question. I enjoy face-to-face classes. I certainly am happy to have online classes, and I take them very seriously. I want to make them vibrant, but I still really enjoy being in a room with people and having a dialog. Here’s an interesting case. So there was a lady who called herself the Banana Lady. You can look her up online. She dresses in a banana costume, and she does like corporate events. But what she really does is she tends to sue people who take pictures of her at her events and claim that these are copyright infringements. And in the federal court system here in Indiana, the Seventh Circuit Court of Appeals not only ruled against her, but one thing that they said was she was trying to sort of gin up litigation. But they said that people taking photos of her aren’t infringing on her copyright because she wasn’t recording herself doing her banana woman sort of dance routine stuff. So that’s the starting point. If you’re not recording yourself in class, then you’re not creating a tangible medium expression of your teaching, no matter how creative it might be. So, A) are you recording it? B) is who’s recording it, because if your university is recording it either automatically or that you use university technology to record it, I would regrettably argue that that’s not your property. It’s probably your employer’s property if you’re using its technology as the conduit for creating a tangible medium of expression. If you’re recording it yourself and your university is recording it, I think you could claim you have ownership over what you yourself are recording. But if you’re not recording it, there’s no copyright, and that goes to everything. If you put notes on a blackboard, you’re putting it in a tangible medium of expression. But if you’re erasing it, it’s gone anyways. And so the question becomes: “Well, then, what do students own if they’re taking notes from what I’m doing in class?” If they’re taking notes from what you’re saying and you’re not recording it, they have a valid claim that they own their notes. Again, something can be not worthy of copyright, but still be your property. Copyright is a type of property, so I could own something that’s not worthy of copyright. That’s what trade secrets are all about. So that a business who has a client list they believe is a trade secret property that’s not copyrightable, but it’s still their property. So I’m not saying a student owns the copyright in their class notes because that’s a copyright creativity question. But they could argue they own those notes if they’re taking notes off of what I’m saying. If they’re taking notes off of my powerpoints, and my powerpoints belong to me. I would argue that they’re actually just simply transcribing what is already mine, and they don’t own that. There are some cases, very few, but they’re out there, where faculty have sued publishing companies because in some really large universities, publishing companies might sell you notes for the classes that you’re taking that are highly common large enrollment classes, and faculty have sued saying that’s actually my property, but students are selling those notes to publishing companies, who then create study guides out of them. That’s a really thorny question because if they’re simply transcribing what is your property, you’re not transferring your property to them. So that would be a copyright infringement. One area is what’s called transformative use, which is a part of what’s called the fair-use doctrine, which allows a person to raise a defense to copyright infringement by arguing a statute in the code called the fair-use doctrine. And fair use is about limited usage of some of those copyrighted work for reasons that not are allowable, but are part of the variables for determining if you’re committing a non-violation of a use of some else’s material. Transformative use says if I use your material in a way that’s completely different than how you used it, I have transformed your purpose into a new purpose.

So you could perhaps find a situation where a very creative student is creating a graphic novel out of their class notes that might qualify as transformative use, and they might say that’s a brand new thing, and it might belong to me separately. So just to sum that up, if you’re not recording your work, then someone could claim ownership in it, and that could include notes. If you’re in a situation where students are taking down as notes what you’ve already created in a tangible form, generally speaking, that’s not their property. That’s your property.

Rebecca: So, on the theme of recording, we’ve talked a bit about posting materials on the learning management system and maybe sharing through the learning management system. But many faculty also use third-party sites like YouTube or Vimeo that have licenses very similar to what you described at Purdue. [LAUGHTER]

Kent: Yes, right.

Rebecca: So, how does copyright work for those situations or ownership in those contexts?

Kent: That’s a really interesting and strategically powerful question, and I show my students when it comes to the section in business law and intellectual property, I show them Instagram and Meta’s terms of service for example because what they may not realize is that they’re actually granting co-ownership of their property to those sites, and the language is quite broad. It’s hard to find, but it’s quite broad. So you’re raising an interesting question that if I’m putting instructional materials that I’ve created on a third-party site, as opposed to my own hosted site, I’m actually transferring rights to that site. That is a really scary question. I would say the answer is yes, just because vacation pictures aren’t the only thing that social media companies care about claiming as their property as well. But as a general question to conflict of university ownership, you could make an argument that the university can only license to themselves what’s in their system, and therefore there’s no way for them to use what’s in a third-party system because they have no access to it. So if you provide students links to external sites, which may include third-party sites, I think you can get around that licensing issue because it faces like an efficacy problem where it’s just simply not feasible for the institution to get access to it. It wouldn’t take very long for their lawyers to draft new policy that would say you can’t use third-party sites for our classes because even though I don’t think FERPA would have any bearing on this, you could just simply call it a FERPA risk issue. You could call it anything that a lawyer would want to call it. So they could marshal those back into the institutional LMS. They could also say that you must provide them copies of what you create for your courses out of a courtesy of making sure the curriculum is consistent. So they could get around that, and then you’re back to where you perhaps have started. So that could be an issue down the road. But I think initially, certainly on your own created site, you would maintain all ownership over those materials. And if you give students access to those materials, again, you’re not transferring copyright to them. In a way, you’re almost giving them a limited license to have access to your materials, not really sharing ownership of it with them.

John: On a related note, suppose that you’re writing a book and you share drafts of it with your students in an LMS. Are you giving up some of your property rights to that material if the course was created as a work for hire?

Kent: That is a super scary question, and I would say that generous heart of saying “let me share some materials with you” might inadvertently cause you to sort of step on your own rake head there and find out that you got slapped with the realization that you’re losing ownership rights over that material. If it’s protected by copyright… I have put drafts of books that I’ve written, chapters or parts of chapters, into an LMS to share with students as well as an alternative to maybe the book I choose for the course and say here’s an alternative way of looking at the material because that’s already owned by the publishing company. I’m not granting them any rights because I don’t have rights to own it as well. But unpublished work that you maintain the copyright in because you own it as soon as you write it, putting it in an LMS in a way at least grants a license to that, unless you figure out a way to scrub it from your LMS at the end of the semester and remove it as an archived piece of information.

Rebecca: AI has been a very popular [LAUGHTER] topic on this podcast and just in general, and folks are certainly utilizing generative AI platforms to edit materials, enhance their work, use it as a dialog for feedback, and those kinds of things. So, does the user own any rights to images, text, and other content that they’re using generative AI to create?

Kent: Boy, here we go. Who can take another AI discussion, right? It’s like when the zeitgeist has become full tilt, and we just can’t take it anymore. But this is relevant, and it really is a needful question for faculty to think about. So, if I can just do a background on AI and give you a lawsuit involving an Indonesian monkey. So a British photographer named David Slater became friendly with some monkeys in Indonesia as a wildlife photographer. He gave them names. One was named Naruto. He gave them his 35 millimeter camera, and then they started taking pictures of themselves. He then published a book of monkey selfies, which is charming in a variety of different ways. And in came people for the ethical treatment of animals, PETA, who on behalf of this monkey named Naruto sued the author of the selfie book in American court, alleging he violated the copyright of the monkey who took the pictures. I wish I could make that up. I lack that creativity. The upshot of that lawsuit, as it relates to AI, and it does, is this: unlike Solomon, everybody lost. PETA was thrown out of court in effect for saying you have no standing to bring a lawsuit on behalfof a monkey in Indonesia. The photographer lost because he didn’t take the photographs, so he didn’t have the copyright in them. Now you might say, “Well, didn’t the monkeys take them for him?” No, they’re not his employees, so it’s not a work-for-hire monkey selfie book he was working on. Poor Naruto lost as well, ultimately because of this. Copyright law is clear. To be an author, that doesn’t mean just for written word, that means for anything that’s copyrightable. to be an author, you must be a human being. Now, by way of contrast, corporations are persons under the Constitution, but the word “person” is not the word “human being.” So, to be a copyright author, you must be a human being. That moves us a few years later to AI. Nothing that comes out of large language model AI is copyrightable. Now, if we get to this, it’s to me copyright infringement, but it’s not copyrightable because a computer is not a human being. So no matter what it is, if it comes out of generative LLM AI, it is not copyrightable. We know of this for two reasons. One, there’s an interesting case where a super smart guy created his own AI system and had it create some sort of impressionist style painting that he gave a title to, I think it’s called “A Recent Entrance to Paradise,” and he sought a copyright registration for it. It was denied. He went through the court system as far as you can go, and he lost every time because AI is not an author and it can’t be. He tried a second bite at the apple by saying that the computer system he generated was a work for hire, and he owned the work itself. And the court’s like, you know what? Again, work for hire start with a copyright author, that has to be a human being. Now that means we know that from case law, and the Copyright Office in 2023 issued a very long pamphlet statement, which clearly states this: there is no copyright in AI-generated work at all. Here comes the however, and there is one, and they did say this, and there’s a case on it. They said there could be, on a case-by-case basis, a copyright granted to what the copyright office called AI-assisted work, not AI- generated work. I would just say, by a quick analogy as an example, if I ask AI to write an essay for me, or let’s just make it worse, let’s make it a paper, let’s go there. I ask AI to write me a paper I want to get published, and then I take what they give me and then I edit it myself. Not only is that unethical and should be unpublishable, it’s not copyrightable because that came out of AI.

But suppose I write an essay that I’d like to get published on today’s educational issues going on in America, and the requirement is that it must be 2500 words, and I just can’t get down from 2700. If I put that into the AI system and ask it to get me to 2500, I would argue that I still own the copyright in that because I used AI to assist me in a relatively minor editing method, as opposed to, in effect, co-generate with me. So case-by-case basis for AI-assisted work, but AI-generated work there is no copyright whatsoever.

Rebecca: I’m so glad you shared some of these stories. These are some of the cases I like to share with my art and design students. [LAUGHTER]

Kent: Well, can I do an art design for you? [LAUGHTER] Because the case I’m thinking of is… and I’m terrible with names. The name has slipped me, but a person used AI to write a graphic novel. I’m old enough to call them comic books, but I guess that shows I’m too old to not know they’re graphic novels. But a really interesting looking graphic novel in which AI was used to generate all the art, and when that author of the graphic novel saw the copyright in it, it was denied because it was AI-generated art, and then the author sought a rehearing, you might say, of that, and they got a Solomon-like outcome in which the copyright office said, “Okay, here’s the thing: you own the words that are in the bubbles in your graphic novel, and you own the placement of them with the AI-generated art, but you do not own the AI-generated art.” So that means if I wanted to use any visual from that graphic novel, I am free and clear to do so because nobody owns it. But I can’t use the cells because the cells are copyrightable, and certainly the language inside those thought bubbles are copyrightable as well.

John: What if you write something and then you run it through AI and it makes some suggestions for change and you adopt those suggestions? Does it depend on the extent of AI assistance in terms of that case-by-case analysis, or would it be determined on where the thoughts originated, whether they were human originated thoughts that were then cleaned up and edited by AI or rewritten or restructured. It seems like there’s no clear dividing line between the work of the human and the work of the AI.

Kent: That’s why, and this sounds like a cop out, and I apologize, but that’s why case-by-case basis. While I’m saying I don’t think the copyright office was wrong in to say that, it creates a line that you’ll never see the end of, of finding out have I done enough for that to work. Like in corporate law, shareholders can be personally liable for the debts of the corporation if the corporation is undercapitalized, but that’s not based upon a mathematical basis of examining debt to equity. It’s done on a case-by-case basis. So, in your scenario that you gave as a hypothetical, I would say to me that sounds like that would belong to the human being who started with the work. But how much of that goes from doing fine tuning to actually creating really what is a new thing with the use of AI? When it becomes a new thing, as I read that copyright advisory document, it’s really not protected by copyright. And the scary thing to think about is this, and I know that some AI companies will tell you they’re still not doing this. You believe them at your own peril. But once you put your own work into an AI system, in my view, you’re contributing [LAUGHTER] voluntarily to the copyright infringement that is the reason for the existence of AI in the first place, and so the web scraping that goes on to create an AI system, which is the subject of a variety of large copyright infringement lawsuits that are still ongoing, you’re at least volunteering your work in an AI system that may allow your work to then be used for training and other purposes that maybe you don’t like as well. So there may be systems that claim that you can eliminate that, or prohibit that, or stop that from happening. Again, the next question asked is how would you take that to court? Because we live in a world that’s controlled by arbitration agreements, and if you have a membership or a subscription to a paid AI system, they’re not going to want you to sue them. So just like with your cell phone agreement and your credit card agreements, you’re going to have to arbitrate that, which would be a nightmare anyways. And so in a way, it’s almost like I would say when they say don’t lend money to family, treat it like you’re giving it away. I would presume to say the same thing, if you’re going to put your work in an AI system, presume you’re giving it away for purposes you can’t control. But if the purpose why you need to use it in an AI system overwhelms that, then just go ahead and do that.

Rebecca: So I want to like pull a thread that you’ve hinted at in a few different places already, which is some of the work that we may have created that are our own intellectual property has been scraped and is part of these systems. And this is certainly something that comes up a lot in my land of art and design, where particular images of artworks or whatever may be scraped and now are part of these large language models that are now used to create these images that are not copyrightable, [LAUGHTER] right? And so faculty may be using AI to generate images and things to illustrate their ideas or other things, but it’s pulling on a database of things that don’t necessarily belong to them. Can you talk just a little bit about that angle?

Kent: Yeah, the question. might be, if I’m hearing this correctly, is it possible that a faculty member is infringing on the copyright of somebody else when they’re using AI materials that they’re pulling out, even if they’re strategically asking for what they want to ask, because what has gone into AI is, to me, an infringement of someone else’s copyright. And I think that’s a big question because even though there’s no copyrighting what comes out of it, what went into it, in my opinion, has violated the copyright of the author. And so I would say that faculty aren’t infringing on the copyright of an AI system by using AI for whatever they’re using it for. The lawsuits that are ongoing are still ongoing, and there have been some settlements. I’m an author of a textbook that’s owned by Cengage so I know I’m in a settlement with the AI system Anthropic, and that settlement is in, I think, the billions of dollars. But what it gives is to each copyright holder a couple thousand dollars. And I think it’s kind of unfortunate, but that is the way that it’s going, is that the capital that those AI companies have is so huge that they can settle a class action lawsuit. And some authors have said, “I’m not going to be a part of this. I’m going to sue individually.” To them, I say, “Godspeed.” But those settlements allow them to just simply wipe that as an expense and then continuing what they’re doing, but the ethical issue is: Am I doing something I really should be doing? Am I using AI to generate ideas for myself or quick answers I can’t find, or am I using it to actually create work that I want to claim as my own if I massage it with my own assistance when in fact it came originally from another author or even an academic author who sort of lost their rights to it when it was web scraped into an AI system. It’s a deep question. It’s not a copyright infringement to the AI system, but it could be sort of an ethical infringement to the original person who didn’t choose to lose their work into an AI system.

John: Now let’s take that just a little bit further. Suppose you used it as a faculty member to generate some imagery that goes into a book that you sold to a publisher, and then the imagery that was created by the AI platform involves a copyright violation of the original copyright holder.

Kent: Wow.

John: Who would be legally responsible for that violation? Would it be the AI platform that generated that, or would it be you as the author who is using essentially slightly modified copyrighted material?

Kent: Yeah, that is not only novel and really brilliant. It is, to my knowledge and my research, unanswered as of now, so I’m only projecting what I think is the case, and I do see a world where perhaps an original owner, let’s say, of art that ended up in an AI system that ended up again in someone else’s work, could sue that end user, saying you infringed on my copyright, and you could find a situation where a faculty member might have to ask themselves, “Do I have an insurance policy that would cover the legal costs of defending myself in an infringement lawsuit?” Because let’s say that suppose that happened to me, perish the thought, is that a situation where Purdue is duty bound to defend me? But if, like you said a book, what we’ve said is these books tend to be not owned by institutions because their copyright is exchanged for some fee structure. But if you read your publishing contracts carefully, you owe the publisher a hold harmless clause to make them whole if they violate copyright because of what you put into a book they published, and so that’s a very scary financial question. And again, that’s something to think long and hard about before using AI-generated art. And again, it doesn’t matter if you have that background knowledge like Rebecca does. It matters if you’re simply using work that really is sourced to somebody else, because the middleman, in this case AI, is not liable for it, at least not yet, and maybe not ever. But you, the end user, could be.

Rebecca: We haven’t talked a lot about the role that students play. Students are often creating work; they’re submitting it to you, maybe as a faculty member to review. Now you have a copy of it in your possession, perhaps. Do students own their own classwork?

Kent: There is a lot to say about students. It’s not just the cheating part, although that is worthy of discussion if you want to talk about cheating and AI detectors and things. But students own their work for the same reason that faculty might own their work as well, and I think students should be treated respectfully with respect to their work. And so, if we say that the student work passes the modicum of or spark of creativity test we talked about, that does belong to them. There can be some exceptions. Sometimes institutions have policies for graduate students if they’re creating work that’s part of a project that is sponsored by the institution. Sometimes there might be policies that say students don’t own their work if it has been hired out by an outside party. For instance, suppose a marketing class has an instructor who’s connected to a business that wants a project done that the students can free help the business, right? Well, who owns that work is a huge question. Policies might carve exceptions out there, but as a general basis, students own their work. So I have, in instances, particularly in online classes, said to students, “You know, some of your work here is so good. Is it okay if I use it anonymously as a template or a model of what ideal student work would look like?” I shouldn’t take their work without their permission. If I take their permission, I seek an email that I can archive and say they granted me permission. There is a case, there are more than them, but there is a case out of a university in Detroit where an art student was asked by her instructor to have students create artwork that would be used by the Mercedes-Benz exhibit at the Detroit Auto Show, and she knew what she was giving up, but she didn’t realize that Mercedes, with the knowledge of her instructor, used more of her work that she created than she had acknowledged to give over to Mercedes. And she sued Mercedes-Benz for copyright infringement, and she won. And it didn’t matter that she didn’t have a market value in her work because there’s another part of copyright law that allows you to get statutory penalties for simply infringing on someone’s copyright. And then she sued her professor for what is commonly known as a breach of a fiduciary duty to her as a student. I don’t know where that lawsuit ended up, but the court allowed that lawsuit to go forward. So I would say that faculty should be mindful that what students create in their coursework, including artwork, musical compositions, and just essays and projects, really do belong to the students, absent some external independent contractor work for higher situation like the Mercedes Benz was for three of the four artwork pieces that ended up at the exhibit. So I do think faculty should be mindful of that. And there’s a separate issue, which is a faculty member could say, “Yeah, but there’s no fair market value in this student’s work. So what does it matter?” Well, that question alone shows why it does matter, because faculty should be respectful of their students as individuals and as owners of their own work. So whether there’s a market value for it to me is not the question. It is, “Does it belong to them?” And if it does, and it does, then I should be respectful and just ask to use it. And if they say yes, get it in writing somewhere, and you’re good to go generally,

John: Following that up a little bit, if we upload student work into a plagiarism or AI detection software package, or perhaps just to an AI platform to get some feedback on student work to assist with grading, would that be a violation of students’ rights? I was thinking in particular of how if there was some artwork in that student’s work that was uploaded that was used for training it, it would be in some ways a similar sort of violation.

Kent: I totally agree with that. Often AI and students, from an HR standpoint at work, is discussed with respect to FERPA. Well, I mean, you’ve got to be an idiot to use AI systems to create education records that would violate FERPA. But in terms of what you’re saying here is, am I really transferring without their even knowledge, perhaps, their ownership into an AI system by using it for evaluation purposes? That is a serious thing to consider. Now, would anybody know that I did that? They could argue the only way that my work ended up in an AI system is I turned it into you, and you stuck it in there. So it wouldn’t be terribly hard to figure that out. But that’s an ethical question and a legal question as well. One issue to think about with respect to AI detection, if I can just mention that, is when you’re using it, you really then transferring copyright because the original plagiarism detector that we all knew… I never used it… in fact, I didn’t pronounce it correctly unless someone corrected me on it is Turnitin, but students from a high school class sued Turnitin for copyright infringement, and I say “you go, good for you,” and they said you took my papers and violated my copyright by sticking in to Turnitin, and the students lost because the court said this is a transformative use of someone else’s work for plagiarism detection, and there is no infringement on their ability to publish their work by being archived for plagiarism detection purposes. That’s not, however, what’s going on now. Because even Turnitin’s AI system has been turned off by some universities. So I would say this: here’s some free legal advice, which means it’s worth nothing. Don’t use AI detectors. Simply don’t do it. It’s not worth it for a variety of reasons, and the most important reason is to me it’s error prone, and the last thing a faculty member should do from a legal protection standpoint is accuse a student of cheating based upon an AI detection system because they have been known to be error prone, they’ve been proven error prone, and even if they’re improving through retraining, they are error prone. If I haven’t said that yet, and if you’re saying I’m giving up my sort of autonomy as a judgment maker to an AI system, which is based upon predicting the next logical word, that’s why they fail. That’s why they make mistakes because they’re based upon perplexity, “What is the next most logical word coming up that we think you want?” So if I say I’m dropping my children off at blank, logically it’s going to be perhaps school or the playground or the pool. That’s the lower perplexity. Well, you’re giving up your autonomy, your judgment decision when you’re saying to AI, check if this has been cheating. Now, if you can reverse engineer that by saying I put your work in an AI system and it showed that it came out of already published work. That’s a traditional plagiarism detector. But AI systems also flag often non-native English speakers as cheating because of the way they write sentences with low perplexity and what’s called burstiness, which is an interesting word. Burstiness is the variation we use in the sentences we write: short, long, those kinds of things. So, non-native English speakers and sometimes autistic students are flagged because of the low burstiness in their sentence structures as being cheating. So, to me, just don’t use AI detectors because AI detectors also sell humanizers to students as well. AI humanizers are like police officers selling radar detectors on weekends. So if you want to avoid being caught by us by this product, when you’re dealing in a world where AI humanizers are already being used by students to bypass your using AI detection, then the question is, what are you doing for your job? Is that your job to be the hall monitor of all hall monitors? And the cheating is rampant. Just recently, I think a Brown University award-winning economics professor realized that allowing his students to do a take-home exam basically the AI cheating was rampant, and he was just horrified by that. So we know that it’s going on, and I’m not saying that we shouldn’t care about it.

What I’m saying is, out of all the time you have in your life, of which a lot of it is work, how much of it can you give to hunting down the possible AI cheaters in your classes? I don’t have that kind of time, and if I’m going to be wrong, I’m really wasting my time as well. So I think there’s a legal issue with using AI detectors. MIT, who you can, if you want to, you can source MIT as the first creator of an AI system, says don’t use them. So if MIT’s teaching and scholarship learning center says don’t use them, why do I need to seek a second opinion? I’m not going to use them, and so I’m not saying you’re wrong to use them. I’m saying you use them at your own peril because of the issues involving the last thing I want to be accused of is committing an act against a student that is known as arbitrary or capricious. Sometimes it’s called arbitrary and capricious. One of the few areas where a faculty member faces a legal risk is when they act arbitrarily towards a student, and I believe using an AI detector as your reason for accusing someone as cheating is getting you into that arbitrary world that I don’t think is worth risking.

Rebecca: On the flip side, students often submit tests, assignments, and other materials created by the instructor into generative AI platforms [LAUGHTER] without the consent of the instructor. How should faculty address this issue?

Kent: One, and I know you don’t need me to say this, so I apologize for preaching to the choir here. But have a syllabus policy first of all, because this may seem too smart by half, but a student has a good point by saying I didn’t do anything wrong because you didn’t say I couldn’t. And so if we presume they’ll know they shouldn’t do something, then we’re wrong as being in charge of our classroom. So, have a syllabus and specifically delineate what you don’t want to be done. You, again, can’t control what they’re doing on their own time, and unless you get back to that Paleolithic world where we had everything done in class, either on paper or in front of people, it’s hard to catch those kinds of things. People sometimes have students sign honesty pledges attached to syllabi. Those are all well-meaning. They have no legal value. So, other than policies that allow you to go to the dean of students or have some sort of institutional disciplinary process for that if you catch it, outside of a clearly delineated policy and examples and encouragement. And again, this may sound hokey, but like I tell my students, you’re in a business school. A business school is a professional school. What does it mean to be professional? Like, what does it actually mean? It’s not about making money. What does it mean to be a professional in a professional school? How does ethics fall into that? And again, that may seem like I’m wasting their time if they don’t care, but at least they know that I’ve thought about it ahead of time, and that I tell them it’s not worth getting caught, and you’re not going to feel better down the road because you cheated here, and so I just try to be that good angel on the one side of their shoulder, reminding them that it’s just not worth it. Again, that may be a waste of my time to tell them that, but I can’t hunt down all the possible cheating unless I bring everything back into the classroom, which some faculty are again trying to do, and we’re now living in a like a record player world again, where we’re saying, you know, something, let’s go back to the record player and let’s bring the classes not just back into the classroom, but bring the assessments down to more of that low technology world, which cuts down the chance of cheating.

John: We always end with the question. What’s next?

Kent: Well, I’m thinking about or trying to work on creating a podcast for myself, not to compete with you at all, but on legal issues for faculty, and I’m trying to work out how that might come together, and hopefully that will come to fruition sometime in the near future. And I’m working on some conference presentations to interested faculty members and book groups where we get together and discuss issues from this book that I’ve written on legal issues for faculty, and then the fall semester is around the corner, so that’s on its way as well.

Rebecca: Yeah, summer never fails to slip by.

Kent: No, I finished my summer semester about a week ago, and now I’m thinking about the fall semester already.

John: And it’s always good to see more podcasts related to higher ed, including legal issues in higher ed. We’ve lost a few good ones in the last year, so it’s nice to see new ones coming online.

Kent: Well, if I can overcome the technology difficulties I had prior to this recording, that may be something to figure out. Can I actually make this happen? So, thank you for the encouragement on that.

John: If you need any help, just let us know. [LAUGHTER]

Kent: Well, that’s recorded now, [LAUGHTER] so you can’t walk that back. Rebecca didn’t say it; you said it. So I have one person to go to, and I can pinpoint when you specifically said it. So thank you for that invite. [LAUGHTER] Il I appreciate it more than you might care to know.

John: Well, thank you. It’s great talking to you. And these legal issues are ones that many faculty are thinking about. So this was really helpful.

Rebecca: And I always enjoy these conversations. So thanks for joining us.

Kent: Thank you very much for letting me be here. I really have a wonderful time talking with the two of you. So thank you again for allowing me to be on your podcast.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

456. Improve Critical Thinking

Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, Michael Bishop joins us to discuss an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.

Michael is a professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases.

Show Notes

Transcript

John: Students with strong critical thinking skills can make better decisions that have real consequences on their quality of life. In this episode, we explore an approach to developing these skills that appears to have a persistent effect on students’ ability to avoid common cognitive biases.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Michael Bishop. He is professor of philosophy at Florida State University. His research interests focus on two areas: rationality (how should we think about important things) and happiness (what is lasting happiness, and how can we get it). He is also a co-author of a recent study investigating the impact of critical thinking classes on common cognitive biases. Welcome, Michael.

Michael: Thank you. Glad to be here.

John: Thank you for joining us. Today’s teas are:… Michael, do you have tea by any chance?

Michael: No, not today. I usually drink tea in my office, and when I do, it’s usually a Chinese black tea.

Rebecca: Always a nice choice. Today I have an English breakfast tea, nothing that exciting,

John: And I just have nothing that exciting either, a Lady Grey today.

Rebecca: We got to step up our game, John. So, Michael, we’ve invited here today to discuss the field experiment that is described in your co-authored paper entitled “Critical Thinking Classes Can Reduce Common Biases: Results from a Field Experiment.” But before we discuss the study, can we start a little bit with the concept of critical thinking? You teach critical thinking. What are you trying to teach? And what makes someone a good critical thinker?

Michael: Yeah, I think that’s a great question. One of my favorite sayings is the philosopher George Santayana said that a fanatic is someone who, having lost sight of their goals, redoubles their efforts. And so, in teaching critical thinking, I think it’s important to start with an idea of what you want to teach. And so there’s a lot of possible answers. Some people think that a good critical thinker is someone who uses certain rational methods or rules like logic or probability for the scientific method. Other people think that a rational thinker would be an expert debater, someone who can detect fallacies and build a coherent case for their views. Other people view good critical thinkers as virtuous thinkers; someone who is careful, open-minded, curious, intellectually honest, intellectually humble. And I actually think these views are either wrong… so I think that good debaters are often poor reasoners… or incomplete. They capture part of good reasoning, but not everything. So basically, what I am trying to teach in my classes is teaching people to ask good questions. Ask what I call fruitful questions, questions that help us get better answers, not killer questions that end debates, but questions that, if you’re trying to answer them, help get us better answers. So when you view that as the goal, how you go about teaching critical thinking becomes much easier.

John: And as you suggested, people have many different descriptions of what critical thinking is. I know whenever we talked about the importance of this in our Gen Ed program, everyone agrees that critical thinking is extremely important, but they all disagree as to what exactly it is. So it’s helpful, I think, to start with that statement. In this experiment, you’re focused on five common biases. Can you describe the biases that were selected for this study and provide some brief examples of those biases for our listeners?

Michael: Sure. The way psychologists think about these is in terms of biases. The way I think about them is in terms of cheat codes. So there are certain cheat codes people use in debating or in arguments, and cheat code arguments are arguments that you can use to defend pretty much any view you want. And so cheat codes are really tools of manipulation, more than tools of trying to figure out what the truth is or what a good decision is. And so when you think about good critical thinkers as thinkers who can ask good questions, the way I think about it is when you detect a cheat code, when you detect somebody using a cheat code on you, you should be able to ask them a question that diffuses the cheat code and points them in better directions. So let me tell you some of these cheat codes. One I call the focalizing cheat code, and it might go something like this: We should go to war because it has certain benefits, so, if an argument like that, or we should not go to war because it has certain costs. If you’re convinced by an argument like that, I think that you’ve fallen for a cheat code, and the best way to see this is to consider an example where where it’s clear that it’s a cheat code. So, suppose I gave you two envelopes, and I said you can look in these envelopes, and you can take all the money that’s in the envelope that you choose, but you can only choose one envelope. Suppose I open one envelope, see that there’s $5 and I take it without looking in the second envelope. In that case, you’d think that I’ve made a mistake, that I haven’t reasoned wisely about this particular decision because I didn’t look in the second envelope. If I say, “Look, the reason I chose the first envelope is that having $5 is better than not having $5.” It’s true that having $5 is better than not having $5, so there is an advantage to choosing envelope one. But when you think about it, what makes a choice a good one isn’t that it makes you better off than you are now. What makes a choice a good one is that it makes you better off than what else you might have done. Now, economists call this opportunity cost, as I suspect you guys know. And so, one thing I’m trying to do is get my students to, when they see someone argue for, “Hey, we should go to war because it’s going to bring us benefits.” I want them to at least stop and be able to ask the question, wait a minute, what are some alternatives to going to war? Because even if it’s true that going to war brings us benefits, and we can argue about that, but even if it’s true, it might not be the best thing to do, in the same way that choosing envelope one, even though it brings us benefits, might not be the smart thing to do. So I call that the focalizing cheat code. Another cheat code that I teach is what you might call the correlation cheat code. So I say, “Hey, a new healthcare plan is passed, and healthcare costs have gone up 7% a year since this plan is passed. Boy, this plan really stinks. It’s caused our healthcare costs to go up a lot.” This again, the question isn’t is the conclusion right in the war case or in this case, the question is: Is this a cheat code argument? And it is. And the reason is, what you don’t know, and what you really need to know, is how high would have costs been had this healthcare plan not been in place. So again, I say this healthcare plan is great or it’s bad, and I point to something that happened afterwards that. That’s what I call the correlation cheat code because you’re taking this correlation and you’re taking it as evidence for thinking that this policy has caused this particular result. And so again, what I want people to do is just stop and ask the question: “Well, wait a minute. Maybe without the health care plan, costs would have gone up even more, or maybe they would have gone up just the same, or maybe they’d have gone up less.” But unless you ask the question, it’s easy to get manipulated into accepting views that aren’t true. Another cheat code is the regression cheat code. This is a very common one. So lots of coaches and parents think that punishment is effective at molding behavior. I know that when I played sports as a youth, my coaches thought that after a poor performance, if you punish the players by making them work really hard in practice, they’ll do better next time. And sure enough, after a poor performance and after we were punished, we did tend to perform better. But again, the question is: “Did the punishment do that, or was it just a return to our normal level of performance?” So that’s the regression cheat code, rather than just saying, “Hey, maybe you’re returning to your normal level of performance,” saying that “it’s the result of a punishment or the result of a reward.” The third one is sunk cost cheat code, another one that’s a favorite of economists, defending a decision because of the resources already spent on it, so let’s stick with this policy or this war because of sunk costs. Those sunk costs are gone; they’re gone no matter what you do. So you might as well do what’s best for your future. And the last one that we looked at was the gambler’s fallacy, and that is basically treating statistically independent events, as if they were statistically dependent. So, if a coin comes up five times heads, and it’s really a fair coin, thinking that it’s more likely to come up tails next time. So, those were the five cheat codes we looked at.

Rebecca: I think the soundtrack that we have going on with the thunder is really emphasizing that these are cheat codes. [LAUGHTER]

Michael: Indeed, indeed. [LAUGHTER]

Rebecca: It’s very ominous. [LAUGHTER] Can you talk a little bit about how you chose these particular cheat codes?

Michael: Sure. So this goes back to my view about good reasoning. So 20 years ago or so, me and a friend defended a theory of what rational reasoning was, and what we said was that the good reasoning is reasoning that’s cheap, it’s accurate, and it’s focused on significant problems. And so the way I chose these cheat codes is because I thought that they were significant. That is, people use them to manipulate you when it comes to really important issues. So I see these cheat codes a lot out there being used in the world for really important issues. And the cheap part came in because I think that in thinking about how to teach critical thinking, I think one thing you want to do as a critical thinker is to diagnose the problem. Why are my students thinking badly when they do think badly? And I think there’s three general answers that people tend to think about, although often this doesn’t get talked about enough. So one reason students might be thinking badly is that they don’t know these rules, so it might be an ignorance problem. And if it’s an ignorance problem, then it’s going to be really hard to teach them to use these rules in a way that they keep using the rules after they leave the class. Because not only do they have to learn the rules and learn to apply the rules, but when they leave your class, they need to stop thinking in their old ways and use these rules, which tend to be hard to remember, hard to apply in the real world. And so, one reason that I think studies have not been successful or have had limited success in teaching people to reason well in a way that sticks once they leave the classroom, is that too often we teach critical thinking as if the problem is an ignorance problem. So we need to teach our students new rules. So one reason I chose these rules is that I speculated that they already knew these rules and they sometimes applied them correctly. And if that’s right, if they sometimes apply these rules correctly, then I don’t need to teach them a new rule, and I don’t need to teach them how to apply the rule. They’re already doing that. All I need to do is get them to see that it’s the right rule, and then ask them, “Hey, why aren’t you using it in these cases over here, where it seems like you should still be using the rules.” So, for example, think about the two envelope problem that I talked about before. None of my students think, “Oh, yeah, choose the $5, that’s the way to go. None of them think that. They know perfectly well that the way you evaluate that choice is not in terms of “Hey I’m $5 better off than I am now,” it’s “Hey if there’s $1,000 in the other envelope, then I’m 995 dollars worse off than if I’d have chosen envelope number two.” They all see that, so I don’t have to teach them any new rules. All I have to do is when I say “Hey let’s go to war, it has this advantage,” and if they say, “Yeah, that sounds good,” I can say, “Wait a minute. How is this different from the two envelope problem?” So the main reason I chose these, and these aren’t the only rules I teach in my critical thinking class, but the main reason I choose the rules that I do is that I think that my students already know these rules and at least sometimes apply them, and so when I introduce them to these rules in cases to which they are inclined to apply them, they give me what I call the old bovine stare. They look at me like, “Well, that’s obvious. You’re not teaching me anything I didn’t already know.” And that’s what I want. I want them when I introduce these rules to say, “I already knew that. Why are you teaching me something so obvious?” Because then the next class period, or a week from now, or a month from now, when I introduce an argument where they ought to be using this rule that they said was so obvious and they’re not, I don’t have to say, “Hey, you guys have made a mistake.” I can just ask them, “Okay, wait a minute, you said this rule was obvious, why aren’t you applying it here? Either you were wrong on day one about that rule being the right rule, or you’re wrong right now in not applying that rule.” And so there’s a real sense in which the reason I chose these rules is because I think I’m really not teaching my students anything new. So here’s one way to think about it: rather than teaching them like calculus, like a chain rule, like a new formula that they don’t know, what I’m really trying to do is get them to recognize cardinals. They already know what a cardinal looks like, so I’m just trying to get them to recognize cardinals, and so that’s why I think that the way I teach critical thinking tends to work and is way easier than starting off with the assumption that “Hey, my students don’t know anything, and I need to teach them a whole bunch of new rules.” So that’s why I chose these rules, and that’s why I choose the rules that I use in my critical thinking class.

John: And these are rules that, when you think about them as you framed it, they’re pretty obvious. But in practice, they’re rules that people violate all of the time in their decision making. What really impressed me about the study was that you did get some [LAUGHTER] results out of this because so many studies of behavior modification have been found to be ineffective beyond a very short period of time, often not lasting through the end of the class.

Michael: That’s absolutely right. In fact, one of my old professors, he said this about logic classes, but I think it unfortunately is often true of critical thinking classes, but he said that logic classes teach you to do one and only one thing, and that’s take logic tests. And I sometimes worry that that’s the case with a lot of our attempts to teach people to think better, is that what we’re really doing is teaching them to take tests because they will use the rules on the test. But if you want them to think about this afterwards, I really do think that the main thing is to get them to commit to the rule, and then get them to recognize, “Hey, wait a minute,” to be sensitive to when they see it in the real world because it’s all over the place. Once you get sensitive to it, you start seeing it everywhere. And indeed, one danger of my class, of my critical thinking class that I talk to my students is what I call “the turd in the punch bowl problem,” and that is being a good critical thinker isn’t necessarily going to make you popular, and so you need to be careful about when you point out to people that their arguments aren’t very good, because if you do it too much, you will end up being like the turd in the punch bowl at a party. People will look at you and say, “Oh, I’m going to stay away from that person.” And so, at the end of the semester, I will ask them, “Have you had any turd in the punch bowl experiences as a result of this class? And at the end of the semester, a lot of the students – and I don’t know if this is good or bad – but about half of the cases involve the sunk cost principle, and usually it involves the destruction of a relationship. So people will end a relationship because the only reason they’ve been sticking with the relationship is all the pain and effort and heartache they’ve put into the relationship.

John: And I know behavioral economists have noted that quite a bit. That once they really get this message through, it leads to a breakup of so many relationships of their students. But even people who know these things sometimes fall into these traps. And one example I think I mentioned on the podcast before, and I know I’ve mentioned it to Rebecca, is one time I was at a conference with a few other people, and we went over to Universal park, and it was just really hot that day, it was like 97 degrees, and we had spent quite a bit for the tickets for the day, and we were just kind of miserable. And one of the other people said we should just go. And my response to a historian was that we spent $230 and we’ve gone on two rides. We really should stay and get our money’s worth. And her response was, “But that’s a sunk cost. It shouldn’t matter,” and it’s easy for us to fall into that.

Michael: Absolutely, and indeed, this is one of the reasons I choose the rules that I do, is that I think many of them are significant. They really can make a difference to the quality of your decisions and the quality of your judgments.

Rebecca: Can you talk a little bit about the two-stage training program that was used in class?

Michael: Sure. So I gave you a little bit of an example of it already, but let me give you an example with another case with the correlation cheat code. So my go-to example here is I’ll tell the students, “Well, you know, yesterday I ate chocolate and today I broke out with zits. Boy, that means that eating chocolate causes zits,” and they immediately say, “No, that’s terrible.” So the first stage, what I’m doing is I’m introducing examples where they are disposed to use the rule. So nobody says, “Yeah, that’s great evidence for thinking that eating chocolate causes zits.” And then we talk a little bit about, “Okay, well, what would count as good evidence?” And they come up pretty quickly with maybe just a little bit of help, “Well, you want to know what happens when you eat chocolate and when you don’t eat chocolate, because what you really want to know is what difference does eating chocolate make to your complexion, and so what you want to know is what happens when you eat chocolate and you don’t eat chocolate, and everything else relevant to your complexion is the same, and what difference is there in these two cases,” and so that’s the first stage. The first stage is just introducing them to these rules, these cheat codes. So I give them the cheat code, but really a cheat code which they detect right away as a cheat code, and that they know as a cheat code. And then I spend the rest of the semester giving them arguments that use that cheat code, but are not as easy to detect. And often these are political arguments, and one reason I don’t get into trouble – at least I haven’t so far – in giving political arguments is that these cheat codes you can use them to defend views on the left, on the right, in the middle. They’re tools of manipulation. It doesn’t matter what the truth is. You can use these cheat codes to defend the true view, the false view, smart decisions, dumb decisions. And so, what I’m really interested in doing is getting them to be able to detect these cheat codes. So a first step is just introduce them to these cheat codes in a way that they will be disposed to accept and think is obvious. And then after that, I’ll give them lots of examples in which they’re, at least many of them, are not disposed to think is obvious. And then rather than me sort of coming from on high and saying, “Well, you know, you made a mistake there, dummy.” I just step back and I say, “Wait a minute. We seem to have an inconsistency here. You insisted that this rule was right on Monday, and on Friday you’re not using the rule in this case. What’s going on?” And I have yet to have a student, and I’ve taught 1000s of students, I’ve yet to have a student say, “Okay, I’m going to rethink that rule.” They all say, “Oh, well, I made a mistake here. I reasoned badly,” and by the end of the semester, they’re not reasoning badly.

Rebecca: Sounds like you spend a very small amount of time on stage one and significant time on stage two. Do you do all of the cheat codes at the beginning of the semester and then work on utilizing them throughout the semester, or do you kind of do units on different cheat codes? How do you organize?

Michael: So I will introduce a cheat code and then I’ll give them examples using that cheat code and all the cheat codes we’ve covered. So once a cheat code is introduced, they might see it at any point in the future, and I think it’s really important they keep practicing it and keep being able to recognize it, and not just think, “Hey, I know this is the cheat code we’re working on today, so I just need to be on the lookout for that one.” Another thing I do, and I think this is really important, is I will sometimes give them arguments that are promising, that if you grant the truth of the premises, they at least give you some reason to accept the conclusion. Because I don’t want the students just to be skeptical of all arguments. I want them to be able to distinguish between “Okay, that’s a cheat code. That’s definitely a bad argument. The conclusion might be true, the conclusion might be false, but that’s not a good reason to accept anything” and then “Okay, well, what would count as good evidence for thinking that this is a smart decision or that this policy has led to increased costs or decreased costs?” And I want them to be able to say, “Okay, look, that argument right there, that’s pretty promising. This argument over here definitely uses a cheat code.”

John: So you start off with a little bit of retrieval practice with some applications of the same concept, and then you move into some interleaved practice, which raises the cognitive difficulty just a little bit. But it’s one of those desirable difficulties where the more cognitive effort that students have to place in identifying what is at issue, the more deeply they tend to learn it, and the more likely it is that those concepts will stick and also transfer into other applications, which is what you’re building to in this program.

Michael: Absolutely, absolutely, the interleaved practice is really important, and I wouldn’t go back to sort of the blocking practice.

Rebecca: So we’ve talked a little bit about what the intervention is. Can you talk a little bit about how the study was conducted, like the size of the treatment group, the control group, etc.

Michael: Sure. So in the paper, we really describe just the second study we did, we actually did two studies, and in both studies we had almost 400 students. In the first study we had 395 students, and the second study we had 397 students. And some of the students were taking a critical thinking class from me, and the rest of the students were taking other philosophy classes. So in the study that we report on, we had 113 critical thinking students and 284 control students. Then what we did is, in the first week of class, we gave everyone a pretest, and this involved the five cheat codes, and then two good arguments, or at least promising arguments. They weren’t necessarily great, but they didn’t use any cheat codes. And the reason we included these, what we called skepticism checks, was just to check if all we were doing was creating skeptical students rather than students who could distinguish between promising arguments and cheat code arguments. Then, at the end of the semester, we gave them the same survey, same seven questions, and not surprisingly, in both of the studies, the training group students did much better than the control students. In the first study, we waited three months. So basically, the way to think about it is that these classes all happened in the fall. We waited till the end of the spring semester. It might have been four months, and gave them same questions again in the first study. We gave them the same questions again. The trained students did much better, and then we gave them different questions, and they still did much better. So in the second study we did, which is really the only study that we report on in the article, we gave them pre-test at the beginning of the semester, post-test. Then we waited a full academic year plus the spring, so we waited 16 months and only asked them different questions. So the pretest and posttest were the same questions, and again my students did much much better. But then we waited 16 months and asked them seven different questions, and the trained students were still doing much better, detecting the cheat codes and recognizing that the good arguments were still pretty good arguments.

John: DId the trained students do significantly better with all 5 of these cheat codes?

Michael: No, they did better on 4 of the 5 cheat codes. They did better on the correlation, regression, sunk cost, and opportunity cost cheat codes, but in all the studies that we did, on all the surveys that we asked, we just couldn’t get the trained students to do better on the gambler’s fallacy questions.

John: But still, having students do better on 4 out of those 5 cheat codes so long after the initial intervention is a really impressive result. One of the things you talk about near the end of the paper, though, is that you note that there are some possible sample selection issues. Could you just talk a little bit about what source of sample selection might have affected the magnitude of those results?

Michael: So one worry that one might have is that the students were not distributed randomly. So the students chose to take the critical thinking class, the students chose to take other philosophy classes, and so there might be something about the students who chose critical thinking classes that would incline them to be better critical thinkers, despite the fact that on the pre-tests, all the students in both groups did equally badly. So that’s one worry one might have, and the thing to do there is, although it’s not really easy given the institutions of academia, is to randomly distribute students into critical thinking classes and other classes. So that’s the main selection worry. Another reason why we might have gotten good results is that I had those students for 15 weeks. Some interventions talk about full- semester interventions, but often they’re fairly short interventions. I was able to have the students for 15 weeks and made them practice a lot.

Rebecca: It sounds like in the design of the experiment in your class, just generally, there’s lots of opportunities for feedback and correction throughout the semester.

Michael: Yes, absolutely. There’s a lot of feedback, a lot of correction, and I have a big bank of good and bad arguments that I can give them over the course of the semester that use these cheat codes or that don’t use the cheat codes. And one of the things I do for an essay in this class is ask the students to put themselves in the shoes of someone who’s trying to come up with a cheat code argument and ask them to give me bad arguments that use these cheat codes, and they actually turn out to be pretty good at it. And many of the examples that I use on later students are examples that prior students have come up with.

Rebecca: Sounds like it makes it a little bit fun too.

Michael: Yes, I do ask them for any humorous examples or examples that us boomers would have a hard time understanding. [LAUGHTER]

John: One of the things we appreciated about this study is that the data, the results, and the analyses that were done are all reported in more detail on the Open Science Framework website. We’ll include a link to that website in the show notes for this episode.

Rebecca: So we always wrap up by asking, “What’s next?

Michael: A few things are next. One is I’m writing a textbook that uses the materials that I teach. The way I teach critical thinking is different from the way other people teach it. I haven’t found a good textbook that teaches critical thinking the way I like to teach it. So I’m currently writing a textbook that hopefully will help other people get good results. I’m also planning to build, if I can, a game that anybody can play to see whether they can avoid being bamboozled [LAUGHTER] by cheat code arguments, this is sort of a longer term project because writing a book is going to be useful, hopefully, for people to teach this. But what’s really important, in my view, what makes this stick is practicing on lots of examples. And so, in a game where people see an argument and you say, “Well, is this promising argument, or is somebody trying to manipulate you?” we’ll hopefully give people practice at being able to detect those cheat code arguments. So those are two of the projects that I’m working on.

John: On the latter project, will it be a board game? Will it be an electronic game? Will it be an online game, or is that something that’ll be decided further in the future?

Michael: No, it’ll be an online game. Is what I’m hoping to do. I want to do something that can be used not just by my students and that’s easily accessible.

Rebecca: Sounds like some great projects. Thanks for joining us today.

Michael: Oh, you’re welcome. It’s been a pleasure.

John: Thank you. I was really impressed by this study [LAUGHTER] because, as I mentioned before, it’s so rare to see effects that are so significant and persistent. It’s, I think, something that people can learn from in terms of other types of issues where it’s hard to convey this sort of long-term impact on students.

Michael: Well, thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

455. Transformative Coaching

Higher ed has become an increasingly complex and challenging environment to work in. In this episode, Susan Hrach and Kitty Maynard join us to discuss the role trained coaches can play in supporting agency, resilience, and career growth for college employees.

Susan is the author of Minding Bodies: How Physical Space, Sensation, and Movement Affect Learning. She’s served as a Fulbright Canada Distinguished Research Chair, and recently appeared on the TEDx stage in Columbus, Georgia. Kitty is the Director of the Teaching and Scholarship Hub at the University of Richmond. Her higher ed career began as a Professor of French Studies, and she has taken on leadership roles in faculty development at Washington College and Virginia Commonwealth University before moving to the University of Richmond. She is the author of Reveries of Community: Epic in the Age of Henri IV and co-editor of Polemic and Literature Surrounding the French Wars of Religion. Susan and Kitty have both received training from, and are members of, the International Coaching Federation. They are also two of the editors of Transformative Coaching for Faculty and Staff in Higher Education.

Kristin Croyle, a psychologist and the Dean of the College of Liberal Arts, Sciences, and Engineering, fills in for Rebecca in this episode.

Show Notes

  • Hrach, S. (2021). Minding bodies: How physical space, sensation, and movement affect learning. West Virginia University Press.
  • Maynard, K. (2017). Reveries of Community: French Epic in the Age of Henri IV, 1572–1616. Northwestern University Press.
  • Kendrick, J., & Maynard, K. S. (Eds.). (2019). Polemic and literature surrounding the French Wars of Religion (Vol. 68). Walter de Gruyter GmbH & Co KG.
  • Rice, K. G., Hrach, S., Linder, K. E., & Maynard, K. S. (Eds.). (2025). Transformative Coaching for Faculty and Staff in Higher Education: Powerful Tools to Address Institutional Challenges. Taylor & Francis.
  • Little, D., & Palmer, M. S. (2011). Coaching-Based Framework for Individual Consultations. To Improve the Academy, 29(1), 102-115.
  • Cannon-Bowers, J. A., Bowers, C. A., Carlson, C. E., Doherty, S. L., Evans, J., & Hall, J. (2023). Workplace coaching: A meta-analysis and recommendations for advancing the science of coaching. Frontiers in Psychology, 14.
  • Jones, R. J., Woods, S. A., & Guillaume, Y. R. F. (2016). The effectiveness of workplace coaching: A meta-analysis of learning and performance outcomes from coaching. Journal of Occupational and Organizational Psychology, 89(2), 249–277.
  • Reynolds, M. (2026). Coach the person, not the problem: A guide to using reflective inquiry. Berrett-Koehler Publishers.
  • Pope-Ruark, R. (2022). Unraveling faculty burnout: Pathways to reckoning and renewal. JHU Press.
  • International Coaching Federation
  • ICF coach finding tool

Transcript

John: Higher ed has become an increasingly complex and challenging environment to work in. In this episode, we explore the role trained coaches can play in supporting agency, resilience, and career growth for college employees.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Susan Hrach and Kitty Maynard. Susan is the author of Minding Bodies: How Physical Space, Sensation, and Movement Affect Learning. She’s served as a Fulbright Canada Distinguished Research Chair, and recently appeared on the TEDx stage in Columbus, Georgia. Kitty is the Director of the Teaching and Scholarship Hub at the University of Richmond. Her higher ed career began as a Professor of French Studies, and she has taken on leadership roles in faculty development at Washington College and Virginia Commonwealth University before moving to the University of Richmond. She is the author of Reveries of Community: Epic in the Age of Henri IV and co-editor of Polemic and Literature Surrounding the French Wars of Religion. Susan and Kitty have both received training from, and are members of, the International Coaching Federation. They are also two of the editors of Transformative Coaching for Faculty and Staff in Higher Education. Welcome back, Susan, and welcome, Kitty.

Kitty: Thank you, thank you so much.

Susan: Thank you,

John: Rebecca cannot be here today. So, once again, our Dean of the College of Liberal Arts, Sciences, and Engineering here at SUNY Oswego will be filling in, and that’s Kristin Croyle.

Kristin: Thank you. I’m delighted to be here. So, today’s teas are, John, what are you drinking today?

John: I am drinking a spring cherry green tea.

Kristin: Oh ho ho. and for our guests, what are you drinking today?

Kitty: Well, I’m drinking a tea that is called Richmond Blend. I’m here in Richmond, Virginia, and the tea is from Boscovs, which is a sort of an iconic coffee and tea purveyor here in Richmond, Virginia, and it’s sort of a spicy cinnamony clovey tea. It’s probably better for Christmas time, even though we’re in July, but I definitely wanted to play along, because I’ve noticed, anyway, because I listened to your podcast quite a bit, that a lot of your guests have not been following and really respecting the conceit of the show, [LAUGHTER] so I wanted to make sure to do that today with my tea.

Susan: Oh no, and I’m a violator [LAUGHTER] because it’s going to be a swelter today here in Columbus, Georgia, and so I am having a Limoncello LaCroix, or La Croix, [LAUGHTER] as fancier people might want to pronounce it now. [LAUGHTER]

Kristin: Now, if this was afternoon here… we’re recording in the morning… I would be drinking a cold Earl Grey, but morning… just water today.

John: And going back to your earlier comment, one of our serial offenders has been Derek Bruff, who brings his tea for teaching mug, which he refers to as his coffee mug. [LAUGHTER]

Susan: Oh no. [LAUGHTER]

Kitty: It’s heresy. [LAUGHTER]

John: It’s always great to have him on, though, when we do get a chance to talk to him. [LAUGHTER] So, we’ve invited here today to discuss transformative coaching for faculty and staff in higher ed. Could you tell us a little bit about how this book project came about.

Susan: So, this is a book that was born at the Pod Network Conference in 2023 in Pittsburgh, and Karen Gonzalez Rice and I decided to pitch a birds of a feather session, which POD attendees may know more about. They’re very informal conversations. They usually take place first thing in the morning on one of the conference days, and they’re opportunities for people to informally discuss something that’s of sort of emerging interest. And so we pitched a birds of feather session on coaching. Kitty was in the group of attendees that day, and she was already working with Katie Linder and her coaching program for academics, and so within a couple of months after that November conference, the four of us had kind of connected and expressed interest in doing an edited collection and put together a call for proposals, and we were off to the races.

Kristin: Excellent. And who’s the intended audience for the book?

Kitty: Well, it’s really interesting, because we started out thinking that this book was going to be mainly for other coaches working in higher ed, or possibly people who we call coaching curious, just those who are wondering what coaching is about, and maybe if they want to become a coach. But it was really interesting as we started to talk to potential collaborators and contributors, our vision of what this book could be really expanded, and we realized that coaching has applications that really go beyond what we had initially proposed, so while the most obvious audience would be those who support faculty and staff in higher ed, that’s to say, people like educational developers, or maybe even folks who work in HR, several pieces relate directly to, for example, faculty roles like chairing a department, or even teaching, and so we ended up adding some other chapters to have a more capacious vision of what coaching could do in higher ed, so we have chapters from mid-level administrators, we have some other chapters for people working in different parts of higher ed, like instructional design, and I guess I would say our real wish for an audience, our sort of shadow audience, is upper-level administrators who hold the purse strings and can really hopefully have an opportunity to see how coaching might benefit their institutions and those working in them, and how it might be deployed throughout the university, but I think our overall takeaway here is that coaching is a really nimble way to support individuals at a complex moment in higher ed and the world.

Kristin: Excellent.

John: How did you select the authors of the chapters of this book?

Susan: We circulated our call for proposals widely among all of the coaching networks that the four of us were already a part of, and encouraged them to share freely. One reality that we recognized in seeing the need for this book was that there aren’t that many… there really aren’t any… formal organizations for those who work in coaching in higher ed, and so it had to be sort of a grassroots solicitation, and we got a really generous, lovely set of submissions that then we got together, talked about where there might be some topics that we wanted to see, but no one had submitted on, recruited a few more people who we knew had expertise in those areas, and overall we’re just super excited to have almost 30 contributors, or maybe a little more than 30 contributors to the book.

Kristin: Yeah, I was struck in looking through the book at both the unified voice, but also the really personal voices in it, and having that many contributors gives you an opportunity to have a lot of different voices in a kind of personal way. The book is divided into six sections. You were talking previously about all the different aspects that you wanted to bring in. Can you give us an overview of what kind of thing is addressed in each section?

Kitty: Sure, I’d be happy to. We start with a section on the foundations of coaching, which is really an introduction to what coaching actually is and is not. Susan just referred to the fact that sometimes coaching isn’t even really well defined, and can even have a questionable reputation, depending on who you’re talking to. So, this book, in the intro, we really established the norms of the International Coaching Federation, which is where most of us have been trained, or we’ve been trained in with organizations that are adjacent and use some of those same principles, and that is defining coaching as a collaborative process between a client and a coach to support the client in reaching goals. And then the second part is about growing coaching skills, and it really explores how to develop some of the essential skills and competencies that we associate with coaching, and how those can support folks in higher ed. So we talk about cultural awareness, we talk about knowledge of the profession, and we also talk about ways we can support the entire spectrum of neurodiversity with our clients. The next section is about applying coaching skills to specific scenarios, and so we really explore how coaching can be used in teaching, research, and leadership positions, and a subsequent section talks more about coaching for specific audiences, so we look at coaching for folks who are in roles like senior administration or student affairs, instructional technology, and also people who may just be having certain experiences in higher ed, for example, those experiencing burnout. And then we have a section on coaching the career cycle, where we focus on how coaches can help develop or help others develop vitality in the different stages of their careers. So that one’s a little more faculty focused, although not exclusively so, and then our last section is about group, team, and peer coaching materials, and that section we think about how we can scale coaching to reach more people than just a typical one-on-one individual interaction, so we think about how coaching can play a role in organizational transformation, or how we can use coaching to develop community.

Kristin: I appreciated your use of the term coaching curious, because I think having that variety of topics really gives people a chance to see the range with which coaching can be used in higher ed.

John: What led you to see coaching as a useful framework for faculty and staff development in higher ed?

Susan: Well, I had begun my academic career working in writing centers, and in that field the writer is always given the agency and autonomy to be in charge of their own work, and you work with the peer tutors, or the peer consultants, as they’re called sometimes, to respect the control of the output to the student, and so when I first learned about coaching, I immediately recognized a similar kind of ethos, which is different than mentoring and different than consulting, and different from therapy in an important way that maybe we can address later, but mostly in that a mentor or a consultant is someone who has the knowledge that they’re planning to impart to the person being mentored, whereas coaching respects the client or the coachee as the expert in their own situation, and the person who the answers really need to be drawn out from. And so I also connected with it as an educational developer, and thinking about active learning, because telling people what to do just doesn’t work, [LAUGHTER] as wise as it may be, and telling them about the knowledge or skills you want them to develop is just not as effective as allowing them to come to those realizations for themselves, and so I saw the value in it. I was already committed to its usefulness in learning, and coaching, in the end, is a lot about the learning process of the client.

Kristin: And John didn’t mention, but I’ll say, I’m a clinical psychologist, that’s my background, so I’ll just double down on what you said there. [LAUGHTER]

Susan: Yes, thank you.

Kitty: And I would also say that coaching for educational developers, coaching is kind of in the air anyway. There’s a really canonical article from 2011 by Michael Palmer and Deandra Little, which is called “A coaching-based framework for individual consultations,” and a lot of us use that, and I used it for years to help me be a better consultant when folks come in for specific pedagogical reasons, or even just to talk about career questions, and so I was already using some of those principles, and so it was a really exciting opportunity to develop those coaching skills and really take them further.

Kristin: Beyond some of the iconic articles in the field, your book cites several studies concerning the impact of coaching in workplace settings. Tell us a little about the research foundation here.

Kitty: Absolutely, so we have a very short literature review in the book that will point you to some helpful resources that talk about what coaching is and how it works. In particular, there are two meta analyses, one that came out in 2016 and one that came out in 2023 that have both looked at the literature to demonstrate how coaching is an effective workplace intervention, and more specifically, what that could mean is that coaching has been linked in the research to things like greater optimism, work-life satisfaction, engagement in work, productivity for workers, and it’s also been studied in terms of how it can improve management skills for those who are in supervisory roles. So, there’s a recognition there that more research is needed, that it’s really an evolving field and has become more and more common in the workplace over the past 20 years, but there’s also a lot that’s very promising there, and I would just say that what it really does for me is confirm the positive results that we tend to see firsthand as coaches in our sessions, or even those of us who’ve been coached ourselves, so it sort of confirms that anecdotal evidence that we already had, that it’s a very powerful practice.

Kristin: Thank you.

John: Your book title uses the word “transformative.” What makes coaching transformative rather than simply supportive or developmental?

Susan: Yeah, we address in the introduction just briefly the fact that the word transformative is more than just an adjective for a number of theories of leadership, and we want to signal with that word that we are taking a particularly open approach to multiple different ways of thinking about transformation, but I can say about the way that coaching, in particular, signals transformation to me comes from one of the leaders in the field, Marcia Reynolds. She wrote a book, I think, called Coach the Person, Not the Problem, [LAUGHTER] in that one of the most important questions and transformational questions in coaching sessions can be what makes this important, and so you’re drawing out from the client not just a solution to their immediate problem, but something deeper, something that would signal a shift that is a new perspective for them, a new insight that might change their behaviors or even beliefs, and so in that respect it’s more than a surface- level change.

Kristin: I know that none of us are unaware that higher ed is going through a lot of change right now, and that we have many faculty and staff colleagues who are experiencing burnout, who are feeling particularly uncertain, who may have difficulty seeing what their future is in higher ed and aligning their own values to some of the changes in their regions of the country. Tell us a little bit about how coaching can be useful in a setting like this.

Kitty: So I think to go back to our conversation about that word transformational, I would say that coaching is really particularly useful for people navigating transitions and change, and frankly, that’s when most people start to seek out coaching support is when they’re going through something where they’re really not sure what to do, [LAUGHTER] they’re really not sure where to go next, and so anytime there is a lot of uncertainty and change in someone’s life, coaching can be a really helpful response. It kind of gives people back that sense of agency in some way or another. In terms of burnout, in particular, we have a really great chapter in our book by Rebecca Pope-Ruark, who wrote the book Unraveling Faculty Burnout, where she really explores this in terms of faculty, and in that chapter she offers some tools for coaches to help people rediscover purpose, compassion, connection, and balance, and those are the four things that she notes that are pillars of burnout resistance, the things that help us find ourselves when we’re facing those challenging moments in our careers. And I would say more globally, as we’ve started to think about coaching as an ecosystem that could exist within higher ed, I see it as a hopeful space where we can actually build more humane and inclusive higher education workplaces that really respect the individual needs of the people who work here.

Kristin: Thank you.

John: Teaching centers and faculty developers often serve a role where they answer questions for people, they provide models of effective teaching practices. They do workshops and so forth, but the coaching approach is a bit different than that type of interaction. Can you talk a little bit about how a coaching approach might change the nature of interactions with teaching centers and what a typical coaching session might look like?

Susan: So, the most conventional kind of coaching is one on one, and so it’s not a workshop typically, not that a workshop can’t employ coaching strategies, but as we were discussing earlier, it’s about drawing out the expertise of the person that they already possess, so a coach would ask: “Where are you now? What would be different if you achieve this change that you want to make? What are your options for getting there? What obstacles can you anticipate, and how will you respond? What’s the first step?” And so those are each pieces of a coaching framework that the client has to answer for themselves, and so if you think about it in terms of a client who might be also interested in changing their teaching strategies, let’s say it’s slower but might be ultimately more successful route to making that change. You’d have to acknowledge that it’s going to take significant time and resources to adopt new teaching strategies and refine them, or to envision a different career path, or a purposeful retirement. And coaching is allowing you to get clear on why making that change is important enough to invest that time and energy. And so if you don’t know what strategies you might want to employ, then the coaching conversation might include, “Let’s talk about what your resources are. Who do you know? Where could you go? What actions can you take in order to find those things.” So again, you’re not providing the answers, you’re sort of helping the person think about where they want to pursue those.

Kristin: And I know many faculty are well versed with what an individual teaching consultation might feel like, but how would that differ from how a typical coaching session might feel or look like?

Susan: It’s different in that, and this is very important for clients to understand, it’s usually part of the initial meeting in which we take time to talk to them about their expectations and how coaching is different from mentoring or consulting, which is that the coach is not going to hand you the answers. [LAUGHTER] If that’s your expectation, then this is not, maybe the interaction, or the…

Kristin: Oh, but that would be so much easier., wouldn’t it? [LAUGHTER]

Susan: It would…

Kristin: You’d just have the magic answers on hand.

Susan: It would and yet, it wouldn’t be as successful in leading to a lasting change, or growth, or learning, which is hard and takes effort on the part of the person who’s seeking out that growth.

Kitty: And I would just add to what Susan said by saying there are times when coaching isn’t the solution. Sometimes, in a pedagogical consultation, your expertise does come into play, that people really do want to know, like, what should I do in my classroom, what kind of warm-up activities should I do in my classroom, and do you have any ideas for me that I could use? And that’s not a coaching conversation, that’s a consulting conversation, and that’s okay. It doesn’t all have to be about coaching, but I think that coaching skills are valuable, and I think to get back to Susan’s point about her writing center work, the idea that you are seeing the person who’s coming to you for a consultation or a coaching session, you’re seeing them as someone who has more answers than maybe they even realize, and helping draw that out for them with powerful questions, and sometimes coaching tools and deep listening, that is valuable in a consultation too, even if it is a moment when you maybe are going to give somebody a more direct answer, or you’re going to be like, “Here’s a book you could take a look at, and it has all sorts of great ideas in it.” So, they both can be very helpful in this work.

Susan: That’s right.

Kristin: Yeah, I like the writing center analogy, because the best writing teachers are helping people find their voice and express their own voice, they’re not telling them what to write.

John: So, part of coaching involves having the clients identify their own goals, but one of the problems that we all have sometimes is having the motivation to achieve those goals. How can coaches help their clients remain motivated to achieve their goals?

Kitty: Well, this is a foundational piece of what coaching does, and I think the first thing is that it helps people identify their values as well as their true priorities, which sometimes can get lost in all the doing that consumes so much of our time, so it taps into intrinsic motivation, which means that if we are able to set goals that matter to us, that helps us make meaningful progress on those goals. I think sometimes we have goals in mind and they’re actually not our goals, their someone else’s goals. [LAUGHTER] I mean, sometimes that has to be the way it is in our jobs and in our lives, but a lot of times there are ways that we can find to tweak the goals, so that we are actually more naturally motivated to accomplish them. But coaching is also really helpful with project management. It’s helpful in how to plan to find mentors and to find the help that you need to accomplish a goal, and what kind of resources are needed, and then it also can provide accountability structure, which is another thing I think, in particular for academics who do a lot of independent work, where deadlines are a little bit more fluid, or sometimes incredibly fluid, depending on where you are. It helps kind of stay on track. Also, sometimes, for example, in coaching, I actually will write follow-up emails to people, so that’s part of my support is checking in to say, “Hey, last week you said you were going to do this, how’s it going for you, how is that going? And sometimes that’s enough to help people kind of loop into the goals that they want to achieve.

Kristin: And I know that faculty who have worked in a more traditional faculty role versus ones who are working in an administrative role, where there’s more of a team environment, will sometimes find that different structure also to be helpful. So, yeah, it’s finding ways to build your own structures in, yes?

Kitty: Absolutely. And that’s one of the other types of conversations we would have with a client, is: “What’s going to help you keep on track. What do you need to keep on track?” and “How are you going to build that into your life?”

Kristin: What role can coaching play in helping faculty and staff make more sustainable decisions about their time, their energy, their professional identity? How they can make decisions that will pay off for them in their own satisfaction for years to come.

Susan: Yeah, thank you for that question. I think it’s been really interesting to me over the past few years of working with colleagues how much it matters to have someone give you their undivided attention and presence for a number of hours as your coaching series evolves. Even when I was feeling very sensitive about my own development coming out of training as a coach, I would be very conscious of where I had not asked the best question or had fallen through the cracks of where the conversation might have gone and yet there was no question by the end of the conversation that the person was just so pleased to have had the chance to talk and have someone really listen to them carefully, and how desperately people needed that, [LAUGHTER] and so I would say that the sustainable decision making is partly an outcome of having made the choice to invest time and energy in a coaching relationship, like you’ve already taken the first most important step, which is to say, I am going to carve out some time, which, as we know, time is maybe the most precious resource that most academics have, we’re not in it for the money, but we sure do spend time. So, if you’ve decided I’m going to intentionally dedicate an hour on several occasions to talk through an important issue for me, then you’ve already kind of signalled to yourself that this is something you’re prepared to sustain and be committed to, and as Kitty was saying earlier, the accountability that coaching provides is one of the really important things that we practice in between each session, and that ultimately by the time we reach the point where we’re ready to at least temporarily bring that series to a close, t he person needs to have identified what their own accountability practice is going to involve, and so there’s a sustainability priority that’s kind of built into the coaching conversation.

Kristin: I can imagine that some people who have not experienced coaching may listen to this and may think, but I have a very supportive colleague or partner or friend that I have these kinds of conversations with. What do you see as different in the coaching relationship than a great supportive personal relationship?

Susan: That’s a great question, and it’s a question that I asked myself when I was starting the training, because I thought, “Well, you know, I have decent conversational skills. How hard can this be?” And then I realized that it’s actually incredibly hard, because coaching conversations do not follow normal social turn taking. You, the coach, are not an equal, like, interlocutor in that relationship. The client should be doing way more talking and way more thinking, and you are posing powerful questions for them. And so it is not a normal conversation, [LAUGHTER] and it’s hard to develop the skill to sort of be quiet and make space for their thinking to emerge. Sometimes there’s long silences while you’re allowing people to process their thinking, and that’s not the typical friend or colleague kind of conversation.

Kitty: And coaching sessions are actually very highly structured. We follow a pretty standard structure of every coaching conversation, so in that sense, as well, it’s different than having a good supportive friend or colleague, although we all need that too. So I certainly don’t want to say that those conversations are not incredibly valuable, but I would say in my own experience, when I’m having a conversation with a colleague or friend, a lot of that is about getting advice, specific advice from people, which is also incredibly helpful, but coaches aren’t going to give you that advice, they’re going to draw out your own advice for yourself. One of my favorite coaching questions is something that Katie Linder taught us, which is a lot of times clients come to you and they actually expect to be getting advice from you, and so that’s something that we sort of have to prime with our clients, is that we’re not going to be giving you specific advice, because that’s something we’re going to trust you to find yourself, but when somebody does come to you and they really want advice, Katie’s question is, “What would you like me to give you permission to do?” because it turns it back on them again to say, “Hey, what do you really want to do, what are you looking for me to say to you that you want to do yourself?” which I think is a very clever question, and it’s really fun to ask it, [LAUGHTER] it’s really fun to see what kind of answers you get.

Kristin: And I’ll add, too, that having a separate relationship, a coaching relationship, you don’t have to worry if I say this thing, how is my friend going to judge me, or if I say I want to do this, am I ready to talk about this piece with my partner? There’s other complexities in our long-term personal relationships that in a coaching relationship you don’t have to worry about how are they going to change how they see me or what is the impact of this going to be, where there’s a reciprocal kind of emotional care where you’re trying to take care of the other person. Coaching gives people an opportunity to not have to care about [LAUGHTER] not in the same way, to not to take care of the other person, that that’s the time for them to think about how to care for themselves.

Kitty: That’s an incredible point. I think it’s really, really important, and gets us back to some of the principles of coaching, where we are confidential. It’s really our relationship with our clients, it is 100% between us, it never leaves that conversation unless it’s some professional obligation, because someone is expressing harm or something like that, but otherwise it is absolutely confidential, and so people can try on things that they might not feel comfortable trying on with a colleague or a partner.

John: It sounds a lot like psychological counseling in many ways, doesn’t it?

Kristin: In just certainly non-directive approaches, there are similarities, and many of the roots come from the same roots.

John: One of the things you address in the book is how coaching needs will be different for different people at different stages of the career and in different roles. How might coaching needs differ for early career faculty, mid-career faculty, senior faculty, chairs, and staff leaders?

Kitty: I would say in many ways the coaching approach doesn’t change much with all of these audiences, because our practices are similar. We are going to ask powerful questions, we’re going to be listening deeply in a way that empowers the individuals we work with, but coaches often develop expertise in a particular area, depending on where they’re working and the spaces they’re working in, and when that happens, coaches tend to see similar issues come up, so they become more skilled at asking the right question at the right time, or understanding a particular tool is going to be really useful for somebody in a particular career phase. So to your question, John, it’s not unusual, for example, when I encounter people in their early career to be struggling with finding mentors or to be trying to figure out what kind of workflows are going to actually be effective for them as they’re juggling all of the new things that they may have not had to juggle in graduate school. On the other hand, someone who’s a little bit more advanced in their career might be thinking about legacy or thinking about retirement. Someone in mid-career might be just getting tenure and trying to figure out what’s next for them after they’ve jumped all these hurdles and they finally come to the end of the hurdles and are looking ahead and saying, “Well, what do I do now?” So, really, I think a general coaching approach can help with any of those concerns, but a coach with a background in those areas is likely to have the tools to address a particular need. And so, one of the nice things about our volume is it has some great chapters to help both coaches and those who are kind of going through those different career phases.

Kristin: How can coaches help faculty and staff navigate conflicts between their own goals, values, and commitments and institutional pressures, particularly in settings where coaching may be available within the institution.

Susan: This is a really important question, and it’s one that touches on some things we’ve already mentioned having to do with ethical commitments. The International Coaching Federation, in fact, requires an ethics training as part of the credentialing process and re-credentialing process, and so for internal coaches this is particularly important, and as Kitty mentioned, the confidentiality piece is primary, but the other thing that kind of surprised me when I first started the training and remains really important to my sense of the value of doing this in higher ed is that the coach’s primary responsibility is with the client, not the institution. And so, I was really kind of shocked when I asked the question in my training: “Well, what if the person wants to quit? Then what do you do?” And the response was, “Well, then you need to explore that with them.” This is not a place where your job is to make sure that they stay with the institution, no matter what. You’re helping them think through a very difficult professional crossroads, and so if maintaining their affiliation with the institution is really a valuable and meaningful priority for that client, then okay, we know that that’s one of our anchors. We’re going to have to figure out how to manage this conflict in other ways, but if the conflict turns out to be more significant than staying in that job or role, then the job of the coaching partnership is to think about how to manage this transition. What would be a better fit? What does that look like, how are you thinking about that, and so there’s a great chapter in the book, in particular by Chinasa Elue, that is about coaching. Well, let’s see, what does she call it?

Kitty: Walking the tightrope…

Susan: Walking the tightrope: Coaching faculty clients who decide to leave the university.

Kristin: And as a Dean, I will say that if someone talks with me, and there are faculty and staff who talk with me about their quandaries about whether they should stay or go, I actually see my role in a very similar way. I want them to be thriving. If they’re thriving here, it’s good for the institution, but if they’re not thriving, it’s not good for them, and it’s not good for the institution,

Susan: That’s right,

Kristin: But that’s a very different conversation, certainly, than it would be with a coach.

Susan: It is, and I have noticed that it’s particularly difficult and regrettable that the system of tenure makes many people feel that there is no possible way for them to walk away from this. They grabbed the gold ring, and so, boy, they better stay. And so it’s a life crisis. [LAUGHTER] And so coaching is a really wonderful tool to have to be able to work through that kind of crisis.

John: So, who should be trained in coaching skills? Clearly, from the main theme of the book, teaching center staff should be, but might this also be appropriate for chairs and deans and student affairs professionals and supervisors and peer mentors, and might it even be helpful for faculty, because our relationship with students is also a coaching relationship in many ways.

Kitty: Yeah, I would say absolutely to all of those things. I think coaching skills are really helpful for anyone, and in particular, anyone in a type of role where you’re being asked to mentor or support or guide others in some way. I think one of the biggest revelations for me during my coach training was the breadth of occasions where I found deep listening and powerful questions to come in handy, not just professionally, but even personally, in some moments. So, I think when I think of faculty members, John, as you noted, working with students, coaching skills can be so helpful, or if you’re doing advising with students in any way, but I’d also say with colleagues, I think one of the things that I have experienced myself as a former faculty member is we’re not always good at talking to each other, we’re not always good at listening to each other, we’re not always good at asking the right questions of each other, and so those skills can be applied in all sorts of contexts.

Kristin: What mistakes should campuses avoid when they first try to introduce coaching?

Susan: In the beginning of the book, we have a couple of chapters. These are Katie’s on how to seek out certified training programs, what questions to ask, what are all of the options you might consider, and for what reasons. But training is key. So I hope that our book is an important contribution to people who are at the coaching curious stage, but I wouldn’t want someone to read our book and then decide, okay, we’re ready, [LAUGHTER] let’s do it. There’s some very important standards that we want coaching professionals to be able to meet and uphold, and institutions can also look into outsourcing, just like with any other professional development. The imperative to develop your own internal coaches doesn’t have to be where you start, and in the corporate sector, executive coaching has been a thing for a long time, and I think upper-level administrative leaders get a little bit of that because they fall into that kind of executive identity, but what we’re really hoping to do is democratize this resource, and so where deans or provosts or others who have control of institutional budgets might be interested in whether coaching could be a useful way to help people who could benefit. I think there are other places to begin.

Kristin: Yeah, I think the influence of kind of corporate leadership coaching has been a positive in many ways, because people see it, they see the value, they think, “Well, if that’s something that is a given in those settings, and people will pay a lot of money for it in those settings, that it must have actual value.” So that people who are not responsive to the research foundation may be responsive to that argument. Instead, I think the first time I hired a coach for someone was 15 years ago, saying, “Look, you have great leadership capacity and skill, but it’s not something that you’re able to bring out. Let’s see if a coach can help bring that out.”

Susan: Oh, thank you for saying that, Kristin, and if I could just add something else about ways we would prefer not to see coaching implemented is as remediation. Your impulse was to see someone who has great leadership potential, because coaching is about where are you now and where do you want to go, and I would love to h ear more from you about how you see therapy and counseling as distinct from that kind of orientation, because my understanding, or the way that I explain it to my clients, is that if we get to a place where you seem unable to move past something that happened to you in the past or some conflict you had with someone in the past, and we just keep being drawn back to this issue that clearly needs to be resolved or healed in some way, then I’m going to make a referral to you to a professional counselor, a professional therapist. A coach should be trained to recognize when referrals are the appropriate action.

Kristin: Absolutely, absolutely, and of course it depends on the therapeutic approach. Some therapeutic approaches focus more on skill building as well, or someone may not have developed the skills that they need to really thrive in interpersonal relationships, for example, or they may have significant substance abuse backgrounds, or there may be all kinds of things going on, and a great therapist is tooled to be able to handle complexity and to be able to recognize how those complex pieces act together and help somebody move forward in their lives and their goals. I certainly would think that the ultimate goal is very similar. We’re all trying to help people in education as well identify the things that they see as really valuable in their lives and the capacities that they can see that they have to live a meaningful and joyful life, that is our goal across the board. I’m sure that is the goal of many of our faculty as well, with the students that they work with. We just take a somewhat different approach in doing so. I’m also thinking that listeners may have one other question. You may have gotten them really excited, so can I ask the question, if someone is looking for a good coach, what might you suggest in how they could find one.

Kitty: Well, I would say one thing to know about coaches is that most of them do a free half an hour consult with their coaching practice, so there’s nothing lost if you find someone who looks interesting to you, and you just want to have a conversation to see if you vibe with that person and their approach. In terms of finding a coach. I would look for someone who has an ICF credential personally, just because having gone through that training, I know that it’s quite rigorous and quite demanding, and the people there have been trained quite a bit with ethics, and they’re trained to recognize the difference between therapy and coaching, and know where their limits are. So, I would definitely look for that as well. And I might just ask around a little bit to see if anybody you know has had a great experience. I know that’s how I found my career coach that I had back in the day, that I had a really excellent experience with, which was one of the things that got me into coaching in the first place. So, Susan may have some better advice than I do on that one.

Susan: No, I think you answered that really nicely. There is, I believe, a coach finder tool on the ICF website that would be the whole breadth of coaches all over the globe, so that may be a deeper dive [LAUGHTER] than you’d want to do right at the beginning.

Kristin: And I’ll also double down on a short consult session is a great idea. Sometimes people just don’t jive, and it’s no reflection. The coach may be a fabulous coach, it doesn’t mean that the person requesting coaching is all screwed up because they didn’t jive with this coach. Sometimes there’s just not a connection, and it’s not a reflection on either person, but it’s worth that first conversation to say, “You know, this just didn’t seem to go. I’m going to just try again,” because having a good connection can provide a really good foundation.

Susan: That’s right. And coaches are typically very generous. I have also had other coaches make referrals to people who might be better suited, for example, when I was seeking out mentor coaches for my re-accreditation, and I appreciated that generosity.

Kristin: Tell us about your experience of writing the book.

Susan: I would really like to praise our contributors to this volume, not least of which for being the most organized, well-managed time writers that I think have ever existed in a co-edited collection of this size. It was such an interesting experience to work with people who all sort of work with other people about time management issues and writing, because they were on it. No one had to be repeatedly nudged into getting their submission in. It was a really smooth process, and it was such an enjoyable experience to work with Kitty and Katie and Karen, because four people is a nice group as a editing team. When one of us had a life got in the way of being able to keep our balls rolling, the other three people would always pick up the slack, and so I think we were able to make incredibly smooth progress on getting this book to the press.

John: We always end with a question, what’s next?

Susan: I am actually heading full force into this coaching world. As of this summer, I’ll be leaving my role as a director of Center for Teaching and Learning to focus on coaching for individuals and for organizations, in addition to the speaking engagements and workshop facilitating that I have been so honored to do with a number of universities, my focus on active and embodied learning and my coaching practice specifically offers an embodied approach, because I like to conduct outdoor audio only sessions, in which clients can be walking or enjoying some fresh air to develop new perspectives and spending time outside.

Kristin: Excellent.

Kitty: And for me, I’m sort of in the midst of a lot of reflection and planning for the academic year here at the University of Richmond. So, I’ve been particularly busy thinking about ways to support folks in our mid-career. We revamped our early career support programs with the Provost Office last year, and this year we’re starting to focus on mid-career folks, so really post-tenure, or for those not in a tenure-line position, more advanced teaching associates, and so I last year did a group coaching with the Associated Colleges of the South across all the schools of that consortium, and we’re going to do one here at UR next year. So, I’ve been working a lot on that and thinking about group coaching and how we can apply that more effectively in my institution.

Kristin: Sounds so interesting.

John: Thank you both. It’s great talking to you again, Susan, and it’s nice meeting you, Kitty. I really enjoyed reading through your book. It’s a really nice collection of resources.

Kitty: It’s been a pleasure.

Susan: Well, thank you. Been great to be featured on Tea for Teaching, because it’s a little outside of the normal topics, and so we’re so happy to spread the word.

Kitty: And I’ve been a fan of Tea for Teaching forever, so it’s so exciting to finally make it on the show. [LAUGHTER]

Kristin: Well, I would love to hear about the group coaching and how that goes, [LAUGHTER] but that’s just putting in a listener request. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

454. Designing Impactful College Courses

Student motivation is enhanced when the framework of self-determination theory is used to design courses that support student autonomy. In this episode, Daniel Guberman and Chantal Levesque-Bristol join us to discuss a new resource designed to assist in the development of such courses.

Daniel and Chantal have leadership roles at the Center for Instructional Excellence at Purdue University. Chantal is the Executive Director and Daniel is the Assistant Director for Student-Centered Pedagogy.  They are the authors, with Emily Bonem, of Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy Supportive Learning Environments, which was recently released by Routledge.

Show Notes

  • Bonem, E. M., Guberman, D., & Levesque-Bristol, C. (2026). Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy-Supportive Learning Environments. Taylor & Francis.
  • Kuh, G. D., & Levesque-Bristol, C. (2023). Student-centered pedagogy and course transformation at scale: Facilitating faculty agency to IMPACT institutional change. Routledge.
  • Purdue University’s Instruction Matters: Purdue Academic Course Transformation (IMPACT) program
  • Lang, J. M. (2021). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • The Grading Conference
  • Mentzer, N., Asad, A. M., Koehler, A. A., Mohandas, L., & Mammadova, E. (2025). Engaging Learners Everywhere: The Interactive Synchronous HyFlex Method. Purdue University Press.
  • Student Pedagogy Advocate Program at Purdue
  • Center for Self-Determination Theory

Transcript

John: Student motivation is enhanced when the framework of self-determination theory is used to design courses that support student autonomy. In this episode, we discuss a new resource designed to assist in the development of such courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Daniel Guberman and Chantal Levesque-Bristol. Daniel and Chantal have leadership roles at the Center for Instructional Excellence at Purdue University. Chantal is the Executive Director and Daniel is the Assistant Director for Student-Centered Pedagogy. They are the authors, with Emily Bonem, of Designing Impactful College Courses: Applying Self-Determination Theory to Unleash the Potential of Autonomy Supportive Learning Environments, which was recently released by Routledge. Welcome, Daniel and Chantelle.

Daniel: Thank you. It’s great to be here.

Chantal: Thank you. Thank you so much.

Rebecca: Today’s teas are:… Daniel, are you drinking tea with us today?

Daniel: Yes, I have a Bengal spice tea, which is basically like chai without the caffeine, as I understand it.

Rebecca: That sounds nice. How about you, Chantal?

Chantal: Yeah, I just made some tea. I have some green tea with some notes of apple spices and oranges.

Rebecca: That sounds nice.

John: Very nice.

Rebecca: How about you, John?

John: And the Bengal spice, I think, is the first time we had that on this podcast.

Rebecca: I think yeah, it might be a first time. It’s special. [LAUGHTER]

John: And I have an English breakfast tea today.

Rebecca: Okay, and I think I have a Scottish breakfast that’s mixed at this point. I think with an English afternoon, it’s all mixed together. [LAUGHTER]

John: Well, as we approach late morning, it sounds like a good brunch time. So, we’ve invited you here today to discuss Designing Impactful College Courses. Can you tell us a bit about how this book project got started?

Chantal: Yeah, so really I’m going to go back to a first book that I wrote on the Impact program, was published in 2021 by Stylus, which is now Routledge as well, that was really talking more about the structure of the Impact program and a lot more for faculty developers, and this book is sort of the sequel to that, but really looking a lot more about how you would design specifically a course, and this one is a lot more for faculty themselves or instructors, people that are actually teaching that are really interested in understanding how to use the principles of self-determination theory to design courses that are autonomy supportive and self-determined.

Daniel: And maybe I’ll add a little bit about the Impact program, for those who don’t know, which is Purdue’s course design program, which has been going on for over 15 years now. I’m a few weeks away from my 10-year anniversary at Purdue, and Chantal is a few years longer than that. It predates both of us, and it’s a semester-long course design program based on a faculty learning community, and it’s done in coordination with our teaching center, libraries, and teaching technologies. So, we get groups of faculty, we break into groups of three to five, and they each work with members of those three teams, and it varies every year on what our capacity for support is, but we’ve worked with over 600 faculty across this time, and so we’ve had semesters where over 90% of students at Purdue are taking a course that went through this program, so it’s been a really impactful program, pardon the pun. It’s a good name to just like constantly have puns, although unfortunately, whenever I write the word impact out, I write it in all caps, so that I need to go back and edit all my documents. So it’s been this really long and successful program, and we presented on it, and we recognize that a lot of faculty, instructors, graduate students don’t have access to this type of program, and even instructors at Purdue, for various reasons, can’t commit a whole semester; their schedule doesn’t work. So, we’re really excited to bring a lot of these lessons and ideas to a broader audience.

Rebecca: It’s incredible to have such an impactful program, certainly reach so many people, and to build on that in your book. One of the things that we observed as we were just even skimming the table of contents is that the organization seems to have something in common with a philosophy or a strategy that instructional designers often recommend, which is backwards design. Can you talk a little bit about the organization of the book, what it does have in common with that strategy?

Daniel: So, backwards design was at the core of the book, and the program, that program followed a backwards-design structure even before Chantal arrived and brought self-determination theory, and we’ve kept that. Probably the biggest change is that we brought this theory as the foundation, which in a lot of backwards design work does not have a sort of core theory that is structured to follow throughout everything that you’re doing. So we start with the theory, and then following traditional backwards design, for those who don’t know, we talk about the outcomes and objectives of a course, then the assessment, which we frame around grading specifically, because we found a lot of traditional course design approaches talk about assessment, but don’t tie it to grades, and then at the end they just sort of throw some numbers on all of it, and it feels like that can undermine the connection between the assessment process and what students are experiencing. And then we go to assessment types, which we have chapters on a variety of different assessment types, and also activities and core structures that are basically from a backwards-design perspective, going from what will students know or achieve by the end of the class, how will you know, which is the assessment part, and then how will you get students to the point where they can demonstrate that effectively. So that’s sort of our broad picture overview of the book, and we think it makes a lot of sense for someone who’s designing a course, which isn’t to say we view it as a linear progression, but rather we’ve tried to structure it in a way that you can skip around and you can go back, so you might get to thinking about, “Oh, I really want to bring experiential learning into my class, and that means a group project, which means I need to go back to assessment, “so we don’t think of backwards design as a linear process, but more sort of like a spiral that you move through.

John: And it may nudge people to think about it, because faculty don’t always think about course design using a backwards-design perspective. In fact, it’s often the opposite, but in addition to having this backwards- b design structure through the book, after going over the basic theory, you also have a consistent structure for each of the chapters. Can you describe this structure and talk a little bit about why you chose this particular organization?

Daniel: So each chapter we start with a vignette that comes from a student perspective, because it was really important to us to think about how students are really a central part of the learning environment, and they’re often omitted from the course- design process, and this has been something we’ve been doing throughout Impact for a number of years, is better integrating students into that process. The vignettes are based on true stories from students, but adapted a little, actually, a lot of the students, they came up with overly positive stories all the time [LAUGHTER], we’re like, “Tell us some of the negative ones too.” So then, after the vignette, we have an intro to what the topic is. So some people might not be familiar with what we mean by the experiential learning, or even when we say, like, this is a chapter on lecturing, like, what does it mean to write a chapter on lecturing or class discussion, then we get into the basic needs, and we talk about how the basic needs can be met or hindered in each of these contexts. Then we always follow the same order from competence, autonomy, and relatedness. So I’m sure I’ll talk more about what those mean in a bit. And then after that, we go into case studies. So one of the really fun parts about writing this book, is we got to go back to some of these 600 faculty members who have worked with this program, and we interviewed them and had really meaningful discussions to understand the ways that they’ve actually applied it, because as anyone who’s worked in a course- design program will know, there’s a lot of work that gets done in the design process, and then students come into the picture, and things change, and that’s important. We don’t want someone to design a course, and then students arrive, and they’re like, “Well, you are not viewing the course in the way I do.” So, we want courses to adapt. So, it was really cool to hear about how these courses adapted, and how approaching course design through a self-determination theory lens helped them make those adaptations. After the case studies, we have a series of guiding questions to help someone get started, and we have our short reflections about each of those, inspired by James Lang’s Small Teaching, which I feel like everyone knows by this point. We have a section called Starting Small, which are designed to be small things you can do, because while it’s a course design book, we realize not everyone is going to approach it thinking I’m going to redesign my entire course from the ground up, and again we have some sort of reflections and ideas about what that might look like, and then we just end with some short takeaway bullet points.

Rebecca: You’ve hinted at this already a little bit, but can you talk a bit about who your intended audience is for this book.

Chantal: This one is really for anyone who’s teaching a college course, or anyone who’s even interested in teaching. So we could be thinking about graduate students, for example, that may be asked to teach a part of a course or an entire course with little preparation. This book would really help them understand how it is that they can really approach teaching from a really a deep perspective in really understanding students, which I think is very important, as Dan mentioned, that we start with a vignette, which brings in the student voice first and foremost to really think about them as part of the learning and teaching endeavor. So also it could be for someone who is working at an institution where they don’t have their own faculty development program, or it could be for some faculty developers that do not have a very established structure at their institution. How can they talk about teaching and learning in a different way to their administration for example, and also I would say anyone who’s interested in self-determination theory and the teaching and learning principles of self-determination theory and autonomy-supportive environments, and how would that look like in a course-design environment?

John: So, you mentioned self-determination theory, and we talked about this a little bit on earlier podcasts, but it’s been a while, and many of our listeners may not have caught that, so could you talk a bit about what self-determination theory is? Because it does serve as an organizing framework throughout this book.

Chantal: It does stop me if I start talking about it too long. That’s definitely my passion. I say this is how I’ve been raised in terms of an academic, so my research area is also in motivation, and this is as Dan mentioned. When I came to Purdue University in really the impact program was just starting, it was really something that I brought in terms of thinking about how we design courses. So to summarize really quickly, the core principle of self-determination theory is the existence of three basic psychological needs, right, that all humans have. So, it’s a humanistic theory of motivation, and it’s about competence, autonomy, and relatedness, which is how we structured the book. And they’re really baked into every chapter. And competence is something that we understand very easily in higher education. It’s this idea of developing your skills, mastering, becoming proficient, but often we focus very much on competence to the exclusion of the other two needs, and they’re also very, very important. And when we talk about autonomy, which is probably one that is a little bit harder to understand, autonomy does not mean independence or doing whatever you want to do, right? That’s not what it is. It’s not about that independence, it’s about choice and volition, and really empowerment, and really feel like you are sort of the agent of your own development, and that applies for students and for instructors as well. And then relatedness, which is very important, which is the sense of connection with other human beings, with the instructor, with other students, it’s very relational, but also we think of it as relationship with the content that we’re covering as well, which also is highlighted in many of the chapters, and when we are creating environments where the satisfaction of the needs is met, where those basic needs are satisfied, then we see a progression toward more self-determined types of motivation, so from extrinsic type of motivation to more self-determined or intrinsic type of motivation, and this is really one of the very important contributions of self-determination theory, is to think about extrinsic motivation in four broad categories of extrinsic motivation. So, you may be familiar with extrinsic motivation as doing something for the rewards or to avoid the punishment, which is only one type of extrinsic motivation. So, extrinsic motivations are about means to an end, you’re doing something to achieve an outcome, but there’s different levels, so after external regulation, which is really doing something to attain a reward or to avoid punishment. So the typical understanding of grades would really fit within that type of extrinsic motivation. You’re trying to get an A because that’s how you get the rewards, and you go to class because you want to avoid losing points, that’s really the type of extrinsic motivation that we’re talking about in external regulation. We move towards still something that is not self-determined, but we think about introjection, which would be doing something because you feel guilt or shame, so you’re trying to avoid feeling those sorts of feelings, so it’s more internalized, but it’s still not self-determined, right? It’s an internal pressure, it’s perceived as an internal pressure. And then the next two types of extrinsic motivation are more self-determined. Identification, for us, and we talk a lot about this in the book, is sort of the sweet spot for us in academic endeavor. It’s really when you start to understand the value of your behavior and how it makes sense, how it’s meaningful, how it’s important to you, so that’s really how identification is formulated. And although you’re trying to do a behavior or you’re trying to still achieve an outcome, the outcome is perceived to be important and valuable, and then the last form is integration, which is about an alignment with the self, so it’s something that is so in line with yourself that you’re doing it because it’s part of who you are. So, I think this is important to keep in mind in terms of the different types of extrinsic motivation. I’ll stop here, see if Dan has anything to add.

Daniel: And what I’ll share is my background. Emily and Chantal are both psychologists by training. My background is as a music historian, so very different. But when I came to Purdue and was introduced to this theory, I just found it really compelling. So, I think this is what we’re hoping readers from a variety of disciplines will find, is that they read about the theory, they hear about the theory in places like this, and they find that same connection when you think about, “oh yeah, I can see examples of myself in each of these types of extrinsic motivation, and I can see how in environments where my basic needs are met, I feel more internally motivated to do something.” If you start to think about it, you really start to feel that, like,”oh, this resonates, it works, it’s not this abstract theory that, like, we hope will apply in our class, that we can feel it throughout our everyday lives,” and then it’s a matter of how can we apply this meaningfully in our classes as well.

Rebecca: I think many faculty experience moments in their classes where maybe particular material or concepts are not overly exciting or interesting, but maybe very foundational, or maybe students are in a general education class and it’s hard to get motivated around that. What are some strategies for helping students develop motivation to develop their knowledge and skills in these areas that maybe we deem as important, but maybe initially they don’t.

Daniel: Yeah, I think that’s such a great question that gets at the core of the book, and one thing that I think is really important, and one of the key misunderstandings or like the threshold concepts, if I may, when it comes to self-determination theory is the recognition that when we talk about these basic needs for autonomy, confidence, and relatedness, it’s about the individual’s perception of those needs for themselves. I think a lot of people come and we do an introduction the first week about, like, what are the basic needs, and they’re like, “Oh, that’s great, I’m giving my students choice, so I do unessays, so I give them choice that they can choose what format their project is going to be, and therefore I’m meeting their need for autonomy,” but and I’ve experienced this myself, is that that doesn’t necessarily work for all of my students, that some students will see, like, “oh, I can do anything,” and they’re like, “well, that’s scary, I don’t know, I guess I’ll just write a paper, and it’s not because I really want to write a paper, it’s because that’s the thing I’m used to doing, and I know that’s a safe thing to do.” So that student has not perceived that they’ve had autonomy in this assignment, even though I’ve told them, like, “oh, you can do a poem, you can make a video, you can bake a cake,” but at the same time they haven’t seen paths for them to do that, and I haven’t necessarily provided… at least when I started doing this, I really did not provide guidance to make a meaningful choice, and that’s the key when we think about autonomy, is how do I create a structure where a student is going to see this choice in my class and feel like that choice is meaningful to them, so when I give them the unessay and say you can write a poem, I need to then say, like, well, what does it look like to write a poem for this, and what resources, because I am not someone who could help them write a poem, but what resources are available on campus if this is something they want to explore that can help them do that in a meaningful way, and then I can also provide the feedback that I’m capable of providing, in thinking about how to tie the content of the class with the poem, and really help them then also feel that sense of confidence that all these needs get intertwined all the time, because that’s also then about the relationship I’m able to have with students to help them see that, like, oh, when I give you a choice, I mean it, and I’m also here to help you make a meaningful decision for you. So, I think the key is always to try to think about the students’ perception and experience. When we start doing that, we start to help overcome these motivational boundaries. And I say this, coming from a background in music history, where even when I taught music majors, a lot of my colleagues on the performance side of that school would, like, tell them, “Well, the music history and theory classes aren’t that important, you should spend your time practicing.” So at that time, I felt like it was always the struggle of, like, I need to motivate students for this class in their major, and I didn’t know about self-determination theory at the time, but now, like, oh, I missed all these opportunities to help them really understand why these things were important, and to connect with them in doing so.

Chantal: Yeah, and I want to add that it’s really important to think, like, it’s not a recipe, it’s not a recipe book. You’re not going to find in this book that, like, this is exactly how you need to do it, and here’s step one, two, and three, and that’s why the student voice is very important, that student perspective and the perception, as Dan mentioned, because it takes getting to know your students to really understand what meeting the basic psychological needs is going to mean in your context, and also what you’re able and willing to do as an instructor, so you could implement a technique that would be very effective at meeting the basic psychological needs, but if you do it poorly, you’re going to have the opposite effects. So, if team-based learning is not your thing, and you don’t think you can implement it well, that is not why it works, if that makes sense. It’s really important to understand that it’s not the what that is as important as the how you implement that pedagogical strategy.

John: One of the things you note in this book is that grading is one of the things that faculty tend to really not enjoy particularly much, and it’s also something which can be somewhat demotivating for students. What suggestions do you have on trying to shift the focus on grading, so that it can be a bit more motivating, and perhaps not quite as unpleasant for faculty too.

Chantal: That is so Dan’s question. [LAUGHTER]

Daniel: Yeah, [LAUGHTER] and I’ll say, as we’re recording this, just before the Grading Conference begins this summer, and I’m a co-organizer of the Grading Conference. If you’re familiar with the Grading Conference, you might understand where my perspective on grades is coming from, but with that said, I don’t think that means we need to, like, do away with all grades, which is some answers, but I think the key is getting away from the view of grades as a mechanism to control students’ behavior, because ultimately, what self-determination theory is, in contrast to more traditional theories of motivation, and self-determination theory is like 50 years old now, so I guess it’s also traditional, but theories before that were really about mechanisms of control. I’ve just got a new dog a few weeks ago, so like how you train the dog is like, dog does good thing, I give treats, dog does bad thing, I don’t give treats, [LAUGHTER] and I think a lot of early motivational theories were rooted in that, of like, “oh, student does good thing, I give grade, student does bad thing, they lose points towards grade,” and that still happens all over our campuses. So, I said, my background is in music, so, like, one of the things we do in music departments is we want students to go to our concert, so what do we do? We give them extra credit for going to a concert, and what we don’t do in doing that is explain to them the value of going to that concert and give them opportunities to engage with that concert in ways that they might not otherwise, and that connect that concert to the experience of what they’re learning in our class, and I think that’s what we really would advocate for in self-determination theory, is to really think critically about grades and what grades mean, because ultimately grades, at their best, are a form of information, they’re a communication form between instructors and students, that is unlike basically any other form of communication, and that can be used effectively, [LAUGHTER] or it can be used for harm. And I think too often we view grades as a mechanism to control. So students after COVID, my class attendance has dropped, so I’m going to give more points for students coming to class, or more penalties for not coming to class, and instead we can think, well, what is it that’s happening in class that we can make more valuable for students, and then that doesn’t mean you can’t have grades tied to it, but like, how do we then make the grade tied to the learning that they’re doing in class that’s tied to the learning outcomes and objectives that will help them in their long-term career personal goals, whatever those may be, and trying to really build that connection, so students don’t see their grade as well. This instructor liked me or disliked me, and therefore they gave me this grade at the end, but really that this grade reflects what I’ve been able to achieve, and my growth, and then I can go talk about that, and like what it meant to earn an A or a B in this class, and what I gained from it, and that grade is one piece of information that is sort of a tightly packaged representation of that.

Rebecca: This conversation’s making me think a little bit about self-determination theory related to instructors, too. We have needs, especially around grading, and some of these other things, too. I’m wondering if you have any thoughts about that side of the coin in thinking about some of the basic needs of instructors and how some of the techniques that you’re recommending actually [LAUGHTER] serve in both directions.

Chantal: Yeah, absolutely. I’ll just kind of get started, and then Dan can jump in, but when we go through our Impact program, which Dan described, our 13-session semester long program, we really emphasize that to faculty, and that’s the beauty of self-determination theory. As you mentioned, it’s a theory of human motivation, so it’s about humans, so it’s really the humans that are doing the teaching and learning endeavor, and that’s really the most important thing, and what we really need to focus on. So, we tell faculty, you also need to think about the satisfaction of your basic psychological needs, and that will revitalize your teaching, how you approach your teaching, how you think of your teaching. So, again, if you encounter a technique that you think is the next sexy, flashy thing, but you don’t think it’s going to work for your class, or it’s not going to be something that you can do that you can implement effectively, you’re not going to be in a position to satisfy your basic psychological needs, so that will translate into something that will hinder the needs of your students, because you’ll feel forced, controlled, and not self-determined in applying those techniques. So we really engage in conversations with faculty around that, and to find, through asking questions of the faculty, to really find the thing that’s going to create the best possible autonomy-supportive environment for them and their students.

Daniel: Yeah, and I’ll just add this is what, to me, is really fun about being on the I mpact team and working with faculty every semester in this way because I might be a person who thinks ungrading is the greatest thing for my classes, but I’m not there to convince them of that, I’m there to try to understand what they’re doing and what they’re achieving and what will be meaningful to them, and to me that’s a really fun thing because it makes every semester unique and different, and our hope is in having all these different case studies, readers will be able to see how people have gone through this same program, but come out with very different classes, but all classes that are at their core achieving the same basic goals of creating a learning environment that’s going to support students to internalize their motivation and to succeed and to value the things that they’re learning.

John: We talked a little bit about the structure of the book, but we should note that you have chapters on pretty much all aspects of teaching that should apply to pretty much any faculty member out there. You have chapters associated with creating tests, writing assignments, group projects, lectures, labs, and experiential education, and in each of those cases, you provide some really nice suggestions on creating an autonomy-supportive environment. Could you each highlight one or two ways in which the approaches that you advocate in either one of these chapters, or generally across the chapters, might differ from what we see in many classrooms today?

Chantal: I love how you set up that question, and I want to kind of not push back on it a little bit, but “different” is not how I would really think about it. It’s not like many of the techniques that we have in there, or suggestions are necessarily different. There’s a lot of the things that are the same things, but sometimes it’s just to think about them in a context that is really based into self-determination theory. So I would say not necessarily different in kind, but it’s really about using the self-determination theory lens and the lens of the satisfaction of the basic needs to make meaningful choices and decisions about your class and your content, and maybe how you’re delivering it, so that’s I think is important to understand that you’re not going to read this book and say, “Oh, I’ve never heard about this before, this is amazing.” That’s not really at all what it is, and again, the suggestions are suggestions, we don’t intend them to be recipes. But if I give a couple of examples that are in the case studies that I really find appealing is that Nathan Mentzer, for example, is a faculty who we talk about in our technology chapter, and really he used HyFlex during the pandemic as a way to really survive as we were re-engaging and Purdue came back to in-person instruction very quickly, and he needed to really continue to use his HyFlex model. He actually wrote a book about it, so that’s also another very interesting piece of it. But he says he will never go back. The HyFlex and what he learned about it, and actually he did research on it using self-determination theory as a framework, allowed him to provide choices and options to students and build connection and relationships with the students and amongst the students that he was not able to do before, so he will never go back. He will always do HyFlex and he talks about how he really talked to his students about how important it was to attend in person if they could, but he was not forcing it. He was not taking attendance in person. He was not making the students feel guilty if they could not attend in person. And then the virtual option was something that everybody could connect to, everybody, even in person. Every student could log in with their teammates and be in this in-person and virtual environment all at the same time. It allowed students to have flexibility to have choices of whether they could show up in person that day. If they were sick, they didn’t have to explain why they were joining in virtually, and they didn’t miss anything. They had all of the content, they could continue to be engaged with their group. He really also focused on the autonomy piece by providing a rationale. He would say, I think it’s important and valuable if you can come in in person, but if you can’t, that’s okay. And here is the content. He also talked a lot about whether you should have your cameras on or off, but again, he wouldn’t force the students to behave or do certain things in a certain way, but he would really spend time explaining the rationale. This one is a great example. I’ll let Dan jump in with a few, and we can go back.

Daniel: So, I’ll share one. It’s actually not the story in the book about this person, but by Lindsay Hamm, who is in the book, I think also in the technology chapter about what’s called Charlie, which is her AI writing feedback tool, and she teaches 500 students sociology class, so it’s like functionally impossible without something like this to give feedback to students, which is very important to her. But what I want to share is actually she is someone who is very interested in the development of AI from the very beginning, and she’s become a campus leader on this, not just in her use of this AI-based tool for feedback, but she took her writing assignment, and in the middle of one semester she was confident that AI at the time was not like providing results that had to make her concerned about students using it, and then in the middle of one semester, an update came out, and she was like, “Oh no, [LAUGHTER] it’s time, it crossed the threshold,” and the question then was, “Well, what do I do now?” And this is where I think self-determination theory is so useful, because it really gives you a framework to think, “Well, now I’m in a situation I didn’t expect, and what do I do?” So, what she did was she first met with some students, because this is 500 students in this class, so she didn’t want to just go in front of all of them first. So, she met with some students, because she trusted students who had taken the class already, and she talked to them and said, “This is what’s going on, and I want you to talk through some options with me.” So then she came to the class and had three different options for like what we could do going forward, which some were just like rebalancing other assignments, some were creating something new, but all with the idea that okay, there are different paths forward, I’ve vetted these and now we’re going to give students in the class, different options, and I think students could actually pick which they wanted, so it wasn’t like, okay, 51% voted for this thing, and it stinks for the rest of you. So she was able to make it so that every student could have a meaningful choice, and she could progress the class in a way that she still felt good about meeting the learning outcomes and objectives for the class, and then over the summer, she could do [LAUGHTER] a much bigger, like, “okay, what does this mean now, and how can I adjust and adapt in a more meaningful way, that’s not just like throwing a brand new syllabus at this 500 student class mid semester.” So to me that’s like a really fascinating example of how you can take these principles and respond to the unexpected, which it feels like now every semester we’re always responding to the unexpected in a way that at least I didn’t feel 15 years ago [LAUGHTER] when I started teaching in higher education. So to me that’s one of the real appeals of this kind of framework for thinking about course design.

Rebecca: Stories for responding are always very valuable. [LAUGHTER] One of the things that you’re just hinting at in this instructor’s approach is kind of checking in with multiple students to develop some choices, and kind of at the start of our conversation, you mentioned the importance of bringing students into the conversation of course design, but of course, students have competing interests and values in what motivates them? Can you talk a little bit about kind of balancing the very different needs or perceptions of needs of different students in a class?

Daniel: I’ll start. So, part of what has enabled this is we have a Students as Partners program at Purdue called the Student Pedagogy Advocate, so that has provided a structure with students who are not like heavily trained but are introduced to self-determination theory and are really interested in working with faculty to make meaning of their classes and really centered around achieving the instructor goal, so we have some students who are prepared to do this, which certainly helps when we bring them in, but I think that’s to me part of the exciting part of teaching is when you are teaching the same class sometimes for decades, for some of us, what makes it exciting is that the students are different. And to me that’s at the core of what it means to teach, is to really think about the connection with the students that you can make and the students who are in the room, and how you can be responsive to them, and I think again thinking through self-determination theory gives us a way to integrate students into that thought, because it gives us a way to understand students and a set of goals for what does it mean for a student to feel a sense of competence, and whether that’s that they aced the test, but maybe guessed on half of it, but we might not know that, or they might have struggled on the test, but they felt good going into that, so then what does that mean for our discussion, or they struggled on the test, but they say, “well, I know I didn’t have time to study, so it’ll be fine next time,” but like what does that mean to help us, for us to help them think about what strategies they’ll take. So yeah, I think to me it really helps us think about how we engage with students, both to build a sense of relationship, but also to really understand what will be meaningful to them.

Chantal: Yeah, and I would add to that, that in some of those classes it’s not like instructors all have to do this, but some have used this very effectively, is to really work with students as co-creator of this class and this content, and even how it’s delivered, and sometimes it’s not possible; in certain situations or certain courses it’s not possible. But to the extent that you can think about students as partners, as Dan mentioned, not just the program that we have at Purdue, but also in terms of its philosophy of how can you really engage with students as guides and partners, as opposed to in a hierarchical sort of relationship, that’s really helpful when we’re thinking about our students as coming to this learning endeavor as co-creator of their pathway, and that’s really the essence of what we mean by autonomy and supporting autonomy. It’s not doing whatever we want and letting the students do whatever they want, but it’s providing a structure and then guiding within that structure. One word, one sentence that I really like to use to think about this is choices within a structure, so structure is also very important within that, so it’s not like there’s no boundaries, there’s no structure, there’s the course objectives, but the alignment, as Dan talked about, and we talk about in the book, the alignment between your learning outcomes, your objectives for the course, and how you assess the kinds of activities that you engage your students in, it’s very important to think about.

Daniel: And one other thing that I’ll add, we’ve been collecting student data on this program through a self-determination theory lens for over a decade, and one of the great things about self-determination theory being a theory that’s been put into practice around the world for 50 plus years is there’s a lot of great existing survey tools, and we can probably, in the show notes, have a link to these, but you go to the Self-Determination Theory Center website, and you click on metrics, you can download all sorts of them. One we really like is called the Learning Climate Questionnaire, and there’s a short version that’s, I think, eight questions. Chantal can correct me, but it’s eight questions. [LAUGHTER]

Chantal: That one is actually published in the first book, so the student-centered impact book that I talked about earlier, and we have some of those scales published in there.

Daniel: To me having eight questions that I can give to my students and get a clear sense of how are they perceiving the learning climate, and we tried to get these into the end-of-semester surveys with some luck, [LAUGHTER] but not as much luck as we’d want, and also you can ask these earlier, you can ask these in the middle of the semester and get really valuable information to understand how students’ basic needs are being met or not being met, and then you can take action from there to address those gaps. So I think that’s one of the other benefits we don’t talk about enough is that having these set surveys, because making class surveys is really, at least for those of us who are not psychologists by training, really hard, and also then we make them, and we’re like, well, did students interpret this question how I thought, [LAUGHTER] and the general answer is probably not, so having these set surveys that have been tested in hundreds, if not 1000s, of classes can be really valuable to collect some quick feedback and understand, like, okay, is this achieving what I wanted when I designed the class in this way.

John: We’ll include a link to the survey site in the show notes. Now, we wouldn’t be having a podcast in the last three years without some mention of [LAUGHTER] artificial intelligence, we keep trying, but… [LAUGHTER]

Rebecca: I keep trying to run away but it’s not helping. [LAUGHTER]

John: However, that is certainly an issue that does relate to educational technology, and you do address this in your book a bit. Can you talk a little bit about ways more generally in which educational technology can support student learning and autonomy, and also perhaps some ways in which we can redirect efforts, so that we don’t see students engaging in excessive cognitive offloading that can be harmful to student learning and skill development.

Chantal: I’ll start with a general comment, and then I’ll let Dan answer the tough part of that question [LAUGHTER] …so I’ll run and avoid it. To me, the general message that I really want the listeners to think about is that technology, including AI and others that will come in the future, are simply tools, and I don’t mean this in a sort of dismissive way, but tools are there to help us create environments that will foster learning, just the same way that our computer and typewriters and smartphones were tools, and some when those came out, were really afraid of those particular tools, but the challenge is to really think about how to integrate them effectively into your teaching, into the learning of the students, and what does it mean to support the basic needs with those tools? So, again, they’re tools. Your voice is a tool, like we’re doing a podcast now. So, your voice is a tool. How do you use those tools to support the learning is important. So, I’ll just put AI in that same category. We just need to think a lot about it. So, now Dan is going to answer the hard part of the question. [LAUGHTER]

Daniel: So I guess in thinking about, like, how do we prevent students from using this as not just a tool, basically to avoid the class or the learning process, because ultimately meaningful learning is hard. That if we haven’t challenged ourselves and haven’t struggled, we probably haven’t learned that much, and our society, our human nature is we try to make things easier. So it’s human nature to look for ways to skip hard things and make them easier, but what that means is we really need to think about motivation, and we need to think about how we internalize motivation, and this goes back to when Chantal described the different types of extrinsic motivation, is we need to really think about how can we help students identify the value of what they’re learning and the skills they’re developing, or ideally even integrated, and feel like it’s a sense of who they are. And often this means really focusing on the basic skills that get before the stuff in their major, especially like the stuff that they perceive as fun. So when I taught the intro music theory class, not many students were very excited to do like the basic lessons and learning how to read music in multiple clefs that they didn’t necessarily deem apply to them, but you need to have those skills really solid before you can do much else, so how do we help students either internalize the importance of developing those skills at a really high level, or to integrate it and feel like, well, I’m a musician, a musician, this is who we are, and we need to have these skills. And this is where the idea that when we create environments that meet those basic needs, students are more likely to have that sense of internalization, and we’ve talked about lots of ways to do that. But really I think that at the core for us is to think about that relationship element, because ultimately students need to trust us when we say this skill is important, and we also need to believe it, and I think this is something we sometimes forget, because most of us now use AI for some of these things, and there’s a difference. I would say using AI for something that I can do and it will save me time, but then I also know if the output is good or not, because I’ve done this enough times, and doing something where, like, I really don’t know if this is good, it will save me time, but it’s not something I could do otherwise and I think those are two very different places, and a lot of students are in the “it’s not something they could do otherwise,” and we really need to get them to the point of recognizing the value of those basic skills and not skipping them, because when they skip them now, they’ll run into challenges later, and we can say that all the time, but unless we really develop a sense of trust and meaning, and have students really buy into the core elements of the class, they’re not going to believe us, when we say it. So that’s where we need to create this environment that’s internally motivating, and an environment where students trust that we have their best interests at heart, and I think most of us do, but I think also there’s, at least when I started teaching, I felt like I wasn’t supposed to say that, I wasn’t supposed to tell students I cared, and I think we’ve turned a corner on that. I think now it’s more acceptable for students to believe we care and us to articulate that we care, and I think now it’s more a matter of like really creating an environment where students internalize that we care, we want them to succeed, and what we’re sharing with them are strategies that we know will help them succeed in the long run.

Chantal: I’d just like to add something, because Dan, you made me think about this, is that with regard to the offloading, yes, humans will tend to find things that they can do that tend to be easier, but humans are not lazy by design. If you really think about self-determination theory, and you really go to the origin of the theory, it’s really about humans are really wanting to grow and develop and become self-determined… self-actualized is another word that’s been used in psychology. It’s like one place in the right environment where the needs are supported, humans will do a lot, and if you think about humans that are alive and vital, they’re doing things, they’re just finding ways to write, to read, to consume, like students read and consume a lot of information, maybe not the textbook, but that’s a question that we can think about, like why not, why is it, but humans are not inactive or lazy, like we really engage in things that we care about, that we value, and that we want to be intentional about. I think that’s kind of the way to think about this in a broader perspective.

Rebecca: So, Chantal, you mentioned the last question is a little challenging, but some folks find our last question the most challenging, which is [LAUGHTER] we always end by asking what’s next.

Chantal: I’ll answer with something that’s kind of on my mind right now, and in things that we’re really thinking about at Purdue is, and you mentioned technology, so it has to do with AI, but really thinking about how critical thinking gets developed and is fostered, including how we think of AI using critical thinking, and this is a program that actually Dan is part of. We’ve been sort of pioneering this, piloting this for the past couple semesters, but in our conversations with faculty, we’re really thinking about how do we develop this sense of problem solving and critical thinking, because when you talk to faculty, right, it’s often something that they’ll say, “Oh yeah, I want my students to become better critical thinkers, but it’s just going to happen, it’s not something that I’m going to build into my class,” and we disagree with that, we say no, we really need to build in how you define critical thinking, how you think about it, how it evolves throughout the curriculum, and how is it that students progress from an emerging skill in critical thinking to really proficiency, and what does that mean, and what does that even look like. So that to me is what I’m rethinking about right now, and what is sort of our next steps in terms of applications.

Daniel: And I’ll just add, because I’m also involved, I’ve really loved this program, that one thing that’s been interesting is we’ve approached this through the same Impact framework, so it’s still based on self-determination theory, but the idea that you need to create a self-determined environment to get students to engage critically with material, but we’ve changed it, as we mentioned earlier. Traditionally, the program we’ve explicitly brought together faculty from different departments, and that was one of the core aspects, was this is a time to go outside your discipline and to realize that teaching has significant differences, but also commonalities, and for this program we started bringing together groups from the same department, and this has been like really interesting in different ways, and part of that is the recognition that a lot of faculty don’t actually talk about their teaching with their colleagues in their department, so this has been one of those opportunities for them to get together and think about what does critical thinking look like in our context, and how can we actually connect our classes in meaningful ways that help a student, whether it’s the 100 level class, start to develop certain skills that will be applicable in that context, and then how can those be reinforced, and how can we use similar language and structures, or one department we’ve been working with that I’m really excited they built a whole curricular long rubric with like here are key critical thinking skills and here are where we think students will develop and actually the department is Speech Language and Hearing Sciences, so the vast majority of students go on to get a master’s degree because you need that to be a practicing professional. So they actually said, well, at the top level we don’t even expect you to be here until you’re in graduate school, and the idea is to share this with students from the beginning, and obviously figuring out how you share it with students at your 100- level class and say we expect you to end this class at level one, maybe two, right, [LAUGHTER] but then show that that progression, and then when they’re applying to graduate school, give them the structure and language to talk about how they’ve developed and what they still plan to learn, because ultimately you go to grad school to continue your learning and development to become a professional in this field, so that to me is also really exciting, and like a twist on this program, but our early success shows how this basis in self-determination theory can really apply in lots of different contexts.

Rebecca: Sounds like a really great initiative. Well, thanks so much for joining us.

Chantal: Thank you for having us. This was a lot of fun.

John: Thank you. It’s a really good book, and it’s one I’d recommend to anyone who is beginning to develop or redesign an existing course.

Daniel: Yeah, thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

453. Prequestions and AI

A growing body of research indicates that asking questions about concepts students have not yet studied increases learning. In this episode, Steven Pan and Judith Schweppe join us to discuss how generative AI can facilitate this technique by generating prequestions.

Steven is an Assistant Professor of Psychology and the Director of the Learning Sciences Lab at the National University of Singapore whose research focuses on evidence-based teaching approaches. Judith is a Professor of Educational Science with a focus on teaching with digital media at the University of Erfurt. Her research focuses on working memory, long-term memory, and learning.

Show Notes

Transcript

John: A growing body of research indicates that asking questions about concepts students have not yet studied increases learning. In this episode, we discuss how generative AI can facilitate this technique by generating prequestions.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Steven Pan and Judith Schweppe. Steven is an Assistant Professor of Psychology and the Director of the Learning Sciences Lab at the National University of Singapore whose research focuses on evidence-based teaching approaches. Judith is a Professor of Educational Science with a focus on teaching with digital media at the University of Erfurt. Her research focuses on working memory, long-term memory, and learning. Welcome, Judith, and welcome back, Steven.

Steven: Thank you.

Judith: Thank you, thank you for having us,

John: And thank you for agreeing to come on at times that are a little bit challenging. Could you mention for our listeners what time it is where you are?

Steven: I’ll go first. It’s a bit after 4:30 in the morning [LAUGHTER] here in Singapore.

Rebecca: Starting your day right.

Judith: And for me it’s 10:30 pm, so I’m about to go to bed after this recording.

Rebecca: Best way to end the day. [LAUGHTER]

John: Today’s teas are:… Judith, are you drinking any tea?

Judith: Yeah, in fact, I’ve prepared a ginger and lemon tea.

Rebecca: A good one for the evening.

Judith: Yeah, exactly, that was the idea.

Steven: Well, I sometimes drink tea. Actually, I should drink more right now. I just have water.

John: And Rebecca,

Rebecca: I have Golden Monkey today.

John: That’s been a while.

Rebecca: Yeah, it’s been a while.

John: And I have an Irish breakfast tea today.

Rebecca: Awesome. So we invited you both here today to discuss an article in the Journal of Applied Research in Memory and Cognition that you both co-authored with three additional co-authors, entitled “Using ChatGPT Generated Pre-Questions to Improve Memory and Text Comprehension.” But before we discuss your study, it might be helpful to talk a little bit about what is meant by pre-questioning, and what past research has suggested about the impact.

Steven: Well, pre-questioning is a learning technique that involves attempting to guess the answers to questions about information you haven’t formally studied or know much about, and then there is, after you’ve attempted to answer those questions, the opportunity to learn the correct answers from some form of instruction, such as a lecture, watching a video, or, for example, reading a text passage. So you guess first, then you discover the answers afterwards. That’s pre-questioning. It’s similar to, it could you even be considered a part of a class of strategies that sort of involve what you might call pre-instruction learning activities: do something in advance of formal instruction, and pre-questioning, along with those other strategies or techniques, can improve learning in a variety of circumstances, improving, as I mentioned, learning from videos, learning from lectures, learning from text. There are a growing number of studies showing that it has these kinds of benefits.

Judith: What’s remarkable about this is that learners typically don’t know the answers to these questions, because they only learn afterwards, so they guess wrongly or can’t come up with an answer, they don’t get any feedback, but still they benefit on the criteria test questions after learning.

John: What are some of the possible explanations for why pre-questioning is so effective in enhancing student learning?

Steven: So this is an active area of research, and there are many possible explanations. Just to name some of them, for example, one explanation is that pre-questioning sort of activates attentional processes that either make you aware of your gaps in knowledge and want to focus on finding the answer, or when the answer is presented, you notice it better, or simply when the instructional opportunity happens afterwards, you are more focused, and you are more primed to learn from it. So that’s a sort of an attentional class of explanations. Then there is, for instance, an error signal type of explanation, where you make a guess, and usually it is incorrect because of the lack of prior knowledge or instruction, and your awareness of that error, particularly when the correct answer is presented, there’s a discrepancy. You guessed this, and then the answer is something else. And there are potential mechanisms in the brain that are responsible for learning from those kinds of situations, and the pre-questioning triggers those mechanisms. So that’s a second possibility. There are other ones that involve, for example, your memory of making the error. Okay, I guessed this, and now this is the correct answer, and you sort of integrate this, and that becomes a more powerful learning event, because it’s not just you learn something, it’s you made a guess first, it was usually wrong, and then you have the correct answer, and that becomes a richer, more integrated representation that helps you better remember that information. So, those are three candidate explanations for how pre-questioning can improve learning. There are more beyond that, but those are some of the ones that come to mind.

John: One of the other ones that was mentioned in your paper was that it also activates prior knowledge to help prepare you to integrate new knowledge into that framework.

Steven: Yes, that’s also a potentially useful way to consider how pre-questioning may enhance learning, and this idea that it may activate that prior knowledge, so you are then better able to integrate the new information, you may be primed to learn that information, and so on.

Judith: And that explanation might even be more important in real life in class, because even when you ask students questions for a new topic, they will have some prior knowledge that they can activate, whereas in the lab we usually use texts that they know hardly anything of, so that the importance of prior knowledge is even stronger, maybe in class.

John: And however it works, it does seem to work. [LAUGHTER] So it’s an effective technique, whatever the specific mechanism.

Rebecca: And despite the evidence on the value of pre-questioning, you know that the practice is rarely used in the classroom. So why might this teaching strategy be so rarely used?

Judith: I think there can be several reasons. For one, the research field is still very young, so there was not that much time for the evidence to distribute to the public, as compared to retrieval practice, for instance, which has been far better known in the past years. And another reason may be that it may also feel weird for a teacher, for an instructor, to ask their students questions that they know they won’t be able to answer, and that might refrain some people from using pre-questioning.

Steven: Yeah, those are, I think, very good reasons for why pre-questioning may not be used that widely. I’ll add another one that’s relevant to the paper that we’re going to talk about, which is this idea that you need the materials to engage in pre-questioning, and particularly for students who know nothing or very little about the topic, which is sort of definitional requirement of pre-questioning, it would be very difficult for them to come up with questions on that topic to practice with, and also even getting them from instructors, instructors commonly, for example, in surveys, they’ll say that they’ll provide practice questions on occasion, but typically after instruction has occurred, rather than beforehand. So all of this points as sort of a materials or implementation challenge. Not only do learners and instructors not necessarily know much about pre-questioning, and it also does feel weird to use it, but even if they wanted to use it, they might not have the materials, either the time to come up with the questions or access to questions that are appropriate for pre-questioning. So there are a host of obstacles that make it difficult, at least in many circumstances, to use this particular technique.

Judith: That was the main starting point for our study, because it is not only recommended as a teaching approach, but also as a learning strategy. But for learners to use it on their own, it’s a lot harder than retrieval practice, because, as Steven said, you cannot come up with new questions before you know about the topic, you’re totally reliant on somebody else providing these questions.

Steven: Yeah, exactly. And with retrieval practice, sometimes it can be as simple as: “take out a sheet of paper and write down as much as you can remember about whatever you’re trying to learn.” You just retrieve it. You don’t even necessarily need to have a database of questions to practice, but with pre-questioning you really do need those questions, and if you don’t have those easily available, then it’s very difficult to engage in the strategy or technique.

John: Which brings us to your paper. What are the primary questions you’re investigating in the study?

Steven: Well, there are sort of a sequence, we sort of went from very simple and basic, and we got more and more specific. The first question was simply, if you use generative AI to come up with these questions, and you use them for pre-questioning, do they actually work? That was sort of the most simple question, and that was the first experiment focus. We then advanced to, okay, how do those questions generated by AI compare to questions that a relatively expert human being creates? Are they similar or different in efficacy if you use both human-generated and AI-generated questions for pre-questioning? And that was the second experiment. And then, for the remaining two experiments, we went into more fine-grained issues, specifically comparing AI-generated questions made using what we called a sophisticated or elaborate prompt that was very carefully engineered to generate the questions we wanted to see, versus a simplified or more basic prompt that any person could use without much training, so different prompts to generate these questions, and we compared that to a condition that didn’t use pre-questioning at all, but did complete an alternative activity. In this case, it was studying an AI-generated outline, so that was sort of a more rigorous test of the effects of pre-questioning to see whether using pre-questioning compares favorably to spending the same amount of time studying something else.

Judith: It’s an additional question concerned whether answering pre-questions generated by either an AI or human would generate to a different source, because if you think of a regular learning situation, you might generate as a learner your question with an AI. Well, meanwhile, the instructor might also generate the exam questions with an AI, but they might also be generated by the teachers. So, it might be worth knowing whether there is any decline in the transfer from one source to the other, and the same with the more elaborate prompt, compared to a basic prompt. If maybe a student would use a more basic prompt than a teacher.

Steven: Right, and that issue speaks to this idea of, shall we say, transfer or generalizability. So you ask someone in pre-questioning a particular question, and then you assess learning using that same exact question, that’s one way to measure learning, but as Judith just said, it could be different questions that the instructor generates, either by themselves or using an AI, or using different prompts, or whatever, and so a critical question really is, “What happens when the assessment, the questions used on that assessment differ in some way from the questions used earlier for pre-questions?”

Rebecca: So we’ve talked a little bit about the research questions. Can you talk about the design of the study itself?

Steven: So each experiment, these were online studies with online participants, adult learners, and in each experiment the learners would be assigned to either a pre-questioning condition of some type, or in some experiments a control condition where they didn’t have pre-questioning, or they studied an outline, or some other activity happened. So that happens at the beginning, they’re randomly assigned to these different conditions, and then they would engage in some learning activity, for example, pre-questioning, attempt at least two pre-questions one at a time. All participants would then, and to be clear, all of these experiments involve learning from an encyclopedic text, in this case, I think it was about brakes, the function of brakes, different types of brakes, so it’s sort of a great proxy for certain kinds of information that you would find, for example, in an engineering course, something like that. And so they engage in the initial learning activity, such as pre-questioning. Then everyone reads this passage on brakes, which contains the answers to the pre-questions, and then after that, after a short delay, then they take an assessment of learning, which is a series of questions about brakes, which include questions the same as used for pre-questioning, but also other questions that were not necessarily exactly the same. That’s the broad design. Of course, across experiments, there are variations, so to get more specific, experiment one, the one condition was pre-questioning, where they attempted AI-generated pre-questions before reading the passage and taking a test. The other condition had no pre-questioning, so they just read the passage and took the test. In experiment two, it was AI-generated pre-questioning versus human generated pre-questioning. So one group got the AI-generated questions, one group got the human generated questions. They all read the passage, then they took the test. And then experiments three and four were either what we call elaborate generated pre-questioning, the prompt that was used, or basic generated pre-questioning, again referring to the prompt that was used, or studying an AI-generated outline. So, depending on the group, they would have different activities before reading the passage, and then taking the test.

Judith: And in any of the experiments in which we contrasted different questions, both types of questions were also used on the criteria test, so that we could see differences between the tested questions and the untested questions, and in some cases even between new questions generated by the same source, either both AI-generated or both human-generated, and transferred to new questions generated by the other source.

John: A very well designed experiment, and one of the things that I appreciated about this was that it was an experiment that was pre-registered. That’s not something that happens in all cases, in all disciplines. Could you tell us a little bit about this process? It’s something I think that came up because of people attempting to replicate some past studies and not always finding that results were replicable. So, could you just give our listeners a little bit of background on pre-registration of studies?

Judith: We try to do it in a brief manner. You’re right, this originated in what we call the replication crisis in psychology and social sciences overall, in which several well-published studies did not replicate, and so there’s a counter movement right now, the open science movement, and the idea is to restore trust in science and psychological research. We want to be as transparent, as open, about what we are doing as possible, and there are several reasons for why some research findings are not that reliable. There’s a tendency for significant findings to be published more often than non-significant findings, so that we are rewarded for getting significant results, and even more for getting our predictions right. There’s a study that indicates that more than 90% of hypotheses in psychological studies turn out to be right, and that’s not because we’re so clever, that might be due to some questionable research practices. So pre-registration is one way to make sure that we write down in advance what we predict, how we conduct our study, how we analyze our data, how we decide whether a certain data point is an outlier, how we decide whether to include any covariate. So to write that down in advance, so that we’re not subject to our own biases, so that may be the one analysis that brings about the best result that we’re convinced that that was the one we wanted to do right from the start. No, we can look that up because we’ve written that down, and to make it transparent to everyone, you pre-register publicly, so you are the one who decides whether others can see it, but when you submit a paper, you can include this pre-registration, and the reviewers can see that the date of the pre-registration is really before you collected your data.

John: You mentioned that so many studies, and this is true, it’s not just in psychology, it’s also true in pretty much all social science disciplines. And two of the explanations for that are publication bias and something called p-hacking. Could you just explain a little bit of those two things for our listeners?

Judith: Yeah, the publication bias, so that indicates, as I just said briefly, that of the data that are published, most of them are significant, and that’s not realistic, because a lot of studies we conduct, they yield non-significant results, and the idea is that this is due to several biases in the publication processes. So reviewers and peer-reviewed journals are more enthusiastic for significant findings, researchers don’t have that much time to write up their findings, so they tend to focus on those that are more publishable and leave the others in their drawers. So this is also called the file drawer problem, that there are a lot of non-significant findings in the file drawers of lots of researchers in all the ivory towers, and this leads to us overestimating the size of published effects, because we don’t know about this heap of unpublished studies, and that’s a problem for evaluating how important certain effects are for us. And p-hacking is one way where one type of questionable research practice, because p is the value that indicates statistical significance, and the field has agreed that p that is lower than .05 is the threshold for importance, and this statistical significance is often mistaken for importance. And there are several ways to analyze your data that can influence whether your findings yield a p value below this threshold, and all these ways to treat your data are called p-hacking. So, you can decide to include gender as a variable that was never part of your hypothesis, but you can always see whether it makes a difference what gender the participants are, and that might bind some variance, so that your main effect is significant. You can first analyze your data, and the p-value is just above .05 and you think, well, just a few more participants, and they might be below the threshold, or you might notice that there’s this weird participant, which must be an outlier, and that often happens after your first analysis, and helps you get below this very important threshold. This often is not outright fraud, it’s just us being human beings and being subject to our biases, and that’s why the open science movement thinks research science needs the self-control mechanisms to be open with our data, with our materials, and with what we predict. And we are both committed to that, so it was important for us to pre-register the study and also to publish the data and the material.

John: Another way of expressing the issue of p-hacking is that if you torture the data enough, it eventually will confess. [LAUGHTER]

Judith: That’s nice.

Rebecca: So gruesome. [LAUGHTER]

Judith: Something else that’s related to this is also HARKing, that’s hypothesizing after the results are known, so that you don’t actually make predictions, but rather post-dictions, and then there’s this analogy to the Texas sharpshooter who draws the target after his shots, so that that is easy to hit bull’s eye, and that’s also often used to compare this research.

John: But your results are not subject to any of those issues, because you’ve done all these things to prevent that with pre-registering and sharing the data.

Rebecca: Seems like a good transition to results, huh? [LAUGHTER] Can you talk a little bit about the impact of providing the AI-generated pre-questions? I imagine that folks are really interested in this particular piece because it might save time.

Steven: So we were very excited to see that the AI-generated pre-questions were effective at enhancing learning, or producing meaningful learning, shall we say, across all the experiments. So, for example, in experiment one, just presenting two, I think, AI-generated pre-questions before reading a text passage, that was enough to enhance learning from that passage, and that enhancement extended not only to later questions that were exactly the same as the questions presented in the beginning, but also new, never before seen, questions, so there was a generalizable benefit to some extent. In the second experiment, we found that the AI-generated pre-questions yielded similar levels of learning as questions generated by competent human beings, so that suggests that you don’t necessarily need a human being to invest the time and effort to come up with the questions. An AI can potentially, an AI chatbot can come up with similar questions of comparable efficacy, and then in the third and fourth experiments, again, they were similar designs, those two experiments, both AI-generated pre-questions that were generated using a more sophisticated, detailed prompt, as well as those generated using a very simple prompt, they both enhanced learning relative to studying an AI-generated outline, so that shows the power of pre-questioning extends to not only using AI to generate the pre-questions, but even compared to using AI to generate some alternative materials that are used for some alternative activity. So those are the broad outline of the key results. Now there’s, of course, that other finding about, or other issue about what happens when the questions on the test are not the same as those that you were practiced on, or they were generated by a different source, and whether, for example, you practice with AI-generated questions, but then some of the questions on the test are generated by a human being or generated by a different AI prompt, or things like that, and I think one general way to describe those results, because they varied across experiments, was that we could still see across experiments some evidence that learning still benefits in those cases, so that it’s not essential that the questions that are used on assessment have to be exactly the same as the ones that were seen earlier. There is some degree of generalizability, although there was sort of a trend, I would say that as the questions get more and more different in source for the content targeted, the learning benefits may not be necessarily as strong, at least numerically, but there’s still evidence that the learning is not restricted to the exact same questions that you saw earlier.

Judith: And we were very happy about that, because that’s not always the case in pre-questioning studies, because often these effects are restricted to post-test questions that target the very same information, so that the transfer to these untested questions really excited us about these findings, and also the case that this very, very basic prompt, which just said, “Can you generate two questions for this passage?” worked, that really sounded feasible.

Steven: One thing I would add to that, we were really very excited by that finding, because literally that means that anyone practically could just use a prompt that is one line and come up with questions that are useful for pre-questioning or potentially other purposes. One thing I did want to note, though, is that when we were conducting the sequence of experiments that we were using ChatGPT, the version of ChatGPT was getting more and more advanced as the experiments progressed. So, by the time we got to the latter experiments and we were comparing different types of prompts, we were using a more advanced GPT version, and in that case, the questions made using, again a very elaborate prompt versus a simple prompt, they seem to be equally effective for the most part, which is again I think in part a reflection of how advanced the technology has become, and also how just generally how easy it is now to use.

John: Given that you’ve got these results, and it doesn’t require a very elaborate, detailed prompt, or even much knowledge of the subject, might you suggest that this would be a useful way of students learning things more deeply as part of their studying? Because students often use it to generate an outline, which was one of your control conditions there, and that was not as effective as having these pre-questions. Might this study suggest that this would be an effective study technique for students to use on their own when they’re learning new things?

Judith: I guess we would say yes. That was the idea, yeah. That was the hope to find evidence that it can work if they generate the questions on their own. We might also add a cautionary note to that, because we found transfer to untested questions, but that may not necessarily be the case for other questions. We noticed that there was some overlap between the questions generated, because all of them were constructed in a way that they were integrative, so they targeted more than one piece of information, and that resulted in somewhat overlapping questions, so the suggestion is a bit different compared to other studies, in which the new questions are totally different, and it might still be more helpful to get the pre-questions from your instructor, who will also post the questions for the exam, because they also tell you what this person thinks is most important, and that may be a valuable addition, but if your instructor doesn’t want to give you any pre-questions, because they don’t believe it’s effective, or because they don’t want to waste their good exam questions on students preparing for learning, that might still be good alternative to ask a generative AI to be a question generator.

Rebecca: I’m imagining that faculty hearing this are excited about the possibility of integrating pre-questions into their teaching methodologies. Do you have some advice about how they might do this, how they generate effective questions, or even like how many questions would be useful?

Steven: So, there are some tentative recommendations that we could offer, but again, these are tentative because we have barely scratched the surface of implementing pre-questions. We are still sort of exploring the efficacy and mechanism stage. Nonetheless, in terms of possible ways that the pre-questions could be generated in an effective manner, well, firstly, one thing I would note was you mentioned number of questions, so it seems at least in the studies that have been conducted thus far, the number of questions is not that many. For example, I don’t think we went more than four or so pre-questions, depending on experiments, and sometimes there’s just two. Why is that? Well, in part, because if you ask people many, many, many questions on something they know very little about, and then they have to try to find the answers afterwards, trying to keep in mind, say, I don’t know, 10, 20 questions is difficult, and so a limited number of targeted pre-questions on perhaps the topics that are most crucial is probably a good way to go in terms of number. Now, in terms of type of questions, you just mentioned the questions that we used were integrative questions, and one of the reasons why, besides what was mentioned, is that there’s some prior work on different types of pre-questions. Some pre-questions that have been investigated are so-called detail-oriented, that might just target a little fact or a number or a little detail, and then there are these integrative questions that might connect two concepts or different parts of a text passage, what have you. Those seem to be more effective at improving learning. So, in addition to the number of pre-questions, the type of pre-question probably I would recommend to instructors to think more about integrative pre-questions. Even if you want a student to learn one particular small detail, maybe couching it within an integrated pre-question might actually be better. And then there is a third sort of practical point that I just wanted to raise, which is that definitely keeping in mind that pre-questions, in a way, serve as almost like a introduction to the topic, so these are not the kind of questions used necessarily to really assess mastery in a retrieval practice sense: “Explain this concept or this theory that you’ve just learned.” If you ask such a question to someone who has not learned that concept, that theory, that question may be so difficult they don’t even know how to begin to answer such a question. Then it’s probably not the best type of question… more like a question that is in some sort of currently perhaps nebulous sweet spot of they can come up with some sort of guess, it’s probably not correct, but it’s also not so impossibly foreign that they look at this question, and think, “Well, I don’t know where to begin.” So, those are at least three tentative recommendations that I have about how pre-questions could be developed by instructors for effective use.

Judith: Something else I would like to add is that you can also first think about what do you want your students to learn the most. So, what are the most important aspects? Knowing that these pre-questions work best for what they ask for, you could make use of that and target the most important aspects with the pre-question. That’s also something related to our study, to think of the cognitive levels associated with your learning objectives, and to see that the questions are generated in a similar way, so that also guides the learners. We also looked at this in our pre-questions, and noticed that for the first experiments, the AI-generated questions were mostly on the lower cognitive levels of Bloom’s taxonomy, so mostly remember and understand, whereas the human-generated questions we used that were used in an earlier study, use comparisons, and that was not the case with our AI questions. So that was the focus of the fourth experiment, where we engineered a prompt that asked for comparison questions, and then the AI generated these questions, but only then. So that might also be something to take from this study that the Gen AI might need an additional prompt to generate this kind of question, but that might also be the case for a human being that thinks of questions that maybe we also need an additional prompt to not just ask remember and understand questions, but also target these higher levels that we name in our learning objectives, and to kind of reverse engineer our questions from what we want learners to take away from the learning material.

Steven: Yeah, and if I may add that Judith raises a very important point, which actually qualifies the finding about basic versus more elaborate prompts, even if you use a basic prompt, you know, generate questions, you can still add in there, generate questions that involve comparison. You can add some details that’s not necessarily required multiple premises in a very complex prompt, but some of those things actually might be necessary through even a process of a little trial and error to come up ultimately with questions that meet certain characteristics that are desirable for learning, including higher order type questions, questions that tap different cognitive levels, that promote the types of learning objectives you want to promote.

John: One of the things you were doing in the study is comparing AI questions with human-generated questions. Was there any apparent differences between the types of questions that humans generated and the types of questions that AI generated?

Steven: We did have sort of as an additional measure in the study… we didn’t tell them these were AI-generated questions, we asked them to guess who made these questions, and we would just give them all the questions that we had in a particular set and some were human-generated, some were AI generated. We just asked them, and they just could not tell, and so this is, I think, part of a larger challenge now, which is differentiating between AI-generated and non-AI-generated materials. Now, not only are the questions as effective as those generated by human beings, but also human beings cannot tell that they were AI generated, but I thought that was kind of a fun little thing, and it speaks to this larger challenge that we have, whether it’s trying to detect AI use in our students’ essays, other things like that.

Rebecca: We always wrap up by asking, what’s next?

Steven: Well, in my research group, and also in collaboration with Judith, we’re really interested in exploring a variety of aspects of pre-questioning, in terms of, for example, ways to implement it in authentic educational circumstances more successfully. So, for example, studies in the classroom, or studies exploring how you might integrate pre-questions into learning activities that are very common, like reading a textbook chapter, things like that. So we’ve explored AI-generated pre-questions, we’re very interested in implementation that could be easily translated to real-world practice, so that is one area of very active interest related to pre-questioning.

Judith: And then a related question in our labs… Well, I thought of it because it’s somewhat related to the current idea, because I’m currently working on a study with a PhD student where we want to investigate whether spacing, distributed practice, which is recommended both as a teaching strategy, but also as a learning strategy, but it’s investigated almost always so that the entire schedule, sticking to the schedule, that’s decided by the experimenter, and our idea was we really don’t know whether that works as a learning strategy, so we implemented it in a way that gives some responsibility for the learning, for sticking to the schedule, to the learner, and see whether it’s still an effective strategy, and also to see whether something like gamification might help learners become better spacers.

John: You’re doing some great work here, and actually that reminds me a little bit of the last interview with Steven and his co-author on another study that looked at classroom implementations of interleaved practice, because a lot of the interleave practice studies had been done in the lab and it was nice to see what happens when it was actually introduced in the classroom environment. You’re both doing some really good research, and we’re curious to see what you’re going to find in those new studies. Thank you.

Judith: Thank you.

Steven: Thank you.

Rebecca: Yeah, thank you so much for joining us.

Judith: Yeah, thanks for the great conversation.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

452. Revisiting A Curated AI Framework

Shortly before the start of the spring semester, Tim Curry joined us for an episode discussing his ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology.

Show Notes

Transcript

John: Shortly before the start of the spring semester, we had an episode discussing an ambitious plan to create structured learning experiences for students using chatbots. In this episode, we check back to see how this experiment developed.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Tim Curry. Tim is a lecturer in the Department of Health Sciences at Northern Arizona University. He is developing AI applications for autism research and tribal health service while building Northern Arizona University’s Applied Health Analytics program. He’s also working on the completion of his PhD degree in health equity technology. Welcome back, Tim.

Tim: Hi, it’s great to be back here.

Rebecca: Today’s teas are:… Tim. Are you drinking tea with us today?

Tim: I’m not right now, but I was, and the tea I had this morning was a spice blood orange tea. It was quite tasty.

Rebecca: Looked like it was in a beautiful mug.

Tim: Oh, yeah, yeah.

Rebecca: turquoise… ribbed…

Tim: Yeah, with a little bit of a, almost a goldish highlight on the ribbings.

John: A bit iridescent.

Tim: Yeah.

Rebecca: How about you, John?

John: I am drinking a Lady Grey today.

Rebecca: Oh, I have its companion, a British breakfast.

John: Very good. Earlier this year, Tim, right before the start of the spring semester, we discussed a plan that you had developed in which students would engage in a structured sequence of interactions with a chatbot tutor that you created in each of the course modules in your health science course. The course was scheduled to begin actually the afternoon of the day when we last talked. We’re following up today to see how this plan has worked out once it hit the actual classroom. First, though, could you provide our listeners with an overview of the course, because not everyone may have listened to that, or it’s been a little while, and just remind us of what the course is and how the AI activities were structured.

Tim: So, as a rewind here, so I actually use this in two of my classes. The first one being our HS 300, which is a class focused around the diseases we get as people and how we work in that space, in more of a public health focus. And then the second class was a fitness wellness class that’s focused on how do we do physical activity more of a clinical population focus, and it had that as the topic, and then it’s also our writing-intensive higher-level course. So, in both of the classes, we had a tutor bot. The way that it functioned was I gave them a file that I built out that was a learning file that the bot would read and would have an idea about what are the topics that the student needs to learn and then would work with them in a very Socratic style to help them learn the content and the depth that was required as kind of the minimum foundation points for them walking into class in the first day and they did actually fill in the questions I had in that file, and it was required to hand that in, as we could call it proof of learning.

Rebecca: So I’m picturing the first day. How did students react to your plan?

Tim: About what I expected, honestly, which was tentative. And that first day, and I’ll actually expand that, and say that first week, we had a lot of discussion in class about what does this look like, what’s the purpose of it, as we had, I believe, discussed about six months ago. One component of that first week was about roughly half of one of the first two days of class in that first week, they had built-in time in class for them to actually dive in, learn about this GEM that was the host of our bot, and actually get to start working with the first file for that first week, and get their hands on. And I’ll say that first day it was very interesting to hear about the reactions, and some students, within the first five or 10 were like, “this is not for me,” which is fine, because that was how it was built, was that they did have an option B that they could do without having to engage with the bot. Quite a lot of them had said, “I’ve never ever engaged with an AI like this before, and I’m a little uncertain, but I’m willing to go and give it a shot.”

John: Roughly what proportion of students chose to continue with it, and what proportion chose to opt for plan B?

Tim: Yeah, that’s a great one. And in fact, I’ll say that was one I was watching throughout the entire term, because they have to hand in this like proof of learning, which allowed me to kind of track what was the proportion. Roughly, the proportion was like 70% continued with AI, and 30% chose to go the more traditional text-heavy route. I’ll go and say this. I would argue in a 16-week term, I would say just off the top of my head that by about week six the students had settled into the way that they were going to engage.

Rebecca: That’s not too bad, about a third in.

Tim: Yeah, it’s not too bad.

Rebecca: How did it work?

Tim: Really well, I would say. Overall, there were definitely things, that within the first two weeks I hit the fun point that I’m sure all three of us have at some point, which is Gemini pushed a major change in their models and updated, and suddenly the AI tutor [LAUGHTER] wasn’t working the way I was expecting it to work. When I had trained it and done the last round of kind of making sure it was working, that was end of December, start of January. By the time we hit like week two of the class, the model had been adjusted and suddenly it just kind of wanted to go off in its own in right field and treat the student as if they were like a medical student and go in way too much depth. So after that, I realized very quickly in that week, too, like, “oh, we have a problem,” so very quickly went back, adjusted the system prompt, put up some more forms of constraints in a guardrail to rein it back in, bring it back in, and effectively the way I did it was to get the bot to do less explaining and focus more on explicitly what the student needed to learn and allow the student to drive where they wanted to engage further. And after that adjustment it seemed like that solved the issue and the students were able to better engage with the bot overall and I’d say overall, for both the AI bot and the textbook route, the big change of having these effectively notes that they were expected to hand in dramatically improved their readiness to engage in a flipped classroom, which is the way I teach, and so I would count that as a win period for the AI bot, the fact that at the end of the day the grades were effectively the same regardless of the way that they engaged. So I would go and put that up as a positive sign overall that the AI at least performed as good as the traditional way.

John: And in a first iteration that’s probably a very positive outcome, because often new experiments have some rough edges on them. I was especially curious about the performance of the Gemini Gem, because I was working with a group of 18 faculty in SUNY from various campuses who were experimenting with chatbots, and many of them started with ChatGPT, but then moved to Gemini, and they had very similar experience, that the GPTs in ChatGPT seemed to pay much more attention to the guardrails that were put in, while Gemini Gems seemed to go off in directions that were different from what the instructions were, and that caused a bit of frustration for several of the faculty, because one of the reasons they moved to Gemini Gems was because their campuses had Gemini, and they thought that would allow all students to have more equitable access, but they had begun in ChatGPT because the grant had funded Chat GPT accounts for them, but again students were complaining that the free accounts didn’t allow that much interaction with it, while those who had paid accounts had much better quality of interactions, so it was a trade off, and ultimately pretty much everyone who had that choice did stay with the Gems, but it did require quite a bit more training. So, I think your experience there was not unique.

Tim: And I’ll just go and say this as well. I did also one more adjustment I did in week three which further helped was, in Gems, you can have certain variations of it, right? Just go and turn on as the default that the student can go and turn off if they want, but generally they don’t, and I defaulted on the guided learning mode, and that further enhanced that Gem as well. 8:11

Rebecca: So it’s always interesting when you try something new and something goes awry, and obviously it sounds like you adapted well. How did the students adapt to this minor interruption, especially because it happened early on.

Tim: I think the biggest item in terms of the feedback was that I was very open that first week of “this is an experiment, we don’t know what’s about to happen. I don’t know, I know what I’ve done for my testing, no idea what’s about to occur here, and I want to know your feedback, so let me know. We’ll go and talk about this.” And opening up that window of this is an experiment, you are my group. If this doesn’t work, we’ll adjust and change. I believe that it allowed that space for a dialog to happen, and that first week we did half of one class where they had a chance to go and get in there, work on it. I got a lot of feedback from them in that class, then once we got in to week two, that feedback of like, “Hey, this took a really long time because it kept like wandering off over here, and I had to bring it back, and it was really annoying,” I think was the biggest response I had, and then after we got it tuned in, I began to hear more feedback about, “Okay, I kind of like this, because if I’m totally lost on what is this problem with the heart, why do I care about it?” It really allowed them to explore if there was an area or a disease that they really found fascinating, and it just gave them more freedom to go and explore that. So, overall, the response that I had from the students was, I’d say broadly, it was kind of two broad groups. One was: this was very intriguing, and it was about the same as what I’d expect if I read the book. I’d say out of the 70%, like roughly I’d say like maybe 10 to 20 of that was that group of like it was nice I didn’t have to pay for the book, and then the rest of the group was generally that they liked the ability to have that freedom to explore or to ask the bot for more help in a space as they felt need.

John: Going back just a little bit, you mentioned you have these chatbots that are trained as Socratic tutors on that module, and then students complete an assignment at the end. Was the assignment built into the chatbot or was it something that was a standalone, and how did you receive all that? Did you get a transcript of the interactions along with some responses or something else?

Tim: So what I did was I gave them a file that was in Canvas, and the chat bot understands how the file is laid out, and it has a set of questions and tables and cases and all of that stuff without the correct responses put in there, and so the student’s job was to start a chat with this bot and to give it the file, and so that the bot knew how to drive the engagement, and then what they did was, as they engaged with the bot, they would go through and fill out each of the components of this file, and then they would just go, and it was a hand in before the first day of class that week that we did through the Canvas site. Now, a big part of not only these classes, but just the classes I go and teach across the board is that I do a contract grade, so for these items, because the focus is on “I want you to engage with the content, I want you to learn it, so in class we can work on like going further and building out skills,” so they knew that as long as they completed all the work, it got marked as done. I wasn’t about to go through with a comb and dock them points for everything that wasn’t exactly right, but once they had that handed in, it unlocked the key for them, and then they were in charge of ensuring that they went through and reviewed the key to make sure that they were in the right space.

Rebecca: Seems like a good plan, when otherwise that would sound like a lot of grading.

Tim: It did work well.

Rebecca: And you said that the students who chose the textbook method essentially did the same assignment, right?

Tim: Yeah, so it was adapted for the textbook, but exact same idea. The plan, as well, was to keep the overall load on the students the same. It seems like, based on what they reported in evaluations, that was the case, that regardless of path, it was about the same amount of time to complete each one.

Rebecca: Besides the tech surprise, were there other surprises, things that you didn’t expect? Anytime you go into an experiment, things come up that you don’t expect.

Tim: I think the one that was unexpected and interesting, and I suspect both of you, being faculty, you’ll hopefully laugh at this, was HS 300, in particular, is a content-rich class. I mean, not only is it a 300 class, but human diseases, there’s a lot of them, [LAUGHTER] like there’s far more than we can fit in one class, so we only focused on functionally the major diseases for each of the body systems, and still there’s just a lot of content that you need to know, and the feedback that I had at the end of the day kind of made me laugh, which was regardless of the path that they picked, because now we were requiring them to show at least foundational knowledge prior to walking into class as a core component of this class was that they were spending three to four hours a week outside of class doing all this work and going and like preparing for class and that it was more load than what they were used to or expecting and a lot of the feedback was “and I feel like I learned so much more because of this.”

John: That’s a nice positive outcome. We don’t always get that. We often get the “this class is so much more work than I expected,” but it’s not always combined with “and it was worth it.” That’s a really positive outcome.

Tim: Yeah.

John: You had mentioned that you were using a flipped classroom approach, and earlier when we talked to you, you mentioned that you were using active learning activities. Could you talk a little bit about what a typical class would look like in terms of a typical day in the classroom?

Tim: I’ll actually expand, I’d say a typical week, because usually things span like day one to day two. So, a typical week, the first day of class, we go and start off, and the first thing that we do… well, after the “hey, this week in class… , and don’t forget about this,” all of that fun at the start of class was we would do a tRAT, and for those who don’t know, a tRAT is a team readiness assessment test, and it is classically done with these cards that look like a card I’d buy for a lottery game, where we would code effectively a multiple choice quiz to the card, and then they work as a team to go and figure out what do they think is right. If they’re not right, they have to keep working on it until they go and get it right. I’ve created a digital version of that, so we begin with that, not only to kind of like warm their brains up, because I would get them Monday evening at 5:30.So, a lot has happened. [LAUGHTER] So that helped them kind of get their brains back into our content for the week and help them understand, like, “What do I recall now? What do I need to go work on?” and then we would dive into one of two things, if we were having a bigger application week, which was a lot of the more impactful work, we’d walk through, here’s our work for day one, day two, here’s our task for the end of the day, and then they would just dive right into it, and in those cases, day one was like they were working with a small, and by that I mean no more than four of them in a team, a small team of their peers, and they would work on doing more of the engagement part, and then day two would be wrapping up that, and then doing usually a presentation to go and share out. So, an example of a week, this one was a hit. It was new in this format, and it seemed like it went over well. When we were talking about diseases of the lung, the pulmonary system has a lot of diseases that are infectious, a lot of ones that we would have to report to the state if you happen to get it, and even if it isn’t directly involving the lung, that might be the way it’s spread. So we wanted to introduce them to the fact that there are these things called an investigator that works for the state, and all they do is investigate potential forms of these that might spread that we might need to do some public health work on. So in the first day, they were handed their initial case that they had to go and talk with this AI bot to figure out what had the patient done, where were they at? They knew that they had TB, but they didn’t know anything else. And then they were allowed to expand their investigation to potentially up to three additional bots, and they were told you might have to talk to all three of them, you might only have to talk to one of them. You have to figure this out on your own. You have to do the investigation. So they went through the path and the process to go through and investigate. As I’ll just go and say they did have to talk to all three. The first one went and led them to two of the three bots, and then one of those two went and led them into the last bot. And then they had to go and figure out what do we think happened, and then our day two of class, they wrapped up that work, and we had a chat as a whole class about what did you find out, how do you think this spread, what do we need to do next, and it was actually quite a complicated case, but that’s the way that it spread throughout that week was to go and show them, like, hey, this is an airborne thing, it’s a dangerous thing, and this is a real job you might be able to actually get, and by being able to engage with the AIs as effectively a patient, they had to go through the process of the fact that one of the bots was extremely not helpful, in fact, actually almost like was very rude, and the reason for it was because of the circumstances built into this bot if they were found out to have a TB test that was positive, they had significant potential losses that would be a major problem for them. So that allowed them to actually engage and attempt to work through real-life experiences that they might actually have to go and deal with.

Rebecca: That sounds like a fun activity, that of course leads me to want to know how did you build those bots? … and a little bit of detail about using AI in that context.

Tim: Yeah, so using AI in this format is, I’ll say for myself, has not been a new thing, it’s actually been something I’ve been doing for a while. It started for me out pre- 2022 and the world exploding and changing on us, but it started before that, because in my exercise physiology course I used to teach, engaging with a client is really important, and learning how to do that, so back in the pre-AI time [LAUGHTER] I had used IBM Watson bots that you had to build very crudely when we go and look at then and now, where they had to work with this fake form of a client to pull out all the risk factors and try to understand what were their goals and how do I work with that, and it worked okay. It didn’t work great. It worked okay, because of just the inherent limitations. After we got into our current era, the last four years, roughly, that expanded greatly, and what I would do is I’d build out a prompt for the GPTs first, and now for our gems that included all of the different things I wanted, as like building, so it did include things like here is the background of you as a person. Here is a personality. Here are the, for example, in this case, if they had had any type of testing done, what are the tests? What are the results? What are the tests that we don’t go and know about within all of those? What are the ones that you immediately share? And then there might be additional forms of like a test, like maybe an X-ray of the chest, that unless the student asks, the bot won’t just offer it up. So, what are you allowed to just go and bring up? What are you allowed to bring up only after they go and ask about this thing? And then I did find that when we get into the disease space, where I want them to model the signs and the symptoms of a certain form of a problem. I need to give them exactly what are the symptoms you’re feeling, and I never give them strictly the textbook, because that’s usually not the reality. They’ll have some, but not all. And then I found that it was also very important that I put a very strict box around the bot, this is what you have, so you are a person who has a problem A, it occurred because of B, C, D, E, F, G. You’re not allowed to share any of this. It’s only for you to use to better drive your responses to the students, and so that as they ask about a thing like, have you had a fever, if I hadn’t defined that, it at least knows which way it should kind of push.

John: One of the things you had mentioned when we talked last time is that you had students engage in some self-reflection, and you referred to this a little bit earlier. What did they reflect on? Was it a metacognitive type of reflection, or was it reflecting just on the material? Could you tell us a little bit more about that?

Tim: So this actually, I won’t say that I invented it, because it’s not mine. I actually borrowed a resource from the American College of Sports Medicine annual conference three years ago in Boston, where there’s a really good discussion about flipped classroom approaches within the exercise physiology space, and one of the items that was brought up that I borrowed because it had been used in some research work was this weekly graded survey. So, in the Canvas shell, it’s arranged as a quiz, but it’s marked as a survey that is graded, and all that that means is once they go and do it, it’s just marked as check done, and it runs through a range of different things, and it’s the very last thing that they’re in charge of doing every single week of the term. So it begins with exploring how was the week overall, and I don’t say just for this class, I just say how was the week overall for everything, and that’s a multiple choice, and then we go and walk through, how well did you feel prepared for this week in class? How was the week for you overall in your life? Because we all know that we teach folks who are increasingly more stressed out right now, like I don’t know about both of you, I might have brought this up before, but I have students who, 50% plus of my class, is in six to seven classes and working one or two part-time jobs. They are incredibly busy, and then there’s other life that happens on top of that. So, having a feeling of like maybe they were off this week and maybe their life just took a bad turn in this week, so that makes sense. And then we went and moved into some more open-ended stuff, which included: What did you learn this week? What do you have in terms of a question? Were there any types of like a moment that was an aha for you (it’s fine if there weren’t)? And then we do a bit of goal setting at the very end, which is: “What do you want to do next week to improve on what happened this week? And then when you look back at this week: Did you accomplish the goal that was set for the current week? And then finally, just kind of a broad: “Anything else you want me to know, like did your car break down on the drive up from Phoenix, and you were stuck on the side of the highway for a whole day? And it’s been a thing that I’ve used for a couple semesters now, and overall it seems like it works very well, and the feedback has been very positive, in that a lot of the students are like, “I’ve never had a class require me to, and I’ve never thought about, stopping at the end of the week and reflecting over how things went,” and it’s helped them with a thing like, especially my students, who are those “got to have an A” students, which was me, in fairness of, they might have a bad week, and then when they have to stop and reflect, they’re like, “Oh yeah, I had to randomly work more hours than I thought, and I found out that my dad has to go in for a cardiac stent, of course it was a hard week.” And so it helps them kind of put the week in context better.

Rebecca: In the spirit of reflection, now that you’ve had [LAUGHTER] this experience over a semester, would you continue the approach? What are some things that you might do differently next time? This is your moment to reflect. [LAUGHTER]

Tim: Yeah, so reflection for me would be this, I think I wouldn’t, and I’m not planning on moving forward in exactly the way that we did. Learning that just the fact of Google will push adjustments to their models regularly, and it’s not always easy to go and track it down, that makes it a little bit unpredictable, and I think there definitely is time that could be clawed back because there are times when AI was just inherently not efficient. So in the current model I think there are improvements that could be made, and how I would improve this is I am very much a fan of OER, especially at the current time when things are just expensive, so I would want to create and provide content that provides that foundation that is a little bit more controllable, so I have a better idea of like this is the time that they’ll spend here, so I don’t say like it’s going to be an hour or two, and then bringing in that AI in a more limited way, but in a higher impact way, where instead of using the AI to learn, they’re going and using it as a way to apply and test their own knowledge in a manner that’s going to give them immediate feedback and allow a challenge to the way that they actually approach the content for that week.

John: One of the big concerns with the use of AI models is personally identifiable information. Were you concerned about that issue? I know the contract that Google provides with campuses is that they don’t use the data for training, but were there any residual concerns about that issue?

Tim: This is a great point, because this is a big thing that I work with our faculty on a lot, because with AI’s advancement so fast, it’s so easy to ignore the legal terms of use, which I know everybody reads in fine print. [LAUGHTER] And so for this spring semester I was not worried about that, number one, because as you’ve alluded to, we have a Google ed license, which explicitly prohibits them from training on our chats that are through our accounts, and as a part of that, I made sure that we had a pairing up with that, with ensuring to be very clear, I want you to log in through your college accounts onto this, that’s the only way that it’s like a guaranteed private, and part two, even with that, I don’t want you to put in anything that you wouldn’t mind having as like a TikTok post, as like a public thing. It’s here to help you go and learn the content. It’s not here to help you troubleshoot why you’re not [LAUGHTER] feeling great today. So I did go and build in a bit just to help our students better understand, like, what are the safe and unsafe ways to engage with AI in this format, because I don’t think that’s talked about enough in the world. Now, with that said, I will say that there’s a very interesting change on the horizon, and that’s because our institution has been… for quite a number of years now. We have both Google and Microsoft. Don’t ask me why, I don’t know why [LAUGHTER]… but moving into this fall we’re moving to purely a Microsoft license, so we no longer have the Google license, which has been an interesting change to find out about, because it’s been very recent that we’ve found out about it, that really affects this work. So, moving forward, we, we being myself and the faculty who go and teach these classes, we’re going to have a conversation over the summer about how do we manage this, because the Microsoft format is through the Copilot, and unfortunately the way that it works is I can make these – they’re called an agent in place of a Gem, however, once I go and push them for a student, at least right now, from what we’ve been told, it costs us tokens, which means that there’s an ongoing cost that’s not like a Gem, where it doesn’t matter how much they engage, it’s all wrapped up in that fee. This is a big change, because, as you know, as I suspect, both of you are feeling like finances are a worry for a lot of institutions, so we want to be very thoughtful. So we’re having to go and talk about this, about how do we move forward in the future, is this a thing where we keep it as a G em, and we’re just explicitly clear about you should put no private information in here, because even with a free account, there’s a fair amount of use you can do with these, so that’s a big change we have to go and figure out as we’ve moved away from the Google group.

John: That’s really unfortunate, because I know in campuses where people have had both the choice of faculty is to use Google tools and to use Gemini and Gems, both because of the cost and because Copilots development is behind the others. Even though it’s based on the Open AI model, the training and everything else is unique to Microsoft, and it’s designed for a business environment to keep information locked down and also to build based on the use of the chatbots created by the businesses that are running Microsoft, because it’s designed more for commercial enterprises than for academic enterprises.

Tim: Yeah, well, and I’ll just say, as an expansion, too, because depending on the institution, I mean, one of the things I want to look into as well is there are increasingly more open source models that we can use, and there’s plenty of techniques that, if you have the compute capacity, you could potentially train a larger model, just going and thinking, like maybe you train a Llama 4 and use that as a teacher to train a student that’s smaller that then we can host locally. So there’s plenty of alternates, but it requires you to have the proper not only compute and server capacities, but also the fact that you understand how to make that work. So that is a thing I want to look into, because I do agree with you, Copilot is much more on the business versus straight up education side. So it’s going to be interesting to see how this changes, and perhaps that’s a point to go and have a chat about in like a year after we’ve had a year to explore how have we adapted to the Microsoft model.

John: And that is probably going to be a much more of a transition than adjusting to the change in Gemini.

Tim: Yes, it will be. [LAUGHTER]

Rebecca: It’s a good reminder how quickly this landscape is shifting in the role that the changes in technology have, the role that these companies play in what we’re doing, the finances around it all, they all have an impact on what we do in the classroom. We always wrap up by asking: “What’s next?”

Tim: Yeah, [LAUGHTER]I think I might have alluded there. So, what’s next is for this fall, for both of the classes where we’ve used this in. First of all, one have to figure out what to do with the fact that we will no longer have G ems in an institutional license, so have to figure that out. And then two, how are we going to do adjustments with the classes to provide content in more than just text, because that gets a little bit boring. So, how do we provide the content foundation, and then back it up with applied learning through the use of an AI to further enhance learning? So, we want to keep moving forward with this idea. We just have some hurdles we have to go and jump over first.

John: It’s going to be a challenging experience moving forward, but I hope things go well.

Tim: Me too. [LAUGHTER]

Rebecca: I hope you’re a hurdle jumper, because they’re going to keep coming. [LAUGHTER]

Tim: Yeah, I think I might be sore by the end of this fall. [LAUGHTER] It’s going to be interesting to see what happens.

John: Going back to the multimodal thing, one of the people I worked with this year from one of the other institutions in the health sciences field had put a microphone in a mannequin and a speaker there, and they had voice interactions with a patient who was basically just a plain mannequin, but they were interviewing it and asking questions. They enjoyed the multimodal interactions, probably a bit more than just text interactions.

Tim: That’s a really interesting idea. I might have to follow up with you more about that, because that’s a really clever idea.

John: It wasn’t mine.

Rebecca: I immediately saw your gears turning. [LAUGHTER]

Tim: Yeah.

Rebecca: Well, it’s always a pleasure to talk to you, Tim. Thanks for joining us again.

Tim: Yeah, thank you. I’m sure we’ll follow up again in the future about how did the transition go. [LAUGHTER]

John: You’re not alone, because institutions are cutting costs, and many institutions, including us, have access both to Microsoft and to the Google tools, so we’ll see.

Tim: Yeah, we have some strong feelings as faculty.

Rebecca: Yes, we do. [LAUGHTER] All the big feelings. [LAUGHTER]

Tim: Yes.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

451. All Day TA

 Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, Joshua Gans joins us to discuss an attempt at creating an AI platform that accomplishes this.

Joshua is a Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.  Joshua has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours.

Show Notes

  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Prediction machines, updated and expanded: The simple economics of artificial intelligence. Harvard Business Press.
  • Agrawal, A., Gans, J., & Goldfarb, A. (2022). Power and prediction: The disruptive economics of artificial intelligence. Harvard Business Press.
  • All Day TA

Transcript

John: Imagine having a virtual clone of yourself to assist your students whenever they need help and would track where students are struggling. In this episode, we explore an attempt at creating an AI platform that accomplishes this.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is

Joshua: Gans, Professor of Strategic Management and Jeffrey S. Skill Chair of Technical Innovation and Entrepreneurship at the University of Toronto’s Rotman School of Management with a joint appointment in the Department of Economics. His work focuses on the economics of innovation, entrepreneurship, and artificial intelligence. He has authored numerous scholarly articles on the economic impact of generative AI and, with Ajay Agrawal and Avi Goldfarb, of Prediction Machines and Power and Prediction, two books that have helped shape public understanding of the economic implications of AI.

Joshua: has been thinking deeply about what generative AI means for higher education: how it can support teaching and learning, how colleges should prepare students for AI-augmented careers, and how AI may reshape labor market opportunities for today’s students. At the University of Toronto, he and Kevin Bryan developed an AI teaching assistant, All Day TA, which has been used to provide students with support outside traditional class hours. Welcome,

Joshua:.

Joshua:: I’m happy to be here.

John: We’re very glad you are joining us. Today’s teas are:…

Joshua:, are you drinking tea by any chance?

Joshua:: I am not. I’m sorry. I see it in the title. I don’t see it in the instructions for the podcast, and I would have followed them. [LAUGHTER]

John: Most of our recent guests have not.

Joshua:: I guess the listeners out there should pause this now and go get their tea.

Rebecca: Sounds like a good plan. I have English afternoon today, John.

John: and I have an Irish breakfast tea today.

Rebecca: Going all imperial. [LAUGHTER] At the initial release of ChatGPT in late November 2022 a primary concern of faculty was really on the impact of generative AI platforms on academic integrity, but now that we’ve had more experience with generative AI tools, what have faculty and institutions learned about the risks and opportunities of generative AI?

Joshua:: Well, those initial concerns that people were worried about have only got worse [LAUGHTER] in many respects. Nothing in the development of AI has brought any comfort to those concerns, at least insofar as the foundation models are concerned. I think this was always on the cards, which is one of the reasons we started working on this problem back in November 2022 when ChatGPT came out. Put simply, it’s not just a student issue. ChatGPT helps you do stuff, and it helps you do stuff that you don’t like doing. It just so happens that we have set ourselves up for a challenge there with regard to students, because we give them a whole lot of things to do, for most of students, most of the time they don’t like doing it, [LAUGHTER] relative to other things they could do. And I think they very much understand why they should be doing it, but unfortunately, our students, it turns out, are quite human, and so a temptation is right there that they will take it. That said, there are actually things we’ve learned from this experience. I think actually students have been learning faster than faculty about AI. One of the challenges we have is, well, how do we prepare the students for the world that they’re going to be working in, where they’ll have AI tools, etc. when we, as faculty, and everybody else doesn’t know anything about that, [LAUGHTER] and really it’s very difficult to even anticipate. Well, one of the fortunate answers is that, amongst all groups in society, students by this very mechanism have actually spent more time with AI than most people, [LAUGHTER] and so they are more keenly aware of where it works and where it does not work. That may still pose a challenge for assessment, but I think students I’ve noticed have got better at using AI, even in their assessment, which has got to bode well for something.

John: We give a lot of workshops here, and there’s two sort of responses we tend to get from faculty. Faculty tend to be really excited about all the things that AI can do for them when they learn about them, but they’re really troubled about all the things that students can use AI for. What sort of changes should we be making in teaching practices to deal with an environment where student access to AI tools is pretty much ubiquitous?

Joshua:: I think, when it comes down to it, the problem we have is the problem we’ve always had, [LAUGHTER] which is how do we engage students and get them to learn for their own good. Now our tried and true method is to basically hold a set of carrots and sticks, or some mix between the two, and then come up with surrogates for learning, which is essentially not proof that you’ve actually learned or deeply understood, the things that are correlated with it, and they may be more correlated with it, or more appropriate for certain types of students, and certain types of learning patterns, and neurodiversity, we call it, and things like that, but that’s always been the case, and so basically our approach has been: you should learn this or else, [LAUGHTER] and the “or else” is also done with a caveat as well. It basically says “I’m a professor, I can come and I can give you lectures and talk with you. I can spend a little bit of time in office hours if other people have additional questions, but by golly, I hate grading. I hate setting assessment. I hate grading assessment. So I’m going to give you assessment a couple of times a semester, and we’re going to be all in on that, you either do well there or you get whacked with a stick.” Now, that creates a very high-stakes, high-pressure environment, but I don’t believe there is any scientific merit that that’s a good way to learn. The way to learn is the way we actually assume students learn, but none of them do, which is every week you prepare for the lecture, you do the readings, you come in and you pay 100% rapt attention in it, and then you repeat. That would be the nice way to do things, [LAUGHTER] but we haven’t matched the assessment with that at all, we’ve made the assessment to do with cramming for final exams and things like that. Now, to be clear, if you do do what the professors want you to do and pay attention in a methodical way, you do quite well. And you know which group of people actually did that? The professors themselves, when they were studying, they generally tended to do that. That’s how we select professors. So we basically created a mythical system with practical implications that are very predictable, and I don’t like to condone cheating, but if you tell people their livelihood depends on something, which is a critical moment, and they haven’t learned all this stuff, they’re going to do things. [LAUGHTER] I’m an economist, and that’s what we’d expect, and we’ve set up everything about assessment, actually, on the presumption that they will try and do those things as well. So, throw in ChatGPT, [LAUGHTER] you’ll get more of that. The sort of good news is that ChatGPT, and AI in general, is so powerful. We’ve got a whole group of professors saying, well, I guess I’m going to have to spend more time creating exams and proctoring it, and some people are talking about oral exams, and a whole lot of things. I guess I’m going to have to do all that, but the alternative is, or maybe not. Is there a way we can actually use the AI in a way that will actually alleviate the fundamental problem, which is high-stakes assessment, and so that was our motivation for creating All Day TA. We haven’t got to that aspiration, but that’s where we were starting from. Is there a way that we can reduce the incentives of students to go to ChatGPT and instead pay attention to the things that the professor wants them to pay attention to.

Rebecca: What I’m hearing you describe are faculty that sound a lot like students and students that sound a lot like faculty.

Joshua:: They have a lot of commonalities. [LAUGHTER] If you are honest with yourselves, we are all balancing all sorts of things, and in many respects, depending on your perspective, they can look the same way, which is one of the reasons why I don’t jump up and down with the moral force and outrage of some [LAUGHTER] of my colleagues, because we’re not exactly pure in this endeavor. So you can think about it, what if we were so pure and given that as professors we actually wanted to do the right thing by learning, what would we do? We’ll spend all our time on it. Then we might come up with something closer to what they do in Oxford and Cambridge, which is they have two or three students a classroom sessions. They teach all the time. They are able to monitor the students’ progress every week. It’s in an intimate setting, so everybody’s got an incentive to study and learn as they go along, and then who cares about the final exams? [LAUGHTER] They probably don’t mean anything whatsoever, and who cares about AI, because you’re sitting right there, you can talk to people, you can find out if they know things. So we know that if we alleviate the chief cost for faculty, that is their own time, then we can design the right system. So then there comes the question is: “Well, can we use AI to do that same thing?” And our answer is yes. [LAUGHTER] Yes, I think you can, because it is possible to take your own material, the things in the way you like to teach, transcription of your own lectures, add other stuff to it, and I know it’s not a popular phrase, but you sort of create a digital twin of yourself, and it’s a twin that can interact, and 95% of the time will give exactly the same answer to the students that you would if you were spending time there. But there’s a big difference, even in this ideal Oxford Cambridge world, I only have 24 hours a day, seven days a week, even in that ideal world. When we first did a very crude version of our chatbot for our own courses, trained on our own material, back in 2023, we had 200 students in the class, and they asked, over the course of semester, this chatbot, which is ChatGPT 3.5 which is like the very first version, 10,000 questions. There is no professor at Oxford and Cambridge who is answering 10,000 questions, and so you can see that. And now most of those questions came near the midterm and near the final… what a surprise… because it’s actually a very useful tool, given the current system, but at the same time, when we’ve done things since then, you can now use these same materials to generate questions to ask the students, and they use it for self-testing, and it turns out they like that even more, and that’s not even assessment, that’s just practicing for midterms and finals, they like that even more. And we got up in one class in Texas was 80,000 of these, they were using it all the time, and they’re very high quality because they’re trained on your data and with your learning objectives in mind, so it’s only a small step away from that to saying, “Why don’t the students do something a bit more like the Khan Academy does for elementary, middle, and high school, where it does little tests, and you don’t get to learn the next bit or do the hardest stuff until you’ve mastered the other thing.” What if we had these multiple choice short answers, or whatever things we might evolve to, if we put our minds to it every week, and then the students will be sitting there: “Well, I could go to ChatGPT for these questions, but the problem is by the end of the semester, ChatGPT won’t be able to answer them, and so I’ll need to answer them. So it may be just easier for me to just answer them now when they’re easy [LAUGHTER] and spread out my learning of it.” I don’t know if that’s possible. We have another thing we deal with in this, which is IT departments and integrations in universities. We haven’t had that sort of access to really put in the tests, but that’s the goal. The goal is, if we could alleviate the professor constraint, we can have low- stakes assessment at regular intervals, paced at the student’s own pace, that allows us to do things, for instance, like making it not by pass and fail, keep doing it until you know it, which is very low-stakes indeed. So that’s why I think there’s a lot of promise, but we’ve had a wake-up call. But the wake-up call reaction is not to run away from AI. I think it’s to run directly towards it.

John: Now, does All Day TA track student progress and provide reports to the instructor, or is it just something used by the students themselves without that sort of information. You’re getting some information on the number of uses, but do you have other information about whether people have attained mastery?

Joshua:: Oh, yes, we definitely do have other. So we don’t identify students by their questions and their performance, and you could imagine a world where some, for actual assessment, they give permission, and of course those get recorded, but there’s a reason for that. Is because we saw the sorts of questions students were asking, even if we didn’t know who they were, and like 40% of them were ones that they would be embarrassed to ask a professor, but it’s so much easier to ask the chatbot than to ferret around in their notes and textbook for the answer. So it’s really a smoothing tool, but if they knew the professor was looking over them, they probably wouldn’t want to do that. You want to encourage the students not to fear this thing and not to worry about, but at the same time, look at all this information you’re getting as a professor about what the students are asking about, where they’re struggling, and there could be also things that you expressed incorrectly. If the chatbots answering something incorrectly, chances are we know it’s because you failed to describe it properly, [LAUGHTER] and not because of the chatbot making stuff up. That’s not the way we’ve designed it. So, we provide basically, like you would with a really good TA, you’d sort of say, “Oh, where are the students struggling?” So, every week you get a summary of what they’ve been looking for, what they’ve been concentrating on, or where the difficulties are. You get access to all the questions, even though you don’t know who’s asked them, and you can look through all that. You can adjust your lectures next week because of the information you receive. You can also adjust the answers right on the spot, and say, “Oh, that’s not the answer I want to give here,” and I can adjust that. It all depends on how much time, of course, you want to spend with it, but all that information is there, and our experience is that professors love getting real-time feedback on what students understand. It turns out, for all our wanting to save time on things, we are interested in using our time well.

Rebecca: You’ve indicated a little bit about how many questions, the sheer quantity [LAUGHTER] of questions that students have asked. Do you have more feedback on how students are responding to the platform, and maybe what profiles of students are benefiting the most from All Day TA?

Joshua:: So, it’s varied on things. We’ve rolled out this platform that can be used in any course, regardless of the discipline, regardless of the subject, where it really has a nice fit, especially. And basically, the thing that gives them a nice fit is where those assessments are very high stakes, ironically, where you’ve got midterms, finals, and increasingly so at the moment, or where you’re in a profession like nursing, where not only do you have to do an exam, you’ve got to score high on it, very high usage of All Day TA for those circumstances. Where people have a PhD course, where they’ve got the assignments getting done, and other courses, which are a bit more discussion-oriented, and things like that, not so. It all depends. Now, we haven’t done nearly enough to know all those gradations, but we do know that as people currently move back towards in-person exams because of the AI issues, at least until we sort out something else. All Day TA is a huge component for that, because it allows you to do that without getting simultaneously overwhelmed with students in those exam periods, panics, and what have you. And we haven’t had a study of this, but I imagine also it alleviates student anxiety. I mean, we notice that most of these questions are coming out outside of work time, and that’s why we called it All Day TA, [LAUGHTER] because it really is around, and it’s there when the students actually need it, and so I think it can reduce anxiety just by its mere existence. Even though I would like it all to evolve into another system.

John: You mentioned this evolving into another system. Where would you like to see this go in the future?

Joshua:: Well, our business [LAUGHTER] is convincing universities to adopt this. I don’t know what your impression of university is, but fast to adopt technology is not their characteristic. The students are quick, [LAUGHTER] but the universities tend to be slow. Imagine you could integrate this into the LMS, and the students, you had all their courses and stuff like that. Imagine they’re taking a third-year course and they’re studying for it, and the previous prerequisite was in first year. Can you imagine that they have access to their first-year TA still in third year? That’s got to be a good thing. And then when you start thinking in those terms, you say, “Hey, why would we stop when the student leaves the university?” So the possibilities are quite enormous in terms of being able to provide something better for the students to learn and to work, taking into the very real account that they’re human beings [LAUGHTER] and have those sorts of issues with them. And so the immediate thing that I want to do is, I would like to, and we’ve had a couple of successes in this, being able to integrate right into the LMS at an institutional level. Fortunately, AI is actually not that expensive in terms of it requires authority, because doing things with LMS should require authority, but it’s not like a big budget item. [LAUGHTER] It’s actually quite small for what it is doing, and so I would like to do that, because I think that’s the path to providing a better student experience, but also providing the information we need to design whole curricula and other things. What is being remembered? Wouldn’t it be great to know in your third-year class what people are asking about on day one about the first-year class? Wouldn’t that be really helpful to know what they have retained and what they haven’t, and just be aware of it? We currently do all this kind of blind, and then once we’ve done that, working out how to put the assessment in, and the low stakes-assessments, and other things, is all possible, but at the moment, access to student records, and student assistance, and student privacy is extremely important, and we think it is the hallmark of this business, and these sorts of products, so it needs universities to be very proactive in wanting to do that.

Rebecca: One of the things that you just hinted at a bit is the idea of access to AI beyond the classroom and in jobs and beyond. So many college students are constantly expressing concern about the impact of AI on their future jobs, and what occupations might AI tools serve to complement, and where might they serve as a substitute for highly educated workers?

Joshua:: Oh, well, that is a very big and important question that we haven’t got a resolution to. We could have an entire other podcast about that issue, but the way I sort of thought about it in our modest example was, you’re a student, you’ve started your new job, you’ve had some on-the-job training and other things like that. Then something comes up, and they’re assuming you remember whatever, and you don’t. [LAUGHTER] So you can go to the internet, but it may be something that you recall your professor having talked about, and so you could then basically ask your professor again without emailing them for those things, and it could be brought back up, and it triggers the memories, and it makes those connections better, and it might just be much easier to take the knowledge that we’ve given the student, pour it into their head, force them to cram, but maybe they’ll just remember a bit of it, and they can pull that straight into the workplace and make them more productive on day one. That’s the kind of thing that I just feel we seem to say, “Well, you’re on your own,” [LAUGHTER] and everybody’s talked about lifelong learning and other ways of staying connected. Well, here’s another potential opportunity here that I can see how it might work.

Rebecca: Or an opportunity for quality control over the sources,

Joshua:: Yeah. Well, what if there are updates as well? It could be that, yeah, you learned it this way, but now there’s some additional things. I don’t know, the possibilities are endless. And, of course, this is not something any centralized company is going to be able to provide. It’s going to be the people who taught you in the past who are updating their own systems, and so long as you can push that out. It’s interesting, though. I think there’s a lot of faculty members who are concerned about such things. [LAUGHTER] I can imagine dire scenarios in which that occurs, but I have to admit I still have more than enough faith in what we add, in both developing content and also structuring it in nuance and thinking about learning and stuff that the replacement option isn’t there. I’m all about replacing the stuff that we don’t like doing, so we can concentrate on the stuff that we do like doing.

John: And colleges in general haven’t always done a very good job of encouraging that lifelong learning, because, for example, they set up systems to buy back student books, so that students don’t get to keep their books, and LMS systems, almost every institution will lock them down shortly after the semester ends, so all the material that they had and resources disappear. The practical reason for that is, if students have to retake a course, they don’t want them to have access to similar assessments, but for most students, having access to those resources would be really helpful, especially if that could continue after they left college, as you mentioned.

Joshua:: I mean, again, think about what having access to recent assessment is. Why is that the case? That’s the case because we kind of like to reuse things that work, but with AI, you can tweak it up enough that that wouldn’t be an issue either. And also that presumes that these are high-stakes assessments that people want access to, as opposed to low- stakes ones that they don’t. So, really, from my perspective, you can really see how we could push to a new system given the right set of circumstances. I’m hopeful one day one university will sort of like realize this and we’ll be able to evolve to it and really experiment with it. We’re doing a lot of that here at the Rotman School, but even here, where it’s been in our classrooms for three years, and a larger number of classrooms, we’re still not integrated with the LMS, because that’s controlled by the University of Toronto, which is one of the largest universities in the world, [LAUGHTER] and I must admit, with the current security issues that hit Canvas, no one’s jumping up and down to do things, so it’s a challenging environment.

Rebecca: You spoke a bit earlier about the concerns of faculty not necessarily having a lot of experience with AI, but still needing to help students prepare for a workforce that does use AI. Can you talk a little bit about additional ways that we can better prepare students for that work environment?

Joshua:: That is a very difficult question. One of the problems is that things are changing so quickly, so even when we get some research on what might be working for students or what they might need in the workplace, that was from a model that was a year ago, and now is it relevant? Now that will settle down eventually for that stuff, but I don’t see any short-term thing there. I think it’s more we have to understand that we’re running big experiments all the time now, and not only the students learning, but we are learning as well as we do them. A lot of faculty I’ve encountered have chosen: Well, what if we just pretend that the thing doesn’t exist?” [LAUGHTER] And the problem with that is you will not learn, you cannot learn, you cannot serve your students if you pretend the thing doesn’t exist. Now, there could be great rationales for trying to work out what you want the students to learn without AI and what you want them to learn with AI, and we’ve had this challenge before. We had the same challenge when we had calculators, and we had the same challenge when I was told, when I was going through high school, I wasn’t allowed to type up my assignments at all, because computers started to have spell checkers, and that boy did they not want me to use computers to spell correctly. I should just somehow do it myself, even though let’s face it, the second I was out of there, I never had to worry about that again. So [LAUGHTER] that’s the kind of things that we’re dealing with, but there are things you have to learn, and any new technology really sort of forces you to think about what is it that I need to get into a student’s head, and will AI get in the way of that, which it could.

Rebecca: Probably like the need to prioritize limited resources, limited time with students. You have to decide what’s the most important to focus on, and what maybe we can offset to technology.

Joshua:: Indeed.

Rebecca: It’s always about resource management, isn’t it? These economists….

Joshua:: Ah yeah, exactly, [LAUGHTER] that’s correct. [LAUGHTER]

John: Going back, though, to the labor force, as Rebecca had mentioned before, there’s a lot of students who are really anxious about what their job prospects are like, and we’ve seen in the last couple of years that the job prospects in the U.S., at least, for a new college graduate have deteriorated quite a bit, and it’s not clear how much of that is due to AI, how much, because of the random policy shifts that are occurring at the federal level, or whatever else might be causing these things. One concern that’s coming up a lot in discussions is whether college is still going to be relevant in a world in which people have access to AI tools, and can use them to substitute for the types of learning that occurs in college. Do you think college is going to disappear or change fundamentally in some way?

Joshua:: Well, that is a, of course, a great question, but it’s not a question that just exists because of AI. I think, actually, with proper introspection, we should wonder about whether we have been, for the past few decades, serving students well. Now the evidence is that they earn more, they can get more jobs and other things for a college degree than with not. In the U.S., at least they’re quite expensive. It’s not clear what the value of that is, since at other places it’s much cheaper and students seem to get similar benefits. So we could discuss that. So there’s always a constant problem of what is relevant, what is out of college, and also half of what you get out of college is not related to the formal education, and it’s related to other things, growing up and socializing, and becoming an adult, and all those other things we list [LAUGHTER] to justify our existence. But the student anxiety and stuff, I’m not surprised, students are anxious about AI, because all of the big proponents of AI are going running around saying they’re going to wipe out all the jobs, and I have never bought into that. I’m, amongst my colleagues, a very prolific user of AI. I use it for all manner of things, and some of those things are quite excellent. I’m a head of department, and I use AI now to get rid of all of those reports and things that I have to write for the university that might have taken a day. They now get done in seconds, and it’s great because no one was ever going to read those reports. They just wanted to see that they existed, and now that I can use AI, they exist, much like students producing assignments, and so I can’t blame them. Students are producing assignments, and there’s producing for professors who, for the most part, don’t want to actually read them. Anyway, nonetheless, [LAUGHTER] the thing that I’ve seen from that is that anything that is enjoyable and similarly meaningful to you, and also involves judgment and taste, these are things that you can’t get from the AI. These AI models, as we wrote in our first book, are prediction machines. They’re good at certain aspects of intelligence, but not others. They’re quite miraculous in the way they’ve evolved over the time I’ve been studying them. No doubt about that. They’re amazing things, but in all of it, I haven’t had any reason to doubt the value and usefulness of what humans do, even though it’s sometimes very difficult to define. It’s come in my own experience, it’s come in all of the data that is generated. As I said, we can’t identify whether people have been losing jobs from AI. Will some people lose jobs from AI? No doubt, because it’s going to reorganize things, that happens, but is there going to be the mass 50% unemployment that somehow engineers in Silicon Valley can imagine, but no one else can. That’s not going to happen. And fundamentally, it’s this: if you believe enough that humans matter, that even with the best AI, humans provide something of value. If you believe that, you can simultaneously believe that AI is going to be great, and AI is going to be great for people. It’s only where you are in a mindset that I think is what drove the Silicon Valley types, is that humans are crappy, stupid, and should be replaced, that you get big use of AI and big displacement of humans. And so, fundamentally, that’s the different assumptions, people who are anxious have to say: “Is there something… what is it that I can do?” Now, I mean, the problem is that, being older folk, we sort of have more opinions about our value [LAUGHTER] as we’ve learned it over time, and university graduates, all they’ve been doing is not getting that, [LAUGHTER] not yet. And so it’s sort of like you would have to take that on faith, but I must admit I’ve seen no evidence to the contrary. That’s my resident belief at the moment, and so I’m on team human.

Rebecca: So pro human, I hear.

Joshua:: And I was perfectly open to the idea that we weren’t. I mean, when it comes down to it, my use of AI has been a whole lot of things that I’ve decided the AI is better than I am. I’ve decided that literally answering student questions, I think it’s better than I am… [LAUGHTER] doing those sorts of things. But ther’re other things, there’s enough of the other stuff, enough of the other stuff that I think it works, and I can spend more time doing that, so even better.

John: And at two in the morning, AI tends to be much better at answering student questions than we do.

Joshua:: Exactly, and the students know that too. The students don’t like going… there’s some students who like seeing professors, students who are really at the top of the class want to see it, but not the seventy on the test. And then the students who are struggling also like to go because they need that extra help. And the beauty of it is, is if you use All Day TA, you don’t have the mass in the middle coming and bothering you, and you can concentrate on the parts where you can actually provide the most value. It’s absolutely true. What do we get most value out of a professor? It’s engaging with really smart students who we think are going to go somewhere, or a struggling student who really was close to giving up, that you sort of explain things to, and they’ve been able to come through. Those are our rewarding parts. That’s what we want to do. We don’t like the students to come in the middle saying, “Will this be on the test? They can ask the AI.

John: And there is a fair amount of research that suggests that first-generation students and students with imposter syndrome are very unlikely to go to their professor to ask questions, because that would reveal that they don’t understand things that well, and they’d rather keep that hidden, and many of them would just give up. So having access to free tutoring service without any judgment 24 hours a day is a valuable service.

Joshua:: I would go beyond that. Well, I’m in one of the elite universities, University of Toronto. The vast majority of tertiary institutions are not that, they are ones where a lot of students struggling financially, they have other jobs, they’re working part time, they’re studying part time, and things like that, and those are the students who drop out. And why do you imagine they drop out? Because they’ve got these assessments, because they have under-resourced teachers as well. So, they have these high-stakes assessments, and it’s basically on the night before, they haven’t been able to learn it, and they can’t get any help. And they said, “Well, I can’t do this anymore.” Now, if just a few of those students, [LAUGHTER] because of All Day TA, can get the answers they need and be able to hump over those moments, they can complete their degree. And so, if we can just move a few percentage points on dropout rates, I will be more than thrilled.

Rebecca: Well, we always wrap up by asking, what’s next?

Joshua:: Well, what’s next? It’s very interesting to think about what’s next, because normally when we talk about what’s next, we’re thinking like in five or 10 years. Now I’m talking about what’s next is maybe later this year… [LAUGHTER] would be the sorts of things we’re thinking of, and I actually no longer have a very clear view of it. I kind of feel, “Oh, it’ll be exciting and new,” and I get excited anytime some new thing comes forward. I think what we’re going to see, what’s next, more is that the AI is going to start coming into the fore in the other parts of our jobs. As I already said, it can help administration, it can help teaching. We kinda know those. I think in research it’s going to be a very interesting development, where we will be confronted with our desire to know what the answer is versus what used to be a complimentary desire to be the one who found it, [LAUGHTER] and so we will have to reckon with that in regard, and everybody will have sort of views to it on that, but it’s not until it actually happens, or it happens to you that you really know how you feel about.

Rebecca: Much to look forward to, that’s for sure.

Joshua:: It’s going to be exciting. [LAUGHTER]

John: It is, and the rate of evolution of AI tools has been phenomenal. I don’t recall seeing anything else develop as rapidly as this.

Joshua:: No, it’s crazy. It is crazy, and I was studying this seven years before it occurred, [LAUGHTER] all the latest stuff, and it was like a surprise, how quickly it happened.

John: And just a few years ago, there were all these reports about people saying AI will be there in the next five years, and that kept getting extended, and all of a sudden it was there.

Joshua:: There it is. We arrived through the Turing test, the whole thing. Amazing.

John: Well, thank you. This was a really interesting conversation, and it’s a really interesting time to experience these things.

Joshua:: Thanks for having me. I’ve enjoyed it greatly.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

450. Collaborative AI Guidance

With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, Kathleen Landy and Joshua Foster join us to discuss a collaborative approach involving student government, administration, and faculty.

Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College.

Show Notes

Transcript

John: With the rise of AI, institutions are scrambling to provide guidance and policies for students and faculty. In this episode, we discuss a collaborative approach involving student government, administration, and faculty.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Kathleen Landy and Joshua Foster. Kathleen is the Provost and Vice President for Academic Affairs at Genesee Community College. Prior to this, Kathleen has served in leadership roles at teaching centers at Cornell University, Queensborough Community College, and Mercy College. She has taught in multiple modalities and also has extensive K-12 teaching experience. Kathleen also served on the Pod Network Core Committee from 2022 through 2025 and currently serves as a co-chair of the SUNY Council on Assessment. Joshua is the Vice President for Academic Affairs for the Student Government Association at Genesee Community College. Welcome, Joshua, and welcome back, Kathleen.

Kathleen: Thank you so much. Great to see you again.

Joshua: Thank you for having me here.

John: Thanks to both of you for joining.

Rebecca: Today’s teas are:… Joshua, are you drinking tea with us today?

Joshua: I am. I am having some Japanese Sencha green tea here in my GCC mug.

Rebecca: Amazing. And how about you, Kathleen?

Kathleen: Well, I don’t know if this is like wearing the concert T-shirt to the concert, like the band T-shirt to the concert, but I have a Tea for Teaching mug, and I’m defaulting to Irish breakfast tea.

Rebecca: Clearly one of the cool kids. [LAUGHTER]

Kathleen: That’s right, that’s right. [LAUGHTER]

Rebecca: How about you, John?

John: I have a Twinings English Breakfast tea today.

Rebecca: Oh, nice.

Kathleen: Very lovely.

Rebecca: I think I have… maybe I should verify…

John: Another imperial tea?

Rebecca: I do, [LAUGHTER] I have a Scottish breakfast tea today.

Kathleen: That sounds lovely. I don’t think I’ve ever had a Scottish breakfast tea.

Rebecca: It’s got a little more oomph to it. [LAUGHTER]

Kathleen: Okay.

Rebecca: A little bolder, I feel like.

Kathleen: There you go. Okay, I’ll have to check it out.

John: This is a follow-up to our earlier podcast, where we talked a little bit about program- level AI policies, and you mentioned that you had been working with a member of the Student Government Association to come up with some recommendations for faculty and students. Could you tell us a little bit about the origin of this collaboration on the policy?

Joshua: Sure, I would be happy to do that. As a liaison between students and faculty on campus, students often approach me with various concerns, and one concern that came up multiple times during the fall semester was students having their legitimate work flagged as partially or even completely AI-generated. This initially prompted a discussion in one of my classes, at which time I learned that many students actually avoid the use of, or even the mention of, AI in order to avoid the potential of a suspected academic integrity violation, but I also learned how frequently academic integrity violations related to AI do occur. So I realized this was a topic that necessitated a more direct and open conversation involving both the faculty and the student body, and that is what led Kathleen and I to develop these guidelines as a way to guide those conversations,

Kathleen: I’m happy to add to that, Joshua, because this came up organically in one of our routine meetings. So, Joshua and I, as part of his role at the SGA (Student Government Association), and my role in Academic Affairs, we meet every three weeks, and so this came up in one of our conversations, and it was so timely, and it was totally a function of happenstance that at the time I had been having conversations with academic program directors and multiple faculty about the necessity of having course-level AI policies, and so it was very serendipitous, almost, that there was this student concern that Joshua articulated, and I had been exploring this question with faculty, and it seemed like a really lovely opportunity to bring everyone together and explore this topic.,

Joshua: And I had actually heard at the time members of the faculty discussing that they did have concerns as well, and so it sounds like the people who we serve both had concerns, and so we could kind of bring those concerns together, merge them together in the development of these guidelines.

Rebecca: Can you talk a little bit about how you facilitated the creation of the guidelines? Who participated in developing this initial round and this initial draft?

Kathleen: So, Joshua and I, we opened a shared Word document, and we just got some thoughts together, and we thought initially it was a no-brainer to have this set of guidelines have two parts, one that was faculty facing and one that was student facing, and we knew that they would kind of run parallel to each other and reinforce each other, but we started with a little bit of a brain dump, and we just kind of mapped what we thought were good ideas and good starting points and points of entry for the conversations about AI and academic integrity, and we landed on, I guess, we would call a sacrificial draft that we thought we could both bring to our respective colleagues and get more input. So it was with real humility and curiosity that we did a little road tour. We hit the ground running with this document that we’ve shared with you as well, and I personally brought it to colleagues in Academic Senate. We have a standards committee, and so I brought it to the chairs of that committee and shared it with our academic program directors, and we went from there. We solicited feedback, and it was great feedback that we got. And Joshua, I don’t know if you want to speak to folks that you brought it to. I don’t know if it was the SGA cabinet.

Joshua: Yes, so I initially didn’t bring it to too many students. What I did rather was I sort of took what the guidelines were without actually taking the draft, and I took that and I talked to different stakeholders about their concerns, and I brought up where different points that were covered in those draft guidelines and got their feedback on each point that I thought was relevant to them, and then I would take that and contribute it back in, and then I would also look at what some of the different faculty members are who I talk with and get their feedback and kind of combine that feedback with the feedback from the students and bring that back to Kathleen and to her colleagues, and I think that was really a good collaborative way that we took this, to include both levels of feedback, but not have the students edit the guidelines, but have them indirectly contribute through my contribution, I think, allowed them to really be able to express their thoughts and their concerns more clearly.

John: We have a student committee on learning and teaching here, and this came up in all of our discussions with that group, and the same sort of concerns that you mentioned, Joshua, appeared, that some students were really upset that they were accused falsely of engaging in academic integrity violations, and also others were concerned about whether it was safe to use it, because they’d like to be able to develop skills in this, because it may be relevant for their careers. It’s a topic that I think applies to a lot of people. Could you give us an overview of the guidelines, both for faculty and for students in this draft document?

Kathleen: The guidelines are surprisingly simple, but we strove for them to be very clear. So, the guidelines include having a clear AI policy that’s articulated in every syllabus, even though that’s a soft expectation, we wanted to make it a little bit more concrete and a little bit more pronounced, and so we wanted to encourage faculty to have a clear policy that includes specific examples of acceptable and less acceptable uses of generative AI, and a disclaimer or an explanation about any possible AI detection platforms that might be in use for the course. So that’s kind of an equity issue. It’s a social justice, it’s a transparency related issue that if there are detection platforms in play, students should be aware of that. That should be a known element in any learning environment, and importantly, this same AI policy should have a clear list of consequences for initial and possibly for subsequent violations of the class policy. So we wanted to start with having something concrete in place, something that’s published for students and faculty alike to refer to. And then the second part of the guidelines for faculty involve posting open and candid conversations at the beginning of any course or assignment, something that came up in the previous discussion that we had was this idea that in sympathy for the schedules that students have and the potential for them having five different course level policies to keep track of, it’s really not realistic for any given course level policy to be expressed once and then never mentioned again, or to be published on page eight of the syllabus, and then never revisited again. So, the idea of not only having a clear AI policy, but revisiting that and explaining it and communicating it frequently and clearly, that’s a kindness that we can show to students. It’s a good practice. So, this idea of hosting ongoing and candid conversations within the class, especially at the beginning of a course or the beginning of an assignment, that’s really critical. And then finally, the guidelines involve one or two strategies for fostering greater student buy-in, for lack of a better term, or a greater sense of accountability among students for the policy, so things like having students sign an integrity statement, a commitment to acting within the confines of an academic integrity policy, having students sign an integrity statement, or including a question about AI use and implications within the context of a syllabus quiz, or something that’s a little bit more interactive. We thought that would allow for a structuring of engagement with the policy, to not just have it live somewhere, but to have students actually engage with the policy.

Rebecca: It seems really important not to just have something exist on paper or some virtual ether, right? [LAUGHTER] It’s like have people actually interact with it.

Kathleen: That’s right, especially because for many students, and this will change over time, of course, but they might not be accustomed to having to check with the course-level policy. These are relatively new and emerging caveats or elements of a syllabus. So, how could we not redirect students to say, “Hey, remember we have this policy, we have this expectation?” It’s really a kindness, I think, to students. And then Joshua, I’m happy to turn it to you for the student side.

Joshua: Yeah, thank you, Kathleen. So, the guidelines for students really do mirror and complement the guidelines for faculty, and I want to start by pointing out that, similarly to the guidelines for faculty, the guidelines for students do not dictate when the use of AI is appropriate or not appropriate. Instead, the goal of the guidelines is to help students learn how to communicate with their instructors to be sure that the course policies are clearly understood prior to any misunderstandings happening. So, with that in mind, the guidelines mentioned topics like requesting a course policy on AI, if one is not provided, and asking questions of the professor or the instructor to understand that policy. They encourage attending office hours, sending a professor an email with questions, or even asking a question in class, so that other students can benefit. Maybe other students have that same question, and maybe they don’t want to ask for the reasons that we discussed earlier. So, the guidelines, in addition to this, include steps for how a student could start a conversation with a faculty member if that student’s work has been flagged as having been AI generated, regardless of whether or not that work was correctly flagged as having been AI generated. The guidelines stress being respectful and aware that misunderstandings do occur, but they also encourage students to advocate for themselves when necessary, and I personally believe that student self-advocacy is a very important skill for students to learn early on during their college experience. The last major focus of the guidelines for students is the importance of building strong and collegial relationships with faculty members, which can be difficult or is something that is even avoided by some students, especially younger or first- nyear college students who are still adjusting to leaving a high school environment, and I believe that this is actually one of the most important parts of the guidelines.

Rebecca: I think I hear some really common themes around transparency and relationship building.

Kathleen: There’s no doubt. I was speaking with Joshua. We had a debrief of an event that we co-facilitated this past Friday, and I was struck by the extent to which the following statement is true. Almost all of my recommendations or strategies or suggestions for how to handle issues involving academic integrity and the use of AI, in particular, almost all of my strategies involve the human elements in a learning environment, and they’re all relational. It all comes down to either communicate… not all, but it often comes down to communication and relationships rather than the tech itself, and so I think that’s a really interesting relationship: that the tech challenges that we encounter often have very human or relationship-centered solutions. That’s been my experience to date. So I agree with you, Rebecca. There are trends in the solutions that we keep coming to.

John: I believe you also did some sessions with faculty, staff, and students about this. Could you tell us a little bit about that, and what sort of reactions were received?

Kathleen: So, we’ve had one session, and we hope it will be the first of many, and I have to tell you, it was awesome. We were delighted, we had over 50 participants on a Friday afternoon, no less, which is no small feat. We were very excited by the turnout, and we tried to be very mindful of how we engaged folks from the beginning. We wanted to set the right tone, and so we started with a little bit of a series of short interactive polls to let folks know that we’re all coming at this from a point of intellectual curiosity, and we wanted to build a little bit of a sense of community around this topic, and so the very simple question that we started with, it was a simple qualitative poll everywhere to just text your answer in, and we asked a very simple question. We asked what word best captures how you feel about generative AI in your classes, very simple, and we knew we could get anything, and so we gave everyone simple directions. Everyone had the option to pass if they wanted to, but everyone had the option to text a single word, and we generated a word cloud right there. I can send you, if you like, a picture of this word cloud. I’m looking right at it. I have to say, it was really lovely to see the extent to which students could see the faculty’s human responses to this question, and faculty could see students’ human responses to this question, and so the responses were anonymous, but the word cloud was really thought-provoking, and it was populated with words like conflicted, concerned, curious, complicated, stressful, uneasy, overwhelmed. I think it set a tone where everyone could kind of check their egos at the door, or check their personas of being an academic, or a scholar, or an honor student, what have you, fill in the blank. Everyone could check that at the door, and we all kind of became students of this together, and it was, I thought, surprisingly effective at just setting a nice tone of inquiry for the remainder of the session. The session was about an hour long, and so we had interactive polls and a bit of an overview of the guidelines. We distributed a physical copy, like a paper copy of these brief guidelines in the room, and we posted them for the hybrid. It was a hybrid session, that’s worth noting. We posted them for the online participants, and it generated rich discussion. We got some great suggestions from both instructors and staff alike. We had students there who weighed in. We had folks, both students and instructors, share experiences, times where they wish they had these guidelines. Some expressed a concern about having or desiring more robust guidelines, like they wanted more of the same. And I’ll maybe stop there, because Joshua, I’m sure you want to add to that, but I was surprised at how eager folks were, and I was gratified by the enthusiasm that the group expressed.

Joshua: I was very excited as well, after having been at that session, and I joked with Kathleen a little bit at our follow-up that I wish we had had two or three more hours to go on, because I think that we certainly could have, but I think that the thing I was most pleased about was that we had a good amount of both faculty and students in attendance, and that they could see that, in most cases, they are in agreement with each other on these issues, or at least there is some mutual common ground between our students and our faculty, and I think that was the first thing that was really important for both of those groups to see, and then the second thing was that a lot of students came to me afterwards and said that they were really excited to have learned about how AI is becoming a common issue and how interested people are in it. As an accounting student, I have found that within my classes I’ve had a lot of people who are both interested in AI and who are both very hesitant about it, and ironically, if I had to make a really, really wide swoop with a brush to say who is more pro the implementation of AI, at least in the specific classes that I have been in, it’s the faculty, not the students, and so I think that the students being able to see that was really helpful, that that may be what we students imagine that the faculty are thinking about, and what the faculty maybe imagine that the students are thinking about maybe those are being too heavily stereotyped, maybe those aren’t really, really how we are seeing this, and so I think that was really good to have experienced, and I just think it was the meeting was very well received. There was obviously a lot of desire for follow up to it, for more conversations, but I think that the conversation that we did have was a very good starting point, and it helped to really demonstrate that where the direction that we’re going is really the direction that we should be going.

Kathleen: Thanks, Joshua. The only thing I would add, because I don’t want to paint this picture that it’s all rainbows, and we all sang together at the end of this session. I did have a couple of faculty members, and I was super grateful that they attended. They didn’t express frustration, per se, but their response was more like, “yeah, and…” like they were just ready, and I’m loosely paraphrasing, but this is close to the exact language. They said, “this is good, this is fine, Kathleen, but this isn’t the real work. The real work has to come in the articulation of departmental policies, and what’s that look like?” And I guess my response to that was, “well, yeah, of course, that’s the departmental next step, and that’s not for Academic Affairs or SGA, for that matter, to dictate to departments what their departmental or course level policy should be,” and so I think the workshop work is what should follow. So, there was a little bit of maybe frustration, that might be too charged of a word, but there was a little bit of feedback that was, “yeah, and okay, like, get to it, get to the next step,” which indirectly I interpret as affirming feedback that we’re on the right track, that folks are eager for guidance or at least engagement on this topic.

Joshua: And I believe that the same is true for students. I think that students really want that greater level of course-level policy or program-level policy, but I think that we’re not to that point yet, and so at this point a student came to me today and asked me. My response would be, “think about communication, because that’s what we are able to do now. That’s what we can do now.” Those other more clear things will come, but now we have to focus on what we can realistically do together.

Rebecca: It seems like a really valuable opportunity to demystify what people are thinking about, and that’s a really important first step into moving towards policy, or something that’s more formalized. You’ve certainly presented this as a working draft, and some revisions, and things like that. What do you see as the next steps to maybe formalizing this work at an institutional level, and then obviously there’s some work that would need to follow at the departmental level.

Kathleen:I think there’s a few next steps, and one has to deal with instructional practice and faculty development, and the other has to deal with steps to take leading towards policy articulation. I think the immediate next step involves endorsement, to take the feedback, the great feedback we’ve been given, and refine these guidelines, share them again, probably have a few more rounds to get some greater input, but then go through the formal channels of shared governance to have endorsements, formal endorsements, so that we can say “here are some endorsed guidelines, and those endorsements are coming from Academic Senate, from our local union, from Cabinet, from SGA, from program chairs.” We can get a lot of structures on board to kind of promote these and broadly share them. On the instructional side of things, and this is connected closely, of course, to faculty development, I think we can generate conversations about all the things that we’ve been discussing and all the things that are focused on in these guidelines. One of the polls that we asked, one of the questions we asked this group, and we had some questions restricted to students and some restricted to faculty in this one- hour session that we had, and one of the faculty questions was very simple, yes or no question: “Do you presently have a clear published AI policy for your courses?” And 57% of faculty said yes, and clearly 43%, my follow-up question is, “why not?” Let’s get that number a little higher of those that do have a clear policy, and I think that’s really good for that 43% of faculty to see, like, “oh, wow, 57% of my colleagues present at this session have something in place. Let’s talk about what that is, and maybe I should have that,” and I think that those collegial conversations, and then doing the work of adopting, crafting your own AI policy, and publishing that in the context of whether it’s a Brightspace shell or a syllabus, likely multiple places, that’s part of what’s involved in the next steps from my perspective.

John:I know in our campus there’s been a fairly wide range of perceptions concerning AI detection software. Some faculty trust them very strongly, even when the evidence suggests otherwise. Other faculty very reluctant to rely on them at all, except maybe as a signal that they may want to contact a student to talk about the issue. I was really pleased to see that your guidelines explicitly stated that AI detection software is fallible. Could you talk a little bit about that, and whether people in general have accepted that message there?

Kathleen: I’d like to think they have accepted it, I think, and I also think it’s a case that folks are still exploring what detection platforms do and how they operate, and how they might be of use to them in their own classes. So, I do think it’s variable in terms of comfort with and knowledge of these platforms, but I think there’s a lot of false positives, there’s a lot of false detections, and I think disclaiming that and modeling how to have the conversations about that, that’s something that we hoped the guidelines would get into. I think dragging that issue into the light is part of the intent of these sessions. I think folks are pretty receptive to it, because there might even be a self-consciousness about what we don’t yet know. So once again, I do think there’s a dotted line to faculty development there, and I think we have to model the questions that we ask as we analyze the efficacy of these platforms. So I think it’s time for candor, and I think it’s a time for inquiry, and just the humble recognition that we have to talk about what we need, what we want, and how good these platforms are. How efficacious are they?

Joshua: And something that one of the faculty members, who was present at our session, brought up, and something that I really appreciated from the student perspective is the acknowledgement that because these AI detection platforms are not perfect, they can actually serve to erode the level of trust that has been built between students and faculty members. And so it’s unfortunate that they do that, but that’s one thing that will have to be accepted when using an AI detection platform, and so, as a student, I appreciate that faculty are acknowledging that reality, and so that means to me that they will be more sympathetic and more understanding, because obviously in some cases those tools, those positives, are accurate, but in other cases they may be a false positive. And so just having that awareness and the fact that faculty members are acknowledging that level of student trust that may be called into question, I think is really good for them to be able to recognize.

Kathleen: Well, it’s great that you say that, Joshua, and I do want to respond to it. It prompts me to underscore this. I think the disclaimers regarding the effectiveness of AI detection platforms, it drives the conversation to prevention rather than detection, and I think that’s a huge part of not only instructional and assessment practices, but also the learning environment, and it goes back to the relationships and the trust that you mentioned, Joshua. So, I think that’s a huge part of it, the shift from detection to prevention, that’s a nice point of entry for me, and the conversations with faculty.

Joshua: And I think that that level of trust between students and faculty may be the first step. probably is the first step, in that prevention.

Kathleen: That’s right. Agreed.

Rebecca: So, you’ve talked a little bit about wanting, or even departments, maybe, being eager for like the next steps. Can you talk a little bit about what movement there has been in terms of the course-level or program-level policies on AI, or what you maybe expect to see as we enter into the fall?

Kathleen: We’ve had a few departments, a few programs, or program clusters, I should say. We’ve had a few folks self-organize, and they’ve proven to be self-starters in this domain, and they’re really gathering and just meeting routinely to articulate policies and to explore what other institutions have to craft that together, and that’s inspiring. That’s really great to see. Our business department comes to mind immediately. They’re doing great work, and there are others that I think are curious, and folks are kind of coming to those realizations of, hey, we need to huddle up and figure this out. I think folks are coming to that conclusion on slightly different yet overlapping timelines. So, I think there’s a role for my role and Academic Affairs Administration to prompt that collaboration a bit more intentionally, whether it’s the articulation of a strategic objective relative to having a target, the number of academic programs to articulate as policy, regardless of what it is, I think we can prompt that a bit with the strategic planning that we do, with the faculty development planning that we do, and I think to a certain extent we’ve done that, but I think we can turn that up a little bit. I think we can be a bit more intentional about it, but it’s happening just at different degrees and varying by program.

John: We talked a little bit about AI detection and how perhaps that could be used to nudge people in ways to use AI within the guidelines of the course, but are there other strategies that faculty are using to encourage students to use AI ethically rather than to shortcut their learning?

Kathleen: It’s a great question. I think, and this will vary by discipline, so I don’t want to speak in too broad of terms here, but I might end up doing so. I think to the extent that AI can be leveraged to enhance or scaffold or facilitate a process, I think that’s always a good thing. And I think if the use of AI can underscore the importance of the integrity or ethics-related outcomes for a course or a discipline, that’s great. I think there’s incredible potential for use cases, in terms of here’s how to use, here’s how not to use, here’s how this could go really well, or here’s the quickest way to work yourself out of a position, a passion, a process, whatever is appropriate there. So, I think there are examples, but they’re highly discipline-specific, and I know that that’s not the most robust answer, but I do see evidence across disciplines that folks are incorporating AI in really transparent ways that scaffold learning rather than inhibit it.

John: Joshua, do you want to take a stab at that in terms of things that you’ve seen where faculty are encouraging students to use AI in productive ways,

Rebecca: …or even how students are doing that on their own. [LAUGHTER]

Kathleen: Yeah, yeah.

Joshua: True. Yes, I have seen it. So, Kathleen shared briefly earlier that one of our questions was how many faculty members have an AI course policy, but another one of our questions that was restricted to students was, “How many of you, how many students used AI to complete their coursework?” …and that’s not meant to be AI that’s sanctioned, that is meant to be using AI in appropriate ways, assumingly, and that answer was 89%. 89% answered yes. And this is an interesting thing, because in my personal experience, the people I’ve talked to, I would say that that percentage is probably lower, maybe more around 60% but I found that answer quite interesting, and from the students that I’ve talked to, even the faculty that I’ve talked to, or other tutors, people who work with students, a lot of students are using AI in ways, for example, they may ask for help coming up with a topic idea, or maybe to brainstorm a topic, or maybe they’re brainstorming a first paragraph, or a direction that they could go within an assignment, or maybe they’re comparing scenario, and I think that those things, those getting started is one way that I’ve seen a lot of my peers using AI. Another thing that I have seen is where they use it for research. Now that will get into another topic, which is the reliability of that research, and so that’s something that would be important to talk about, but I do see students using it. I also see a lot of students using it, and they have very good reasons for that. But I think that it is something that is becoming more in use. I’ve also seen faculty members, and not in particular any faculty members that I have personally had a class with, but who I have talked to, or have actually had assignments in which they ask students to use an AI tool to begin an assignment. For example, there was an assignment in one class that was used for the creation of a resume, and one of the parts of the assignment was to take the resume and run it through an AI tool and have it give you suggestions and feedback on where you could make some words stronger, and so I think that those are ways that AI could be used in ways that are actually helpful or productive for learning.

Kathleen: If I can add one additional comment, and Josh, your comments prompted this thought. One of the more valuable pieces of feedback that we received on the guidelines was our oversight or negligence in not mentioning the library and the learning center in the guidelines, and so to your question, John, I think it’s a really good opportunity, a great idea, in fact, to leverage the use of AI to facilitate engagement with the other support services and the library on our respective campuses. So I think to build that kind of analytical info literacy related analysis of AI generated content as part of a process for a course-based assignment, I think that kind of a no brainer that we have this fodder, this generated content that we can use to engage with tutors and librarians, and of course, faculty, TAs, fellow students to kind of engage in the analysis, the fact finding, the verification of information, all of the info lit and AI enhanced info lit competencies that we’re looking to develop. I think there’s opportunities there as well.

Joshua: The other thing that I have had some students tell me is that they turn to AI because they genuinely need help thinking through something, or they struggle with a certain topic. So, for example, I know a couple of students who use grammarly because they struggle with their grammar, and I think that one thing that we are encouraging faculty to do with these guidelines, and thus also encouraging students to do these guidelines, is to have faculty make clear when these tools are allowed, and have students understand when they’re not allowed, because sometimes there’s a fine line between when is something like Grammarly, when is that allowed, or when is it allowed to use Chat GPT to think through your thought versus when is that prohibited, and I think that having those guidelines is essential, including those in your course policy is essential. The other thing that Kathleen was mentioning with the library and with the tutoring, is those are alternatives, so when AI is not permitted, there are other things that can help students. They aren’t just left on their own. There are tutors. Tutors can help you think through your ideas. There are librarians. Librarians can help find information. They’re very good at information literacy and finding those information. They can also help to verify research, and maybe if research was done via an AI tool, maybe that information needs to be verified from another more academic source. And so librarians are very helpful with that as well.

Kathleen: Now, Joshua, I kind of want to jump in, and I want to challenge one phrase that you mentioned, and I’m doing so as your thought partner in this, so you’ll appreciate the sentiments and the collegial approach here, but I would not want to take away from your comments to be that librarians and the Learning Center are alternatives to generative AI. I think that’s a dangerous framing, and I know that you don’t mean to say that they’re like alternatives to AI, but I think that distinguishing between the tool and the resource that many AI platforms offer, distinguishing between that and the human-centered expert librarian and the qualified and well-situated, well-resourced tutor, they can help us use this tool to our advantage and help us navigate the best uses, for that’s often a distinction. I wouldn’t want any of our librarian and tutor colleagues to say, “Wait a minute,” of course, if you don’t mind me clarifying what I think I know you mean by that. I think that’s important.

Joshua: I don’t mind, and I think that there are going to be times, though, when students choose not to use AI, and when they choose that they would rather have that more experienced human they can go and talk to, and I think that’s what I was trying to say, is that you don’t have to use AI to succeed in your learning. There was so long that this wasn’t available, and so I think that it’s coming available, and it’s a great tool, but it’s not the be all end all. It’s becoming a great tool, but you still need to be able to learn as a student and be able to talk to people, and I think that is really, really important.

Kathleen: Yeah, and I think going back to the spirit of our session, we have to figure out what this means together, what AI use means in our respective disciplines and our respective classes, which is absolutely how we approached this set of conversations. So, I think it may be the be all end all, it may not be. I think we got to figure that out together, which is the spirit of these guidelines, certainly, and the conversations around them. It’s important.

Rebecca: And pointing students and faculty to all the available resources and the roles that they can play in learning is always really important.

Kathleen: Yes.

Rebecca: So we always wrap up by asking: “what’s next?”

Kathleen: In terms of what’s next. I think it’s a clear next step for us to continue bringing faculty and students together, that was so elucidating. It was really great for that exchange of perspectives and for that shared exploration of this really important and impactful topic. But I think we got to keep at it. We got to keep bringing people together in thoughtful ways. That’s what’s next from my perspective.

Joshua: Yeah, I agree with that, and I think whether it’s bringing students and faculty together again or talking among students, talking among faculty, and then coming back together, regardless of how it’s done, just continuing that conversation is really the next step. You know, I agree, Kathleen, having that continued conversation is just really important.

John: We know AI has been somewhat disruptive for students and for faculty, and it’s nice to see this sort of collaboration, and it’s also something that can take place really nicely at the class level as well as at the institutional level. And it sounds like you are making some really good progress there at GCC.

Kathleen: We’re really trying. Thanks so much to you both. Thank you.

Joshua: Thank you. It’s been wonderful to be here to talk with him today.

Rebecca: Yeah, it’s great to have both of your perspectives, because I think the model you’re sharing around the topic of AI could be applied to many other challenging topics that challenge both faculty and students, and bringing those two groups together to work through those challenges and establish guidelines and other strategies is really helpful.

Kathleen: Thank you. I think so too. One of the goals is to model. We’re trying to model as a campus community for each other, whether you’re a faculty, staff, student. We’re trying to model how to problem solve and how to figure things out together and do so in a way that’s ethically sound and mission-centered, and focused on learning. So, we’re trying, we’re hopeful, but we’ll get through it together.

John: And when both students and faculty are involved in making these policies and have a say, there’s going to be a lot more buy-in to the policies than if they were just imposed.

Kathleen: That’s right. We agree.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

449. Student Access and Earnings

The “One Big, Beautiful Bill” contains a provision eliminating federal loan availability for programs in which past graduates had early career earnings that fall below a designated threshold. In this episode, Debbie Furlong joins us to discuss the statistical measures used to determine federal loan eligibility.

Debbie has served for over 30 years doing applied higher education policy analysis at public comprehensive universities, most recently as the Director of Institutional Research at SUNY Oswego. Prior to her arrival at Oswego, she worked and taught at the University of Wisconsin at Green Bay, St. Norbert College in Wisconsin, and the American University in Washington, DC, where she had completed her PhD degree in international relations.

Show Notes

Transcript

John: The “One Big, Beautiful Bill” contains a provision eliminating federal loan availability for programs in which past graduates had early career earnings that fall below a designated threshold. In this episode, we explore the statistical measures used to determine federal loan eligibility.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Debbie Furlong. She has served for over 30 years doing applied higher education policy analysis at public comprehensive universities, most recently as the Director of Institutional Research at SUNY Oswego. Prior to her arrival at Oswego, she worked and taught at the University of Wisconsin at Green Bay, St. Norbert College in Wisconsin, and the American University in Washington, DC, where she had completed her PhD degree in international relations. Welcome, Debbie.

Debbie: Hi.

John: Thanks for joining us. Today’s teas are:… Debbie, are you drinking any tea today?

Debbie: I am waiting for my tea to cool off. It’s a decaffeinated green tea, because it’s too late in the day for me to have any caffeine.

Rebecca: Green tea is a nice choice at this time of day. I have Blue Sapphire today.

John: Back to an old standby, and I have ginger peach green tea today.

Debbie: I’ve had a ginger peach tea. It’s some of my favorite flavors.

John: It’s one of mine too. This is a Republic of Tea variant. Rebecca particularly likes flavored teas.

Rebecca: It always smells good. [LAUGHTER]

Debbie: Yes, it does.

Rebecca: We’ve invited you here today to discuss the growing focus by the federal government on using earnings data on students who evaluate the success of college programs. The One Big Beautiful Bill Act prohibits universities from awarding federal loans to students in programs where its former students have earned low wages. Can you tell us a bit about how we got to this point?

Debbie: Yes, it’s been a long trip down. Some might say the government has been providing post-enrollment earnings data for several years now through a tool called a scorecard. Assessing the quality and value of a college degree in terms of students’ earnings has been growing in popularity for a long time. Increasingly, we’ve all talked in higher ed about people viewing their degree in terms of what it’s done for me as a private good, as opposed to focusing more on the public good of education. We can see data where this trend has been growing for 50 years. I think that trend was really exacerbated when public universities, when states started having to trim public funding, and parents and families were bearing more and more of the cost, and when the price, the stated prices were going up, up, up, this trend became even more predominant. Universities really have not helped at this, because, like, our marketing areas for recruiting have really leaned into the job placement as something that needs to be touted as an important thing, so we’re all sort of involved in telling students they should pay attention to how much they’re going to earn. I think the original interest of the federal government – I’m just speculating… based on timing and the administration that was in play at the time we first got the scorecard… I think the original interest was sort of a consumer protection kind of position. We had had an explosion of for-profit schools that really had some very serious problems. Students were not completing the degrees, and even when they were, they were unable to earn enough to pay back their loans, and the loan default rates were going up for these schools. So it was like, “Well, let’s protect the consumers,” so about 10 years ago, when Obama was actually still in, the Department of Ed launched the College Scorecard, and the College Scorecard was really the first time that families and students could look at a university’s profile and see: this is the graduation rate, this is how much it cost, this is what percent of students get loans, and here’s the average income a certain number of years after graduation. And here’s the percent of students that are paying back loans, so, bringing it all together. Where we’re at right now, the new state that we’re in right now is the Department of Ed looking and saying, “Gosh, we have all this data. Let’s begin using it to stop supporting programs where students are not earning enough money. Since we have all the data, we should protect not just the consumer, but protect the taxpayer. The taxpayers shouldn’t be asked to pay through Pell grants and through loans for programs where students aren’t even able to earn as much as they could earn if they went and got a high school degree.” That’s how we got where we are, and that’s where we’re at. I hope you can hear some of the passion of my voice about this stuff. [LAUGHTER] I really think faculty and staff need to understand where the data come from and understand how we’re all in this together, and to feel confident to go and explore and learn on their own.

John: So, this started off as something designed to protect consumers, but it’s moved on to something a little bit bigger, and because of that, it’s become increasingly important to understand the data, and you spent a fair amount of time researching the data and how it’s measured, and you’ve raised some concerns with it. Could you talk a little bit about the primary sources of data first?,

Debbie: Yeah, there’s two really big national data sets. One is from the Department of Education, and one is provided by a group called the Carnegie Classification, which is a branch of the Carnegie Foundation, which is a nonprofit. The Carnegie Foundation has teamed up with the American Council on Education, ACE, which is a big nonprofit player in the higher ed sphere. Both of these organizations are providing data that is publicly accessible and downloadable, and they both have a website search engine that sits on top that enables you to look up one school or to compare a few schools, but the data that’s running those engines is fully accessible. So this winter I was bored, and I got in, and I started playing. It’s like I just really want to learn how they’re going to do all this, and what does this stuff all mean. So the College Scorecard is the most important resource. Like I said earlier, the Obama administration really launched the scorecard, and this was the first time that consumers could see, for one institution, all the stuff about debt and debt repayment and post-enrollment earnings, and they could see those metrics right next to the data about the tuition and the costs. Department of Ed could not have done this without technology advancing to the point that we could really wrap our minds around really truly big data sets. This downloadable data set I just referred to has over 3500 variables, so it’s not for the faint at heart. In order to measure graduates’ earnings, or the earnings of formerly enrolled students, because both groups are looked at. In order to do that, you got to have two things: you have to have tax records that indicate was the person employed, were they employed full time, and what was their income, their earnings, and you have to have something with a matching field that says what school did this student go to, did they graduate, and if they graduated, what field? No single university has this. So, the way this works is the Department of Ed has the student identity information from its loan and grant recipients, and all the earnings data is only measured for students who have taken out federal financial aid, either through a loan or by getting a Pell Grant. So, the Department of Ed has these files that have individual identification and information about whether the student graduated and what field. They send these files over to the Treasury Department. The Treasury Department does a merge and match with their records and sends the information back to the Department of Ed. As a little sidebar, we’re going to be talking in more and more depth today about this, and the fact that we’re limited to just looking at loan and grant recipients is a significant limitation. The sidebar I’ll drop in right now is I am surprised the Department of Education has not yet asked all of us to provide full identification records for all of our enrolled students. The reason I’m surprised they haven’t yet done this, there’s two things, [LAUGHTER] the first one is the Department of Ed did ask for student identified records earlier this winter when they started asking for information about access, information so that they could make sure that students who are black or low income were not being admitted when they had lower test scores and high school grades. The data file that the Department of Ed requested this winter is student identified, so they’ve gotten over their concern about privacy issues. [LAUGHTER] The second reason I’m surprised is they’ve been asking states to provide voter record identification, so I think it’s only a matter of time. The other reason, there is another organization out there that’s not really related to this topic we’re talking about, the National Student Loan Clearinghouse, or National Student Clearinghouse. The NSC has student-identified data, and I don’t see how the Department of Ed is going to feel too happy knowing somebody else has it and they don’t have it. So, I think it’s only a matter of time before that little limitation goes away. So, the Department of Ed sends these files over to Treasury, they get back the earnings data. The second data set out there has used the Scorecard file, and this other data set is the . I’m going to be referring to this as access and earnings, or more just Carnegie, the Carnegie framework. The Carnegie framework about access and earnings was released about a year ago, but it was in development for quite a bit longer. Carnegie’s real contribution that’s relevant for what we’re talking about today is they began trying to deal with the fact that in the scorecard the only earnings that people were getting compared to were national, and more recently, statewide data, and Carnegie’s framework argues that earnings for a school’s students should be compared to the earnings of high school graduates in the geographic areas where those students come from, not necessarily to the whole nation, and not necessarily even just to the state where the school is located. This issue about adjusting the earnings to reflect regional differences in incomes is definitely confounding the Department of Ed right now. The OBBBA is in negotiation. There is a negotiated rule making going on right now, and one of the last things that the Department of Ed is trying to figure out how to deal with before they roll out this framework, where if your program’s earnings are too low, you can’t get loans. The last issue they are mucking with is the regional differences in earnings, and I don’t know how they’re going to do it. The Carnegie folks did it one way and sort of got the ball rolling. Carnegie’s real contribution here, I think, is showing how incredibly difficult this is going to be. We’ll have to see, we’ll have to see how they do it, and it may take a few years before they really figure out how to most appropriately set a benchmark for earnings for every program in every school.

John: So, are you saying that the earnings of people in rural Alabama might be different than Manhattan? [LAUGHTER] Who would have thought?

Debbie: They are. Yes.

John: Among the other problems there that you’ve noted is that there’s a fair amount of sample selection in the sample that they’re working with, and you’ve already referred to one aspect of that. Could you talk a little bit more about that, and how that might affect the validity of the earnings measures?

Debbie: Yeah, yeah, so what John just referred to is the fact that only students who get federal financial aid are currently included, and for many of us that doesn’t seem like very much of a limitation, because so many of our students have to get loans and a huge percent of students get Pell grants, but there are a bunch of schools out there where those percentages are really, really small, and I went and I looked up Harvard’s data, just because everybody knows Harvard, you know, Harvard, Harvard. And I went and looked at Harvard because I anticipated Harvard students would not need to rely a lot on federal aid, not because I think Harvard students are rich or poor, I don’t particularly know, but Harvard is very, very wealthy, and they’re able to provide a lot of institutional aid, which keeps those students out of this federal tracking, tying together with your Treasury data. So I looked up Harvard. In a recent year, 20% of their students got Pell grants, and 4% of their students took out loans. So by the time that cohort gets through the system, only a quarter of the students would have their wages reflected in these median wages. I don’t really know what this is going to mean. Is this good? Is it bad? It makes me feel uncomfortable about the validity of the data. Another thing that I know with this coverage, I have been watching that the percent of students in the United States agreeing to take out loans fell after COVID and did not rebound. I think this past year they have started rebounding a little bit. What this means is that the coverage overall, nationally, school after school after school, this coverage rate is going to be going down for the next several years. So, this is another reason why I am expecting the Department of Ed to be knocking on everyone’s doors very soon, saying: “The gig is up. We want a student-identified file. We want to know everyone who’s enrolled every semester. Here’s the portal. Drop the data.” Another issue that kind of deals with coverage that gets very confusing has to do with the timing, and this is kind of devastating to me, is a validity issue. And what I mean by timing is, how long do you wait after a student has left the institution, or after a student has graduated. How long do you wait before you go and grab their earnings data? Okay, so timing issues can be very confusing, and what I would really encourage anybody who’s paying attention to this stuff, don’t try to sink into your mind exactly how these things are measured. When you go to one of the websites, look for the information notes, they will tell you exactly what they’re measuring. Carnegie’s Access and Earnings framework originally relied on a measure… they looked at students’ earnings eight years after the student initially enrolled, and there are reasons why they made that choice. One of the things that higher ed has been deeply criticized about is the poor retention of students after the first year, and there’s a huge amount of tax money that goes to students who are at the institution for one year, and then they never really do anything with that education. So looking at students based on where they started is not stupid, but it is kind of misleading. I think when people say, “Oh, here’s Cornell’s earnings data,” I think people assume that’s the earnings data of students who graduated from Cornell [LAUGHTER] or from whatever school, and the current version of the scorecard has flipped over and is relying almost exclusively on earnings data measured four years after the student graduated. So the cohorts are now measuring students who actually completed the degree, which is probably more valid for what people actually think they’re looking at, and they give students four years, so now it’s like, okay, four years… is four years long enough to get a valid measure of a student’s earnings potential? I think this is a huge limitation. Many of the highest earning fields, and I’m thinking like medicine and law, those are going to be either really hard or downright impossible for a student to complete the whole thing and be out and earning. So a lot of fields where students go on to complete masters and doctorates, those students are ultimately going to be the highest earners, but they’re not going to be earning that stuff four years out, so I was thinking about, like, where I was four years after my undergraduate degree. I had gotten my first full-time job, but it was just a job to pay the bills, because I was working on a PhD. So, was I earning more than a high school student? Maybe. I don’t know. I was mostly a graduate assistant, getting paid a graduate fellowship. Even fields where students don’t go on for advanced degrees, I think that there’s a lot of research that indicates people who complete a baccalaureate degree will ultimately have higher lifetime earnings, even if they don’t get an advanced degree. We know this lifetime earning data exists, so is four years long enough to predict the lifetime earnings of a particular person in a particular field of study from a particular university? I think it’s a big validity question. The last thing that I wanted to mention is this timing, all this stuff about the validity of the timing of when the data are grabbed, it’s grabbing data right smack at an important time for young families that are deciding to have kids. I looked up before coming today, because I knew I wanted to make this point, because I think it’s important, the average age for women in the United States who have had children at this time, the average age of when they had their first baby was 26 that’s just right in this four years… you graduate when you’re 21 or 22 So I mentioned this because it’s very common for young families, for either the husband or the wife, the mom or the dad, or one parent, to back off a little bit from working full time or working at the high pressure job. So there’s a lot of problems with the data.

John: Going back to what’s referred to as the Mincer equation, which was first estimated back in the early 1970s. economists have been estimating the overtaking age, the age at which the earnings of college graduates overtake the earnings of high school graduates with equivalent demographic and ability characteristics, and back then was about seven years after graduation. More recent studies suggest that it varies a fair amount by discipline and by occupation, but it’s generally between six and eight years after graduation from college, which means on average the fact that people are earning more than high school graduates puts them way above where you would expect them to be had they not gone to college, or people with similar characteristics would receive had they not gone to college. And so it’s really troubling that they came up with a metric that’s so evidence-free, basically.

Debbie: Yes, [LAUGHTER] problematic.

John: And also, you mentioned law school graduates, so they might be a year out of law school, but some of the people who come out of law school are going to be working with large firms for ridiculously low pay, and those are the people who are going to have the highest rate of growth of earnings. Other people will start their own practice and will have higher starting salaries than those who go for those large law firms, at least in terms of hourly wages, they’ll be paid a lot less, but they will end up with a slower rate of growth of earnings, and there’s another classic economic study back during this period on on-the-job training and the fine structure of earnings, which notes that holding other things constant, people in the same occupation with lower starting salaries tend to have higher rates of earning growth over time, because part of what they’re picking up on the job are some skills that will improve their productivity, and they’re essentially accepting lower wages during those times in return for the increase in training or human capital development they get on the job, which is certainly true in medical school, because then they’re starting their residencies, where they’re paid again ridiculously low amounts, but they end up with much higher earnings just a few years later.

Rebecca: I knew John would be passionate. [LAUGHTER]

Debbie: Yes. And everything he said is true today, the research is still valid and important.

Rebecca: One of the other things that you mentioned is those regional variations that Carnegie was developing, or kind of initiated. Can you talk a little bit about these regional variations, and like, what does their framework mean? How did they do it? How maybe should they do it? [LAUGHTER]

Debbie: Yeah, so like I said, when they first put the scorecard out, my perception is that was kind of a consumer protection thing, like here’s the tuition, here’s the percent of students who have loans, here’s the percent of students who have paid back their loans, and here’s the average amount that their graduates earn, without judging whether or not that average was adequate or met a benchmark. Well, those days are behind us. The OBBBA wants to establish this benchmark of acceptable earnings, and the negotiation is moving forward. It seems very clear that they’re going to be using this high school degree population as the benchmark, but obviously people earn different wages based on different regions and different wages based on whether or not you’re in an urban area or a rural area. So the Carnegie framework is the first place I’ve seen where they tried to take this on. So let me tell you how they did it. A little segue, when Carnegie adjusted for regional variation, they have also adjusted for variances based on race and ethnicity. Carnegie’s argument was, if a student goes to a certain university, when they get done, that student should earn as much as high school graduates matched by the racial and ethnic distribution of the service area of that university and weighted to reflect the earnings differences of different racial groups, so they did this racial weighting. I’m not going to talk too much about that, because the OBBBA, the language I have seen, specifically says that the earnings benchmark cannot be adjusted for race. They want a race-neutral approach, just like they want race-neutral admissions. This is part of what we’re going through and working on with the Department of Ed right now. So let me talk about how they do the regional stuff. What Carnegie looked at, they said, “Okay, let’s look at where the entering class of a university is coming from.” The Department of Ed, for 20 or 30 years, has collected this data. It’s called a migration file. In the migration file every second fall, you’re not required to submit migration data every year… I don’t know why not… but in the file, the institutional researchers at the university have to go in and look and say, “Okay, how many people are in my freshman class, and where did they come from? How many of them came from New York State. How many of them came from Pennsylvania?” And the file goes through all of the states, so that data exists in IPEDS. IPEDS is the federal government’s collection of data about colleges and universities, and it’s a massive data set. So Carnegie said, “Well, gosh, since IPEDS has this migration data, let’s use that to figure out this concept of a service area.” What is the service region of this university? So, if a university has 90% of its students are coming from New York and 10% are coming from Pennsylvania, its earnings should be similar to the high school graduates of New York, weighted a little bit to reflect the fact that people from Pennsylvania have lower earnings. Now, so a 90%, 10%. In the Carnegie framework, it takes the top three states, so it says, okay, school X. If you had 50% of your students from one state and 40% from a second state and 5% from a third state, your earnings benchmark will be the weighted average of those elements, and then the remainder, the last little bit, we’ll just use the national data. This sounds kind of reasonable, I guess. Carnegie was very careful developing this. They got feedback, they got pushback from the two year schools saying, “Well, wait a minute, everyone at our university comes from one state, but they don’t come from all over the state, they all come from my county, or these two or three counties, because we have a lot of two-year schools out in rural areas.” This is a really serious issue, because the state earnings levels are very influenced by what’s happening in the big cities, and rural incomes tend to be much lower than urban incomes. So, Carnegie said, “Well, this is a reasonable complaint, so we’re going to back off, and we’re not going to use your state, where you have most people from. We’re going to use this core-based statistical area to gather your data.” There are a couple of other problems before we really get into the CBSA, and what that is. There are a couple of other things about relying just on the migration data that concerned me. My professional background was all in public comprehensive universities, and every place I ever worked had a huge pool of students who transferred into my universities, so I know transfer students may or may not be accurately reflected in this migration data in the schools where I’ve worked. In the systems where I’ve worked transfer students had a very different geographic distribution than new freshmen, and I think this is true for a lot of public comprehensive universities that the freshman class tends to come from all over the state much more, but the transfer students tend to be students who started at a nearby community college, and now they are your own community students, so that migration data that we’re using may not really be a great fit. The other thing that’s problematic is we are just assuming that the geographic distribution of graduates is the same as the geographic distribution of new entering freshmen. It may be, it may not be, and it may or may not matter, but these kinds of assumptions that are baked into the method, I think, cause people to feel a sense of discomfort about really feeling confident that the data really are measuring what we think they’re measuring,

Rebecca: So Debbie, I have another question. It seems like most of the data that you’re talking about is really focused on the states, so not necessarily international students and entirely undergraduates.

Debbie: Yes, and no, all of the earnings data is for US citizens only, and I will remind you that in order to have a student’s earnings get studied, they have to have gotten federal financial aid. International students are not eligible for Pell grants and for federal loans. So even if a former international student got OPT training and landed a job and became naturalized, we would have no way of knowing… the Department of Ed would have no way of knowing to tell the Treasury Department, “send me that person’s salary and earning data.” So universities that have huge international student bodies, especially if they’re doing really well, and supplying really fabulous students into high-need areas here in the United States. All of that is just kind of a loss. Now, your other question, like “Debbie, all you’re doing is talking about undergrad, undergrad, undergrad. What about graduate students?” The research that I really focused on this winter was about this Carnegie Access and Earnings framework, which is exclusively about the undergraduate experience. The scorecard data does include earnings data for graduate students, and the OBB BA work going on right now is also going to be relevant for graduate students. Graduate students are eligible to get federal loans in certain circumstances, and those loans will be cut off if a student gets a graduate degree from a school where their program is not earning on average as much as baccalaureate earners in whatever state or region the OBBA decides to look at, so it’s the same kind of framework, notched up one level.

Rebecca: The other thing that I was curious about is the data that you were talking about, is where students are coming from initially, which may not be where they are working.

Debbie: Yeah.

Rebecca: Am I right about that? [LAUGHTER]

Debbie: Oh, yeah. [LAUGHTER] And I’ve got a bunch of notes about this thing, and I definitely wanted to hit this. So you’re jumping into something that I already also agree is really a problem. Two issues: do students go back to where they came from? No. The other thing that’s really problematic, I think, is that the earnings data are all aggregated on the basis of what the student studied, not what they’re actually doing. So, I was thinking in my head, like examples, I know your field is art and design, design arts. Someone who studies design arts and graduates with an art degree and goes and does marketing in New York City is going to have a very different earnings experience than someone who completes an art degree at the same school and stays in that local area and runs art programs for the YMCA. These are both wonderful jobs, both jobs that would require an art degree to be really effective, but they’re incredibly different fields. I mean, one is marketing and the other one is social services, and that’s not accounted for. So, what a person’s actually doing with their degree is not part of the mix.

John: You talked a little bit about these core-based regions that are used for measuring geographic effects. What are they, and could you tell us a bit more about how they’re defined, and whether there’s some problems with the use of those.

Debbie: Yeah, the two year schools kind of pushed back and said statewide data doesn’t make any sense for us. So the Carnegie Foundation said, “Okay, the regional adjustment for two-year schools, we should use the data from the closest geographic region to their service area.” The Census Bureau has these geographic entities called core-based statistical areas. A CBSA is a geographic area that contains a large urban center, and then the surrounding areas that are kind of integrated economically with that urban center. Like, I live in Oswego, New York, which is about 30 miles north of Syracuse. I’m in the Syracuse CBSA, which makes perfect sense. There’s so much stuff that we depend on Syracuse for, and people go back and forth there all the time. So, Carnegie said, “Okay, let’s use core-based statistical areas for the two-year schools, and let’s also use them for these public regional universities.” There’s this category of universities called a regional public university, or an RPU, which is most public universities that are not focused on research. So, in New York, Stony Brook, and Albany and SUNY Buffalo, those are all research-focused institutions, they’re not RPUs. Almost all of the other four-year public universities in New York are considered RPUs. So, Carnegie said, “Well, they’re called regional, they must be serving their region. Let’s do all of our benchmarking for them based on the core-based statistical area.” Well, I really dug into this with my research, because the truth of the matter is most of these regional public universities are not recruiting heavily from our own regions. At most, RPUs would have maybe 20 to 30% of the students from their region. We’re all recruiting heavily in the big urban areas, because that’s where the students are. This is the whole piece of this enrollment cliff. Not only is certain states and regions losing students, but the students who are left are in the cities, and if we want to maintain universities out in the smaller communities, we’ve got to move the students out into these smaller communities. So I did some actual sensitivity analysis to see, does it matter? Does it impact the outcomes for these RPUs in the Carnegie world if you change from looking at them through a core-based statistical area lens, or using just their regular state lens?” And there’s a pretty big impact. It really can make kind of a big difference. One of the other things I looked at, just to sort of confirm for myself that this is kind of questionable, is I was thinking all these RPUs have… not all of them, but many of them… have huge dormitories, [LAUGHTER] and we have the ability to house huge percentages of our student bodies. So, while most two-year schools don’t have dorms, and the students have to come from right there in that community, that is just not true for the RPUs. I did a lot of work looking at the impact of this flip between CBSA and state data on the metrics measuring diversity. I didn’t do as much looking at the income piece, but it’s definitely a problem. Most RPUs are not getting the bulk of their students from their CBSA, and yet their benchmark salary is being determined, in very large matter, by the incomes in that CBSA. Is this good? Is it bad? I don’t know. It is a validity question, because we’re measuring something, thinking it means one thing when it really might not.

Rebecca: One of the other metrics that you were talking about is how the Department of Ed is comparing to median salaries of high school graduates. Can you talk a little bit about how they’re doing that, and what concerns you might have around that, in terms of like how they’re benchmarking which programs get loans.

Debbie: I’ve been speaking mostly about how the Carnegie made adjustments in order to accommodate regional variation. I want to make it clear at this point that the Carnegie framework did not consider field of study, which is where I think you were kind of starting to head, Rebecca, because it’s the field of study that the Department of Ed wants to use to cut loans off. They are not suggesting cutting off loans and grants to entire universities, although that is a possibility if a university has a really large proportion of its programs failing to meet the benchmark, then they could talk. I don’t know. But it really is this program level data. So, my understanding is that the Department of Ed, right now, is trying to figure out how to do some regional salary adjustments that may or may not look like how Carnegie did it, but the OBBBA data set is going to be doing these regional adjustments on salaries aggregated to reflect program of study, so you can go in right now to the scorecard, and you can go into fields of study, and for fields of study that have an adequate number of graduates, you can see what is the median earning for those graduates from that program four years later. On the same page, it has a little bar that says here is the average earnings of high school students in the United States at this point in time, and then there’s a bar right beneath that that said, and here’s the earnings for this school’s state. So right now OBBBA would be leaning on a data set, or the scorecard is saying here’s the field of study, but everybody gets matched up to the same national high school graduate earning data, or state high school earning data, and what I mean by high school earning data, it’s earnings of people who are employed full time in that year whose highest degree is a high school diploma.

Rebecca: So basically the numbers are comparing someone who studied this particular topic with anyone who’s in any job with a high school degree [LAUGHTER] in any field. [LAUGHTER]

Debbie: Yes, yes, and you know something that I haven’t really thought too much about, but it’s one of those things that I think I think about it in the middle of the night. The federal government right now is spending a lot of time and effort to make more federal funding available for short- term academic programs, which is like technical training. I feel like the federal government is working hard to improve the salary and earnings [LAUGHTER] of people with a high school degree, you go and get a certificate in a particular field, which is great, and I completely support this, but these high school earnings figures, I think, are going to be coming up. If the federal government is successful, the benchmark that they’re going to be comparing the four-year schools to is going to be going up more. I’m not really sure that’s going to happen, because I think earnings are really more a reflection of the labor market than they are a reflection, particularly of field of study and variances between different programs. Another problem, though, that I really want to talk about, and I hope this doesn’t give you heebie jeebies, Rebecca, because I know you’re more of a qualitative person and less of a quantitative person, but all of this benchmarking is going to be made based on comparing median earnings of a group of people to median earnings of a larger group, the high school comparison, so we really have to ask the question, we hot to dig back and think, okay, what is a median, or even what is an average, and what does it mean to enact a policy where we treat an entire group of people in a particular way based on the average of a group that that person happens to be a member of. This is a bigger problem, really, for Carnegie, because they were putting people in groups left and right, but this is a problem. This fact that the median is not necessarily particularly representative, because there are people who have much higher salaries and people who have much lower salaries. So, how fair is it to treat everybody in a program one way because of a median value, there’s two real issues here. The first is the issue of the cutoff value. I’m trying to think of a really good example to illustrate the cutoff value problem. In higher education, it’s very common for universities to say “everybody with a high school GPA of 3.5 and above is automatically admitted, directly admitted.” The cutoff problem is that there’s going to be a lot of people who have a 3.49 who have very, very good reasons for arguing that their 3.49 justifies their admission to this university to much greater extent than some other people that have a 3.5. Maybe that person went to a very exclusive, difficult high school, maybe the person began high school speaking a foreign language, maybe the person had trauma in their last year, all that stuff, and we all know colleges have these vast mechanisms for making exceptions that are required because of the cutoff value problem. Whenever you’re going to take a quantitative element and put people in groups, you’re going to have a cutoff problem. It’s like for the OBBBA, the cutoff problem is there’s going to be programs where the median high school salary is only $1 or two or three, it’s very, very close to the median for this group of students, for whatever reason. And that median could be low for a lot of reasons, maybe it’s low because that program has a bunch of students in grad school, maybe it’s low because those people all decided to take a mission year and they’re just now starting their careers. Whatever it is, you’re going to have people really close to the cutoff value. The second numeric problem, you have to think back when you were in stats class and they made you learn all about the standard deviation and the variance around the midpoint, this is a huge, huge problem. So, just to illustrate this, if you think about a couple of universities, and you think about business school graduates, you could have one university where the average business school grad earns 50,000 and another one where the average earns 40,000 so there’s a $10,000 difference between those two. Well, I can guarantee you, if you looked at either one of those, at the 40,000 you would find a lot of students earning over 50 and even over 60,000 and you would find lots of students earning 30,000 and below. The definition of a median is that half the people are above and half the people are below. There’s been a lot of research in higher education, specifically the research I’m most familiar with was done by the National Survey of Student Engagement, the NSSE. About 30 years ago, the NSSE started collecting student experience data, and very quickly they started putting out data showing that the variance of student experiences within any single school is vastly greater than the variance of student experiences from one school to another. This is just by definition, this has to be true in this earnings data. S o we’re very likely going to be eliminating programs that could have served students where we would have predicted that at least half of those students would have done just fine in the earnings metric. Now the scorecard has a ton of data about this variance measures. For every single program, they show the standard deviation, they provide a mean, they provide a median, and they provide the quartile values at the third quartile and the first, so that you can measure and see how wide is the gap within which 50% of students fall. The fact that they’re providing all this variance data tells me the people at the Department of Ed deeply understand the importance of variance, but deeply understanding it mathematically doesn’t explain to me how they’re going to accommodate for the variance when they start deciding which programs get cut off, because they have not talked about that at all. So there’s a lot of stuff really coming out here, [LAUGHTER] a lot of questions still on the table.

John: And you mentioned the cut off value, and how it’s somewhat arbitrary, and it’s very precise, though, they come up with a specific value, and yet so many of the benefits of education are not measured in salary. People with college degrees have lower unemployment rates, they have longer lifespans, they have higher quality of life, as measured by all sorts of happiness surveys. They have lower probabilities of divorce, they’re less likely to commit suicide, they’re less likely to commit crime, and so there are many aspects of jobs and the returns to education that are just completely being ignored by this focus on one specific value, and that’s kind of troubling.

Debbie: It is troubling, and it’s very, very sad. I hope that we can eventually, as a country, return to a time where we value learning and personal growth. I think our labor market is kind of not really supporting all of our people particularly well right now. I would like to live in a world where if someone is called to ministry, that they don’t have to worry [LAUGHTER] that they can’t go and get a degree that they can afford to pay for, because being a minister is not going to pay very much, or if a person is called to be a pianist, concert pianists don’t always earn a lot of money. Does that mean we taxpayers, with our taxpayer support, that we shouldn’t support that person’s education? I don’t want to live in a world that doesn’t have concert pianists who have been supported all the way through getting a Doctor of Music Arts, because the people who have those degrees are amazing in the way they connect our world to our history through music, and I’m picking on fields that are often sort of the punching boys for “don’t study theater, you’ll never make an income.”

Rebecca: But these are the fields that often contribute a lot to a community, and these measures don’t actually have anything to do with community benefit or the public good.

Debbie: Yep, the public good.

Rebecca: So, on that note, [LAUGHTER] we always end by asking, what’s next?

Debbie: I’m so glad that you are ending with that, because I think there’s no question we will be… we in higher education… will be controlled by these student earnings data for a while. This final OBBBA ruling is supposed to come out of its last session of a negotiated rule making at the end of May, we’re sitting here, May of 2026, we are speaking about this as it’s happening. If you’re starting to pay attention to this, or you’re looking for information about it, I want to enable you to learn more and be empowered. So, really, what’s your favorite publication for higher ed stuff? Is it the Chronicle of Higher Ed, is it Inside HigherEd or even The New York Times and will be covering this information. What you’re looking for is, they might speak about this ruling as financial value transparency work. The old language, which some of us are still kind of stuck in, we talked about this as gainful employment regulations. I have just started seeing reference to a new acronym, STATS, which stands for Student Tuition and Transparency System. I’m not sure, but I think that stats may be how the Department of Ed plans to rebrand the Scorecard, so that when they bring it out as the basic thing undergirding this new regulation of limiting students’ access to loans and Pell grants, that STATS is what we’re going to be looking at. So those are some keywords to look for and listen for, and the last thing is wherever you are, figure out who your institutional researchers are. Make sure that they are pulling up these data files. If you Google Scorecard and higher education, if you Google access and earnings, or if you look for Carnegie classification, all of these will take you to these websites where you can look for the data and download the data and learn about its structure. So don’t wait, be empowered. The federal government has a lot of power in this conversation right now, because they control the data, but we can at least begin to educate ourselves, which is why I wanted to come and talk about this, and I really appreciate that you guys gave me the opportunity, and you ask me great questions, and you listen to my answers, and I hope you’ve learned some about how the data are put together and what we’re going to be looking for in the next couple of months.

John: So one implication of all this is that, in terms of admissions decisions, would this whole incentive system encourage, perhaps, colleges to only accept white males from higher income households because they end up with higher salaries on average.

Debbie: I don’t think we’re going to go there in higher ed. What I think is more likely is an acceleration of what we’re already seeing. I have a daughter in school in Indiana, and she keeps me apprised of what they’re doing out there, so I pay attention. I think many states have used the OBBBA work as a reason to go through and cull their programs, not particularly on the basis of earnings, but to get rid of all the small programs, and these two things tend to go together. They just do. So, I know Indiana has chopped a ton of programs. Universities are eliminating small programs left and right under an argument that they’re saving money, that they don’t have enough resources to maintain these programs. Any university with an IR person worth their salary is making sure that universities do not eliminate small programs that have very high earnings. I worked at a university that had a really small engineering program. They’re not going to eliminate that program, [LAUGHTER] even though it’s really hard and expensive to run, but those students do earn really high salaries. I think, getting to your point, John, universities are so desperate for tuition revenue, I do not think that they are going to filter for the potential future earnings of white men. Some might, but I don’t know. I’ll do that research four years from now, and I’ll come back and let you know how it’s going. [LAUGHTER]

John: It’s just one of the worst possible outcomes of the incentive system that they’ve devised.

Rebecca: I think it’s really helpful for faculty to be more apprised of this information, because it’s kind of zooming past us as policy and regulations are changing so swiftly, and so having this explainer is very helpful. [LAUGHTER]

John: Yes, thank you. This was really good and it’s things that I think most faculty are not that aware of right now.

Debbie: I think people are viewing it as something that is happening to us, and as I close, saying yes, that’s kind of it, it is happening to us. There may eventually come a time when we can speak back and push to recreate a more equitable, more reasonable understanding that incorporates more of the community values and the public values that we get from higher ed.

John: And we are lucky to be in a state where there is much more support…

Debbie: Oh, yeah…

John: …for the positive aspects of education.

Rebecca: Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on Apple Podcasts or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing Assistance provided by Madison Lee.

[MUSIC]