312. Alice: Finding Wonderland

Many of our disciplines are unfamiliar to students until their first encounter in an introductory course. In this episode, Rameen Mohammadi joins us to discuss his first-year course that introduces students to computer science using an approachable hands-on experience.

Show Notes

Transcript

John: Many of our disciplines are unfamiliar to students until their first encounter in an introductory course. In this episode, we look at a first-year course that introduces students to computer science using an approachable hands-on experience.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Rameen Mohammadi. Rameen is the Associate Provost at SUNY Oswego and an Associate Professor of Computer Science. Welcome, Rameen.

Rameen: Thank you. Thank you both.

John: It’s good to have you here. Today, for the first time ever, we’re all drinking the same tea, which is Chimps tea, a black tea with honey flavor, fig, and thyme. And this is a gift from one of our listeners. Miriam in France. Thank you, Myriam.

Rebecca: Yeah, so far. It’s really tasty.

Rameen: Yeah, really excellent tea. We love it.

Rebecca: So we invited you here today, Rameen, to discuss your First-Year Seminar course that combines animation and storytelling using the Alice 3 programming environment. Before we discuss that, though, could you provide an overview of the goal of first-year seminar courses at Oswego?

Rameen: This is not a standard first-year seminar. First-year seminar courses are designed to extend orientation, familiarize students with resources, and things like that. Our perspective about this type of course, which we call signature courses at SUNY Oswego, is that you are welcoming students to the intellectual community that we have. So as first-year students we desire a number of outcomes to be met by these courses, one of them is critical thinking, they have to have a significant critical thinking component. Also, these courses need to have both writing and oral communication embedded in them. And one of my favorites is that they have to enhance the cultural competency of our students, we’re a very diverse student body, there’s quite a bit of opportunity to make sure students experience other perspectives. And I think courses of this type really need to address that. Our provost, Scott Furlong, brought the idea to us even during his interview at SUNY Oswego, about what they had called, at his previous institution, passion courses. Now, as I said, we call them signature courses here. But those of us who love our discipline certainly can understand when somebody uses the term passion. So what makes you excited about your discipline? That’s what the course should help students experience.

John: So, you’re using the Alice 3 programming environment. Could you talk a little bit about the types of things that students are going to be doing in the class?

Rameen: So Alice 3 is a VR programming environment. So what you do is you build a scene, you can bring avatars of various types, could be people, could be aliens, could be dogs, into a scene, and you have props, trees, mountains, buildings, that you could bring into a scene, and then you learn to program something. So they can talk to each other, they can move from point A to point B. And it actually turns out, they’re able to, and they will be, writing reactive programming, which typically is what we do when we design games. So the user acts in some way, and then you program the reaction in the VR world in that context, or things run into each other. And obviously, when you’re designing games cars, or other things may run into each other, and you have the ability to detect that and actually act on that. But at this point, they are already running about a month ahead of where I thought they could be in just about a month of the semester. So I’m really hoping we can get that far.

Rebecca: Can you talk a little bit about why you chose the Alice platform, and what you were really hoping to foster with students.

Rameen: So, just a bit of background about Alice, Alice is supported by researchers in Carnegie Mellon. I think Randy Paudsch, when he was at University of Virginia, is really the person that began the innovation with Alice. And he thern moved to Carnegie Mellon. Many people in computer science would know who he is because of his work in VR, but what he should universally be known for is The Last Lecture, which is a pretty amazing hour plus lecture he gave before he died from cancer. But that group has been working on Alice for a very, very long time, and of course, has had new actors along the way. Don Slater is one of the people that has been part of that group for a long time, and he’s very much involved and was at the time when I met him very much involved in advanced placement. And that’s something I’ve been involved in for a long time, advanced placement for computer science. So one of the things we do in AP readings, we have people do professional development activities, and he gave a talk about Alice and this is a long time ago. But when I first listened to him talk about it, and he showed the features of the system, I really didn’t have a place for it in anything I taught at the time. So it has been brewing in the back of my head as a thing to build a course around for a long time, but really couldn’t have done it until the opportunity came along to build a signature course.

John: For those students who do go on to computer science. It seems like a really nice way of introducing the concept of object oriented programming. Does it work in that way?

Rameen: So the thing to understand about object orientation is that most of us who are software engineers by discipline, database types, we are very comfortable thinking of a student information being an object and the fact that we have a container of student objects and so on. But it turns out that’s not necessarily as comfortable for students as it is for those of us who do this for a living. But when you say, here’s an avatar, and you put this on the screen, and you could tell it to go from point A to point B, that seems like a very natural idea to students, and the fact that this particular avatar, so suppose it’s a bird, has wings, and opposed to a person who has legs, you don’t have to explain that. It’s a concept as inherent in being a human being and 18,19 years old. So some aspects of object orientation that often is difficult for students that are really obvious in this context. So any object like an avatar of a person, dog, cat, whatever, they can all be moved from point A to point B. So they share a set of expectations and attributes. They have a location in a 3D world, and you can move them from A to B, piece of cake, they understand. But then you say, “Well, this one is a bird, it has wings.” So the fact that you can spread the wings or fold the wings would be a characteristic that exists only because of it’s a bird. So inheritance, which is a concept that we like to teach in computer science is just built into the way the system behaves. And no student will say, “Well, what do you mean, a bird can spread this way or fold its wing.” People just naturally know what it all means. And believe me, it’s not always natural, in some of the other things we try to do with students to teach these topics. So it does lend itself extremely well, in understanding that objects have attributes, they have functionality, and it’s all there on the screen, and they can see it.

Rebecca: I think Alice is really nice, because it is so visual. And so you get those immediate, “I can see the thing I did,” whereas I remember when I started learning some code, I was building a database for car parts, and it was completely abstract. And I cared nothing about car parts.

Rameen: Yeah.

Rebecca: So it didn’t make it that accessible to me.

Rameen: It’s not, exactly, then the other aspect of it that I think we need to think about, about the platform, is that you don’t write a single line of code, you generate 1000s and 1000s of lines of code, but you don’t write any. So if you have a particular avatar as the object that you’re processing at the time, in building your code on the screen, you could just drag and drop functionality it has into your code. If you need to loop and repeat steps, you drop a loop into your code and then put the steps you want to repeat inside that loop. So all the typical barriers a student had with syntax or various languages, whether it was Java, Python, C++ kind of wash away, because you don’t really have to know syntax at all you, need to know “what are you trying to do?” and what will enable you to do it, and then you can execute that. So far, clearly, that’s not a problem for them. Here is the screen, this part of it is dedicated to X, that part of it is dedicated to Y and they’ve been able to handle it probably from week one. So all the standard things that tend to take a long time, don’t take any time. And besides doing 3D graphics, if you are a computer science person, in my mind, is super senior level type of activity. You got to teach them an awful lot about data structures and other event handling elements that they must learn because that is what we all had to learn. But guess what, you can learn it with Alice in short order. And this is the course is proving that you can.

John: Now one of the challenges that I could imagine you might have as that students come in with different levels of prior knowledge or interest or engagement with computer science. Some students may have not written a single line of code in any language, while others may also be taking other CS courses at the same time, or have some prior programming experience. How do you address the differences in background?

Rameen: So my sample case here is small, I only have 17 students, but this is not a computer science required course. So this is one that has art students in it, it has biology students, and and it does have a few computer science students and then maybe this one with an AP computer science background from high school, and none of them are doing any better than these other kids. So I guess the point is, it levels the playing field in a pretty significant way. If you can think a thought you can probably write code in Alice. And I’m finding it quite interesting, since I’m not preparing them for another course… not only this course doesn’t have a prerequisite, it’s not a prerequisite for anything else. So the way I designed the course going into it, I went into it with thinking, okay, so if storytelling and writing a really cool story within groups is the best I can get out of them, great. If I can get them to a point where they can write new functionality for objects, and I can help them write reactive programming so they react to the mouse click or collisions of objects and so on, maybe I’m dreaming, but that would be fantastic. At this point, I’m pretty certain I can get him there without other stuff. But that was kind of the key coming into the course, I walked in with a mindset of being flexible, that if they are struggling, I’m not going to keep pushing it like I would typically do in a CS course, which is partly why you would also lose students, but at least in my experience with these kids, and I can’t say until I teach it again (and hopefully I can) whether it will be the way it works is that you show them how to do something, and then they go to work, and they start doing it and then they make mistakes, and we all do, and then you give them a little bit of a hint about potentially maybe a different technique they could have used to accomplish the same task. I’m just going to give you an example. So you want the bird to go from point A to point B, so it’s on the ground, needs to go up on top of the tree. So Alice lets you put a marker where you want it to go on the tree, because you can’t go to the tree, you’re going on a branch of a tree. So you need to know how to put a marker there. So you put a marker there. And then it just goes from point A to point B, it goes from the ground to the top of the tree, then you say “Wait a minute, that’s really not the way birds fly.” So now you got to figure out, well, how am I going to flap his wings to go from point A to point B, to go from the ground to the branch on that tree? So it turns out and I’ve come up with a solution to this myself, obviously, you can’t really teach these things if you haven’t thought about how you could possibly solve them. And one of my students, after like three weeks of instruction, she figured out how to do what we call in Alice a “do together.” So as it’s moving from point A to point B, the step that is happening at the same time is the flapping event of spreading and folding the bird’s wing and she made it very clear that the bird was flying [LAUGHTER] from the ground to the tree with no interruption. Then we need to talk about well, do they really need their legs hanging out as they’re flying? I don’t know much about birds, but I think they fold their legs back. So now we have to learn how to address some kind of a functionality that is about a part of the body of the bird. So this is the way the learning is happening in the course, kind of naturally, you’re trying to make a realistic action on the screen in the animation. Well, how are we going to do that? Well, we have to now address the joints like the hip joint or the knee joint or the ankle joint to make that much more natural in the way it works. And there’s no persuasion here, the student is trying to make an interesting thing. And then I’m there to help them figure out how do you make that much more realistic.

Rebecca: What I really love about these courses, and in what you’re describing with Alice, as someone who’s also taught code to students, particularly ones that are not in computer science, is that they’re thinking like a computer scientist, and you’re really getting them completely within the discipline, you’re hooking them right in because they’re leading with their curiosity. They’re not satisfied with the way something looks, so they’re digging in and digging in and digging in. And unlike our traditional way of structuring curriculum, where we think this is the foundational information, and this is the next thing we build on, it almost turns it totally on its head [LAUGHTER], and does it like backwards from what we traditionally do. And it’s really fun.

Rameen: Well, I think the students are at the center of that type of a decision, that for years, you see human beings that probably could do this kind of work, but shortly after they try and they get errors after errors after errors, they say, “Hey, listen, this is great that there are geeks like you would like to do this kind of thing. It’s not for me,” and they walk away from the discipline, even though they could have had great contributions in computer science. So for me if some of these concepts are introduced this way, where syntax and semantics, which is typically what slows people down when they first begin, even the systems we use… like how do you type your program? And how do you run your program? …there’s a whole bunch of instruction around how do you do anything. You just go to alice.org, you download Alice 3, but once you do, it’s here you go, you click here, and then you set up your scene; you click here, you begin writing code. Well, how you write code? Well, the object is on the left side, you drag the command from the left to the right, how far do you want it to go? Well, you gotta choose a certain distance that it needs to travel… really, really easy for students to take to right away. And I just had no idea what I should expect. You watch a lot of YouTube videos. I mean, I certainly do when I was preparing this course, of all these different people, young and old, building things and being proud of what they had built. And I thought, if I could bring that to a course for our first-year students, that would be really, really awesome. And I think that’s what has happened.

John: You mentioned that the students are able to interact. Are they all in one virtual shared space for the class? Or do they have to invite the other students into the spaces that they’ve developed?

Rameen: So this is a really good question, John. So when I imagined how the course was going to work, I had to think of a number of things. One, I asked our technology people to install Alice on all lab computers because I can’t assume or assert that every single student that will take a course like this will necessarily have the equipment that could enable them to run it. Even the Mac kids who had trouble at first installing the thing, and I needed people to help them to get it installed, even they could continue to work because we had the software on our machines. The type of collaboration that I advocate for in class is a little untraditional. At least I think you could argue that it may be. So like the other day, I gave them a 10-question quiz. So they answer the 10-question quiz. And then I said, “Find a couple of other people and persuade them why your answers are correct and their answer is wrong.” So now the whole room starts talking about the quiz. I don’t know if they’ve ever had an experience where somebody says, “This is not cheating, what I’m asking you to do.” Who gives a quiz that says, talk to everybody else to see what they answered for the quiz.

REBECCA; John does. [LAUGHTER]

Rameen: And that’s not surprising, but in my mind, is it about the learning process? Or is it about assessing or giving a grade? This is a very low-stake experience. So why exactly would I care if you talk to someone else about it? So why not persuade someone else that your answer is right? That’s a very different tactic than to say, “Do you know the answer? And are you right or wrong?” Persuading someone requires talking to them, requires thinking for yourself, first of all. Well, why is this answer right? And then opposite of that, you hear the explanation. Are you persuaded that what they said is accurate? Or do you think they’re wrong? In which case now you’re giving them back a different perspective, and then they change their answer. And of course, you could change your answer for the wrong reason. That’s just one example. I really want them to collaborate and work with each other. And every time somebody does something interesting, like the young woman who built the code that I had not been able to myself, having the birds fly from point A to point B, looking very natural. I had her come to the front of the room, plug in with a connector that is in the room and show everybody how she wrote her code. And we’ve done that at least a dozen times so far, where people just come up, plug their computer and show everybody their code. So we often are worried about students cheating and using other people’s work. And if it is about collaborative learning, then you really have to cultivate the idea that, you know, that was a really good idea, maybe I can do that. And I think hopefully, the course will continue to behave that way where I’m confident everybody’s learning from it. That’s the concern that I’m the only one who knows something, whoever I am as a student, and everybody’s just copying me or whatever. That is not my experience so far in this course. They’re just trying to do it better, and if you have a better idea, maybe I can take that and move with it. Then maybe I’ll make it a better idea. And you see that also with students. One of them figures out how to make somebody walk more naturally, and then the other one even enhances that, even makes it even more realistic in the way you would walk. And that’s kind of what I like to see happen and is actually happening.

Rebecca: So how are students responding? Are you cultivating a whole new crop of computer scientists?

Rameen: So, this is an interesting question. I am wondering, those who are not computer science students, whether or not they decide that this may be something for them. But I’m also… with Rebecca here, is good to bring this up… they might become interested in interaction design as a discipline to pursue and become passionate about. Those of us who do this kind of work for a living and have done it for 40 years or whatever, the engagement aspect is the critical aspect. If they are really invested in the learning process, they can overcome an awful lot of barriers, that frankly, I cannot persuade you to put the time in, if you’re persuaded that you could do something a little bit better, then I’m done. As a teacher, I’ve set up the environment in the way it should be where you are driving the learning process yourself as a student, and they look like that’s what you’re doing at this point. And we’re only within a month into the course and they are behaving that way. Now whether or not they will continue to take more computer science courses, and get a degree in computer science. I’m not really sure, but, hopefully, if they do interaction design, there’ll be better interaction design students, because some of the structures that they have to learn here would definitely benefit them in that curriculum too.

Rebecca: Yeah. When can I come recruit?

Rameen: Anytime. I’ve already had Office of Learning Services. That was one of the things I wanted to point out, that I’ve had Office of Learning Services, they came and they talked about the learning process. And besides what they have done for me and talking about the learning process, and it’s all research-based discussions, which is really critical for students to hear things that actually do work, and we can prove they can work. I talk about the learning process on a regular basis with them. And I’m very interested in them understanding why we’re doing anything that we’re doing. I mean, they may be used to somebody standing in front of them talking for an hour and I just don’t do that. I may talk for 10 minutes and then have them work on stuff and then as we see gaps in what they understand then I talk for 10 more minutes, maybe. I try not to talk a whole lot. I want them to be working. So the time I’ve spent is on building what they are supposed to do to learn, not so much talking to them on a regular basis during the 55 minutes that the class goes on. Unfortunately, I’m not quite sure that that’s the experience they’ll have in other courses that they take, because to me, there is a freedom embedded in the way the course is designed that is hard to replicate if you have to cover from A to Z of some topic. You get to “H” and people are having trouble, well, you just keep going. Well, I don’t if everybody has the luxury to say, well, maybe we need to pause longer because we can’t get past this point. I mean, what’s the point? We get past this point, you’ll never catch up to where the end is. So I am hoping some of them will decide to be CS majors as a result, but I’m more interested in seeing how they will do if they take more CS courses. I mean, if they take a CS 1 course, are they going to do better than a typical person taking a CS 1 course if they move on and take data structures and other courses we require? Will it come to them easier? I think it’s a really interesting question. And I think there’s a lot of research that advocates for the fact that they will do better, but I like to see it firsthand.

John: One of the things I believe was mentioned in the title of the course was storytelling, what types of storytelling takes place in the class? Is it the design of the scenes or is there some other type of communication going on?

Rameen: So the way you do animation, in general… and I probably should back up and say I spent about an ungodly amount of hours trying to learn how to do this, but I went about it backwards because I went after event-driven programming and interesting things that I knew Alice could do to write games. But then you step back from it and say, well, students can’t start there. I mean, that’s just not a good place. So then you look at the alice.org website, which gives you tremendous amount of resources and say, “Oh, designing scenes happens to be the first thing they teach you to do. Oh, maybe I should learn how to design a scene.” So you put the pieces that you want in your story on the screen, and if you don’t want them to appear, you could make them invisible. But the way Alice works, you have to put all the components in on the front end. It’s a little different than the way we do object-oriented programming. When we do object-oriented programming, you create things when you need them, you don’t think about setting up the scene on the front end. So that took a little getting used to, but that’s what you do. And then the characters you put on the screen can move, they can talk, they can fly, they can do whatever you need them to do. And my biggest interest was storytelling, when I was conceiving of the course was that I really want students, especially those from other cultures, other backgrounds, to tell their story, find a way to tell their story. And this is probably going to be starting as a group project for my students in a few weeks. And we’ll see how that actually goes. And obviously it has to have a beginning and middle and an end for it to be an actual story. But I’m just excited to see what they will actually decide to do and how they actually do it. Along the way, though, they’re going to need some tools. And that’s kind of what my contribution will be in making sure that they can tell the story the way they want.

John: It sounds as if developing this course required you to learn quite a few new things that were outside your normal teaching experiences. Do you have any advice for faculty who are working on the development of similar courses?

Rameen: So for those out there who teach for a living, the opportunity to build something from the ground up, especially something that frankly, when I first thought of it, I thought it’d be a lot easier than it turned out, because there was so much that I didn’t know how to do, but when you don’t know something you don’t necessarily know that you don’t know something. People who were doing it and I was watching them do it made it look very easy to me. But once I began to do it, I discovered how much work there was to come to a point that actually orchestrates a course, I mean something that is meaningful, and has a clear direction to it. So if you have that opportunity, even after 40 years of teaching, to start over in some ways and build something that you feel not particularly comfortable about, I really highly recommend people do that. Because that is what your students are experiencing every single day when they are trying to learn this stuff that you know so well. So having a little bit of a taste of what it takes to learn something you know very little about, I think is critical. So my message to the faculty who are listening, is that if that opportunity arises, by all means, take it.,

Rebecca: You certainly get a lot more empathy for what it feels like to not be an expert, when you’re learning something brand new again.

Rameen: Well, that’s the thing about most of the things we do. I’ve been programming for 50 years. So it’s one of those things where you’re completely in tune with the idea of: understand the problem, solve the problem, whatever. But where should the camera be in a 3D world in order for it to point at the person talking just the right way? I had to figure that out. It didn’t come naturally to me. The first bunch of programs I wrote the camera was always in the same spot and then I began learning that “Oh I have control over where this camera goes. [LAUGHTER] So maybe it needs to be somewhere else when this person is talking versus this other one.” That’s been a lot of fun to get a sensibility back into the system here that this stuff is not as obvious as it may seem.

John: We always end with the question: “What’s next?”

Rameen: So I certainly would like to teach the course more, and I also want to do some presentations with the faculty in computer science, and if there is interest in graphic design faculty to do some for them, too, because I think the platform is extremely powerful. It doesn’t cost anything, the resources that exist have been getting developed for a very long time and they’re pretty mature. And again, back to no cost. We all know how much books cost. You really don’t need one, you just use the exercises that they give you at the Carnegie Mellon site for Alice and go with it. So I really want to advocate for faculty to consider using it beyond just this first-year signature course.

John: Well, thank you. This sounds like a really interesting project that can really engage students.

Rebecca: Yeah, it sounds like a lot of fun. I can’t wait to come visit.

Rameen: Yeah. Thank you both. Really, this was fun. Thanks for the tea also.

John: Well, we have Myriam to thank for that.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

311. Upskilling in AI

With so many demands on faculty time, it can be difficult to prioritize professional development in the area of AI. In this episode, Marc Watkins joins is to discuss a program that incentivizes faculty development in the AI space. Marc is an Academic Innovation Fellow at the University of Mississippi, where he helped found and currently directs the AI Institute for Teachers.

Show Notes

Transcript

John: With so many demands on faculty time, it can be difficult to prioritize professional development in the area of AI. In this episode, we examine a program that incentivizes faculty development in the AI space.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Marc Watkins. Marc is an Academic Innovation Fellow at the University of Mississippi, where he helped found and currently directs the AI Institute for Teachers. Welcome back, Marc.

Marc: Thank you, John. Thank you, Rebecca. It’s great to be back.

Rebecca: We’re glad to have you. Today’s teas are:… Marc, are you drinking tea?

Marc: I am. I have a Cold Brew Hibiscus, which is really great. It’s still very warm down here in Mississippi. So it’s nice to have something that’s a little bit cool. That’d be refreshing.

Rebecca: That sounds yummy. How about you, John?

John: I am drinking a peppermint spearmint tarragon blend today. And it’s not so warm here. In fact, my furnace came on for the first time yesterday.

Rebecca: Yeah, transitions. And, I have English tea time today.

Marc: Well, that’s great.

John: So we have invited you here to discuss your ongoing work related to ChatGPT and other AI tools. Could you first describe what the AI Institute for Teachers is and its origins?

Marc: Sure, I think that when I was last a guest here in January of this year on your show. And it seems like 1000 years ago [LAUGHTER], but during that spring semester, I really took a much deeper dive than the original pilot with a lot of the generative AI tools in the fall. And we started noticing that the pace that big tech was deploying these tools and integrating these with existing software from Microsoft and Google was only accelerating. So in about April or May, I went to my chair, Stephen Monroe, and said, “I think we need to start training some people to get them prepared for the fall,” because we kind of thought that fall was going to be what it is right now, which is a chaotic just sort of mash up of sort of everything you can imagine that some people dive in deeply, some people tried to ban it, some people are trying to do some critical approaches with it too. So we actually worked with the Institute of Data Science here at the University of Mississippi, and we got some money. And we were able to pay 23 faculty members $1,000 apiece to train them for a day and a half about everything we knew about Generative AI, about AI literacy, ethics, what tools were working in the classroom, which wasn’t. And their whole goal was to go back to their home departments over the summer and serve as ambassadors and help prepare them for the fall semester. And we started that, we’ve had funding for one Institute, and now we’re doing workshops, and searching, as we all will, for more funding for doing,

Rebecca: How did faculty respond to (A) the incentive, but (B) also [LAUGHTER] the training that went with it?

Marc: Well, not surprisingly, they responded really well to the incentives, where you can pay people for their time, they generally do show up and do so as well. We had quite a few people wanting to take the training both internally from the University of Mississippi and then people started finding out about it, because I was posting it out on Twitter, and writing about it on my substack. So when we had interest from graduate students in Rome, interest from other SEC schools wanting to attend, and even interest from a community college in Hawaii. Definitely seen a lot of interest within our community, both locally and more broadly, nationally.

Rebecca: Did you find that faculty were already somewhat familiar with AI tools? I had an interesting conversation with some first-year students just the other day, and we were talking about AI and copyright. And I was just asking, “Hey, how many of you have used AI?” And I and another faculty member indicated that we had used AI to make it safe to indicate. And many of them really kind of shook their heads like “No, they hadn’t,” and they were unsure. And then I started pointing to places where we see snippets of it, in email and in texting and other places where there’s auto-finishing of sentences and that kind of thing. And then they’re like, “Oh, yeah, I have seen that. I have engaged with that. I have used that.” What did you find faculty’s knowledge?

Marc: Extremely limited. They thought of AI as ChatGPT. And one of the things we did with the session was basically frame it out as “Look, this was not just going to remain as a single interface anymore.” One of the things that actually happened during the institute that was completely wild to me was the last day, I woke up that morning. And I’d signed up through Google Labs, and you can do it as well, to turn on the features within the Google suite of tools, including in search and Google Docs, and Sheets and everything else. And they gave me access that last day, right before we began. And so I literally just plugged in my laptop and said, “This is what it’s going to look like in Google docs when you have generative AI activate in Google Docs. it pops up and immediately greets you with a wand with a phrase “Help me write.” And what I tried to explain to them and explained to faculty ever since then, is that it makes having a policy against AI very difficult when it shows up at an existing application with no indication whatsoever that this is in fact Generative AI. It’s just another feature that’s in the application that you have grown up with, from many of our students’ perspectives their entire lives. So yeah, we need to really work on training faculty, not just in the actual systems itself, but also getting them outside of that mindset that AI that we’re talking about is just ChatGPT. It’s a lot more than that.

John: Yeah, in general, when we’ve done workshops, we haven’t had a lot of faculty attendance partly because we haven’t paid people to participate [LAUGHTER], but what’s been surprising to me is how few faculty have actually explored the use of AI. My experience with first-year students was a little different than Rebecca, about half of the students in my large intro class had said that they had explored ChatGPT, or some other AI tool. And they seem pretty comfortable with it. But faculty, at least in our local experience, have generally been a bit avoidant of the whole issue. I think they’ve taken the approach that this is something we don’t want to know about, because it may disrupt how we teach in the future. How do you address that issue, and getting faculty to recognize that this is going to be a disruptive technology in terms of how we assess student learning and in terms of how students are going to be demonstrating their learning, and also using these tools for the rest of their lives in some way?

Marc: That’s a great question. We trained 23 people, I’ve also been holding workshops for faculty too, and again, the enthusiasm was a little bit different in those contexts, too. And I agree that faculty, I feel like they feel overwhelmed and maybe some of them want to ignore this and don’t actually want to deal with it, but it is here and it is being integrated at phenomenal rates in everything around us too. But if faculty don’t come to terms with us, and start thinking about engagement with their technology, both for themselves and for their students, then it is going to create incredible disruption that’s going to be lasting, it’s not going to go away. We’re also not going to have things like AI detection, like it is with plagiarism detection to come in and save the day for them too. And those are all things we’ve been trying to very carefully explain to faculty and get them on board. Some of them though, just aren’t there yet, I understand that. I empathize, too. This is a huge amount of time that you spend on these things to think about and talk about as well. And we’re just coming out of the pandemic, people are exhausted, they don’t want to deal with another, quote unquote, crisis, which is another thing that we’re seeing too. So there’s a lot of factors that are at play here that make faculty engagement, less than what I’d like to see.

Rebecca: We had a chairs’ workshop over the summer, and I was somewhat surprised based on our experience with other interactions with faculty, how many chairs had used AI. The number was actually a significant number. And most of them were familiar. And that to me was encouraging [LAUGHTER], it was like, “Okay, good, the leaders of the ship are aware. That’s good, that’s exciting.” But it’s also interesting to me that there are so many folks who are not that familiar, who haven’t experimented, but seem to have really strong policies around AI use or this idea of banning it or wanting to use detectors, and not really being familiar with what they can and cannot do.

Marc: Yeah, that’s very much what we’re seeing across the board too, is that the first detectors that I’m aware of that really came online, I think, for everyone was basically GPTZero, there are a few others that existed beforehand to IBM had one called the Giant Language Testing Lab. But those were all based on GPT-2, you’re going back in time to 2019. I know how ridiculous is it to go back four years in technology terms and think about this… that was a long time ago. And we really started adopting that through education or seem to be adopted in education based off of that panic. The problem is in incidents of education putting a system like that in place, it’s not necessarily very reliable. TurnItIn also adopted their own AI detector as well too. A lot of different universities began to explore and play around with it, I believe, and I don’t want to be misquoted here or misrepresent TurnItIn. I think what they initially came out with it, they were saying there was only 1% false positive rate for detecting AI. They’ve since raised that to 5%. And that has some really deep implications for teaching and learning. Most recently, Vanderbilt Center for Excellence in Teaching and Learning made the decision to not turn on the AI detection feature in TurnItIn. Their reasoning was that they had, I think, in 2022 some 75,000 student papers submitted. If they had the detector on during then that would give them a false positive grade about 3000 papers. And they just can’t deal with that sort of situation through a university level..No one can. You’d have to go through it investigating each one. You would also have to get students a hearing because that is part of the due process. It’s just too much. And that’s one of the main concerns that I have about the tools that it’s just not reliable in education.

John: And it’s not reliable both in terms of false positives and false negatives. So some of us are kind of troubled that we have allowed the Turnitin tool to be active and have urged that our campus shut it down for those very reasons, and I think a number of campuses, Vanderbilt was one of the biggest ones, I think to do that, but I think quite a few campuses are moving in that direction.

Marc: Yes, the University of Pittsburgh also made the decision to turn it off. I think several others did as well, too.

Rebecca: It’s interesting, if we don’t have a tool to measure, a tool to catch if you will, then you can’t really have a strong policy saying you can’t use it at all. [LAUGHTER] There’s no way to follow up on that or take action on that.

Marc: Where we’re at, I think, that for education, that’s a sort of conundrum. We’re trying to explain this to faculty. I think much more broadly, in society, though, if you can’t have a tool that works when you’re talking about Twitter, I’m sorry, X now, and understanding if the material is actually real or fake, that becomes a societal problem, too, and that’s what they’re trying to work on with watermarking. And I believe the big tech companies have agreed to watermark audio outputs, video outputs, and image outputs, but they’ve not agreed to do text outputs, because text is a little bit too fungible, you can go in and you can copy it, you can kind of change it around a little bit too much. So, definitely it’s gonna be a problem, too when state governments start to look at this, and they start wondering that the police officer taking your police report is writing this with their own words, the tax official using this as well, too. So it’s gonna be a problem well outside of education.

Rebecca: And if we’re not really preparing our students for that world in which they will likely be using AI in their professional fields, then we’re not necessarily doing our jobs and education and preparing our society for the future.

Marc: Yeah, I think training is the best way to go forward too and again, going back to the idea of intentional engagement with the technology and giving the students these situations where they can use it and where you, hopefully if you’re a faculty member, you actually have the knowledge and the actual resources to begin to integrate these tools and talk about the ethical use case, understanding what the limitations are and the fact that it is going to hallucinate and make things up, and to think about what sort of parameters you want to put on your own usage too.

John: One of the things that came out within the last week or so, I believe,… we’re recording this in late September… was the introduction of AI tools into Blackboard Ultra. Could you talk a little bit about that?

Marc: Oh boy, yes indeed, they announced last week that the tools were available to us in Blackboard Ultra. They turned it on for us here at the University of Mississippi, and I’ve been playing around with it, and it is a little bit problematic, because for right now, what you can do is with a single click, it will scan your existing materials in your Ultra course and it will create learning modules. It will create quiz questions based off that material, it will create rubrics, and will also generate images. Now compared to what we’ve been dealing with ChatGPT and all these other capabilities, this is almost a little milquetoast by comparison. But it’s also an inflection event for us in education, because it’s now here, it’s directly in our learning management system, it’s going to be something we’re going to have to contend with every single time we open up to create an assignment, or to do an assessment. And I’ve played around with it. It’s an older version of GPT. The image version I think is based on Dall-E, so you would ask for a picture of college students and you get some people with 14 fingers and weird artifacts all over their face, which may not be the one that would actually be helpful for your students. And while the other learning modules there are not my thinking necessarily, it’s just what the algorithm is predicting based off the content that exists in my course. We have that discussion with our faculty, we have them cross that Rubicon on and saying, “Okay, I’m worried about my students using this, what happens to me and my teaching, my labor, if I start adopting these tools. There could be some help, definitely, this could really streamline the process, of course creation and actually making it aligned with the learning outcomes my department wants for this particular class.” But it also gets us in a situation where automation is now part of our teaching. And we really haven’t thought about that. We haven’t really gotten to that sort of conversation yet.

Rebecca: It does certainly raise questions about, obviously, many ethical questions and really about disclosing to students what has been produced by us as instructors and what has been produced by AI and authorship of what’s there. Especially if we’re expecting students to [LAUGHTER] do the same thing.

Marc: It is mind boggling, the cognitive dissonance, with having a policy and saying “No AI in my class,” then all of a sudden, it’s there in my Blackboard course, and I could click on something. And, at least at this integration of Blackboard, they may very well change this, but once you do this, there’s no way to natively indicate that this was generated by AI. You have to manually go in there and say this was created. And I value my relationship with my students, it’s based off of mutual trust. I think almost everyone in education does. If we want our students to act ethically, and use this technology openly, we should expect ourselves to do the same. And if we get into a situation where I’m generating content for my students and then telling [LAUGHTER] them that they can’t do the same with their own essays, it is just going to be kind of a big mess.

John: So given the existence of AI tools, what should we do in terms of assessing student learning? How can we assess the work reasonably given the tools that are available to them?

Rebecca: Do you mean we can just use that auto-generated rubric right, that we just learned about? [LAUGHTER]

Marc: You could, you can use the auto-generated rubric separately from Blackboard. One of the tools I’m piloting right now is the feedback assistant, it was developed by Eric Kean and Anna Mills. I consulted with them on this, too. She’s very big on the AI space for composition. It’s called MyEssayFeedback. And I’ve been piloting this with my students. They know it’s an AI, they understand this. I did get IRB approval to do so. But I’ve just got the second round of generated feedback, and it’s thorough, it’s quick, it’s to the point. And it’s literally making me say, “How am I going to compete with that?” And maybe the way is that maybe I shouldn’t be competing with that, maybe it’s I’m not going to be providing that feedback. But then maybe then I should be providing my time in different ways. Maybe I should be meeting with them one on one to talk about their experiences, maybe that way. But I think you raise an interesting question. I don’t want to be alarmist, I want to be as level-headed as I can. But from my perspective, all the pieces are now there to automate learning to some degree. They haven’t been all hooked up yet and put together a cohesive package. But they’re all there in different areas. And we need to be paying attention to this.Our hackles need to be raised just slightly at this point to see what this can do. Because I think that is where we are headed with integrating these tools into our daily practice.

Rebecca: AI generally has raised questions about intellectual property rights. And if our learning management systems are using our content in ways that we aren’t expecting, how is that violating our rights or the rights that the institution has over the content that’s already there.

Marc: A lot of perspectives of the people that I speak with too, their course content, their syllabi, from their perspective is their own intellectual property in some ways. We get debates about that, about the actual university owns some of the material. But we have had instances where lectures were copyrighted before in the past. And if you’re allowing the system to scan your lecture, you are exposing that to Generative AI. And that gets at one aspect of this. The other aspect, which I think Rebecca is referring to is the issue with training this material for these large language models itself could indicate that it was stolen or not properly sourced from internet and you’re using it and then you’re trying to teach your students [LAUGHTER] to cite material correctly too, so it’s just a gigantic conundrum of just legal and ethical challenges. The one silver lining in all this, and this has been across the board with everyone in my department. This has been wonderful material to talk about with your students, they are actually actively engaged with it, they want to know about this, they want to talk about it. They are shocked and surprised about all the depths that have gone into the training of these models, and the different ethical situations with data and all of it too. And so if you want to just engage your students by talking to them about AI too, that’s a great first step in developing their AI literacy. And it doesn’t matter what you’re teaching, it could be a history course, it could be a course in biology, this tool will have an impact in some way shape or form in your students’ lives they want to talk about, I think maybe something to talk about is there are a lot of tools outside of ChatGPT, and a lot of different interfaces as well, too. I don’t know if I talked about this before in the spring, the one tool that’s really been effective for a lot of students were the reading assistant tools, one that we’ve been employing is called ExplainPaper. They upload a PDF to it, it calls upon generative AI to scan the paper and you can actually select it to whatever reading level you want, then translate that into your reading level. The one problem is that students don’t realize that they might be giving up some close reading, critical reading skills to it as well too, just like we do with any sort of relationship with generative AI. There is kind of that handoff and offloading of that thinking, but for the most part, they have loved that and that’s helped them engage with some really critical art texts that normally would not be at their reading level that I would usually not assign to certain students. So those are helpful. There are plenty of new tools coming out too. One of them is called Claude 2 to be precise by Anthropic. That just came out, I think, in July for public release, it is as powerful as GPT-4. It is free right now, if you want to sign up for it as well too. The reason why I mentioned Claude is that the context window, what you can actually upload to it is so much bigger than ChatGPTs. I believe their context window is 75,000 words. So you can actually upload four or five documents at a time, synthesize those documents. One of the things I was using it for as a use case was that I collected tons of reflections for my students this past year about the use of AI. It’s all in a messy Word document. It’s 51 pages single spaced. It’s all anonymized so there’s new data that identifies them. But it’s so much of a time suck on my time, just go through to code those reflections. And I’ve just been uploading to Claude and having it use a sentiment analysis to point out what reflections are positive from these students, in what way, and it does it within a few seconds. It’s amazing.

John: One other nice thing about Claude is that has a training database that ends in early 2023. So it has much more current information, which actually, in some ways is a little concerning for those faculty who were trying to ask more recent questions, particularly in online asynchronous courses, so that ChatGPT could not address those. But with Claude’s expanded training database, that’s no longer quite the case.

Marc: That’s absolutely correct. And to add to this rather early discussion about AI detection, none of the AI detectors that I’m aware of had time to actually train on Claude, so if you generated essay… and you guys are free to do this on your own, your listeners are too… if you generated and essay with Claude, and you try to upload that to one of the AI detectors, very likely you’re going to get zero detection or a very low detection rate for it too, because it’s again, a different system. It’s new, the existing AI detectors have not had time. So the way to translate this is don’t tell your students about it right now, or in this case, be very careful about how you introduce this technology to your students, which we should do anyway. But this is one of those tools that is massively popular, a lot of people just haven’t known about it because, again, ChatGPT just takes up all the oxygen in the room when we talk about Generative AI

John: What are some activities where we can have students productively use AI to assist their learning or as part of their educational process?

Marc: That’s a great question. We actually started developing very specific activities for them to look at different pain points for writing classes. One of them was getting them to actually integrate the technology that way. So we built a very careful assignment, which called on very specific moves for them to make both in terms of their writing, and their integration of the technology for that. We also looked at bringing some research question, building assignments that way. We have assignments from my Digital Media Studies students right now about how they can use it to create infographics. Using the paid for version of ChatGPT Plus, they can have access to plugins, and those plugins then give them access to Canva and Wikipedia. So they can actually use Canva to create full on presentations based off of their own natural language and use actual real sources by using those two plugins in conjunction with each other. I just make them then go through it, edit it with their own words, their own language too, and reflect on what this has done to their process. So lots of different examples, too, I mean, it really is limited only to your imagination in this time, which is exciting, but it’s also kind of the problem that we’re dealing with, there’s so much to think about.

Rebecca: From your experience in training faculty, what are some getting started moves that faculty can take to get familiar enough to take this step of integrating AI by the spring?

Marc: Well, I think the one thing that they could do is, there are a few really fast courses. I think it’s Ethan Mollick from even from the Wharton School of Business put out a very effective training course that was all through YouTube, I think it’s like four or five videos, very simple to take, to get used to understanding how ChatGPT works, how Microsoft’s Bing works as well too, and what sort of activities students can use it for, what sort of activities faculty could. Microsoft has also put out a very fast course, I think takes 53 minutes to complete about using generative AI technologies in education. And those are all very fast ways of basically coming up to speed with the actual technology.

John: And Coursera has a MOOC through Vanderbilt University, on Prompt Engineering for ChatGPT, which can also help familiarize faculty with the capabilities of at least ChatGPT. We’ll include links to these in the show notes.

Marc: I really, really hope Microsoft, Google and the rest of them calm down, because this has gotten a little bit out of control. And integration of these tools are often without use cases, they’re often waiting to see how we’re going to come up and use them too. And that is concerning. Google has announced that they are committed to releasing their own model that’s going to be in competition with GPT4, I think it’s called Gemini by late November. So it looks like they’re just going to keep on heating up this arms race and you get bigger models, more capable and I think we do need to ask ourselves more broadly what our capacity is just to keep up with this. My capacity is about negative zero at this point… going down further.

John: Yeah, we’re seeing new AI tools coming out almost every week or so now in one form or another. And it is getting difficult to keep up with. I believe Apple is also planning to release an AI product.

Marc: They are. They also have a car they’re planning to release, which is the weirdest thing in the world to me, that there could be your iPhone charged in your Apple Car.

John: GM has announced that they are not going to be supporting either Android or Apple CarPlay for their electric vehicles. So perhaps this is Apple’s way of getting back at them for that. And we always end with the question, what [LAUGHTER] is next, which is perhaps a little redundant, but we do always end with that.

Marc: Yeah, I think what’s next is trying to critically engage the technology and explore it not out of fear, but out of a sense of wonder. I hope we can continue to do that. I do think we are seeing a lot of people starting to dig in. And they’re digging in real deep. So I’m trying to be as empathetic as I can be for those that don’t want to deal with the technology. But it is here and you are going to have to sit down and spend some time with it for sure.

John: One thing I’ve noticed that in working with faculty, they’re very concerned about the impact of AI tools on their students and student work. But they’re really excited about all the possibilities that opens up for them in terms of simplifying their workflows. So that, I think, is a positive sign.

Rebecca: They could channel that to help understand how to work with students.

Marc: I hope they find that out, there’s a positive pathway forward with that too.

John: Well, thank you. It’s great talking to you and you’ve given us lots more to think about.

Marc: Thank you guys so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

308. Design for Learning

We tend to design courses for ourselves because we are the audience we know best. In this episode Jenae Cohn joins us to explore how user-experience design principles can help us create effective and engaging learning experiences for the students we have right now. Jenae is the Executive Director of the Center for Teaching and Learning at the University of California at Berkeley. She is the author of Skim, Dive, and Surface: Teaching Digital Reading. Her newest book, co-authored with Michael Greer, is Design for Learning: User Experience in Online Teaching and Learning.

Show Notes

  • Cohn, J. (2021). Skim, dive, surface: Teaching digital reading. West Virginia University Press.
  • Cohn, J., & Greer, M. (2023). Design for learning: User Experience in Online Teaching and Learning. Rosenfeld Media
  • Global Society of Online Literacy Educators
  • Horton, S., & Quesenbery, W. (2014). A web for everyone: Designing accessible user experiences. Rosenfeld Media.
  • Web Accessibility Guidelines
  • Copies of Design for Learning may be ordered at the Rosenfeld Media website. The discount code for listeners is TEA20. It’ll be available on Wednesday, 9/27 and will give listeners access to 20% off the book for one month (i.e. 30 days).

Transcript

John: We tend to design courses for ourselves because we are the audience we know best. In this episode we explore how user-experience design principles can help us create effective and engaging learning experiences for the students we have right now.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Jenae Cohn. Jenae is the Executive Director of the Center for Teaching and Learning at the University of California at Berkeley. She is the author of Skim, Dive, and Surface: Teaching Digital Reading. Her newest book, co-authored with Michael Greer, is Design for Learning: User Experience in Online Teaching and Learning. Welcome back, Janae.

Jenae: Thank you. I’m so glad to be back.

John: It’s good to see you again.

Jenae: …Good to see you, too.

John: Today’s teas are… Jenae, are you drinking any tea?

Jenae: I sure am. I’m always prepared to drink tea. Especially when I’m talking to the two of you. But I went for a classic English breakfast tea this morning. Do you both have some tea with you?

Rebecca: Yeah, I have English tea time.

Jenae: We’re matching…

Rebecca: Yeah…

[LAUGHTER]

John: And, I’m not. I have [LAUGHTER] ginger peach black tea today.

Jenae: That sounds really good, though.

John: It is.

Rebecca: Sounds like a good way to start the day, for sure. So we invited you here today to discuss Design for Learning. Can you talk a little bit about how this book project came about?

Jenae: Absolutely. So my colleague Michael and I have a lot of shared interests. Michael and I both are trained in rhetoric and composition. And we both are people really interested in online writing, online reading, and online learning, broadly speaking. We both served on the board for the Global Society of Online Literacy Educators, which is an organization dedicated to supporting folks who teach reading and writing online, broadly speaking. Through that organization, we got to know each other better. And we just realized how much we wanted to talk about what it really meant to create quality online learning experiences. And something that kept cropping up for the two of us. And I should say that both of us have had like a hodgepodge of jobs in and around higher education. We kind of joke that we were both sort of like these misfits in higher ed, people who have kind of done a bit of teaching, a bit of admin. He’s worked in publishing, I did a lot of work in instructional design and just higher education pedagogy. And something we noticed, just in the various roles that we were in was that educators, professors, faculty could learn a lot from user experience frameworks. And we were reading a lot about UX and UI in the work that we were doing around instructional design and for him publishing, and it just dawned on us like, why are we not bridging these conversations between the work of thinking about designing learning interfaces and the work of building really good, high quality learning experiences. I think we notice that in higher ed, there is this tendency to kind of try and reinvent the wheel around defining what a good teaching experience, especially what a good online teaching experience is by just creating really kind of exhausting templates and tons of checklists and rules. And we really thought those are useful, but wouldn’t it be more useful just to remember that students are people navigating devices online? And can’t we use the frameworks that help inform those design decisions to inform the design of learning experiences to make them better? So that was really the genesis of this project. We started off thinking we’d write a bunch of blog posts and then it struck us that blogs and articles were great, but wouldn’t it be even better if we wrote a book [LAUGHTER]. So we put it all together, and it resulted in this book.

John: So who’s the intended audience of this book?

Jenae: We really are targeting a broad audience with this book, I’d say primarily folks who do instructional design style work in mind. So in higher ed, that could be faculty, a lot of faculty play the role of instructional designers, as well as facilitators and teachers, of course. But, we also hope that this book would really reach folks who do dedicated instructional design support. We also hope that this would just reach people who are having to teach online or do trainings or workshops online, and who are still really struggling with it. This book, I would say, was written before the pandemic happened. We were, I would say, drafting and conceptualizing it before the pandemic. And of course, the pandemic shaped the drafting as we went, there’s still COVID-19 out there, so I don’t want to say we’re beyond the pandemic. But in this moment where we’re beyond perhaps like a peak point of the pandemic, let’s just say. There may be folks who are still wanting to be more intentional about what it means to provide more equitable access to online learning experiences, who want to be designing in a more intentional way, and who want to be really thinking critically about how to create more sustainable online learning experiences, as well, that really work. I think we were also on a mission with this book to prove that really, anyone can do this, you just need to keep some known principles in mind that, again, this is not totally new territory, and scholars and user experience and human computer interaction have been thinking for a very long time about how to make information accessible online, and how to make sure that information and interactions are easily navigable. And so that was really the literature we wanted to tap into. So that’s all to say that I think people who benefit from reading this book is really anyone who wants to be creating a better online learning experience for whatever teaching situation they’re in.

Rebecca: I’m, of course, super excited about this book, because I’m a UX designer. I love that you use that framework to write this book. Can you talk a little bit about why you chose this approach?

Jenae: Absolutely. I’m so glad that you appreciate this book exists. We’ve gotten really good reception from the UX community on it as well. I would say that we use this framework because we felt like it really centered the learner in an important kind of way. I think that in a lot of teaching situations, people who educate or design learning are often more thinking about the content: What information do I have to deliver? What are the main things that I need to make sure people know how to do? Those aren’t bad things to focus on, we need to cover content, and we need to make sure that there are clear outcomes. But I think it’s most important to really think about how is someone engaging with that content? How are they understanding it? What are their opportunities to understand that content in a variety of different ways. And I think what a user experience framework allows us to do is to center that reminder. Learners have these embodied experiences that shape how well they’re going to be able to learn, how well they’re going to be able to interface with the information. And if we’re talking about that in an online context, in particular, it’s impossible to do so without addressing what it means to, again, engage with and use these online environments effectively. So I think a UX framework really just allows us to be more centered in reminding ourselves who really benefits from the learning experiences we design, and who really needs to have access to [LAUGHTER] the information to be successful. And I think UX frameworks just really help us center that.

John: Can you talk a little bit about how this approach centers the user in terms of practical ways in which that’s built into the design process?

Jenae: Sure, one way to sort of think about that is to really take a step back and try to remind yourself just who is taking your class in the first place. Starting there, starting from the place of trying to be curious about who your learners really are. I think that it’s easy to make assumptions, I’ll just say in higher education, in particular, since I think that’s primarily the audience for this particular podcast. I think a common misconception, for example, is that all students entering their class are traditional college age, 18 to 21 years old, but like, I should put a big asterisk on that and say, that’s probably not the traditional age at most institutions anymore. But that’s the stereotype of kind of who a college student is. And there may be some assumptions about what their prior learning experiences were like that brought them into a college classroom… about the prior knowledge that they had. And so what I think a user-centered design encourages us to say, “Do we know that? How do we know that? What information do we need to gather to remember who’s actually coming into our rooms?” And I’m not suggesting that any educator has to like, do deep dive demographic data work to find out who their learners are. But I think most of us can kind of anticipate the range of people who are coming into our classes. We might anticipate just the different types of learners that we may encounter. And by that, I mean, it’s worth, I think, before you start designing, just trying to remember, what are the different motivations that students have for coming into the class? What are their purposes for being there? What are the main things that students are going to want to do by being in your class or your training or your workshop at any given moment? So starting by just sort of trying to map out who those people are? And then try to anticipate, okay, given this motivation, or this purpose that this learner may have, what kinds of things might they be looking for… literally looking for my online course? What things will they click on first? Which links are they going to want to access most frequently? Which resources are going to benefit them on the site most? And then trying to design your learning management system course site, or if you’re not using learning management system, your course website, broadly speaking, to really privilege the resources, the links, the activities, the pages that are going to be best aligned with what you anticipate your users or your learners may need. And Rebecca, I’m sure, can speak to this given her expertise, too, but UX design really is a whole process of trying to consider how the visual information, how even like the tactile information, say how your keyboard was set up, how your device is set up, how that allows you to most easily use and engage with products, so to speak, that you’re building. And in this case, we want to think about how can you build the best online course that you can, in a way that allows users to most easily find the information you anticipate they will most frequently need?

Rebecca: So one of the things I’m hearing you say, is really thinking about the wide variety of learners that we have and the different needs that they have and trying to address that. One of the things that’s really popular in UX design and that you talk about in your book are personas. Can you talk a little bit about how learner personas can help us think through the different kinds of learners that we have in our class in a really practical, tangible way. You just kind of provided that theoretical framework, but I love that the personas is such a practical application of that.

Jenae: Yes, thanks for asking that. Rebecca, I was debating whether to dive into that with the last question. But let’s dive into it now. So for those who aren’t familiar, personas are an exercise where you really try to create a character sketch, I would say, of the user you’re imagining is going to engage with your course or in this case, try to imagine an example of a student who’s going to be in your class. And by creating a character sketch, I mean, I encourage instructors, if they have the time to sit down and say, “Okay, what might be the name of someone in my class? What might be their age? What might their prior experiences with learning my topic have been? Why are they here? What brings them to college? Or what brings them to this class in the first place? What are going to be some of their biggest challenges? What are going to be some of their biggest hopes? What are the things that they’re going to be most excited about doing in this class?” And again, it’s a bit of an imaginative exercise. And so I think it’s easier to do with more teaching experience. But it’s also not impossible to do even if you’ve had relatively limited experience. It’s really just an exercise in trying to think through who might be the real people that you are engaging with, I do want to say that there’s been a lot of conversation in the UX community, and again, Rebecca, you may have some thoughts on this, too, about sometimes the stereotypes that personas can perpetuate. For example, I think there have been concerns in the UX community that when you try to characterize, say, an older user, of an online interface, a stereotype might be that they struggle more or are more challenged with using technology than, say, a younger user. And that that might be a challenge to anticipate. And so I want to be mindful, for example, that if you are going to be in the practice of building personas, which we talk about in the book, because I do think is a useful exercise to kind of try and make concrete for yourself who is going to be on the receiving end of your experience, that you do try to check yourself a little bit on reinforcing stereotypes to the best of your ability. It’s easy to do, stereotypes exist because we notice patterns sometimes in how people behave. And that can sort of reproduce some harmful assumptions about who those users are. But again, to the best of your ability, attempt to anticipate what the needs might be based on what you do know about who might be in the room, just again, kind of reminding yourself that you’ll want to think about your personas in nuanced ways, and not necessarily make assumptions about who they are. And I would say one solution to that, how am I supposed to write a generalized description of a persona, while avoiding all possible stereotypes about who they might be? I would say again, time allowing, try to run your personas by other people, and just see what their reactions are to reading them. For example, if you have a trusted colleague, or a friend who teaches a similar class, or who you work with regularly, just show them what you’ve drafted and say, “Does this feels like a real person to you?” And to attempt to ask diverse people about how your persona sketches are landing or how realistic they feel to them. That’s always a good way to kind of gut check, and just make sure that as you’re anticipating your users’ needs, you’re not falling too much into your own biases about who the people are that you’re supporting in your course.

Rebecca: So one of the things, I think, people do sometimes run into when they’re making personas is to create the ideal student that doesn’t exist, and also to recreate themselves. And so one strategy that I often recommend is thinking about creating aggregates of people that you do know. Because then they’re more realistic in terms of the way they might interact. So if you’ve taught a class before, you might have a real pool of people you could draw from [LAUGHTER] and to create a persona from, obviously, that’s more difficult when it’s a new place. And I was also going to offer up in terms of thinking about disability and thinking about accessibility, that there’s a book called A Web for Everyone, and they have a lot of resources. It was published quite a while ago, but they have a lot of resources still online, they have some personas for people with a wide range of different kinds of disabilities. And sometimes that can be really useful in just thinking through kinds of scenarios that you might not think of on your own.

Jenae: That’s fabulous. I would love to see that resource about sort of supporting accessibility, especially. That’s such a huge issue in designing online learning experiences, particularly. I’m so glad you mentioned those resources. That’s fantastic.

John: And while there may be those types of biases that you might have, those who’ve taught classes multiple times do know some of the types of problems that past students have had. So those issues that they’ve experienced in the past could be built in. But one of the other things you suggest is doing a pre-course survey, so that you get some more information about the actual students in the room rather than those who may have been thinking about when you initially designed the course? Could you talk a little bit about that survey?

Jenae: Yes, I’d be happy to talk about the pre-course surveys. So this is a practice that, I think, has multiple benefits. So in a pre-course survey, I think instructors have this wonderful opportunity just to ask students what their motivations are for engaging with the class, what brought them here, how they would characterize some of their prior experiences with learning similar topics, if any, and just to voice what concerns they have, or what things are exciting to them about the term ahead. I’m giving a lot of examples of possible questions and I just want to acknowledge that not all instructors will want to ask all of those questions all at once. But those kinds of questions that really get at motivation and concerns, I would say, in a nutshell, can be really critical, both for adjusting, I think, those persona expectations. So, creating personas should be an iterative process, I should say, as well. It’s not a one and done thing where you anticipate who your learners are prior to the course starting and then you’re like, “Okay, I figured it out, I know who all the students are. Knowing who the real students are, can then allow you to go back to what you anticipated. I think, and both of you, Rebecca and John, were speaking to how you could use prior information from prior terms to inform your kind of current term or current course. Great, you could sort of just align your prior understanding with this current information you might get from these surveys to then go into your course website, or your course learning management system, your syllabus and say, “Okay, is this design going to work for the group of people who are actually here based on what I’m reading?” …recognizing, of course, that nothing’s gonna be perfect for everyone. But you can do the best you can to try and make the materials as good as possible for the group that you have. In front of you. I would say that you want the survey to feel less burdensome for your students to complete. I’m giving a lot of examples of questions that I think are ideal as open-ended questions. Some of these, you could turn into multiple choice or kind of Likert-scale style questions, because you can just use it as an opportunity to take the temperature. “On a scale of one to 10, for example, how confident do you feel in your understanding of your ability to pick up new quantitative concepts?” …for example, if you’re teaching in a STEM-style discipline. Or “On a scale of one to 10, how comfortable do you feel as a writer or with writing tasks?” …if you’re teaching something more humanities- or writing-centric. You can get really creative in trying to solicit some feedback. And I also encourage instructors to be judicious in what they’re asking in these pre-course surveys to kind of try and ask questions, with the end goal of helping you as the instructor make small tweaks to the design of the course. Think about this information as a way to say “Okay, are there certain links I should put on the homepage that I didn’t think needed to be on the homepage? Or should I reorganize the menu on my learning management system in a way that highlights some resources more than others based on the information I’m getting in the survey? Should I reorganize a module to introduce some content before other content, because I’m seeing a trend in the surveys about less confidence in one area of the course than I was expecting in another?” So thinking about how the answers might inform your design, a research-based perspective really, I think, can make your course really even stronger. And I think it’ll feel better, both for you and the students because it helps the students see that you’re curious about them, you want to know who they really are. And we know that engaging personally with people really matters for good teaching. But the instructor too, it can be really frustrating. If you design something and it doesn’t land with your students. You feel like you spent a lot of time building something that didn’t work. That’s a really disheartening experience. So getting the feedback might allow you to avoid [LAUGHTER] having or feeling so disappointed if the information didn’t land the way you were expecting it to. And this isn’t foolproof. There’s always room, again, for iteration. But I do think the surveys can at least help you anticipate a little bit better how the progression through your course could go.

Rebecca: I can imagine that some of those surveys with open-ended questions could lead to better understanding how students name things or label things which could give you a lot of clues about the actual user design of a course by just how you might name or provide quick descriptions of things. In your book, you talk a lot about instructional text design, which obviously has lots of skills in online learning from instructions for assignments to just how we might label a folder [LAUGHTER]. There’s lots of skill there. Can you talk a little bit about the basic principles that you’d recommend for course designers to follow when they’re writing instructional text?

Jenae: Absolutely, and I realized, as you were talking and responding, I was nodding along. And then it struck me. It’s like, “I’m on a podcast, no one’s going to know that I’m nodding and agreeing with you right now.” So [LAUGHTER] for the listeners sake, like I was nodding along quite vigorously with that entire response. Instructional text, I think, is one of the most underrated and one of the most important things to design for any online course experience. I think that online course designers have a real tendency to rely too heavily on video and on images. There’s an assumption that if you’re working online, everyone’s just using video all the time, or everyone’s just wanting to engage with the flashiest multimedia possible. That is still important. I mean, we have two chapters in the book, all dedicated to video. So I don’t want to undermine that. It is important to engage with multimodal artifacts and building multimodal interventions, when you’re teaching in a multimodal environment like the internet. However, for students who may have low internet access and low bandwidth, for students with disabilities, text remains one of the most accessible and easiest ways to find information in an online course. I’d also say text is one of the most mobile-friendly pieces to think about. And we know that increasing number of students are accessing their courses or coursework through their smartphones. I’ll answer your question directly now, but I wanted to provide that context. I would say when it comes to designing instructional text, I really encourage instructors to think about two big things, to think about the hierarchy of the information that they’re writing, and to think about the discrete chunks of information that they’re wanting to communicate. So when I talk about the hierarchy of texts, I think it’s important when we’re writing to consider what are the sections of our text? Most academics, most instructors, are used to, when they’re reading or writing, creating headers, and sub-headers, and paragraphs that denote a certain order of information. And when you’re teaching online, especially, I think even more critically about how are you labeling the text? How are you indicating which things are instructions versus content? How are you labeling the order of the content that you want students to read in? How are you even labeling the order of instructions, like there is usually multi-tiered sets of steps. So using header text and different layers of header text, is a really important web accessibility measure. And again, it helps readers see visually and if they’re using a screen reader tool, it helps them navigate that text more easily. So I should take one step back and say when I’m referring to header text, I mean that when you’re working in a rich text editor, on any website, you can typically see an option to select different layers of headers, like the header ones are usually the highest, biggest level header, header twos go below that header, threes go below that. So just being mindful that just increasing text size is not the same thing as using headers is one really, really simple way to create hierarchy. And again, to denote the correct order of reading the text information. And when I say chunking text, this is as simple as just thinking about paragraphing, making sure that you are spacing out pieces of content in really critical ways. So anyone who’s read a piece of writing with super long paragraph knows, that’s a lot harder to kind of discern, it’s a lot harder to see how one idea moves from one to the next. Shorter paragraphs are typically easier to get a sense of when you’re moving from one idea to a new idea. And so even though long paragraphs have their purpose, perhaps especially in scholarly writing, or even in more, I would say kind of creative writing, in some cases, when you’re doing really instructional or technical work, which you’re often doing when you’re designing a class, shorter is better, more chunked is easier to access, because you’re assuming that people are doing things with your information. So those are the two qualities I would just be thinking about with instructional text. There’s a whole other component that we didn’t really address in the book, but I’ll just stick to very briefly here, which is also thinking about just the visual appearance of your text. A lot of accessibility folks speak to some best practices and guidelines around font face, and font size, and some of these factors when you’re designing text as well. I’m not an expert, I should say, in like type of graphic design or font size, but I want to point out anyway, because I think if you are designing online, it’s important again to do the best that you can to try and anticipate those needs. So I think as a general rule, making sure your font sizes are not super teeny tiny, or super large. Making sure that you’re using standard font faces: Arial, Helvetica any sort of sans serif font is typically considered a best practice. The rules around this change all the time, Web Accessibility Guidelines change as technology evolves, so I never like to give super hard and fast rules, and again, it’s not my area of expertise. But it’s another piece to keep in mind that visual and verbal information is intertwined. Text is a visual medium, online learning experiences are largely a visual medium, by default. And so the more mindful we can be of what that looks like, and the more mindful we can be of how the visual experiences we design online, are compatible with accommodations for disabled users. We just anticipate our users’ needs, our learners’ needs more proactively, and it raises the boats for everyone. It just gives everybody a deeper chance to succeed if we’re just thinking about these interface choices in more deliberate ways.

Rebecca: I love that you’re really talking about how the instructional text is also part of digital accessibility. It’s important to have plain language, it’s important to chunk your content and these sorts of things. So I’m really excited that you’re incorporating that into the work that you’re doing.

Jenae: Thank you. It is exciting. I think it’s one of these things that, when Michael and I were first discussing this book, it was a real lightbulb moment for us that there was such a robust literature out there that discussed all these great principles for making sure that online information was easy to find. And it just was striking to us that a lot of folks in teaching professions weren’t getting access to that information or exposure to that information. And we started thinking about this, again, prior to the pandemic, kind of in the mid 2010s. And even at that point, online courses were growing, mobile access was becoming a more common way that students were engaging with courses. So, why not tap into these existing sets of conversations that are industry best practices, for engaging with online interfaces, in spaces like higher ed, and in spaces just like learning and development, where these dialogues seem not to have met each other as fully as they could.

Rebecca: Our chief technology officer and I were having a conversation about some of these things yesterday as we’re talking about our student body is diversifying and that we have far more students with disabilities who are able to attend college and have access to college in a way that maybe they haven’t in the past. And as you were talking about headings and paragraphs and things, something that people might not know, is that if you use a screen reader, you’re not necessarily visually interacting with the text. Instead, you’re thinking programmatically, and so just like kind of vision centered [LAUGHTER], the user might skim headings visually, it’s the same way someone might use a screen reader. So by choosing a heading level two, it allows someone to find that section easier. And by breaking things into paragraphs, and delineating that’s that kind of content that allows a screen reader user to be able to jump to a particular part of the content. When we don’t do that, a screen reader user has to listen to everything from the top to the bottom of the page.

Jenae: Great example. Yes, and that’s such a frustrating experience to have to do that. If we can be just a little bit more attentive to the information architecture of sort of what we’re trying to communicate and convey… information architecture is a technical term, but it’s also a metaphor [LAUGHTER]… we have architecture and we have design to help create solid foundations for places that we live. Similarly, when it comes to information, we need to be building solid infrastructure to help people navigate their way through a course. One of my colleagues a while ago used a metaphor for online learning design I’ve never forgotten and we’ve alluded this a bit in the book, which is that when you’re building something online, it’s like you’re just building a whole house [LAUGHTER]. When you walk into an in-person classroom, the architecture is literally there, and you make assumptions about the room in the space, the second that you walk in the door. When you’re designing text online, or just when you’re thinking about the whole online learning experience, it’s a total blank canvas, you have to build that architecture and those hierarchies. If you’re not attentive, you’re absolutely right, the consequence is that it can be a big overwhelming mess of information. And I think it’s a useful practice for instructors, even when they’re not teaching online, to think about these things. It’s also just a great exercise and getting really very focused on what information do you want to prioritize when you’re communicating assignment instructions or when you’re picking out content-based readings for your students? What do you want them to focus on? What are the big things you really need them to learn or pay attention to? And so if your course design, your visual design can align with the hierarchy of choices you’re making as an instructor or the priorities that you’re setting, it just makes it easier for everyone to have equal access that information so that more time can be spent for students to focus individually on how they’re processing, applying, doing higher-order thinking with that information. They don’t have to spend so much energy just trying to intake the kind of basics before they have the opportunity to really work with it and apply it meaningfully,

John: You provide a lot of other information in your book, and we encourage people to read your book. If they want to find out more about creating videos, about providing effective webinars, and so forth, there’s some really nice hints and suggestions throughout. But one of the things you end with in there, is ways in which instructors can continuously improve their courses, in terms of soliciting feedback to make the course better each time. Could you talk a little bit about how you would encourage instructors to continuously work on developing their courses?

Jenae: Sure. So I really like that section of the book, because what I hope that section communicates is that thinking about your course design is a reflective and an iterative process. I don’t think a course is ever really fully perfect and done, there’s always things you can do, and modify each time you teach or offer the experience. So, I don’t think getting feedback on the course has to be hard, I don’t think it has to take a ton of time. We talk about multiple ways of getting information about how the course is working. And I’m going to start with I think some of like the easiest and most passive ways to get information and then we’ll sort of work our way to some of the more perhaps active or personalized interventions for getting information about the course. So one thing I think is worth really paying attention to, after you finish teaching a course, are some of the analytics that are available in your learning management system or your course website. And I recognize that some folks are really reluctant to look at the analytics, because there is a surveillance economy implicated in the tracking of course analytics. Every site on the web tracks your movements, every site in the web knows how long you’ve stayed on a certain page, what things you’ve clicked on. And a learning management system is no exception to that. Unfortunately, that information can get weaponized to discriminate against students, discriminate against users in problematic ways. In the web, outside of learning, for example, analytics can be gathered and sold to advertising companies to spread information about your activities for profit. So I just want to note that context, but you can also use this information for good and for some useful things as well. So seeing which resources students are clicking on the most in your class can be really useful information for you to say, “Huh, seems like a lot of people found that resource useful.” You don’t have to necessarily identify which individual students looked at which resources but you can look at this data, typically in aggregate, and again, most learning management systems have an analytics dashboard, you can access to look at this. I think that’s incredibly useful just to see what was clicked most often and what wasn’t. You might also want to track, for example, which pieces of information students did spend more time on. It could indicate a couple different things, it might indicate that something was really challenging, if students spend a lot of time on one particular piece of content over another or if they found it useful. You’d have to contextualize that data based on what you were seeing in the course. But I think if you’re willing to look at that information, again, in the context of how your term went, it might just give you some passive information that could surprise you. I would even look at, for example, with assignment submissions, how many delays were there on certain assignments versus others? In which assignments did students request more extensions more than others? Again, this is just information that might help you inform whether the pacing was appropriate for the course, whether assignments were sequenced appropriately. That kind of thing. If you want to get more active, if you gave a pre-course survey, you can do a post-course survey. Most institutions, of course, have formal evaluations of teaching, but we know that institutional student evaluations of teaching can be fraught. Sometimes they ask the kinds of questions we don’t always want to ask or find most useful as instructors. So if you do your own very brief post evaluation, you could focus it on the design of the course itself. I think it’s worth asking students at the end of the course, how easy was the course site to navigate? How accessible did the materials feel for your ability to learn? You could return to some questions from your pre-course survey. If you asked a Likert scale about rating your confidence with learning something on a scale of one to 10 at the start of the course, you could ask them by the end, “How does your ranking change?” Even referring back to the original data that they might have submitted to you with the pre-course survey. So those are another way to ask them. I think if it’s possible to, what I love to at the end of the course is even a little brief post interview with students if possible. We mentioned this a bit in the book. Again, it’s time consuming. But if you have a small-ish class where you could have conferences at the end of the term, and have a moment with just a five-minute conversation to ask students: “How did it go? What aspects of the course design did you like most? Which were most challenging to you?” That’s another way to get information. Finally, I’ll just do one more technique we write about in the book, which is never discount your own reflection on your experience as well. This is another form of user research. Even though you are not the end user for the course, you are the designer, and so I think it’s always useful just to jot down a few notes and treat that as research when you’re done, too. What did you notice about user interactions on your course site throughout the term? What things surprised you? What things went exactly as you expected? You can use those notes to iterate and improve your experience for the next time that you offer it. So those are just a few techniques and many, which again, are drawn from the field of user experience research surveys, and interviews, for example, are pretty common user experience research practices… other UX research practices that, again, just depending on your time, depending on your resources, it’s great if you can see students engaging in the course as well, asking them, just really seeing what it looks like for them to interact in the course. That’s a good way to get at good information about it. I just want to encourage anyone who’s teaching not to shy away from getting that kind of feedback, because it does make, I think, teaching more satisfying when you’re getting more information about what’s working and what isn’t.

Rebecca: So, you know this question’s coming…[LAUGHTER] We always wrap up by asking: “What’s next?”

Jenae: Yes, I do know it’s coming, and it’s funny, because I was thinking about it. What am I [LAUGHTER] doing next? So to be honest, I don’t have a clearly defined project, I’m doing a lot of little things, I might be taking a little break, because I have written two books in about two and a half years [LAUGHTER]. So that’s been a lot… wonderful. I think I’ve been bitten by the writing bug, for sure. And so I suspect there’s more writing in my future, but nothing immediately next. I’m still very curious about what it’s going to mean to keep designing really good online learning experiences in the future, I don’t think we’re done with that conversation. I’m really curious about how that’s going to evolve in the context of creating more inclusive and equitable learning environments for students. So I imagine those are topics I will continue to explore to some extent, but we will see how, of course, with AI too, and the impacts of that on online learning, I’m sure there’s gonna be a whole set of ways to think about these topics that will continue to evolve. So I’m kind of keeping my eyes open and my ear to the ground on how things are developing. And we’ll just kind of see what ideas emerge from there.

Rebecca: Well, it’s always a pleasure to talk to you, Jenae. Thanks for all the work that you do.

Jenae: Likewise, thank you, again, for having me and for engaging with these excellent questions. And if you listened to this podcast, we’ll put in the speaker notes, I’ll give you a little gift of a promo code. If you’d like to buy the book, we can give you a 20% off discount with thanks to Rosenfeld Press who published this book.

John: Well thank you. We’ll be sure to include that in the show notes and it’s always great talking to you.

Jenae: Wonderful, and likewise, thank you again.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

306. Gender Bias and Timing of SETs

 A number of studies demonstrate gender bias in course evaluations. In this episode Whitney Buser, Jill Hayter, and Cassondra Batz-Barbarich join us to discuss their research that looks at the timing of when these gender differences emerge and theories for why they exist.

Whitney is the Associate Director of Academic Programs in the School of Economics at Georgia Tech. Jill is an Associate Professor of Economics in the College of Business and Technology at East Tennessee State University. Cassondra is an Assistant Professor of Business at Lake Forest College. Whitney, Jill, and Cassondra are the authors of an article entitled “Evaluation of Women in Economics: Evidence of Gender Bias Following Behavioral Role Violations.”

Show Notes

Transcript

John: A number of studies demonstrate gender bias in course evaluations. In this episode we discuss research that looks at the timing of when these gender differences emerge and theories for why they exist.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Whitney Buser, Jill Hayter, and Cassondra Batz-Barbarich. Whitney is the Associate Director of Academic Programs in the School of Economics at Georgia Tech. Jill is an Associate Professor of Economics in the College of Business and Technology at East Tennessee State University. Cassondra is an Assistant Professor of Business at Lake Forest College. Whitney, Jill, and Cassondra are the authors of an article entitled “Evaluation of Women in Economics: Evidence of Gender Bias Following Behavioral Role Violations.” Welcome Whitney, Jill, and Cassandra,

Whitney: Thank you for having us.

Cassandra: Thank you so much.

Rebecca: Today’s teas are:… Whitney, are you drinking tea?

Whitney: I am. I have some jasmine tea.

Rebecca: Always a good choice. Jill. How about you?

Jill: Harney and Sons Hot Cinnamon Spice.

Rebecca: Oh, that’s such a good choice. I love that one. It’s a family favorite at my house. How about you, Cassandra?

Cassandra: Yesterday, we made a sun tea on the porch. So it’s sweet peach tea.

Rebecca: This is a good variety. How about you, John?

John: And I have ginger peach black tea from the Republic of Tea.

Rebecca: So we’re combining choices here [LAUGHTER]. And I have Awake tea, despite the fact that it is early afternoon here.

Jill: I also had three cups of coffee this morning.

Rebecca: It’s one of the most popular kinds of tea, Jill.

John: We’ve invited you here today to discuss your research on gender bias instudent evaluation of instructors. Could you tell us how the study came about?

Whitney: Jill and I have been working on this for about six years, believe it or not. It’s been a long process for us. And actually at the very beginning we had a different third working with us. And the original three of us, we met at the conference, and we had just attended a session that talked about teaching evaluations. And afterwards, we just naturally began talking about this, because we all had these really, really strong feelings about teaching evaluations. All three of us at the time were young, young in our careers, young age wise. We were female PhD economists. And we were all earning tenure, or I think Jill had just earned tenure. But we’re all in this similar experience of having what we felt like was a very positive class climate, and a lot of camaraderie between ourselves and the students until the grades were returned for the first time. And then we could feel a definite shift and it was upsetting to all of us. We all got into this because we love teaching and we want to do a good job in that. It was just something that we were picking up on. So that was our anecdotal experience, Jill had a little data on it herself, because she would do mid-semester evaluations herself, just to gauge the class climate and see what students were needing. And I had an experience where in my first position, they did a surprise midterm evaluation, just to kind of see how the new professor was doing, that I didn’t know about. And I got glowing reviews from the students, everything was very, very positive, wonderful and six weeks later, same students but grades returned, evaluations looked a little different. And the comments were a bit different. So we had a little data to backup this idea too, and one thing if the people listening today haven’t read the literature, there’s an extensive literature on course evaluations. And it consistently finds gender bias in those. But the thing about that literature is it only looks at evaluations, which are typically done on the very last day of class, maybe even after that, maybe a couple of days before, but at the end of the semester. And we really haven’t seen anyone look into how these opinions of students evolve over the semester, or how students feel at the beginning or the middle of the semester. So that’s what we wanted to do with that. And in my opinion, and this is just me speaking here, Jill can have her own other motivations, or our other co-author that has worked with us before could feel differently. But for me, it was really important to acknowledge that society has come a long way in the past several years with gender bias. And I don’t think that modern students are shocked by female faculty any longer, I don’t think they have an explicit distaste for female faculty. Anecdotally, I feel that my students are actually happy when they meet me. And they have expectations of me to be warm, comforting, approachable. But I do think that when you expect someone to be more comforting and approachable, and they give you a grade back, that’s not always an “A” in a difficult quantitative subject like economics, you can get a bit of a Grinch Who Stole Christmas effect. I thought it was going to be one way and now my expectations are taken down. We all know no one likes that dopamine depletion of having expectations not met. So, to me, if we’re going to talk about gender bias, we really have to talk about it in this nuanced way, so that it doesn’t get automatically dismissed by people who don’t see an explicit bias and then say, “Oh, hey, there’s nothing here.” And then the last thing that I think is really important here for the motivation for the paper is that we have this expectation that bias would grow over the semester. So if bias grows over the semester, that means the earlier in the semester you evaluate, the smaller the bias will be. And one thing that the literature is missing is a very concrete objective way to deal with bias. What we were hoping to find was: move the evaluations up in the semester a bit, and you minimize or eliminate bias and that’s a concrete objective. Towards the end today, we’ll talk about what we actually found and whether or not we knew that. But that was one of the motivations.

Jill: So that’s how the original paper found in terms of motivation, but then Cassandra, she is a PhD in Psychology until she had read and she was doing work in the area. And she had reached out to Whitney and I. She had read our paper, she had read the results of our paper. And so then a second paper with Cassandra takes a more psychology approach in terms of a lot of what Whitney is talking about and Cassandra is going to talk about it later, with respect to the role-incongruity theory, social role theory, and she’s going to talk more about that later. And Whitneys described the motivation of that first paper, the second paper takes a very different perspective and looking at it from a more psych perspective. Cassandra, you might want to chime in?

Cassandra: Absolutely. I think you summarized it well, I joined the paper, as Whitney and Jill were trying to find a home for it. And we thought that our interests, though coming from very different backgrounds ,would blend nicely for this particular topic, as there’s a lot of scholarship in psychology that looks at understanding reasons behind this bias. And so I was brought in to really help kind of think about how do we frame that in a way that might appeal to even a broader range of audiences.

Rebecca: At the beginning of the paper, and Whitney, you’ve kind of pointed to this today about being a young faculty member, you also noted in the paper that women are underrepresented among economics faculty, especially at the level of full professors. Can you tell us a little bit about the extent of this under-representation?

Jill: Women have earned more than half the doctoral degrees for over a decade. But particularly among tenure-track faculty are underrepresented. In the paper we cite 36% of full professors are females. In economics, that’s a smaller percentage, 17 and a half percent of full professors are females, in the area of economics, although 35% of PhDs in econs represent females. It’s a smaller percentage of female faculty receiving full professor rank in economics. That’s what we mean by that under representation. In terms of economics, specifically, it’s oftentimes left out of the STEM fields, and depending on which university or college that you’re out at, economics can sometimes could be found in the social sciences and in the arts and sciences, or it can be found in the business school. So at my institution, Whitney’s institution, I believe, and Cassandra’s I think we’re all represented in the business school. But sometimes, you know, economics wanted to put in there with the social science field, it’s not thought of as being this more quantitative, heavy subject, and it oftentimes is, it is by nature of it. And so females in those more math heavy classes, like the STEM classes. I think my students when I started off, and I think Whitney was getting at this, with us being more junior faculty members. I can considered by students peer, instead of the professor in the course. And that made it tough, because to Whitney’s point about that returning grade feedback and the perception that students had of me a day one versus midway through the course, I was now coming across as someone that was handing back maybe less than 100% or “A” grade. So in my business school, my principles of economics courses are required. They might not even want to be in there, but they have to be in there to get a business degree. Earlier on, that was a challenge I faced, I’m 13 years into my career. I’m going up for full professor this summer. But starting off was really a challenge. And I remember having female mentors in my graduate program. They tried to prepare me for this, they tried to say it’s going to be challenging early on, you’re going to have to go against some of these perceptions, alot of the perceptions that we measure in this paper..

John: To what extent is the underrepresentation of women faculty due to a cohort effect where women have become a larger share of PhD economists in the last few decades, but that was less true 20 or 30 years ago and how much of it might be due to the impact of gender bias on evaluations on career pathways for women?

Jill: Really what this paper looks at, the standard evaluations of teaching and the bias or potential for bias, that exists there. So I’ll just speak to that and that where I currently am, evaluations of teaching are weighted heavily for retention of faculty, promotion of faculty, tenure and promotion decisions. And then when we’re hiring new faculty, looking at any previous course evaluations and experience with teaching. At every level in academia, these are used as some gauge for teaching effectiveness. I think one of the questions that we’re looking at and accrediting bodies are looking at is whether or not this is the measure that should be used. And looking at different measures that might be options for measuring teaching effectiveness, we know that they’re flawed, that our study is showing that they’re flawed, but also previous literature has suggested that they’re flawed as well. And so the fact that for most schools, this is the single measure that’s being captured… and I know that it’s different depending on again, at my institution, some departments don’t give them a whole lot of weight in tenure and promotion decisions. But certainly, my experience in my College of Business and Technology that these are weighted heavily. And so in thinking about a junior faculty member starting off, when Whitney and I met at the conference, if my evaluations were lower, I’m putting a lot of time into my teaching and improving and bringing up those scores. My male colleagues, in discussion just with them, didn’t have the same experience that I was having with respect to these SETs. And so we think about allocation of time and resources as a tenure track junior faculty member, I’m putting more in what I would consider just catching up, getting those SET scores higher, so that it’s reflected in my tenure and promotion packet. And that’s less time that I’m allocating toward research or other things. That’s my view on it. I think Whitney has a couple other thoughts on that.

Whitney: One of the things we tried to make clear in the paper is that the literature is very clear that evaluations do have a gender bias. And if these evaluations are being used, and they are, in hiring decisions, annual evaluations, promotion, tenure evaluations, and merit pay raise decisions, then they’re being used at every single level of advancement. It’s not one small piece. It’s a piece that’s used throughout and very integrated late in the process.

Rebecca: You mentioned at the top of our interview that the second paper shifts more towards psychology, and specifically describes ways in which both social role theory and role-congruity theory may explain the bias against female faculty in student evaluations. Can you briefly summarize these arguments for our listeners?

Cassandra: So social role theory was a theory that has been put forth for decades by Alice Eagly, a very prominent scholar in the social psychology world, as well as her colleagues. And this has been used as a framework to really understand the complexities and origins of gender gaps in our workplace in particular, whether that be inequities and experiences, the expectations that are different for women, and of course, the outcomes such as promotion at work. Essentially, social role theory suggests that the reason we see these gender inequities today in society or that they originated from men and women being distributed into social roles based on physical sex differences, so that women biologically were able to have children, men, on average, were physically stronger, which those differences 1000s of years ago, had an evolutionary benefit to a well functioning society, people were supporting in the ways in which they were best equipped to do so. And the assignment of men and women into these roles led get them to adapt role-specific qualities and skills. So women who were bearing children were friendly, helpful, sensitive, concerned with others, kind, caring. We refer to these now as more communal qualities, and men and the provider, the protector, role led them to have attributes such as ambition, being assertive, authoritative, dominant. These are qualities that now we label as agentic. So while technology of course has since caught up and made these biologically driven role assignments unnecessary, society continues to see a division of labor along these lines in the modern world and society at large. And society at large still holds the belief that women do possess these traits, and should possess these traits, these more communal qualities, and men do and should possess more of these agentic. Relatedly, role-congruity theory helps us understand the consequences when men and women fail to fulfill these expectations. And we know the failure to fulfill these expectations are more consequential for women, this experience of bias driven from the failure to behave in communal ways. In other words, violating these cultural expectations can be seen in all areas of society, but particularly in traditionally male-dominated positions, like college professors, or in male-dominated fields like economics [LAUGHTER]. And so women that are in these roles are already going to experience some degree of backlash for being in gender-incongruent positions. But that is especially true if they are also going to behave in traditionally more agentic ways, being more assertive, demonstrating their power, which we argued was what was occurring when you give critical feedback back to students.

John: To approach this, you gave evaluations to students at two different points of the semester. Could you tell us a bit more about the study design, how large the sample was and how many faculty and institutions participated in the study?

Whitney: Sure, we had a really rich data set for this study. That’s one of the reasons we were able to get two different papers out of it, and maybe even some future research, because we took all of this data, and we collected it in person on paper and entered it, which was an arduous process. As I said, we had been working on this project for about six years, about a year and a half of that was just data collection. And we have a lot of people to thank that did that for us for no author credit on this paper, so we had males and females across the United States gathering that data for us, that we’re really appreciative to have. So in the end, we wound up with about 1200 students in total, we weren’t quite 50/50, we were 60/40, favoring men, which is typical for economics classrooms, even though it is required in a lot of majors (that’s where you’re getting a lot of the women taking it). And like you said, John, we surveyed them twice. We surveyed them on the second day of class, we wanted as close to a first impression as possible without having a major sample issue with drop/ad. And then we surveyed them the day after they got their first midterm grade back. So we got the first impression, and then we got the way that they felt after they had had their first grade returned. We did this at five different colleges and universities, we had three male professors contributing data and four female professors contributing data. One of the big questions that people have asked us over the time is “Well, how does race play into this?” And that’s something that’s beyond the scope of our research, I will say that we only had one underrepresented minority in our sample, again, typical of economics professors, it was one of our male instructors. So, we would expect a downward bias from race and maybe an upward bias from gender, or getting those two, at least watching one another out in the paper. And when we asked these students about how they felt after their grades were returned. This was about four weeks into the semester, so still pretty early in the semester. What we did was we really wanted to ask about the specific qualities that had been hypothesized in the literature as drivers of bias or drivers of differences. So we just asked students to rate their instructor on a bunch of different qualities. Cassie really helped us out here because she came in and she says, “Well, you know, we can categorize these qualities into communal qualities and agentic qualities and neutral qualities…” which was really the way to approach it because of course, we get different things in communal versus just qualities. So we asked our students things like: “How knowledgeable do you find your professor? How challenging? Do you find them to be approachable? Do you find them to be caring? Are they interesting?” And then we asked a couple of very general questions: “Would you recommend the course?” All of this set us up to have a really nice dataset where we could look between genders and across time as well.

Rebecca: So I think everyone’s probably dying to know exactly what you found. [LAUGHTER]

Jill: I’m just going to provide an overview of the results because we do a number of different specifications and use different econometric methods in the findings. And so you can get all of those results there in detail. But in general, on the second day of class, we find that women are receiving lower ratings across the five agentic and gender-neutral instructor characteristics that we measured. They were rated higher on that second day of class on those more communal characteristics. And not all of those differences were statistically significant. Immediately after the first exam grade was returned to students, women were receiving lower ratings for all seven measured characteristics. Each difference was significant except for those caring and approachable, more communal characteristics. And then men were now having higher ratings in all the different aspects relative to time, or the second day of class. Over time, what we see was that men’s evaluations were getting higher on all characteristics from the second day of class to the period after the first exam was returned. And then in contrast, women’s evaluations were not trending upward. So we had a couple that were staying the same, but overall, they were going down. So those are just some overview findings. Again, those more specific results, by specification, can be found in the paper.

John: We will include a link to both papers in the show notes too, so people can go back and review them. To summarize, what you found is there was relatively weak evidence of significant gender bias on the second day of class, but that gap increased fairly dramatically after the first graded exam. So what do you attribute that change to, was it because of the feedback students were getting from grades as Whitney had mentioned before?

Whitney: We were attributing, and Cassie can talk about this with more authority on the theoretical point, but we’re attributing that to backlash theory, this idea that if I expect one thing, and I don’t get it, there’s this need to back off so that things go in congruence.

Cassandra: Exactly, Whitney is spot on there. What we thought this was evidence of was women behaving in gender incongruent ways, women are supposed to be warm and caring and friendly. And when you get a perhaps grade that maybe wasn’t an “A,” that feels harsh and critical, and a woman is asserting their power and dominance in the classroom, which again, they already are in a male dominated field profession. And those two things together combined can result in this backlash.

Rebecca: So if we take these findings, and think institutionally, what are some things that institutions might want to think about moving forward?

Whitney: That’s a good question. If you remember, from the very beginning, we were saying, we’re really hoping to find this nice objective concrete solution, we anticipated finding it through timing. And that’s what I would really like to do with future research is to be able to find something concrete and objective to treat this with. We weren’t able to do that because we found bias from the beginning. And we found that it came so quickly in the semester that it’s not something that we can just move back evaluations to midterm or something like that. Since we can’t do that, we’ve talked about other ways for institutions to take this. And one takeaway really is just an awareness that these gender biases exist and that these evaluations are flawed. This is really well established in the literature, but not necessarily in the general sphere of knowledge. When we published this paper, Georgia Tech did a little feature in their daily digest, and I had two female engineering faculty email me and say, “I knew this in my gut for years, but nobody’s ever quantified it.” That to me, is just evidence that it’s not in the general sphere of knowledge, even though the literature defines it well. Some of the impact of the concrete solutions that we have seen is we’re seeing a lot of schools and accreditors, like AACSB, they’re starting to require multiple indicators of teaching effectiveness and evaluation. So evaluations and peer reviews, or maybe something else to see the observation, something to that effect to where we have more of a global and inclusive way to look at someone’s teaching effectiveness. So this is a great takeaway, hopefully that will reduce the weight of the impact of evaluation just by having other factors in there. And just one final point that I want to make. And this is just a really big sticking point to me for the paper is that all of us are researchers, we all deal with statistics and statistical significance, and robust research methods. And then when those of us in Chair and Dean roles go to look at evaluations, all the sudden, all that training completely goes out the window, and we look at the difference between a 4.2 and a 4.4. And I know those differences sound really small, they are that small. And we say, “Oh, well, this person does better than this person, this person deserves to be hired over this person.” Never in our research, or in a formal presentation, would we ever compare two means that small without significance testing, number one, and without making sure they’re actually comparable, and say, “Oh, there’s a difference.” It’s just something that I think we need to recognize, we would not recognize this as good research or good methodology in any other area of our work. It’s just something that we should keep in mind as we move forward with this.

John: Now, you mentioned the use of peer evaluations as another way of providing, perhaps, more balance, but might they be subject to the same type of bias?

Whitney: Yeah, all the things that we would see for student evaluations, I can imagine how you would see with peer evaluations as well.

Jill: But there are creative ways to do peer evaluations that I think here at ETSU, we have a Center for Teaching Excellence. And I’m confident Georgia Tech, and Lake Forest has their own version of that. And so there are creative ways. And again, not that SETs are necessarily bad, but knowing what we know about the flaws in them, that, coupled with an additional measure or two, can be a lot more insightful, I think, to the teaching effectiveness, like true teaching effectiveness of instructors.

John: And one thing I’m wondering is if the measured effect might be larger in economics, because at least at many institutions, grades and economics and STEM classes are often lower, which might magnify the effect of this difference. It would be interesting if there was to be a study that also included some classes, maybe in humanities, to see if perhaps there’s less of an effect because of that role-incongruity issue there. It may not appear to be as severe in disciplines where grades across the board tend to be higher.

Whitney: I think you’re right about that, most people when they take economics, it’s a required class and certainly the grades are a big factor, then the two things that showed the most significance outside of our key variable of interest was interest in economics, and expected grade. Those were the things that across the board… now we still found gender bias controlling for those things, but it mattered.

Rebecca: So we talked a little bit about things that institutions might want to start thinking about: institutional policy and things that might shift how we use teaching evaluations. Are there any other strategies that institutions or instructors can use, or adopt, to try to reduce this bias in the short term?

Cassandra: That’s really the million dollar question. Because this type of bias exists in a lot of different domains, whether we’re talking managers and their subordinates, teachers and their students. One thing that’s often suggested or recommended is simply making people aware that this bias exists, and providing training on how to better approach evaluations, whether that’s how to use a rating scale and ensuring that you aren’t engaging in a halo effect, for example. Another strategy is requiring that people justify their ratings that are provided with qualitative comments… that if you’re just asked to fill out on a scale, on how competent is this person? Well, bias may creep in more if you aren’t asking for a justification of why that particular rating was given for competence. A last recommendation that I’ll share here is making these evaluations more public. So if there are a couple of people, say peers, that are evaluating myself or Whitney or Jill in the classroom, well, they need to come together, share and publicly disseminate their evaluations that they had given to us. This social accountability can help to mitigate bias and for people to ensure that the ratings that you’re giving are, in fact justified.

John: So we’ve got a long ways to go with this. It’s a problem that’s been recognized for quite a while with a lot of studies. But there hasn’t been that much done to address that. And those are some good suggestions that institutions may want to try. We always end with the question: “What’s next?”

Cassandra: [LAUGHTER] That’s a good question. Of course, I think that the three of us collectively would say we do hope that administration and decision makers start asking questions about their use of student evaluations of teaching and how they might seek to mitigate this bias, based on the recommendations Whitney had already shared. But we also hope that women faculty perhaps feel more empowered to advocate for themselves when it comes time for promotion and tenure decisions to be made. My Institution, a part of the promotion process, is writing letters, and going through interviews. So speaking to this, bringing an awareness to the people who are making the decisions that this exists, and that it is not just an opinion, that there is empirical evidence of its existence. But we are really interested in exploring more fully how providing feedback, particularly critical feedback, like in our study, where the professors are giving back grades might impact the perceptions of men and women in other contexts as well. So is this a phenomenon we would see, for example, between a manager and their team? Do people respond differently to critical feedback from a manager because of their gender? And how much are these differences, perhaps, driven by perceptions of how communal or agentic they are in their delivery of that feedback? So in other words, are we seeing the same pattern in other contexts? Ultimately, we hope that by better understanding how perceptions of communion and agency impact interactions that women have at work, particularly women in male-dominated or gender-atypical roles, this greater understanding will allow us to also discover ways to alleviate some of that backlash through more targeted interventions and training and perhaps better timing. Because at a minimum, it’s important to highlight the various ways gender bias continues to persist in our society. Because without that awareness, nothing can be changed.

John: Whitney, Jill?

Jill: I think that was great. [LAUGHTER]

Whitney: Yeah, I think, Cassie, you did a great job. And Cassie certainly helped us out with bringing formal language and theory to things that we felt as intuitive and we felt in our gut as important. We don’t have a lot of language for that in the economic space. And so blending these two disciplines together has been very helpful for looking at the situation.

Rebecca: Well, thank you all for joining us. And the research that you’re doing is really important and impactful. So we hope our listeners will use it.

Whitney: Thank you so much.

Cassandra: Thank you.

Jill: Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

304. ChatGPT Inspired Course Redesign

AI tools such as ChatGPT have the potential to significantly disrupt how we work and how we learn. In this episode, Don Donelson joins us to discuss a course redesign strategy that could help prepare students for a world in which AI tools will be ubiquitous. Don is a senior lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of the Spring 2016 University of Miami Excellence in Teaching Award and the Dean’s Excellence in Teaching Award from the Miami Herbert Business School.

Show Notes

Transcript/h3>

John: AI tools such as ChatGPT have the potential to significantly disrupt how we work and how we learn. In this episode, we discuss a course redesign strategy that could help prepare students for a world in which AI tools will be ubiquitous.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Don Donelson. Don is a senior lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of the Spring 2016 University of Miami Excellence in Teaching Award and the Dean’s Excellence in Teaching Award from the Miami Herbert Business School. Welcome back, Don.

Don: Glad to be here.

John: Today’s teas are: …Don, are you drinking tea?

Don: I am. It’s the same tea that you’re drinking, black currant, and it’s great.

Rebecca: It’s a John favorite for sure. I have a Tazo Awake tea today.

John: Does that mean you’re woke? That may be an issue down in Florida.

Don: …not in a private school.

Rebecca: It means that I couldn’t make a pot of tea. I didn’t have time. So I had to use a single tea bag. [LAUGHTER] That’s what it means.

John: And I am still using the mug from Australia that Clare McNally gave me with kangaroos all over it.

Rebecca: I like that mug.

John: I do too.

Rebecca: I look forward to seeing it in person.

John: Soon.

Rebecca: Yeah, you’ll be back soon, right? A couple weeks.. I’ve got my grad studies mug. We’ve invited you here today to discuss your plans to revise the graduate and undergraduate core courses in critical thinking and business communication at Miami Herbert. Can you tell us a little bit about this course?

Don: So this course started at the grad level, MBAs in particular, in 2008. That’s what I was hired to teach. And it grew with the program, expanding into specialized master’s programs. And then it went out into the undergrad program. And it’s a core course required for all full-time business students, undergrad and graduate. This past year, we had 46 sections of undergrad courses and 21 sections of graduate courses, about 900 students or so in total.

Rebecca: So a really small situation going on here.

Don: Oh, yeah, very small, [LAUGHTER] no problems with scaling or anything like that.

John: What was the typical focus of this course in the past before this revision that you’re working on?

Don: So the course was called “Critical Thinking and Effective Written and Oral Communication,” and it lived up to its name. It was about those three things. At the time that we started in 2008, we called them soft skills. We don’t use that phrase anymore. We like to call them fundamentals, something of the sort. We think that soft skills sends a bad message. But it’s been overhauled three times, this will be the third overhaul since. And the things that we would do in the courses, from the very beginning, the main evaluations would be based on writing memos and giving presentations

Rebecca: Which should be about the kind of communication you’d be doing in business. [LAUGHTER]

Don: Yeah, it’d be based on hypothetical cases, some non-hypothetical cases, the standard Harvard Business publishing 10, 20 page case on “How did Netflix beat Blockbuster?” or something of the sort.

Rebecca: What prompted this big overhaul?

Don: Well, the accreditation body, AACSB, required program evaluation. And it’s sometimes an annoying task that people do just to go through the motions. But what we found, the first time I went through it, was that we actually learned a lot from going through those motions. And so in my department, at least, we institutionalize curriculum audits on a semester basis. And so, in between the fall and spring semesters, we have a shorter meeting, where we kind of look at what happened in all the instructors, faculty teaching in that space in the fall, what happened and what worked, what didn’t work, and we might make some minor revisions. And then at the end of the spring semesters, we’d have a little bit of a longer meeting. And the last few of those had turned out some opportunities for change.

Rebecca: Can you talk a little bit about what some of those opportunities were?

Don: Yes. So we found that this is a challenging course to get buy-in from the students. And so we still haven’t figured out 10% of what we can figure out about teaching in this space, I’m sure. But one of the things that we’ve seen is that it is somewhat of an innovative curriculum, and one of the challenges with being innovative is students haven’t had material like that before. It’s a core course. And so students have to take it. And there’s always challenges with that. But this is a bit of a different challenge. And so I was talking with John the other day, he has some core required economics courses for business students, and some of the challenges that come with that. But this is a bit different in that those students know that economics is a field of study. And they know that people take economics courses, and there’s a textbook and critical thinking and communication. They’ve been taught kind of as separate. They’ve been add ons in other courses, not as a discrete course itself. And so we think there’s some challenges with that. And really, the challenge that we’ve seen, in addition to that, is that from the faculty view, critical thinking and communication are not separate things, they are one thing. And so critical thinking, I would call it problem solving, is really what we’re teaching. But communication is a component of that. And from the student view, we’ve had a hard time getting them to see those as integrated. And so when they do a memo, that’s an evaluation metric, they see it as: “Well, that’s just looking at the writing and not critical thinking.”

Rebecca: That’s interesting. Some of the things that we’ve done in our design courses around critical thinking and writing across the curriculum, my department, which is art and design, is doing some of those same things. We would do projects and embedded in those projects would be things like memos and other ways of communicating as a way to critically think about the decisions that our designers were making on things. But we would run up sometimes against the same kinds of challenges, like how do you really make that feel practical, that’s relevant, and then also keep it interesting. And it helped, I think, in those cases, because it was tied to a project. So is that a challenge that you face in this particular class is because there are these kind of standalone case studies, and it’s hard for students to buy in or get them into a business space?

Don: That’s one of the things actually I think that’s going to be changed is more of an arc to the course. And one of the things that I’m looking at is more integration of assignments. And so things building more towards the other assignments, and so we have skills building on top of each other. But, ideally, the assignments that they’re doing all build towards one culmination assignment, capstone type project.

Rebecca: Where does this course fit into their other required courses? Is it something that happens in the beginning? or in the middle? towards the end?

Don: So that’s partially an administrative question that is dependent on staffing. We see some students wait until the very last semester to take it, particularly the students who don’t have English as a first language, but they can start taking it as early as their sophomore year. But usually, it’s junior.

John: What’s the difference between the undergraduate and the graduate versions of the course?

Don: So the graduate versions are taken on a quarterly basis, and the undergraduates on a semester basis and so there’s more contact time in the undergraduate version of the course. They use different materials, and they’re more in depth. And so much like you would see with undergraduate economics classes, the graduate version of the economics classes might have similar titles, but go far more in depth into the material.

John: So one of the main issues is that students don’t see the critical thinking aspect of it as being important in their writing. How are you going to change your course to focus a bit more on the development of those critical thinking skills?

Don: Well, this is where I need to go back and add more to what Rebecca asked before about what prompted this because of course, ChatGPT prompted a lot of the revisions as well. And so ChatGPT, AI in general, while it’s kind of an independent axis of revision, we were thinking about some of these other problems well, before ChatGPT even became a thing that people were aware of. But they go hand in hand, really. A lot of the problem that I’ve seen with the writing assignments, and why students don’t necessarily view them as critical thinking and focus on the writing, is because there’s writing for aesthetic, and then there’s writing for substance. And if you’re teaching anything about writing, you kind of have to be teaching both. But when you’re teaching both together, the students tend to focus more on the aesthetic. And they connect it back to English composition classes that they might have taken in ninth grade or 10th grade. And those classes are certainly very important. But they’re a bit different than what we’re doing in these classes. And so I think it primes them to approach the course in a way that is not really conducive to getting what we want out of it. And so with AI, well, it remains to be seen, but it looks to me like you don’t need to be teaching the aesthetics of writing so much anymore in a class like this. And so I’m going to experiment with just not. [LAUGHTER]

Rebecca: It’s interesting, because in design classes where we were doing some similar kinds of things, aesthetics obviously always come about, because if we’re doing visual design, aesthetics are a part of that conversation. But we would have the same thing. It was like, “Well, that looks nice. That reads nice. It just doesn’t say anything.” [LAUGHTER]

Don: Right. Yeah. [LAUGHTER]

Rebecca: So it’s interesting that we bump up against these same kinds of challenges across a wide variety of disciplines. And that ChatGPT does offer some opportunity to focus on some different things.

Don: Absolutely.

Rebecca: I’m curious what exactly you’re going to focus on and how you’re gonna leverage ChatGPT in the context of this class.

Don: So I think ChatGPT is an insane, wild, amazing tool. And it’s going to only be more wild, more insane, more amazing next month, [LAUGHTER] or six months from now. But I see it really changing the way that I teach, the way I prepare for teaching. So kind of on the end of creating lesson plans and what it does for me as a teacher and the kinds of things that I can do in a class that I wouldn’t be able to do in a class before without a lot more hours in the day. And then also, from the student side, changing the way that they do assignments. They’re not going to be writing memos outside the classroom in the way that they have in business for decades and centuries. They’re going to be using ChatGPT. And so if, in this course, which is meant to be a practical course, we can’t make it practical if we’re not allowing them to use the tools that they’re not only going to be encouraged to use, they’re going to probably be required. And so if they don’t use ChatGPT in the future, they’re going to have bosses saying, “Why are you spending X amount of hours on this client memo instead of doing something else.” And so we really need to prepare them for that world. There’s been some early research and I’ll get John the citations, but we looked at research over the break between fall and spring, this past year, some preliminary research about the kinds of jobs in the way the labor market is going to be affected in the future by AI and ChatGPT and the jobs that were predicted to be the hardest hit in terms of reduced wages, and just reduced demand are jobs that involve writing and the jobs that were predicted to be the most insulated from Ai were jobs that involve problem solving and critical thinking. And so really, when you look at that research, it doesn’t even give us a choice. Even if we weren’t thinking about making some kind of revisions before, we’d probably need to just on that alone.

John: So is the focus now shifting more to the critical thinking skills and a little bit less on the basic structure of writing?

Don: Yes. And that’s really where even though the impetus for the revisions were independent, in practice, they’re not going to be that independent. And so it really dovetails nicely. And so I’ll give you an example, if a student is writing a memo, where a business is making a decision between two or three different courses of action, and one of the main criteria is the profitability of those courses of action, the structure is kind of guided by the math of profitability. So if you’re not talking about revenues, independently of talking about costs, you’re not proving profitability. And so when we talk about structure in this course, that’s really what we mean. But students very often, because of some of the things I’ve talked about previously, they’re looking at it as far as like the five-paragraph structure. And that’s not really what we mean. And so by being able to focus less on the aesthetics of writing, and more on the substance, I think we’ll be able to undo some of that priming,

Rebecca: …almost like this shift to articulating the decision making…

Don: Yes.

Rebecca: …rather than talking about writing, because articulating, it could be verbal, it could be in written language, it could be in a lot of different formats. But the point is that you thought critically about the issue, and how you made the decision. [LAUGHTER]

Don: Yes, exactly. In presentations, I’ve never had as much of the same problems as we have in the memos. Part of that, I think, it’s because of contemporaneous feedback. My students early on learned that this comment is kind of a trolling comment. And it’s not really meant as a attaboy or attagirl. But, sometimes a student will give a speech, and when they’re done, I’ll say, “I’m very impressed with your public speaking skills.” And they think, at first in the early parts of the class, that that’s a compliment. But they realize that that’s actually not a compliment. What I really mean is “No one would be buying what you’re selling, no one would be buying this stock, no one would be making a decision based on this, but you have very impressive charisma and confidence.” And that’s not really what we’re about, maybe in politics, but that’s a different question.

Rebecca: I’m curious about integrating ChatGPT as part of the process. Are you thinking about requiring students to reveal and discuss how and why they use ChatGPT in particular instances, and how they leverage the tool.

Don: So I think part of it is going to be first showing them how ChatGPT is not a critical thinking tool. And so I think it’ll be kind of walking on the escalator backwards for a bit just so that we can walk forward. It’s not going to be ChatGPT’s here, so you should use it. Go. ChatGPT is like a personal assistant, who is extremely capable and competent, but will do precisely what you tell it to do, and nothing else. The input you give it determines the quality of the output. And so if you go to ChatGPT, and you say, “I’m writing a letter of recommendation for Rebecca, and she was a great student, and she’s applying to law school period,” it’s going to give you about what you would imagine…it’s going to make up some stuff about Rebecca, it might even not get what program you’re in right. It’s not going to use the last name because I didn’t give it one. And it’s going to give you a very fluffy, perhaps disingenuous response. Now if I give ChatGPT a really robust stream of consciousness almost about Rebecca Mushtare was a student in the spring of whatever and she got this grade and she did phenomenal in these areas. And this assignment she really stood out most because of this, this, and this, it might give me a much more usable response that I can then play with. And so I think that’s going to be the first to instruct students on: what it does not do, which is critical thinking. And from there, I think they’ll have to use it however they feel comfortable. We’re still going to have some writing assignments that are scored. But what I’m hoping for is that these changes will make it so that they’re focused much more on the critical thinking parts of it. And so for some students that might look like writing a fairly complete draft on their own, and then putting it into ChatGPT and telling it to edit this for brevity and clean up grammar mistakes or do something of the sort. For some students, it might be much more of a back and forth kind of a conversation with ChatGPT, which I think a lot of students will be surprised to learn that it functions in that way. And when I find myself using it, it’s mostly as a conversation. Like, I didn’t like what you did here, cut that part out and do this again.

Rebecca: It’s funny that we don’t always think about it as a chat tool, despite the fact that chat is in its name.

Don: Yeah, exactly.

John: Before making this major change in your curriculum, have you experimented with any changes in this course recently to put more focus on critical thinking skills before introducing ChatGPT?

Don: Yes, so in some of the sections, especially at the graduate level, since we have so many different master’s programs, when I first started, it was MBA, and pretty much that’s it. Now with where the business world is going, there’s a lot more demand for specialized skill sets. And so we have, in addition to MBAs, we have a Masters of Science in Finance, a Masters of Science in Sustainable Business, so on and so forth. And each of those sections afford some opportunity to take things in a different direction, really, not even just an opportunity, but we kind of want to, to be more responsive to those fields. And so in the graduate sections, we’ve had some isolated ability to experiment with more problem-based learning, which I think ChatGPT goes really, really well with on the faculty end as far as creating problem-based learning curriculum. But we haven’t experimented with the AI component of it yet, really, because it’s so new, and it doesn’t feel like it right now…it kind of feels like it’s 20 years old, but yet haven’t used it. But it’s very new. And so I don’t know about every other institution, but we don’t move at the pace of jets when it comes to curriculum revision at the University of Miami. We move, I think, faster than probably most but still, it takes time. And so we haven’t had the opportunity to do anything with the AI yet. But we’ve revised in the past couple years to focus more on some of the problem solving in some of the graduate sections.

Rebecca: The faculty member in me heard I can use ChatGPT to help me with problem-based learning classes, and I want to know more about that.

Don: Oh, yeah. [LAUGHTER] So if you type into ChatGPT, you have to give it really, really good direction: the who, what, when, where, why…that you are a professor teaching a negotiation class. And it is a upper-level, undergraduate course. And you are going to create simulation practice for negotiation in which you play one role and the student plays the other, and you will create a scenario and interact with the student, but wait for the student’s response after each of your responses. And then at the conclusion, give the student feedback based on what you know about the science of negotiation from a management sciences perspective, as well as a legal perspective. And then you hit go, you will be blown away with what ChatGPT starts to create. And so it will give you a little blurb. A couple of weeks ago, I did something of the sort, and it said, Sally is the owner of a handmade furniture manufacturing company in North Carolina and has been contacted by so and so that owns a furniture retail store. And so and so has been impressed with Sally’s furniture and wants to arrange a distribution agreement. The meeting begins over the phone and so and so ask Sally what her goals are in this arrangement. And then that’s where I would type in and I said my goals are to reach this level of profitability and to have a productive long-term relationship with the other party, and it responded back. So it can create an entire dialogue that you can then ask afterwards, once you tweak it and say, “Well, I liked this part of it. I didn’t like this part of it, write the Python code for this, and it will write the whole Python code and allow you to turn it into a web-based interactive program. It’s really quite wild.

John: So basically, it gives every faculty member the ability to create interactive simulations for their classes, which could be done for pretty much any topic I would think.

Don: Absolutely. In the past that kind of thing was in some courses, probably a bit aspirational. It’s the kind of thing that would probably require some kind of course leave to develop it. And for faculty who become really comfortable with it, it will get to a point where it’s doable within a day or two of a lesson. And so you can on kind of miniature scale, you can do these on a daily basis, really high quality ones.

Rebecca: It sounds like something that we can use in a lot of contexts in higher ed, including if we want to do simulations for interviews for new positions or other things as well, if you’re trying to better understand how someone might approach a problem.

Don: Absolutely. I think that’s a very good application, in fact.

John: One of the things though, that I think has generated some panic for a lot of faculty is the effect that this may have on how we assess student learning. So how can faculty address issues related to the ethical use of artificial intelligence?

Don: Well, I’ve never known any faculty to ever panic over a technological innovation…sarcasm ended. So I think faculty have to assess this on their own, but also part of the community. One thing that I think’s going to be an early problem are faculty doing things in a different way, I think that’s probably unavoidable. And so I say all that as kind of a disclaimer that my approach and what I think our approach is going to be in my department, and even if the disclaimer applies to that, I don’t even know that for sure, is perhaps going to be different than others’ approach. And so since this course is supposed to be so much of a practical course, and the writing is on the wall, no pun intended, well, it’s in the AI software, I view that we really have no choice. And so there’s been a lot of commentary in The Atlantic magazine, a lot of commentary in the higher education journals. And most of that I have seen focused on this question, but using as an assumption that it’s wrong to use ChatGPT. And so the easiest way to make it not a question of cheating is to allow it to be used, and then it’s not cheating. And so that’s the direction that I’m leaning in. And I think, ultimately, for the practical tools, for the practical courses, that’s going to be the direction it goes. But again, I can’t even speak for my own department on that, because we’re so new in this.

John: And that will be an issue, I think, everywhere, as it has been in the past with things like calculators, or smartphones, or even Apple watches, I remember getting all these memos coming in from various places at one point to make sure your students are not using a smartwatch while they were taking an exam, because somehow the answers are going to miraculously appear on that tiny little screen for the test that you’re giving them.

Don: Right. And I think you can’t really separate the assessment design and the student response to the assessment in this. There are going to be some courses, I can imagine, in different disciplines, that they’re focused on more fundamental foundational skills, that it’s going to be more of a challenge for them. Well, I’m not saying that students don’t need to know and learn about the aesthetics of writing; that has to keep happening, but not in this course. And so I don’t know how the faculty in those spaces and really the 9th and 10th grade composition teachers that I talked about before, I don’t know how they deal with it, probably in person assessments, that sort of thing. But for this practical application course, I would view it really as kind of training track runners to hop on one foot. And so that wouldn’t be very practical. And so if you have a cheating or plagiarism or honor code policy that requires them to only use one foot, then it would be plagiarism for them to run on both feet. But that wouldn’t be very helpful. And so I really viewed ChatGPT as the same thing in a practical sense, if you’re plagiarism or honor, code policy defines ChatGPT as out of bounds, you’re training them to run on one foot.

Rebecca: So we’ve talked a lot about the writing component, and really building in stronger structures to focus on critical thinking. One of the other issues that you identified was that students don’t necessarily see the intrinsic value of the course or like get the buy-in. Can you talk a little bit about the ways that you’re redesigning to help with that piece of it as well?

Don: Yeah, and so certainly a lot of students do get that. But it really depends on how intrinsically motivated they are. And I think it requires faculty to kind of sell it somewhat. And so in my courses, I’ve found success with selling it in that way, which I really don’t like to do. And it’s something that a lot of faculty probably think is kind of an icky thing to do. But for instance, I will repeatedly tell students, I’m not here to make myself feel good. I’m not here to make myself feel smart by putting you all down. I’m here to help you all get jobs and to get promotions at those jobs and do well in your careers. And so I will focus a lot on kind of pointing things out as criticism that I also tell them these are not affecting your grade, however, X, Y and Z. And so little things like in a presentation if they go…you know, we have to have time limits for presentations because it’s basic math, we have X number of students, 75 minutes in a class session, we have to have time. And so when it comes to a student has five minutes and they go over, what do you do? I’m not going to take off on grades for that. But I am going to point out for a student that, in some settings, if you’re given a time limit, that’s because the CEO has another meeting five minutes after you start and you will be cut off, not because they don’t like what you’re saying not because you haven’t followed the directions, but because they’ve got somewhere else they need to go. And so a lot of the problem I think, is just students are so focused on grades, to the shock of everybody, [LAUGHTER] that when the things that you’re grading and are affecting their grades are these kind of…and aesthetic isn’t the right word, but they would view grading something like that as a bit ticky tack. And when you’re scoring things like that, it’s much harder to have a serious conversation about the nitty gritty substance, and how, if you’re trying to prove that this course of action is more profitable than the others, and you didn’t provide any support for the change in costs, you really can’t have accomplished your goal. You don’t get the same attention from the students in the same response, if you’re also talking about things like, well, “you went five and a half minutes when you only had five minutes,” or “You didn’t use 10 point font when you were told to use 10 point font and that ChatGPT, with that second example with that 10 point font, if the instructions said 10 point font and the students input the instructions it produce it in the appropriate formatting.

John: And I know in the past, when I’ve graded student papers, I, as many other people do, spend far too much time correcting grammatical errors, reminding them that there’s a difference between singular and plural or the difference between all the various homonyms out there. Might be easier for us to evaluate student work when we can actually focus on the arguments they’re making, and their ability to engage in critical thinking, rather than getting ourselves so tied up in all this minutia, which I always try to avoid doing, but when I see so many errors in student work, it’s hard not to at least correct some of it so that they could become more proficient. In the future, they may not need to have that type of correction.

Don: Yeah, John, I think you really hit the nail on the head there. You really feel like you have an obligation to correct those. And when communication or writing is one of the titular topics of the course, even more so. But I have always felt that you get diminishing returns on the things that you focus on. And every time you’re talking about grammar, you are not talking about the critical thinking, and the grammar does matter. I can tell you that I have lots of conversations with CEOs with HR directors, etc, in the ongoing effort to make sure that my curriculum is responsive to what’s happening in the market. And one thing that I consistently hear is grammatical errors, spelling errors on slides or in cover letters are catastrophic. And it’s not because they’re nitpicky, it’s because the markets are so competitive, that they get a window that’s maybe 5% or less of what someone’s actual quality as a candidate may be. And that’s just something that there’s going to be some other candidate that is just as qualified and equal in every other way that didn’t make grammatical mistakes in their PowerPoints, and so on and so forth. And so it is important, but it doesn’t matter how good your grammar is, how compelling your vocabulary is, if you are missing some of the logical components of the argument, you cannot be correct.

John: And when students get feedback, where they see dozens of comments on it, the easiest strategy is to focus on correcting those small grammatical errors that are riddled through it. You might also have told the student that they don’t have a very substantive argument. But if they’re going to make a lot of corrections, it’s easier for them to focus on correcting the grammar and ignoring the more fundamental problem.

Don: Right. The very first writing assignment I ever did as a graduate student was a 50-page memo and I got back no comments anywhere except for on the front page: “Do you talk like this?” I think probably somewhere [LAUGHTER] in between those two is ideal. But you’re exactly right, that the more that we focus on things like grammar and tense and such things, the less we can focus on the meat and the critical thinking.

Rebecca: It’s funny how that often is the level of polish would be something that goes from someone that’s got like a really high grade to like an excellent grade.

Don: Right, exactly.

Rebecca: Something that’s foundational, that’s often not how our feedback structures work. And even if we keep form and function feedback separate and even weight them very differently. It’s really easy to address the form issues, because it’s almost like a series of checkboxes and it doesn’t require a lot of thought because the critical thinking part’s the hard part. And so it’s funny that even if they’re weighted differently, and to keep the comments separately, students will always flock towards the thing that’s kind of easy to fix. I mean, who wouldn’t? Then it becomes a checklist.

Don: And that’s exactly really kind of where this boils down to me, it’s not to say that those things aren’t important. They’re still very important. But in the world in which ChatGPT is a real thing, which it now is, and will continue to be and only be more powerful than it is, the juice that we get out of spending time in class or in feedback, in office hours, whatever it may be, talking about those sorts of things, is getting much less of a return than it did before ChatGPT. I am not a walking detector of 100% perfect polish by any means. But it seems to me that the product that ChatGPT can produce, in terms of those things that you were speaking about, Rebecca, is pretty dang good and hard to distinguish for me from highly polished products. But again, where it is easy to distinguish is this is a load of crap [LAUGHTER] that is fluff and has no substance to it, but a very polished load of crap, but nonetheless…

Rebecca: It’s pretty crap. [LAUGHTER]

Don: Exactly. It’s very pretty crappy with a nice bow,

John: …which reminds me of some work that I graded just the other night, where spelling and grammatical issues have mostly disappeared in student essay responses since the advent of ChatGPT, but the substance is not always there. And there were many responses that I provided feedback on which said, “this is a really nice response, but not to the question that you were asked to address.”

Rebecca: Yeah, or you spent two paragraphs and you haven’t actually said anything yet.

John: So teaching students how to use ChatGPT or other AI tools more effectively might allow them to be more productive in their learning as well as beyond their college experience.

Don: And might allow us to make for more productive learning environments as well.

Rebecca: So we’ve talked a lot about course content, and what to maybe focus on and not focus on. One of the most important things a course has is its syllabus or course outline. Can you talk a little bit about course policies and the way that you might make change in that realm?

Don: Yeah, so I think you’re gonna have to be more detailed than you probably are used to being in terms of putting language and syllabi, very specific and upfront. And so some of the policies that I’ve seen that I’ve liked elements of and are going to end up including in the syllabi the explicit weaknesses of ChatGPT. It is not a critical thinking device, it will produce responses only as deep or as shallow as you instruct it to. You are still responsible for the critical thinking, essentially, and very explicit in terms of what’s allowed, what’s not allowed. And I think also, it would probably be a good idea for faculty to be putting in explicit language that what is allowed in this course, is not necessarily the same as what will be allowed in other courses, and it is incumbent on students to navigate those differences themselves.

Rebecca: And part of the reason why things might be different across courses is because the focus of those courses is different and really helping students understand that there’s reasons why policies might be different in other classes. It’s not necessarily arbitrary.

Don: Right, exactly.

Rebecca: So we always wrap up by asking: “What’s next?”

Don: Well, what’s next is I figure out how to do all this stuff, [LAUGHTER] and not just to talk about it.

Rebecca: …and you’re gonna send us a memo, right with that in it. [LAUGHTER]

Don: Oh yeah. Yeah, I’m happy. [LAUGHTER]

John: …or at least have ChatGPT generate a memo explaining….

Don: Exactly. So yeah, what’s next is to put this stuff into action. Of course, as I mentioned, some of the things here have already been experimented with, the non-ChatGPT parts of it at least, but really kind of integrating them and seeing if what I am imagining is what comes to fruition in terms of do these things dovetail as well as I think. I really think that they do. …that kind of pre-existing urge to go more towards the critical thinking element, and really, I think, does dovetail well with the AI, but putting it into practice, it will be over the course of probably all of next year. And so there’s going to be some experimental sections, most of the sections are probably not going to look very different than they did in the spring. And I think that’s probably a very good plan. But there’s going to be some experimenting in some of the sections at the undergraduate level, and part of a faculty learning community on problem-based learning. This course is going to be participating in that in the fall. And so a lot is going to come out of that, I think, as well.

John: Do you think there’ll be much buy in from other people teaching the course?

Don: So, students, by and large, do not like writing. Faculty, by and large, do not like grading writing. And so I don’t think this is one of those political monsters of how are we going to get this through? How are we going to make these changes work? I think there’s probably a lot of people who have nervousness about how you would make these changes. But with those two facts that I don’t think you’d get much disagreement from, I think even across disciplines, I don’t think it should be that difficult for this to be implemented

Rebecca: Well I hope you’ll join us after you’ve implemented some of the things to share some of your reflections and let us know how it went.

Don: I’m happy to.

John: Well, thank you, Don.

Don: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

303. Higher Ed Then and Now

Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, Todd Zakrajsek joins us to reflect back on the changes that have occurred in higher ed during our careers.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson.

Show Notes

  • Zakrajsek, T. and Nilson, L. B. (2023). Teaching at its best: A research-based resource for college instructors. 5th edition. Jossey-Bass.
  • Zakrajsek, T. D. (2022). The new science of learning: how to learn in harmony with your brain. Routledge.
  • Harrington, C., Bowen, J. A., & Zakrajsek, T. D. (2017). Dynamic lecturing: Research-based strategies to enhance lecture effectiveness. Routledge.
  • EdPuzzle
  • PlayPosit
  • ChatGPT
  • Wayback Machine

Transcript

John: Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, we reflect back on the changes that have occurred in higher ed during our careers.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Todd Zakrajsek, and I am with Todd here in Durham, North Carolina. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson. Welcome back, Todd.

Todd: Well, thank you, John. Well, this is exciting. And Rebecca may be a long ways away, but I have never been arm’s length from a person who interviewed me for a podcast before.

Rebecca: Isn’t that cool?

John: And we’ve really done that before either at a conference or at Oswego,

Todd: I feel very special.

Rebecca: Well, we can celebrate with our teas. So, today’s teas are:… [LAUGHTER]

Todd: I’m drinking a peach mango that I got from some teas that John brought, which are fantastic.

Rebecca: John, how about you?

John: I am drinking a Tea Forte black currant tea, which I brought from Oswego, in a new mug that was given to me by Claire McNally, when she visited this area last week.

Todd: Love Claire, she’s fantastic.

John: And it has kangaroos on it.

Todd: Yeah.

Rebecca: And I can’t see it. Let me see it, John. Oh, that’s a cool mug.

Todd: It’s a good mug. I got a mug from her university. But I didn’t realize I should have brought it. So I feel bad about that. But it is a podcast. So I didn’t think about what it would look like.

John: That’s true, we generally don’t do a lot of visuals on here.

Rebecca: And I have a blue sapphire tea in my Tea Rex mug.

Todd: Well, that’s a nice mug,

John: We’ve invited you back to talk a little bit about how some of the changes you’ve observed in college teaching across your career have impacted how you teach today. When did your work in higher ed begin?

Todd: Actually, it started when I was a graduate student. So back in 1987. So there’s no reason to try to figure out how old I am. Now I’ve basically specifically dated myself here. I started teaching, I got to teach an introduction to statistics course. And I had so much fun that I taught again the following year. And by the time I left my graduate program, I had taught more courses in that program than any other graduate student had ever taught in the psychology department there. I really loved teaching right from the beginning, when from the beginning, very concerned about student learning, and just getting rolling.

Rebecca: What was it about the teaching, Todd, that really got you hooked?

Todd: Just watching the studentsis. it’s the same thing as it is today, when you have an individual who’s struggling with something, and suddenly they get it and you realize that they may eventually get it on their own, but you realize how much you’ve helped them to move that along very quickly. And facilitating the learning process, I just really love that. That doesn’t mean I was fantastic at it. But I really did love it.

Rebecca: Sometimes the things we love the most are things that we’re not great at to start with.

Todd: That’s true.

John: My experience was similar, actually, I started in 1980, with a course where I had a fellowship, so I didn’t have to teach. But there was a sudden shortage in the department. And they asked me to fill in. And I was planning to go on into research. But it was just so much fun teaching that I’ve never stopped.

Rebecca: I taught as a graduate student too, and taught the whole time I was there. But I started a little bit later in 2003.

Todd: Alright, so that was a couple of years later.

Rebecca: Just a couple.

Todd: Yeah, I had kind of a funny start, I will mention that when I first started that after the first semester of teaching, my students got almost all As and Bs. And the department chair called me in and he said, “I’m not going to have you teach any more courses.” And I said, “why not?” And he says, “Well, you give grades away like candy, we have to have better standards than that.” And I said, “Well, how are you basing that?” And he says, “Well, you know, we looked at the grade point averages.” And I said, “Well, how about if I bring in my final exam, and just walk through it, and then you can tell me how it could change to be more rigorous.” And so it was great. I showed it to him at the beginning. And like the bottom of the first page, the students had to calculate a statistical value, then I had them explain how they came about that number. But if they had used a different test how might it been inappropriately found and what the interpretation might have been, based on the fact that they had done it wrong with a different test. I thought it was important for them to understand how these things can change. The Chair said, “I can’t believe you have your students in the first class actually talk about various tests like that.” And I said, “Yeah, I did. Then we turned the page he says “You did nonparametric tests?” I said, “Well, yeah, we did parametric tests, but then I thought they should know the equivalent.” And he said, “We never do that.” And then he turned the last page and he said “You had them do a two-way ANOVA? You’re only supposed to go through one-way ANOVA.” I said, “Yeah, but we’d finished everything and we still had a week left. And I figured I might as well introduce the next concept to them. And so I showed them how to do a two-way ANOVA and they ended up with all As and Bs. So if you could help me in how to push their grades down and give them lower grades, I’m perfectly happy to do that.” And he then set me up with two courses the next semester, but it’s that reliance on the teaching evaluations is always funny.

Rebecca: Todd, it’s just funny, as we’ve gotten to know you through the podcast [LAUGHTER] it sounds so perfect that that was your first experience. [LAUGHTER]

Todd: Yeah, I’ve lived my entire career on the edge. [LAUGHTER]

John: And those sorts of arguments are still occurring in a lot of classes today about rigor and the need to keep grades lower.

Todd: Yeah.

John: They’re less severe than they were a few years ago.

Todd: Yes, but also looking at how well a person’s teaching based on student evaluations. I mean, we should be looking at authentic assessment. Some things have changed through the years, some things have not changed through the years.

Rebecca: Well, technology is one of those things that has changed.

Todd: Woosh, yeah.

Rebecca: Can you talk a little bit about what tech was like in the classroom when you first started and how it’s evolved a bit?

Todd: Yeah, I know you have some listeners who have been teaching for a very long time. So those of you have been teaching for like 30 to 40 years, just stop and think back about what it was like when we first started. For those of you who have been teaching like Rebecca since 2003, let’s just mention that technology back then was mostly pens and chalk and chalkboards. So back then, of course, there’s technology, there’s always technology, but we were using overhead projectors. This was long before the internet came along to really be used in the classes. LCD projectors were not out yet. Canvas, Blackboard, Sakai, all those learning management systems were not around. We didn’t have any of the ways to email individuals, you couldn’t email your students back then. And there was no ChatGPT to write your papers for you.

Rebecca: But there were calculators that could do all the work for you.

Todd: Yes, but this is the cool part. Back when I started teaching statistics, I’m glad you mentioned the calculators, huge debate back then was whether or not the students should calculate the statistical values by hand using the calculator, because computers had just come onto the scene and we could punch the data into a computer and have a computer run an ANOVA for you. Should you calculate it by hand? Should you run it to the computer? And there was a huge camp that said you should do it by hand or you will never understand a statistical value. And I said, “You know, we’ve got the technology there. Why don’t we have the students use the computer to do the mundane stuff, and we’ll have more time to talk about the theoretical and the important implications.” But even back then we were having the discussions about whether to use the technology at hand or not. Oh, and by the way, we are also hanging grades on doors. So we would figure out the grades, we’d tack it to the door, and then the students who want to know what their grades were for the class would swing by and look at the door.

John: And they were sorted alphabetically, to make it easier for people to find where they were in the grade list.

Todd: Yeah, it was great. We listed them according to their social security number, [LAUGHTER] which was a little different back then. And yeah, we actually did that back then. But as John pointed out, they were listed by number so nobody knew whose number went with whom, except, surprisingly, they were alphabetical on the door. So not only could you figure out Armstrong’s exam score, you’d get Armstrong’s social security number as well. Yeah, times have changed.

John: And it was also back in the day of dittos and mineos as well, which was the only way of disseminating information on paper.

Todd: This is so much fun. We’ll get to some real meat of this thing. But that walk down memory lane has some fun stuff too. The dittos…

Rebecca: I remember dittos, just for the record, okay.

Todd: Yes. So you probably remember, if you dittoed just before class, and you handed it out in class, the students would all pull the ditto up to their face, so they could smell the ditto fluid. And they got that smell. I was running dittos one time in the graduate student office, and I noticed when I looked down because it ran out of fluid, and I had to put some more fluid in, and I looked down and I noticed that the floor was kind of eaten away by this ditto fluid. And then… this is the best part… About a month later, I was digging for something in the closet and I found extra tiles and I thought they should put these tiles down to replace the ones that are all eaten and on the side of the box it said these tiles were long lasting and durable, reinforced with asbestos. So that ditto fluid was eating through asbestos tiles. That’s some strong stuff.

John: …to make it a little bit more friable so that it would disseminate in the air nicely.

Todd: Well, there had to be something to help the faculty members who were running all their own dittos to not mind doing it, and one way of doing this is to have them use ditto fluid, because I’ll tell you, you may not have liked it when you started, but by the end, it was all right. [LAUGHTER]

Rebecca: It’s funny that we’re taking this walk down memory lane, because on our campus, I was in our historic lecture classroom today in Sheldon Hall.

John: What are some of the other changes that have occurred and how have they influenced how we teach?

Todd: Yeah, so it’s interesting, I did the walk down memory lane and we were chatting about this stuff. It’s all fun, but thinking about how the changes have taken place. I think that’s really important. So there have been massive changes. I think that we tend to forget, it’s so easy to communicate with students now. Heck, people are texting now so that you can text back and forth with students. But think about how that has transcended or gone through time. There was a time when I would have to call and leave a message for a student on an answering machine, and then they would call back and we would try to find a time that we could talk on the phone. If we wanted to have a conversation. I could either leave a note for the student or I could call and leave a message that says least come see me after class. So even having a conversation with a student was difficult, then it became easier with email because you could start emailing back and forth. And now we have Zoom. And the equity in the way that this has changed, just think about the difference of this, if I’m leaving a message for a student, they may not even have an answering machine, if they’re living off campus with limited means back then. So even getting in touch with a student would be challenging. Now I can have a Zoom conversation with a student who doesn’t have to hire a babysitter, who doesn’t have to find reliable transportation, who doesn’t have to drive across town and burn gas, and to do all of those things that it would take to have a 15-minute conversation that in the past would have been really hard, and even four or five years ago would have been challenging. The grades, why in the world would a person have to leave… and I was teaching in very northern Michigan, there were days that the wind chill was 75 degrees below zero… and students would leave their dorm rooms and walk across campus to see a grade on the door. It’s actually physically dangerous. And now we have learning management systems, we could post things for students. Interlibrary loan used to take weeks to get a document that you can now go on and get. People can lament all of these technological changes at times, but we’re actually creating more and more equity within the higher education system as we make certain things easier. Not saying that we’re anywhere near an equitable system yet, but we’re moving in a really good direction. And a lot of those changes are helping us to get there.

Rebecca: I’m thinking about all the times when I get to go to the door or meet after class, it really assumes that students are a certain kind of student, they’re full time, they have time. And our students now are working [LAUGHTER], and where they’re juggling a lot of different schedules and things.

Todd: Yeah, and I mean, we want to be careful too. And I agree with you 100%. But they were juggling back then too. But some of the things we were doing, for instance, I taught a night class. Now I would probably suggest if I was going to teach a class from 7 to 10pm that I would teach it through zoom, because there’s a lot of reasons that it’s good to do. But I had students that I noticed in class, would very quickly at the end of class would start talking to other students and I couldn’t figure out what it was doing because a lot of buzzing and stuff. And what I found was that there were certain students who were uncomfortable, and we were in a very safe campus, but they were uncomfortable walking to their car at 10 o’clock at night. So I started saying to the students, “Hey, I’m gonna park a car… and when we showed up, there were quite a few cars there… but I’ll be under the second light, I drive a little red Chevette, not a Corvette, a Chevette, but I’ll have my car there. If you want to park near me, we can walk out together.” And there were students that were not paying attention to almost any of the class because they were fearful of how they were going to get to their car safely. When you think about Zoom and stuff, it’s even safety factors, I would never have a review session now like I used to at 8 to 9 pm the night before the exam because I’m exposing people to potentially dangerous situations. Now we’d have zoom sessions. But I could tell you 40 years ago, there was no even concept of what zoom would be and how it would work. Even Star Trek didn’t have stuff like that.

John: And there was also, besides the inequity associated with people who were working, many campuses had a lot of commuting students who could not easily get back to campus for office hours. Or if they were just taking classes on Monday, Wednesday, Friday and your office hours was on a Tuesday, they’d have to come in that extra day, arranging childcare, or their work to be able to fit that into the schedule.

Todd: Yeah, it really did start to change that system. So we got a little bit more equity, and like you were saying too, the commuting students, the part-time students, the students taking distance courses. When I first started teaching, I was writing… oh my word, remember the correspondence courses? …and you mail away and get a packet of material, you take a test at a local library and, and they talk about distance education being not as good as on campus, but at least better than nothing. And now we’re finally getting to a system where we can stop assuming that those folks who are coming in for part-time courses and stuff are just getting something better than nothing. They’re actually getting something similar to full college courses, which some of those online courses are actually as good or better than college courses that are on campus. But all that’s changing with the technology. It’s crazy.

John: And there’s a lot of research that supports that in terms of the relative learning gains with online and face-to-face, as well as hybrid courses, which seemed to outperform others in a few meta studies that have been done. But those were options that just weren’t available back then. And the early online courses were often designed to be replicas of face-to-face classes, and they probably didn’t work quite as well. But we’ve learned since that, which brings us to the issue of research. During the time that you’ve been teaching, there’s been a lot of research on teaching and learning. While some of it was taking place, it wasn’t very widely disseminated to faculty.

Todd: Yeah, that is true, too. It’s so much easier to get technology out. It’s easier to gather data, it’s easier to write it up. It’s easier to edit it so all of those types of things that are happening now that couldn’t happen before. And as a result, we’re learning a lot more about how people learn, you know, the book I did on the New Science of Learning, looking at a lot of the ways that students learn. And part of it’s just the ease of getting to information. But also part of it’s just being able to investigate how people process information. I used to teach Introductory Psychology back then, we would talk about the stages of sleep. And nobody really knew, for instance, what REM sleep was about, we knew that you had to have it or else it caused some problems. Deep sleep we knew was important, we now have indications that deep sleep for consolidation is necessary for semantic memory. If your sleep is interrupted, you can get eight hours of sleep. But if you don’t get deep sleep, the information doesn’t get consolidated. Procedural memory, how to give shots and kick balls and do anything procedurally looks like it’s more solidified during REM sleep. So again, the different types of sleep are associated with us learning long term, different types of information. We never knew that before all this technology was running around. In fact, back then I gotta say, I remember from my intro psych class being told that you were born with a certain number of neurons, and as you live through life, neurons would die. And if you killed them by drinking or doing something like drugs or something, they were gone forever, and you would never get more. And if you broke a connection, it was broken forever. That’s just simply not true. But it’s what we thought back then. So technology has really allowed us to look better at how people learn, different ways of helping them to learn and different ways they can even study. By the way, before we move on, we now have this physiological demonstration that staying up all night and cramming the night before the test. Even though it gets you slightly higher grades on the test, we now know that because the information is not consolidated that it won’t be there a week later or two weeks later. So we’ve always told students, you shouldn’t do it, but now we can actually show them why it doesn’t work.

John: And the LMS itself has offered a lot of ways of giving more rapid feedback to students with some automated grading with some things to give them more low-stakes testing opportunities. And those were things that we just couldn’t easily do back when you started teaching.

Todd: No, John, that’s a really good one. And we know that one of the most consistent findings right now in all of learning and memory stuff is that the more often you do something, the easier it becomes, long-term potentiation. Which means the more frequently you retrieve information from your long-term memory, the easier it is to retrieve. And just like you’d mentioned, we can now do LMS systems that are set up so that you could do practice quizzes, you could do dozens or hundreds of practice quizzes and keep pulling that information out over and over and over again. That was just not possible before this. And so the LMS helps with that, it helps by giving feedback, really good feedback so that students know what they’re doing well, and what they’re not doing well. And it helps faculty members to design feedback specifically for certain types of projects, and so that I can more easily give more feedback without spending a lot more time on it. So LMSs have done a tremendous amount of work. And that’s not even mentioning the fact that you can have all of the LMS systems loaded with the content. So students can log in and get their information without leaving their house. If there’s fiscal challenges with your class, you can put in articles, the students may not have to buy a book, they could read the articles. And so we’ve got students who were able to come to classes because they can afford to be there. By the way, I remember being on a committee when I was a graduate student, and we were looking at financial aid and different financial systems. And I remembered asking the Chief Financial Officer, I said, “What increase in tuition does there need to be before you start to see students drop off because they can’t afford to be here?” And this was about 40 years ago, but he said $100 for a year, if they have to pay $100 this year more than last year, some students won’t come back. If we look at the price of textbooks now, textbooks can cost $400. So, a book like that is definitely going to make a difference between some students being able to take the class or not. So LMS systems make this possible.

John: And they also make it easier to share OER resources that don’t have any cost for students, or some less expensive adaptive learning platforms, giving all students that first-day access. I remember, not so long ago, when I was still using textbooks in some classes, students would wait several weeks before they got that book. And that put them at a severe disadvantage. And the people who were being put at a disadvantage. were generally the students who came in with the weakest backgrounds because they came from lower resourced school districts.

Todd: Yeah, if they had the resources, they would have the better background foundational material, but they’d be able to buy the books. And you mentioned OERs. So open educational resources are really another thing that are really valuable because back then, before the technology, you couldn’t produce something that would be readily available like throughout the world. And so this project that’s going on now where they’re doing introductory level books in all the different disciplines, you can get an OER introductory psychology textbook that students can log in and read. None of that was possible before the technology. So even the creation of OERs has changed so much.

Rebecca: Well, speaking of digital materials, libraries have changed significantly too over time from having completely physical collections and interlibrary loans and things that take a lot of time to having a lot of digital resources, which changes access to research and materials that you can populate into your classes, but also can aid students in the work that they’re doing. Can you talk a little bit about the change in libraries and how that’s impacted how you’ve taught?

Todd: Yeah, you know, libraries have been fascinating to watch over the last 40 years, because it used to be the biggest challenge librarians had before them was which books to put on the shelves because there was a finite amount of shelf space. And there were lots and lots of books. And so that was the big thing. We used to take out journals that weren’t used very much to make room for other journals. Through time, little by little, they started digitizing all that stuff. And I can remember chatting with librarians, one conversation I had was back around 2001. I said, gonna be interesting, because there’s gonna come a day where there’ll be no books in the library, and the Dean of Libraries said “Well, there’s always going to be books.” I said, “Not always, potentially.” But even if we reduce them, I said, “What is your foresight? How is the library going to change?” And so he had a couple of ideas. But what it basically boiled down to our conversation is, I always felt like a library was like the brain of the campus, it had the books, and it had all of the information that you could go and get. As the books left, and things were diversified in a way that you could find this stuff, you could get all the information right from your dorm room, or from your apartment, when the internet came along, you could get anything you needed, then the library was still a physical space that was in the middle of campus. And what it should become is a learning commons, a place where people go to share and to learn from one another. And I think that’s what’s really changed is individuals still just pile into libraries and use the space, but they use it in different ways. They go there to meet other individuals to work, which they did before. But they took away that aspect of going there for the book part. And it meant all of those shelves got emptied, and they started pushing them out. And you can go into libraries right now that have very few shelves. But they have webcams, they have smartboards, they have spaces where folks can plug in their computers and share with one another. They’ve got screens set up so that you can project and have students sitting around a table, they’ve got Google Glass set up, all of these types of things that bring students together to use technology to learn from one another.

John: And they have cafes to help support that to make it easier for people to gather.

Todd: Yeah, you could swing by and get a cup of tea.

Rebecca: It’s funny, even when I was in high school, my sister and I would rely on going to the library to have access to a computer so that we could even type of paper, because we didn’t have one at home. And that kind of place of having the technology started a long time ago, but it’s amped up quite a bit over the last 20 years.

Todd: Yeah, and I agree completely. And the computers that are there. I mean, even right now, with the books dissipating, there a’re still large numbers of computers. And oftentimes, they’ll even be an area in a library that’s carved out with really high-end computers. But it gives students an opportunity to go. We make this assumption that everybody has a computer and they don’t. But libraries give them that opportunity.

John: Yeah, for those students working on smartphones or Chromebooks, that gives them access to all the tools that students with $2000 or $3000 or $4,000 computers.

Todd: Yes, because smartphones can work for lots of things. But they’re a little tough to write a paper on

John: When I started teaching, and probably when you did too, the predominant mode of instruction, which actually still is often the predominant mode of instruction in many departments, was lecture. That’s changed quite a bit since then. Could you talk a little bit about the shift from lecture-based courses to courses that involve much more active learning activity?

Todd: Yeah, or they just involve a lot more of everything. The concept of flipped classrooms, which was almost impossible 30, 40 years ago, because you really couldn’t get the information to the students. Yes, it was kind of possible, but whoo, if it was hard now, it was really hard back then. But the ability to get information out to students that they can read it before they come to class. But coming back to the lectures… So I’m going to take this moment and those of you who know me know that I’m going to do this, is that we still have no evidence that lectures are bad, but there’s something that we need to really keep in mind. I think this is vital. I do think it’s important for us to be able to talk about buzz groups and jigsaws and fish bowls and lectures and Socratic lectures, discussion lectures, all those different methodologies out there so that we know what we’re talking about when we chat with one another. But I do think it’s time that we stop talking about lectures being more effective than one thing or fishbowls being more effective than something else and look at the components of what is valuable in a learning experience.

John: And a good reference for that is a book on Dynamic Lecturing, which you happen to be a co-author of.

Todd: That is true and in fact that there’s the Dynamic Lecturing. And then there’s a chunk in that about The New Science of Learning. And then there’s a whole chapter in that about Teaching at its Best, because that’s a good point, John, thank you.

Rebecca: It’s almost like you’re trying to slip it in everywhere you are.

Todd: Because the research… people keep talking about one methodology being better than another. Here it is, folks, you can be a hideous lecturer, you can be a phenomenal lecturer. And if you’re a hideous lecturer, you’re not going to learn anything. If you’re a phenomenal lecturer, students will learn from you but they won’t learn all the time, it depends on some student factors. I’ve actually been exposed to group work in flipped classrooms that were awful. And so that concept is we start thinking about and this is why it’s going to come back to the technology, we think about the elements that need to be there, that are necessary for learning to take place. I’m just going to do this, because it’s not the topic I’ll make it very brief, is let’s just go with three things. If you don’t have your attention, as a teacher, if my learners aren’t attending to what I’m saying, if they’re on their phone or thinking about bacon, then they can’t process what I’m presenting. And if you’re having a think-pair-share, if they’re not attending to the person they’re sitting next to, you have to have attention. Number two, they have to have some value. If I’m hearing somebody or I’m reading something, and this has no value to me, it’s really hard to get it into your long-term memory and to learn it. And number three, I have to have a clue of what’s happening, I got to understand some aspects. Now if we think about attention, value, and understanding, now we can flip back to the technology. This is why gaming works. Gaming draws the attention, it increases the value, because you want to win the game, and it has understanding. We have all played games. You open up the old board games, and now it’s digital, where you don’t have a clue what the game is. It’s like, if you advance a player four pieces and the opponent advances five pieces, you have to go back three spaces, unless it’s a Tuesday. When those instructions are that complicated, you don’t understand. So we can use technology to help with attention, we can use technology to help with the value of what’s going on. And we can use that technology to help with understanding. Those are things that were very difficult before. And they allow us to do things like a mini lecture and then shift over to an active learning exercise, and then say, take all this information and create a Zoom session tomorrow that will go over it again. So the technology has really helped us to be able to do all of these things to get at the core of learning, a topic I barely care about. [LAUGHTER]

John: That’s an important one, because people often see this as this binary issue where you lecture or you use active learning. And there are some really effective ways to combine them. And in fact, in that book on dynamic lecturing, it was suggested that lecture can be more important in introductory courses, when students don’t have as much of a knowledge base.

Todd: You’re absolutely right. Discovery learning is a really great way to learn if you’ve got a lot of time. I can just put you into a room with some other people and say, “Here’s some data, and here’s some things we need to know. Go.” And if you don’t have any foundational knowledge at all, it takes forever to figure it out, you go online, you know what to look for, I could do a five-minute lecture, and at the end of five minutes, set it up and say, “Now go and work with your neighbors. In fact, here’s what we’re going to do, we’re going to have you each work in small groups in class, I’m going to open up a Padlet. At each table, I want you to go in and add your information or put it into the column that corresponds with your group number.” As an instructor, I can watch everything develop in front of me. While I’m in the room, I can look at my laptop and see it and walk over to a table and say, “looks like you’re struggling a little bit.” I’ve lectured, I put them into small groups, I’ve had them use technology, I’ve created a little bit of competition on who can come up with what and I’ve had a way for me to monitor it and give them feedback. That is so different than what teaching used to look like. So pulling it all together, that’s what we do.

Rebecca: The tools to be able to monitor have been really helpful in my own teaching and being able to get a better pulse on what’s going on and get a nice overview and then be more targeted in how to interact with small groups rather than just kind of wandering around more aimlessly like I think I did initially. [LAUGHTER]

Todd: Yeah, and this is all going to be great until we get our cognitive load headbands that I’m waiting to be developed. So anybody who’s listening, take this idea, run with it, you can make a bazillion dollars and then take me out to dinner or something. I want a headband and the headband has a light and it measures brainwave activity. And then as I’m teaching, if you start to be a little bit like it’s a little bit too much, you’re moving out of that zone of proximal development, the light turns from green to a yellow. And then when it hits red is like when you’re trying to put together an Ikea bookcase and someone comes by and says “What do you think of this?” and you say, “Errr, I’m working on an Ikea bookcase right now.” …that shutting down with that red light. I’m telling ya, that’s going to be the technology we’ll want next.

Rebecca: It would be so helpful. [LAUGHTER]

Todd: You can actually look and see somebody else’s zones of proximal development and their cognitive load. Whoof. Which by the way, there’s a little party game that they’ll do periodically at parties. It’s like if you’re a superhero, what would you want your superpower to be? And I was in a room one time and one person said they wanted to fly and somebody else said that they wanted to be invisible, which real quickly in my head, I’m thinking, what could you possibly gain that wasn’t illegal or creepy if you’re invisible. So aside from that, transporting and everything else, and they got to me, and I said, “I want to be able to see people’s zones of proximal development. If that were my superpower, I’d be the best teacher.”

John: I bet that went over really well at those parties. [LAUGHTER]

Todd: Yeah, my friends all said “You are amazingly smart and quite insightful.” They used different words, but that’s what I heard. [LAUGHTER]

Rebecca: They didn’t start with what is that? [LAUGHTER]

Todd: As soon as I start talking, most of my friends just shake their head and drink whatever beverage they have near them. [LAUGHTER] So yeah, it’s good times, good times. They’re all impressed. They don’t say it all the time, but I know they are.

Rebecca: I think one of the things that often happens with technology is that it allows us to get things quickly and move through things quickly. But sometimes, as you just noted, learning doesn’t happen quickly.

Todd: Yeah.

Rebecca: Can you talk a little bit about speed and the difference between maybe not having all the technology and all the things really quick versus maybe now where we have it at our fingertips, but do we always want it at that speed?

Todd: So there’s another study that I’m waiting to see. This is an easy study, folks, somebody can run this one quickly. We all know that students are listening to any recorded lectures or recorded material that they have to watch, 1.7 is about the best speed that we tend to see people listening. 2.0 is a little bit fast for some folks. 1.0 is like normal speed, that’s no good, too slow. So what I’m curious about is the space between words and between sentences that our brains, because they move so fast, we can listen faster than somebody can talk. And we have all this other stuff going on is I can be thinking and processing while you’re talking to me. But if I bumped that up to one seven, I think we close the gaps. And I hear it a lot faster. But what I don’t think is happening is the cognitive processing while I’m listening. The active listening component to it. So I think technology can create concerns in those directions. And students who do try to process material too fast… we’ll wait and see.

John: And that’s especially important in flipped classrooms where students do watch these videos outside. One of the things I’ve been doing with those, though, is embedding questions in the video so they can watch them as quickly as they want. But then they get these knowledge checks every few minutes. And then if they find they’re not able to answer it, they may go back and get their attention back and watch that portion again.

Todd: Yeah, I think that’s a really good way to go. EdPuzzle’s kind of a fun technology to use. I don’t know if that’s the one you used.

John: I’m using Playposit, which is a bit more expensive, it works beautifully. I love it, they did just double the price this year, though, it was bought by a new company.

Todd: This is the tricky spot now as the prices are going up. You know, inflation is a terrible thing to waste. Anytime somebody can raise prices now it’s like, “ooh, inflation”. So you know, prices double, inflation is 8% with runaway, now it’s back down around three. But when inflation was 8%, they doubled the price and say, “Hey, we’ve got to,” but yes, it’s some of them are expensive. There’s lots of things that are less expensive. Oftentimes we pay for functionality that help us but the freemiums kind of thing. So stuff that’s inexpensive. I just wanna let everybody out there know just about anything you want to do in class or can think about doing it, there’s a way to do it for either free, or probably under $100 a year, which I know $100 can be expensive for some people, it’s about eight bucks a month. And so things like Padlet that I think might be up around 140 now, so maybe $12 a month, can change how much time you spend doing things, and how much time for students. But yeah, I love the embedded questions to help slow things down.

Rebecca: I think that the cognitive load can happen really quickly if we’re piling lots of information in but not always providing the time to process and use that information in some way in the kind of activities that you were talking about. Or knowing when everybody’s red light is going off in the class.

Todd: Or when people try to do multiple things. I mean, now you’ve got the technology around. So if students are trying to listen to an assignment while they’re texting their friend and have a TV on, I mean, we’re living in an age where there is a lot going on, and people believe they can process lots of things. Evolution doesn’t happen quite that fast. And so I think we have to be careful with that one.

John: One other thing that’s happened is back when you and I both started teaching, the only way students generally communicated their learning was either on typed pages or on handwritten notes. Now we have many more types of media that students can use. And also we’ve seen a bit of an expansion of open pedagogy. How does that help students or how does that affect student learning?

Todd: Wow, that’s really changed a lot as well. Blue Books. Remember the blue books? I think they still sell blue books in the library. They may cost more than the I think it was eight cents when I started, but the concept of writing things down, you turn them into the faculty member, the faculty member would grade them and turn them back. One of the big things that I caught years and years ago was so much wasted cognitive energy in terms of what they produced. I’d read a paper from a student and think this is amazing, and no one will ever see it. It was written for me, I graded it. And now it’s done. I think the technology has changed so many things. One of the biggest things, I would encourage all the listeners, any faculty member out there is, whenever possible, create something that will take the students’ work, the things that they’re doing, and use it to make society better. It’s not that hard. There’s assignments that you can do on Wikipedia. Anybody who wants to complain about Wikipedia, if you don’t like it, I’m gonna go back to Tim Sawyer, who is a faculty member of mine, my very first time I ever did TA work. I was complaining about some students. And he said, “You can complain three times. And after you’ve complained three times, either stop talking,” he was a little bit ruder about that, “or do something about it… just shut up or do something.” And so I complained about Wikipedia for a while, that it wasn’t all that effective. And I thought, well, if I don’t like the page on cognitive load on Wikipedia, I could give an assignment of my cognitive Psych class to go on to Wikipedia and fix it. And so you can have Wikipedia assignments, there’s so many things you could do. Here’s one for you. If you’re doing one on communication, you could have your students go and take pictures or short videos somewhere on campus of something that’s meaningful to them, and then jot down why it’s meaningful, take that compilation of stuff and send it over to the office on campus that does publicity. What better way of drawing students to campus than to have all of these students that have said, I love sitting by the pond because… and in the past, we would have had students write a paper about someplace on campus that you think is effective, put it in the blue book, we would grade it, we turn it back to the students. And that is a waste of possibilities. And so I think we do have lots of ways that we can get the students involved in helping through technology,

John: One of our colleagues in SUNY, Kathleen Gradel, had an assignment for a first-year course, where the students went out, took pictures, geocoded it and added it to a map layer that was then shared with other first-year students about useful resources on campus and their favorite spots on campus, which is another great example of that type of authentic learning.

Todd: Yes, for the authentic learning, there are just so many possibilities because of the technology. If anyone doesn’t have ideas, ask deans, ask the provost, ask the president on your campus, like what kind of information would be helpful, either for the next round of accreditation or for just helping the campus and we can design those things. Another one I did was we took students to the museum. We’d go to the museum, almost any class could kind of find some way to tie museums in, and through the museum, not only would they write stuff that the folks at the museum who did curation would help use, but also just helping the students to see how issues from the museum, how artifacts and things can be used in their own life, to better understand.

Rebecca: When I first started teaching, community-based learning was popular, in fad at the time, and I think having the experience of being a student in a class like that, but then also a faculty member teaching classes like that has really informed the kinds of projects that I do. Maybe they’re not always community-based learning, but they’re often community oriented, whether it’s the campus or even the surrounding community that the campus is situated in to help students get connected. There’s so many nonprofits that need partners and love, there’s always a project that can be done. [LAUGHTER]

Todd: There is. And I used to be a director for a service learning component of the campus. And yeah, there’s just so much out there that we can do to help others.

Rebecca: And students always had such a strong connection. And they didn’t want to fail because other people were depending on them. And so there was a real investment in the work that they did on projects like that.

Todd: I will admit that I’ve never experienced it myself. I’ve never even heard of anybody that if the students are doing some kind of authentic learning, that their authentic learning is then used to help somebody else. I have never heard students say “What a waste of time” or “I hate that class,” or “those assignments are just busy work.” They’ve never used those terms.

John: One common sort of project is to create resources that could be shared with elementary or secondary school students in the disciplines. And again, they can see the intrinsic value of that.

Todd: Yeah. Students could write short manuals on how to learn and then pass that on to the first-year students. And so upper-division students could be helping the lower-division students because not everybody can get a copy of The New Science of Learning, third edition.

John: …available from… [LAUGHTER]

Todd: Available at… used to be Stylus. Since Stylus was sold to Routledge, now it’s available at Routledge. [LAUGHTER]

Rebecca: Given the historical background that we’ve walked through today, what if we think about the future? Where do you see technological changes or learning theory changes impacting the future of higher ed?

Todd: Yeah, we’re living at an interesting time. I like to point out to folks that when you go back to Socrates, Plato kind of time there was a thought that if you wrote something down, it would weaken the mind so we shouldn’t write things down. Luckily, some individuals wrote things down or we never would have known. We’ve gone through several iterations of those kinds of things. Samuel Johnson, I believe it was, who said “With the ready availability of books, teachers are no longer needed. If you want to learn something, you could go get a book on it.” Well, that was a couple of 100 years ago. And we still have faculty members, we have students writing things down, we’re reading, I don’t imagine how you could teach without writing things down and having books. The internet came along, as we were discussing earlier, while we were teaching, we watched the internet show up. And there were people who said, “Well, with the internet, there’s going to be no need for teachers anymore, because students can get whatever they want.” I can’t imagine teaching without the internet right now. So as we’ve gone through each of these iterations, there’s been this fear that maybe we’d be supplanted by some technology followed by “I don’t know how I’d work without that,” it’s a little trickier now, because with generative AI, we’re talking about not just something being available, but actually creating something. I don’t know what that’s going to look like. But there’s some real possibilities that the generative AI ChatGPT, could do things like help students who have writer’s block, get started. And that’s an individual that maybe could produce something really cool, but just can’t get started. I didn’t publish my first book until about seven or eight years ago, because I’m one of those individuals who has a terrible time from a blank screen. I just have a terrible time with that. And so now, I don’t use ChatGPT to actually write anything significant. But I will tell you that I will use it for the first paragraph. That’s all, just one paragraph. And then I completely rewrite that. And there’s no actually trace of it. But it’s something that gets me going.

John: So can we count on more than a book a year going forward? [LAUGHTER]

Todd: No, no, no, no, you can’t. So exhausting. But the concepts that will help students that can do that, I think that’s going to be helpful for them. So there’ll be a type of student who couldn’t have produced before, but now can. We are definitely going to run into some challenges, though, with students who are going to just use generative AI and use artificial intelligence to actually create and to hand something in instead of doing the work. So I do think we’re in a challenging time right now. And I wouldn’t make light of that. There’s actually something that I find fascinating from this. Right now, more than ever before, we can actually have artificial intelligence create something for us, especially in higher education, this hasn’t been done before. The tricky thing is that we were the ones to be able to make that possible, because we learned things. If we let a machine do that work for us, we’re not going to be put into the situation or our students coming along, will not be put into a situation where they’re intelligent enough to do the things that need to be done when they need to be done. And so I do think we’re facing a real dilemma right now. If my students, for instance, always do use some artificial intelligence to create a paper and hand it in, if I can’t catch it, they may end up with an A in that portion of the class. But there’s going to come a day when they’re going to have to write something or be able to read something and tell if it’s written well. And so I’m a little bit nervous, we’re entering a phase where by bypassing some cognitive processing that needs to be done, we may be limiting what we’re able to do in the future. Wrapping this up, though, I don’t want to be the person who says if you use a calculator, you’ll never understand this statistical test. So I don’t know where the balance is. But I do think we’re going to have to have decisions coming up that we’ve never had before.

John: Generative AI is drawing on that wealth of knowledge that has been produced. And for that to continue to grow in the future, we do need to have some new materials being created. So that is an interesting challenge, unless it goes beyond unless….

Todd: …unless it creates it. So that was one I thought about, by the way, sometimes you’re sitting around just thinking about stuff and it’s interesting. I was thinking how do I acquire new information. And the way I acquire new information is I go read articles, I read books, I read a ton of stuff. And then I say I think this is valuable, I don’t think that’s valuable. And then I put it together and say here’s what I’m thinking. And now I’m looking at this generative AI who goes out and scans the environment and pulls these things and then creates something new. It doesn’t have the cognitive processing that I have at this point, but…

John: it’s in the early stages.

Todd: We have some folks who are very concerned out there, especially in European countries that are starting to put some guardrails out, because at the point that it keeps grabbing stuff, and then generating and then it grabs the stuff it generated, then it’s going to be interesting. But as of right now, I just read another article, I think it was yesterday, that they’re going out and grabbing the most popular or most frequently written things and then putting it down as if that is right.

Rebecca: The way that you might prioritize as a human with an expertise in something, is going to be really different than a system that’s prioritizing based on popularity, [LAUGHTER] or like how current something is like when it was last published. That’s a really different value system that really changes priorities.

Todd: Yeah, and I think it changes how we teach. I think the way we teach is going to fundamentally shift because we’re going to have to work with students with all these things being available and explain to them and talk to them about the learning process and the value of the learning process. And keep in mind, this isn’t just about ChatGPT writing papers, everybody’s freaked out about that right now. We shouldn’t lose sight of the fact that you could get fresh, cleanly written papers that have not been plagiarized at all, we’d be able to do that for 20 years. There are paper mills, I can either write away or contact somebody and say, “Please write me a 10 page paper on Descarte, and they would write it, I could turn that in. What actually has happened recently is that everybody can do it, even those who can’t afford to have a paper written at $10 a page or whatever it’s costing. And so equity comes back again. [LAUGHTER] Now we’re an equal opportunity cheater. So we have to be careful with that. But I think the way we teach is going to change because all that information is going to be available, kind of like the internet on hyperspeed. And then what do you do with that? It’s going to be really intriguing. I think it’s an exciting time.

Rebecca: So Todd, this episode’s gonna come out right at the beginning of this semester. So you’re saying we need to be thinking about how to change our teaching. ChatGPT’s here, what are you doing for the Fall differently?

Todd: Well, I think the biggest thing is what we were just talking about, looking more at the learning process, which has been a big thing for me for the longest time, is explaining and talking through the learning process, I can hand you all this information, but if I hand it to you, you don’t learn. In fact, one of my favorite examples came from a friend of mine, and it was the gym, if you want to get in better shape, I could pay somebody to go do sit ups for me. And then I could somehow log in the book at the gym that 100 Sit ups were done, use the passive voice there, and somebody else did them for me, I’m not going to get in better shape unless I do the situps. So I have to do the work, I have to run, lift weights, do the situps in order for me to be able to gain. We need to just turn that into a cognitive process for our students to really gain cognitively they have to do the work. And so I think more than ever, it’s how do we convince students of that? And for the faculty members who say, “Well, that’d be great, but my students just want the grade.” If that’s the case, we have a bigger problem than whether or not some technology can write a paper for them.

John: So how do we convince students that it is important for them to acquire the skills that we hope they get out of college?

Todd: I think this is probably going to come down to the community building, it’s been there forever. If you really want your students to do the work, the best thing you can do in my view, and that’s why I’m gonna say, Rebecca, I don’t think a lot for the way I teach, has changed. You build a community, you build relationships, you talk to the students about importance of things, if you’re sincere about that, and they get that then yes, there’s going to be some students that are going to mess with the system, they have always been there. But you’re also going to get a lot of students who will say, “Yeah, that’s a good point.” And then they’ll do the work. I don’t teach as many undergraduates as I used to, I’m teaching more faculty than ever because of being the faculty developer. But there were years that I would have to tell my students don’t put more time than this in on your paper, you have other classes, you need to do the work in the other classes. Because, and I’m telling you, I am very proud of this, my students would spend a ton of time on this stuff for my class, because they didn’t want to let me down. And I would say you’ve already got an A, I’m proud of what you’re doing, please go work on your other classes. That kind of scenario happens when you build community. And I’m not saying it’s easy, I would never say it’s easy. And it’s not going to happen for everybody. But it is the foundation of good teaching.

Rebecca: So we always wrap up by asking, what’s next?

Todd: There’s just so much going on right now. I think that what’s next for me is I am still in that headspace of coming kind of back from the pandemic, anybody who says, “Yeah, but the pandemic’s all over,” wait for November, we won’t know, we’re going to see. But I still think that’s next is kind of thinking about how we teach and learn in this environment. So moving in that space, it’s probably not surprising. I’m working on the next book here. One of the things I want to do now is the last couple of books that I’ve done had been pretty heavy books. And now I want to write something that’s a little bit lighter. So it’s going to be more of a quick guide with more narrative and having some fun, I love telling stories. I love having fun with people. So I’m going to try to create a book that’s kind of like a science of learning and teaching at its best but really accessible and more of a story-based kind of way of looking at things.

Rebecca: Who is your audience for that book?

Todd: Anybody who will read it? [LAUGHTER] Anytime I write anything, I have to have the audience firmly in mind and think about who am I talking to. And I really believe there is a pretty big overlap with students and faculty who don’t know specific things. And I’m not saying this in a mean way toward any of my faculty colleagues at all. But there’s a lot of people who aren’t taught about things like long-term potentiation and deep sleep in terms of semantic memory, and looking at depth of processing and those types of things. So the same type of thing we can say to a student, we know you shouldn’t cram, but here’s why you shouldn’t cram… faculty learn a lot from that as well. And so my audience for this book is going to be faculty and students, students, because I think it’ll be more fun to read about how to learn in a narrative form like that. And faculty because it’s more fun to learn when you read in that kind of a format for some people. we’ll see.

John: And if faculty design their courses to take advantage of what we know about learning, it can facilitate more learning.

Todd: Wouldn’t that be cool? We could just keep rolling, rolling. What a great amount of work. I mean, a huge amount of work that faculty do. They’re hard working folks that are just cranking away all the time. Number one, making their life a little bit easier by helping to understand things would be great. And just having a little bit more fun would be fun, would be nice way to go to0.

Rebecca: Hey, anytime you can save time, so that we can have more play in our lives is better.

Todd: Yeah, just to do whatever you want to do.

John: Yeah, ending on a note of fun is probably a great way to end this.

Rebecca: Well. It’s always great talking to you, Todd. Thanks for chatting with us and going on the Wayback Machine.

Todd: Oh, you know, I love the Wayback Machine.

Rebecca: I love it too.

Todd: For those of you who don’t know about that, you should check out the Wayback Machine

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

299. My Professor Cares

Students from marginalized groups often question whether or not they should be in our classes and disciplines. In this episode, Michal Kurlaender joins us to discuss an easy to implement intervention that faculty can use to improve retention and student success. Michal is a Chancellor’s Leadership Professor in the School of Education at UC Davis and is a co-Director of the California Education Lab. She is a co-author with Scott Carrell of a National Bureau of Economic Research working paper entitled “My Professor Cares: Experimental Evidence on the Role of Faculty Engagement.” (This article is forthcoming in the American Economic Association journal, Economic Policy.)

Show Notes

Transcript

John: Students from marginalized groups often question whether or not they should be in our classes and disciplines. In this episode, we discuss an easy to implement intervention that faculty can use to improve retention and student success.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Michal Kurlaender. Michal is a Chancellor’s Leadership Professor in the School of Education at UC Davis and is a co-Director of the California Education Lab. She is a co-author with Scott Carrell of a National Bureau of Economic Research working paper entitled “My Professor Cares: Experimental Evidence on the Role of Faculty Engagement.” This article is forthcoming in the American Economic Association Journal, Economic Policy. Welcome, Michal.

Michal: Thank you for having me.

John: Our teas today are:… Michal, are you drinking any tea?

Michal: I am not drinking tea, but I did have some not too long ago today.

Rebecca: Do you have a favorite?

Michal: I’ve come back to Earl Grey. I used to be an Earl Grey person. I left it for a while, and it’s just made a comeback for me.

Rebecca: Nice. It’s a classic. I have Christmas tea today, despite the fact that it’s June.

John: And I’m drinking a ginger peach black tea today.

Rebecca: I have my Christmas tea because we had our presidential announcement today. And it was like celebration tea.

John: White smoke has come out of the towers [LAUGHTER] and we have a new college president here.

Michal: Congratulations.

Rebecca: So we invited you here today to discuss the study of the impact of specific faculty behaviors on historically underrepresented minority student success. How did you decide on this specific intervention?

Michal: My colleague and collaborator Scott Carrell and I do a lot of work to try to understand College Access and Success. And in particular, we’re interested in understanding inequalities in graduation rates at more open access institutions, like the California State University system, which has, across its system, some more selective campuses and some more open access institutions. But in particular, what we’ve noticed for years is that the graduation rates of students of color, particularly male students of color, black and LatinX men, were really much lower than other groups. And this was a puzzle to us, largely because the eligibility to get into four-year colleges, including the CSUs is quite substantial. These are primarily B-plus students who have finished a comprehensive set of courses required to be eligible for the CSU. And so to see their graduation rates lag so much behind other students was really troubling to us. And so that’s why we decided to focus particularly on the CSU system. And we focused on one campus in particular, that’s a less selective CSU campus.

John: What was the intervention that you used?

Michal: We didn’t go in knowing what intervention to use, we actually started with a focus group with particularly men of color at this campus and asked them what their challenges were. In particular, what we learned was that their challenges were not necessarily social or more broadly campus level, they were primarily in the classroom, and that is they felt disconnected from their instructors and from what to do to be successful. These were all students who reported feeling quite successful in their high school, feeling quite connected to their high school instructors who encouraged them to go on to college. Many of them were in competitive fields like STEM and engineering. And then they felt like they really struggled in college, and in particular, how to seek help and how to understand what instructors wanted from them. And so we came in quite agnostic, I would say, about what could work, what is helpful here? Is it more writing centers, more coaching, more nudging? We didn’t know. And what we came out with is feeling quite sure that we needed to tackle the classroom. And in particular, I think what we wanted to think about was this untapped source of potential support or hindrance in that is faculty, and think historically, we know that many times we just think of the classroom as kind of untouchable, and we put other support centers, writing centers, and tutoring centers, and other supports for students. And we kind of leave the classroom alone and leave faculty, including ourselves… we’re both faculty… to do what they will. And instead, here we wanted to really think about, could we intervene with faculty to provide more support for students?

Rebecca: Yeah, it’s funny how we often overlook that particular option given that’s a key touchpoint with our students, right?

Michal: Exactly. So we came out deciding that we’re going to do an intervention that was classroom based, and that was going to try to work with faculty to give students more information about what it takes to be successful in the classroom, how to seek help. And then we decided to pilot that to see if the proof of concept worked. And we piloted it at a large, more selective institution on a small-scale pilot, and found some promising results and then launched it at full scale. This article describes that whole kind of research process as well, which we think is also an important contribution to the literature… to not just immediately do something, but to actually think about the way in which it might function and just really to understand from students what they tell us they need.

John: Your initial pilot used a light-touch intervention, could you describe that intervention?

Michal: So initially, in the pilot, what we did was a slightly underpowered pilot in the sense that we took students who didn’t complete their first homework assignment in a classroom where you have to complete a set number of homework assignments. And you could miss one, but historically, we knew that students who missed the first one often struggled in the class. And so that’s the point of randomization that we took for the pilot. That is we took those who didn’t submit their first homework assignment and to half of them we sent an email saying, “Hey, we noticed you did and submit your homework assignment. Just to remind you to do well on this course, here are some things that might be helpful.” We also provided some information on what’s coming up and reminding of the office hours and how to seek help. And then we did two others. Importantly, the two other emails provided information that showed the students that we knew how they were doing in the class so far. So after the midterm, and before the final, and we found again, it was underpowered, but we found positive effects among the treatment group. And then that was conditional on some other ex post characteristics that we added to the pilot, but then we decided to launch it at the CSU campus that we worked with at full scale across a random sample of introductory courses.

John: For those listeners who are not familiar with statistics, when you mentioned that this is an underpowered test, could you just explain that in terms that…

Rebecca: …Rebecca can understand? [LAUGHTER]

Michal: Absolutely. So they were underpowered statistically, to detect a statistically significant difference between the treated students, those who got the emails, and those who didn’t. And so for that, we need a large enough sample size of treated versus control students, particularly if we’re going to add other kinds of observations about them, like their gender, or their race, or their prior academic achievement. And so when we say something’s underpowered, we might see the positive effect that is better achievement in terms of the final or in satisfactory progress in the course. But that difference may not be large enough in statistical terms to consider it statistically significant, even if the mean differences are actually in the direction that you expect. So to get that, to be able to actually detect significant effects, you need a large enough sample size. And that’s when we launched into the population that we were specifically focused on, which was the population at this less selective campus.

Rebecca: At that less selective campus when you scaled things up, did you keep the intervention the same? Or did it change? What did that scale up look like?

Michal: Great question. So the first thing we did was we really focused on introductory courses. This was also piloted in an introductory course. But we wanted to focus on particularly large classes, especially because the information was going to come from the instructor and we were doing a randomized control trial, that is some students are going to get this treatment and others are not. So the class had to be large enough for it to not be weird that some students were getting it and others were not. If you’re in a classroom of 30, that might be strange if you’re talking to someone next to you, and they get this email from an instructor, but basically what we did is we recruited faculty, we randomly chose 30 large undergraduate courses. And then we recruited those faculty and said, “Will you be part of an experiment with us? …and here’s what you need to do.” And the important thing here is that we’re not trying to dramatically change faculty style in the classroom, we all have our own style, the own way we write a syllabus and what we expect from students. What we wanted, we had several key principles. The key principle first was that faculty need to directly communicate with the students showing them that they know who they are. So it very much said in an email, Dear Rebecca, or Dear John. They needed to provide information that was specific to their class, it couldn’t be quite generic. We provided them some templates, but the goal was for them to personalize them and say, “Here’s this upcoming unit, here’s what to look out for, here’s how I would study for it, Here are my office hours, and so it provided information. So the way we changed it is… it was a semester-long courses. and so we requested that faculty sent a minimum of three emails to students, one after the first assignment or exam, sometimes earlier if they didn’t have an exam, and two later, we wanted it within the first three weeks of the semester, one after the midterm, and one before the final. And the important thing is, in the second two emails, those were further personalized to sort of say, again, “Dear Rebecca, I see that you’ve gotten a 72 on your midterm, it’s not too late to improve your grade in this class. Here are the things that you could do. And so it was personalized also just showing that we know how you fared in this class. And so again, the goal was to let faculty, in their own words and in their own course formats, personalize these emails, with the principal being information to students, personalization to students, and help seeking behavior advice.

John: And this process is a personalization, was this done in a mail-merge type format? Because I would think to scale this intervention would be a lot easier if you did do it either using the tools within the LMS or using some type of a mail merge.

Michal: Great question. So again, this was a grant-funded study and where we could provide some support to faculty, some faculty didn’t need additional RA support from us and either knew how to do a mail merge, it really worked with their course management system, like Canvas or Blackboard and found it very efficient to work on the own. Others you may or may not be surprised, did request our help from our graduate students. And we did provide support including one actually helping a faculty member directly write individual emails for students to support. You’ll probably ask me how the faculty feel about this. And I will say we actually asked them how long it took. It didn’t take more than a minute an email and so we do kind of try to guesstimate the investment on the faculty’s time to do this, and it very much varies on their comfort level with the course management system.

Rebecca: In the scaled up version of this study, did you continue only interacting with students who had struggled and missed their first assignment or is that a shift from the pilot to the big study?

Michal: Yeah, thank you for catching that. No, it is a shift. We did open it up. We believed actually, theoretically, our priors were that anyone could benefit from this. So if you were a B or on the cusp, we have lots of researchers suggest students, especially in introductory courses, some students, particularly first gen students might take a B or a C as a signal that they shouldn’t be in a particular major. We really did want to encourage across the achievement distribution for everyone. As to John’s earlier question, as you scale this up, or as people have talked to me since this experiment said, what if I want to do this, but I teach 400 people, you could one year, one term, try it with your lower achievers, another term, try it with those at that C range, or others. And so we did in this initial intervention want to do it across the board.

John: How large was the sample on the scaled up version?

Michal: It was 20 faculty, some were in multiple classes, we had 22 classrooms overall, and roughly 3000 students.

John: Excellent. How large was the estimated effect in the scaled up version?

Michal: First, it’s important to note, as we’re talking about findings, that our findings are really concentrated on first- and second-year students or new students, and who are from underrepresented minority backgrounds, so URM students, and we find that their treatments are about five percentage points more likely to earn an A or a B in the course by comparison to control students. So again, just important to note, we find overall positive effects for the whole of treated students. But they’re only statistically significant for the URM students that we target, that our intervention aimed to focus on.

Rebecca: Was the impact limited to just the classes the students were in, or was there an effect beyond that individual class?

Michal: Yeah, so that’s a great question, and we do find what are called spillover effects. And that is that those same students, those URM students, had a positive effect of being in the treatment group, even in courses that were not part of the experiment. That positive effect was much smaller in magnitude, it was like three quarters the size of the effect of the actual treated course. But still statistically significant at the ,10 level.

Rebecca: it seems so easy, just three emails.

Michal: Yeah, it does. It takes an investment. But yeah, I think it does beg the question, I think, for me, and maybe this is something that you want to talk about a little bit later. But we do so many things to introduce first-gen students or to get students in the classroom and again to provide support externally, but we do tend to sort of assume that they’ll just survive or just be okay in the classroom without training faculty on how people might experience their classroom differently. And so, again, we do test for other subgroups. I focus on first-gen, because it’s a concept that is helpful for people to think about students who don’t have at home, at least, people to tell them what to expect in the college environment and how you might go to someone’s faculty office hours, and they’re not there, or they’re there, but they’re sort of like, “Yes, did you have a question about the material?” Whereas to know, you could go just to review the material or you don’t have to have a question, you can just show up, things like that. Being able to feel comfortable asking questions in class and others who don’t or just step in at the end of class and sort of say, I found this part of the lecture really complicated, will you be reviewing it again the next day? So things of that nature. And so I don’t think these emails did those, but they sort of remind us that there are things that faculty can do to remind students that they see them, that they see that they’re in a classroom, and that they know that they may be enjoying parts or struggling in parts, and that there are some actions that they can take to be more successful in their classroom.

John: Did the effect size vary with class size? Was there a more substantive effect in larger classes than smaller classes? Because I would think it might be easier for students to feel more lost in a larger section, especially as a first-gen student?

Michal: It’s a great question. We aren’t able to test that, our sizes were all pretty similar, and we didn’t have enough. We actually chose the largest classes that exist in this campus, which don’t get much bigger than 150. I think it is worth testing. Absolutely. I can tell you from the pilot, that, in particular, that was a class of 400 students, and both for the pilot and for the full scale up the types of emails we got, that faculty got back. So that’s one thing we could talk about is how did students respond more generally, and many students emailed instructors back, and in particular, in the pilot, but also in the scale up, we got those emails back from the faculty who were in our experiment, and they very much appreciated the email and said, “No faculty members ever emailed me before,” or “especially in a class of this size,” or “I so appreciate the email. I’m working really hard.” So the first and sort of overwhelming finding from these emails is just gratitude from students that a faculty member emailed them, particularly we noted that in the pilot when the class was quite so large.

Rebecca: Yeah, I can imagine that just something that feels personalized, whether or not it’s super personalized or not, just feels personalized, really helps students feel seen.

Michal: Exactly that. That’s right. I should also say in addition to just grades and you asked me about graduation outcomes, we also included a survey, both in our pilot and in the scale up, to try to get at some of the mechanisms and in particular, we asked questions like, “Did you feel this instructor supports your learning? Did you feel you could reach out to your instructor?” And we do find consistent with our intervention that students in the treatment group reported more positive outcomes on these dimensions.

Rebecca: That’s so fascinating, because it’s so easy. Like it just isn’t that complicated.

John: In your study, you also examined how this effect persists over time, which certainly relates to the graduation thing. What did you find in terms of the persistence of this effect over time,

Michal: We did look at long-run treatment effects. That is, we waited to see what happened several years later, we presented on this paper in the shorter term outcomes. And we tracked these students and worked with this institution closely, we really wanted to know did it affect the outcome we care most about, which is graduation? In other words, we care about student success in a particular classroom, and maybe they’re slightly better grade, or not dropping out of the course. But we really care about their longer term outcome of finishing. Again, for this specific group of interest. We note that the treatment results in that 7.3 percentage point increase in persistence, one semester later, and then a higher four percentage points difference in graduation. So we do find positive effects on the likelihood of graduation.

Rebecca: We’ve talked a bit about the impact on students and how students have responded, how did faculty respond to participating in this intervention?

Michal: First, I’ll just say, again, we had to recruit faculty to do this. And so we do track the faculty who said no, and we did as much reconnaissance work, if you will, to understand that we did need to self select faculty, keeping in mind that if we did self select faculty who just had a proclivity to help students or being this intervention, if anything, we perhaps under reported some of our findings, but we do as much effort to understand how representative our faculty are, which we did determine they are, and in the paper, there’s some details about that. And they represent faculty from a real diversity of disciplines, from music to engineering, across the board, humanities and social sciences, we had the whole range, we had the range of faculty. So you know, “I do this a little bit sometimes. And the two of you are, I would have said, I’d do the same with a student who doesn’t show up or doesn’t complete to sort of track them down. But I’ve never done it systematically. And I’ve always wondered if it even matters.” We had everything from that kind of faculty member to a faculty member that’s like, “Well, I’ll do this, but I don’t think it matters. I mean, at the end of the day, the students who want to put in the work put in the work.” And so we had a whole range in their attitudes. We did offer a stipend to do this, because we did believe it takes time. And we wanted to sort of show that faculty who do have a lot of demands, especially at teaching institutions, that this was going to take some time. And so we haven’t done it again without an incentive, or with an institutional incentive, that’s part of like performance evaluations or something. So that’s yet another thing that in terms of where to take this in terms of where institutions might take this, for our perspective, it was externally funded, they were only accountable to us in their efforts to do this. And so we talked to them multiple times in the term, again, some were in both waves of the study, because we did it over two terms. And then we surveyed them at the end and really got some details from them. Some of this is in the paper around how they felt. And I would say most expressed similar to what you expressed, Rebecca, which is like “Wow, this some emails and I made this difference, especially in underrepresented minority students lives and in their classroom, and it felt really good about the impact.” …keeping in mind, we talked to them also, before we knew the results. And so at that point, they just were sort of documenting how much time it took to do the emails, and what kind of emails they got in response from students and most felt, I would say, humbled by the thank yous that something so small, like an email, got so much gratitude back from students. We did have some faculty that sort of said, this takes a lot of time, and I’m not sure it’s much of a help. In our last survey with faculty, we actually provided them the full scale results and said, “Here’s, by the way, what we learned from the study,” and then asked them to respond or to reflect on that. And many said, “Wow,” like, again, similar to your reaction, “a few emails could make such a big difference, I will be sure to continue.” We ask them directly if they will and we report this in the paper. Most do say that now that they know the positive impacts of this study, they’re likely to continue with these emails.

Rebecca: I imagine the workload for a faculty member isn’t necessarily in drafting those initial emails, but maybe the responses to the emails [LAUGHTER] you might receive back.

Michal: It’s a good question. I don’t know how many continued the conversation once an email was sent, the standard emails, part of the experiment and the student wrote back and said, “Thank you so much.” I’m not sure they continued. We did a lot of qualitative coding, which we don’t report in the paper, we report some but then we did a lot more internally. And there definitely were a lot of hardships described among students who did reply, the extent to which faculty replied with those hardships offering extensions or any other kind of augmentation to their requirements, we don’t know.

John: In an introductory microeconomics course I use the Lumen Learning Waymaker package which actually does automate emails to students based on their performance on weekly quizzes and so forth. And even though the students in the class realized it was automated, they’d still write back to me and I’d respond to it. And they’d often apologize, saying, “I had a bad week and I know I need to work harder.” But it did often start a dialogue that might not have occurred otherwise. And I’ve often wondered how large that effect was, but because it’s automated for the whole class, it’s hard to measure the differential effect of that. So it’s nice to see this result, that that type of approach works.

Michal: Yeah, I think that what you’re describing is exactly right, this sort of feedback. Our whole intervention is built around theories, not just from kind of behavioral economics or nudging or information source, but on the education literature on feedback, and the important role of feedback, and the timing of it. And it’s most useful if it’s not just performative like feedback, like your grade on an assessment like a “C,” but that actually gives you more information about how you’re doing or what to do to improve. And so this kind of thing you’re describing John is exactly right. I think we know that more touch points with students through assessments, as opposed to all hanging on a midterm and final also support students to get more feedback about how they’re doing.

Rebecca:I think sometimes students know that they’re struggling, you get a grade back, you know, if you’re doing well or not, but I think a lot of students need more coaching around what to do to improve or to better understand how the grade is calculated, to just take the time and attention. It’s there. It’s in the syllabus. But sometimes they don’t realize what they should prioritize. And including some of that kind of messaging makes a lot of sense. I know that when I’ve done that with my students, they’ve been really appreciative because they didn’t realize that they were putting all their energy into something that didn’t really matter as much.

Michal: Yeah, that’s right. And I think coaching is a great choice of words around what to do with it. I also think many students come to our universities with really uneven or unequal preparation for those courses. And so I think a lot about students who came from a high school where they took an AP course in economics or chemistry that might as well have been a college level course. Many of these questions are great on a grade curve. And so that C might be an excellent grade for them, given the type of preparation that they had, but they don’t know that necessarily, and they might, to them, signal that maybe this isn’t the right major for them. And so I also think coaching around what to do with the grade when you’re kind of passionate about a subject and not to give up on yourself too quickly. Many are juggling jobs, we know for some, it’s their only work is to get through this term, and others are doing this while working and taking care of family members or whatever. And so that grade that they got often conveys information that we as faculty don’t necessarily know anything about how they’re interpreting,

John: I would think just a signal, as in the title of your paper, “my professor cares” might create a sense of connection and belonging that might otherwise be missing for someone who is a first-gen student who might not feel that they belong in the institution.

Michal: Yes, I agree. And I think there are more and more studies coming out, particularly in social psychology, but elsewhere about the importance of belonging. We know from the K-12 literature that it’s having a teacher who cares about you matters, actually. And again, nothing dramatic happens once you get to college. But we assume it’s completely different, where in fact, I think having an adult or particularly your instructor care, you feel a sense that that instructor cares about your learning, or how you’re doing in their class, irrespective of the grade per se, just that you’re making progress in the class or feeling comfortable in the class, I think is really important. And I think it’s hard to test. And most of the belonging literature has been on survey type research, “I feel like I belong here.” And it’s not as much in the classroom, although there’s increasingly more studies about belonging in the college classroom beyond just a university at large.

John: A while back, we interviewed Peter Arcidiacono, on a paper that looked at the impact of differential grading between STEM and non-STEM courses. One of the things you just said reminded me a little bit of that, because one of the things that was noted in that paper is that many of the people, particularly female and underrepresented minority students who switch their majors out of the STEM fields had some of the higher grades in the class, but it was below their expectations. And I’m wondering if this type of intervention might have an effect of letting them know that that performance in that discipline may not be all that bad. Since we’re probably not going to eliminate grading differentials between STEM and non-STEM disciplines, perhaps some type of personal communication might help preserve some of those connections so we don’t lose as many people in the STEM fields where the returns to education are the highest.

Michal: Absolutely, I think well said. That’s exactly right. And I think that is among the reasons we wanted to get across the grade distribution, not just those who are really, really struggling. And also because we do think that students might give too much meaning from a signal of a grade early on in their academic pursuits where they can get through a certain amount of courses and then maybe where the fun stuff of their major where they really see that utility of a particular course for a future career path matters. And so I think that’s right and I do think Peter’s work and other people’s work looking at the impacts of particular kind of grades and grading distributions or signals of grades, I think, are really important. I think that’s an area that’s blossoming in economics and in other fields to sort of better understand heterogeneity or differences between subgroups around how particular information like an assessment, grade or a test score.

John: One of the things we’ve been seeing in a lot of studies is that many changes in education, such as using more active learning techniques, providing more course structure, benefits all students but disproportionately benefits those students who are historically underrepresented. And it seems like this study just provides more evidence of that, that good teaching practices benefits everyone, but especially benefits the people who are most at risk in higher ed. And those are often the people who can get the greatest benefit by persisting in higher ed.

Michal: Yes, I think that’s exactly right, if we’re really going to address disparities in college outcomes, and I think one really important source to go to is the importance of information gaps. And that would be particularly for first-generation students, for students of color, but also for students who come from unequal K-12 backgrounds. Colleges and universities often know and often are recipients of systematically particular high schools in their states, especially public flagships, community colleges, others, and so they are aware, they offer relationships with those K-12 high schools that are feeders to their institutions. And that is an important source of information that they can provide to high school students as they enter college for a kind of a warmer handoff, if you will, but that also faculty teaching introductory courses can provide. And so I think, again, if our goal is to address inequalities that we see in college outcomes, then I think information, particularly for those for whom their information gaps, is particularly key. Students want to be seen and not necessarily yes, there’s the anonymity of a large lecture hall, that maybe don’t want to be called on. But that doesn’t mean you don’t want to know that your faculty member sees you and knows whether you’re doing well or struggling, or how you feel about the class or how to succeed on the next exam or in the next course in that sequence.

Rebecca: So if there’s other faculty who think, “Hmm, three emails, that seems easy,” What recommendations would you make to those faculty?

Michal: Yeah, I mean, I think what the first recommendation I would make is to try it, to do it. I think thinking about how you communicate in your syllabus that about your forms of communication are important. So if you’re going to do an email, I think one thing that we would have loved to test and if we were to continue further is the format for the information. And so I think letting also students know that you want to hear from them over email or through other means, I think is useful. So first, deciding on the kind of medium like how you’re going to communicate this, I think email makes sense. When faculty start texting students, maybe we’d move to a text them information. But that’s not the case for most of us, so it’s through the course management system or email. I would say focus on again, what we know from the literature on feedback is that for it to be as specific as possible about what students can do with this information, and so that is looking at your syllabus closely, knowing… we typically do as faculty… where students trip up in the material, what’s complex, what’s up ahead. And so giving that kind of feedback as well about how to prepare for the next assignment or exam, what has tripped people up in the past, what you know, might work for them is really helpful. Again, other research has suggested the importance of going to office hours might matter. But that means you need it to show up, you have to think about how you structure your office hours. Incidentally, we did try to track that… quite hard to do whether it actually promoted more office hours in the pilot, we believe it did promote more office hour usage. More broadly, it’s something we’d like to test, the actual help-seeking behavior of students. So I would say faculty should do it if you’re teaching a 400-person class and you can’t imagine doing this for 200 students or even 100 students, maybe start with students who you see as struggling based on that first assignment, as we did in the pilot and see what you can learn from that… maybe do as John suggests, which is kind of get savvy on a mail merge and think about ways to do this so it’s more efficiently done, so that you can reach as many students as possible.

John: We just switched recently to Desire to Learn’s Brightspace platform, and it does have intelligent agents and it does have replace strings so you can automate an email conditioned on the grade on either your overall course grade or on a specific item. And if you do it on overall course grade, (which I just set up, by the way, for my summer class last week), students get an email saying, “I see that you’re struggling, there are some things you might want to try.” It would be nice if I could put in their grade without having to go to mail merge, but I don’t think that would be possible. And I’d like to scale this up. In any case, I’d encourage them to contact me during my office hours or to make an appointment to talk to me. The first iteration of that went out last week. None of the students responded, but it’s a small summer class. So I’m curious to see how this might work. And your paper helped encourage me to try this. I had other reminders out there, but this was one that I thought might be useful, using a specific grade trigger.

Michal: That’s great.

John: We always end with the question. What’s next?

Michal: Well, I think our lab and Scott Carrell and I continue to do this work, and in particular, we’re also spending a lot of our time these days doing work at community colleges in California, which serves one in four community college students nationwide. So also persistence outcomes are quite weak at community colleges, historically, and we’ve seen real declines in enrollment at community colleges since the pandemic. And so we are definitely doing some work at community colleges. We continue to track and follow graduation rates, particularly inequality in graduation rates at CSU. And we’d love to launch another intervention. So stay tuned on that. I can tell you, we are quite committed to understanding the college classroom beyond college settings more generally, and so hope that the college classroom continues to be a source of important information for the field about how to better support student success,

John: You’re doing some really wonderful research. And it’s really nice to see some of the attention that this got because your article has been mentioned in The Chronicle. It’s been mentioned in Inside Higher Ed, and I’ve seen people tweeting about it ever since it came out. It’s good to see this research becoming popularized.

Michal: Well, we appreciate it, especially since Scott and I did not succeed on that front, that is becoming those people who do good social media. Other people are better at that than I am. And I’m always a little troubled when I talk to more junior faculty around “Do you need to do all that?” …and 10 years ago or so I would say “No, just do good work and it doesn’t matter.” And now I confess the sort of buzz that some people are better able to develop around their findings in their papers seems to matter and so it’s really nice when it happens to you because we didn’t do it ourselves. [LAUGHTER] So I appreciate my friends Sue Dynarski and others who’ve done a really nice job promoting this paper.

Rebecca: Well, thank you so much for joining us and for sharing your work with our audience.

Michal: My pleasure. Thanks for having me.

John: And we will include a link to your study in the show notes and we encourage our audience to read it.

Michal: Wonderful.

John: Thank you.

Michal: Thank you so much. I really appreciate it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

295. Equity-Minded Teaching

As our student body diversifies, higher ed needs to respond and adapt. In this episode, Bryan Dewsbury and Mays Imad join us to discuss equity-minded strategies we can use to redesign or incrementally improve our courses. Bryan is an Associate Professor of Biological Sciences and the principal investigator of the Science Education and Society research program at Florida International University. Mays is an Associate Professor of Biology and Equity Pedagogy at Connecticut College and is a AAC&U Senior Fellow. Bryan and Mays are co-authors, with Flower Darby and Isis Artze-Vega, of The Norton Guide to Equity-Minded Teaching.

Show Notes

Transcript

John: As our student body diversifies, higher ed needs to respond and adapt. In this episode, we discuss equity-minded strategies we can use to redesign or incrementally improve our courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Bryan Dewsbury and Mays Imad. Bryan is an Associate Professor of Biological Sciences and the principal investigator of the Science Education and Society research program at Florida International University. Mays is an Associate Professor of Biology and Equity Pedagogy at Connecticut College and is a AAC&U Senior Fellow. Bryan and Mays are co-authors (with Flower Darby and Isis Artze-Vega), of The Norton Guide to Equity-Minded Teaching. Welcome Bryan and Mays.

Mays: Thank you. Thank you for having us.

John: Today’s teas are: …Bryan, Mays, are either of you drinking tea?

Mays: I am.

Bryan: I am not.

John: Mays, what tea are you drinking?

Mays: I am drinking chai masala, that I prepare the night before, and I wake up and it’s the first thing on my mind.

Rebecca: Oh, that sounds amazing. I have an awake tea this morning, John.

John: And I have a ginger peach black tea today.

Bryan: So I’m it odd one out with a cup of black coffee.

John: That’s not uncommon. [LAUGHTER]

Rebecca: One of the most common teas we have.

John: We’ve invited you here today to discuss the Norton Guide to Equity-Minded Teaching. Before we discuss the book itself, could you tell us a little bit about your own pathway to this project?

Bryan: Mays, you want to go first?

Mays: Sure. So my career started at the community college. And in fact, I was a postdoctoral fellow when I started, a postdoctoral fellow studying the cellular mechanisms of learning, which is vast and complex. And as a graduate student, as a postdoctoral fellow, I had this growing understanding that learning can happen, and it can happen spectacularly well, provided the right environment is there. So when I began to teach at Pima Community College, I began to see how so many of my first-generation working students, coming, showing up, doing all what is expected of them, and having a really hard time academically. And I began to understand the notion of the system and the complexity of learning. So really, for me, it started as a recognition that I, as a teacher, as a fellow human being, have a moral imperative to address what is going on. The inequities that I was seeing are not an inherent self-evident part of the system. It’s by virtue of the human-made system, that I had a choice and a chance and an obligation to start to shift and address and interrogate and even transform. So that’s how I began and fast forward to a few years ago, when Isis reached out to me, it was something that very much spoke to my heart and I said, “Yes.”

Bryan: yeah, a little bit of a similar story, I guess. I mean, without maybe recounting my whole academic career, all of the authors on this guide are people with whom that I’ve worked with in different contexts. And there’s a sense in which projects like these tend to be a combination of conversations that you’ve been having for years. And obviously, there’s a plan, there was a strategy, we carved this out and really tried to think carefully about what would be the most impactful. We also recognize and appreciate all the other books and publications out there addressing inclusive teaching. So it’s not to replace any of those, it’s really to just kind of add to the conversation nationally. We are an interesting mix in that, in terms of our individual careers, and that Isis is a provost. I’m a research faculty, but also do a lot of faculty development. Mays will probably describe herself the same way and Flower Darby is nationally known a lot of times in the online space, but really her work expands to everything. So I bring that up to say that you probably will see a lot of that complexity come to bear in the way the book is written and the kind of things we try to think about, that our faculty would need to think about, when they’re designing an equity-minded classroom.

Rebecca: So it is nice to hear all your different backgrounds and thinking about the authorship of the book, because it does help us think about, as you mentioned, the complexities of how it was written and also just the complexities of the things that we need to be thinking about when we’re teaching in this space. Your book is divided into three sections. The first section addresses class design, the second addresses the day to day operations of the class, and the final section focuses on critical reflection at the end of the course. During course design, what are some of the most important factors that faculty should consider when trying to design an inclusive course?

Mays: I’ll share some of them and Bryan, please feel free to jump in and add. So one of the things that are critical is that we approach this work with intentionality and explicit intentionality that right from the get go, even from before the students get there that we design the course to have equity in mind. And one of the factors, when it comes to course design and curriculum, is that the course has to be relevant for the students. And what the research that we found is many students find that the materials are not relevant. Now, when I think about how the brain works and how we have limited energy, oftentimes, when I see my students struggling, and I try to dig deeper and try to see how I can engage them, I usually find things like they say things like, ‘I don’t know how this is going to make me a better person,’ or ‘I don’t know how this is relevant.’ And so the brain is going to disconnect, it’s going to focus on something that is more urgent, more relevant. So we talk about relevance being very important… to explicitly make connections between students’ lives and what is in the course. The second one is transparency, why we’re doing what we’re doing. And there is a lot of research about transparency and how there is more buy-in from the students when we articulate for them how things connect, and that it’s not just busy work. And then a third factor that we say it’s really important to consider when you’re designing your course, is rigor. And we talked about the research behind rigor, and we problematize rigor, and what is the definition of rigor, and so on. And here we’re talking about academic challenge. We owe it to our students to challenge them academically, so they could succeed in their next courses, and we understand that this is multifactorial, and that we have to also find the resources so they could succeed when we challenge them academically. And so those are some of the factors that we talk about to take into consideration before you even meet your students: who are the students? what matters to them? and why it is important to cultivate the space where, when we challenge them academically, they can succeed.

Bryan: The only thing I will add because you know, we both wrote the book. So [LAUGHTER] refer pages 25 to 50, for your question kind of thing. But the only thing I will add, because this phrase wasn’t used in the book. But this phrase comes from another wonderful book called Radical Equations by Robert Moses, a civil rights leader from the 60s. And there’s a phrase they use during that time, which is then applied, he then came to apply to the Algebra Project, which he founded, which is “Cast your bucket where you are.” And that really speaks to what Mays just mentioned about getting to know your students and your context, especially at an entry level, it almost sounds like provocative, radical advice. But a lot of these things that are really important for good course design have nothing to do with the content of the class. This is not us saying that the content doesn’t matter and they should just all be signing Kumbaya for 15 weeks. What this means is that teaching is a skill and a skill that involves the psychology of the individual, the social context, both in real time but also what they brought before they showed up to that classroom. And this whole conversation is taking place because of the years we’ve been ignoring that. So I really want to kind of center that. Because I think a lot of times when people ask that question, the first thing they hear is, “Well, how do I make respiration more exciting?” Well, yeah, we’ll get to that, but this is a human being that you’re trying to build a relationship and a connection with. And that needs to take front and center before you get to inspiring them with the really beautiful content.

John: The second part of your book deals with maintaining the class, with the ongoing running of the class. And one of the things you emphasize throughout is the importance of creating a sense of belonging. Could you each talk about some strategies that could be used to help create that sense of belonging within the classroom.

Bryan: I don’t want to run the risk of listing your audience to death here. So I want to offer what I’m about to say as maybe the thing that bubbles up to the surface to me as an instructor. There’s certainly a lot of things one can do to help students feel a sense of belonging. And the approach chosen might differ depending on the class that’s been taught in the institution. One of the classes that I teach most often is intro bio and it’s a really special class for me, because it’s it’s a privilege to teach that class. I’m grateful to have the chance to welcome students into a wonderful discipline and a way of seeing the world that’s relatively unique to that space. I also like the challenge of showing a lot of students who before may not feel like this is a place they belong or see themselves doing this 10, 15, 20 years in the future. I like the challenge of showing them that this is the real thing. You can be as much a biologist as anybody else, anybody in the book, anybody you’ve seen on TV, etc. But it is a very technical space. And any technical space kind of requires a slowly evolving comfort level [LAUGHTER] as you navigate through the technicalities of it. And I think that tends to be a place where some students and faculty get stuck into how you keep that door open and welcoming while navigating this environment that really requires a lot of time and attention on the cognitive challenge. So one thing, and it might sound simple, but how feedback is given really, really matters. Without saying my age here, but I certainly went to school at a time when it was your tool, just try harder, right? …just study harder. And any grade you got you there was this sort of assumption that any grade you got was just 100% due to whether you did try well or didn’t try well enough or knew this stuff or didn’t know it. There was no discussion of the way the teaching happened. There was no discussion of your actual approaches to studying. There was no discussion about what motivates you to even do this class in the first place. And now we know and we probably knew it then too, but we know how much all of that matters. So when a student sits in front of me and they have a C, it’s not just that “Well Mays, you need to do better than this.” It’s: “Tell me how you prepare for this moment. Tell me what is motivating you to be pre-med or to want to go to grad school.” And all of those things come to be in a conversation, the goal of which is for me to see you shine in the way that I know you can. How does it build a sense of belonging? It shows the student that I am not questioning if you can become a biologist, I’m actually assuming that. What I’m working on are things that are fixable. I’m working on strategy. I’m working on things that you can do something different and see a result. And so once you know I kind of have your back in that way, your effort then becomes different because it’s just a matter of specific things we can work on.

Mays: Yes, so thank you for that. Bryan. If I were to add to your beautiful answer, I would say take the time to find out what belonging means for your students, I think we often make assumptions about inclusion and belonging and what they want and what they don’t want. Of course, I start with the understanding that wanting to belong is a human need. We’re social beings, we want to connect and we want to belong, but on a day-to-day basis in my classes within my context, what does that mean and what it would look like that’s going to be different. And while I’m going to apply what I learned, and what the research says, I also want to take the time to ask students do you want to belong? And what makes you want to belong to an academic setting or a social group? And what are some of the things that make you want to belong? And what does that word mean for you? So I think starting with that is really important. It can be really informative to our practices.

Rebecca: I really related to what you were saying, Bryan, I teach in a really technical field as well. And that technical challenge can really discourage students if we don’t make those assumptions that they can indeed be in the field that they are studying. So thanks for sharing that as your top. I feel like that’s one of my top ideas, too. And I love, Mays, about thinking about our audience, and including our audience in the design. As a designer, I really gravitate towards those kinds of ideas. One of the things that you already kind of mentioned is this idea of connections and relevance. So what are some ways that faculty can help students connect course content to their lived experiences?

Mays: I think one of the things I asked them is I talk about how learning is very relational. We are relational, learning is relational, knowledge, when we co-create it, is very much relational. So one of the things I ask them is, “Why should you care about this lesson, this topic, this context? How does it relate to your life? How does it connect to your family? How does it impact the people you care about?” And throughout the semester this is a recurring set of questions that I ask. And in the beginning of the semester, I pause, and I model the answers. I care about this, because this is how it connects to the community, the people I care about, this is how it connects to something I feel passionate about. And then it becomes an exercise that they do regularly. We do this exercise at the beginning of the class of “What is your why? And what is your ‘why?’ beyond taking this class? Why is this important? And so we connect those many exercises or “Why should you care about the acid-based physiology” to that initial “why?” exercise?

Bryan: You don’t imply this in your question, Rebecca, but it does come up a lot about relevance and how it works. And I think one danger that I want to ask faculty members to avoid is the notion here is not that every aspect of the content has to tie back to something. You may fall into that trap if everything is not connected to something, but, you know, sometimes a cell is a cell is a cell kind of thing. But there are several things that do. For me, it’s actually less a case of connecting it to their lives, per se and more a case of communicating that human beings do science, human beings do the discipline. That, in and of itself, by definition, means that there’s a social component to how it’s practiced. It introduces the cultural context within which science is done and it introduces the bias that occurs with some scientific decisions, good, bad, in-between. It brings up a different conversation once you recognize that none of these things are really apolitical, values free. The second point is, to connect something to students’ lives, you have to first know student’s life so that this way you understand, it doesn’t feel patronizing or facetious. There has to be some authenticity there about Mays’ story about getting to know the students, I don’t think she just meant know them individually, but just also their broader context. When I taught in Rhode Island, it was important to know the local and the very ancient, but also more recent history of the state, of the city, of the neighborhoods the students came from. I knew the high schools that were big feeders. Those things made the relationship a lot more authentic, because the knowledge was there first. So I think once that all those precursors are present, honestly, the connection part is not that hard, it almost comes naturally. Because you just naturally want to teach in a way that builds a history of these beautiful people. And you kind of… I don’t want to say just is automatic because you are intentional… but it is so much, so much easier, and you have this desire to do it. I’ll just add one last thing, a lot of times with teaching, and I’m speaking now, as a faculty developer, here. I get good course design, I get you want to have learning outcomes, they’re going to be measurable, etc. But sometimes, even conversations like this can get stuck within the structure of what higher ed is, which is you get 120 credits and 15 weeks at a time you take a suite of classes, and those classes give you a grade and those grades average into a number, and then you become a 3.1. Student or a 4.0 student. We get it. We get how all of that stuff works. But I know Mays well enough to know that we have much more radical visions for what education is and what it can be. We come from the Freireian school of critical consciousness and preparing students to be civically engaged to have agency and power and to see themselves as agentic parts of a democratic experiment. So what’s driving all of this is who I want you to be, not necessarily getting your good grades in 15 weeks. So from that perspective, even simple things like group work, for example, it really is actually practicing deliberative democracy. Even things like how we talk about experimental design, they bring up other questions like whose voices are not at the table when you think about these questions, like whose perspectives are you not considering? Or who are you. So if you think about this a little bit more broadly, beyond like, I just want people to do well, in this really nice subject I have a PhD in, that you tap into the more socio-political aspect of education, which is a beautiful thing, I think, but it makes the connection to social life and just broader life in general easier to do.

John: One of the things that’s especially challenging is that we’ve always had students come in with very diverse prior backgrounds in terms of their training and what they know coming into our classes, but the experience of remote teaching during COVID resulted in much greater variations in their preparation. We want to create courses that are challenging for everyone. How can we do that in a way where we’re challenging the students with really strong preparation without losing the students who don’t have as much background? What types of support can we provide to make sure that all of our students are challenged, but also have the resources to be successful in our courses?

Bryan: We could just give everybody a trophy right? Now, that’s what they say this generation is. When I put on the intro bio hat again, and that’s a place where the differential readiness is really, really apart, I would have students who last science class you take, not bio, science class, was their first year of high school next to students who were in AP, went to private school or things like that. And that’s fine. It’s fine in the sense that I’m able to design for that. And I would say there’s kind of two things I’ll say to this point. Number one, you have to be able, as an instructor, to have things in place to accurately detect whose readiness might be further behind. And specifically, what areas of readiness that need addressing, and then have the tools in place to quickly respond to that. So a lot of times just my own history, it’s hard to study. A lot of times it’s confidence. Honestly, it’s fixed mindsets around who gets to do science and what counts as doing well in science and things like that. So the first month of that class is almost like a battle, that I generally win, I think, in convincing them that this is a place for them if we consider these specific things I’m trying to show you. So a lot of times, yeah, there’s class or whatever. But there’s a lot of times spent in office hours, which we actually call student hours, because it uses a smaller group. There’s a lot of time sneaking in conversations about: Have you tried this? Have you tried that? Emails that are sent to direct people, etc. Your students will come in and I’ll just say for now, maybe kind of fly in from the get go. There are other things that they can do to grow. So those are the students I might say: Have you thought about joining our research lab? I know it’s your first semester, but it’s a good time to get to know a professor more personally and see how science is done a professional way. A lot of times those students will come to student hours. And what I would do is I put them to work. I will say, “Mays, can you make sure this group of four understands glycolysis as well as you do,” and those students will typically go on to become my learning assistants in a future semester. So my point here is that everybody has a space to grow and has a direction and an amount that they can transform. The pedagogy is you being able to figure that out? And figuring out how to respond to what growth that is needed for that individual.

Mays: Thanks, Brian, I have a couple things to add. So number one, obviously, as I hear you, John ask the question, I think it’s complex. So yes, there are students that are going to come in that may not have the academic background, perhaps they come into my pathophysiology class not really knowing the cardiac output and the cardiovascular system as they should in their previous class. And then there are students that come in ready to be challenged. At the same time, there are students that are going to come in that are going to have perhaps subtle and not so visible skills that other students who maybe have the academic background are going to have. So it’s not just that I want to bring everyone up to speed academically, I want to also bring them up to speed when it comes to issues that really matters for citizens, their empathy, their non-academic problem solving, their collegiality, and so on. So there are a couple of things I do, I want to get to know who knows what. And I tell them that I’m very transparent why I do what I do. And I tell them that this is a way to give me feedback, so I could know how to maneuver forward. I also bring the tutors and the preceptors myself from my previous classes. And I rarely bring the ones who got A’s, I bring the students who came into the class, not quote, unquote, having the background. And then they picked up the background. And they succeeded. And they ended up with a solid B, and sometimes even a C, and I bring those students and we meet on a weekly basis. And so the culture of my class is very much we’re going to work together with the tutors. And then the third thing is I tell the students, we come from different backgrounds. And I use my story that when I started graduate school, my background was in philosophy, and all of a sudden, I am studying neuropharmacology. And many of my peers in graduate school were steps ahead of me. And I was trying to figure out just basic things in cellular neurophysiology, or cellular neuroscience. And so I talk about the notion of some of us have this background and not the other background as a way to celebrate the diversity. And I say the way I’m going to approach this is I am going to start slow, because I want to review for some of you and I want to bring up to speed others, those of you who are going to feel like this is slow, I want you to stick with me. And I want you to think about what’s coming ahead. And then sometimes I’m going to speed up and some of you are going to feel like this is so fast, I also want you to stick with me, I’m going to slow down. So it’s very much relational. I’m telling them what’s going on, I’m working with the tutors, they come and talk with me. At the beginning of the class, I do a lot of exercises where students answer on a form, it’s a one-item form: should I slow down or speed up. So I’m getting this real live feedback, slow down or speed up. I’m not learning anything new, this is overwhelming. And it gives me the opportunity to change and accommodate. No class or no session is going to be the same as the previous one. So those are some of my approaches. Again, be transparent with the student, get to know what they know. focus on not just what they perhaps don’t have academically, but what other assets they bring that perhaps the students that have the academic background don’t bring and celebrate those. Those are also important and constantly seek feedback from your audience.

John: And it sounds like using peers in the classroom to provide feedback and using group projects would be another way of leveraging the strength of all the students so that those who do have better strength in a topic can assist those who are still at an earlier stage of development, which benefits both types of individuals because by explaining it to other students, they’re going to reinforce their own learning and the students who have things explained to them by their peers are going to be able to connect to that in a way they don’t often if we were explaining it to them.

Rebecca: You just kind of talked a lot about some feedback loops that are necessary in learning. And one of the things that you advocate in your book is a process of critical reflection at the end of the term that relies on self examination and engagement with course data. Can you talk a little bit about that process and what data faculty might use to assist in this kind of reflection?

Bryan: I guess for me, I really would like the conversation about how we look at semesters that we teach, to move away from a hyper focus on whether we grade or not grade, or just assess learning. And I’m not saying that assessing learning is not important. But I would like to move away from that and broaden it to evaluating an experience. And that language matters. Because in the latter, by definition, when you’re doing any kind of forensic analysis of an experience, you naturally have to think about all the factors that contribute to whether that experience was successful, however its success is defined for that situation. And the factors include ourselves, the factors includes things we did well or didn’t do well. The factors include the physical environment of the classroom or the virtual environment. The factors include the support structures that were available for both the instructor and student to be their best selves. And so once the conversation broadens in that way, it just by definition necessitates some critical reflection. It really presents the class with that question wrong. It’s probably not them, it might be me. So just things like that. It gets us away from what I think is a little bit of a false dichotomy on that kind of issue. Should you put a ladder or should you not put a ladder, that is what it is, but this is what our section is really asking us to respect. If it’s a humanist process, then every human involved will have some questions to answer about how well it went or didn’t go.

Mays: So teaching, as I mentioned earlier, is very relational. And it’s a work in progress, as learning is. And so the feedback that we see from our students is critical to help us look at what we’re missing, look at assumptions we made, assumptions we didn’t make. It really helps us move forward in a kind, equitable, and liberatory way. So, first of all, we advocate for feedback throughout the semester. Talk with your students, listen to your students. It’s a co-creative process. And then the feedback that we seek at the end, we of course, problematize student evaluation. And we say that, on the one hand, there are so many biases and so many problems with how instructors are not evaluated equitably, especially instructors from racialized backgrounds and women instructors. And at the same time, the feedback is so important, because it is right now, it’s arguably the only source of information we get about students’ experience, not so much about their learning, but about their experience. And that’s really important. We do talk about ways to enhance the reflection, too. So we could get more in-depth information about students’ experiences. And we talk about this idea of reflection, why it is so important. Parker Palmer talks about how we teach who we are, and how our inner being is so critical in the teaching process. And when we reflect on even the most harsh evaluation, what we’re doing is trying to find the truth, find a truth within that, with the intention that this truth can help us grow as teachers, as instructors, as facilitators, as human beings. So it could help our future students. So those are some of the things we talked about that the reflection is so critical, and it’s been written about in bell hooks’ work and Paulo Freire’s work, and certainly Stephen Brookfield and Parker Palmer and Laura Rendon. And those are really important. I hope when I finish a class, that it’s not just my students who are changed, that I change as well, that it is a process where as bell hooks calls it, liberating mutuality, where the classroom, whether it’s online or in person becomes a space where both the instructor and the students are transformed by the end of that experience. So those are just some of the things we underscore in that section of the book.

John: We always end by asking, what’s next?

Bryan: I will say what’s next, for my program in general, is we’ve been keeping our ears to the ground on the political landscape in the US, particularly around the kinds of things that we’ve been writing about over the past several years. And I think one thing the field needs more is we need to continue to have scholarship around these ideas and think critically about how to have education systems that allow everybody to thrive. But I think there also needs to be really well designed communication projects that message this and perhaps ways other than how academia typically messages this, which is through the research process. This is not trying to throw that process out of the water, but I think getting into more storytelling type projects, podcasting, narrative, op ed. So the short answer to you “what next” question is there’ll be more projects like that are coming down the pipeline, because we think that there’s a gap and a real need for that.

Mays: For me, it’s really bringing the wellbeing and mental health to the equity conversations. I think for so long we’ve done equity and inclusive work, kind of like in a vacuum without taking a holistic approach and that exclusion can have a profound impact on our sense of wellbeing or even mental health. I mean research shows just how systematic exclusion and microaggression can impact our cortisol level, for example. In my own work, I’ve been just saying how they intersect mental health and equity and inclusion and justice. And I want to be more intentional to, I guess, bring that to national conversations. What are you doing about mental health? I know you have this and that initiative for equity-minded education. Where does mental health fit in within that?

John: Well, thank you. Your book is a tremendous resource that can be really valuable in helping people build a more equitable classroom environment.

Mays: Thank you for having us.

Rebecca: We appreciate you joining us today.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

294. The Allure of Play

Student learning is enhanced when active learning activities are used in instruction. In this episode, Victoria Mondelli and Joe Bisz join us to discuss how principles of game design can be used to create engaging active learning experiences. Tori is the Founding Director of the University of Missouri’s Teaching for Learning Center and is an Assistant Teaching Professor in the Department of Educational Leadership and Policy Analysis. She had previously served at the teaching centers at Mercy College and at the CUNY Borough of Manhattan Community College. Joe Bisz is a learning games designer and Full Professor of English at CUNY Borough of Manhattan Community College. Victoria and Joe are co-authors of The Educator’s Guide to Designing Games and Creative Active-Learning Exercises: The Allure of Play, which was published in March this year by Teachers College Press at Columbia University.

Show Notes

Transcript

John: Student learning is enhanced when active learning activities are used in instruction. In this episode, we discuss how principles of game design can be used to create engaging active learning experiences.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Victoria Mondelli and Joe Bisz. Tori is the Founding Director of the University of Missouri’s Teaching for Learning Center and is an Assistant Teaching Professor in the Department of Educational Leadership and Policy Analysis. She had previously served at the teaching centers at Mercy College and at the CUNY Borough of Manhattan Community College. Joe Bisz is a learning games designer and Full Professor of English at CUNY Borough of Manhattan Community College. Victoria and Joe are co-authors of The Educator’s Guide to Designing Games and Creative Active-Learning Exercises: The Allure of Play, which was published in March this year by Teachers College Press at Columbia University. Welcome Tori and Joe.

Joe: Hi.

Tori: Hi.

Joe: Thanks for having us.

Rebecca: Today’s teas are:… Joe, are you drinking tea?

Joe: Oh, yes, classic chamomile.

Rebecca: …in a mug that looks handmade?

Joe: It looks handmade. But that’s just to make people think I have other skills as well. [LAUGHTER]

Rebecca: How about you, Tori?

Tori: Today I have a Starbucks espresso and cream in a fancy, fancy glass because it’s so special to be with you today.

Rebecca: She has upped to a stemmed glass. [LAUGHTER] How about you, John?

John: I am drinking a peppermint and spearmint tea in a Tea for Teaching mug.

Rebecca: That’s good. I have a cacao tea.

Tori: Very nice.

Rebecca: Cacao tea… the cocoa plant. And so it’s actually made from the plant rather than with tea leaves. So it’s not chocolate flavored. Made from the cacao plant. It’s very tasty.

John:I heard cat cow?

Rebecca: Cacao.

John: Well, I was thinking of the animals.

Tori: I was thinking of the yoga pose.

John: That’s right.

Tori: Cat cow. [LAUGHTER]

John: We’ve invited you here today to discuss your book, The Educators Guide to Designing Games and Creative Active Learning Exercises. What are the advantages of using playful activities in learning?

Joe: Well, I think I can take this start. Basically the foremost advantage of being playful is that your exercise can recapture the ageless allure of learning. As children,the first way that we all learn was by probing our environment, playing with it, inventing mock scenarios and then changing up the rules. As adults, I think that in our rush to instruct, our exercises can often become top down, and we can miss opportunities to inspire wonder.

Tori: Yeah, and I’d like to add, that’s why Joe and I really wanted our book to show teachers how to build their own creative active learning exercises to be easily appliable. We mostly come at it from a non-digital aspect. So we are recommending using basic materials, paper, cards, tricky questions, puzzles, point systems and role plays as some of our most common tactics.

Rebecca: Although, lucky for us, there’s plenty of digital technology that can make all those things happen in other environments too.

Tori: Exactly, and the book does have that. We have different sections of the book “design online,” and so recommending those tools as well.

Joe: Low-key though, low hyphen key.

Rebecca: So we know that students play games for hours, often weeks, months, and engage in things for long periods of time as they master new games and get engaged with their friends. What are some principles of game design that we can use to create and maintain student engagement in our classes?

Joe: This actually began some of our early research before the book, thinking in this direction. When designers make mainstream video games, or even elaborate board games, escape rooms, or interactive installations in museums, whether for children or adults, there’s often a lot of difficult content that needs to be taught. So as you mentioned, the designers rely on tried and true principles of interactive design to make these experiences feel engaging. In fact, there’s been a lot of work by other researchers in the past 20 years, exploring variations of these principles, and suggesting that we teachers can learn from these principles. But most of the focus was on proving the research rather than articulating how to apply these principles in a practical manner. This led to our own research collaboration, and eventually our very successful and hands-on professional development workshops. They use our methods.

Tori: So we have several sets of principles in the book, and I think this will be a perfect opportunity to tell you about our engagement principle. And we have five of those. The first one we call “narrative and fantasy,” the basic idea being that people need to hear stories that give context to what they’re facing in their own lives. The next one is “networked or sensory environment,” the idea that people thrive when they’re communicating in social groups using digital smart tools and multimedia. The next one, we would call “fast or random access,”
many people like processing information non-linearly and even simultaneously. Next is a classic one, “challenge,” people really want to challenge them feels achievable and personal. And the last one is also sort of a classic, I’d say, and that’s “rewards,” what we call “frequent rewards” or “feedback,” that we really as human beings crave feedback on our choices, even if that feedback is abstract, like .4 miles or something along those lines.

Rebecca: Can you share a couple of examples of what those engagement principles look like in the classroom?

Tori: Yes, Joe, do you think now’s a good time to talk about the simple mechanics?

Joe: Yeah, I think so. And this is basically a way that learning activities can be, as we could use the word converted, or transformed into some kind of playful activity. So there’s a spectrum, we think, to making an exercise more playful, or engaging. Of course, the first step is to just have it be active, have the learner actually be doing something other than just listening, the next step is to add a fun element. These are elements which researchers have shown add engagement or playfulness, we’ve classified some of these into what we call the five simple mechanics. And let’s see if we can explain them using audio only. So make the pace of your exercise a little bit random, or a little bit rapid. That’s two of the simple mechanics, rapid or random, or you can make its goal involve some kind of reward, such as points, badges, or some kind of rivalry, a little bit of competition, or students cooperating against a challenge, which still has competition, or some kind of roles. And we define roles pretty broadly, it could involve themes that have concentrated moments of emotions like feelings of suspense, it could be playing a historical, or fictional character, or just the students being on teams, and each student has a particular responsibility. So if just a part of it has one of these simple mechanics, not the whole thing, then you’re already on your way to creating something playful, so at least those moments would be playful.

Rebecca: So you’ve mentioned the five simple mechanics, can you give an example of what that looks like? So you’ve provided a frame, provided some playful things that we can think about, what would an activity maybe look like using something like that?

Tori: So one of the games that I like to play in the Teaching for Learning Center with faculty, and then what usually happens is they figure out ways to use it in their classroom, is a game that Joe designed a few years ago, we call it icebreaker bingo. So you can imagine a bingo card where there are prompts on each of the squares. And when people come in the room, and they’ve never met each other before, but there’s a common thread, those prompts will relate to that thread. And you circulate around the room and you go up to different people, you obviously introduce your name and other important details, and then you get right to it. You can ask them up to two of the prompts. And if they are true for them, they sign their initial, and then keep moving through the room. So the first person to get five across or five down or five of a kind, we have categories would shout “bingo.” They can think about the simple mechanics at play in that wonderful icebreaker certainly is random. Usually people don’t come up with a strategy for it, they just go up to random people and ask random prompts. And then they also can kind of get into the role of being an interviewer and selecting different prompts based on the ones that they like. The rewards is really strong in that particular icebreaker because there’s that joyous moment of explaining bingo, and then we’ll usually have a prize. And the rival factor is very heavy in icebreaker bingo. So that’s one that has a lot of these engagement principles, and is very popular with faculty and students alike, and can be just for general getting to know each other, but also could be used a little bit as a kind of quizzing mechanism as well.

Joe: And I like that Tori gave an example of what we would call basically a more full fledged game. But our listeners can also think about the simple mechanics as just little sprinkles onto something. So for example, for random, we have something that we named monkey wrench challenges, where you have any kind of standard activity with the students that are working on something. But then you make a point of changing things by calling out a sudden challenge that requires the students to alter their approach. So maybe it’s a new perspective by which they’re looking at the material and other concepts and kind of new information that they have to find. And even though you’ve planned it out, probably, unless you have a deck of these things that you’re drawing from, it’ll have a nice taste of random and nonlinear and break the order that students were doing something and therefore, for reasons known to these researchers, feel somewhat pleasing.

John: Can any active learning activity be converted into a more playful structure.

Joe: I think so because if we look at it as a spectrum, moving from active learning to adding a little bit of play, even if the rest of your exercises is exactly as you’ve already designed it, then anything can be converted. It’s just a question of whether one, after analyzing the need for what you’re trying to achieve, whether it needs it. The longer you have an activity that goes on that you might consider or your students might consider to be passive, we would argue the more the need for some element of it to change the pace, or to change the goal, so that it could feel a little bit more playful.

Tori: And Joe, in our decade plus time of doing this work, have you ever had an occasion where somebody came to you for advice on making a learning activity more playful or gameful where you’ve had to turn them away? I could say I have not, we always find something.

Joe: No, definitely not. Especially if they’re already crossed the chasm, and they’re actually interested in making a more playful,

Rebecca: …perhaps the very first step, right? So you note in your book that learning activities might be focused on skill and drill, or focus more on deep learning. Can you talk a little bit about how we can do a playful activity or a game for each of those different kinds of learning?

Tori: Yes, definitely. And if you’ll let me just kind of explain a little bit on how we classify skill and drill versus going more towards deep learning. So skill and drill, what we’re really talking about here are sort of facts, factual, and foundational knowledge that’s needed either in a discipline or sort of a general education curriculum. And so it’s really things that people need to commit to memory, just basic ease and facility with those facts or figures, that kind of thing. And then when we’re talking about deep learning, we start to move into the ability to, for those who love Bloom’s Taxonomy, is like moving into manipulating information using higher order thinking, but to do that in a way that really will be long lasting, because even over time that certain things can evaporate. That’s just human memory. So when we talk about games for deeper learning, it’s not enough to use the engagement principles and the simple mechanics. And there are ways of structuring it so that we get students to grapple more deeply with content and do things with content. So we’re delighted to bring the complex mechanic to people and readers with the ALLURE method. Joe, do you want to talk about that?

Joe: Yeah. So it’s perhaps a little unexpected in our book, because a lot of people who are talking about games are focusing on the engagement element. And immediately, we do talk about that. But we’re actually firm believers that, in addition to being engaging, well designed games and playful activities, can carry, as Tori was saying, deep learning principles taken from the cognitive sciences. So before the book, I started doing some research on ways that faculty were already approaching learning activities in the classroom. And I came up with a classification system that Tori also expanded on with me for the purpose of the book. And we looked at how these things could carry some of what we call deep learning principles, such as identity situated meanings, and the ability to pass information to students through cycles of expertise, so that the students slowly becomes a master and then is challenged by new information. We call these the complex mechanics, and to them will instantly be very familiar, I think, to most of our audience. One of them is called “trivia questions.” It’s pretty much the most common type of playful activity that educators use in the classroom. Another one is called “simulations,” which is also very common, especially at the high school level. And that can involve debates and roleplays, which in our opinion, are an example type simulation, also arguments. Very complex mechanics, we call “cut ups,” basically about sequencing puzzles, reordering information, mixing it up, and having to put it back together. Other ones would involve classification type exercises, such as sorting or matching. So these are already hard-core activities that are done sometimes in the classroom. But we’ve linked them to some of these deep learning principles and have explained how these teachers want to design something but they don’t want to reinvent the wheel, they could just look at one of these nine complex mechanics and they’re sort of gameful approaches, creating something. We also have deduction exercises, common to scavenger hunts, brainstorming, which is definitely playful and feel like a kind of game and interpretive exercises such as improv. I was just at conference a few days ago, called Playful Learning and improv was everywhere. It was the heart with all these teachers just talking about and it was so interesting to see all these examples of playful improv in order to illuminate the concept.

John: In your title, you have the ALLURE of play, but ALLURE is used there not just as part of the title, but it’s also an acronym. Could you tell our listeners what the acronym stands for?

Tori: Yes, A – ask where to apply the play. L- list the mental moves. Second L – link the mental moves to the play. U – understand how the learning principles operate. R – run the activity game. And E – evaluate the learner experience. So that spells ALLURE, and that’s our six-step method, the backward design guide. And we’re so happy that it worked out because we just love the word “allure.”

Rebecca: Can you walk us through those steps in how a faculty member might approach their class and designing an activity?

Tori: The first step really, an educator wants to ask themselves or work with a small group to really inquire in their own curriculum about where, to use a business term, where would the return on investment be for your student learning? Joe and I are believers that if it isn’t broke, don’t fix it. [LAUGHTER] So where are students struggling? Have you taught this class before? Is there a bottleneck in learning? Is there something that’s holding them back from the learning? And we think game-based learning is the ultimate in active learning pedagogy, as really, really strong to help students through and be successful. We advise educators to look at the curriculum and go where the student learning is being held up for some reason. Maybe there’s a student learning bottleneck, because cognitively it’s a difficult concept. Or maybe there’s an affective blockage. So Joe and I are strong proponents of game-based learning as the best active learning strategy. And so we think that you can go to your toughest student learning challenges and apply game-based learning there. So that’s what we’re really asking in step one is, where do you think you’ll have the most bang for your buck? Because game-based learning does take a good amount of talent and time put into it. So making judicious decisions. Is there a place where students are bored? Is there a place where just, semester after semester, you lose half the class, and to go ahead and apply your energies there. The next step, list the mental moves, really gets at when you have a student learning outcome that you’re working with for students in a class, oftentimes, that’s written in a way that is a little obtuse, you have to break it down to really get at what the student is actually doing mentally, what are the mental acrobatics that are going on there? So this is where we will say, “Let’s get granular. What are we really talking about? What are the mental moves?” Joe and I are both very inspired by the decoding the discipline scholarly community, and they have had, for decades, really strong practices in faculty working together, two faculty from different disciplines to help each other break down what the expert is really doing. And so the first couple of steps of the ALLURE method are really picking up on that tradition of where do you want this learning game to be successful in your curriculum? And then what is really going on at a minute level, the kinds of thinking steps, the kinds of mental moves that students need practice with. And then the third, I like to think of link the mental moves to the play… it really is the magic of the whole ALLURE method. Because when Joe developed the complex mechanics, he categorized them according to Bloom’s Taxonomy level. Now, what this does is it really takes a lot of guesswork out for educators who want to apply the method. Now all of a sudden, they can drill down on nine of the complex mechanics and say, “Well, I want the complex mechanics that are really going to help my students. say with analysis.” And then that gives them a subset of choices, that they can choose the complex mechanic that foster more analysis and build that in as the game play, so students are really practicing analysis, getting feedback on that analysis. And it’s really quite time efficient, thanks to that categorization.

Joe: So Tori just reviewed the first three steps: ask, list, and link. The last three steps are understand, run, and evaluate. So as soon as the teacher has finished thinking about what are the actual steps the students are doing, and connecting it to one or more of the complex mechanics, this is the moment when we ask them to brainstorm roughly what their playful activity will look like. So in the book, they take 20 minutes, putting together some ideas, making connections, then they have some tea, take time off and we move on. So this is all about the end of link moving into understand how the learning principles operate. Within understand how the learning principle operate, they then do a debrief and they’re examining what they came up with. And they’re comparing it to all the mechanics and principles that we’ve been teaching them so far in the book, not because their activity needs to have all of these things, that they’re looking for. have perhaps missed opportunities, or things that can be refined, or mechanics that they were working on, but perhaps didn’t come across as thoroughly as possible. We call these checklists. And they’re just going through and reevaluating their approach, and then tweak it one more time. In the next step, we have run the activity game. So ideally, the teacher would talk to either some play testers or some fellow colleagues in their discipline, and talk to them about what they’ve come up with, one more last chance for feedback. And then they would present it in the classroom to their students. And we have different methods for collecting student feedback and taking notes on what they’re saying. We realized pretty early on in our own workshops, that a lot of teachers are actually just uncertain how to even run an activity game in a classroom, much less one that they’ve designed. There’s so many facilitation issues to think about. So within our step, run the activity game, we also walk teachers through thinking about common facilitation issues that will come up such as: Do I have the right kinds of desks to hold these games or activities? Do I need a projection screen? How many teams are there going to be? Should students be in teams. In all these concerns, they can think about them ahead of time and have a good strike plan. After the activity game is run, we go to the last step, evaluate, evaluate the learner experience. And we have different rubrics and ways that the teacher can think about how successful was the students’ experience of this? And how might the activity game be iterated upon and changed for next time? Because we all know, we slowly work on our exercises, and we make them better year after year, till perhaps we get tired of them, or we put them aside for a while.

John: When you present this material, I imagine there’s some faculty who are resistant to the whole concept of building play into classroom instruction. How do you address that with faculty?

Tori: I never force it on them. We start with: “What brings you in today?” And we talk about we sort of have a co inquiry process of like, “What’s going well, what’s not going so well?” And then wel don’t always bring the play or the games at first. It seems like mostly when I advertise a game-based learning community of practice or a workshop, you’re sort of already getting people that are open to it. But I can tell a nice anecdote about working with the political science department. I started working with them about four years ago, just on a course redesign because their student success, or the grades, were not where they needed to be… a large enrollment course. And so we just started working together to boost student attendance, students really weren’t coming to class as they should be, and to boost presumably student learning, but they really had some assessments that I thought could be improved upon. So I was even questioning how much learning are we really evaluating here. So we began and worked with each other for a couple of years, just doing classic active learning redesign, before I even broached the subject with them about playing games. And what brought that about was an internal grant for $100,000 became available to do innovative creative teaching and learning. And so when I pitched that idea to them, I think our relationship and my credibility with them, that we’re not just adding bells and whistles, we’re really talking about some proper teaching and learning principles to not only boost engagement, but to deepen learning. And it’s been a wonderful project. And we have new board games and card games, we’re going to have our first video game. We’re working with Adroit Studios, our gaming lab on campus. And so they are very, very pleased and had never really been in game-based learning before.

Joe: And adding to what Tori’s saying, I think often the teachers just want to see like a little hook, a very quick understanding of what they could do to be playful. And then from there, they start getting other ideas. So we often lead with talking about simple mechanics. So we might say, well, for example, you make something a little bit more rapid, maybe you’re giving a certain amount of time for students to answer the question, not like a test question that’s worth something, but during an activity just to make it a little bit more playful, 10 seconds to think about something. Or taking another aspect of the rapid simple mechanic, one might have a very long activity that students have to do. There’s been research showing that if you can chunk that activity into shorter segments, perhaps that are assigned a certain amount of time, not really above timers, but just to get through all of them the day, students can more easily see the discrete tasks and operations you’re supposed to be performing in each of those segments. Make something a little bit more random. We mentioned earlier about the monkey wrench challenge, and try to use a little bit of roles in what you’re discussing. Roles are linked to narrative, and it really does help to give a theme or some kind of narrative directivity. For example, for discussing an essay, or a textbook entry, or any concept, tell a quick story that illustrates some of the concepts from that reading. Or ask the class to actually help you write a short creative introduction to the boring essay that everyone had to read or that you’re reading in class, and to use a narrative hook in that short introduction. So now we’re using the power of analogy and of fiction, well, not really fiction, but sort of narrative to help students see sort of a more macro way of tying concepts together before doing a deep dive. So these are all little ways to think how to quickly and briefly be playful.

Rebecca: I hear both of you describing some really interesting things in the classroom. But I also know that faculty can be really anxious about trying something new, or trying to make something happen. And as exciting as it can be, taking that first step can be scary. You’ve mentioned some of the simple mechanics and I heard you say something about low stakes and the way that you’re applying about points and not being a big test or a test question. Can you talk about the little ways to get started?

Tori: My favorite way to introduce faculty to this whole new world is to invite them to play a game Joe made years back called: “What’s your game plan?” And this is a wonderful card game in a team. And it’s cooperative play. And it’s a brainstorming game. It has been just a joy bringing faculty together where they can be playful together, and see what they come up with. It uses a lot of the simple mechanics, so it’s a great way to introduce them to the simple mechanics. And they can either bring a lesson objective that they have, or Joe’s deck has some common lesson outcomes. So it really takes the edge off. And they can just relax in what Joe and I like to call the sandbox, and just say: “Be creative.” …give them permission to play themselves. And that is one of my go to. I mean, there’s not a semester that goes by that I don’t lean on What’s your game plan? So grateful to Joe for making that. And then I guess we could also say: “Well, let’s start at the beginning with the ALLURE method.” Step A: ask where to apply the play and once you have that student learning bottleneck that you want to look at, maybe you’re not ready for the complex mechanics, but you can think about two or three of the simple mechanics that you can add to an existing activity you have for to convert even maybe a lecture or a mini lecture into something a little bit more playful, where the students are interacting with you in some way.

Rebecca: So I’m always curious to ask what’s your favorite, or one of your favorites… because we obviously can’t have favorites… your favorite game or playful activity you’ve seen implemented in the classroom.

Joe: I talk about this example a lot, my heart goes there immediately. One of the complex mechanics I was talking about, I said something about one called cut ups, which comes from sequencing ideas together. And I was trying to think about how to use this in the writing classroom since I’m an English professor. I was getting a little bit stuck looking at it on the level of just words, like cutting up a sentence, at the level of the word… reordering it. This is really only teaching grammar and I wanted something a little bit more sophisticated that was about reading and writing. And about this time, one of my colleagues, Julie Cassidy, without us really even having a conversation told me that she’d come up with something. She showed it to me. And basically, she had taken, I think it was a four paragraph essay, that was published about crab fishing. So we have something that’s explaining a procedure, how to do something, to cut up this essay on the level of the sentence. So, there are like 25 sentences, carefully inserted in an envelope so they don’t all get lost. And then we had a low-level writing classroom where the students were very weak. She brought it in, in groups of five, dropped an envelope in front of them, said, “Okay, put it in order.” She’s describing this to me, and I’m very excited. So I did the same thing about a week later. And it was incredible just to see their faces so utterly focused on the task in front of them. And one of them would like, move their hand forward to move a piece of paper with a little bit of timidity, but with deep interest, and then the other one would gently touch your hand and say, “No, no, it goes over there.” Then like “ah,” [LAUGHTER] and they’re like stroking their chin and moving. And this went on for like 12 Intense minutes before the groups had basically all solved the order for a possible order that made sense of the essay. So of course, they’re looking at transitions, their reading for sense, so there’s close reading here, they’re thinking of ways that information can be ordered for their own writing. And then a great way to follow up an exercise like this is to ask students to write their own mini essay. It’s teaching some kind of procedure, like creating a recipe. So this was a great example because it goes beyond just a quick moment of play into something that’s a longer activity and where you can really see the deep learning happening with the students. That’s one of my favorite examples.

Tori: And I have one also that’s quite well known for deep learning, and that is the whole suite of games called Reacting to the Past. So I’ve had the honor of attending and playing a lot of these games, and they are deeply immersive role-playing games. Most of my experience is with history games, but they’re all across the curriculum, even in some of the sciences now. And here I just marveled at the amount of hard work that students will do in order to play their characters really well and meet their objective. So, the writing of speeches and other kinds of rhetoric and communication, the behind the scenes faction politicking, they’re willing to do, the, I would say, transcendence of one’s own identity to kind of widen perspective of other that goes on in these games. Kudos to Mark Carnes and all of the reacting creators and trainers around the country and around the world who are doing such great work with that. At the University of Missouri, we have recently made a game that’s akin to Reacting in the sense that it’s a role playing game. But it’s more of a card game than a reacting game, and it teaches students about bureaucracies, and how frustrating bureaucracies can be, but that hiring more people isn’t always the right scenario. So it takes the role-playing mechanic and then we throw in unexpected stress bombs, we’ll call it, of the government goes on furlough, all of the workers have to cease what they’re doing for 20 minutes or something like that. And it just has all of these likely events or unlikely events that can happen. And the students are really in their roles to solve a problem. So we’ll take things from current events, like recent scandals or problems and have the students work through that. And then there’s a peer review piece of it, where they’re actually scoring each other’s solutions. And that’s been really wonderful to help the faculty create, and we’ve been play testing it and we’re really excited to bring it to a live real class in the fall.

Rebecca: That sounds really exciting.

Tori: Thank you. Yes.

Rebecca: So we always wrap up by asking what’s next?

Tori: Well, Joe, and I do a lot of professional development events, we are often going to conferences. I’m really excited coming up soon, I’m gonna get to take ALLURE abroad. I have a trip to Thailand to Prince of Songkla University to present there. And then I’m going to Germany in the fall for the Decoding the Disciplines conference. Joe, I know you’re always doing lots of things, if memory serves, you’re going to Gen Con, this summer to do a lot of workshops.

Joe: Yep. So for any teachers listening who will have a strong interest also in playing tabletop games, there’s a conference, a convention called G-E-N C-O-N, Gen Con. And they have a pre-day, an earlier day, which is all about using learning games and thinking about games of education. And I’ll be running at least four or five workshops there, just for all you applying the ideas behind learning games.

Tori: And we always love for people to get in contact with us. You can reach us through AllureOfPlay.com, our website. We have a growing contact list that we’d like to keep people apprised of our different online workshops and in-person opportunities. And there’s a lot more on that website.

Joe: Yeah, we’ve even …was it two years ago, Tori? …there’s a whole section on our website about using playful activities online. So on your discussion board, your Zoom class, we made a video that walks teachers through thinking about these ideas, also how to design in the lowest easy possible for the online space, basically through using PowerPoint, and creating little things students can manipulate and using your class should that be of interest to you, and also a lot of other free resources and activities and games that you can download.

John: We’ll share a link to that in the show notes file that will accompany this episode.

Rebecca: Well thank you so much, Tori, and Joe for sharing ALLURE with us.

Tori: Thank you, so great to be with you

Joe: Thank you very much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

290. Transparency in Learning and Teaching

While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, Mary-Ann Winkelmes joins us to discuss what happens when instructors make their expectations transparent to their students.  Mary-Ann has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd.

Transcript

John:While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, we explore what happens when instructors make their expectations transparent to their students.

[MUSIC]

John:Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John:…and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Mary-Ann Winkelmes. She has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd. TILT is an acronym for Transparency in Learning and Teaching in Higher Ed. We are very much fans of the TILT approach and have referred to it often in workshops on our campus (and on previous podcast episodes). Welcome, Mary-Ann.

Mary-Ann: Thank you. I’m really delighted to be here with you. And I appreciate the opportunity to speak with you on Tea for Teaching.

John:We’re very happy to have you here. You’ve long been on the list of people we’ve wanted to invite. So we’re very pleased that you’re here today. Today’s teas are:… Mary-Ann, are you drinking tea?

Mary-Ann: I am indeed. And I’m drinking a Sencha green tea today. That’s my new favorite kind of green tea, Sencha.

Rebecca: Nice. I have English breakfast today.

John:And I am drinking a mixed berry Twinings black tea…

Rebecca: Hmmm.

John:…which I haven’t had in a long time. I wanted to mix it up a little bit today.

Rebecca: …mixing it up with mixed berries. So, Mary-Ann, can you tell us a little bit about how the TILT project came about?

Mary-Ann: Sure. This was years back, I want to say in the early 2000s, late 1990s, where I was working at the BOK Center for Teaching and Learning at Harvard University. And I was leading a seminar group discussions about teaching and learning. And we began to think about the question: “What happens when you tell students why you’re teaching how you’re teaching, just what happens when you tell the students more about your choices as an instructor, how you’re choosing to shape the learning experiences for the students?” And that’s not often something that we think about first when we’re thinking about what’s the content of the course. But we began to think about that a lot. And we had a kind of metaphor about the Wizard of Oz, and pulling back the curtain to show what was happening behind the scenes to build the experience. And then somehow through that conversation, the word transparency emerged. And that became the word that we used pretty regularly from that time on. When I moved to the University of Chicago, that was the word we were using, and it kind of stuck. So that’s kind of where it started. And it started alongside of my career as an educational developer. And it’s kind of been, for me, in the background or on the side, as something that I’ve been kind of tracking along with as a project. It’s still there, it keeps going. And just about a year ago, I began to work on TILT as my full-time job, which I’m really happy to be doing now because it gives me an opportunity, not just to do a guest talk here or there, or a keynote address, which is usually a one time-interaction. But now I have the flexibility to connect with institutions around a longer-term project. So if there’s a faculty learning community that emerges from a first talk that I would give, I get to follow up with them later and see what’s happening and check in with them. Sometimes I get to see the assignments before and after, which I really like. And I invite those now, because we’d like to publish some of those on the TILThighered.com website. And there are some schools that I’ve been working with in the state of Washington for several years now running with their TILT projects. And that emerged from a project we did with the entire state system of Community and Technical Colleges in Washington State. So I have opportunities now like that, where I can work with larger scale TILT projects that take more time, because this is my full-time job now. And I’m really happy about how that’s working, because I feel like it’s getting larger beneficial impact for students in a way that’s more efficient than when my full-time job was at an individual institution.

John:Could you give us an overview of the TILT framework?

Mary-Ann: Absolutely. So the TILT framework is meant to be a very simple tool that is a framework for an ongoing kind of communication among teachers and students. And in all of our studies, we asked teachers to use this framework in their own way at their own discretion, because we know that it’s not really possible to expect that people would do the exact same things with it. So our research is based on the premise that people are using this framework in their own way, at their own discretion, in a way that feels consistent with their teaching style. So there are three parts to this framework: purpose, task, and criteria. And what we ask in all of our studies is for teachers to engage students in conversation about three aspects of a particular assignment or a project or even an in-class activity. Before the students do a piece of work that we want them to complete, we’re asking for teachers and students to have a conversation about three aspects of the work before the students start working on it. And those three aspects are the purpose, the task and the criteria. Now the purpose kind of consists of two pieces. The first part is talking about the skills that students will practice while they’re working on the assignment. And then how are those skills useful, not just now in this course, or maybe in college and other courses, but how are these lifelong learning skills that will be useful for the student in their careers after college or in their lives ongoing? And then the second part of the purpose is about the content knowledge. What new information or what disciplinary information will the students be researching, or gaining, or applying when they’re working on the assignment? And how will that be also similarly useful to them, not just now, or in college, but beyond in their lives? The task, that’s the second part of the TILT framework, and the task is sort of about what are the teacher’s expectations about how students will approach the work? And for the students, it’s kind of like mapping out their game plan, like, what’s the first thing they will do? Will they Google something? Will they go to office hours? Will they go seek out a research librarian? Will they go into the lab and start mixing something like, what’s the first thing they’ll do? And then a sequence of what they plan to do after that until they submit the work. In an ideal world, the teachers and the students would have similar expectations about how that would go. In some cases, though, teachers have a pretty legitimate pedagogical reason for hiding that, that they don’t want students to know how to do the task. And I found this to be the case, particularly in fields where creativity is really important: performing arts, studio arts, even engineering or some STEM courses, where teachers really want students to cast about for a while and kind of use their imagination and see if they can come up with something unique, if not into the discipline, at least unique for the student to try to figure out some new process. And there’s value in that. When teachers want to do that, we did have some pushback from teachers in our original TILT research studies, where they said, “What happens if we don’t want to tell students how to do the work, like part of the task is for them to figure out how to do the work?” So in that case, we asked for those teachers to just say something like, “Part of the purpose of this assignment, in addition to the skills and the knowledge we’ve talked about, part of the purpose is for you to struggle and feel confused, while you invent your own approach to the question.” And we think this is what helps to preserve the student’s sense of confidence and their sense of belonging. Because instead of having that moment of panic of “Oh, no, I don’t actually know how to do this, I don’t even know where to start, I don’t know where the resources are, I don’t know what I’m doing. Maybe I shouldn’t be doing this, maybe this isn’t for me, maybe I shouldn’t be in this major, or in this course.” Instead of going to blaming it on themself or to questioning whether they’re up to the task at all, students can say instead, “I am totally lost right now. And that is exactly where I’m supposed to be. I know I’m on track. I’m doing great. This is the confusion part that comes before the clarity. And I know that because we talked about that and the professor said, this is part of what we expect to happen. This is intentional, this confusion, you’re supposed to feel lost right now.” So that’s sort of what we can say about the task. And the benefit of students knowing upfront what the task is, or knowing how they plan to approach the assignment or the project, the benefit there is that students get to spend 100% of the time they’ve allocated to work on this project, doing their best quality work, and they don’t lose time trying different approaches to see if this or that is going to work or looking for resources that aren’t what the teacher intended for them to be using. Instead of losing time, on the “how,” students get to spend their time so that what teachers receive then is most of the time what we’re looking for, which is “What is the student’s highest capacity right now?” Let’s see an example of the best work that the student can do right now, so we know where they’re at and we can bring them further so that they can advance in their learning. But if we accidentally end up in a situation where a teacher didn’t intend for the students to be confused, they expected the students to take a particular approach that they may have even mentioned at some point in class. So that’s why they think the students know that that’s the expected approach. I don’t want to say the correct approach but at least what they expect students to do. So if we think that students know how to do what we expect them to do, and the students don’t know what we’re expecting them to do, then there’s this chunk of lost time, where what we’re measuring then in the end is what happens after the students spend a chunk of their time lost trying to figure out how to approach the work, and then whatever time is left after that doing their best quality work in the amount of limited time that’s left. So part of the “task” piece of the framework is about what do we want to measure? Right? Do we want to be assessing the best quality work that students can do? Or do we want to be assessing what happens when you give a really varied, diverse group of students a particular assignment to do and you don’t give them 100% clarity about how to do it, and then kind of what you’re measuring is which students have, through no fault of their own, not encountered that information in their lives before coming to this course. And then you also get to identify who are the students that maybe because they had some other kinds of privileges that not all the students had, who are the students that can figure it out faster, because they come equipped with those privileges. So you can begin to see that this is an equity issue. So if talking about the purpose of the assignment kind of speaks to the student’s motivation, and to the value that they will gain from doing the work, and maybe to their ability to assess if they’re getting that value while they’re doing the work, the task speaks to even more of an equity situation where we’re trying to get all of the students to the same starting line of understanding of how to do it, and of having all the resources they would need to do the work to complete the work. And we want to make sure that students are all at that same starting line before they start the assignment. So that’s kind of the equity piece of this. And then finally, the third part of the framework is about criteria. We want students to be able to understand while they’re doing the work, how well are they doing. We want them to be able to make corrections, if they end up with a finished version that doesn’t look like what successful work would look like in this kind of a scenario. But if the students have never seen what successful work looks like, and they probably haven’t, because why would you assign them to do something that they’ve already seen many examples of; they wouldn’t be learning anything new. So kind of by definition, students aren’t going to know what successful work looks like when it meets this or that criterion in the discipline. So what we encourage teachers and students to do there when they’re considering the criteria is to offer students more than just a checklist or a rubric, because the words on a rubric or checklist might mean something different to the student who hasn’t done this kind of work than they do to the teacher who’s really immersed in this kind of work. An example I sometimes offer is, let’s say, I asked students to write up an analysis of a 15th century wooden painted sculpture of the Madonna and child from when I was teaching Italian Renaissance art history courses. In an art history course, the word analyze, like the tasks, the actions that you take when you are analyzing something, that’s a very, very different activity than analyze in the context of an economics course, or in the context of a chemistry course. But if the student hasn’t done this kind of analysis before, you can’t know for sure that they know what you’re asking them to do. So we kind of have to talk that through and students are going to need to see some examples of real world work in the discipline so that they can, with you, in a class meeting, talk about how do we evaluate analysis in this example from the real world, or in that example from the real world. And you won’t find any one example that matches every criterion of the assignment you’re asking students to do, usually, so you need several examples. The benefit of several examples is also that you can begin to talk about the relative success with which different examples are meeting a particular criterion as well. So once we’re in a conversation with students, and we hear back from them, that they’re telling us, what we had hoped they would understand about the skills they’ll practice and the knowledge they’ll gain, that purpose, about how they’re going to approach the work, the task, and about how they’ll know that they’re doing good quality work, the criteria, once we hear students telling us that, that’s the moment that things have become transparent. It is that activity of communication, that conversation with students about purposes, tasks, and criteria, that’s where the transparency comes from. And when we are done with that conversation, we know that students are at the same starting line of readiness In terms of their understanding of what they’re going to do, and also, in terms of their confidence that everyone has the resources that they need, in order to complete that work

Rebecca: What faculty believe is important for students to learn doesn’t always align with the goals of students. Can you talk a little bit about some strategies for bringing these into better alignment?

Mary-Ann: Sure, I think that this kind of speaks to the purpose part of the transparency framework. And often teachers are expecting students to learn something that is very valuable, we wouldn’t spend our time teaching things that don’t have a lot of benefit for students or that they would only use today and it wouldn’t be useful to them later in life. We like to teach things that have value. And so, when we are communicating with students about that value, we’re talking about the skills that students will be practicing. They won’t perfect them on this assignment, but they will begin to strengthen a particular kind of skill set. And they will gain some sort of disciplinary knowledge that can be useful to them later. And we know that sometimes disciplinary knowledge changes over the years as people discover new things and publish new things in any field. Sometimes that knowledge changes. But having some knowledge now does give you important value if you’re going to continue in that discipline or if you want to understand basic principles of a discipline that you might find useful elsewhere. So if students and teachers have a transparent conversation or communication, it could be a written communication, it could be something that they record and put on a website, it could be an asynchronous kind of conversation in an online course. But whatever form that communication takes, I think students and teachers when they’re on the same page about what the knowledge is, what the skills are, that are the focus of this assignment, students will feel more motivated to do the work, because they’ll see that it has benefit for them. And it doesn’t feel like a rote exercise, or just churning out another problem set or another art history analysis paper. There’s some value here that the students know upfront what that value is. And when the teachers hear the students reflecting back to them in this communication, that this is the value that they will be gaining, then we know that students have a kind of motivation to benefit from this assignment.

John:One other issue is that students have come up with some way of learning while they’ve been in elementary and secondary school. But those methods that they picked up are not generally the ones that are most effective. How can we encourage students to adopt learning strategies that they may be resistant to because for example, students, when there have been surveys of what types of learning strategies they found most productive, students often say they prefer to be lectured at, because they learn more from the professor that way. And also, many students don’t like active learning strategies. While they learn more, they don’t perceive it that way. Partly because of those desirable difficulties you referred to before, that when they’re struggling with something, it’s a little bit less pleasant than sitting there nodding and smiling and having everything seem to make sense. How can we encourage students to accept those desirable difficulties associated with learning so that they can learn more effectively,

Mary-Ann: I want to say that this is something that the TILT framework can definitely help us with. And this is not an uncommon phenomenon at all, I even find in my TILT workshops that I do with instructors, that instructors don’t love collaborative learning either. And in fact, many of these TILT workshops that I do will begin with some kind of a research review about “How do we know TILT works? What are the studies and what do they tell us and show us the data?” So we get off on this kind of role, where we’re almost in a traditional lecture format, where like someone’s delivering some information, and people are listening, and then they have questions about it. Or maybe they have challenges to say, “Wait, this doesn’t make sense, let’s talk about this.” And then I kind of switch the method that we’re using. And I’ll ask people to break off into small groups and begin to analyze a particular assignment and talk about where do they see the purposes, the tasks, and the criteria? Before I do that, I acknowledge the fact that we are shifting gears, and that we were doing fine with this sort of Q&A format. You know, look at the research and then think about it and talk about it. Ask questions. Why would I switch that up now? Like we were on a roll, we were doing great. Everybody was sort of on board. Why would I change that now? And so I use the TILT framework to talk about why we’re shifting gears now. What is my purpose in having you use this different method? So if it’s a peer learning method, as it is in the workshops, or as it might be with students in a class, we want to tell students: “Why are we now manipulating your learning experiences this way? Why would I do that to you when I know that sometimes students resist this, when I know that it can be uncomfortable, because I don’t personally always like to do it when I’m in a learning experience?” So if we can tell students, here’s why this is going to benefit you, because you don’t just hear it, but you have to struggle to apply it, you have to fit it not to the situation that I was talking about, where it all sort of makes sense when it rolls over you and you’re hearing it. But you now have to take the principle of what we were talking about, and apply it to this new unfamiliar scenario. And the benefit of that is that you will discover you will hit a barrier at some point in that process, where you will discover the exact piece of information that’s missing for you. You will discover exactly where you hit a barrier to your understanding. And you will have an opportunity right now, right here with me, the teacher in this class, to address that confusing point. And the benefit of doing that now, as opposed to later when you’re doing a graded assignment, is pretty obvious, you get the benefit of having the difficult learning experience in a safe environment that doesn’t lose you any sort of points on your grade. It doesn’t have any negative impact on you the way that it might if you waited until the end of the term to do some massive project and you hadn’t really done a lot of the homework or done a lot of the practices and so you didn’t really know what you didn’t understand until it was kind of too late to do anything about. So I think in short, what I’m trying to say is when we’re asking students to do something uncomfortable, that has a really solid pedagogical reason, that has evidence behind that, it is an evidence-based practice, we want students to know that upfront, because that then will increase their motivation to do it, because they see how they’re going to benefit if they do this thing.

Rebecca: One of the things that students often struggle with is when they start new courses with new faculty, and new ways of doing things and determining what the instructor will expect out of them and out of that learning experience. Can you talk a little bit about how the TILT framework could allow students to shift their focus to learning if it was adopted in the design of the course rather than just an individual single assignments?

Mary-Ann: Yes. And in fact, this is a way that lots of faculty are using the TILT framework, is to think about how do I TILT not just a single assignment, but a whole course. So usually, when people are introduced to the TILT framework, the original ask for all our research studies is would you please do this two times in an academic term, just twice? Because we wanted to see how little change could you make and have a beneficial impact on students’ learning, because small change is much more likely to happen than massive change. But once you’ve made that small change as an instructor, and you see that when you do this with two assignments, there’s some real benefit for students. And on the TILThighered.com website, there are publications by faculty who talk about not just how the quality of students’ work increases, but how the teachers experience in grading, or in responding to students, or in how many students will ask for an extension at the last minute, like these difficulties that teachers often face are diminished, while the benefits for students and the quality of students work increases. So once you begin to see this in the small scale of assignments, teachers then, maybe in the subsequent term, will think about what else could I TILT? Could I TILT in-class activities? Could I TILT a unit of this course? Could I TILT the whole course? And then the effects or the applications can grow. So we can apply this to a single assignment, we could TILT a whole course, we could TILT a curriculum in a department, we could TILT a program, we could TILT an institution’s learning outcomes and thread them through not just all the courses, but through all the co-curriculars too so that students might discover in their work-study job that they’re practicing one of the critical thinking outcomes, that’s a goal for the whole university that connects with what they were doing in their accounting class. And then we can even think about this in terms of a national framework of learning outcomes as well. So there are many scales at which you can apply that to a framework. And one of the things that I’m really enjoying about doing TILT full time, is that I can work with groups of schools, groups of institutions, so not just the Washington State group that I mentioned to you, but several weeks ago I was in the state of Kentucky where working with teams of teachers from institutions across the state, for the whole state system, to think about aspects of how do you map out a path for students to succeed in fulfilling their curriculum? And then how do you pursue that path? How do you complete that path? And in that case, we were using the TILT framework as a strategic planning framework to think about once we know what the plan is, like, once we’ve mapped out our plan for how students can effectively complete their degrees, how do we then communicate the value of that degree, not just to the students who are doing the degree, not just to the students’ families who may be contributing to the costs of doing that degree, not just the costs of the student’s tuition, but the cost of the student not being an earner in that family. And we want to communicate this to all the stakeholders, so the students, their parents, faculty, and staff at the institution, to state legislators who may be voting on packages of funding to higher education in their state, to individual grantors who might be funding particular scholarships. And we want to be able to communicate the value of this degree to every stakeholder in a state system that way. And the TILT framework is very helpful for thinking across multiple audiences, because that’s a pretty difficult task to communicate clearly to all of those different kinds of audiences. But it’s pretty essential for the success of higher education in this country. And so we spent a couple of days using the TILT framework as a strategic planning framework to think about how do you communicate the value of a degree? There are lots of ways that you can apply the TILT framework. Another example is I was working with a school in Texas over the summer, and they were TILTing their entire college success course. Many institutions have that kind of course in the first year, and some of them had TILTed individual assignments. And they decided they wanted to put the team of all the teachers together, and then subdivide that so that a smaller team of teachers was working on each week of the course. And then all the assignments and the lectures or discussions that would go into that week. And then we use the TILT framework as a larger framework to connect that whole course. So that from week to week, the purposes, tasks, and criteria were pretty clear. And students understood the path for all of their learning across that course.

John:Have you tried taking on the Florida Legislature? [LAUGHTER]

Mary-Ann: I have not.

John:That’s a real challenge, I suspect.

Mary-Ann: Yeah, I have worked with schools in Wisconsin. Last week, I was working with a school in Tennessee, right after a couple of their legislators were expelled temporarily. This kind of a framework, I think, can be effective in a lot of different higher education systems and contexts. That’s one of the beauties of it. Because this is something that teachers can do, starting right now, to complement any kind of larger, institutionally driven or federally funded program that might focus on student success. A lot of the time, those programs don’t necessarily feel like they’re directly connected to what faculty members are doing in the day to day in their classes. But using this TILT framework is something that you can do that will advance students’ success that will then make you feel more like you’re connected to these larger ongoing efforts that might be focusing on something that you don’t do directly, like targeted scholarship funding, for example. But that’s part of the beauty of the TILT framework is that it can work in many, many different contexts, and across different scale sizes of projects, as well.

John:And it works nicely for faculty because you end up getting work of the quality and the type that you expect, rather than getting student work that you find disappointing. And similarly, students end up doing work that they’re much more happy with, because they were not guessing at what the instructors want. So it just seems really, really logical. But it’s not always so widely practiced. Your efforts are really helpful for all of this.

Mary-Ann: I think one of the reasons why people might be hesitant to use the TILT framework, you don’t necessarily want to try doing something different that could suck up time that could take time away from delivering important content in the course, and what teachers have discovered and written about and published in the National Teaching and Learning Forum and other places you can see on the TILThighered.com website, what teachers have discovered is that if you take some class time to talk about the purposes, tasks, and criteria for a project before students do it, by the time that practice is completed, everyone has saved time; that time gets recouped, and students have learned a larger quantity of what we had hoped they we’d learn because when we deliver content in a course, we don’t know that students are absorbing it the way that we’d hoped or that they could apply it the way that we’d hoped. So I think by the end of the course, if you’ve used the TILT framework a couple of times, you’re in a situation where you’ve worked in a way that is more time efficient, somewhat, and you arrive at a place that, as you say, is more satisfying for students and teachers, because more of the time has been spent with the students doing the highest quality work possible.

Rebecca: I think one of the things that can be challenging for faculty initially is that if you’ve never communicated in this way, it’s hard to do it the first time, because anything you do the first time is difficult. But once you have a little practice doing it, it’s easy to adopt and expand across a course or across a set of courses.

Mary-Ann: That’s so true. And I think that the way that we’ve structured the TILT framework, it looks so simple, it’s a three-part framework. Applying it then gets you into some complexities that are important to clarify. I think you’re absolutely right, the first time we try anything that’s unfamiliar, just like for students, it’s more difficult. And then we kind of get the hang of it. And then it comes much smoother, and much easier. The TILT framework for starters, is pretty simple. It’s got three parts, right? And I think you could probably share a link to the one-page version of the framework that we give to students, that sort of spells out the framework: purpose, tasks, criteria, the knowledge and the skills. And then at the bottom, there are some of the evidence behind why we know this works and some footnotes, so that students can see on one page, this is a real thing. It works, it helps you. It is, in some cases, equitable, and it is probably worth giving it a try. And if you can see all that on one page as a student, then you might be more willing, especially in a context where a teacher is describing to you why this will be good for you, why this is a benefit for all of us. And then for teachers who have not encountered the TILT framework, when students can bring in this one pager that has some studies listed at the bottom and footnotes, they can see that when the student is asking me, why should I bother? This is actually a legitimate question. This is not a troublemaker student, this is a student who actually knows that they will benefit from knowing a little bit more in advance about this assignment that they’re planning to do. So we try to make it as easy as possible to implement. And then we also try to say only a little bit of this will make a statistically significant difference for students’ learning, so that you only have to try it a couple of times in a whole term. And you’ll probably see the kind of differences that we saw in terms of increases to students’ confidence and their sense of belonging, and their metacognitive awareness of the skills that they were practicing and developing. So if you’re doing anything new or different for the first time, yes, there’s some difficulty to that, but this one is a very, very desirable difficulty. [LAUGHTER]

John:We’ll share a link to that one-sheet document as well as to your website in general. And you do have a lot of research cited on your website. And there’s also some ongoing projects. Could you talk a little bit about those?

Mary-Ann: Yes, we are sharing all the resources that we possibly can on the TILT higher ed website, because we want for everyone to have access to this. Some of the places that benefit most are places that might have the least amount of money that is allocated for faculty development or educational development. So we want to make sure that this is accessible to anyone who would want to try it. And then the studies that we’ve done in the past, there are a few studies that have indicated to us a number of the benefits of TILT. One of the first studies we did was the national study we ran with the Association of American Colleges and Universities. It was funded by TG Philanthropy and my colleagues working on that project were Tia Brown McNair and Ashley Finley. And what we did there was we worked with a group of seven minority-serving institutions from across the country that represented every possible type of minority-serving institution, as well as a range of educational contexts like urban and rural, two-year, four-year, research university, really small in scale, large, residential and non residential campuses because we wanted for teachers to look at our results and see, “Oh, well, this worked for those faculty at that institution, and there are students like my students in that mix, so maybe this would work for my students. And in that study, we started with 35 professors at seven schools and we surveyed about 1200 students and we saw that, for the students who received the more transparent instruction, their competence and their sense of belonging and their metacognitive awareness of the skills that they were developing, those increased, those were higher for the students who got more transparent instruction than for those who got less transparent instruction. And then we also saw in that study some differences that showed us that while all the students were benefiting to a statistically significant level, underserved students were benefitting slightly more. So first-generation students in their family to attend college and ethnically underrepresented students and low-income students have slightly larger benefits than the benefits for the whole group. And then in our second study, we focused on how long does this effect last. So we worked with a group of University of Nevada – Las Vegas students. At the time we were working with that study, University of Nevada – Las Vegas had the most diverse undergraduate student population in the nation, according to US News and World Report. And we know from other studies, like Walton and Cohen’s, 2011, Science Magazine article, for example, we know that when students’ confidence increases, when their sense of belonging increases, they tend to persist longer in a course. So in courses that have higher levels of confidence and belonging, fewer of the students would drop the course, for example, more of them more likely to complete the course. And we wanted to see how long does that last. Is it just that course? And some studies indicate that this could last for a year. And what we did was we kept looking at the retention rates of these students to see how many of them were still registered a semester later, a year later, two years later. And we saw that by the time students were in their third year of university as undergraduates, those students who had received transparent instruction in one of their large gateway intro courses in their first year, those students were a little bit more likely to be still registered as students in their third year. And we’re now tracking that out to six-year graduation rates. So we saw that not only does transparency have a beneficial effect, it’s statistically significant, but that effect lasts for a good long time. And then in the state of Washington, we’re now writing up that study I mentioned with the Community and Technical College System. And I think that TILT is particularly helpful in that environment, because the population of community colleges and technical colleges is a little bit more diverse. And we have more students who belong to that underserved category of students, first-generation, low income, ethnically underrepresented. And what we’re finding from that study is we’re understanding a little more about how does transparency work, and I want to thank all of the researchers who are contributing to all of these studies too, because I’m not an educational statistician, so Daniel Richard, and Carolyn Weisz and Kathryn Oleson are contributing to this study and doing a lot of the analysis, along with help from some graduate students who have been working on this project over the years. What they’re discovering is that transparent instruction has a direct impact on students’ awareness of the skills that they’re learning, and it has a direct impact, similarly beneficial, on students’ sense of belonging. And then separately, sense of belonging has a direct impact on students’ metacognitive awareness and skills that they’re developing. So TILT has this direct effect. And then there’s this other effect between belonging and skill development as well. So we’re finding out more about precisely how TILT works for the benefit of students in these studies. And I think in terms of next studies, I want to be asking questions that really matter to populations of faculty and students around the country. So we open up the TILT research team to anybody who’s curious about this, and a number of faculty have asked about, can we say something more about how this works in an online setting, in an online synchronous setting in an online asynchronous setting, and we’ve got a few publications up on the website about that, but others are looking at that a bit more. And then we have another person who’s looking into just the impact on low-income students to see if we can find out more there about the details of how this works. And I’m really curious to see if we can work with large state systems, what can we find about the most time efficient, most beneficial ways to apply transparency and learning and teaching in community college settings. And I’ve also noticed that as I begin to do more work internationally, because I now have more flexible time to be able to do that, the colleges of applied sciences, like in the European Union, for example, they have a kind of three-year degree that is similarly focused on students’ learning something from their degree like they do here in a community or technical college that will lead them on a path into sustainable long-term employment and a career. So I think that this is going to be a really beneficial place to focus TILT efforts and to do some more research about how can we long term have an impact on not just students’ education, but how that is a pathway into a career. And I’m hopeful that we can find out more about that, like the longer long-term effect of TILT. But I’m also really open to inviting anyone who wants to do more research with the mountains of data that we’re sitting on, to discover something that is of interest to them about how students are learning, and how we can help students succeed more.

Rebecca: I really love all the resources and examples and research materials, worksheets, that are on the website. They’re really handy for folks who are starting out. We always wrap up by asking: “What’s next?”

Mary-Ann: What’s next for me, and then what might be next for teachers and students too. So we’ve talked a lot in detail about how TILT works, and how we know it works, and what more we want to discover about how it works. But I want people to remember that this is really a small effort, it’s a very easy lift that has a really large benefit from the size of that lift. And so I would really encourage teachers and students, if they’re going to do anything at all, even if they have no time to adjust any assignment prompts or to adjust anything about the way that they’re teaching or learning in a classroom. If you use any one single thing, I would say use that framework that we built for the students that has the footnotes at the bottom, and it’s called the “unwritten rules” and that framework, and I think you could probably provide a link to it, that’s what I would hope people would do next, just take that framework with you to anywhere that you’re communicating with your students. And the students will tell you how to make the work more transparent for them. Ask students what they see as the purpose, the task, and the criteria. And you’ll discover very quickly, very efficiently, how you can make that work more transparent so that all students are starting to do the work with the same understanding about what’s expected and with the same set of resources that they need in order to do it. So that’s what I hope is next for teachers and students.

Rebecca: And I hear all the faculty cheering about efficiency, and quick. [LAUGHTER]

Mary-Ann: That’s good. Yeah. So that would be the most time efficient thing to do, I think is to have students teach us more about how to be more transparent. And then in terms of researchers, I’m hoping that researchers will think about what can we learn more about? Can we learn more about what motivates students? Or what forms students’ sense of belonging? Is there anything in our survey data that would shed light on any kind of work you’re doing around that? Is there anything in our survey data that would shed light on more of the research on neuroscience and how that’s impacting learning? Or is there anything in the research that we have in our survey data that might help clarify what would be most beneficial for the very most at-risk students? So if we look at federal government statistics, National Center for Education Statistics about retention rates and graduation rates of different populations of students? Can we double down and look at those students with the very lowest graduation rates? And can we find something about TILT that would be the most beneficial for that population of students? To me, that’s a really important and interesting question. And then I really do want to be finding more locations where TILT could be useful, small scale for teachers and students, large scale for state systems or national systems to be thinking about how to apply this all for the good of students success, and for the satisfaction and time efficiency for teachers work as well.

John:If you’re finding these results of long-term persistent effects from just a single intro course, imagine what would happen if all intro courses use the TILT approach. I imagine the effect would be magnified if it was adopted at a broader level and it is being adopted at many institutions at a broader level.

Mary-Ann: I absolutely agree with you that applying TILT across the largest introductory gateway required courses at any institution would be probably the most efficient way to improve retention and graduation rates. Because if you go for the largest group of students as they enter, and you reduce the number of those students who might be thinking or doubting or wondering if they should continue, and if you increase the number of students who feel confident, who are aware of the value of what they’re learning, in terms of skills and knowledge, and if you increase the number of students who persist from the first year on, then that’s where you’re going to have the best success in increasing retention and graduation rates. I agree with you. I think that’s a really strategically wise place to invest TILT effort.

Rebecca: Well thank you so much. We’re looking forward to sharing this with our listeners.

Mary-Ann: And thank you so much for the opportunity to talk with you this afternoon, I really appreciate it

John:Thank you for all the work you’re doing.

[MUSIC]

John:If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]