Category: higher education

353. Beyond ChatGPT

Posted in accessibility, artificial intelligence, edtech, educational technology, engagement, equity, faculty learning community, higher education, inclusion, learning, teaching
August 7, 2024

Faculty concerns over student use of AI tools often focus on issues of academic integrity. In this episode, Marc Watkins joins us to discussion how the use of AI tools may have on student skill development. Marc is the Assistant Director for Academic Innovation at the University of Mississippi, where he helped found and currently directs the AI Institute for Teachers.

Show Notes

Transcript

John: Faculty concerns over student use of AI tools often focus on issues of academic integrity. In this episode, we explore other impacts that the use of AI tools may have on student skill development.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by

John: , an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Marc Watkins. Marc is the Assistant Director for Academic Innovation at the University of Mississippi, where he helped found and currently directs the AI Institute for Teachers. Welcome back, Marc.

Marc: Thank you guys. I really appreciate it. I think this is my third time joining you all on the pod. This is great.

Rebecca: Today’s teas are:… Marc, are you drinking any tea?

Marc: I am. I’ve gotten really into some cold brew tea and this is cold brew Paris by Harney and Sons. So very good on a hot day.

John: We have some of the non-cold brewed version of that in our office because the Associate Director of the teaching center enjoys that Paris tea so we keep it stocked pretty regularly. It’s a good tea.

Rebecca: Yeah.

John: My tea today is a peppermint spearmint blend.

Rebecca: Sounds nice and refreshing. I have a Brodies Scottish afternoon tea, and it’s hot. And it’s like 95 here. And I’m not really sure why I’m drinking hot tea in this weather. But I am. [LAUGHTER]

John: Well, I am here in North Carolina, and it’s 90 degrees. So it’s much cooler down here in the south, which is kind of nice. [LAUGHTER] And actually, it’s 71 degrees in this room because the air conditioning is functioning nicely.

Rebecca: Yeah, my studio at home… the one room where the air doesn’t work. So hopefully I don’t melt in the next hour.

John: So we’ve invited you here today to discuss your recent Beyond ChatGPT substack series on the impact of generative AI on student learning. Many faculty have expressed concerns about academic integrity issues, but the focus of your posts have been on how student use of AI tools might impact skill development. And your first post in this series discusses the impact of AI on student reading skills. You note that AI tools can quickly summarize readings, and that might cause students to not read as closely as they might otherwise. What are some of the benefits and also the potential harms that may result from student use of this capability?

Marc: When I first really got into exploring generative AI, really before ChatGPT was launched, there were a lot of developers working in this space, and everyone was playing around with openAI’s API access. And so they’re like, ”Hey, what would you like to build? And people would go on to Twitter, which is now X, and Discord and basically say, “I would like this tool and this tool.” And one of the things that came about from that was a reading assistant tool, which was called Explainpaper. And I think I first played around with this in the fall 2022, and then deployed with students in the spring of 2023. And the whole idea that I had with this and that design was to help students really plow through vast amounts of papers and texts, and so students that have hidden disabilities, or announced disabilities with reading and comprehension, and also students that were working on language acquisition, if you’re working in a second or third language, this type of tool can be really helpful. So I was really excited and I deployed this with my students in my class thinking that this is going to help so many students that have disabilities, that will go through a very challenging text, which is why I set it up as, and it did. The students initially reported to you that this was great. And I met with a lot of my students, and one of them said that she’d had dyslexia her whole life and never wanted to talk about it, because it was so hard and this tool for her was a lifesaver. And so that was great. But then the other part of the class basically said, “Hey, I don’t have to read anything at all ever.” And they don’t have any issues, they were just going to offload the close reading skills. And so I had to take a step back and say, “But wait, that’s not what we want this to actually happen. We want you to use this if you get into a pain point in your reading process, and not completely offload that.” So this really became this kind of a discovery on my part that AI can actually do that, it can generate summaries from vast amounts of texts. There are some really interesting tools that are out there right now: Google’s notebook LM, you can actually upload, I think, 4 million words of your own text to it in 10 different documents, and that will summarize and synthesize that material for you. And like the other tools we played around with the Explainpaper, it can change the summary that it’s generating for the actual document to your own reading level. So you could be reading a graduate level research paper, and you’d like it to be read in an eighth grade reading level, it will change the words and the language of that. So yeah, that could have helpful impacts on learning. It could also lead to a lot of de-skilling of those close reading skills we value so much. So that’s really how this started, was kind of coming up here too, and thinking about “Man, this was such a wonderful tool. But oh my gosh, how is this actually being used? And how has this been marketed to students through social media?”

Rebecca: How do you balance some of these benefits and harms?

Marc: By banging my head against the wall and screaming silently into a jar of screams?

Rebecca: I knew it.

Marc: Yeah, the problem with the jar of screams is every time I open it, some of the screams I put in there before escape before the new ones can come in. That’s a great question. So every single one of these use cases we’re gonna talk about today has benefits but also has this vast sort of terror of being offloading the skills that we would associate with them that are crucial for learning. The most important thing to do at this stage is just to make sure the faculty are aware that this can happen and that this is a use case, that’s the first step. Then the next step is building some friction into the learning process that’s already there. So for reading as an example, something that we do usually is assign close reading through annotation, whether that’s a physical pen and paper, or you could use digital annotation tools like Perusall or Hypothesis to help you go through that, that slows down that process if you’re using AI, and really focuses on learning. So when I say friction, it’s not a bad thing, and point of fact too friction… it’s actually sort of crucial for learning. The one challenge we’re faced with most of these tools is that they’re providing or they’re advertising a friction-free experience for students. And we want to say to them, “Well, you may not want to offload these skills entirely, you want to make sure that you do this carefully.” The main thing too I would think about this is I could never ban this tool even if I wanted to, because you don’t have any control over what students use to read outside of the three hours or so that you’d have in class with students a week. And it would be very beneficial for those students. So we can discuss to look forward to that had all those issues to use it. It’s just basically persuading them to use this in a way that’s helpful to them.

John: It reminds me a little bit of some of the discussions years back on the use of things like Cliff’s Notes for books and so forth, except now it’s sort of like a Cliff’s Notes for anything.

Marc: Indeed, Cliff Notes on demand for anything you want, wherever you want it, however you want it, too. And so how we could do that… what I’d said to my students at the time to kind of get them to be shocked of this is that, “You know, what would your reaction be if I used this to read your essays instead of going through and reading all of it and just giving a nice little generative summary” and one of my students said, “Well, you can’t do that. That’s cheating, you’d be fired.” And I had to explain to them, no one even really knows that this exists yet. There’s no rules. There’s no ethical framework. That’s something we’re going to have to come up with together, both faculty and students talking with each other about this.

Rebecca: It seems like the conversations you were having with students about how to maybe strategically use a tool like this, in this particular way, was an important part of harnessing the learning out of the tool, rather than the quote- unquote cheating aspect of the tool.

Marc: Oh, absolutely. Yeah, I mean, the thing we’ve been seeing with every single generative tool that’s been released too, whether it’s for text generation, or for augmenting reading, or doing some of the other use cases, we’ll talk here today, it does take a lot of time and effort on the part of the instructor to basically say, “Look, this is how this tool should be used to help you in this context in our classroom. How you use this outside of the classroom, that’s gonna be on you. But for our intents and purposes here, too, I would like to advocate that you use this tool this way. And here’s the reasons why.” Now asking every educator to do that is just too much of a lift, right? Because most of our folks are just so burnt out with everything else that they have to do. They’re focused on their discipline-specific concerns. They’re not really even on the radar, the fact that this technology exists, let alone how to actually deal with it. Trying to do part of the series is obviously advocating for people to be aware of it. But the next step is going to be building some resources to show how they can use things like annotation and why that matters. And a very quick way for teachers regardless of discipline to start using in their classes.

Rebecca: Your second post in this series examines the effect of AI tools on student notetaking skills. Can you talk a little bit about what might be lost when students rely on AI tools for notetaking and how it might be beneficial for some students as well?

Marc: Yeah, so a lot of the tools are using assisted speech generation software to actually record lecture like we might be using right now on this podcast and a lot of other people are too, and how they’re being marketed to students is just to sort of lean back, take a nap and to have the AI listen to the lecture for you. And some of the tools out there, I think one of them, it’s called Turbolearn.ai, will also synthesize the material, create flashcards for you, create quizzes for you, too. So you don’t have to do that processing part within your mind, which is the key thing. So, notetaking matters. In fact, it can be an art form. I’m not saying that our students treat notetaking like an art form either too, but there are examples of this that is somewhat of an artistic talent, because you as the listener are not just taking down verbatim what’s being said, you’re making these critical choices, these judgments to record what matters and put it in context of what you think you need to know. And that’s an important part of learning something. One thing that I did too as a student when I was in a community college in Missouri as a freshman, I volunteered as a note taker, and back then we did not have assistive technology. I had a pad of paper for myself for my notes and I had a pad of paper that had larger areas to write in for a student who was functionally blind. So I would do two notes at the same time. One in a font that was my size, one was a larger font that he could read with an assistive magnifying glass from one good eye that he had, it was shocking to me that this is what they did. So the first part of the class is do we have anyone who could help take notes? I was like, “Okay, sure I can.” And that’s how that student had notes for him. Obviously having a system like this in place helps those students so much more than having a volunteer notetaker go through this that’s rushing between one set of notes and another too. And using that in an effective way that’s critical, that is thoughtful about how you’re going to engage with it to, is meaningful for their learning versus just hanging back, sitting down letting the AI listen to you for lecture forty.

John: And another mixed aspect of it is the fact that it does create those flashcards and other things that could be used for some retrieval practice. That aspect, I think, could benefit a lot of students. And not all students maintain a very high level of focus and sometimes miss things. So I think there could be some benefits for everyone, as long as they don’t completely lose his skill. And I think maybe by reminding them of that, that could be useful in the same sort of way you talked about reading. But it’s a lot of things to remind students of. [LAUGHTER]

Marc: That’s lot of things to remind them of, too. And keep in mind, it’s a lot of temptation to offload the skills of learning to something that’s going to supposedly promise you to do that skill for you, or do that time-intensive skill for you too. I would love to have this employed in a giant conference somewhere. In fact, I’d love to go into the hallway of a conference and see all these transcripts come together at once in the overhead almost like you’re waiting for a plane flight at your airport, and you’re just seeing the actual material go through there too. That would be exciting for me too, to see what other people are talking about too… maybe I want to pop into this session and see that as well. So I think there’s tons of legitimate use cases for this. It’s just where’s the sort of boundaries we can put in place with this. And that’s true for almost all of this. I was talking to my wife last night, and I said, “When I was growing up, we had a go kart that a few kids in our neighborhood shared and it had a governor on the engine that made sure that the go kart wouldn’t go past 25 miles per hour, because then you’d basically die because it’s a go kart, it’s not really safe.” None of these tools or these technologies have a governor reducing their ability to impact our lives. And that’s really what we need. The thing that’s shocking about all this is that these tools are being released in the public as a grand experiment. And there’s no real use cases about or best practices about how you’re supposed to use this for yourself in your day-to-day life, let alone in education, in your teaching and learning.

Rebecca: I mean, anytime it feels like you can take a shortcut, it’s really tempting, the idea of turbo learning sounds amazing. I would love to learn really quickly. [LAUGHTER] But the reality is that learning doesn’t always happen quickly. [LAUGHTER] Learning happens from mistakes and learning from those mistakes.

Marc: Absolutely. It happens through learning through errors, it happens through learning through friction in many times. We don’t want to remove that friction completely from that learning process.

John: In your third post in the series, you talk about automated feedback and how that may affect both students and faculty. How does the feedback generated from Ai differ from human feedback and what might be some of the consequences of relying on AI feedback?

Marc: Well, so automated feedback is something that generative AI models, especially large language models, are very good at. They take an input based off of the students writing or assessment, and then the instructor can use a prompt that they craft to kind of guide the actual output of that too. So the system I used in the, I think spring of 2023, maybe it’s the fall of 2023 was MyEssayFeedback designed by Eric Kean. And he’s worked with Anna Mills before in the past too to try to make this as teacher friendly, as teacher centric, as possible, because I would get to design the prompts, my students would then be able to get feedback from it. And I use this in conjunction with asynchronous peer review, because it’s an online class. So they got some human feedback, and they got some AI feedback. The thing that was kind of shocking to me was that the students really trusted the AI feedback because it’s very authoritative. It was very quick, and they liked that a lot. And so I did kind of get into the situation where I wanted to talk with them a little bit more critically about that, because some of the things I was seeing behind the scenes is that a lot of the students kept on cueing the system over and over again, they’d get one round of feedback from the tool, they would try to go back and using air quotes right now so your audience can see this “fix” their essay. And my whole point is their writing is not broken. It doesn’t need to be fixed. And generative AI is always going to come up with something for you to work on in your essay. And one student I think went back seven or eight times saying “Is it right now? Is it perfect?” And the AI would always say something new. And she got very frustrated. [LAUGHTER] And I said “I know you’re frustrated, because that’s how the AI is. It’s not smart, even though it sounds authoritative, even though it’s giving you some advice that is useful to you. It doesn’t know you as a writer, it doesn’t understand what you’re actually doing with this piece.” So that crucial piece of AI literacy, knowing that what the limitations are too, is a big one. I think also when you start thinking about how these systems are being sold, in terms of agentic AI, we’re not there yet. None of these systems are fully agentic. That involves both strategic reasoning and long-term planning. When you can see that being put in place with students and their feedback, that can become very, very scary in terms of our labor for faculty to understand that, because there are some examples of some quirky schools, I think it’s the Health Academy in Austin’s one of them that have adopted AI to both teach and provide feedback for students. And I know there’s some other examples too, that talk about the AI feedback being better than human feedback in terms of accuracy. And that is something that we are going to have to contend with. But when I provide feedback for my students, I’m not doing it from an aggregate point of view, I’m not doing it to try to get to the baseline, I want to see my student as a human being and understand who that writer is, and what that means to them. That’s not saying that you can’t have a space for generative feedback, you just want to make sure you do so carefully and engage with it in a way that’s helpful for the students.

John: And might that interfere with student’s development of their own voice in their discipline?

Marc: I think so. And I think the question we don’t have an answer to yet is what happens when our students stop writing for each other or for us and start writing for a bot? What happens when they start writing for a robot? That’s probably going to change their voice and also maybe even some of their ideas and their outlook on the world too, in ways that I’m not all that comfortable with.

Rebecca: It does seem like there’s real benefits to having that kind of feedback, especially for more functional things like grammar and spelling and consistency and that kind of thing. But when you lose your voice, or you lose the fresh ways of saying things or seeing things in the world. [LAUGHTER] you lose the humanity of the world, [LAUGHTER] like it just starts to dissipate. And to me, that’s terrifying.

Marc: It’s terrifying to me too, to say the least. And I think that’s where we go back into trying to find, where’s the line here? Where do we want to draw it? And no one’s doing it for us. We’re having to come up with this largely on our own in real time.

Rebecca: So, speaking of terrifying [LAUGHTER] and lines, you note about how large language models are developing into large multimodal models that simulate voice, vision, expression, and emotion. Yikes. How might these changes affect learning, we’ve already started digging into that.

Marc: Yeah, so this is really about both Google’s demo, which is I think called Project Astra and also openAI’s demo, which is GPT4 omni. Half of the GPT4 omni model is now live for users, you can use the old version of the large language model too for resources, but the other half is live streaming audio and video. And the demo used a voice called Sky that a few people, including Scarlett Johansson, said “that sounds an awful lot like me.” And even the creator of openAI, Sam Altman, basically said that they were trying to go for that 2013 film Her where she started as the chatbot to Joaquin Phoenix. And basically, this is just the craziest thing I can ever think of. If openAI goes through with the promise of this, it will be freely available and rate limited for all users. And you can program the voice to be anything you want, whenever you want. So yes, it’s gonna be gross and creepy, there’s probably going to be people that want to date Sky or whoever it is. But even worse than that, there will probably be people who want to program this to be a political bot. And they only want to learn from a liberal or conservative voice, if they only want a voice that is of their values and their understanding of the world. If they don’t like having a female teacher, maybe they only want a male voice talking to them. Those are some really, really negative downstream effects of this that go back into how siloed we are right now with technology anyway, that you can now basically create your own learning experience or your own experience, and filter the entire world through it. We have no idea what that’s going to do to student learning. Sal Khan thinks that this is going to be a revolution, he wrote about this in Brave New Words. I think that this is going to be the opposite of that. I think it’s going to be more chaotic. I think it’s also going to become, for us as teachers, very difficult to try to police in our classes, because at my understanding of this, this is a gigantic privacy issue. If your students just come up and you’re having a small group discussion or anything else that’s going on too, and one of them activates this new multimodal feature in GPT4 omni and there are voices streaming, they’re talking to the Chatbot and everything else, anything that goes into that is probably going to be part of its training data in some way, shape, or form. Even Google’s demo of this using Project Astra, part of the demo was actually having someone walk around a room in London, they had stopped on a computer screen that was not the actual person’s computer screen and it had some code running for encryption and it read the encryption out loud. It said what it was. So there’s some big time issues that are coming up here too. And it’s all happening in real time. We don’t even have a chance to basically say, “Hey, I don’t really want this,” versus “Oh, this has now been updated. I now have to contend with this live in my own life and in my classes.”

John: Going back to that issue of friction that you mentioned before, Robert Bjork and others have done a lot of work on the issue of desirable difficulties. And it seems like many of these new AI tools that are being marketed to students are designed to eliminate those desirable difficulties. What sort of impacts might that have in terms of student learning and long-term recall of concepts.

Marc: I love desirable difficulties too, and I think that’s a wonderful framing mechanism outside of AI to talk about this too, and why learning really matters. I think the downstream consequences that if this is widely adopted by students, which I think a lot of tech developers want this to happen, and we don’t see this sort of sporadic usage. which we’re seeing right now… to be clear to your audience, not every student is adopting this, not everyone’s using this, most of them are really not aware of it. But if we do see this widespread adoption of this, too, it is going to have a dramatic impact on the skills we associate with reading, the skills that we associate with creating model citizens who are critical thinkers and ready to go into our role to actually participate in them. If we really do get to the situation where they use these tools to offload learning, we’re kind of setting up our students for being uncritical thinkers. And I don’t think that’s a good idea.

Rebecca: Blah. [LAUGHTER] Can you transcribe that, John? [LAUGHTER]

John: I will. I had to do a couple of those. [LAUGHTER]

Marc: Well, blah is always a great version of that. Yeah. [LAUGHTER]

Rebecca: I only have sound effects.

John: One of the transcripts mentioned “horrified sound” as the transcript.[LAUGHTER]

Rebecca: I think that’s basically my entire life. These are the seeds of nightmares, all of them… seeds of giant nightmares.

Marc: Well, I think the thing too, that’s so weird about this is that, yes, and this is kind of getting into the dystopia version of it, but there’s clearly good use cases for these tools, if you can put some limitations on it. And if the developers would just sort of pause and think not just as someone wanting to make money, but as someone who would use this tool to actually learn or be useful to their lives, what areas do they want to design to actually preserve that sort of human judgment, that human sort of friction in learning is going to be meaningful for that going forward?

Rebecca: Yeah, guardrails and ethics would be great.

Marc: Absolutely.

Rebecca: So a number of these tools are also designed to facilitate research. What’s the harm? What harm might there be when we rely on AI research tools more extensively, and get rid of that human judgment piece?

Marc: Yeah, I think one of the tools I used initially was Elicit and Elicit’s probably the most impressive research tool that’s currently available. It is expensive to use, so it’s hard to sort of like practice using it now. It was free initially. Consensus AI, I think is the best ChatGPT plugin that you can use through the custom GPT store. But what Elicit does is it goes through hundreds, if not 1000s, of research papers, and it automates the process of reading those papers for you, synthesizing that material, and giving you an sort of aggregate understanding of the state of knowledge, not just within your research question, but perhaps even in your field of research you’re trying to acquire. So you’re basically offloading the process of research, which for a researcher to do that, takes hundreds upon hundreds of hours of dedicated work, and you’re trusting an algorithm that you can’t audit, you can’t really ask how it came up with its response. So yes, it’s a wonderful tool, when it works and when it gives you an accurate response. Sometimes the responses are not accurate in the least. And if you haven’t read the material too, it’s very difficult to sort of pick up on where the machine is making an error. So yeah, there’s a lot of issues if we just uncritically adopt using this tool, versus if you sort of put some ground rules and ethics about how to use this, to support your research, to support your learning as well. And I think that’s what we want to try to strive for with all of these. And research is just one level of that.

Rebecca: We all have our own individual assumptions that we make when we do things, many of which we’re not aware of. But when we’re relying on tools like this, there’s many more layers of assumptions that we might not be aware of that are built into the software or into the tools or in the ways that it’s doing its analysis or synthesis that I think seems particularly concerning to me.

Marc: Yes, the bias, the sort of hidden biases that we’re not even aware of. And then the developers I don’t think are aware of either, too, is another layer that we can go into and think about this. I say that layer, because this really is like an onion, you peel back the layer, there’s another layer there, another layer, another layer, you’re just trying to get to the point where it’s not so rotten anymore. And it’s very difficult to do because the way that this has been shaped to do is to just accelerate those human tasks as quickly as you can to reduce as much friction as possible, so that you can just sit back and get a response as quickly as you can from this. And in a lot of ways the marketing of this basically describes this as almost like magic. Well, it’s not magic, it’s just prediction and using massive amounts of compute to get you to that point as well, but there are some serious consequences, I think, to our learning if we just uncritically adopt that.

John: Going back a bit, though, to early in my career, I remember the days of card catalogs and indexes where you had to read through a lot of material to find references. And then finding more recent work was almost impossible unless you happen to know of colleagues doing this work at some other institution, or you had access to the working papers of other institutions because of connections. The fact that we have electronic access to these files, and you don’t have to wait a few weeks for one to be mailed to you, or go through interlibrary loan. And that we can do searches and get indexes or get abstracts, at least for these articles, takes us a long way forward. And one other thing is that I do subscribe to Google Alerts in some of my popular papers. And then I occasionally, maybe once every month or so when I see some new ones, I’ll just look at the article and about half the time the person who cites the article gets it wrong, they actually refer to it in a context that’s not entirely relevant. I think in some ways, maybe relying on an AI tool that generates some summaries of the articles before people add them to their bibliography or footnotes, might actually, in some cases, improve the work. Going back again to the early days, one of the things I enjoyed most when I was up there in the periodical sections of the library were the articles around the ones that I was looking for, they’d often lead to some interesting ideas. And that doesn’t come up as much now when you’re using an online search tool, but as you’ve noted all along, we have both benefits and costs to all this. And in this issue, I’m kinda thinking some of the benefits might be worth some of the costs, as long as people follow through and actually read the articles that seem relevant.

Marc: I think that’s the key point too. So long as this leads you to where you want to go. That’s just like what Wikipedia basically is, that’s a great starting point for your research, it just leads you back to the primary sources to actually go in there and read to do it. The challenge that I think we see, and this is what it goes back down to where we go back that onion sort of analogy, is that a lot of the tools that are out there now …I think one of them is called ProDream AI or something like this… will not only find the sources for you, but then it will draft the lit review for you as well. So you don’t have to go through that process of actually reading it. And obviously, that’s where we want to pause and say this isn’t a good idea. But I agree with you completely. John, we are in a digital age, we have been for over 25 years now too. And in fact, when I students is: “This was a terrible experience because I can’t navigate this thing. This is just so horrible for me to do.” And yet every time I’ve done this with the AI research for my students, the interface design is much more easy for them to actually establish and look at sources and go through this and think about it, and part is because the algorithm is now using some of those techniques to actually narrow down their sources too and help them identify them as well. So yeah, there’s definitely benefits to it. It’s not all black and white, for sure.

Rebecca: There’s a lot of gray. [LAUGHTER] I think one of the things that you’re hinting at too is this difference between experts using a tool and novices or someone who’s learning a set of skills. And the way that these tools are designed, an expert is going to be able to use a tool and have a judgment call about whether or not what’s provided is accurate, helpful, relevant, etc. Whereas a novice doesn’t know what they don’t know. And so it becomes really challenging for them to have the information literacy skills that may be necessary to negotiate whether or not this is a path to follow or not. For me, that’s one of the biggest differences when we’re talking about using these tools in a learning context versus using these tools in a professional context are ways to save time to get to the point or get to an end result more swiftly.

Marc: Oh, absolutely. I think that thinking about the audience who’s using it too: a first-year true freshman student, using a tool like this versus a third-year PhD student working on their thesis is a totally different audience, totally different use case. For the most part, the PhD student hopefully has that literacy needed to effectively use these tools already, they might still need some guidance, might need some guardrails and some ethical framing for this too, but it’s a very different situation from that freshman student. I think that’s why most faculty aren’t thinking about how they’re using these tools, because they already have many of those skills already solidified. They don’t need to have a refresher course necessarily on research because they’ve done this now for a large part of their career. For their perspective, adopting these tools is not going to necessarily de-skill them, it might just be necessarily a timesaver in this case.

Rebecca: And what skills we’re offloading to a tool. Some things are just repetitive tasks that take a long time that a tool is great to solve. Just a kind of a waste of time versus really like critical thinking or kind of creative aspects of maybe some of the work we do.

Marc: The tool I want, and I think this exists, I just haven’t found it yet is when I’m trying to write a post and instead of trying to search for the URL to go into the actual title that automatically just finds the URL for me to click on it. I’ll review it for a second, because it takes me so much time finding the URL for the page when I’m doing either a newsletter or I’ve tried to update a website, that would be amazing. Those are some of the things that we could use really easily to cut down on those repetitive tasks, for sure.

John: In your six post in this series, you talk a little bit about issues of ethics. And one thing that I think many students have noted is that many faculty have extremely different policies in terms of when AI is allowed, if it’s allowed, and under what conditions it’s allowed, which creates a lot of uncertainty, and faculty aren’t always very good at conveying that information to students. What should we be doing to help create perhaps a more transparent environment for our students?

Marc: Well, I think transparency is the key word there. We want to, if we’re using these tools for instructional design, be transparent about what we’re using this to, just to model that behavior for our students. So if I develop a lesson plan or use a slide deck that has generated images, I want to clearly identify what part of AI was in that actual creation and talk about why that matters in these situations. What concerns me is that these tools are being turned on left and right for faculty without any sort of guides or best practices about that. I actually asked for Blackboard to have a feature built in with a new AI assistant, so it could identify what was AI generated with a click of a button. There’s no reason why you can’t build something that tracks what was generated by AI within the learning management system. And the response that I’ve gotten to is: “Who basically cares about that?” Well, I kind of care about that, and I care about this for the effects we’re trying to do for our students as well. But yeah, I think adopting a sort of stance of transparency as a clear expectation, both for our own behavior and our students behavior is going to be more meaningful than turning to sort of an opaque AI detector that’s only going to give you a percentage about if this is aggregated content or human content or completely misses the entire situation and misidentifies a human being as AI or vice versa. And that’s something I think we want to focus on as being that human in the loop situation here too. And really not offloading ethics in this casein just trying to teach it. It is hard to do that when the technology is changing rapidly before your very eyes, though. And that’s what this has felt like now for the last two years, I think.

Rebecca: You’re really concerned when faculty lean on an AI detection tool as the only way of identifying something that might be AI generated or an academic integrity violation of some sort. Can you talk a little bit about the effectiveness of these tools, and when they might be useful and when they might not be useful?

Marc: Yeah, to me, they’re not very reliable in an academic context, there’s far too many false positives. And more importantly, too, the faculty that employ them, for the most part, aren’t really trained to actually use them. So some universities have invested in academic misconduct officers, academic honesty officers, or whatever you call them, for offices of academic misconduct, where they actually have people who are trained to both use these tools and provide this to faculty. I might be a so-called expert at AI, again, I’m gonna use air quotes here too, because I’m self taught like everyone else is. But I don’t think I would be comfortable in an academic based conduct investigation, trying to use these tools, which I barely understand how they work, trying to come up with a case for students to do so. The few areas that I’ve looked at that have engaged AI detection, do so as part of a process. And that process is just one part of the AI detector, they have independent advocates usually coming in talking with the students and talking with the faculty member, they don’t go to taking students up on charges at the first step, they often try to look at a restorative process to see if that’s possible. So if the first instance of a student using this technology, they would sit down, and they would be like a third party between the instructor and the student, and talk about if something could be repaired within the relationship. And if the student would acknowledge that an ethical breach actually happened here, not rule breaking, but an ethical breach that has damaged this relationship. And can that relationship be basically restored in some way. So to me, that’s the gold standard of trying to do this, that takes a whole bunch of resources to set up, lots of training, lots of time, versus let’s buy an AI detector for our entire university, turn this on and here’s a little one-page guide about how to use it. And that, to me, has set up this recipe for just chaos in the world too. And it doesn’t matter what detector you’re using. They all have their own issues. And none of them are going to ever give you a complete picture of what’s going on with that student. And I think the big challenge we’re seeing too is that we’re moving well beyond AI detection into some pretty intense surveillance. We’ve got some companies going to stylometry and going through keystroke logging, tracking what was copied and pasted into a document, when it was copied and pasted to. And these are all interesting novel techniques to try to figure out what was written and who wrote it, but they also have some downstream consequences, especially if they don’t involve training. I can imagine certain faculty using that time stamping technique to penalize students by not spending enough time on their writing, whether there is AI in it or not, they’re looking at: “you only spent two hours on this essay that was assigned over two weeks, that’s not showing me all you’ve learned, other students spent 5, 6, 7,12, 14 hours on this. So I think we have to be really careful about what comes online these next few years, and really approach it critically, just like we are asking our students to, so that we don’t look for a solution for this problem that’s based on technology.

John: One of the things you discuss in this essay, though, is the use of digital watermarking, such as the work that Google has been doing with synthID. Could you talk a little bit about how that works, and what your thoughts are about this.

Marc: So watermarking has been sort of on the perpetual horizon in AI for a long time. I think Scott Aaronson, he teaches at the University of Texas, he has been working with open AI for the last two or three years, he has really been very vocal about his own research into watermarking. And supposedly, he has a watermarking system at OpenAI working in the background, they just have not deployed it in public. Google’s synthID is not just for text, it’s for images, it’s for audio, it’s for video. And it’s really designed for what our world is going to very soon look like when you can have an AI that makes the President say anything, do anything, and deal with this vast amounts of misinformation and disinformation, too. And so what synthID is is their actual watermarking technique, and watermarking starts at the source of the generation. So their model was Gemini. And when watermarking comes online, it uses cryptography to put a code into the actual generation, whether that’s a picture, a video, music, or text that can only be deciphered from a key that they actually have. And so watermarking is this really interesting technique that it can be used to try to identify what was made by machine versus a human being. The challenge is, the last time I checked, there’s almost 70 different models on the market now that use multimodal AI or large language models. And those are the only ones I’ve been tracking, I’m sure there’s probably hundreds of others that are small that people have been developing, Google’s synthID model is specific to Google’s products, all the other watermarking schemes will be absolutely specific to OpenAI or Microsoft or Anthropic or any other companies. So it’s going to be the situation where you’re going to use a tool, then you have to rely on the tool to give you a classification if this is accurate or not. And from what I’ve also read, it’s pretty easy to break, because you can feed it into an opposing system’s AI or an open model. And it will simply rewrite it, removing the actual code in that process. So I don’t think watermarking is going to be a long-term solution, I do think it’s a good first step towards something that we can actually do. But it’s just a little bit too chaotic right now in the space. And we would need some massive sort of multinational treaties with different countries who don’t like to talk with us to try to get a sort of unilateral watermarking scheme in place that everyone will agree upon. And then we’d all have to cross our fingers that that key would never be released to the public. Because if that ever happened, that’s when the whole sort of house of cards falls apart.

Rebecca: So that’s kind of a fantasy.

Marc: …kind of a fantasy, but part of this stuff, I think, is marketing based. So like Google wants their products to be both safe and secure. You can’t have that safety and security unless you have some sort of system between there. And that’s what synthID is. I think that it can possibly work for audio, for video, and even for images. I think text is a lot more fungible than anything else, because it’s very easy to start copying and pasting things out there too. It’s also easy to write as yourself as a human being into a document. And that becomes very difficult to sort of gauge what was human versus AI using a watermarking type of program like this.

Rebecca: The final post in your series addresses the use of generative AI tools to design instructional content and activities. Instructors often find the use of AI tools to be very useful for these purposes, even if they ban it for their students. What concerns do you have about relying on AI tools in this context?

Marc: My concern there: “AI for me, not for you. It makes perfect sense to me going forward.” Yeah, obviously we go back into this phase of trying to model ethical behavior using the tools too and understanding why this matters. If you’re going to use a tool to grade or design rubrics, you want to be open about it. You want to be attributing what you use this tool for too, because your students are going to be looking at you and seeing “Well, how are you using this in your job? How am I going to be using this in my job when I graduate from here too?” That’s the actual grounding framework we can do for this for our students and for ourselves. If we can think about that and do that, then we don’t have to necessarily rely on technology as being the sole solution for this, we can start talking about “this is the ethical behavior I’m modeling for you too, this is the ethical behavior I expect from you too. Let’s work together and think about what that means.” Now, that’s not always going to be the solution for this situation, some students are going to listen to that, other students are going to smile at you and go back and happily generate away and try to get past it. But the fact is, we do have that agency in our part too. And that is something I think we should be leaning into right now. Because the connections we’re developing with our students too are, as of this time, still human-to-human base, for the most part. I want to value that and use that to try to persuade them on an ethical pathway.

Rebecca: Modeling our use of technology leads to so many different interesting conversations with students. I know that when I’ve talked about using assistive technology in my classes, having something to read to you if you’re having trouble focusing or using some of these tech tools to solve barriers that you’re facing in getting your work done. And sharing the ways that you use tools to do the same can be really helpful in leading to student success. So I can see how doing the same thing when it’s an AI product is relevant. I know that I used AI to generate a bunch of little case studies for one of my classes and I just told the students that that’s what I did… fed it in a prompt, and I made some tweaks to it, but this is where it came from. And they found it really interesting, and we ended up having a really interesting conversation about when it might be most relevant to use particular tools and when maybe it’s not as wise to use a particular tool, because it isn’t actually helping you in any kind of way. Or it’s defeating the learning, or it’s not really creating a good product in the end.

Marc: That’s a wonderful use case. I mean, sitting down there talking with them and saying how I use this, why use this, let’s get into discussion about this, maybe even a debate about that, too, is part of the learning process. And I’m glad you focused on the fact that about the assistive technologies, I want my students to use this technology if they need to, they don’t need to announce that they have a disability. We need to really be focusing on this fact, for education beyond. At our university, they have to go through a very formalized process to be recognized by the Office of Student Disabilities. It’s very expensive, it’s time consuming, that is out of reach for the vast majority of students, even if they felt comfortable going out there and advocating for themselves that way or if they had parents or other resources to do that. I want to design my classes so that students are aware that these tools exist, that they can use them and that they can be able to trust them to hopefully be able to use this in a way that is effective to their learning too and to trust them for that. That’s what I want. Now if that’s going to happen is another case, indeed. But that’s going to take time. The one thing I will say too, and I think that something that popped up here at a recent story that I read is that professors were moving from a point of despair to anguish with this technology, and I want us to avoid that more than anything else. Because that’s not the sort of stance we need to be taking for ourselves when we deal with this technology with our students. We can navigate this, it’s just going to take a lot of time and a lot of energy. And I hope administrations of various institutions are listening to that too, that they really need to focus on the training aspect of this technology, both for students and for actual teachers. This isn’t just something you flip a switch and turn on and say: “You guys now have AI, go learn how to use it…” that has been a recipe for disaster for it.

Rebecca: It’s definitely a complex topic, because there’s so much hope for equity in some of these tools, especially for students with disabilities. But then there’s also the really scary parts too. [LAUGHTER] So finding that balance, and making sure that both enter conversations when we’re having conversations about AI, I think, is really important. And I appreciate that today we’ve done that, that we’ve talked about some of the scary aspects, but also there’s some real benefits to having these tools available to our students and incorporating them and really having deep and meaningful conversations about them.

Marc: Absolutely. I think that one of the most powerful things I’ve done from the AI Institute is when you can get a skeptic and an early AI adopter at the same table together talking about these things back and forth. You really do see how people come out of their sort of silos and their positions and they can kind of come together and say “Yes, this is an actual use case or two. This is actually meaningful. This is good. How do I make sure that I can put some boundaries on this for my own students and their learning?”

John: So, we always end with a question which is so much on everyone’s mind concerning AI, and that is: “what’s next?”

Marc:Well, what is next indeed? So I think we’re all holding our breath to see if OpenAI is going to fulfill its promise and asking if they’re going to turn on this new multimodal system that lets you talk with it, lets it see you, because they have not done so yet. So we have a little bit of time. But that is going to be on everyone’s mind this fall if they do so. Because having an AI that can listen to you, talk with you, and have a voice that you get to program it, is going to be a new set of challenges that we have not really come up with yet.

John: Well, thank you. This has been fascinating, and your series is wonderful. And I hope that all faculty think about these issues, because a lot of people are focusing on a very narrow range of issues and AI is going to affect many aspects of how we work in higher ed.

Marc: Thank you, John. Thank you, Rebecca. This has been great too. And hopefully I’ll be putting some more resources into that series [LAUGHTER] when I have a chance to do so here.

John: And we will include a link to your substack in the show notes because you’ve got a lot of good information coming out there regularly.

Marc: Thank you.

Rebecca: Well, thanks for joining us. We hope to talk to you again soon.

Marc: I appreciate it. Thank you guys.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

352. Enhancing Inclusive Instruction

Posted in diversity, equity, first-generation, higher education, inclusion, learning, pedagogy, race, racism, reflective practice, social justice, teaching
July 31, 2024

We often don’t have the opportunity to hear directly from students about inclusive teaching practices. In this episode, Tracie Addy, Derek Dube, and Khadijah A. Mitchell, the authors of Enhancing Inclusive Instruction, join us to explore how student perceptions of inclusive teaching practices align with the growing consensus on what constitutes inclusive teaching.

After serving as the Associate Dean of Teaching and Learning at Lafayette College, Tracie will be transitioning to a new role this summer as the Director of the Institute for Teaching, Learning, and Inclusive Pedagogy at Rutgers University – New Brunswick. Derek Dube is an Associate Professor of Biology and the Director of the First-Year Seminar Program at the University of St. Joseph in Connecticut. Khadijah A. Mitchell is an Assistant Professor in the Cancer Prevention and Control Program at the Fox Chase Cancer Center in the Temple University Health System and Affiliated Faculty in the Department of Epidemiology and Biostatistics at the Temple University College of Public Health.

Show Notes

Transcript

John: We often don’t have the opportunity to hear directly from students about inclusive teaching practices. In this episode we explore how student perceptions of inclusive teaching practices align with the growing consensus on what constitutes inclusive teaching.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane , an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are: Tracie Addy, Derek Dube, and Khadijah A. Mitchell. After serving as the Associate Dean of Teaching and Learning at Lafayette College, Tracie will be transitioning to a new role this summer as the director of the Institute for Teaching, Learning, and Inclusive Pedagogy at Rutgers University – New Brunswick. Derek Dube is an Associate Professor of Biology and the Director of the First-Year Seminar Program at the University of St. Joseph in Connecticut. Khadijah A. Mitchell is an Assistant Professor in the Cancer Prevention and Control Program at the Fox Chase Cancer Center in the Temple University Health System and Affiliated Faculty in the Department of Epidemiology and Biostatistics at the Temple University College of Public Health. Welcome back Tracie, Derek, and Khadijah.

Derek: Thank you so much.

Tracie: Thank you.

Khadijah: Thank you so much.

Rebecca: Today’s teas are:… Tracie, are you drinking tea today?

Tracie: I am drinking tea. And I love that I’m drinking it from my Tea for Teaching mug that John gave me at the POD conference in November. And the tea that I’m drinking is just a simple kind of peppermint tea.

Rebecca: Sounds wonderful, and in the best mug possible. [LAUGHTER] How about you, Derek?

Derek: I’m just drinking water. But I would also like to shout out to my mug, which is actually a mug that I recently received from an instructor in my first-year seminar program here where the second semester of our first-year seminar program is a service-based learning course. And this mural that you see on the mug, and also behind me today, is one that their class created as a means to, as it says, inspire and engage and unite the globe around topics of digital and media literacy and things like that. So shout out to the mug, more so than the beverage within the mug.

John: It’s a very impressive mug and image behind you.

Rebecca: And something good to drink to, for sure. [LAUGHTER] How about you, Khadija?

Khadijah: So, I am drinking organic spearmint tea, and my mug has my name on it and in the spirit of inclusivity, this is probably the first thing I’ve ever had with my name on it. [LAUGHTER] When I go on vacation, I never get a keychain. So I’m very excited to drink the spearmint tea out of this mug.

Rebecca: I love a special mug. John and I don’t really have special mug.

John: Well, I do…

Rebecca: Oh you, do. Right.

John:…from the SUNY Oswego School of Ed, because we moved to a new recording room that they’ve loaned to us after we lost our other one. And along with the room, we got a tea mug, a nice insulated one.

Rebecca: And what kind of tea are you drinking?

John: In line with what we’ve just heard, I am drinking a peppermint spearmint blend today.

Rebecca: I missed the mint protocol today, unfortunately.

John: You didn’t check the calendar?

Rebecca: I didn’t. I’m sorry. [LAUGHTER] But I have a nice Jasmine black tea this morning, which feels nice and springy.

John: We’ve invited you here today to discuss your new book: Enhancing Inclusive Instruction: Student Perspectives and Practical Approaches for Advancing Equity in Higher Education. Your previous book, What Inclusive Instructors Do, was the basis of a very well received faculty reading group that we ran at SUNY Oswego, in collaboration with Jessamyn Neuhaus at SUNY Plattsburgh. So this book is a superb follow up. Could you tell us a little bit about the origin of this new book and how it connects to your earlier work?

Tracie: Absolutely, John, pleasure to do so. So when we think about inclusive teaching, and when I think about it, I would say in general, we know that instructors started at different points in their journeys. And when we wrote What Inclusive Instructors Do, we really knew that it will be really helpful just to have the voices of instructors who do that work, and their thinking about how they think about it, their mindsets, their values, etc. But also kind of distill it into some principles and practices that were very practical, because we know that that speaks well to instructors, but also that would allow them to reflect on their teaching. So after What Inclusive Instructors Do, we knew that instructors aren’t the only voice, right? [LAUGHTER] So we also knew that how important it was to really get students involved and students seeing the benefits of inclusive instruction, how did they experience these spaces in terms of their sense of belonging, welcoming environment, the feelings that it’s equitable or fair as well. So that kind of elicited our next step to really do a study on students and to find their perspectives. And a couple of things that we were able to do with this study, which I think is really helpful and useful for anybody, whether they’ve read What Inclusive Instructors Do or not, is we were also able to identify and ask students aspects according to what are the pitfalls? What have you seen not be inclusive? So that emerges kind of more strongly also in this book than the first book, for example. So we were able to really capture their experiences and these are students from a variety of institution types. They’re coming from a variety of classes they’ve been taking, various identity backgrounds, etc. So it really feels like now we’re talking about students like students [LAUGHTER], it’s their voice in this book. So Enhancing Inclusive Instruction, basically is just that, so that we can have a more comprehensive picture understanding of an inclusive classroom. What does it mean to students? What does it look like for instructors? And the beauty of it, and this is something that I’ve been speaking about a lot is, how are these alignments? What are the alignments between these two perspectives? What are the differences between these two perspectives? And how can we use this wider body of knowledge by having these viewpoints to really inform our practices in what we do in the classroom. So in general, both books, you can read both, you could read one, either/or, but they really were meant to really support inclusive instruction, no matter where an instructor is in your journey.

Rebecca: As a user experience designer and interaction designer, I really love the very student-centered or audience-centered nature of this book, and really bringing those voices to the forefront. Could you tell us a little bit about how you found the student participants to participate in your project.

Derek: So much like we set out with the goal of hoping that our teaching is as inclusive as possible, we also wanted that to be the case for our study, and our recruitment in the voices that we were hearing back from. So we made extensive efforts to have as broad a population be able to give us feedback, to participate in our surveys and our interviews. And that’s an important part, there were opportunities for our student respondents to reply via survey or via a virtual interview where they could potentially even go further and have deeper conversations around some of the topics if they wanted as well. So in order to recruit, after many collaborative meetings between myself and Tracie and Khadija, we came up with a broad spending plan to recruit students through using connections that we had, various educational listservs like POD, social media. We directly communicated with various institutions across the country, their teaching and learning center specifically, but also faculty that we knew that were invested in this type of work. And this included institutions at the community college level and research intensive institutions, liberal arts, HBCUs, colleges for students who learn differently, and basically any type of college you can think of. We also did lean on and take advantage of an opportunity to leverage technology through cloud research, to expand our reach as well. And through all of this, we did end up with well over 300 responses from various students when looked at geographically or institutionally that were very diverse. And we felt good with that as a basis for which to move forward.

John: One of the things we really appreciated when we were running the reading group was the reflection questions you had embedded in what inclusive instructors do. And this book also contains reflection questions spread throughout each of the chapters. While most teaching centers do encourage faculty to engage in reflective practice, it’s nice to see that embedded in the books that they’re reading. But might this be even more effective when faculty are reflecting on the student voices that make up so much of the focus of this book?

Khadijah: Right, so one of the things that stood out to us, when we did survey the inclusive instructors, there was a particular quote saying that inclusive teaching is a perpetual work in progress, informed by research, dialogue, and reflection. And when we think about that, in the context of incorporating student voice, it’s a perfect example of this. So when we think about research, actually collecting data from our students thoughout our courses, and I mean that as a community of inclusive instructors, and really fostering dialogue, and to be honest, what we found from what students say it is, that dialogue means we have to be willing to share and discuss and actually negotiate the space in our classrooms. And when we think about reflection, thinking that there’s no one set of practices for every inclusive instructor that we can act, and we’re not superheroes. So you can’t do all things, but you can do what you can to be responsive to your learners’ needs at the same time. So with that being said, when we came up with these reflection questions, and actually thinking about one of those quotes from the instructor, we saw that there were some mirroring with the first book in terms of course design, and creating a welcoming environment and conducting the course. But really, we saw that there were these 13 key themes, and that, although sometimes they reflected what we saw in the first book, we actually saw new themes emerge. So we really wanted to write reflection questions that captured that, that would help you delve deeper in thinking about the student perspective. And I think to echo something that Tracie mentioned, we really thought about potential pitfalls with these questions that we wouldn’t have thought if we didn’t speak with the students from their perspective to basically avoid certain types of things that may not have been the best for course design or maybe didn’t make everyone feel welcome, or when conducting the course. So that was very encouraging, the way that we could tailor these reflection questions based on what the students said.

Rebecca: And I love that so much. One of the tools that you developed and described in the book is the Protocol for Advancing Inclusive Teaching Efforts, or PAITE. Can you describe this instrument for our listeners?

Tracie: Sure. So the Protocol for Advancing Inclusive Teaching Efforts is a classroom observation protocol that enables instructors, in a formative way, low stakes, to obtain feedback on 15 specific instructional behaviors that are considered inclusive, equitable types of behaviors. And this protocol actually emerged from my work. And it’s a lot of what I’ve been doing in my own work is really thinking about tools that can support our instructors in building more inclusive classrooms. And so this PAITE actually originated from that. And we co-created it with students. So the student partners that work on a lot of the inclusivity efforts that I’ve done at my institution, basically supported this work and helped us measure reliability, et cetera. They were like, you know, the researchers, they helped us think about and devise it. So this particular protocol, I think, has been pretty impactful. We’ve done hundreds of observations with it at the institution that I was previously at. And thinking about all the debriefings, and all the feedback, it really supported instructors in thinking about their inclusive teaching practices. In fact, I like to share some of the responses that I’ve seen or heard them have around it. Some of them and even some of the student partners will actually, in future classes, they’ll say, “Am I doing this?” “Am I doing that?” They’ll actually reflect on the codes that are embedded within this protocol and continue to think about their practice. And that’s actually a very positive outcome of this. If they’re actually putting it in their awareness, awareness is critical for knowing are we actually doing these things that we say or hope we are doing in the classroom, and that has actually helped them really think about their teaching differently. And then some of the students said that actually, when they go to their own classes, they’ll start to think about the codes, because they’re so immersed in actually doing a protocol or conducting it with their partners as well. So that has been really great to see. And so we included it in the actual book, it has like a separate methods kind of paper, or how we developed it. But in the book, it really talks about how it can be used, and how it aligns a lot of the behaviors, and I’m actually doing a lot of this interesting mapping, align with what the perspectives of the students are, and also our instructors in What Inclusive Instructors Do. So it provides a really beautiful way to think about your teaching and monitor your approaches, reflect on your teaching, by actually having that data from your actual classroom. And I think it’s just really been helpful. We’ve embedded it in a variety of different academies and institutes, and different programming at my previous institution.

John: So you mentioned the themes that have arisen from the student interviews. Could you talk a bit about those themes?

Khadijah: Sure. So we saw 13 themes. And interestingly, seven of them centered on inclusive course design. So I’ll elaborate on the two that were the most prominent. We were really interested and surprised to see that maximizing student engagement from the first assignment and throughout the course. So the student has highlighted the importance of that first assignment, and several of them, they really believed that it helped them have a high degree of engagement. But also it was important that it required little content mastery, and built upon their existing knowledge. And there was a lot of positive feedback from successfully completing that first assignment. Students referred to short-term as well as long-term positive emotions from that first assignment. So that was really interesting. And group work… we often think about group work in higher education as a seminal active learning strategy, different learning teamwork, and things like that. So a lot of the students also echoed that, that it maximized the engagement. But what we saw them speak about was the pitfalls from their perspective with the group work, but in particular, those students that were neurodivergent and had disabilities. And so some students along with autism spectrum disorder, and also attention deficit hyperactivity disorder mentioned that, in that particular context, they didn’t feel most engaged with that. Student choice was very prominent. We talked a lot about: flexible deadlines were really important, individual student preferences on assignments and exams and accessibility needs and I think many of us are going to talk a lot about mental health and student well being. So that was very clear there. So those two were some of the key things, but also, including course policies that respectfully consider student time, structures that are easily followed and designed for student success. And students really wanted to encourage feedback on the instructions, diversity of materials, and finally, accounting for the type of course delivery. So I think in a post-COVID19 world, they talked about the importance of in person and hybrid and online themes. So that was the last theme that we saw in that regard.

Rebecca: When you quote students, you include references to their identities. And when faculty first start teaching, they sometimes assume that students in their classes are maybe just as they were as a student, but the mention of the diverse student identities in the book, hopefully help [LAUGHTER] remind faculty about the diversity of their students, and may also help encourage faculty to use a version of the “Who’s in class form.” Can you talk a little bit about that?

Derek: So with Inclusive teaching, I think identity is a key word to be keeping in mind, this concept of identity, this concept that everybody that enters the classroom has their own unique identity that brings with it a wealth of assets, but also potentially some personal bias. And this goes for the instructor just as much as the students if not more, and with the role the instructor is playing, it might be most important there to acknowledge it. So why we include the identity information that our student respondents shared willingly within their surveys was to that reason, for some of our faculty, for our readers, for our instructors, to acknowledge that the students that are responding are not identical to us when we were students, or us as we are now. But also beyond that, to acknowledge that no two students are the same. And we’re seeing different identities across these student respondents. And with all of this in mind that even the idea of teaching to the average student, well, there is no average student really, and that creates significant issues, to the idea of inclusive teaching. So for all of these reasons, we did think it was really important to include those identity aspects of the quoted students that was shared. We think this helps frame the quote, with some context, some important context, and hopefully also makes the quote more real for our readers. This came from somebody, a real person, a real student. So I do think, to your point about the “who’s in class form,” and things like that, that once we as instructors recognize and are aware of the diversity in our student populations, it becomes much more clear that getting to know our students is critical to teaching inclusively, and to including them in the learning process. So you mentioned the who’s in class form, which was part of the first book, and that’s certainly one way to do it that we’ve had a lot of success with. I’ve personally used it for years now in every class that I’ve taught since its development, and had just great personal experience and positive feedback from the students. There was a study that was published that showed student perception and instructor perception also very positive for the who’s in class form. So it’s a tool that we share, because we want it to be available for instructors to use, they don’t need to reinvent the wheel in this situation. It’s a great way to start from the very beginning from the first day like Khadija was mentioning with that first assignment being important… also, that first day, that first interaction, that first relationship building step to be one where it says, “I want to get to know you because you matter in this classroom.” So that’s one tool that we offer up to our readers and want them to be aware of, but there are other ways to do it as well. And I think that it’s important for instructors to think critically about their classrooms and their institutions and their contexts and determine what works best for them within their class to get to know their students.

John: You mentioned the importance of the context for instructors. And one of your chapters in the book focuses on providing advice for historically excluded instructors and their allies, because faculty are often in a very different context in terms of how students perceive them and how students perceive their identities. Could you talk a little bit about some of the things that you emphasize in this chapter?

Tracie: Yes, perfect segue. So in this chapter, we really wanted to emphasize that we see you, we hear you, like understanding instructors from historically excluded backgrounds can have different experiences in the classroom, as you mentioned, John, a bit, but also to give some advice and encouragement and support for how to still do this work because we know that a lot of instructors even from historically excluded backgrounds, in even my experiences and me being one [LAUGHTER], we know that there’s a lot of investment in this. There’s a lot of investment and then the importance of this. So some of the advice we give is general advice. What we know that students really appreciate in general and then also some really concrete advice to really support them in having the evidence that they are teaching well. [LAUGHTER] So the first thing to really think about and we talk about in this section is just care, like, how do we demonstrate care for our students, and regardless of whatever identities, care is critical. We also talk a little bit about other types of strategies. Some of them may or may not resonate with folks, again, all of the stuff we give as options to really think about, to experiment with, to practice with, to really think about what makes sense for you. So we give a couple of examples from people who’ve actually been doing this work as well. So things that have been published around storytelling, thinking about your story and who you are, what you choose to share with your students. And you might withhold some things, you might share some other things, and there’s good reasons for that, depending on your identity as well. We also talk a little bit about engaging your allies and advocates, because you really want to have people in your departments, and at your institution, who really value and can speak highly of your teaching, they know that you’re doing a good job, and they can advocate for that. So engaging them in various ways is important. We have these great tools we talk about, the PAITE, etc. So being able to actually have data that actually shows that you are doing things that align with what the research and evidence says can support students and their success in the classroom. So collecting that information, and just making sure you have it when you need it. And just for your own practice is also really critical, I think, for instructors, in this way, too. So those are just a few of many types of guidance that we do give in this section. I think it’s really important to consider what makes the most sense. I will say, in working with instructors, and also thinking about myself as a black woman teaching as well, it’s so contextual. It’s just like you can’t say that this will happen in this course, or this or that with this instructor. But we know that there’s a lot of variation that can happen. And I think a lot of these different guidance points can really help equip people to really think about what makes sense for them in terms of how they engage with Inclusive teaching practice. And the last thing I’ll share is that even in What Inclusive Instructors Do, if we break down and disaggregate the instructors a bit, we see a lot of instructors successfully talking about doing this work from historically excluded backgrounds. And so we know like it’s happening, we do it in our classes, as well. So this is not being meant to be like discouraging at all, but a very encouraging chapter. And just having some tools and equipment to kind of continue so that you also can be seen that way at your institution and you have that information.

Rebecca: I love these themes of reflection and data and being able to tell your story of who you are as a teacher and why you’re doing these inclusive practices and that there’s evidence that it’s working and supporting your students. Chapter 10 of your book focuses on a topic that we’ve certainly talked about a bit on this podcast, and it’s all over the news and conversations in higher ed, which is AI and instructional practices. Can you talk a little bit about some of the ways in which generative AI can support an equitable learning environment?

Derek: I think it’s a really important topic right now. And on the ground, what I hear most commonly instructors talking about, and faculty talking about are the challenges that AI presents us around commonly topics like academic integrity, like, “Oh, here’s my assignment, my students are just going to go to [insert AI platform here] and complete that assignment and then submit it. And it’s hard for me… the tools for understanding if that occurred or not, can be challenging still at this point.” But I think what our chapter wanted to do more than focus on those challenges is kind of flip the table on that and focus on what are the potential benefits, like you mentioned, to increasing equity in the classroom and inclusivity in the classroom? And they are manifold, when you take the time to actually look at it and think about it. And I think the chapter goes over a number of examples to show how this can be done. In terms of benefits that we see AI being able to be leveraged with in the classroom, incorporating and developing more efficient learning experiences is certainly one of them. It can kind of balance prior knowledge gaps that students come with to the classroom by being a tool which can be used for brainstorming and ideation and things along that line, so that maybe a student can go to a platform with a particular question or broad topic and use that platform as a tool to help kind of narrow down something that they want to explore more deeply, get some ideas that maybe they didn’t have the knowledge that they brought with themselves to class to come up with that idea on their own, but now they can get there, and it can kind of even up that playing field a little bit. There’s also additional benefits to multilingual learners in terms of helping learn the language that their course is being taught in, writing more fluently and efficiently, and better understanding written material that’s present. It can help personalize learning in a really equitable way. There’s a portion of the chapter that speaks to the idea of using AI as an additional support for overcoming kind of the muddiest points in a classroom. So if an instructor is either directly asking students what is the most challenging topic, what is the muddiest point, those points could then be used and imported into an AI platform to help navigate those situations, or it could just be pointed as a tool that students could use on their own when they come up with their muddiest point and when they find a topic that’s challenging for them. Now, of course, all of this has to go within the concept of how reputable is the response that we’re getting, how trustworthy is the output of these different platforms. But that also actually leads into what a really potentially beneficial assignment tool would be with this, which is the idea of using these platforms, seeing what their responses are, and then having the students kind of fact check and revise and edit and think critically about those responses. And there’s an assignment there where students can essentially take a prompt, import it into an AI platform, see what they get back, and then the assignment is actually go through this, revise this, edit it, fact-check it: “What would you change?” And that can be a really great way to enhance the critical thinking and evaluation aspects that we’re really trying to build in a lot of our students with, again, narrowing and balancing what’s needed for kind of background knowledge coming in. So there’s these and many other mechanisms that are talked about in the chapter, including ones for, as instructors it can be a tool to brainstorm ideas for: “What are the most inclusive ways I can teach on this topic? Give me a list of 10 of these?” And then the instructor can look through that list and say, “Hey, which one of these might work best for my class context? And how can I adapt that to work for me?” So there’s lots of ways in which AI can be a really positive tool, as long as we use it in that way, and are appreciative and recognize the potential challenges that it also faces.

John: It’s certainly a bit of a challenging time in terms of professional development for faculty, because as you said, so many people just say, “this is just another tool for students to cheat,” because they haven’t really played with it that must themselves, they haven’t really seen the potential. And I think that’s something that all teaching centers are going to have to continue to work on quite a bit to help faculty become more aware of how these tools can be used to complement other forms of learning, rather than as a substitute, which is what they’re concerned about. So in addition to your two books, what are some other strategies that faculty can use to help create a more inclusive environment in their classes? I know, you’ve talked about some of them, like the “Who’s in class form,” and so forth? But what are some other ways that faculty can work to become more inclusive? Or what are some of the things faculty who have not thought about this much should start focusing on most immediately?

Khadijah: That is a great question. I think, yes, we have a ton of great resources, like books and tools like PAITE, and our lived experience. There’s something that I really think this particular study and process and talking to students that really resonated with us is thinking about nonverbal communication. Often people think inclusive teaching takes so much effort, but even thinking about your personality, you’re smiling. So for example, smiling, warm and welcoming tone, being thoughtful about tone. For example, students say not getting angry. So to throw back to something that Derek mentioned, is thinking about your instructor background, and what type of strategies or the nonverbal communication are you communicating. So that is definitely easy if you’re in person. [LAUGHTER] But if you’re online, some things that I think you can think about is the use of thumbs up, clapping, things like that. Something that recently came up for discussion was when you are online courses, do you tell your students to change the tone of the thumbs up so that everyone is reflected in that space and welcome? Students have actually responded that they feel more included in that space with nonverbal communication. So using emojis in emails, some students actually thought that made their instructor more approachable, but I will say, that being an instructor in 2024, you have to watch that. There is a pitfall there. Studies have shown overuse of the emojis, induce sometimes less confidence in the instructor. I think, being mindful of any strategy that you can use to be mindful of the nonverbal communication. But the verbal communication I’ll say, unequivocally, I keep going back to this… [LAUGHTER] the mug… it is really important to know names, say names correctly, and to use chosen names. So name tents can help with that, using an annotated class roster, seating charts, photos, mnemonic devices, student introductions, those things are concrete. And I think the last thing I would say, is actually telling jokes. So a lot of instructors use humor in college and university classrooms. So that tends to go over very well, there’s a positive effect on students in the learning environment, and making people feel included. But again, a word of caution, thinking about how the joke can land. So it could be very powerful in that way. But also wanting to be mindful if you’re going to, in fact, use that particular strategy to make your students feel welcome.

Rebecca: Well, we always wrap up by asking: “What’s next?”

Tracie: I’ll share a little bit about some of the things that I’m working on. So with regards to my scholarship, I kind of hover between teaching and learning and also educational development in general. And so a couple of things that are exciting, I think, for the future is we’re currently proposing a multi-site PAITE study. And so that study will dig deeper into the impacts of, we’re hoping, of the PAITE in the classroom. Since now, we’ve been using it a lot, but we don’t have all the data from the classroom and the experiences of students and the instructors. So we’re going to be doing a multi-site study in the next year or two. So hopefully, you’ll see that come out. Another project that we actually just finished off, a smaller project, we talked about group work, students experiences with group work, and that being a challenging area. So we implemented an intervention this past year at my previous institution, in collaboration with another center, and several classrooms, several instructors in their classrooms. And so we did collect some data on that. We just are synthesizing it. So we’re gonna hope to write that up and also share the resource that we use, that was an intervention that we created that students use to help empower them to really have more effective group work experiences, more inclusive, more equitable kinds of experiences. And then a couple of things I’m working on in an educational development realm, we just collected data from the POD community and outside on educational development right now in our times, so we can track how it changes. So we’re in the process right now, this summer, of starting to write the book, which will be the next one… it’s written like almost every 10 years educational development in the age of… and we’re gonna ffill that in [LAUGHTER] based on the results of the study, and then some other educational development things. And lastly, as was mentioned earlier, I’m moving to a new institution. So I’m going to launch a new Institute for Teaching, Learning, and Inclusive Pedagogy. So that’s going to be very exciting to really think about what’s next there, it will be launching the first center at Rutgers University, New Brunswick, and it will have embedded within it all of this inclusive stuff. So there’s so many things to bring there and to increase and expand.

Derek: Yeah, for me, I mean, I think coming out of the work of this study that led to this book, I think where I am is continuing to spread the word, but then also putting it into action. And that means personally, for me, in my own teaching, taking all the things that I learned through this process of listening to students, working with my fantastic colleagues, and writing this book, and having conversations like these, putting that into action, and making sure we’re walking the walk, so to say, because that’s the only way this actually happens. And in my role at the University of St. Joseph, again, I’m not only a instructor, a faculty member in the biology department, but I’m also the director of the first year seminar program here at USJSJ.. a two-semester long program. It’s a true first year program, where the first semester is a three-credit course that is kind of half transitioning to college and half an academic component where the students get to dive into college learning. And the second semester is a one-credit course where it’s service based and meeting the mission of our university to engage in compassionate service. So as in my director role, I am thinking a lot about how do we help students transition to college in their first year? And I think there’s a lot that we can gain from listening to students. And that was reported out in this book that can be leveraged there. So building in some of these ideas of really getting to know the students, engaging in opportunities for student choice within our first-year seminar program itself as well, kind of coming out of the work of this, I’ve created a college success portfolio that we’re implementing in the program. We implemented it for the first time this past year that ties a lot of these features in that allows students to talk about right off the bat, their first assignment is nothing to do with the content, but it’s the “Why are you here? Why college? What are your goals, both in college and out of college? And this is something that the students can succeed with, right away to kind of set that form and let them know that their instructors here at the institution care about them as learners, and that the learning that they’ll go through will be personalized for them. So something that we’re piloting here at the University of St. Joseph trying out but long term would be something that we would hope to share out with the greater community and hopefully help those first-year experiences at institutions around the nation and globe.

Khadijah: Well, for me, there’s one thing that really struck me as we worked on these two projects, was that inclusive teaching is very similar to inclusive research mentoring. And so I do a lot of research mentoring. There’s a lot of teaching strategies in dealing with different types of learners in a research laboratory setting, so I’m really interested in going in the direction of thinking about how some of these strategies and things that we learned translate to a research mentorship context. And also, because I’m at an institution now that has graduate students and postdocs, really train the next generation of inclusive instructors, and particularly those that are in STEM spaces. So when I’m working with a mentor to our postdoctoral advisory committee here, and trying to get them excited about being inclusive instructors as they transition to their next phase.

Rebecca: We’re looking forward to hearing about all the things that you’re continuing to do because you’re making an important difference in our teaching and learning community. Thanks for joining us.

Khadijah: Thank you.

John: It’s great to see all of you again and talk to you again. And this time, we shouldn’t let it go quite as long as between our last discussion and this one.

Derek: Thank you so much for having us again. It’s been a pleasure.

Tracie: Yes. Thank you.

Khadijah: This is so fun. [LAUGHTER]

Rebecca: Either that or you’re enough to write books faster, one or the other. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

351. Extending Kindness

Posted in classroom community, engagement, equity, first-generation, higher education, inclusion, learning, pedagogy, teaching
July 24, 2024

Beginning faculty often receive warnings that lead to antagonistic relationships with their students. In this episode, Cate Denial joins us to discuss how a pedagogy of kindness can build productive learning environments for all students.

Cate is the Bright Distinguished Professor of American History and Director of the Bright Institute at Knox College. She is the winner of the American Historical Association’s 2018 Eugene Asher Distinguished Teaching Award and sits on the board of Commonplace: A Journal of Early American Life. She is also the author of A Pedagogy of Kindness, one of the first publications in the new Oklahoma University Press series on teaching and learning, edited by Jim Lang and Michelle Miller.

Show Notes

Transcript

John: Beginning faculty often receive warnings that lead to antagonistic relationships with their students. In this episode, we examine how a pedagogy of kindness can build productive learning environments for all students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by

John: , an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Cate Denial. Cate is the Bright Distinguished Professor of American History and Director of the Bright Institute at Knox College. She is the winner of the American Historical Association’s 2018 Eugene Asher Distinguished Teaching Award and sits on the board of Commonplace: A Journal of Early American Life. She is also the author of A Pedagogy of Kindness, one of the first publications in the new Oklahoma University Press series on teaching and learning, edited by Jim Lang and Michelle Miller. Welcome back, Cate.

Cate: Thank you.

John: Our teas today are:… Cate, are you drinking tea?

Cate: I am. I am drinking Harney and Sons Decaf Earl Grey.

John: …very nice.

Rebecca: It’s a great choice. I have a green tea today.

John: A what tea?

Rebecca: I don’t know it’s just green tea. [LAUGHTER] Sorry.

John: And I have a peppermint spearmint blend today.

Rebecca: Sounds nice and calming. So we’ve invited you here, Cate, to talk about the Pedagogy of Kindness. I think we last talked to you in October of 2021, when the book project was just getting started. And our discussion focused on a “Pedagogy of Kindness,” an essay on the topic that you posted in I think, August of 2019. This essay was such a great resource that helped so many people and I know, our community here was really interested in the subject. Can you talk a little bit about how this essay evolved into a book?

Cate: So, as I wrote the essay, there was so much I wanted to say, and I had to sort of just express it in the most concise form that I could. After I was done, I was sort of pondering the other things that I wish I’d had the space to be able to include. And then Josh Eyler, who is now at the University of Mississippi, messaged me and said, “Hey, I think this needs to be a book.” And I was like “Me too.” [LAUGHTER] So I started thinking of what I would do to make it a longer treatment. And I wrote up a proposal, sent that off, and started writing, pretty much straight away.

John: In this book, you argue for a teaching approach based on kindness. You note that many people, though, confuse “kindness” with “being nice.” How do you define “kindness?” And how is it different from “being nice.”

Cate: So, being nice is a lot of what we do in academia, nice lies, nice will do anything to just get along, it will avoid conflict, and it will just try and paper over cracks in our institutions, whereas kindness is fundamentally honest. So it means you’re going to have some really tough conversations with your colleagues and with your students. You’re not going to mislead them about how they’re doing in your class. And kindness is really about accountability, it’s about positionality, and it’s about responsibility. And those are all things that I think niceness just elides.

Rebecca: I know that I found myself saying many times after reading your essay: “Telling the truth here or letting a student know what their real situation is, would be the kind of thing to do. [LAUGHTER]

Cate: Exactly,exactly. And I often have people, as I go around the country and talk to other faculty, ask me: “Exactly how do you have these conversations, when the student comes to you and says, ‘this terrible thing has happened, and please just excuse this paper,’ or ‘don’t include this in my grade’…and you have to, especially if you are in a professional school, for example, or a medical school or something like that.” And that’s when that distinction between niceness and kindness really comes into play. Because niceness is just about getting along. And kindness is like, “Well, let’s have a real conversation about what’s going on here. Why these expectations are, what they are, whether it’s possible for us to make some kind of adjustment here, or whether this is a moment where we really have to talk about what your future is in this course, or in this program,” things like that.

John: In the introduction to the book, you note that this is not quite how you started teaching, that you started teaching, like many of us, with a somewhat different focus in terms of interactions with students. Could you tell us a little bit about your perspective when you first entered teaching?

Cate: So when I first entered teaching, I was brand new to teaching. I had got my degree in England one month before I stepped into a classroom in the United States. I did not know the United States educational system well at all. I felt like a complete fraud, and the limited training I had had at my graduate program really encouraged me to think of my students as antagonists, to assume they were going to cheat, they were going to try and get one over on me, that they wouldn’t do the reading, that they were going to try and negotiate for a grade that was not what they deserved to get, things like that. And walking into a classroom like that prepares you for battle [LAUGHTER] at every single moment, which is a terrible way to walk into a new relationship of any kind. It’s especially demoralizing and damaging, I think, when you are walking into a classroom, so I had to unlearn the business of thinking that my students were up to no good. I had to unlearn the business of thinking that my students were fundamentally not interested in their own education.

Rebecca: You mentioned your four day visit to a digital pedagogy lab, and how that impacted your teaching. Can you elaborate a little bit on that and share that story?

Cate: Yes. So I went to the Digital Pedagogy Lab Institute in 2017, which was in Fredericksburg, Virginia. It was the most incredible experience. I went because I had the feeling that I was lagging behind in bringing digital tools into my classroom. And I thought this would be where I would learn those things. And I certainly had that opportunity. But the first thing that I was asked to do in the intro track, was to tear apart everything about my teaching, in a really good way. I was in a track that was led by Chris Friend and Sean Michael Morris. And they asked us to reflect upon everything about our pedagogy. Why were we doing things in the way that we were doing them? What was the rationale that we had for the choices we were making, and the way we addressed our students and the kinds of things that we have happen in our classrooms. Everything about that institute was predicated on kindness. And it’s not that someone actually said at some point, “Why not be kind?” But that was the subtext from everything from… it was the first Institute I think I ever went to where there were pronoun pins that people could take when they registered. There was really generous swag, there was this wonderful ethos of sharing, everybody rolled up their sleeves. and really worked hard at turning everybody’s teaching materials into something even better than what they arrived with. There was really no hierarchy. It was just a wonderful community experience of trying to get better at what we do.

Rebecca: Sounds like such a great experience, and an opportunity to really critically reflect on your teaching, which we don’t always take the time to do.

Cate: I didn’t even know I needed to do it. [LAUGHTER] So by the time I went in 2017, I was more than 20 years into my teaching career. And I thought I was pretty great at teaching at that point. But what the digital pedagogy lab made me realize was that no matter what my goal was, as a teacher, all of the documents in my course, my syllabi, my assignment sheets, the way I handled introductions, all kinds of little things, really told a different story. And they told that story of antagonism that I had been socialized into as an early instructor. And I think that is something we are socialized into globally in higher ed.

John: One of the things you talk about is finding out a little bit more about your students. Because when many of us start teaching, we tend to assume that our students are similar to us, they have similar backgrounds, they’ve been prepared in ways that were similar to ours, maybe they weren’t quite as good as students as we were. But many people start off by assuming that students will be just like them. And yet you note in your introductory chapter that as many as 71% of undergraduates in the United States are non-traditional, under a teaching approach that focuses on kindness, how might faculty begin to address the diverse identities and needs of our students?

Cate: So, I think the first thing that we need to do is take stock of who our students are. And so we need to think about the demographics on our campus, that information is usually pretty easily available. As I travel around the country and visit different campuses, I always pause and look up on their websites, what is their diversity rate? And how many low-income students are there? How many people get Pell Grants? How many people are first-gen? How many people are returning after taking some time off? Or maybe have a GED? There’s all kinds of ways in which our students are not that sort of stereotypical 18 to 22 year old, straight white student who is perhaps in a sorority or fraternity or does athletics or anything that sort of gets caught up in the stereotype of who students might be. I’m really interested in what happens behind that. Even if a student walking into my classroom is, for example, a straight white student who is 18 to 22 years old, or maybe is an athlete, maybe is in a sorority. That doesn’t mean that those things tell me who they really are. They don’t tell me what is going on in their home life. Do they have kinship obligations and responsibilities? What is happening with the way that they’re funding their education, all kinds of different things. So I think pausing at the beginning of the term and really getting to know who our students are, is vital if we’re going to meet them where they’re at in our classrooms. So there’s lots of ways of doing that. There are ways of sending out surveys to our students, either before the semester begins or right as it begins. There are ways to have conversations in class to make those introductions a little bit more substantial than just, “Hi, my name is….. And my major is….. , and I’m taking this class, because….. So trying to gather that information, and also reflecting on how my identities are showing up as I walk into a classroom. I think those are really important parts of what we do.

Rebecca: What are some of the ways that adopting a pedagogy of kindness changes the ways that instructors and students interact?

Cate: I think it is all about undermining that presumption of antagonism. If we do not assume that our students are out to get us, But rather, that our students, in the vast majority of cases, are actually really invested in being there, and are perhaps a little bit scared of us, that they are a little bit uncertain, a little bit unsure, maybe they are first years, or this is their first class in a certain discipline, or they’ve been out of school for a while. There’s all kinds of ways that they might be very, very nervous coming into our classrooms. And so I think, taking the time to think about that, and address that, and make space for that, so that our students are not some sort of terrible, antagonistic, cruel set of people who really don’t want us to succeed or for themselves to succeed. That’s just not based in reality.

John: And I would think it would be a much more enjoyable environment, both for students and for the instructor, when there’s that trust between students and faculty.

Cate: Yeah, I actually have a blog post that I wrote, I think back in 2016, called the first-day jitters, that was about how nervous I would get before I would meet my students for the first time every semester. And once I changed my outlook and stopped expecting antagonism, I stopped getting nervous. I walked into the classroom and was excited about what was going to be co-created in that space. And it made an enormous difference to me and to my students. Certainly, I saw it in the ways that they interacted with one another and with me in our classroom spaces, but it changed everything for me about showing up as a teacher.

Rebecca: One of the things that you’re pointing to or hinting at is the idea that the pedagogy of kindness also applies to being kind to yourself. Can you talk a little bit about ways that instructors can be more kind to themselves?

Cate: Yeah, I think the number one way in which we can all be really kind to ourselves is to have impeccable boundaries. [LAUGHTER] I think it’s really important that we not prostrate ourselves, not give beyond what we are capable of giving, and have a really good understanding of what our capabilities are. So for example, if I am someone who is stringing together three jobs to make a livable income, if I have a family, if I have caregiving responsibilities, if I have a really long commute, all of those things are in the mix. And I cannot then be someone who 24/7 is simply available to my students. I don’t think that’s healthy, even if you don’t have any of those other responsibilities or considerations. I think that we all need time to have downtime, to have rest, to have play to, have pleasure. But I think that we need to be able to say, “I am available by email between these hours, you can expect an answer from me within a certain period of time.” Communicate those things incredibly clearly to our students. And that means that we’re also modeling for them that it is okay to take time for yourself and to have good boundaries.

John: In the second chapter of your book, you focus on the syllabus, which is often one of the first forms of communication between the instructor and students. Quite often the syllabi look more like legal documents than they do a welcoming introduction to the course. What are some of your suggestions on how to demonstrate kindness in your syllabus?

Cate: So an exercise that I was asked to do at the Digital Pedagogy Lab was to read my syllabus with a critical eye and ask myself who I was addressing when I wrote it. And what I found was that I was addressing someone who I assumed was going to screw up somehow. I assumed that they were going to disappoint me they were going to let me down, that they were going to try and get away with something… that old antagonism rearing its head. So then I had to stop and go, “That’s not who I think my students are, so why is that reflected in the language I’m using in my syllabus?” So then I stopped and I rewrote. And I kept in mind a student who is very real, I pick students to have actual students in my mind when I’m writing my syllabi, or when I’m revising them. And in addressing those students, I was able to make my language welcoming and warm. It doesn’t mean that it’s soft, it doesn’t mean that I don’t have boundaries and that I don’t have expectations. It simply means that I’m talking to them as I would if we were in person. That’s one thing that I learned to do. Another is that I learned that for many students, syllabi are impenetrable walls of text. And I’m sure all of us, in our time, if we haven’t written that syllabus, have seen that syllabus, maybe when we too were students. Those are really hard for students who have things like ADHD, for example, to be able to parse, to find the important information. They don’t look welcoming. They look like legal documents. And so I’ve changed my syllabi, for example, to have really colorful welcoming headers. I make a point of always saying “welcome to the class,” as the first thing that you see on the syllabus. I make a point of emphasizing relationship early and saying, “This is how you can get hold of me. Here are what my student hours are and what student hours mean.” And only then do I get into the stuff like, here’s what we’re going to read and here’s the assignments we’re going to do. It’s all about establishing warmth and a relationship from the get go.

Rebecca: How do you balance that approach with some institutional policy or other things that you maybe have to include in a syllabus that don’t have that same tone and tenor?

Cate: Yeah, that’s a really big consideration for a lot of people. There’s a lot of boilerplate language that has to go into many syllabi. So there, I borrow from Remi Kalir, who has students annotate the syllabus, and I’ve been doing this for several years. Now, my students get the syllabus at the end of the first class session, and they take it away for homework and they annotate it. I give them a really simple definition of what annotation is. And then in our second class period, we come back and they get to tell us what their questions were, what their comments were, what seemed unusual to them, what really stuck out and we have a conversation about all of that. And if you have a bigger class than I do, this is the sort of thing that you can have small groups do or you can have parents do. And then you can have people report out some of the things that they thought were most pressing. What that does is it returns to students some agency, and they can comment on the language shift, the tonal shift between the things you wrote and the things that you have to include, and you can have open conversations about “So why do you think we have to include this stuff? And what are ways that we could say this differently?” or “What is the way that you understand this best,?” Having that conversation is really important in allowing students to talk back to that document. And to make it clear that it’s a human creation, that it’s not just spit out by a machine somewhere, that there are people behind it, and that we are collectively always polishing and revising our syllabi in light of these relationships that we have on our campuses.

Rebecca: In smaller classes, I’ve done the annotation in a single document. And it’s been really interesting to have students comment on each other’s comments in a digital format, too. And it really brings some interesting things to light.

Cate: That’s a great way of doing it. I tend to have my students do it individually, just because they don’t know each other very well yet. And I found that my students tend to respond better to individual documents, and then a group discussion, but it works differently with every single instructor and every single set of students. So there are multiple ways that people could do this kind of annotation.

John: You did mention that this syllabus is not something spit out by a computer, but last week, we had a workshop where one of the things I suggested was that faculty may want to submit their syllabus to one of the AI tools out there and have it checked for inclusive language or to rewrite it in a more inclusive manner. And I think a lot of people found that, in that case, the computer did a better job than they did [LAUGHTER] in terms of creating that inclusive syllabus. But, at the other extreme, one of the things I’ve often done and I think you mentioned this in your book is in smaller classes, co-create the syllabus with students at the very start of the semester.

Cate: Yeah, I teach a course on history pedagogy. And there I give my students blank sheets of paper and say “Here’s your syllabus,” [LAUGHTER] and then they get to start brainstorming the things that they should put in syllabuses. They’re experts at syllabi, they read them all the time, they see many more than most of us do. And so they brainstorm what they think it should include. And then I put them in small groups. And together, they start to talk about, “Oh, that was a great idea you came up with, I didn’t think of that. Let’s put that in our syllabus.” Then we write their syllabi up on huge post-it notes that we put on the wall. And then everybody gets little post-it notes. And they get to walk around, and I ask them to do what I was asked to do at the Digital Pedagogy Lab, which is to say, “Who is the student that these authors are imagining as they write the syllabus?” Write it on your little post-it note, put your reactions on the wall. And what they very quickly learn is that they too have absorbed the idea that students are antagonists, students can’t be trusted, students are going to mess up somehow. And so then, as a class, we sit down together, and we say, “Okay, we don’t want to do that. We don’t want to make those assumptions. So how are we going to write a document that accurately sets out expectations, that also supports and that communicates trust?”

Rebecca: In chapter three, you focus a bit on assessment. And many faculty note that when students do less well than maybe they thought they were going to or anticipated on a graded activity, they might become disengaged or lose interest or disappear. Can you talk a little bit about ways that we can make our assessments more equitable, and to make expectations more transparent to our students?

Cate: There are so many ways that we can do that. Including students in the process at all points, I think, is a way that we can never really go wrong. But I think that one of the most important ways we can approach assessment is to think about, are we being inclusive? are we giving our students the tools they need to be able to show us what they know?…because so often, what we’re actually grading or assessing or giving feedback on is, how they write about what they know, or how they make an equation about what they know, we’re not actually just finding out what they know, what they have learned. So giving them multiple ways to express what they know, is a great way of leveling the playing field of allowing students who maybe don’t excel at writing to still be able to show “Hey, I have been thinking, I’ve been turning this over and over, I have some new appreciation for whatever it is art, or music, or fishbones, or an equation, or set of chemical compounds.” So in my classes, it’s traditional for history to be assessed through writing papers. And I certainly do have my students do written work, I don’t get rid of it altogether. But I also make an assignment where my students get to create projects. And they can do anything they want to, except write a paper. So they do things like make quilts, and food, and board games, and they knit and make pottery. I’ve had people make raps and sea shanties. I’ve had people dress up as pirates and come talk to us about a day in their life. All of these things are based on historical research. All of these things have a reflective component at the end. But for a student who really struggles with writing, for example, this gives them a way of being able to show me they have been learning that term, that they have done research, that they’ve really engaged with the class material in a way that doesn’t limit them in any way.

John: In chapter 4, you address kindness in the classroom. Could you share a couple of ways in which faculty can display kindness in their classroom interactions?

Cate: I think one of the most basic and important ways that we show kindness in the classroom is we learn who our students are. And that means names. And that is increasingly challenging. It depends on how many, if you’ve got 300 students in your classroom, how do you learn their names? I tend to have smaller classes. So it is really doable for me to learn their names, but I still rely on tools that helped me do that. So I bring in name tags, and for the first several sessions, my students will write their name on their name tag, and if they feel comfortable, they’ll put their pronouns on there too. I know that Vigi Sathy and Kelly Hogan talk about having name tents in their big lecture classes. And those name tents are also color coded. So then that helps them put people into small groups. People can turn those name tents on their end when they want to indicate they want to participate in conversation. They’re wonderful, wonderful tools. So there’s lots of resources out there that sort of suggests there are many, many ways to make sure that we’re getting to know our students, but I think that’s a really basic thing we do that says, “I know who you are. I know when you’re here, I really appreciate you being here and being engaged in a classroom activity.” I also think that it’s important to teach our students how to be in that space together. I think this is really important coming out of the height of the pandemic. We’re not done yet. but we are coming out of those most acute years. And certainly I have seen among my students on campus, that there was a period of time where it was odd for them to interact with other people because they hadn’t had practice. They had been in lockdown. They had been at home. They had been doing school online. And so we really had to sort of think about how do we interact? How do we talk to one another? How do we show that we respect one another? When something goes wrong, how do we show that we are accountable and responsible? How do we apologize, and mean, it. These are all things that we need to teach, as instructors to make sure that our classrooms are kind spaces in which learning can really happen.

John: Is there anything else you’d like to share with our listeners that we haven’t yet discussed?

Cate: One of the things that comes up in the conclusion to my book is that it’s very easy to feel tossed around in higher education, like we’re in tiny little boats on a very large and angry ocean.[LAUGHTER] And certainly, there are structural problems that need our attention in higher ed. But we’re not powerless. And we get to make changes in the spheres that we control. And we get to approach our students, as fellow human beings who matter, we get to treat ourselves as people who matter. And kindness is the fabric of that, it’s what weaves everything together. It is the material from which we build inclusive spaces, and spaces where people’s words and ideas are valued. And without kindness, we end up with places that are disrespectful, we end up with places where people do not feel like someone’s listening to them. So I have the metaphor, that little boat being tossed around in my conclusion, but ultimately say, if we can just be kind to one another, then we can sort of temper that storm.

Rebecca: It’s a great image to end on, for sure in your book. And it’s such a great metaphor for really thinking through what sometimes it really feels like to be in higher ed, not only with our students, but with other colleagues. There’s lots of ways to bring this kindness work into other spaces within higher ed outside of the classroom as well.

Cate: Yeah, I agree. And one of the projects I’ve been involved with over the last couple of years is something called Care in the Academy, which is about showing kindness and care to each other. So focused on faculty, focused on staff, focused on those relationships, and how we show one another that we matter to each other.

Rebecca: Can you share some examples about what that might look like in spaces outside of the classroom?

Cate: Yeah, so some really concrete things that we recommended after 18 months of really thinking critically about this issue were things like extended bereavement leave. Places tend to give us three, maybe five days and they’re immediate. And the idea is that whatever mourning or immediate grieving you need to do will happen in like a week. And that’s just not practical or realistic, or how humans work. And in many cultures, it’s also not reflective of the way that people process death. So we talked about having maybe another three to five days that you can use for a year after someone passes and you can mark an anniversary, or maybe there is a ceremony that you need to attend. Or maybe there is a ritual that you need to perform, and you take time away to be able to do it. It’s a very small thing, but it makes such a difference in sort of saying we respect what a difficult time this is for you. Trying to improve reimbursement culture, there are so many people who are stringing things along on their credit cards, basically making loans to their institutions, and trying to speed up the process by which they are repaid as a basic form of respect, I think is really, really important to the way that we all function around one another. To not assume that people are sitting on vast mountains of cash that they can just make available to do things like go to conferences or to travel to another town or whatever the thing is. I think that’s also really important.

John: That’s especially true for junior faculty, for people who are contingent faculty and who are struggling a bit because this occupation does not pay all that well. It gets a lot easier once your kids are out of the house and you’re past those stages, but it’s a struggle for a very large share of faculty.

Cate: Especially given student loans being what they are. This lasts a very long time.

Rebecca: It’s something that sometimes faculty have in common with students and we don’t always recognize or acknowledge that.

Cate: Yeah, I agree. And I think our students don’t often look at us and think, “Oh, wow, they’re still paying off their student loans.” [LAUGHTER] They tend to see us as something other than precarious. Whether we are in a contingent position or not, very often our financial lives are in some kind of precarious situation, just because the reality of funding higher education careers is what it is right now.

Rebecca: So we always wrap up by asking: “What’s next?”

Cate: What’s next is that I am going to go read a book about biomimicry. And biomimicry is looking to the natural world for patterns that all organisms share, and taking wisdom from those patterns. So it helps us in terms of ecology. But I just heard a podcast that was applying this to the way that we organize our institutions, and thinking about the way that organisms heal after fire or clear cutting or a flood, for example, the different jobs that different organisms have in the recovery process, and thinking about how we make really resilient institutions and organizations by looking to the natural world. So I know very little about it. I’ve listened to that one podcast. So I am going to go read a book that is all about that topic.

Rebecca: That sounds really fun.

Cate: Yeah, I think so.

John: And I think we’ve all needed a bit of resilience over the last few years with the challenges we’ve all faced. So it sounds like an interesting way of addressing that issue.

Cate: Yeah, I’m hoping I will get some fresh ideas.

Rebecca: Reading stuff that’s unfamiliar is always a great way to reinvigorate our own work.

Cate: Yeah, and I have not really considered biology since I was 16 years old and took my GCSEs. So this will be a real treat and a delight to exercise part of my brain that hasn’t been exercised in a while.

John: Well, thank you for joining us. It’s great talking to you. And I really enjoyed reading through your book. It grew from this very nice, useful essay that many of us were reading through during the pandemic, to a much more elaborate discussion of how to be an effective teacher in all aspects of teaching.

Rebecca: Such a needed robust resource.

Cate: Thank you so much. It was such a pleasure to be with you today and have this conversation. And I really appreciate all your kind words about the book.

Rebecca: We look forward to talking to you in the future, hopefully the near future.

Cate: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

350. Nudging, not Judging

Posted in classroom community, engagement, evidence-based practices, higher education, learning, pedagogy, professional development, teaching
July 17, 2024

During the pandemic, faculty participation in professional development activities expanded dramatically. Faculty involvement, though, has been gradually returning to pre-pandemic levels. In this episode, Sarah Rose Cavanagh joins us to discuss strategies for bringing more faculty into discussions of teaching and learning.

Sarah is a psychologist and the author of four books related to teaching and learning. She is the senior associate director for teaching and learning and associate professor of practice at Simmons University and also is a regular contributor to the Chronicle and many other publications. Sarah often serves as a keynote speaker and we were very fortunate to have Sarah join us for a keynote address at our Academic Affairs Retreat in Oswego last August.

Show Notes

Transcript

John: During the pandemic, faculty participation in professional development activities expanded dramatically. Faculty involvement, though, has been gradually returning to pre-pandemic levels. In this episode, we explore strategies for bringing more faculty into discussions of teaching and learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Sarah Rose Cavanagh. Sarah is a psychologist and the author of four books related to teaching and learning. She is the senior associate director for teaching and learning and associate professor of practice at Simmons University and also is a regular contributor to the Chronicle and many other publications. Sarah often serves as a keynote speaker and we were very fortunate to have Sarah join us for a keynote address at our Academic Affairs Retreat. Welcome back, Sarah.

Sarah: Thanks for having me.

Rebecca: Today’s teas are:… Sarah, are you drinking tea,

Sarah: I thought about trying to blow your mind and saying I’m drinking some Bergamot green thing. But no, I just finished my last coffee for the day. And now I’m on to water. [LAUGHTER]

Rebecca: You’re so consistent.

John: My tea today is a black raspberry green tea.

Rebecca: And I have an Awake tea, which clearly I need. [LAUGHTER]

John: We’ve invited you here today to discuss your May 7, 2024 article in The Chronicle, entitled, “After a Tough Year for Classroom Innovation, It’s Time for a Reset.” And I think this is something that’s going to resonate with a lot of educational developers and many faculty. You begin the article by noting that there was a sudden increase in attention to student-centered teaching in response to the pandemic, but has that shift been as persistent as many of us might have hoped?

Sarah: Oh, I think yes and no. I think that a lot of people started focusing more on their teaching. In those pandemic years, we all saw in our teaching and learning centers, higher attendance at events and a lot of engagement, both in these larger events, and then faculty coming to us for consultations. And one hopes, as educators, that learning doesn’t just go poof, that the effect of all of that attendance and thinking and consulting is still impacting people in their classrooms. But I think that there definitely has been a trend on my own campus, and most campuses that I talk to, about declines in attendance at those same sorts of workshops and talks and consultations. And there also, I think, has been a bit of a pendulum shift in some of the topics such as flexible deadlines, where I think we saw a lot of people move in one direction, and then are starting to see a little bit of a shift back. And so hopefully, we still have overall nudged the average over towards student-centered teaching and that hasn’t changed. That’s my hope. But I definitely see a little less intensity and interest lately.

Rebecca: We definitely have seen similar declines in professional development on our campus as well. And sentiments like, “I’m just really tired,” or “I’m done” [LAUGHTER] which you highlighted in your article, and one of our most popular workshops of recent years has been related to burnout. Can you talk a little bit about ways that we can re-engage our faculty or why burnout is such a popular topic? [LAUGHTER]

Sarah: Right? Well, I think that if we’re thinking about kind of a shift away from student- centered teaching, or even a backlash, as we’ll talk about in a little bit, that burnout is one of the many causes of that. I think that faculty are just tired, as you said, and that sentiment of “I’m just done,” like, I’ve heard those words multiple times from multiple different people. And a lot of student-centered teaching, and certainly attending professional development workshops takes time, and there’s just no time. And so there’s the burnout that’s related to the emotions, and the trauma of the pandemic, and just workload being completely ridiculous. But then there’s just also a paucity of time. And I think that part of that is this whole Zoom… we’re here, not on Zoom, but we’re here virtually, and those virtual options open up so many possibilities. But I feel like we took all of our pandemic solutions with the Zoom sessions, and all of that, and we brought them into the future. But then we brought back all the in-person stuff. [LAUGHTER] And so there’s zero time for anyone. And so I think that the burnout is so strong, and that’s why there’s such attention to that. And that is one of the most popular topics right now.

Rebecca: Feeling over scheduled definitely limits the ability to feel creative, or to dream or to think about new things. So if people are used to kind of being in that state, or historically have been in that state, and they’re just feeling like they can’t get to that state. I can imagine that’s why people say they’re done is because it’s hard to come up with new ideas or to put the energy into that kind of thinking.

Sarah: Absolutely. And I think also students are equally, at least equally, burnt out. [LAUGHTER] And so I think another thing that’s happening is faculty put all this time into student-centered teaching, but then it’s not working. They’re not seeing more engagement or better learning. And so you’re investing all this time, but then the students are so burnt out that it doesn’t look like it’s being effective, and that’s frustrating, and I think that that’s also part of the feeling done, like, if you’re just throwing everything you have at bettering your teaching, but then still feeling frustrated by the outcome, then that is demoralizing.

Rebecca: It’s interesting that the way you put it instead of like getting re energized… [LAUGHTER] like, depleted.

John: And we never observe the counterfactual, we don’t see what would have happened, had we not been focusing more on student-centered teaching after some really tough times. So, why do you think faculty are pushing back a bit more against student-centered teaching approaches, because we had a lot of buy-in on that during the early stages of the pandemic, and for a couple of years after that, but we’re seeing a little bit more pushback.

Sarah: Yeah, well, I have a long answer for this one, because I am a psychologist and so I think everything is highly multi-causal. And one major factor is the level of burnout, both on the faculty side and the student side. And so we don’t have to go back to that, but I just want to highlight that I think that that’s part of it, is this burnout, and I love Rebecca’s depiction of like how impossible it is to feel creative and do new things too when you’re feeling exhausted. And so I think that’s a big part of it. On my endless commute yesterday, [LAUGHTER] there was a Red Sox game (I work in the Fenway), I listened to your interview with Josh Eyler, which was excellent, of course, and I think it probably predated the article, but you were talking about just this topic of sort of a backlash, specifically against alternative grading schemes as a student-centered practice. And Josh said, that I thought was very insightful, as everything Josh says is, that he had this analogy of a rubber band, and that there’s just this effect. And I think of it as a pendulum as well, which I already referenced, that there’s just this natural effect when you move in one direction to kind of snap back a little bit. And he saw that as part of the cause of this. And I think that that is true. I think it’s sad, but we love pendulums as human beings. I’m always like let’s just settle into moderation, [LAUGHTER] but we love swinging from one extreme to the other. So I think part of it might just be a regression to the mean sort of effect. But I think there’s a lot of other factors. I think that one is, if you drill down into the recommendations of so many different theoretical approaches to good teaching, you come up with similar recommendations, whether you’re focusing on inclusive teaching or accessible, teaching, engaging teaching, all these things, they all drill down to these recommendations of being warm and welcoming, having a promising tone, seeing students as people, giving them autonomy and giving them choice, all these things. And I think that some faculty feel, after attending several years of the pandemic and all these different intros and frames, but then the same recommendations feel like they got it. [LAUGHTER] And that might be part of it. The final two causes of the backlash are a little more delicate. I think one is the student reactions. And we talked about this a little bit, that students themselves are so burnt out and so faculty aren’t seeing great results when they experiment with student-centered teaching. But I think that there’s also a rise in student incivilities. Beth McMurtrie had a great report in The Chronicle, about a year back on the rise of student incivilities. Another term that people use is reactance, when they’re pushing back against the instructor. I can tell you a brief anecdote. I was on a different campus talking to a faculty member. And she had been using alternative grading practices, it was kind of a standards-based approach where students had multiple attempts at achieving a standard, but they’re passing an assignment and would either be competent or not yet competent. And she was teaching in a male -dominated field. And this student of hers who was male, he got a “not yet competent,” and threw the paper back in her face and said, “You don’t know what you’re doing.” And so I think that that too, when you’re putting all this time and effort into student-centered practices, and then having reactance and incivilities in reaction, that’s also demoralizing. And then finally, just to get myself in trouble, [LAUGHTER] I think that we in the teaching and learning fields, bears some responsibility. I think that especially toward the end of the heyday of teaching and learning years on Twitter, things got a little bit ugly. There was some shaming of faculty pile-ons, and some of the pile-ons were of people engaging in student shaming practices and other things that are ugly on their own. But some of it was also just venting stresses of teaching, and then getting piled on. And I saw this really clearly because I was active on Twitter in the teaching and learning fields. And then because I was more of a research focused faculty for most of my career and all my social contacts in person are more like that, I’ve had all my faculty friends on Facebook, and they would post just some things. They all very care deeply about their teaching and their students, but they would post things and I was like, “Oh my God, if you had posted that on Twitter, [LAUGHTER] you would be dead in the water.” So I think that we, as we talk to each other, and we convince each other, and we get more and more excited about change, and about some innovative practices, that sometimes we get a little out of touch with the majority of faculty who are teaching on the ground, and their constraints and practices.

Rebecca: We spent a lot of time talking about students as humans, but teachers are also human, I think, [LAUGHTER] and I think we need to be, and you’ve mentioned this before, too, is just making sure that we’re treating everybody as humans, and that everybody has constraints, and everybody has different pressures. And in your article, you suggest that the time seems ripe for a complementary revolution, one that recognizes and respects the autonomy and bandwidth of teachers. Can you talk about what this counter revolution might look like?

Sarah: Sure. And this is easy for me to say, because in my position I have no control over budgets, [LAUGHTER] or faculty workloads. But I think that primarily what we need is we need a revolution in how we treat faculty. Faculty have been through the pandemic, I think the coming demographic cliff and all of these things that administrators push for faculty to be more and more responsive to students, more and more student-centered, to take on more and more things. At my last institution, they had us writing letters, personal letters to individual possible students to try to encourage them to put down deposits. All of this emotional labor time, regular labor. And so I think faculty in response need more time and money. And I think we’ve just piled on and increased the workload and increased the workload and expanded the scope of the work exponentially through these pandemic and post-pandemic years. And I think that faculty need course releases, I think they need stipends for some of this work. And I think that we need to put a lot more resources toward our faculty. I think it also means a lot more full-time lines with security as well. And those are going in the opposite direction. I think that the other thing that we need is hands off our classrooms. [LAUGHTER] I think that we need to respect faculty autonomy and faculty choice. And so many of us get into this work, because we want that autonomy, we want that choice. We’re willing to work harder for a lot less money than if we went elsewhere. But it’s because we want to be self determined, and do the things that we want to do. And I think that we need to respect that.

John: One of the things you’re suggesting in the article is that there’s no single approach to student-centered teaching that works for all faculty or for all students. Could you give some examples of approaches that work for some faculty in their context that might not work as well for other faculty, given their context?

Sarah: Absolutely. There’s so many… class size is one, you know, are you teaching 25 students? Are you teaching 750 students? …very different strategies are going to be effective there. Courseload… are you teaching two courses a semester, or you’re teaching five courses a semester, that is going to affect the choices that you can make. Precarity of position… both pre-tenure and post-tenure, but also: Are you full time? Are you contingent? Are you on a semester to semester? Contract… identity, these characteristics of the instructor. We had Chavella Pittman come out and talk to our faculty at Simmons University last semester. She was fabulous. She talked a lot about student incivilities, and how many more faculty of color and other minoritized faculty receive of those student incivilities and what majority faculty can do differently in terms of their policies to help support their minoritized faculty. Your teaching philosophy, I think, is important. And again, autonomy is important. And a little bit your teaching talents. For instance, I’m not funny. [LAUGHTER] Some people are really funny. And that is like being humorous in front of the classroom is a really engaging strategy, but it’s not open to all of us. And some people are gorgeous lectures and actually, lecture might work better in their class than someone who’s not. And there’s just so many contexts, there’s so many constraints that are going to influence this vast buffet of teaching choices.

Rebecca: Sometimes when we’re thinking about offering students choice, that also means there’s an impact on faculty too. And so you’re kind of underscoring a faculty choice and some of the individual faculty needs where they might be balancing other things may have certain kinds of time constraints, physical limitations, there’s all kinds of things that could be at stake. How do we help folks understand the need to balance student needs with also faculty needs, and to make those kinds of decisions?

Sarah: It’s so tricky. I often think of all of teaching as kind of a dance, and there’s an art to it. And there’s a little bit of an intuition to it. I’m mad at myself, because I meant to look this up before this talk, but I will send you a link to something later on. Brian Dewsbury and some other folks put together an inclusive teaching massive online course through HHMI. And I lead faculty through a learning community using it. And they were discussing, and I’ll send you the link and the person’s name after. But it was an approach to teaching that, instead of being learner centered, it’s learning centered. And this balances out this competing tension between faculty needs and student needs, but also between, and I think we’re going to talk a little bit about this in a bit, students to student needs, you know, variations in the needs of students. And instead of centering the learner, centering the learning, and thinking about what is going to be most effective for the goal of this course, which is learning, and what can I do in terms of my teaching choices, and the structure of the course and the types of assessments and things like that, that centers the learning, which is going to center the learner as well, because it’s their learning that will happen. But I liked that subtle shift in how we think about this. And I think that might be one approach to approaching this dance with balance.

John: And that brings me back to when I first started working at the teaching center. The year before that an advisory committee put together a draft statement of the purpose of the teaching center, and they went with learning centered. We’ve still tried to maintain a focus on learning, because that’s kind of what we’re here for. And that’s why students are here. So I think that focus is really good.

Sarah: Oh good. [LAUGHTER]

John: But going back to that issue about different techniques working for different faculty, isn’t the same also true for students? That the flexibility that many faculty provided to a greater extent over the years immediately following the pandemic might work really well for many students, but might that provide a loss of structure which is really important for many of our students?

Sarah: Absolutely. A question I often get when I give talks about emotions in the classroom is about my recommendations for a lot of warm-up activities and small-group discussions, and getting students interacting with each other to maximize the sense of belongingness and community, which we know is important for learning and especially for marginalized students. But I often get the question of what about socially anxious students? What about introverted students? What about neurodivergent students? A lot of these students might really struggle having so many social engagements during a class period. And I don’t know that my answer is very helpful, [LAUGHTER] because I always try in my own teaching to have varied methods. So it’s not always social, that there are some classes that are much more independent, there are options to write in small books and things like that. And in the piece, I referenced Sarah Silverman’s work on access friction, and she didn’t coin the term she gives credit to Areley McNeney, and others, but about this friction that arises when one student’s accessibility needs are different, or even opposing in tension with another student’s. I think variation is good, you know, varying things around and flexibility and talking to your students, using who’s in class surveys to find out about student needs, always respecting students’ accommodations, of course, but having that conversation with students and drawing them in. And one thing that I really tried to do in my own teaching, is to be transparent about why I’m doing a certain thing. So I often start my classes with an explanation for why there’s going to be so many community building sorts of activities and how it is the research suggests that that’s important for their learning. So we think that kind of transparency, varying strategies, not doing the same thing all of the time, and then just acknowledging that certain teaching strategies are going to be more effective for some students versus others. And so let’s not just blast one teaching strategy the entire semester.

Rebecca: I think one thing that often happens is the use of the term universal design, not necessarily universal design for learning but this idea of universal design implies that one size can fit all, but usually universal design, if we’re really following those principles is that there’s variability and is that the space or the technique or the situation can adapt to the person and vice versa, rather than necessarily having one solution that works for everyone. I mean, it’s part of it, but there’s usually this flexibility piece that’s key, but flexible doesn’t have to mean so much extra work for a faculty either. There’s different ways to have flexibility.

Sarah: Yeah, multiple points of entry for students always is great.

John: When I first started teaching, I was told by the department chair at the time, who’s long since retired, don’t waste too much time on teaching, spend your time on research and other things that are essentially the currency of higher ed. We’ve moved quite a ways from the time where many faculty are getting that sort of advice, and those of us in educational development often become really passionate about encouraging student-centered teaching approaches, and we generally provide some criticisms of high-stakes exams, the sole use of lecture, multiple choice tests, and so forth. Those were pretty much considered to be standard practice not too long ago, and are still very common in many disciplines at least. You mentioned Twitter, many people would consider some of those techniques to be sort of like malpractice, where they’re actively doing harm to students and learning. Have we perhaps gone a little bit too far in pushing for changes in instructional practice, in the direction of a one-size-fits-all approach that perhaps we should back away from somewhat to bring more people into the conversation?

Sarah: Sure. So multiple layers there. I think one thing is, this is one of the places where I see the conversation on social media among teaching and learning folks as being a little disconnected from reality, if I can say that, [LAUGHTER] because I think that there’s so much of this still going on. I think that we talk to each other and then we talk to these wonderful faculty who come out to our professional development events. And that’s just like the tip of an iceberg. And we don’t tend to talk to people who don’t come to our events, [LAUGHTER] or don’t engage with us on social media. Where I see in this Facebook versus Twitter differential is I see my friends who are in much more research-focused careers, and all of their demands and rewards are all based on their research productivity. So you say we’ve moved away from that, but I think we’ve moved away from that some, and we have higher teaching demands even on research faculty, but I don’t think that the focus on research for a lot of those folks has changed very much. And when you’re burnt out, and all of the values and reward systems are rewarding you for research and not teaching, and you have TAs doing a lot of the on the ground face-to-face stuff with the students. I think there’s a lot of these so-called traditional practices still going on. And so I think it’s happening, and I think using the word harm might be a whole other big conversation. I would like us to step away from that word when talking about individual faculty and the teaching choices that they’re making in their classes. I think that there’s a host of problems with that word. And I think that part of the backlash, too, is this feeling of feeling judged by teaching and learning center, folks, and I think that that is not going to get us toward the sort of change that we want if there’s kind of these scores of faculty who are not part of the conversation yet. They’re not going to step into the conversation, if they’re feeling like we feel that their teaching practices, that they feel like they’re doing the best they can, are poor or actively doing harm to their students.

Rebecca: And the reality is they’re probably not actually harming students. Students are still learning and they’re having some other experiences and they’re facing challenge. That’s not, as you noted in your article, that’s not harm. Can you talk a little bit about fostering challenge for students?

Sarah: I don’t like to equate high-stakes exams and all these things… this is challenge and then student-centered teaching is low challenge, because I don’t think… and I’m not saying that you’re doing that… but I just want to be clear on my position for that. I think having really high expectations of students is challenge. I think asking students to take emotional risks and be a little vulnerable is challenge. And I think that all of that is done most effectively in a classroom where community has been built where students feel that they belong. I think reducing stakes is a really powerful tool to developing that feeling of safety in terms of willingness to take risks. So I am pro low-stakes. [LAUGHTER] And I’m not arguing that we should all be going back to sage on the stage and three exams, certainly not. And I have worked really hard over the last 10 years to try to nudge us away from those practices. But, I do think that they are not necessarily causing harm. And I think most of us, in our intellectual journeys, thrived under those conditions. I do think that people who have thrived under those conditions, usually people who’ve had a lot of privilege and a lot of resources. And so that’s another reason why we need to move away from these practices. But I think if we want to pull people with us and reduce high-stakes exams and reduce some of these practices that may not have been as effective at bringing all students to the table, then we need to do so with an open hand and also have ways that faculty can make incremental change. I reference in the article, Michael Palmer had a POD presentation at one of the teaching and learning conferences, in which he presented 15.9 million choices [LAUGHTER] that a faculty member makes in the construction of an individual class and argued that if we really want to make lasting change, we should be nudging little changes in all of those choices towards student centeredness, toward more inclusivity, rather than asking faculty to kind of throw everything out and start fresh and do something entirely new, and I found that compelling. And so if we could do that nudging, in the absence of judging, and by suggesting that there might be better effectiveness and equity, and doing that nudging, rather than saying you’re actively harming students with your current practices, I think would be a lot more compelling for a lot of those faculty who aren’t necessarily coming to our workshops.

John: So, nudging not judging, that’s a nice way of summarizing that, I think.

Rebecca: Yeah.

Sarah: That is a much more catchy way of saying it. [LAUGHTER]

John: I was thinking it would be a good title…

Rebecca: Yeah.

John: …or the podcast.

Rebecca: I was just thinking about, you never know what small little change you might make, that actually makes a humongous difference for a learner.

John: To use the technical term

Rebecca: I only use [LAUGHTER] technical terms. [LAUGHTER] So I think what you’re highlighting is really important, because these small little changes can make a big difference for an individual learner, or a set of individual learners. And you just don’t know what little tweak you’ve made, and what kind of impact it might have made. I’ve certainly had the opportunity to have a conversation with a student where they tell me about something that made a really big impact for them. And to me, it was like something that was really small, like, “Oh, alright, well, I should keep that.” [LAUGHTER] But it’s helpful. And they all add up over time.

John: What can we do to bring some of these other faculty who haven’t been as active in discussions of teaching into conversations, and to make it easier for us to do some of that nudging? We can’t really nudge people if they’re not talking to us or not coming into teaching centers?

Sarah: Well, I think more dedication of resources from the administration would be good. I think that bringing people to these events who haven’t traditionally come might happen if there’s more time and money, a course release or something, aa stipend for something else. I saw a presentation by Betsy Barre, a wonderful presentation where they had participation in a one-day teaching celebration event that was at something like 67% of the faculty attended. But it was very well resourced. They had a bunch of really fantastic speakers, a lot of great food, it was an entire day, and the provost canceled classes and like kind of shut down campus for the entire day and said, “This is important, come to this.” And they got something like 67% of their faculty at Wake Forest to attend that. So I think more resources is an answer. I think also having more varied offerings. So I tell the anecdote in the article when I was at Assumption University, one of our faculty came to us and he was from a department that never attended anything [LAUGHTER] that we offered and he said that, and he said, “Actually, none of us ever come because we think that you’re doing harm to teaching with all of these approaches that he felt were not effective, and that were kind of degrading education.” [LAUGHTER] And we had a conversation, and so what the result of that conversation was we had one of our faculty learning communities that we do over dinner with a book, and he gave a presentation called “the art of the lecture,” and it was a defense of lecturing, primarily lecturing as a mode of pedagogy. And then we also had a few other faculty do little flash talks about how they did kind of dynamic innovative lecturing. And then we had a conversation and a debate. So we weren’t advocating for this, but we were showcasing him, and then faculty were engaging in debate and dialogue around when is lecture appropriate? Like what are the ways to make it more student centered. And I would like to say that his whole department came out to that, and they didn’t, [LAUGHTER] but he was there. And I think demonstrating that kind of respect for all the variations of teaching, I think is good. Another that we did at my current institution is we had a panel of faculty and we intentionally chose faculty with varied deadline policies, so on the spectrum of more strict, more flexible, and kind of in the middle, and had them present their rationale and their procedures and then did q&a. And that was really well received and better attended than some of our other events. So I think mixing things up and creating a space where it is clear that we are respecting faculty autonomy, and that we see variation as good. I think it can be powerful.

Rebecca: I’m hearing some themes of celebrating teaching and learning [LAUGHTER] and respecting and listening.

Sarah: Yes.

Rebecca: …listening and respecting a wide range of teaching in all of the examples that you just shared, which I think are two things to remember and keep in mind, because I think you also said not in this last answer, but a couple answers back something about [LAUGHTER] I think you put the words faculty and equity in the same sentence rather than we usually think about students and equity. But I was hoping maybe you might elaborate a little bit more on this idea of some equity around faculty as well when we’re thinking about teaching in particular.

Sarah: I think that probably what you’re referencing in my previous answers was when I was talking about Chavella Pittman’s visit. And I think one thing that we don’t always acknowledge is the impact that our teaching choices can have on other faculty in our department and at our institutions’ teaching choices. And so she was talking about how many more incivilities and classroom disruptions faculty of color and other minoritized faculty get compared to white cis het faculty. And that when we make choices in our own classrooms, that might have an impact on other faculty. And so specifically, when white majority faculty have a lot of policies in their classes, that they have a lot of flexibility, or they don’t have a classroom disruption policy, or the classic example is “Call me Sarah, and don’t call me Dr. Cavanagh,” that sort of thing. That actually makes it more difficult for faculty who, in order to have a good classroom climate, needs to demonstrate their authority. I know we don’t like to use that word when we talk about teaching, but all our friends need to be called by “doctor” and need to have a classroom disruption policy, because otherwise there’ll be a lot of classroom disruptions. If there’s also this perceived differential in those two classes, then those faculty are going to get accusations from students that :”Oh, you’re so strict, you’re so this or that compared to these other faculty who are just like, relaxed, why can’t you be more like them?” And so I think that we do need to think… I’m saying “respect faculty autonomy,” but then also, “let’s talk to each other about our classroom policies in order to make sure that we’re not engaging in these behaviors that are going to impact other faculty.” But again, it’s a dance, it’s a dance and a balance, a little bit of this, and a little bit of that.

John: And I’ll mention that Chavella Pittman and Tom Tobin had a discussion of this in an article in The Chronicle, I believe, and then they joined us on a podcast for that too. And we’ll share links to both of those in the show notes.

Sarah: This has been a great conversation, but I want to be clear that I am so dedicated to student-centered teaching. And that is why I feel really passionate about this backlash because Rebecca, you mentioned listening, and I’ve just been listening and listening. And I’ve been hearing it all over the place. And again, I worry with these pendulums that we could just snap back to a place that we really don’t want to be. But I think sometimes when I have these conversations, I come out as “Oh, pro high-stakes exam and let’s have three exams, multiple choice, strict attendance, policies, all these things.” And that’s not the case, I just think that we need to do the nudging, not judging so that we can continue pursuing change, because I think we had this accelerant with the pandemic, and it was great, with so many people in zoom rooms and in seats, and really eager to like innovate and all of these things, and that has kind of faded, and we now have these other tensions that are pushing in the opposite direction. And so let’s have some new strategies so that we can continue pursuing educational reform

John: And the strategies of having different people with different perspectives discussing it, is probably a really good way of getting people to state their opinions, but also to hear some things that might nudge them in another direction. So someone who only lectures hearing a little bit about the possibility of doing an interactive lecture would be a relatively small adjustment that they wouldn’t have heard if they hadn’t attended an event of that sort.

Sarah: Absolutely.

John: I think that sort of approach is a really good one that we should try doing more of here.

Rebecca: I think also having varied classroom experiences is healthy for students, it’s a good experience for students and helping students to see the value and having just different opportunities, different ways of flexibility being demonstrated to them is helpful to point out to students rather than being really careful when you’re having a conversation with a student about the classes that they’re in, for example, and highlighting maybe some of the skill sets they might be getting from a particular experience rather than kind of bashing a faculty member or particular class and helping them understand they might want a variety of different kinds of classes to help their own workload and interest and that kind of thing. So we always wrap up by asking: what’s next?

Sarah: Well, I have an NSF-funded network in intro bio called TUnE-Bio, and we’ve been holding a lot of qualitative interviews of intro bio faculty and and digging into some of this, they’re thinking about change and the constraints and barriers and facilitators of change and I’m getting excited about kind of expanding that network into the possible future.

Rebecca: That sounds like some interesting work.

Sarah: Thanks.

John: Well, it’s always great talking to you. Thanks for joining us. And we look forward to hearing more about that project as more information comes in, as well as anything else that you’re doing that you’d be willing to talk to us about.

Sarah: Great. Thanks for having me.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

349. Growth Mindset Messaging

Posted in continuing-generation, diversity, engagement, equity, first-generation, first-generation students, growth mindset, higher education, inclusion, instructional communication, large-class instruction, motivation, STEM education
July 10, 2024

First-generation college students, on average, have lower GPAs and higher dropout rates than continuing-generation students. In this episode, Elizabeth Canning, Makita White, and William B. Davis join us to discuss a growth-mindset intervention that has eliminated this equity gap in a large STEM class.

Elizabeth is an Associate Professor in the Psychology Department at Washington State University. Makita is a graduate student at WSU’s Experimental Psychology Program, and William is a Professor of Molecular Biology and the Interim Vice Provost for Academic Excellence and Student Achievement at WSU.

Show Notes

Transcript

John: First-generation college students, on average, have lower GPAs and higher dropout rates than continuing-generation students. In this episode, we discuss a growth-mindset intervention that has eliminated this equity gap in a large STEM class.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Elizabeth Canning, Makita White, and William B. Davis. Elizabeth is an Associate Professor in the Psychology Department at Washington State University. Makita is a graduate student at WSU’s Experimental Psychology Program, and William is a Professor of Molecular Biology and the Interim Vice Provost for Academic Excellence and Student Achievement at WSU. Welcome back, Elizabeth and Makita, and welcome, William.

Bill: Thank you. It’s a pleasure to be here.

John: Today’s teas are:… Elizabeth, are you drinking tea?

Elizabeth: I am drinking tea. I’ve learned from past podcasts with you that I should bring some tea along. So I’m drinking some sweet and spicy black tea today.

Rebecca: Sweet and Spicy… [LAUGHTER] sounds interesting.

Elizabeth: It has some cinnamon and some orange. It’s very good.

John: Very nice. And Makita?

Makita: I’ve got classic Earl Grey today.

Rebecca: Can’t go wrong with a classic. [LAUGHTER]

John: And Bill?

Bill:I have a Raspberry Zinger herbal tea today.

Rebecca: I’ve moved on to Awake tea… this time of day. [LAUGHTER]

John: We recorded another podcast much, much earlier today. So that’s probably needed a bit more. [LAUGHTER] And so that I won’t be awake all night tonight, as I have been for much of the last couple of nights, I have just a pure peppermint tea.

Rebecca: Probably a better choice for this time of day. [LAUGHTER]

John: And for this time of the year.

Rebecca: So we’ve invited you here today to discuss your March 2024 study: “Growth Mindset Messages from Instructors Improve Academic Performance Among First-Generation College Students,” which is quite a mouthful. [LAUGHTER] But before we discuss the study, can you talk a little bit about the difference in outcomes between first-generation students and continuing-generation college students?

Makita: Sure. So when we say first generation, we usually mean someone who’s the first in their family to attend college. For this study, we specifically defined it as a student for whom neither parent has earned a four-year college degree. And these students typically have to face and overcome a lot of social and cultural barriers in order to be successful in college. And although almost a third of the people who go to college are first-generation, they don’t always seem to do as well as continuing-generation college students. On average, they tend to have a little more difficulty adapting to college, they typically earn lower grades and they have higher dropout rates. They also act a little bit differently, they’re less likely to do things like go to office hours, or communicate with their instructors, whether that’s by email or in person to seek clarification on course materials. And just in general, they tend to be less likely to try to access helpful academic resources when we compare them with continuing-generation college students, which would be students for whom at least one parent has earned a four-year college degree. And on top of all of that, the students are also usually… I mean, they have to be… dealing with having less familial guidance than their peers do when it comes to navigating higher education. So if you’re a continuing-generation student, you have a parent who might tell you something like, “Hey, it’s a good idea to go to office hours,” or “You should be asking questions” or “there’s a financial aid department,” important stuff like that. They also may be dealing with a mismatch between the values and culture that they grew up in, compared with the values and expectations of an American university, which is typically very individualistic and may not be as supportive as what they are used to. And of course, they’re also more likely to be working, living off of campus, and they usually get less financial support from their families. But there are other factors that may exacerbate the difficulties that they’re dealing with. And this study is about one of those.

Bill:And one of the things that, just for context at our local institution to keep in mind, is WSU has a lot of, for lack of a better term, brand loyalty. It was not uncommon for me in my class to have fourth generation Cougars. And if you think about that legacy of your great-grandparents had attended this institution, as had other people, that sort of familial knowledge that gets passed along, is a huge benefit for that population of students as compared to the first-generation students as Makita was talking about.

John: One of the things that you mentioned, Makita, was differences in help-seeking behavior. And one of the studies on that is one that we talked about with you and Elizabeth in an earlier podcast, and we’ll include a link to that in the show notes. Could you describe this current study?

Elizabeth: Yeah, so this study we conducted in 2021, I believe. It was the second semester online, so the pandemic year. I know we’re all sort of crossing out that memory from our minds… [LAUGHTER] …that whole year. As instructors, it was a little tumultuous. But Bill and I got together and we designed this study together in his class. He was teaching at the time the introductory biology class at WSU. And what we decided to do is test out some actionable techniques that instructors can do to communicate a growth mindset to their students. So a lot of the research that my lab does looks at how growth mindset messages impact different students from different backgrounds. And at this time, there was a lot of suggestions about how to implement growth mindset ideas in your class, how to change course material, how to integrate it throughout, but there wasn’t really any experimental evidence of specific strategies that instructors can do. So we decided that we would just test it out in Bill’s class. So what we did in this study is we randomly assigned everybody in the course to one of two conditions: a control condition or a growth-mindset message condition. And after the first two major exams in the course, students received an email from Bill, and in the control condition they received a message that is pretty typical of an instructor after an exam has been conducted in their class. It says things like: “exam grades are posted online, here’s how we calculated them,”
“here’s what it means,” and then “feel free to talk to me in office hours, or ask questions about it.“ In the growth mindset message condition, we included all of that information, but we also included a fairly lengthy paragraph… I’ve gotten comments on… but it’s a big chunk of a paragraph that basically says, in Bill’s words, “Here’s what I believe about student performance and exam performance and improving in the class.” And there we integrated growth mindset theory, we talked about how abilities can be improved, we talked about how academic struggles are normal to experience, and that this struggle is part of the process of learning, that is something that is controllable. That you can put in effort that you can use different strategies to make improvements. And I think we used, Bill, if correct me if I’m wrong, we started with a draft of this email that you had already been doing in your class. And we just sort of infused it a little bit more. So Bill really wrote like the first draft of it, and then we kind of went back and forth a little bit to edit it and infuse it a little bit more with growth mindset. But it’s really written in the voice of the instructor. After we implemented this email after the first exam, we did it again after the second exam, changing the language a little bit. We assessed their performance across the semester. So we looked at their exam grades, we looked at their final course grade, and we found that students who were in the growth mindset condition earned higher grades in the course. And this was predominantly driven by first-generation college students. So first-generation college students performed much better in the course, when they received those growth-mindset messages from the instructor.

Rebecca: Can you talk a little bit about the effects and how significant they were?

Elizabeth: Yeah, so when we look at the group differences in the control condition, what we find that’s very consistent across big STEM courses across the university, that continuing-generation students are getting better grades than first-generation students. So in this course, they’re earning about three-fourths of a letter grade higher than first-generation students. So this is a really, really big gap between the continuing-generation students and the first-generation students. It’s not specific to this one course. So we see this in a lot of different datasets. But what was really remarkable in this study is that that difference in their grades was completely eliminated in the growth-mindset condition. So first-generation college students earned about three-fourths of a letter grade higher in the growth-mindset condition compared to the control condition. So this is a pretty large effect on final course grades. And of course, it’s pretty significant in terms of the curriculum that they’re a part of. Bill, if you want to talk a little bit about the students who took this course and how it plays a role in their career trajectories.

Bill:Yeah, thank you for that. I was looking forward to jumping in a little bit. And this is a little bit about my journey too, in this course. So, at this point, just for context, this was the 16th semester that I had taught this course. Largely I had been either the sole instructor or I was the primary instructor of record for the course. It’s taken by about 450 to 550 students each semester, little bit of differential between fall and spring in size, over 50 different majors, or pre-major students take this class. It’s nominally a freshman majors biology course. But there’s a substantial number of students who are neither biology majors or life science majors. And there’s a sizable number of students who are not freshmen. So, in reality, the highest population of my course normally were sophomores, freshmen were the largest population in that spring semester. And that was pretty typical, because it is oftentimes taken as the second semester of an introductory biology sequence. So yeah, and to be honest, over the years before this study, I had been trying to narrow those opportunity gaps between first-generation and continuing-generation students. I knew they existed. I had done many things with the course experimentally over the years, and never had really closed that gap. This was one of the first times, if ever, I saw that gap completely close in anything that I tried, and I tried tinkering in the lecture in the lab, big changes, small changes, a lot had gone on before this.

John: You mentioned that the class is normally about 450 students. Could you tell us a little bit more about the specific course in which you implemented this? You noted that this was during the pandemic. Was this offered remotely? Was it synchronous? Was it asynchronous? And how many students were there in this particular sample?

Bill:So, the course was taught in a synchronous Zoom manner. So both the lecture and the lab were taught using zoom. I believe, if memory serves, I had the largest Zoom Room on campus that semester. I can’t remember the exact number, I want to say my enrollment that semester was somewhere close to 550, or 600

Makita: The final sample is 417, I believe.
Bill:417. That’s right. That’s how many students, so it must have been around probably 450, at the beginning of the semester, or something like that. Yeah, we had synchronous Zoom every day with students. In fact, I learned pretty quickly how to crash Zoom my first semester I taught remotely. [LAUGHTER] So by then, it was a little bit more of a pro, and at least knew how to keep it running for 50 minutes at a time.

Rebecca: And true badge of honor.

Bill:[LAUGHTER] Well, there are many lumps that we took in the pandemic and learning how to teach and Zoom was one of them. [LAUGHTER]

John: I had 350 students in the fall of 2020, and it was not my most enjoyable teaching experience. What are the factors you controlled for? Because you did control for some other variables to separate out the effect of the growth-mindset messaging?

Elizabeth: Yeah, so we controlled for three things in our analytic models. We controlled for students’ prior college GPA… so this was just their self report… GPA that they’ve had in college. Since this was in the spring semester, all the freshmen had had a semester under their belts at that point, to have an idea of what their college GPA was. We included this just because we were looking at performance. So we wanted to filter out or control for their prior performance, that’s a big predictor of their performance in the class. We also controlled for their self-reported personal mindset beliefs. So we asked the students themselves, what do you believe about the nature of intelligence? Do you think it’s something that can improve? Do you think it’s something that you’re just born with? So we measure their own self-reported mindset beliefs, and we control for that and all of the analyses, because we want to look at the effect of the intervention above and beyond student’s own beliefs about ability. And then the last thing we controlled for was their race or ethnic status. So what they self reported to us on a survey. Here, we controlled for that because there is some overlap between race, ethnicity, and first-generation status. We did look at the different groups separately, we don’t have enough power to really look at race ethnicity as a moderator at the intervention. So we control for it in our analyses to look at the effect of first-gen status above and beyond race and ethnicity,

Rebecca: I can imagine, as an instructor who’s fiddled quite a bit in their class to overcome these equity gaps, that it might have been a little surprising [LAUGHTER] how much of a change happened with such a relatively small intervention.

Bill:It was shocking. It’s funny, because one of the things that Elizabeth has really talked about is how we took materials that I actually already possessed in some ways, and modified them slightly in some cases, and more radically in others, but just that transformation was enough to start to really make an impact. And you could see it happening. And that was amazing to me. I remember when I halved my opportunity gaps in my class with an intervention. And I remember dancing and celebrating, [LAUGHTER] and then this one was just on that scale. This was huge and amazing to me.

John: One of the interesting things in this study is you were looking at the effect in terms of grades on subsequent exams, but you didn’t find much of an effect on the second exam from the first intervention. But when you had two of those messages coming out, that’s where you found the relatively large effect. Why do you think the second message was having more of an effect than just a single message?

Elizabeth: Yeah, so we have a couple of theories. We think that one reason could be that they just need the message more than one time. So they need two doses or more than two doses to sort of really get people’s attention and make them change their behavior. That c ould be one reason. Another reason is that it might be enough to just have one message, but it would take time for behavior to change in order to see a difference in their performance. So it could be that after the first exam, they started engaging with the material differently and then that then sort of snowballed to affect their third exam scores, and then again, their final course performance, but we don’t really know exactly what’s going on. Another reason could be that towards the end of the semester, students get a little antsy, and they start waking up a little bit to, “Oh, I should probably be doing something different if I want to have a grade that looks like this.” So it could be timing, it could be dosage, it could be the effects take longer to materialize in terms of performance changes. One of the interesting things that we found here in this dataset is we did try to look at behavior in the way that we had access to, so we pulled all of their activity on the course website. And this semester, our campus was using Blackboard as the course sites. And we pulled all of the Blackboard data. So we looked at whether students in the growth mindset condition were engaging with Blackboard differently. And we found that they were. Students, on average, who were in the growth-mindset condition engaged with the course material 40 instances more, on average, than students in the control group. This is around a 12% increase in webpage engagement. So after they get this message from the instructor saying that intelligence is something that can grow, abilities can improve over time, and here are some strategies that you might implement to realize those changes, they’re going to the course website, they’re clicking on things, they’re re-reading their notes, they’re looking over the course lectures, they’re engaging more with the resources that are posted there. And then not surprising to any instructor ever, [LAUGHTER] that when students are engaging with the course materials that then lead to greater performance in the class. So we found a mediation effect through their behavior with the course material that was posted online. So this is one example of the behavior change that I was talking about earlier, that when they get this message, it can be inspiring, it can inspire a different type of behavior that can lead to better performance in the course.

Rebecca: Those are really strong click rates generally, [LAUGHTER] especially when we’re often complaining that students don’t read our emails or messages. [LAUGHTER]

Elizabeth: Yes. So, we did look at differences in Blackboard in terms of clicking on course material versus going to the page where all their grades were, like the gradebook. And we didn’t find any differences on their engagement with the gradebook. So it wasn’t that they were just tracking it more or anything like that. But it seems to be the case that they’re actually engaging with the substantive course materials more, on average, in the growth- mindset condition.

Rebecca: It’s just incredible. [LAUGHTER] Can you talk a little bit about… and we’re hinting towards this, obviously… what does this study suggest that faculty and institutions can do to improve outcomes of first-gen students, or maybe even, Bill, what you’ve continued or started to do?

Bill:I think, for me, what I took away from it was being very intentional with my students in communicating what I believe about them. And I mean that in the spirit of the old NPR show, or the segment that they used to do, you know, “This I believe,” where people would have those small snippets. I felt like this intervention was something along those lines where I could be authentic in communicating what I truly believed about their abilities. And I did it in a way that, with some small tweaks, obviously had these much deeper and larger payoffs. And I think that’s one of the real nuggets of wisdom here is that when you want to transform, and have better student outcomes, you want to narrow these opportunity gaps for students from certain backgrounds or experiences when they come in. Oftentimes, you don’t have to throw away what you’re already doing. Oftentimes, you already have nuggets of really important things that you’ve built over time. And all you need to do is to work with somebody who sees things just a little differently and can tweak them with you. And I think that partnership with Elizabeth and Makita was really critical for me. They were able to take things that I already had, and show me a different way to just modify what I was already doing. That messaging would go out to every student after that time point. When you see an effect that size, it almost becomes ethical to in fact, it is ethical, in my opinion that you don’t preclude any student from experiencing it. So that should be the way that everyone should be communicating with their students, in my opinion. And you want to make sure that it’s authentic as well, because I started with a set of materials that I had already written and sort of came from my heart, my own experience, that made the authenticity, I think, real whereas if I would have tried to have said something to students that I really truly didn’t believe about their abilities, I think they would have sniffed that out in a heartbeat. And we probably wouldn’t have seen the effect that we did.

Rebecca: It’s amazing what a little bit of intention can do in the design of something, especially because we often don’t realize how big of an impact a simple communication might have. We almost think of it as like a throw away, or a quick thing you might be doing, but you add a little intention to it, and it starts having a really different impact.

Elizabeth: Yeah, I think you mentioned earlier that students might not read the email, you might think that they’re not going to engage with it. And we also thought that, [LAUGHTER] so we think that in designing this intervention, we were really drawing on a lot of the intervention research that had been done previously. So research on wise interventions, mindset theory, all of that was integrated into the design of the message itself. So we started with Bill’s original emails that he was sending out to students and the messages that he had already curated for his class, and timing it in a time in the class in which students are receptive to that information, I think, is also really key. So we don’t know this empirically. But I would guess that if we were to send that email a couple of weeks before the exam, it would not have the same kind of impact. We send it directly when that performance information was posted online. And that’s when students are questioning themselves. They’re saying, “What does this score mean? Does this mean that I’m not good at this? Does this mean that I can continue? Does this mean that I’m smart or not smart? …and that uncertainty, that is the right time to infuse your message as an instructor to counteract those uncertain messages about ability. So I think the timing really matters. I think authenticity, like Bill said, really matters. Makita and I have been working with other institutions to implement a similar type of intervention with instructors. And before we did that Makita did a bunch of focus groups with the students at these institutions and showed them the messages that we used in Bill’s class, and we ended up changing them quite dramatically. The students gave us feedback like “nobody here would ever say something like that,” or “here’s what you need to say to reach students here in my class,” or “here’s what I will be receptive to.” So students in different contexts are going to respond to different messages. And so you really do want it to fit your own class, how you would speak, how you would talk to your students, the needs of the students in that moment might be different from the class they’re in. So it really is not a copy and paste type of situation where you were gonna copy what we did here and put it in your class. I think you do need to kind of go through the process of generating it for yourself and your own students.

Rebecca: I could imagine there would be an element of needing to evolve it over time, too, as your student population changes, and as you change as a teacher.

Elizabeth: Yeah, absolutely.

John: One of the things that I think is particularly nice about this study is that we see higher DFW rates by first-gen students in STEM classes in particular, and so many students end up leaving STEM fields that they intended to major in in their first couple of years. And that’s where the rate of return to education is the highest. And in general, first-gen students come from lower socioeconomic groups and lower family income groups, and if we do want to see more equity in outcomes, getting more first-gen students into the STEM fields could do a lot to help reduce some of those equity gaps in terms of overall outcomes.

Elizabeth: You mentioned the long term nature of these effects. I think it’s a great question. I think it’s something that we’re currently looking at in a project that we’re doing that’s funded by the National Science Foundation, where we’re looking at how these interventions in an introductory course can change career trajectories, that when you get a B instead of a C in a major introductory course, that’s gonna mean something different to that student, that’s going to set them on a path of following that next course sequence, that next step. So that’s the question that we’re asking and the research that we’re doing now, and hopefully, we’ll have some answers in a few years.

John: Peter Arcidiacono at Duke had done a study on the decision to stay in STEM fields. And he did look at some of these questions in terms of the effects on major choice based on the grades that were received. We can share a link to that study in the show notes. In the IRB proposal, did you include the possibility of following up these students in terms of their outcomes a few years later, in terms of retention rates for the students?

Elizabeth: That’s a great question. I believe in terms of IRB, we had students consent to their academic records. So we should be able to go back and look at transcripts from the students who participated in the class. We could look at course taking, we could look at graduation rates, we could look at major. If these students were freshmen/sophomores in 2021, then we’re probably around the time where they would be finishing up so maybe in the next year we could.

Bill:I was just thinking the same. Now’s the perfect time to go back and take a look, because we’ve had enough time since the intervention to see what’s happened. And actually I was gonna go back to one of your earlier points. You were talking about statistics earlier. Basically, this is really the only way that we’re going to increase the number of STEM students going forward. If first-generation students at WSU are 40% of our population today, that means if we don’t improve their outcomes, then we’re going to continue to see the declines in STEM outcomes and STEM graduates that we are trying so desperately to fight against, and also, we’re not going to have the representation in our STEM graduates of the population of the US that we want to see.

Rebecca: I think in addition to the growth-mindset piece, I’m sure that the authentic nature of the messages also just showed deep care. And that can be really important for first-generation students who need that ally or advocate for them in their college journey. So I can imagine those messages coming from Bill in Bill’s voice that make them feel like “Yeah, you can do this, I care about you,” from that perspective is also really important.

John: And there was a study done by a couple of economists, entitled “My Professor Cares,” where they did follow up students four years later, and they did find a significant effect in retention rates four years later by a similar type of intervention, I don’t think it was quite as much growth-mindset focused, but just sending a signal that the professor cared about the students outcomes, a very simple message, made a significant difference in student’s grades as well as longer term outcomes. So it’s nice to see more studies of this, especially focused on issues such as growth mindset.

Elizabeth: Yeah.

Rebecca: So we always wrap up by asking: What’s next?.

Elizabeth: So Makita and I and our lab have been working for the past couple years on following up on these data in different institutions. We used this study as pilot data in a grant application. And now we’re sort of scaling up to other institutions. We have a project with a really large HBCU, a Hispanic-Serving Institution, WSU is continuing to be involved as well, an Asian and Pacific Islander Serving Institution to where we have a number of instructors who vary in different characteristics, different class compositions, and we’re looking at these instructor messages in an experiment in these different contexts to see what matters. Does it help students develop their own growth mindset? Does it change their behavior? Does it work in some contexts, but not others? Do the instructor characteristics matter? So far, in the past two years, we’ve collected data from around 10,000 students who have been involved in this intervention project, and we’re currently in data analysis phase. So stay tuned for future studies and papers that come out of that data set.

John: Makita?

Makita: Well, what’s next for me is Elizabeth and I are actually going to meet right after this to do a debrief and to discuss some of the analyses that I just ran on a study that will be involved in my dissertation.

Rebecca: And how about you, Bill?

Bill:For me, I think about: “How do I proselytize about this work?” Because I think about what little changes make a big difference in an institution. We talk a lot about student retention, we talk a lot about student outcomes, we talk about creating the global citizenry that we want to see emerge from higher education. And the one experience that every student has in common, whether they are a first-generation a continuing-gen, or from some other background, is they have to take classes, and they have to be in classroom environments every day of the week, for the most part, over a course of four years. And I think about what would it look like to have interventions like this, that have evidence that back them, that students get inoculated with repeatedly over multiple time points with different voices, and that faculty engagement in the process of helping our students become better, to find success and to move forward in their careers? To me, that’s the magic of this. I’ve been talking to other people about it and I think about what would a curriculum look like that had small things like this? And how could it transform what we do? And I think that’s a vision that I can buy into, and I hope others are gonna come along for that ride.

Rebecca: I think the scope of the interventions that you’re talking about, that have this kind of impact, are really easy to buy into. Because usually you’re thinking about some drastic, giant investment of time, is what most people are thinking about. And that often is what prevents action from happening. So I’m excited, excited to share this and to hear what comes of your bigger study.

John: And one thing we appreciate is when we see things like this posted on social media so we can hear about it [LAUGHTER] and we can share it out a little bit further. [LAUGHTER] And I hope you’re presenting this at various conferences to reach a larger audience.

Elizabeth: Yeah, we will be presenting some of this work at SABER this summer. So that is the Society for the Advancement of Biology education research, and we’ve presented it in some other psychology conferences throughout the past year as well.

John: Excellent.

Bill:And I gave a plenary talk at the summer educational meeting for the ASBMB, American Society for Biochemistry and Molecular Biology. So I gave a little preview of what we had found. And that generated a lot of discussion and excitement, actually people asking for additional resources and ideas and things. So I think there’s a receptive audience out there.

Rebecca: That’s wonderful.

John: Well, thank you for joining us. It’s always great talking to you and it’s great to hear about this newest study and to meet you, Bill. We’re looking forward to hearing more about how things go in the future.

Elizabeth: Thank you so much. Thanks for having us.

Rebecca: I hope you’ll reach out when your next study is done. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

348. Active Learning Initiative at UGA

Posted in active learning, assessment, engagement, evidence-based practices, faculty learning community, higher education, inclusion, learning, pedagogy, professional development
July 3, 2024

While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, Megan Mittelstadt and Leah Carmichael join us to discuss the active learning initiative at the University of Georgia that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces. Meg is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia.

Show Notes

Transcript

xJohn: While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, we explore a large-scale active learning initiative that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces to better support this initiative.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Megan Mittelstadt and Leah Carmichael. Meg: is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia. Welcome Meg: and Leah.

Meg: Hi.

Leah: Good to be here.

Rebecca: Today’s teas are:… Meg, are you drinking tea?

Meg: I am, I’m drinking double chai. Tea is a big deal. In our center. All of the people who work here have favorites. And we provide tea service at every workshop and I checked this morning, the most popular tea at our workshops is lemon ginger.

Rebecca: Oh, that’s good to know. Another teaching center full of tea. [LAUGHTER]

Meg: Yeah, we even have the teas of folks who don’t work here anymore that we keep stashed, just in case. So, Leah, you know, Colleen. Colleen drank…

Leah: Yeah.

Meg: …caffeine free peppermint tea, and we still keep it stashed, even though she doesn’t work here anymore. [LAUGHTER] Just in case.

Rebecca: How about you, Leah?

Leah: Well, I am a diehard coffee drinker. And so I am enjoying a cup of coffee. But my best friend is a potter, so I have her beautiful mug here to enjoy it with.

Rebecca: Wonderful. How about you, John?

John: I’m drinking a black raspberry green tea today.

Rebecca: Oh, that’s nice. And I’m back to my blue sapphire black tea.

John: Black and blue?

Rebecca: It’s black and blue. [LAUGHTER] We’re recording this at the end of the semester. [LAUGHTER]

John: We read about the active learning initiative in a recent Chronicle article and we invited you here to talk a little bit about the active learning initiative at the University of Georgia. We were really impressed to see that the university had budgeted over a million dollars a year for five years as part of the university’s Quality Enhancement Plan. This is a really large investment in effective teaching practices, and it’s especially impressive that this is happening at an R1 university. How did this initiative come about?

Meg: Yeah, we’re really excited that this investment is happening. And it kind of stems from an initial pilot that happened in 2018. There was a presidential initiative on active learning that resulted from a faculty task force. And so in 2018, our president allocated a million dollars to active learning programs, as a pilot. I feel super lucky to have a president who cares so deeply about the teaching enterprise at UGA. It’s something that you don’t see at every institution. And so I really appreciate it, value it, and count my luck there. Our president allocated a million dollars in 2018 for an active learning course redesign institute and a pilot classroom infrastructure renovation, and also the creation of a one-credit course for our students called Becoming an Active Learner that’s deployed by our Student Success Center. And that first year, we thought, “Gosh, it might just be a one-year investment, so we really got to make this count.” And so all the folks in the center sort of got together and backwards designed the programming for our first Active Learning Summer Institute, which by the way, was one of the most gratifying and enjoyable experiences of my career to get five educational developers in a room and get to go totally meta and apply the backwards-design process to the creation of a Course Design Institute. We nerded out. [LAUGHTER] We had like the room where we had all of our notes on the walls, and it was sort of our Beautiful Mind room for the Course Redesign Institute. We loved it. And so we designed this institute for 2018, and it went quite well. The folks at our institution who applied were way more applications than we could accommodate in the Institute and the folks who got into the Institute had a good experience. And we were able to show a couple of courses that had been redesigned seeing some really significant positive outcomes for students, both in terms of their depth of understanding, but also in some of the metrics in courses like numbers of Ds, Fs and withdrawals from courses decreasing in a few really key courses like our introductory mathematics course sequence. And so we were able to go back to the President and the Provost every year after that and say, “Okay, here’s how the course redesign Institute went, could we get funding to do it again?” And so we continued to do it every year. And this continued for a couple more years. And when we got into the 2020 timeframe, UGA was preparing for our next reaffirmation through SACSCOC. And for SACSCOC reaffirmation, an important component of that reaccreditation proposal is the creation and proposal of a Quality Enhancement Plan (or QEP). And so a committee was assembled to come up with what that topic might be, and they looked at programs that existed on campus as well as the university strategic plan. And luckily, by this point, our strategic plan had been drafted with quite a few key performance indicators related to the adoption of active learning practices. And so they identified this opportunity to scale the Active Learning Initiative at UGA. And so they proposed this topic, it was selected and now the funding has been stabilized for some of the pilot programs that existed, but also some newer initiatives to0.

Rebecca: There’s a substantial body of work supporting the efficacy of active learning and increasing student learning and reducing equity gaps. And it’s so exciting to hear about such a big investment and the interest of faculty [LAUGHTER] having more interest than spots for example, But I know one of the things that sometimes happens when faculty decide to start implementing active learning in their classes is sometimes there’s some pushback from students, but I heard you say something about a one-credit course for students. Can you talk a little bit about that? And how that ties to preparing students for this endeavor?

Meg: Yeah, so because we had this course from the get go, there was already this idea that reaching out directly to students and building their capacities and dispositions as learners to amplify their success in an active learning environment is something we might ought to think about. Of course, we also think about it from the faculty side too. One of our most popular sessions in the Course Redesign Institute is a session on student resistance to active learning, and we rely heavily on Tolman and Kremling’s Why Students Resist Learning for that session and then talk about some of the instructor facilitation and explanation strategies that instructors can use to ensure that student resistance is not cropping up and to help prevent barriers to student engagement. I think what’s really cool about Tolman and Kremling’s book is that they reframe “student resistance” to “barriers to engaging” and so we work with faculty to reframe student resistance as “Where are the barriers to engaging that we need to address as instructors?” And then on the student side too, the philosophy is “Where are there barriers to engaging?” And then “When can we do some really important work to help students appreciate that engaging in active learning is a really actually durable practice for them?” It’s really going to benefit them long term. And since that idea was already in the mix, as we were coming up with the programming for the QEP, there was a really large committee, I think, a 33-person committee of faculty, students, and administrators who proposed the programming that would go into our Active Learning Initiative. And one of the things that they wanted to think about was the role that metacognition plays in active learning. That was identified really early on because we were already aware of the student piece, thinking about metacognition as a key component of active learning. The committee spent quite a bit of time unpacking: “What are the learning dispositions involved in student metacognition? And what are the learning dispositions involved in lifelong learning and other benefits of active learning? And where’s the overlap?” So we looked at things like the AAC&U value rubrics on lifelong learning. And I think the other one was, I want to say, it was like transferable learning, I’m getting the title wrong. But anyway, there were two AAC&U value rubrics that we looked at, as well as some of the literature on metacognition and literature on lifelong learning. And our assessment team identified four student learning dispositions that seemed to exist across all of the categories that we were looking at: students acting more curiously, students engaging with initiative, students becoming more reflective in their practice as learners, and students drawing connections between what they were learning in their active learning courses and developing other knowledge and skills. And so we use this framework of these four learning dispositions to really drive our student programming. And I’m going to pass it over to Leah, so she can share a little bit more about how we’ve expanded beyond this one credit hour course to a whole portfolio of student facing initiatives.

Leah: So, first of all, I would like to say with the instructor development piece, I was an instructor on campus, a lecturer, and was in that first cohort in 2018, an international affairs scholar. And it really was the first time that I had had such a professional development opportunity that felt like it truly revolutionized the way I taught, the way I thought about my students, the way I interacted with my students. And so whereas I was taught to transform one of my courses, I ended up transforming them all. And when I was able to be on the team for active learning, I signed up, it was really excited to do that. So it was an amazing experience. For the student piece, what I really love is that we have not just gone to the instructors and said, “Alright, it’s time to consider something different and then your job is to not only do all this hard work to change and rethink your courses, but then can you do us a favor and sell it to the students?” That’s just such a burden and very difficult. And so within CTL, they do a great job of training us in the Institute to go in first day and frame the course, to craft our syllabus language in a way that would potentially be able to introduce this course before even day one and to help decrease that student resistance. So that was a really important piece. But we also would have these teams of students, these just superstars on campus, who are also part of the programming for active learning. One of my absolute favorites is the peer learning assistants. This is an on-campus paid position for students. They apply and they are able to either return to a class or return to an instructor whom they have worked with before. And they attend a class where they facilitate active learning approaches and activities with the instructor. So the peer learner is responsible for helping with the planning of the course and preparation with the instructor implementing it and then also of course, following along with the course materials. And this is a key: this near-peer experience has been key in allowing the burden to be shared in introducing active learning within a classroom. And the students look at these peers, maybe a semester ahead of them sometimes, maybe years, and they start to see these mentors that are built in. And these students know and care about active learning. They see why it works in the class and they’re returning to the class. And we have found that those classrooms have been just the most wonderful place to see this cultural change among students. We also have active learning ambassadors. And they are trained to do outreach. They do demonstrations for faculty. One of my favorite moments is we have an active learning summit every year open to the public. But we have an active learning summit in which they run a showcase where the faculty walk in, and there are students behind tables with interactive material demonstrating different active learning techniques and how they see it in their class, how they use it for study skills, and how they would enjoy instructors being able to incorporate it in their classroom. The faculty love it. They stop, they ask questions, they think, “Well, what about this? Well, this would be difficult.” And then students give that experience from their perspective of how these techniques work for them. And we see that the faculty really crave that and really enjoy being the learner in that moment, and learning from a student about what techniques work best for them.

Meg: And Leah, I think one thing that’s really unique about our quality enhancement plan that I haven’t seen at other institutions yet, or if they have them, I haven’t seen them, so mea culpa. But one thing that’s really cool about our Quality Enhancement Plan is that we have a partnership with our Office of Student Affairs where they’ve designed some programming through our residential life curriculum to embed active learning in the co-curriculum too, so that students are experiencing and talking about the value of active learning not just in the classroom with each individual instructor, but experiencing active learning outside of the classroom in the co-curriculum. Students who are leaders on campus are trained in active learning facilitation techniques, so that when they go in to serve as leaders of student groups on campus, they have some of these active learning tools in their back pocket that they can deploy and experience on the other side as well. So we’re really seeing active learning proliferated, not just in the classroom, but outside of the classroom too.

John: It’s nice to see such a holistic approach. In so many cases where we’ve seen active learning initiatives, it’s usually just designed for faculty development, to go and fix all those issues and challenges. And by approaching it from all these angles, it seems to be a really effective way of addressing some of the challenges that faculty face when doing this. One of the challenges, though, that faculty sometimes face when they introduce active learning is they go into a classroom where the seats are either bolted down, or are very rigid. I think part of your plan involves some type of infrastructure changes to support that. Could you talk a little bit about how classrooms have been redesigned as part of this process?

Leah: It’s such a key aspect of it, I have the anecdotal experience where I graduated from the Active Learning Summer Institute, went into a classroom ready to be highly engaged and interactive and set up have a highly mobile class design only to find that I was in stadium seating, with fixed seats and I thought, “Oh, no, what am I possibly going to do?” And so we have three foci, within the initiative. And we’ve mentioned the instructor development piece. So key, but as you said, it can’t stand alone without support for the other two. We have the student piece, we also have the classroom piece. And so our goal is to every… usually it has to be in the summer, when it’s much quieter… every summer, we just push for classroom enhancements across campus. And we have three criteria in mind that we see as not necessarily perfect for every single classroom situation, but the baseline for which we want to capture for an active learning classroom. And so the first is: does this classroom have mobile furniture? Are we able to move students around? Could an English professor have a roundtable discussion? And then within the 20-minute transition time, could my students go into that same classroom, which happened last semester, in a highly mobile classroom and be able to set it up into a simulation of roleplay thing for my international politics course? So do we have classrooms with that mobile furniture is the first thing. The second is, do we have student collaboration tools? And these do not have to be high tech. In fact, we ran the survey, and students and faculty said just give us more dry erase boards. We don’t need the fancy, fanciest of smart screens. Please just give us the basics. But let us work on a project huddled over together and be implementing and creating something new. And then finally, it’s just general space, do we have enough space that we are able to not only move the furniture around, but move people around in an effective way. And so 25 square feet per student is kind of our ultimate goal. And so we looked around and I was actually just running the numbers recently. And I think on campus we have, depending on how you measure our classrooms, between 400 to 600 spaces, it depends on if you count labs, and these other learning spaces, about 82% have at least two of these criteria. And so we’re getting there. We’ve made a real push and we’re trying to say: “Yes, instructors, please change but also know that we’re going to meet you with an environment that facilitates this kind of approach to learning.

Meg: Well, I also have to interject and say that the stance of the CTL is that active learning can happen in any classroom no matter the features. And so of course, like a good backwards design course or re-design institute, we always start out with the situational factors of a course. And that might include what’s the likelihood that you’re going to be in a fixed-seat auditorium or fixed table, but moveable chairs, and think about the choreography of that, the active learning techniques that you’ll adopt based on that. But in Leah’s example, in particular, if I remember correctly, Leah, you had a student in a wheelchair in your course the first semester after you redesigned your course. And when you were in the tiered auditorium, that meant that there was a specific area where the student in the wheelchair could sit and they couldn’t move through all of the case study roles that you had available. And so you were able to advocate for a different classroom, you obtained one, thank goodness, and everyone could move through a flat classroom instead. And so sometimes things crop up, even if you have planned to be in a fixed-seat auditorium, that you have to be able to modulate on the fly based on the characteristics of your students each semester too.

Rebecca: Can you talk a little bit about whether or not your program has focused on specific programs or if it started with specific programs and its spread to different programs or what your strategy is for involving faculty across disciplines?

Leah: One of the most important parts that I really liked about this initiative is it’s not a mandate, we are not requiring any instructor to adopt active learning through a mandate. Instead, it really is a support, educate, encourage. And so we really have not been focused on okay, this unit by X year needs to have Y outcome in terms of an active learning goal. But in terms of strategy, what’s nice is we can start to see the people that walk in the door to find the active learning Summer Institute. And the high demand that first year was a nice little kind of bellwether that, okay, there is something going on here. When you have more people that can fit, and that’s consistently. The CTL needs to keep it kind of a quality curated experience with a cohort that goes through and has to do these really difficult re-education, understanding education differently, and then of course redesign. Such a dense set of tasks. So with this cohort like experience really requires so much of the faculty. What we started to notice, as we looked around and realized that instructors from different schools and colleges were coming to the Summer Institute, and we have seen that as the CTL has expanded beyond the Summer Institute and provided workshops, both standalone workshops, as well as a course redesign experience outside of the institute. We see hundreds of faculty each semester coming into the programming of active learning. And really we have, I believe, 19 schools and colleges that have undergraduate programs, and we have representatives from each that participate in this programming.

Rebecca: That’s wonderful.

John: What type of training do faculty receive as part of the institute and other training that you provide?

Meg: It was initially just that summer Course Redesign Institute. And what we started hearing from faculty as we would get these applications every year is the deadline would pass, and we’d hear from some faculty who would say things like, “Oh, I really, really want to do the institute. but summer is my field research season” or “I’m in the clinics in summer,” or “I have childcare duties that I can’t get away during that time of year.” And so there started to be some sort of advocacy for other on ramps into the programming, so that when we had the opportunity to stabilize the funding through the Quality Enhancement Plan for active learning initiatives, we started thinking about how can we make this program one that faculty could find the on ramp that is appropriate for their needs? Is it a faculty member who’s been using active learning for a long, long time and they really want to brush up on their skills in one particular area? And so that might look like engaging in a specific workshop on something like facilitating group discussions in a large lecture environment, or looking at student metacognition and how that plays into a particular student developmental level that they might teach in their courses? Or does somebody want to do the whole workshop series and Course Redesign Institute all the way through, but they can’t do it during the summer, or anywhere in between. And so what we ended up with is a workshop series, we have 12 standard workshops that we offer over time: four fundamental workshops, and eight special topics, workshops, and we rotate through those. We offer a couple of each every year. And then we have the active learning summer institute that happens every May. And then starting next academic year, over the course of the spring semester, we’ve designed a course redesign experience that takes the course redesign elements out of the active learning Summer Institute and decouples those elements from the active learning workshop components of the Course Redesign Institute that we currently have, so that faculty can upskill through the workshop series. And once they’ve taken eight of the 12 workshops, they can apply for the course redesign experience over the spring semester. Through that course redesign experience, they come and meet with us for a couple of hours every three weeks or so over the course of the spring semester. And the idea is that by the end of that course redesign experience and workshop series completion, they’ve recapitulated all of the experiences that you might have during the multi-week summer Course Redesign Institute. We also provide different communities for folks to engage with. We have several robust Faculty Learning Communities on campus. So we have multiple faculty learning communities related to active learning, one specifically coming up this year that Lea generously funded from the Active Learning Initiative budget for folks who are looking to engage in SOTL, Scholarship of Teaching and Learning, on their active learning endeavors, as well as specific faculty learning communities for the folks who have peer learning assistance in their courses, or who are engaged in the work after the course redesign experience, for the alumni of the Course Redesign Institute. We also have opportunities for faculty who are highly engaged and master teachers to serve as active learning mentors to their peer faculty. So we have three active learning mentors every year who work with the folks who’ve graduated from the course redesign institutes to give them opportunities for office hours with their faculty peers for micro mentoring sessions on specific topics of interest. They also create supportive resources. We have similar mentors for our PLA faculty, our peer learning assistant faculty. And for those who are serving in those mentoring roles, the Center for Teaching and Learning provides them with opportunities for additional coaching and support on instructional development, educational development, instructional coaching practices that they might bring to bear in those consultative sessions with their peer faculty, too. So the goal here is that we have some workshops that are entry level, some special topics, workshops, the course redesign experiences, some stuff for the master teachers, who are really wanting to be mentors and wanting to kind of serve as role models within their spheres of influence and for others who are moving through this programming, and we’re helping them build capacity in those areas, too. And then, of course, we have faculty on campus who have been working in active learning for years and years and years, and who published on active learning, who are real role models nationally. And so we really try our best to honor their expertise and bring them in to help frame the ways in which we deploy our programming, but also to serve as guest speakers and to engage with folks who are newer to active learning so that we can help provide opportunities for that cross-collaboration and multidisciplinary discussion.

Rebecca: I love the range of options you have available, and really how much you all have embraced the idea of flexibility for your teacher learners, just like we might want in our classrooms as well.

Meg: Yeah, yeah, that’s right.

Rebecca: Can you talk a little bit about some of the active learning techniques that faculty have adopted after going through the institute or other variations?

Meg: Yeah, so as you’ve heard now, our summer institute is three weeks long. And so not only are we giving sessions on active learning, but we’re modeling active learning throughout. And so folks see a lot of different techniques that they might not have seen before. And they’re trying to identify which things that they’re seeing, which instructor moves they see, that align with the needs of their students, of the needs of their course, and their teaching persona as well. So as you might imagine, folks to leave the institute, select different things that are authentic to their practice and their situational factors for their course. So we go into the Institute, which actually begins today for summer 2024. We’re recording on the first day of the Active Learning Summer Institute. So we’re getting the anxious energy of meeting the new faculty. We start out on the first day of helping faculty appreciate that we don’t see active learning as a homogenous set of teaching activities, we’re not going to give them the recipe. We want them to discover the recipe that works for their course. So our QEP is sort of predicated on the notion that the extent to which students are actively engaged in thinking and applying what they’re learning is of far greater importance than the particular active learning instructional strategies used. And so to that end, the course redesign process helps faculty first hone in on the key enduring understandings and learning outcomes at the course level they want their students to leave with and identifying key learning bottlenecks or threshold concepts that they’ve identified that students tend to either have a tough time with, or they’ve really had to work hard to help students grasp that concept in order for them to move from novice to expert in that field. And then we encourage faculty to design their active learning interventions in alignment with those key features of the course so that active learning is more like the trunk of the tree rather than the ornaments on the tree. And our philosophy there is that when pressed for time, the ornaments are easy to cut, but if the act of learning is baked into the backbone or the trunk of the course, that’s just inherent to the identity of the course and it’s not something that’s going to be cut for time. And also you’re putting your energy into where the biggest impact is going to be for your learners..

Leah: If I may add, this was the revolutionary part of being a participant in the Active Learning Summer Institute. I thought I was going for the tips and the tricks, I’d get my little suitcase of ideas and I would just kind of put it all in there through the Institute, and then graduate and use all those as those ornaments as Meg: described it. And instead, I found myself looking at my course and realizing that the backwards design principle had me throwing a lot of content at my students, and not really aligning the activities in the course towards the student learning outcomes. Once I was able to do that hard work, then the excitement of “Ooh, okay, well, which tips and tricks, which kind of active learning engaging styles would be most useful for this.” And that’s just a really exciting way to do it as an instructor.

John: Most instructors, as they come through their educational process, have not been exposed, particularly not in graduate school, to active learning techniques. How do you address issues of faculty resistance to completely changing the way in which they’re approaching the design of their classes.

Leah: It takes a while, [LAUGHTER] having been one of the resistant faculty, I’ll speak to it from this perspective, I think Meg: referenced the former colleague Colleen, and she was my mentor going through this institute. And I remember her asking me to write all the content that I felt was so essential to get the students to to learn throughout the semester, and I distinctly remember this moment where she said, “Okay, let’s put that in an envelope, all that stuff that you just wrote down, we’re just going to put it in an envelope, and we’re just going to set it over here, it’s still very much matters, don’t worry, it’s not going away. It’s all still important.” And then she said, “Let’s really think about the future student. Who are you teaching? What would be a success?” And I think that’s a moment where I could take a deep breath, as a faculty. At first, it was scary and fearful. But it’s a moment for the first time you take a deep breath and go, if I miss a case in international law, they are still going to thrive in this world. I’m focusing on what skills they actually need to thrive in this future career. And so it’s not about that specific case, it’s about the skills. And once I was able to realign that, and it’s so important to be sitting in a cohort, it’s not an isolating experience, you’re sitting in a cohort with other people doing the same reformulating of their whole profession, with mentors from the CTL, helping and encouraging them. Without that sort of ecosystem, it would be a very lonely and harder… just slow… process, maybe, even if you are willing to change. So I think the quality of that instructional development experience within the CTL cannot be replicated. And then I hope the other two kind of prongs of the initiative, once they leave the institute, they’ll still have the mentorship for the next year with CTL and with the faculty mentors. They then can leave and say: “We have the classroom engagement things.” We are also trying to change the environment to help you and we want your feedback. We are always asking those active learning instructors, what can we do differently within these classrooms? And then also the students… we’re engaging the students. So you’re going to walk into class, and they’re going to know what you’re trying to do, and that it’s recognized as a really good thing to do across campus. So it’s the ecosystem that will create more empathy and support and therefore reduce the resistance.

Meg: Yeah, so Leah mentioned, this idea of you have this mentor that exists for the year after the institute. And I don’t know that we explained that. That’s a pretty unique feature of our Course Redesign Institute. And it comes somewhat from our Center for Teaching and Learning has quite a bit of tradition around investment in longitudinal cohort-based programming. So we have a lot of one-year, two-year, multi-month cohorted faculty development experiences, because we find that that interdisciplinary conversation is so helpful for helping folks build connections and shared disciplinary content, and teaching practices with one another in ways that proliferate those great ideas across campus and seed innovation. And as we thought about the ways that our faculty fellows…we call them Faculty Fellows Programs at UGA… ways that our faculty fellows programs were successful, were in that sort of longitudinal support that was available. And we wanted to replicate that as best possible with our Course Redesign Institute. And when we thought about the outcomes of our institute, and how we wanted people to leave the institute, be prepared to deploy a redesigned course in the coming academic year. And we started thinking about what that experiences like as a teacher, and we thought, “Gosh, people are going to leave the institute, be all excited, go back to their department, where they might be one of few who’s engaging in this sort of robust active learning commitment, especially in the early years of the institute.” How can we support them in being successful when they go into their first class and leave the class and go: “Ooh, that didn’t feel like it usually felt I wonder if it went well?” or they do their first robust activity and maybe it doesn’t go quite as they were expecting? How can we be there for them in those moments? So as folks leave our Course Redesign Institute, they’re each paired with a mentor from the CTL. And that mentor is their touch point for the year following the institute, so that when they have that class session that throws them for a loop or they get some feedback from students, that they’re not quite sure how they’re To respond to it, they have a touch point that they can go to. And it’s someone who knows their course intimately, who’s worked with them through the redesign process, is aware of the ways in which their course has changed, and is there to provide them with that support. And also that mentor is the person who performs… we call them mid-semester formative evaluations… but at other CTLs they’re called small group instructional diagnoses. So at the midpoint of their course, someone from the CTL comes in and talks with their students to focus group them, to collect some feedback on how the course is going so far, and then works with the faculty member at closing the loop on that feedback to help students appreciate that they are such an important component of the redesign process too, because the instructor is engaging in this two-way exchange of information with the students both on the content and on the operational organization of the course and refining the course in response to that feedback. So we want the students to appreciate that their experience and the way that they’re learning and what they think could be happening to support their learning even better, how that could be incorporated into the course as it continues.

Rebecca: I really love the depth of the culture change components that are built into this initiative, from feeling like you’re part of something bigger, the fact that it’s voluntary, your support structures, it’s holistic, both Academic Affairs and Student Affairs, there’s opportunities for feedback loops. Change can be really difficult, but when you’re a part of that bigger network, it can be helpful. The timing of this was really interesting when it kicked off…so, a couple years before COVID, so you had some time to establish the roots, and then COVID erupts, AI erupts, there’s more first-gen students, we’ve got a lot going on, and there’s a lot of change happening. How do you help your faculty avoid burnout with so much change and maintain this momentum?

Meg: Yeah, great question. You’re taking me right back to the pandemic and that experience of everyone just doing the best emergency remote teaching you could do. And one of the things that I remember from that period is faculty who had gone through the Active Learning Summer Institute mentioning that they were so glad that they’d gone through the institute because it made pandemic teaching easier. They had recently conceptualized of their course learning outcomes that identified the most important content that they wanted to prioritize, they had a sense of some lesson plans for how to deploy these things. And even though the institute happens in person, because of the way we were talking about how you would make adjustment to different things in different settings, they could more easily conceptualize of how to engage online or asynchronously with students using active learning principles. So we had this early indication that the folks who’d experienced the course redesign process felt better prepared for emergency remote teaching and pivoting quickly. We also found active learning to be perfectly timed as an initiative as a response to post-pandemic disengagement. So when students were coming back into the classroom and members of our community were craving that deep intellectual engagement and community with one another, but then in some instances, were a little bit disappointed when it didn’t materialize in the ways that it might have before the pandemic, we found that faculty were coming to the institute, saying, “This is something that I’m hoping can meet this need that I have to engage students in rich conversation or to engage students in deep intellectual engagement in this learning community, and how can I address this through active learning?” And also, we’re lucky to have a large number of faculty champions for active learning on campus, folks who’ve been using active learning for a while, and in various departments across the institution, not just in specific pockets. And so they’re serving as role models and mentors to the faculty in their unit. And a message that those faculty champions were helping us reinforce is that teaching with active learning is a joyful way to teach. And so oftentimes, when these faculty champions, these mentors, were talking to folks who were saying things like “post-pandemic teaching is a little bit disappointing,” or “I’m feeling a little bit burned out because of everything that we’ve experienced through the pandemic, and through these various disruptions to higher education.” When their mentors were telling them, “Have you considered bringing some joy into your classroom and making teaching a more joyful experience for you and for your students? And guess what? You can do that through active learning. Let me tell you how I brought joy into my classroom through active learning.” We would see folks come to us and say, “I’m just really looking for some rejuvenation and some revitalization, and I’ve seen my colleagues just transition their course through active learning in a way that they’re just skipping down the hall after they leave their class, and I want that feeling. So I’m hoping that this will be that experience for me.” And so we had people coming to us asking for that, and we hoped that would be the case. And then I’ll just add that like many higher education institutions right now we’re in the midst of an evaluation of teaching culture change, too. And so we’re trying to adopt a more what we’re calling a three-voice culture of teaching evaluations that’s relying upon your individual reflection on your teaching some data from your students and the interpretation and contextualization of that data, as well as peer observation of teaching data and bringing that all together as sort of like triangulated data to provide some insights on our teaching, and then respond to that and have like a assessment loop on this. And so it’s this idea of a culture of continuous improvement with teaching where we’re able to proliferate this message of: “You’re not an award winning teacher necessarily on your first day of teaching, but this is a set of skills that you can develop over time. And also, you’re never done developing it.” And so this culture is helping to amplify the message that taking part in educational development is not just an expected part of being a teacher, but it’s an opportunity to continue your culture of continued mastery.

John: So one of the ways in which this is spread has been from faculty to faculty. One of the things I’ve been noticing a lot on my campus is that it also sometimes spreads from students to faculty. As more and more faculty move from faculty-centered teaching to student-centered teaching, students become a little bit more vocal sometimes about expressing what works for them and that will often result in faculty considering using new techniques that their students have recommended to them. Has that been happening as well at your institution?

Leah: That’s why we wanted to capitalize on this proactively and have channels of growth. And so again, we have little areas in which we have students really playing that intentional role. So rather than maybe a complaining, “I went into this class and I didn’t like it” kind of a customer satisfaction problem area, we really want to make sure it’s more that students, first of all, are trained in the active learning pedagogy. So my active learning ambassadors, as they are chosen, we then say, “Okay, here’s the wizard behind the curtain friends, come join us [LAUGHTER] and see what’s going on behind these intentional classrooms that you really enjoy and don’t quite know why you like it so much.” So we bring them into the understanding of what instructors are doing when they’re in an active learning classroom and why. And then we give them opportunities to then go and front face and tell other faculty, “this is what an active learning classroom feels like for a student.” This is why one of my favorite moments… we have, again, as I mentioned, an active learning summit every year in February, and there was an entire breakout session run by students who were peer learning assistants and active learning ambassadors. And they were talking about the study skills that they have seen through their training, or through their classes that really work in their discipline that they use outside of the classroom to kind of empower themselves to study and to learn. And what was wonderful is that the front of the room were students in a panel and most of the room or faculty and I found that the question and answer period, the instructors would raise their hand and say, “Okay, well, but I’m not in that discipline, but I’m thinking about bringing this into my class in X way, is this something that you would find effective?” And they were asking the students about designing their course to be student centered. And that’s when I think if we put those students that know what is going on, so it’s not a complaint section, it is more a reframing and a sharing out of we know what you’re trying to do and we really appreciate it, and here’s how it works really well, for us. It just starts to kind of change that conversation from being yet another instructor versus student dynamic, which I think we have plenty of those around campuses at any given time to becoming an instructor and student experience.

Rebecca: Having student ambassadors and student voices celebrating this work, is certainly a testament to the success of this work. And you’ve also mentioned some other moments of assessment. Can you talk a little bit about how you’re assessing this initiative?

Leah: There is a rich assessment team who is actually part of the larger Active Learning Initiative. And so I hope I do justice to their very nuanced framework that they have. We have macro level, mezzo level, and micro level indicators that we are trying to capture. At the most macro level demand is one of the earliest indicators of success for us. Are people applying to the Summer Institute? Are ambassadors applying for the program? How many peer learning assistants apply and how many instructors want them, just really kind of those basic quantifiable amounts for demand. And then we also are attempting to capture survey data across the university. We stop the students and say: “We don’t know if you’ve interacted with active learning in the classroom or not, we’re not selecting a given class, we’re just wondering what you would say if you observe a cultural change going on.” So we’re also trying to get these larger macro level indicators. Then at the mezzo level, we’re getting into those lifelong learning dispositions that Meg: referenced earlier, the curiosity, the initiative, the reflection and connection. We have artifacts that we pull from courses that have an instructor who’s either going through all the Active Learning Summer Institute, or who has gone through it already, so we do a pre- and post-analysis. And we have faculty scorers, who can go through these artifacts to see if we are capturing if the artifacts include a prompt about curiosity or a prompt about initiative. And then we are attempting to measure, and we’re working on different ways to do that, but measure if we’re seeing growth over time in these lifelong learning dispositions, and then finally, at kind of the micro level, we’re making sure that the disciplinary knowledge is being enhanced within this that are focused on active learning techniques. One thing we have yet to mention is yet another part of our holistic umbrella. This is why the programming is so large, but we have these active learning change grants. And this felt really good to hear for me as an instructor who went through the institute. And then it was about five years later that the next person in my department went through the institute. What these change grants do is it really allows those who go to the institute to go back to their department or their unit and say, “What if we change several classes at a time through some sort of infusion of deep learning and the CTO will support us, the assessment team will help us assess whether we’re doing what we’re saying we’re doing.” And so on this micro level, the assessment team is really starting to capture some pretty cool indicators, that when you kind of create a unit change, there’s a cascading norm it’s called in international affairs, where you change enough of the unit and enough of the actors in a given scenario, you can then really see that norm start to shift in a unit. And so at each of these levels we’re finding it’s still early. So we’re not saying “It’s a win, close up shop,” [LAUGHTER] but we are finding really robust indicators of early success.

John: Well, you’ve given us and our listeners a lot to think about, and it sounds like a model program that could be copied very productively at other institutions. But we always end by asking: “What’s next?”

Meg: We’ve started to hear from folks that are interested in proliferating initiatives. So, of course, we welcome conversations with folks who are interested in hearing about some of the elements of the program and brainstorming ways to deploy these on other campuses, lessons learned along the way, etc. I mentioned earlier that as folks crave these multiple on ramps, we’re deploying these additional on ramps over time. So starting in the fall, we’ve designed some of our foundational active learning workshops, we’ve taken those and created asynchronous online versions of those workshops that are going to be hosted in our learning management system. So they’ll be available on demand. So if a faculty member doesn’t happen to be available at the time that we’ve offered that workshop this semester, they can still engage, especially in those foundational workshops. And then the spring semester course redesign experience version of our Course Redesign Institute will be deploying for the first time in spring 2025. We’re really excited about that as well.

Leah: I would also say for your listeners, if there is an interest in kind of seeing what we’re up to and helping participate in engaging with this discussion, traditionally the active learning summit has been really focused on UGA community for the first two years, but in the third year, so February 2025, we are going to open up the summit to visitors and encourage them to be a part of these breakout sessions or propose different ideas that they would like to talk about. I know Rebecca mentioned earlier, the challenges of AI, we’re going to try to have a breakout session, though, that focuses on this: active learning best practices in the age of such change. We’ll send out a call for applications in the fall, and we’d love to have anybody who’s interested in learning more, I think it will be Thursday, February 27, and Friday, February 28, on UGA campus, and we welcome all those who are interested in attending.

Rebecca: That’s great. Thank you so much for joining us. There’s so much to think about and so much exciting work that you’ve all done.

Leah: Thank you.

Meg: Thank you so much for having us.

John: Thank you and we look forward to more conversations in the future.

Meg: Absolutely. Anytime.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

 

347. CATs and AI

Posted in access, clickers, diversity, edtech, equity, feedback, first-generation, higher education, inclusion, learning, pedagogy, teaching
June 26, 2024

Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, Todd Zakrajsek joins us to discuss how generative AI can enhance these techniques by providing more immediate feedback.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo.

Show Notes

Transcript

John: Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, we explore how generative AI can enhance these techniques by providing more immediate feedback.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Todd Zakrajsek. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo. Welcome back, Todd.

Todd: Thank you, Rebecca.

John: It’s nice talking to you again. It’s been a while. I think the last time was last summer.

Todd: Really? It seems like yesterday. Time goes by. It was last summer. You came to my office and recorded.

John: That’s right.

Todd: It was fun. You left me a really nice set of tea. Wooh.

John: And a mug.

Todd: Yes, I have a mug for the show. [LAUGHTER]

John: Speaking of teas, [LAUGHTER] today’s teas are:… Todd, are you drinking tea?

Todd: Every time I’ve been on this program I’ve been drinking tea and I’ve had some pretty exotic brands and blends but, I’m sorry, today I have water, the lightest of all teas.

Rebecca: It’s a very light tea. I have English Tea Time tea.

John: I have Lady Grey today.

Rebecca: That’s a good choice.

Todd: Alright, I think I came in third place.

John: We’ve invited you here today to discuss classroom assessment techniques. The first two editions to this book were written by Thomas de Angelo and K. Patricia Cross. In last week’s podcast, Tom joined us to discuss the origin of this book, where you joined as a contributor for the current third edition of this classic book. What was your reaction when you were first asked to join in this project?

Todd: Well, this is actually kind of a funny one, because it goes back to Linda Nilson and Teaching at its Best. And Linda was retiring, and the publisher, Jossey-Bass wanted to do a new edition of Teaching at its Best, and Linda says “Why don’t you see if you get Todd to do it and I will work with him on it and do it.” And we did it. And we got the book done a little bit early and on schedule, and it came out very, very well. So there’s a little plug for that book there. And it must have been about two months later, I had called them and said: “If you find another project like this, I love working with people like this.” And a couple of months later, I got a phone call and they said “We have another project we’d like to potentially have you work on” and I said: “Great, what is it?”… and they said: “Have you ever heard of the book Classroom Assessment Techniques?” And I held it together, and I said, “Well, yeah, yes, I have.” And they said, “Well, we thought it’d be helpful given what you do and what Tom does and everything, is maybe the two of you could work together on this book.” And so he talked a little bit about what there was and everything, and I hung up the phone and did a little happy dance because this book was one of the most influential books in my entire career. But the way the universe works is I ended up doing this one because of the work with Linda Nilson’s. And so that’s how I got started on this.

Rebecca: That’s amazing.

John: And, I agree, that is an excellent book as well. So you’ve got two classic books that you’ve worked on recently.

Todd: It’s really surreal, you know, a first-generation college student and every once awhile is kind of like I just can’t believe this is happening. But it was a really good project.

Rebecca: When did you first start using classroom assessment techniques in your classes? You mentioned that this book was influential.

Todd: I taught my very first course in 1987-1988. This book came out, second edition, 1993. And I was using it pretty much the moment it came out. Somebody had pointed it out to me and I didn’t know anything about classroom assessment techniques, and then I started reading through it and picking some out. I thought it was genius. Much like periodically, what you’ll do is you look at something and think this is ridiculous that nobody started doing this before. In some respects, it’s not that hard, right? You ask people questions, and you find out if they’re learning. It just took Tom and Pat to come along and say, “Why don’t we come up with strategies to find out if they’re actually learning.” So I think it was really influential for me because it kind of struck that chord with me that I always believed we should be finding out how much students were learning instead of focusing on teaching. But that’s not what was going on at the time, everybody was focused on teaching-centered educational practices.

John: What do you find most valuable about these classroom assessment techniques?

Todd: It’s interesting, because there’s a couple different levels. The one that’s really valuable, of course, is you find out if students are learning. Secondarily, you can find out if you’re teaching well, and you can change your teaching practices based on what they’re learning and based on what’s going on. But I’ll tell you the real reason I think this is really, really valuable is it addresses equity issues very, very cleanly. We walk into classrooms with all kinds of preconceived notions and stereotypes and implicit biases. And it’s really, really easy to think that a student’s struggling for a certain reason, unless you ask them. And so, for several of these strategies, I think it’s great because I’ve had students that have been practically sleeping through every class and then I started doing the classroom assessment techniques and he had some of the best responses, this one student did. And I would just totally blown away that this person was just processing it to such a level and I had a preconceived notion based on where the person sat and how they dressed and how they acted in class. I didn’t think they knew anything or they didn’t think they were learning and they really were. So I think it’s huge for equity and overcoming some of the biases we have in our classrooms.

Rebecca: That’s a really good example to underscore the importance of that. We had the opportunity to ask Tom what his favorite classroom assessment techniques are. So I’m really looking forward to asking you the same question: what are your favorites?

Todd: I’m one who says over and over again, in education, that we should always be careful about absolutes. Like, there’s never a strategy that always works. There’s never a type of student that’s always the best. Any situation can change. So there’s like 55 CATs in there. And I listened to the program with Tom and I heard him say that he didn’t have his favorites. And then he quickly picked out his favorites, and I’m sure it’s the same with his daughters. [LAUGHTER] So I have three girls, you know, they changed the favorites, I would definitely go with the standards of the muddiest point minute paper, those two are the big ones. Everybody’s done a minute paper, probably at some point. The muddiest point I like, which was the adaptation. And if you haven’t heard this show with Tom, make sure you listen to that, because he explains that very well. I think I like exam wrappers where you go in and kind of have the students talk about the exam experience. But for me, there’s several CATs that are based specifically on assessments. And it’s the thing that we almost never ask students for, and it’s the thing they can give us the best feedback on. So it’s like, when you have group activities, it’s an assessment on how well the students perceived that the activity worked for them. So instead of me just saying, we did a great group project today, you ask the students: “Was this a good group project?” There’s assessments in there about learning interests, what types of things you’re interested in, your study strategies, exam strategy. So you can ask students those things. And I think those make great classroom assessment techniques, to find out the processes that you’re doing, and how the students are receiving those.

Rebecca: Those are good opportunities to respond in the moment in your class, but also probably plan in the future too, to know what to do next time around, perhaps.

Todd: Yeah, I think specifically for exams. And one of the favorite questions I’ve always had on exams, because the students would get to a point where, and I got this, because of one of the assessment techniques I’d use early on, is students always were frustrated that there was something on the test that they studied really, really hard for, and it wasn’t on there. The problem was, it wasn’t always the same. So the students all had different things they studied. So because of collecting that information, and finding out from the students as they were frustrated, I started adding a new question at the end of every exam worth the same as some of the other questions there, was: “Please describe in detail something that you study that I didn’t ask you about on this test.” And students immediately felt better about the exams, they were happier with the test, because they didn’t spend four hours memorizing something that they perceived wasn’t valued on the test, even though the real reason for all of this stuff is life, not the test. But again, that was one from just asking the students: “What do you like and not like about the test?”

John: That’s a really great application of a technique that I hadn’t thought about, I can see how students would feel that their voices are being heard with that type of thing.

Rebecca: …or that their efforts were validated.

Todd: They got their points for their studying, and it just really seriously changed the affect of the class.

John: In the last podcast we did with you last summer, we talked a little bit about how teaching has changed from the time when you and I both started teaching, and this book was written back around the time when you and I were both relatively new at this. What role do you think this book has played in that evolution of instructional practices?

Todd: Well, first of all, I do want to acknowledge and help people just really understand the difference between 1993 and 2023. Some of the things we took out of the book was “display your results on an overhead projector,” there was no mention of the internet anywhere in that book, because the internet wasn’t here yet. Just stop and think about that for a second. So it was a very different time. A lot of heavy lectures. I mean, a lot of times people were lecturing, there were other activities, or other strategies, there was group work and role plays and those types of things. But primarily people are lecturing, because that’s just what people did, primarily. And so the role that this played, I would say, I don’t know if I’ve ever said this before. I think this book had a bigger influence on higher education than any other single book that’s out that I know of. There are some really, really good books out there. But this fundamentally changed the game. Because what it did was it started asking students about their learning, which means it started to really shift the focus from teaching to learning. This did happen to come along, by the way, if you really look at dates, the whole concept of really pushing active learning in higher education, at least in the United States, really took off in the mid- and late-1990s. So 1996 to 1998, we started seeing people like Richard Hake, do some huge studies about this. American Association of Higher Education was changing their themes from taking teaching seriously to taking learning seriously, all of those things happened about three to five years after this book came out. So I think this was the impetus for a huge, huge aspect of the whole push for moving from teaching to learning. I think it had that big of an impact.

Rebecca: In recent months ChatGPT has hit the news everywhere. We’ve heard all kinds of reports in higher education journals and things. accreditation organizations are talking about AI. And many faculty members are also talking about potential harms of AI or other see benefits. What do you think the impact of AI tools will be on higher education practices? …speaking of evolution… [LAUGHTER]

Todd: I don’t have the statistic right in front of me, but there was one I did a presentation just recently on how long it took to get to a million users. And they were talking about some things like Netflix and other things of taking two and a half years, and then six months and stuff. And I believe ChatGPT was five days. So it’s pretty incredible. So Rebecca, I love the question. I think this is a huge issue. The two biggest things that I heard out of the gate was: number one is how are we going to ever have students do their own work again? It was all academic integrity all over the place. And I couldn’t help but talk to a couple of people who were really freaking out about it and I said, “You should really read some of the research about academic integrity already and the proportion of students who are doing things that they shouldn’t be doing, so to speak, there.” And so that was it… academic integrity. And number two was like, “What kind of assignments should we do now? How do we structure assignments to be meaningful?” And I find this fascinating, because they weren’t talking about the learning process. And I still, to this very moment, I’m just starting to hear it now, but I have not heard much of anything about how can we use generative AI to influence the learning process in a positive way. And so I think that’s the huge thing we need to be looking at. So I guess the people who do know me, know me as the individual who writes a lot about helping students to learn and teaching tips in the classrooms kind of things. I think from a cognitive psychologist’s point of view, we can look at some of the foundational things like: we know that repetition is really good, we know that the process of explaining something really helps because it’s a chance for you to practice at retrieving information. We know spaced recall is really good. And there’s all these things out there that can kind of do that. But I’ll tell you, and I’m coming right back to the CATs, but ChatGPT and some of the other generative AI has some phenomenal opportunities. We know that teaching this is good. So I opened up by ChatGPT 4 the other day, and I was practicing and I typed in: I need to learn about metacognition, would it be okay if I taught you metacognition, and then you tell me how good of a teacher I am. And the chatGPT says, “Yeah, this will be fun. So let’s get started.” And I said, “Okay,” and so I explained metacognition. And ChatGPT asked me some questions like, “Well, what does this mean? And what does that mean? …and I acted just like I was teaching somebody. And then when I was all done, I said, “If I were a teacher, right now, how would you grade me?” And I did not get a good grade [LAUGHTER] which was kind of disconcerting there. But I said, “Why did I not get a good grade on this? And then it explained a section that I had completely forgotten, because I was doing this off the top of my head to try it. So I went back and explained that. And then ChatGPT says, “That’s much better.” I think that’s phenomenal. Another one, I will not get into details, but I typed in, or I gave it the prompt… it’s all about the prompts folks… “That I just learned in my introductory psychology class at a medium-sized, midwestern university. I’m a C-level student who’s worked really hard, and what I just learned were about persuasive techniques. Could we pretend we’re on a car lot, and I’m a salesperson, and you’re going to buy a new car, and you think you want to buy it today, but you’re not going to rush into anything.” And ChatGPT says, “This sounds like fun. Tell me about your best selling model.” [LAUGHTER] And I proceeded to explain all these things. And the scariest one is I said, “This Ford van right here has excellent ratings for safety.” And it said, “That’s particularly important for someone with a loved one and two furry friends.” So the learning possibilities are all over the place for this thing. And so for classroom assessment techniques, this is great, because we can actually teach students some strategies, how in the classroom, how to assess whether they’re learning. We can replicate these and have them do them with AI. So we could use AI to do things we couldn’t do before. But we can also do it to model things for students that they can do on their own. So I think there’s some really, really interesting possibilities coming along.

John: And one interesting one is if you ask ChatGPT for your biography, you may very well turn out to be a car dealer in the Midwest.

Todd: You know what, I’m just gonna jump in there and say that is totally true when ChatGPT was about six months old, because I did it. I said, “Write a letter of recommendation for Todd Zakrajsek… and you probably saw this, why you even mentioned it… and I think it had 19 things I counted in there and this term of hallucinations, but 17 of the 19 things were wrong, including where I went to school where I worked and everything else. That was when ChatGPT was a baby. I asked it the same thing just recently, and it was scary, how close it just nailed everything. But yeah, it was for a while.

John: It’s improved quite a bit, but one of the nice things about it is our students come in with really diverse backgrounds and we can provide supportive materials for students when they have some gaps in their prior learning. But, ChatGPT offers the possibility of having something that’s completely personalized to their needs, so that they can ask questions that are specific to the issues that they’re having trouble with, and not what we might have guessed they’d be having trouble with. So, it’s something I’ve encouraged my students to use in my intro economics course, and they appreciated the fact that they were encouraged to use it,

Todd: I think that individualized instruction is going to take on a whole new level. And so I think that’s going to go crazy. But I will say this, and I think it’s extremely important. I’ve been in higher education for 40 years, I have never seen a bigger possibility for an equity gap to be the widest chasm it could ever be. I think we got equity problems that are down the road. And if we don’t pay attention, it’s going to be awful, because I believe that generative AI is going to be a huge learning aid, and it’s going to help us with lots of things, for the people who have it available to them, they will skyrocket and be able to do stuff; the people who don’t have it available… reliable internet, a good computer, or some kind of a device, a safe place to sleep at night… those individuals are not going to be on that tract and they’re gonna get separated. Right now, there’s a couple of grade differences between people who are privileged and not in some classes. That could widen so much. So I think, everybody, we just got to really be careful about how to make sure that this was as equitable as possible for everybody.

Rebecca: Higher ed leadership at various colleges and universities are certainly dancing around the idea of AI and what kinds of policies and procedures to put into place. What would you advocate for? What would you encourage faculty members to advocate for to really make sure that equity is addressed?

Todd: I think one of the very first things we need to do is teach students how to use it properly. Because one thing we know about learning is, the more you know about something, the easier it is to learn something related to that topic, whatever the thing is, and what that means is that learning is not linear, it’s curvilinear. The more you know, the easier it is to learn something, which means you’re going to know more quickly, and as you know more quickly, you’re going to be able to learn more quickly. And this is why people struggle at the beginning with things and then all of a sudden they take off. So if some students know how to use the systems, and some don’t, that’s where we’re going to run into some huge issues. So I think one of the things we can do is teach students how to use it to help you, and when to be careful that it’s not going to help you. The movie Wall-E is a great thing to be keeping in mind. If you haven’t seen the movie Wall-E, it’s a plug for the movie Wall-E, it came out a long time ago, but that concept of the humans that had everything done for them ended up becoming just blobs that laid in chairs all the time, and I think our brains basically do that. If students use AI to do their work, they’re not going to develop critical thinking skills. If students use AI to help them to practice at retrieval and spaced out practice and do those things, they could become very good. So I think schools need to be careful about helping students to understand how this can really help them. That’s the first thing. Then there’s the obvious, helping people to have access to technology, we could have labs open, we could have laptops that we can loan to people, reliable internet, we could have rooms that we keep open 24 hours a day, keep them 24/7 there. Some libraries during exam time, they don’t close for like four or five days. We could have rooms in the library that people can come and have safe spaces to study and work, reliable transportation is going to be an issue, we’re just gonna have to work out how do we address those issues. It’s gonna be a challenge, but we need to be thinking about it.

John: The use of classroom assessment techniques has been growing steadily since they were first introduced. And each time there’s been new educational technology, it seems to have led to an increased use of those. During that time, we’ve seen the introduction of relatively low-cost computers, computer networks, the internet, mobile computers, and so forth. How do you think the availability of generative AI will affect the use and value of classroom assessment techniques.

Todd: I think there’s a couple things that’s going to really, really change. I think for classroom assessment techniques, number one is we’re going to be able to individualize a little bit more. So we can tailor… we don’t have to ask the same question of all the students, we don’t have to look for all the same type of responses. So we can think about it a little more creatively in how we can use it, almost like an individualized instruction toward classroom assessment. Probably the biggest thing, though, overall, is it’s going to allow us to just process data in numbers and levels we’ve never seen before, and particularly free responses. In the past, we’ve often ask closed-ended questions just because if you’ve got a class with 400 students, you can say: “On a scale of one to five, to what extent did today’s lecture help you to understand something?” And the students can pick a number, we could drop that into an excel sheet, or anything, and come up with a number very quickly. We can now say on a muddiest point, “What are you still struggling with?” Take 400 responses, dump them into a generative AI program and have it spit out five things within 30 seconds:” Here are the five things your students are struggling with.” So I think it’s going to allow us to do more qualitative types of things very quickly. And I totally get that this is not hardcore qualitative research with good analysis of the data. But for what I need in the classroom, I think we’re gonna be able to get those responses very quickly and in real time. So what that means is, we’re actually going to be able to ask something like a muddiest point at halfway through the class, and then I could have the students do a quick think-pair-share, while I analyze the information and a matter of three minutes later, they come out of the think-pair-share, to talk a little bit about what they talked about. And then I could say, I see that you’re still struggling with these concepts, so let’s revisit these things before they ever leave the room. That’s never been possible before.

John: When I’ve been teaching classes of two to 400 students, I’ve used the free response option and word cloud. But the word cloud is just highlighting individual words, the ability to do analysis in more detail is going to be incredible. And I think most providers of response software are working on introducing AI, and some of it is expected to be available fairly soon.

Todd: Oh, there’s some great stuff come check GPT and some terribly scary stuff at the same time. By the way before I forget this, because I have to tell you this because I have ADD, that’s one of my favorite phrases.ChatGPT 5,, one of the things I read about that’s going to be coming up fairly soon is the ability for it to launch his own AI’s as needed. That’s the one that we’ve been waiting for, and thinking, “Hmmm, that’ll be interesting.” But yeah, I think that the ability to read through and look through the information is just going to be a game changer. And online synchronous/asynchronous, face-to-face in all environments. We could look at stuff in asynchronous environments in ways we haven’t before as well. I could look at different ways that students are responding, I could ask classroom assessment techniques. In fact, they’re in the book. We have several of them… ways to use CATs in asynchronous environments, and AI is changing how we’re doing it.

Rebecca: Can you give an example of how that’s happening in asynchronous environments?

Todd: In terms of asynchronous environments, everybody likes to go to discussion boards, but quite frankly, discussion boards are boring. We could have students generate something like the script for a commercial. And after they do that, I could have the CAT that comes in to say, “To what extent was this helpful in the learning process for you?” And the student, as they’re developing this thing and as submitting it they could also submit, right along with it, their CAT that comes with it, and then I could be reading the CATs as they come in one by one right behind each one of them. And so the concept, there, would be kind of a real time CAT analysis that’s not waiting until there’s a whole group and then looking at the group. Because typically, when we think about these CATs, it’s like, the 400 students fill out a muddiest point and we analyze the 400 students. We don’t need to do that. In an asynchronous environment, we can have these CATs coming in and we can be analyzing them as they come in. And we could even add them to the previous ones, to group them if we wanted to. But the bigger one is, I can find out how the students are doing as they’re doing. So that’s probably one way to look at it.

Rebecca: One of the things that you mentioned in an in-person or a synchronous setting was being able to analyze results mid class. And that’s in part because a lot of our students have mobile devices and technology that they bring with them that they haven’t had before. Do you see other ways that these mobile devices are also changing the way instructors might implement CATs?

Todd: The way you just mentioned, though, I can’t think of many other than the concept of as an information gathering device. I will say, just because I happened to read this study just a little bit ago here, the devices are very interesting, in a sense that we’ve seen high schoolers tend to have the same amount of time they spent with their peers was very consistent until about 2012. And in 2012, the numbers started dropping off pretty quickly. And then COVID dropped it even faster, but the slope was already there. And the point here is that people are turning their attention to devices so much that they’re not talking to each other exhibited by when you see two people at a restaurant having dinner, and each one of them are on their device. There’s all kinds of psychology about how those things happen. But the overarching thing that’s the issue is that we’re having students at levels we’ve never seen before disengaged in the classroom, even when we’re trying to do CATs and engage. And part of that is because of the devices. So one of the things that’s kind of interesting about this is when you say: “Okay, everybody gets your devices out, and now we’re going to use them for a CAT.” And then we stick with that and how we use that data makes a big difference of whether they stick with it or go back to what they were doing. Strategically, it’s really helpful to use those devices to engage the students.

John: One concern with generative AI is many of the types of assessments we’ve used before can be answered very nicely by generative AI. What might we do to reduce the likelihood that students will use generative AI as a substitute for learning?

Todd: I think it’s really, really important to talk to students about the long-term implications. I know this is not going to be for everybody. I totally get that. But one thing I’d say to students is: “I don’t understand why anybody would run, why would anybody go jogging? You can just get in your car and get there faster.” So if you kind of pitch that to them, that idea is for most of the students, if there are any kind of health-related fields, or if they want to have a cardiac system later in life, exercise is just one of the most important things you can do for your body, by the way, in terms of that getting 150 minutes per week, and just higher respiration. If you take that away, it is bad for the human body. And so the response is the same thing is true of this. Just talking to the students. If you use generative AI to answer your homework problems, to develop the quizzes, to write a poem, and I’ve done all of these in workshops, where I’ve said to people, I can do this real quickly. I don’t care who’s in this room, I can ask you to write a short story. And I will crush you with generative AI. The problem is that, if I do that, I don’t learn how to think for myself. And so I think the biggest thing we can do for students, number one is to build community in the classroom, to number two, to tell them what this all means, and then number three, ask them, ask them, how it’s helping them, ask them how it’s hurting them, how they believe these things are working. And those are classroom assessment techniques, we can use these assessment techniques to find out to the extent that they’re doing this. While I’m on my little diatribe, we will never be at this spot again in our lives. We have just developed something that can totally short circuit cognitive processing and critical thinking. But we did it with people who have critical thinking skills and cognitive processing. So we developed something because earlier in our lives, we had to learn in the ways of joggers, we had to develop our systems. And now we can talk about the automobiles that would move as faster. The students coming along are going to step right into those automobiles, right into the AI, and if they’re never told that it’s important to go out jogging, they’re not going to develop the systems that are going to be needed later. And so again, just think about this for a second, we’ve developed a system using a system that may disappear if we’re not careful. And so I think that’s where we can change it. So it boils down to community, why we’re using it, and then assessment questions of how it can be used, and where it’s good and bad.

Rebecca: I think students really enjoy having conversations about AI and exploring how it’s useful and not. I have had activities in my class where we intentionally used AI to see what it was like and when it would be useful and also analyze where maybe they tried to use it and it was totally not helpful, and why it wasn’t. And they really appreciated those kinds of conversations and learned a lot from those.

Todd: I think the students do. And we could pick different spots in history. You go back to Socrates and his whole belief that if you write things down, it weakens the mind. It makes sense. If you have to learn it, you’re going to be much stronger than if you write it down. But I can’t imagine right now teaching without students writing things down. And Samuel Johnson came along several 100 years ago and said, “There’s ready availability of books. With books all around, why would we really need to teach this stuff, they can just go get a book.” I can’t imagine teaching without books right now. So these things that everybody got scared about, or thought that just going to change just became integrated. And when the internet came about, we talked about this a while back, in the sense that many of us were teaching before the internet actually showed up. But when it showed up the people who were teaching were freaked out. “How am I supposed to teach when the students can go and get anything they want off their computer?” And now I can’t imagine teaching without the internet. I believe five years from now people are going to say I can’t imagine teaching without writing things down, without using some kinds of print format, without using the internet, and without using generative AI. I don’t even know how I would teach without it. What that’s going to need is the same stuff we’ve done in the past, that’s how do you teach well with the Internet? How do you teach well with generative AI?

John: It’s an exciting time to be in the midst of all these changes. And it’s going to be interesting to see how we all answer those questions as we move forward.

Todd: It is, but before we wrap up, because I can tell that tone, [LAUGHTER] I want to put a general call out that there are several people who are saying: “I wish I could retire right now” or “This is a great time to retire.” The statement I really want to make is this is a hideous time to retire if you’re really good at what you’re doing. Because we have never needed humans with really good critical thinking skills as much as we do now, maybe we will later, but to date I don’t believe we ever have. And for some of the people who are saying “I just don’t want to teach anymore. This is awful…” we might need you more than we’ve ever needed you, so I don’t like this concept of the mass exodus of certain people.

Rebecca: It’s an important point.

John: One other thing I will add is Paul Samuelson once said, in describing the way in which models of the economy evolved, that “funeral by funeral, the science makes progress.” So there is the counter argument there too, that people who are very tied into the old ways of teaching do need to either adapt or perhaps they’ll be replaced by people who are more willing to try new alternatives.

Todd: Do I smell a little Kuhn in there? The revolution of science. Ah yes.

Rebecca: And we always wrap up by asking: “What’s next?”

Todd: I’m trying to keep busy. I’m kind of on a roll right now. And I want to tell you, I’ve been writing books and the only reason I’m unhappy about ChatGPT and gen-AI come along is people are going to assume that I’m using them so that I can crank stuff out. But thus far, I have not used any of this generative AI to write any of these last books that I’ve done, these five books in the last five years or something, and enjoyed working with Tom on this one. But I got another one coming out in about four months. I just turned in the final pages for that And it’s the Essentials of the New Science of Learning: the power of learning in harmony with your brain. That book should be out in a couple of months. And so the what’s next after that, I am working on a book right now for helping with neurodivergent learners. We’ve joked around in the past, I’ve got ADHD is about as bad as it gets. If it were a competition, I could come in probably close to first place, and also on the autism spectrum just a little bit too. And when I started sitting down with some of my colleagues who also have ADHD and autism spectrum, it occurred to me, we don’t talk to students enough about these things. And I think the challenge of that is that the students believe that they’re alone, and they’re not sure they can do things. And I was talking to a student fairly recently that said, “I now believe I have a shot, here you are with a PhD, you’ve written books, if you’ve got ADHD, as bad as you claim you do, maybe I can do something.” And then that’s when I really really was getting serious about we need to help folks out. And the topic I’ve been playing around with a little bit in a couple of workshops is The Ones Too Often Left Behind as the title I’m using. And it’s the students that aren’t built for the system that we developed. And so I think that we need to treat some folks out there a little differently. And I think we can really build up the pool of intelligent folks by helping to teach the people who just learn in a different way.

Rebecca: It’s really important work to have models and put models out in front of students because they need to see themselves in whatever discipline, field, etc., that they want to pursue. I’m excited to hear more about that.

John: And when will these books be out? You mentioned the timeframe for one of them.

Todd: The Essentials for the New Science of Learning should be out in September. And then the other one, if everything goes well, is probably looking at a February date that it would probably be available.

Rebecca: Well, it sounds like we’ll be talking to you soon then, Todd.

Todd: We may be. When you got ADHD pretty bad, you just can’t really predict when that books gonna be. But yeah, I’d love to chat with you when it does finally emerge.

John: And what are you going to do next week?

Todd: You know what? You gave me a great idea. I think we should do a book on procrastinating people with ADHD. [LAUGHTER]

John: It’s always great talking to you. And we’re looking forward to each of these projects coming to fruition and we will be talking to you about each of them, we’re hoping.

Todd: I appreciate the opportunity to come and chat with you. I think you two do a phenomenal job with your interviews and the programs that you pull together on really good topics and I just am honored to be one of the guests.

John: Thank you.

Rebecca: Yeah, thanks. We always enjoy talking to you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC] </p

346. Classroom Assessment Techniques

Posted in active learning, assessment, diversity, engagement, equity, feedback, first-generation, higher education, inclusion, learning, metacognition, pedagogy, teaching
June 19, 2024

Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, Thomas A. Angelo joins us to discuss the origin of these techniques and evidence concerning their efficacy.

Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993.

Show Notes

  • Cross, K. P., & Angelo, T. A. (1988). Classroom Assessment Techniques. A Handbook for Faculty. Jossey Bass Wiley.
  • Angelo, T. A., & Cross, K. P. (1993). 2nd. Ed. Classroom assessment techniques. Jossey Bass Wiley.
  • Angelo, T.A. and Zakrajsek, T. (2024). 3rd. Ed. Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers. Jossey Bass Wiley.
  • Schwartz, Charles (1991). “An Academic Adventure.” March 4. – A description of the origin of the minute paper by it’s originator.
  • Chizmar, J. F., & Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. Journal of Economic Education, 29(1), 3–10. (the article that John referenced as his first reference to this topic)
  • Mazur, E. (1997). Peer instruction: A user’s manual. Pearson
  • Mazur, E. (2014). “Confessions of a Converted Lecturer” – a recording of a presentation by Eric Mazur at SUNY Oswego, 5/19/14
  • Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses. John Wiley & Sons.
  • Pam Hook and Josie Roberts (2018). “The who, what, when, where, and why of SOLO taxonomy.”
  • Angelo, T.A. (1993). “A Teacher’s Dozen—Fourteen General Research-Based Principles for Improving Higher Learning.”
  • Pre-order for 3rd edition of Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers.

Transcript

John: Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, we discuss the origin of these techniques and evidence concerning their efficacy.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Thomas A. Angelo. Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993. Even if you’ve never heard of these books, you may have included some Classroom Assessment Techniques in your teaching toolkit. Welcome, Tom.

Tom: Good to be here.

John: We’re very happy to talk to you. Today’s teas are:… Tom, are you drinking tea by any chance?

Tom: No, I was drinking black coffee.

Rebecca: It’s one of our most popular forms of tea [LAUGHTER] as regular listeners know. Today have some English tea time, John. How about yourself?

John: …and I have an Earl Grey tea today.

Tom: Excellent.

Rebecca: We’ve invited you here today, Tom, to discuss the third edition of Classroom Assessment Techniques, to be published shortly. These techniques described in earlier editions of this book have been widely adopted by faculty in all disciplines at all different types of institutions. And a lot has changed in higher education since 1993. But this book is still the go to reference on classroom assessment techniques, which is pretty impressive [LAUGHTER] for a book that was published 30 years ago. Can you talk a little bit about the origin of this book?

Tom: Well, I think the origin of the book goes back to the fact that when I was a young teacher, I recognized pretty early that many of the things that I was doing didn’t work and didn’t work for my students, and that I had to accept that things that had worked very well for me as a student weren’t working for many of them. So I began to experiment with asking questions in a variety of different ways and getting students to jot down answers anonymously. And through that, I really learned more about teaching and learning, I think, than from anything else that I did. And one of the things that I learned was not to assume. So, classroom assessment techniques are things that I’m sure almost every teacher has done. But what we did in the 1980s. K. Patricia Cross and I, was to really formalize those. And we did that in order to create a resource for teachers so that they could find simple tools that would help them ask students the kinds of questions that would help students learn better, and teachers teach better, as they went along. So it really began at that time, and it was kind of an accidental start, in many ways. We were doing it as part of a big project, and what was then the assessment movement, and it just kind of took off.

John: Most of our listeners are probably quite familiar with classroom assessment techniques and have seen them at lots of workshops and have probably, as Rebecca noted before, used them in the past. Could you define what a classroom assessment technique is, for those listeners who are new to the concept?

Tom: Well, in a formal way, a classroom assessment technique is a formative assessment technique. It’s a way of gathering data to use to improve and inform practice, rather than to make judgments. So we have too, most of us, to give students tests and exams and assign papers, and grade their work, evaluate their work, for grades and marks. Formative assessment techniques, on the other hand, are tools that we use to gather data along the way, before those moments in which we grade them, or mark them, so that we can help students get ready and improve and succeed when they’re going to be graded and marked and so that we can, at the same time, find out where they are and how they’re doing in order to help them improve along the way. So they’re really ways of diagnosing, in a way, students’ learning.

Rebecca: One of the most popular classroom assessment techniques described in your book is the one minute paper. There have been many studies of the effectiveness of this technique and it’s been widely adopted in many disciplines. Why was this technique, in particular, been used so extensively?

Tom: Because it’s simple and easy to use, probably, and it takes almost no time. But seriously, the one minute paper, the minute paper was invented, to the extent these things are invented by Charles Schwartz, in the 1970s, at UC Berkeley, and he used it as both a way to take attendance (in his case, he was asking students to jot their names down), but more importantly, to find out whether students were learning from his physics lectures, in his huge physics classes, what he hoped they were learning. And that idea spread through UC Berkeley really fast. And so soon, there were dozens and hundreds of people using these two questions. And the two questions, famously, were: what are the two or three most important things you learned from today’s lecture, in his case, and what questions do you still have the end of the lecture? So students would jot these down on those strips of paper, remember, there was no internet and no computers, really, in classrooms in the 1970s. And then he would collect them, and read through a sample of the 1000 or so that he had every lecture. And from that sample, he would gather ideas about what students understood and didn’t from the lecture, and also what questions were most common. And then he would begin his next class meeting, by going over some of those points that students had had trouble with, or not understood, or missed them completely. Sometimes it happened that a 1000 students didn’t get one point that he thought was critical, and also answering one or two of the most important questions. And what he found was, he could do all of that in five minutes at the end of class, and about five minutes at the beginning of class. And I think it’s that efficiency that has made it so popular.

John: My first exposure to this was a study that was published in the Journal of Economic Education way back around the time of your last edition. I did try to find it, but I couldn’t find the citation for it, but what struck me was that the instructor had used it in one section of the course, had not used it in another section, and found that there was a very substantial difference in learning outcomes between the two groups, which he attributed to the one minute paper. And it’s been incredibly commonly used. But one of the things that really struck me is that at the time when you came out with this book, including the one-minute paper, and so many other classroom assessment techniques, most instruction at the time was the chalk and talk type where people were at a blackboard, maybe a whiteboard, if they were a little bit more advanced in the technology, where most assessment was done with high-stakes assessments. So I think your book was a fairly important factor in helping people shift to more formative assessments. Why have these techniques become so widely adopted?

Tom: Well, I think that one of the reasons why people have found classroom assessment techniques and other formative assessment techniques, there are other kinds, so useful and so important, I think there are several reasons. One is that the student body in the United States and across the world has become ever more diverse in the last 30 or 40 years. And it turns out that no matter who we are, as teachers, most of our students are going to be pretty different in terms of their experiences and their interests and their skills than we are. And that level of difference has probably grown over the years, at least our awareness of it has grown. So I think most teachers now realize that it’s very important to check in on students and see how they’re doing, before we get to the midterm or before [LAUGHTER] we get to the end of the course, and to check in regularly and not to assume that students are understanding our brilliantly clear presentations or the material that we give them, or the readings, or the problem sets, and to really check early on to see if it’s making sense to them, and what kind of sense it’s making to them. So I think that there’s much more realization of the diversity of human learning, and of our students, than there was 40 years ago or 30 years ago. And I also think that as teaching has evolved, especially as it’s become online, people have recognized that we need data. We can’t just depend on students volunteering that data, they often don’t know that they don’t understand. And that that data, used well, can help us and can help them.

Rebecca: I think one of the things that you’re talking about in terms of assumptions is reminding me of some of the conversations that we had early on in the pandemic, where instructors we’re talking about not being able to see the sea of faces to know whether or not students are learning because maybe students aren’t nodding along or providing some visual feedback, but classroom assessment techniques are really important because they allow opportunities to gather that data, which is probably far more informative than the head nodding.

Tom: I think there are maybe a couple of other things that are important about this that people discover. One is that by stopping to ask students to reflect on their learning and respond, you give them time to think. And you prompt them to think about their learning. And from my point of view, forever, the most important thing about doing this has really been to give students tools to become more aware, we could say more metacognitively aware, of their own learning, and to better take responsibility and control or at least direction over their own learning. So what I found is I’ve had many, many, many very bright students who had never been taught any kinds of techniques to think, or to ask themselves questions, or to monitor their own attention, and having those sorts of tools, those sort of metacognitive tools, I think empowers students to make decisions about their own learning, and also to improve their own learning in those areas where it matters most to them, which may not be my course.

John: One of the things you mentioned was that the one minute paper was originally developed in a class with 1000 or so students. And one of the nice things about many of these techniques is that they can scale that way. Now, there was a time when the book came out when this was all done on paper. But now we have so many electronic tools that we can use to do this. Instructors could put up a QR code with a link to a Google form or something where students can give their feedback within a minute or so or we could use polling techniques, and so forth. And many polling systems have built in one-minute questions or some other classroom assessment technique as a default strategy. So this impact has been pretty substantial. And it’s really nice for instructors, not just for students, having that time for reflection, but also for instructors to get some feedback on what students have learned. Because if you just call on students, you may happen to get the only student who understands the material responding to a question or you might get the only student who hasn’t understood the material. But with these formative assessments, you can get a good feel for where students are. And they also get some feel for where they are in a way that they might not if they would just passively sitting there listening to a lecture. So these techniques have, I think, become increasingly more common from the time of introduction. So these techniques have held up pretty well. But you now have a new edition of this book, a third edition of this book. Could you describe some of the changes in the new addition?

Tom: Well, obviously, and you’ve touched on this, technology has changed. And that’s had an impact on teaching, and an impact on learning. So one of the things that I did, in going back to do this book after so many years, was to take a look at first what was out there, what had people published on this. And there are literally about 3000 published articles of various kinds and various quality, about classroom assessment techniques. There were about 300 that I thought were of sufficient kind of quality and rigor to be able to use. And in the chapter on the minute paper, I refer to 150 articles and studied them very carefully. There are many, many books that refer to the classroom assessment techniques. And I also read [LAUGHTER] all of those references. So I took a few years to work through this material and new things were coming in all the time. So one thing that differs is that we have an evidence base across many, many disciplines for what seems to work and what seems to work well. And also much useful information on how to use these with various technologies. So how best to use formative assessment with online, fully online technologies on totally asynchronous classes. And so that’s been really valuable. And those are experiences that I couldn’t have all of those experiences. So it’s terrific to draw in those from other teachers. The other thing is people have invented new techniques or developed new techniques over time. And so in the third edition 25%, or about 15 of the 55 techniques, are ones that didn’t exist in the second edition. And that’s about one quarter of all the techniques in the new book. So one of the reasons why we have different techniques in the 3rd edition is that some of the ones in the second edition weren’t much written about, weren’t much commented on, and they seem not to be much used as far as I could tell. So I’ve picked up the ones that were most used and most written about, and most referenced so that people in different disciplines can find those techniques. And I’ve also found another 15 techniques to kind of fill those gaps and extend it a bit.

Rebecca: Your book offers many, many assessment techniques. I’m curious what your favorite few are that you’d like to introduce instructors to?

Tom: Well, just like I can’t say which of my daughters are my favorite, I do have favorites. I’ll be honest. I have always used the minute paper ever since I learned about it all those years ago, and I learned about it really when I was at Berkeley, in the mid 1980s. And I learned about it, as everybody else there did, kind of by chance, and thought, “Wow, why didn’t I do this? Why didn’t I think of this?” So the minute paper, the muddiest point, which is a variant of the minute paper, which is developed by a really famous and eminent statistician at Harvard, named Frederick Mosteller. And Fred Mosteller heard Pat Cross talk about this, when we were both at Harvard, and being the kind of person that he was he immediately experimented with it. What he found was that it was more useful for him, in his courses, to just ask one question, and that question for him was “What was the muddiest point In today’s class? What was least clear to you?” Many other people have used that as well and written about that. That’s another favorite. A third one is one that I’ve used a lot, being a statistics teacher. And that’s predict, observe, explain, which is often called POE. And that’s one that I’ve found useful in many situations. And I’ve had colleagues who’ve used that even in literature courses, and other courses where they’ll say to students, “let’s stop at this point in the book or the film, and predict what’s going to happen next, write that down, then we’ll observe it, and then we’ll try to explain why that either met our predictions or didn’t meet our prediction.” And while it’s obvious how people do that in science, and it’s part of the scientific method, it can be used in all kinds of circumstances. And it’s one that I’ve used with teachers who I’ve worked with, about classroom assessment and about formative assessment. I’ve asked them to predict what they thought would happen, then observe it, and explain what did happen and whether or not that met their expectations. And that’s been a very useful tool for me, I think, and the teachers I’ve worked with. So those are three good ones. I’ll give you one more, if that’s helpful. And that’s something that we call the directed paraphrase. So three of those four were in the first and second versions of the book, predict, observe, explain is in the third edition, but directed paraphrase simply asks the questions that are sometimes called journalistic questions, who, what, when, where, why, who did what to whom, when, where, why, and how? And it asks students to answer those questions about some phenomenon that they’re observing, or to answer them about an essay that they want to write, answer them about any kind of question. So giving students a framework of questions that they can work through, to make sure that they really understood either a book that they’ve been reading or an experiment that they’ve been doing, has been really valuable to my students. And I’ve said to them, “Look, this is something that you can adapt and take with you anywhere to use to make sure that you understand what’s going on fully. And to find out which part of it maybe you really can’t answer, and need to do some more work on.

John: In terms of the predict, observe ,and explain thing. I took a MOOC that was put together by Dan Ariely on behavioral economics. And one of the things he talked about was how he would often present the results of an experiment where the results were not what most people would have expected. But then he’d asked people for their reactions only after he summarized the results of the experiment. And they all nodded and said, “Well, that’s exactly what they would have thought would have happened.” So he started revising his presentations, where he asked people to make a prediction, and then revealed the result. And it got people much more engaged. And the same type of thing is done by Eric Mazur with his polling technique, where he asked people to answer a question where the question is not obvious and where there’s going to be some disputes. And then once they’ve committed to an answer, they work with other people and he goes through it again, with a little bit of peer instruction. But the level of engagement when students have made some type of a prediction of what they think the answer is, and then they have an explanation of what the answer is either given by the instructor or by other students, the level of engagement and interest just seems to go up fairly dramatically. So I can see how that would be particularly effective.

Tom: Yes, and I’ve learned a lot from the work of Eric Mazur, as have many others. And I think that that’s a critical thing. Because one of the things that we need to help students with, and help ourselves with, is managing our inbuilt biases and that issue of when you hear something new, feeling that you’ve always known it, even though you haven’t, is a cognitive bias that’s common, I think, to all human beings, myself included. So doing something like that: predict, observe, explain routine gives us a chance to really check and see if we know the answer. And then to find the answer, or discover an answer. And then to realize, because we have a document, we have a record, we have data about what we knew in the first instance, to recognize that we’ve learned something and what we’ve learned and that we might have been misinformed or unclear in the beginning. And I’ve suggested to my colleagues, one of the reasons that students don’t value their education as much as sometimes we think they should, or value our course as much as they think they should, is because when they get to the end, they may think “I always knew that, that was obvious, I knew those things.” And so we can help students realize that there’s a start point. And then there are many points along the road to learning, and that they may have learned more than they thought they did in a given course or given program. And I think that’s important for us in terms of demonstrating the value of education. And I think it’s important for students in terms of understanding and developing some humility about how it is that we learn and how hard it is and how long it takes to learn anything that’s really valuable.

Rebecca: I think, in my experience, students really appreciate the opportunity to slow down and reflect on their learning, because they’re often not given that opportunity. And when you’re doing that throughout an entire semester, they really do get excited about how much they’re learning and they get energized by that and seeing that they can achieve things over and over and over again.

Tom: I agree, and what we found with most of the studies of classroom assessment techniques, the simple ones is: first, classroom assessment techniques can’t really be separated from everything else that’s going on in teaching, they’re just one little part of the whole teaching and learning spectrum. They, at best, seem to make about a 5- 8% difference in how much students learn, which is about half a grade. And I think that’s important, but I don’t want to overstate it. So they’re not a panacea and they’re not a silver bullet, but they’re something that can lift up, to some degree, students’ performance and their learning. And the last thing I’ll say is nothing works for everybody. And so there is no technique or no series of techniques that every single teacher or every single student that I’ve worked with found valuable. And so one of the points of having 55 techniques is to hope that there’s one or two or five in there that people will actually find that work for them.

Rebecca: So you just reminded us about there’s 55 techniques, how do you imagine an instructor using your book? Is it a read from the beginning to the end? Do you have it organized in ways that instructors can easily dip in and out?

Tom: I can’t imagine reading it from the beginning to the end, and I don’t think anyone ever has. That would surprise me, except maybe the proofreaders [LAUGHTER] and they weren’t worrying about what it meant. So it’s organized so that there are many ways, there are actually eight ways to find a technique that might be useful for a given situation or a given teacher. One of those ways is the simplest and that’s alphabetical. So if someone has mentioned a technique to you, you can easily find it that way. The others are a bit more complicated, but I think the most valuable for many people is an index of all the techniques that have been used by discipline and documented by discipline, and the examples by discipline in the book. And there are a couple of 100 examples in the book of how people use these in different disciplines, kind of from A to Z, from anthropology to zoology, and everything in between. So those are ways that many people use. There are also a couple of inventories in the book. One is something called the teaching goals inventory, which was in the first and second edition, and has been used in lots of ways by many people and written about and research has been done on the teaching goals inventory, and it is exactly what it sounds like, an inventory of what we think we’re trying to teach. For this book, I created what I think is kind of a more contemporary take on that, and that’s something that I call the course learning outcomes inventory, or CLOI, and it’s really flipping that and saying, okay, it’s important what we want to teach, but let’s look at it from the point of view of what do we want students to learn? And so the CLOI, the course learning outcomes inventory, the statements are statements of learning outcomes. There are 57, they’re divided up into kinds of learning outcomes, and people can use it as a self assessment. Those items both in the teaching goals inventory and the CLOI are all linked to classroom assessment techniques in an index. So if you pick number 47 goal or number 47 outcome, you can find techniques that are linked to that. Lastly, we’ve done indexing by Bloom’s Taxonomy, and by Fink’s taxonomy of significant learning outcomes, and also by the big solo taxonomy, which is one that’s much used in the Commonwealth world, in the British and Canadian and Australian and New Zealand systems. So there are eight ways to skin a CAT in this book, or to find a cat at least. And I hope those will be useful to folks. And I think different kinds of indexing work for different kinds of people.

John: You mentioned that when this book first came out these things were done on paper. But one of the things that’s grown quite a bit over the years, as you’ve mentioned, is online instruction. Could you tell us a little bit about how some of the techniques can be used in an asynchronous environment?

Tom: Well, I think there are huge advantages to the technology that we have now compared to having to do all this on paper, having to read all those papers after class, or all those index cards, in my case, after class and then waited until the next class, when I saw students again, the time lag was a real problem. So being able to respond almost immediately, or in relatively short time to students in asynchronous courses, to their questions, and to their confusions, and to also let them know that they’re learning some things really well… they really got that… and that they’ve come up with some ideas that I never would have thought of that are excellent. To do those things in real time, I think, is much more effective in terms of teaching and learning than waiting two or three days or a week. Also, as you mentioned, we have technologies for polling students and for analyzing their responses that are ever more sophisticated and powerful. And so those save teachers a lot of time and work. I was always going through things counting and deciding on the number of times people had mentioned things on paper. This can all be done and is all done for you if you want to do it. And that, again, allows teachers to focus on what really matters. And that is: what’s the meaning of this data? What are students telling me? …and to make decisions about when they’re going to respond and what they’re going to respond to, because you can’t respond to everything in data if you have more than two students. And most of us did, and do. So I think that asynchronous learning has huge advantages, it has some downsides, but it has huge advantages. And one of those probably is that it makes it more difficult to assume that by looking at students, you can tell that they’re learning or not, which I’ve always found an odd idea, but many of my colleagues at least used to seem to think that if students were nodding and looking at them, that they were just tracking with them and understood everything that they were saying. And I used to say to my colleagues, “I’ve known many smilers and nodders that probably were thinking of something else entirely, and classroom assessment proved that to me.

John: I’ve seen that so much. We’ve heard so many faculty during workshops over the years who say, “Well, I can tell whether they’re learning just by looking at the expressions on their faces.” And I’ve never found that. There are a lot of students who will be nodding and smiling. And then when you ask them a question, or they ask you a question, it’s really clear that they don’t have the basic idea. At an economics conference, I was presenting a paper and there was someone sitting in the first row taking furious notes while I was talking. So I went up to the person after and asked them what they were taking notes about, because I was curious. And they were actually preparing for a session later in the day. But they looked really engaged throughout the whole thing.

Tom: They were engaged, just not with your lecture. [LAUGHTER]

John: Yes.

Tom: So I think the converse is also true. And that is their students, and people, who don’t appear to be paying any attention at all, or who are frowning or who have had their head down, who are actually paying total attention and learning a lot. And that’s one of the things I learned as well, early in my career, was not to make assumptions based on those sort of superficial impressions. And it’s hard not to, but when you actually learn that some students who appear not to be engaged are engaged in their quiet way. And that’s a humbling lesson, or it was for me.

Rebecca: One of the things that I really like about your book, and also the way the collection of the CATS all in one place is the flexibility, the flexibility of navigate the book, the flexibility of options for the instructor, but also the flexibility of options for learners as well, in different contexts. So I just wanted to highlight that flexibility piece that we haven’t really talked about quite yet.

Tom: Well, that’s the hope. And if this is created as an ebook, and I expect it will be, then I think that will offer people even more flexibility and more sort of usability. It will be a more user friendly book. And again, there are huge advantages to the kinds of technologies that we have available now that we didn’t then. So I think the real key to this is one of the axioms in the book. And that is it’s really about adapting, not adopting. So there are many, many tools in there that people have adapted in a variety of ways, renamed, and that’s all fine. And that’s all exactly what should happen. You should find them, adapt them to their uses, and if they don’t like the names of the CATs, they can change the name.

John: Would this be a good subject for a faculty reading group where people from different disciplines could get together and talk about how either they’ve applied these things or how they might apply them in their classes?

Tom: Well, it has been from the beginning. And we began this whole process working with groups of teachers, initially in community colleges and state colleges, who were interested in assessing student learning. And, of course, the point being trying to improve student learning. And what we did basically was use the teaching goals inventory at that time, you could use the course learning outcomes inventory, with those teachers to get them to focus on what they really wanted to know. And in the book, I’ve written about, in one of the chapters, four different ways you can approach using classroom assessment techniques and figuring out which one to use. But everybody had ideas and theories and questions or problems that they wanted to try to solve. And so we started with that. And then we said, if that’s your question or problem or theory or outcome or goal, what might be a good tool to try to help you gather a reasonable amount, a manageable amount, of data from students that might help you learn, and might help them learn? And that’s really where we began, and those discussions and the collegiality that came from them, and the interactions that came from that, were always the most valuable part of that professional development. That when we ask people: “What did you take away from this?” The first thing that they said was not a bunch of techniques. The first thing that they said was the interactions with my colleagues, the relationships with my colleagues, learning from my colleagues, and then they talked about the actual content that they had learned and the things that they had done. So I think those interactions and those relationships that people can make, by having a group focused around this. This is an excuse to get together and talk about teaching and learning. I think from our experience, we found any excuse to get folks together to talk about teaching and learning, especially if it’s across disciplines, is always fruitful.

Tom: …particularly if there’s food. [LAUGHTER]

John: Yes. The third edition list you as author with Todd Zakrajsek, could you tell us how this collaboration came about and how it worked?

Tom: I think this was, in part, an insurance policy for the publishers. Given my advanced age, they wanted to make sure that they had someone involved in the project, who could take over if I just kicked the bucket. So, in the last six months of the book project, after I had finished writing the manuscript, Todd Zakrajsek joined with me and was really invaluable in finishing the process, in coming up with great ideas and suggesting great ideas, especially for the use of technology and AI with CATs, and in helping me manage the process with the publisher, which is much more complicated than it was 30 years ago. He also wrote the discussion questions for use by teachers and faculty developers, and I think those are going to be very helpful to folks.

John: We’re recording this in May, and the book is scheduled for release in June. We always end by asking, what’s next?

Tom: Well, I’ll answer that. But I’ll first say to teachers, what I always say, and that is, “Feel free to ask your librarian to order the book, so that you can use it, and your colleagues can use it.” Don’t tell Wiley and Jossey Bass that I said that, but I think that’s important. So what’s next for me is I’ve been working for a while with a group of people in, strangely enough, pharmacy education. I’m not a pharmacist, and I’m doing research with graduate students on what we call core concepts in developing what are the core concepts in pharmacotherapy? And that’s been very interesting for me. I’ve learned a lot from that. And so I’m ongoing with that. And we’re beginning to work on educational materials related to that. So that’s one of the things that I’m doing. And I’ve actually, I don’t want to commit to this too much, but I’m thinking about writing a book based on an article that I wrote that people have found useful, called “A Teacher’s Dozen.” And so I’ve updated that. I’ve been working on that actually for a few years, and something may come of that.

Rebecca: Well, we’ll be excited to find out what that is soon, maybe. But you haven’t committed, but maybe. [LAUGHTER]

John: Well, thank you. It’s been great talking to you. And my copy of the book is on preorder. And I am looking forward to receiving that.

Tom: So am I. [LAUGHTER]

Rebecca: Well, thank you so much for joining us. It was great to hear about the origin story of the book and how it’s evolved over time and I know a whole other generation of teachers will appreciate the new edition as well as teachers who have been practicing for a long time.

Tom: I hope it will be useful. Listen, it’s been a pleasure and a privilege to talk with you both. Thank you for giving me this opportunity

John: We should note that this is the first of two podcast episodes discussing this book. In next week’s podcast we’ll be joined by Todd Zakrajsek, who will be discussing his role in this work.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

345. New Era – New Urgency

Posted in authentic learning, engagement, grading, higher education, learning, pedagogy
June 12, 2024

Public confidence in the U.S. educational system has been declining while reports of student disengagement have been rising. In this episode, Deborah Pomeroy and F. Joseph Merlino join us to discuss the possibility of repurposing our educational system to better support the needs of our students and our society.

Deborah has over 50 years of education experience and is professor emeritus at Arcadia University. She has co-directed a Dewitt-Wallace grant, Students at the Center, for inner-city schools in Philadelphia and was actively engaged in the Bioko Biodiversity program in Equatorial Guinea. Joe Merlino has spent 39 years in education. He has been a principal or co-principal investigator and/or project director on numerous federal grants. He currently directs a seven-year USAID grant in Egypt where a team of US faculty are co-developing 180 new undergraduate STEM teacher-preparation courses for five large Egyptian universities.

Deborah and Joe are co-founders of The 21st Century Partnership for STEM Education where Joe has served as president since its founding in 2007. They are also the co-authors of New Era – New Urgency: The Case for Repurposing Education, which was recently released by Lexington Books.

Show Notes

Transcript

John: Public confidence in the U.S. educational system has been declining while reports of student disengagement have been rising. In this episode, we discuss the possibility of repurposing our educational system to better support the needs of our students and our society.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Deborah Pomeroy and F. Joseph Merlino. Deborah has over 50 years of education experience and is professor emeritus at Arcadia University. She has co-directed a Dewitt-Wallace grant, Students at the Center, for inner-city schools in Philadelphia and was actively engaged in the Bioko Biodiversity program in Equatorial Guinea. Joe Merlino has spent 39 years in education. He has been a principal or co-principal investigator and/or project director on numerous federal grants. He currently directs a seven-year USAID grant in Egypt where a team of US faculty are co-developing 180 new undergraduate STEM teacher-preparation courses for five large Egyptian universities.

Deborah and Joe are co-founders of The 21st Century Partnership for STEM Education where Joe has served as president since its founding in 2007. They are also the co-authors of New Era – New Urgency: The Case for Repurposing Education, which was recently released by Lexington Books.

Welcome, Deborah and Joe.

Deborah: It’s great to be here. Thank you so much.

Joe: Thank you.

Rebecca: Today’s teas are:… Deborah, are you drinking tea today?

Deborah: I am. I have a cup of chai right in front of me.

Rebecca: Perfect. How about you, Joe?

Joe: Yes, this is English breakfast tea.

Rebecca: Oh, yum.

John: And I have a black raspberry green tea today.

Rebecca: Nice. We got green tea in the office today. I have a raspberry Jasmine green tea today.

John: We’ve invited you here today to talk about your new book. Could you tell us a little bit about the origin of New Era – New Urgency: The case for repurposing education?

Deborah: Well, that’s an interesting story. We actually started the book over 14 years ago when we were working on the math-science partnership for Greater Philadelphia. And it was the culmination of work that both Joe and I had been doing, both independently and collaboratively, in education reform. And we were beginning to develop our ideas about education reform. And we’ll go into those in more detail later. But we started the book, and we actually got quite a bit of the book written. And then we were interrupted by a meeting with the Minister of Education from Egypt, who came to Philadelphia with a group of colleagues to study STEM schools in the US. And as a result of that meeting, we ended up working on this USAID project in Egypt, helping the ministry develop a series of model STEM schools. And when we got into the work there, which was basically 24/7, 365 days, and we were joking around that many times, it felt like we were drinking water from a firehose, it was so intensive. And we were never able to go back and finish our book. But as we were doing the work, we sort of looked at each other a number of times, sort of laughing and saying, “This is the last chapter of our book,” because in the last chapter, we hit envisioned what real transformation could be in education, and we were actually doing it and it was so exciting. And so we decided that we needed to recast the first part of the book, and then add this whole latter part of it, because we actually have a case where our ideas have been able to be put into place and have worked just amazingly well. And so the book is actually a total of 14 years in the making.

Rebecca: So I’m really curious about the word choice of repurposing because you could have probably picked many, many titles. Can you talk a little bit about why repurposing is your word choice in the title?

Joe: We felt, through our reform efforts that were difficult to implement, that the single most durable element that prevented reform from happening was the inability of teachers and students to articulate why they were teaching what they were teaching, as opposed to all the other possibilities of what you could teach per subject. So why are you teaching algebra? Why not statistics or probability? Why are you teaching it the way you’re teaching? So the idea of purpose, which is answering the why, was what we found to be lacking. That’s why we felt purpose was the central idea of the book.

Rebecca: I knew there was a key reason.

Deborah: When we started work in Egypt, the first questions that we asked them were: What are your aspirations for the future? And for the outcome of these schools? In other words, what do you dream these graduates could do and become, and when they were able to start to articulate that, and we’ll go into that in more detail later, then we built the entire curriculum and assessment system around that purpose. And so we found that that, in fact, was the linchpin for this transformative reform.

Joe: After our meeting in August of 2011. This is like a year after the Arab Spring in Egypt, that’s when the Egyptian delegation came to Philadelphia. Four months later, we found ourselves in Cairo, I did, and two other colleagues of mine. And so we’re sitting in the first school, which is outside of Giza, and it’s in the middle of nowhere. I mean, it’s 30 miles, it’s just dirt and dust in this building. And we’re sitting in the principal’s office, and looking around wondering what’s happening and in bursts the first deputy minister of education in Egypt, and he says, “Welcome to the dream.” [LAUGHTER] And we’re looking around, and we’re saying “there’s nothing here.” So we had the opportunity to rethink things, and for them to rethink things. And I’ll just say one other thing real quick, is that the purposes of education derives from your aspirations for your country or for your community, and that defines what your purpose is. And that’s what we did in Egypt.

John: And you begin the book by talking about how the purpose of education has evolved over time with a little bit of lag, because educational systems tend to adjust slowly. Could you describe how educational systems have responded to changing societal needs?

Joe: Okay, I’ll try to do this in 60 seconds, if I can, the 400 years. [LAUGHTER] For me, it was a tremendous learning experience. We have, in our time, the ability to have the internet, to have digitized libraries, and search engines. So you can go back and actually look at primary sources. So it became a real learning experience for me as I went through it. But we started with the idea that we’ve been in this business for a while now, from the early 80s to now, and we could see the development of the Nation at Risk archetype that has defined our era here. And we wondered about, are there other archetypes? Have there been other paradigms? Have there been other purposes of education other than, right now, preparing kids for the global economy, which was the idea of the Nation at Risk? So we started by looking at the earliest instances of purposes, and we found it was the religious schools in New England, the Calvinist idea of preparing kids to be a part of a religious community, salvation was the key. And then we looked at, “Well, why did that happen? Why was it salvation?” So we looked at the English history and why people came to the United States was to form a new Israel. That was the vision of the Puritans. And then we looked at how this may have changed over a single lifetime. We talk about in our lifetime, how much change there has been if you think about yourselves and your grandparents. So we looked at history through the lens of five lifetimes, laid end to end. We said, “Well, what happened during these lifetimes?” And you see a tremendous amount of change that happened, that’s what we described in each lifetime, but along with those changes have been associated changes in the purposes of education. So I’ll just give you a quick example. So from this idea of the Commonwealth of Massachusetts, from this idea of a religious community, came the idea that as more immigrants came in, there was different religious sects. And there was conflict between these sects, so much so that when William Penn came to Philadelphia in 1682, he brought the Quaker ethos, the idea of the Quaker schools which was based on religious tolerance, and peace. And so it became another purpose of education. And he founded the first Quaker school here in Philadelphia. So that’s an example. Another example is the industrial revolution. There couldn’t be industrial education if there wasn’t an industrial revolution. So that’s what we did.

Rebecca: It’s really done a good job of setting the stage of how change has occurred over time. Can you talk a little bit about how the current system aligns or does not align with our current societal needs?

Deborah: Right now, the paradigm that is pretty much driving education in the US is college and career success. And many schools that I’ve worked in, for instance, have that plus perhaps lifelong learning as part of their school mission and vision. But how does that define the curriculum as such, and there’s really a mismatch, and what would a curriculum look like if we’re really focused on lifelong learning? And while the college and career success is driven so much by standards and by test scores that really are quite empty when it comes to looking at what we need as a society going forward. And going forward, we need people who can collaborate together, we need people who are very well versed in assessing the truthfulness of information and resources, we need people who can listen, we need people who are critical thinkers in terms of being able to deconstruct problems and analyze them and look at their resources and so forth. We need people who know how to recover from a failure and actually see failure as a source of learning and from which positive things can come. Those qualities are not served by high test scores in math and reading and by a curriculum which is so focused on teaching a course to prepare students for the next course. Where does life come in? …students who understand citizenship, who understand the civics of our country, and the government, and our system of checks and balances within our nation and what it really means to be a responsible voter. And on top of that, our students are in a culture in which they are dealing with the effects of pandemics, they’re in a media culture, they’re in a culture of diversity, because our kids below the age of 18 are more brown than white, that demographic is increasing in the country and we’re going to become more and more diverse. We’re dealing already with major effects of climate change. On a more personal level, the kids are dealing with bullying and Infowars, drugs, gun violence, immigration, you name it, and where on earth does the current curriculum prepare students to deal with the culture that they are facing in their own lives and what we need as a country going forward?

John: And I think everything you’ve said are things that everyone would agree need to be addressed. And there have been lots of attempts to bring these things into the school system in terms of an emphasis on lifelong learning, critical thinking, an ability to interpret information, and yet we still see the same basic curriculum being used. Why have previous efforts at reform failed?

Joe: That was the basis of the book. That’s why we wrote the book, because we’ve been doing reform efforts for decades. But we’ve been operating within this bubble of this Nation at Risk archetype, and it’s not relevant to where students are and where our country is. So the curriculum has been set, and then there’s justifications for it, rather than starting at a blank slate and saying: What’s our aspirations? What do you want to do? and then working backwards and deconstructing the subjects to fit that purpose? Like, I’m a math teacher, or I used to be a math teacher. I’m a recovering math teacher, [LAUGHTER] but I also was a philosopher and a cognitive developmental psychology, etc. There are so many different subjects and areas of interest that could be taught, not just in math, but in everything. So why this? Why this curriculum? So when we redesigned the STEM teacher preparation program in Egypt, we had to design an entirely new program. So I went to Cal Poly who was one of our partners, and we had a design session with them. And so it was a group of people at Cal Poly. And I said to them, “we’re going to design a new teacher preparation program.” And I said, “Imagine a teacher preparation program with no courses. What would it look like?” And they were stunned at the question, but eventually, they came around and after three hours, they started to think about it. And they actually did come up with courses, but it was a whole different program than they would have done otherwise. And so I would challenge people to say, what if in high school, there was no college as there was in the first half of the 20th century, it was mostly just high school, there was no college. So what would high school look like? Why would it be there? So the impediment to reform when you get right down to it is the lack of a coordinated coherent philosophy of a school that can inform the curriculum choices that are made. And that this curriculum, that biology talks to chemistry and math talks to each, the humanities are included as a central idea of a school because one of the big problems of people is getting along with each other. So you need history, you need literature, you need to understand human behavior and what triggers you. This is absent, you’re not going to get that from precalculus. And I’m not saying that precalculus isn’t important. But Latin teachers would argue that Latin is important for different reasons. So this is why it comes down to what’s your vision for the kind of life you want to live and the kind of world you want to live in? And then let’s package that in a way that relates and is integral to a child’s life.

Deborah: So, in schools today, many teachers do try to include projects and problem solving with their kids. And we know that that’s very much a part of the pedagogy in many schools today. But the problem solving and the projects are, may be related to the subject matter, and may be of interest to the kids, and maybe not. But the way we did it in Egypt is the schools were designed around the premise that the graduates would be prepared to address the grand challenges of Egypt. And we have grand challenges like climate change, alternative energy, population growth, urban congestion, disease prevention, and so forth and so on. And so all of the courses are designed to provide students with the concepts and skills necessary to address these grand challenges. And so the projects that the students do are projects that are directly related to these grand challenges. And these challenges are things that the kids see every single day when they look out the windows or on their way to school. There are problems that are so pervasive that everyone in Egypt [LAUGHTER] looks around and says “Can’t we solve these problems.” And so what we’re doing is we told the kids, this is what we’re preparing you to do. And so when the kids undertake a project, or work on problems, they’re real, and they’re urgent, and the kids are empowered to think about the fact that they could become agents of change in their country. And that is so different from the kind of projects that we do in our schools today. And when I talk about all the challenges that we face in our society, and the kids are facing in their lives every day, those could be some of the kinds of challenges and problems that a curriculum is trying to address. And those are things that are very, very real and meaningful to the kids. And it sort of takes the kids back to when they were five and six years old and wanted to make lemonade stands to meet the needs of somebody in the community who needs money or something. That altruism, that wonderful altruism of smaller children that somehow gets taught out of them, or many of them, not all of them, certainly. But anyway, I’m trying to take what Joe is saying here and putting it into some very, very specific kinds of things that are so different between a school which is purpose driven, with a purpose which is meaningful, as opposed to a purpose which is really meaningless.

Rebecca: You described in Egypt kind of a context that was ripe for change, and ready to re-envision, repurpose, what would need to happen here to better align our schools for today’s needs in the US?

Joe: I’m glad you asked that. So we brought the process that we used back here to Philadelphia, and we convened a group of 120 people in eight design studios. We had a small grant to do this. And we asked them that question, we said, “What’s your vision for the greater Philadelphia area? What kind of society would you like to live in in the next 10 years?” And we spent time articulating that and then out of that came different purposes of education. It was, you might say, a mission statement… or more than a mission, it was a purpose statement that came out of that vision. And there was like eight elements of that purpose. I’ll give you two of those elements. One had to do with diversity, “how do you live in a diverse society?” And then the second one was, “What are the unifying elements for that diversity?” So if that is your top line theme, you might say then if you’re in biology, how would you exemplify that theme? Well, you talk about biodiversity. And then you talk about evolution as being the unifying principle of biology as one of those things. For chemistry, the diversity of the material world through the elements. And well, what’s the unifying principles to that? So you look at the natural world diversity, which is immense, and then you look at the human world of diversity. What unites us? We have tremendous diversities, not just around race and religion, but personalities and cognitive abilities. So it gets to then how do we live together among all of this kind of human and natural diversity? What unifies us? Kids want to know that. They have to deal with all of us, otherwise, you just get gangs. So that theme of unity and diversity can cut through all of the subjects, look at music, my God, and yet there’s a unifying principle. So those are the two elements, and so when you then sit a group of experts in a room that have diverse subjects, and you say, “Alright, here’s your theme: diversity, unity, you’re a biologist, tell us how would you structure the subjects along those two themes?” And we did the same. And out of that, then we create a curriculum. So it’s like a rug, warp and woof, the threads of a rug becomes an integrated curriculum from that, so that when a student comes into that school, they understand what the school is about, and why they’re learning what they’re learning.

John: How has that worked? How have the changes been received?

Joe: The people who participated were really amazed by the process, and they got a lot out of it. The next step in that is then developing the curriculum, and then implementing it. So we attempted to do that with the School District of Philadelphia. And they were not interested, even though we had this process. But why we were so thrilled and still are thrilled with our work in Egypt is that we talked directly to the Minister of Education, we were in meetings with him, we had his first deputy minister, who is still directing the project. So at the very highest level we had buy in. President El-Sisi visited one of our schools in Minya, Egypt and said, “My God, these students are amazing. I want 100 More of these schools.” And more recently, the minister and the head of the Parliament said we want to have this idea throughout all of our schools,17 million in Egypt, the elements of that idea. So what the biggest change has been in this process is, and the students will tell you, not me, the students will tell you is that they came in as individuals in a competitive environment. They came out of it, they had a personal transformation of seeing something greater than themselves, cooperating with each other, and thinking of it as “we” rather than “me.” And boy, wouldn’t that be great in this country if we had that spirit of let’s just work on this together?

Rebecca: I think your story really attests to the importance of some of the grassroots kind of components of reform as well as real support from leadership.

Joe: Exactly.

Rebecca: You also ventured into Bosnia Herzegovina. Can you talk a little bit about what that project was like there?

Joe: So a lot of this stuff happens through serendipity. We didn’t go out and apply for Egypt, they came to us, quite by accident. So the same thing happened in Bosnia Herzegovina which, as you know, it was a terrible situation back in 1991 with the breakup of Yugoslavia and tremendous genocide there. Save the Children had been working there on a USAID grant to work on a STEM school. This is in 2017. And it wasn’t working out too well. So they called on us, based upon our work in Egypt, and we went over there. And we did the same process with a group of university people and other officials of this design studio. We said, “What’s your aspiration for your country?” And their aspiration was: we want peace, and we have tremendous unemployment, and also people are leaving the country. So we want to be able to have peace, have people work together and not fight each other and be employed. So for them, they came up with the idea that the purpose of education is to be equipped to participate in a knowledge-based economy (KBE). I said fine. Well, what are they? So they listed 10 sectors of a knowledge-based economy? I said, “Fine, alright, so we have design studios, where we list these sectors and the elements of the sectors and then across this matrix was the subjects. And from that they developed a K to 13 curriculum that was integrated around these KBE sectors. So our role there was not to develop the curriculum, but to consult on the process by which they could do it themselves. And so we worked there for a couple of years. And we developed the curriculum app, an application, that was a relational database that had all of this curriculum in four different languages. So, it was interesting.

John: In your book, you advocate moving towards a more purpose-driven curriculum, but we’re living in a country where there’s a tremendous amount of political polarization and a lot of divisions, where there is a already a lot of pressure on schools to move in different directions tied to the local politics of people in that community. What can be done to move towards a consensus on the goals of the educational system in the US.

Deborah: There are a couple of principles that we need to deal with. One is that conversations have to happen across the sectors of community, whether we’re talking a local community, as a community of as large as a state, or national. One of the things of course, we don’t have a national education system as such here. The closest we get to a system is at the state level, but really, decisions are made much more at a local level. And we have tremendous diversity, even within the state of New York, for instance. I mean, you have one of the largest urban areas in the world, and then you have extremely rural areas, and with levels of affluence and poverty at the greatest extremes of the continuum. But conversations need to happen, but they need to be carefully facilitated, because you just don’t get in and say, “Well, what are we going to do?” You have to have very careful guidance and structure in this kind of design studio with everybody buying into the idea that things need to change. And we need to be able to really discuss freely and openly. And I think one of our beliefs is that when you have these discussions, I believe personally, and I’m a bit of an optimist in this respect, that we’ll find that many people, even from very diverse sectors, whether it’s politically or demographically or whatever, really have the same aspirations. I think the key is to identify the shared aspirations, and maybe where they aren’t shared, maybe there’s a way to forge unity between those. So I’m going to stop at this point, and then let Joe continue on with that.

Joe: I think that’s true. I think the key is to have a diversity of voices, and not just the loudest voices. And it needs to be, as Deborah said, very well facilitated in a process that doesn’t allow just the loudest voices to dominate. So that’s part of the process of developing that. But I think you also see that if you do a design studio in Oswego, for example, and then you also did it in Rochester, New York, and you did it in Hazleton, Pennsylvania, I would bet a bag of bagels that you’re going to find a lot more commonality [LAUGHTER] than you would usually think of it, because a lot of people just want to live, they want to have a family or they want to have success. There’s common human aspirations. But we live in a different time period than we did 1800s. So those aspirations have got to reflect the technology and the culture and the context of the times. And why that’s necessary is that you just can’t come in and impose a curriculum, you have to have buy in and acceptance… not to go back to Egypt, what we did was, we had to make sure that it was accepted. So if you have these aspirations, then you design the curriculum, you get affirmation from the curriculum, and everybody knows what the goal is. And once you decide you want to do it, and you have the authorization, if you will, from the decision makers at the policy level, financially, you can do it within a couple of years.

Rebecca: I love that you have great stories about success, and that it’s possible, because I think a lot of times it feels impossible when there’s a lot of division. One of the things that we’ve focused a lot on is the K-12 arena. What role does higher ed have in these conversations? Or what part of the conversation should we contribute to? Most of our audience are faculty and staff, administrators and things in higher ed institutions?

Deborah: I think higher ed has a huge role in several respects. Number one is if you change the way you’re teaching students so that they come out of their high schools with different sets of skills, and a different knowledge base and these kids go into their traditional university classes, there’s going to be a major disconnect. And again, I don’t want to beat the drum about Egypt, but very quickly a story that our students every semester, do a major semester-long, what we call a capstone project. And these are evaluated by outside evaluators. The very first time we did this, among the outside evaluators was a dean of engineering at the University of Cairo. And afterwards, she said, “Oh, my goodness, these kids are at the level of my master’s students or even above, and here they are just sophomores.” She said, “This is just amazing.” And I turned to her and I said, “These kids are going to be coming to your university in a couple of years. What are you going to do?” And she looks stunned. She was absolutely silent. She recognized the disconnect. And so that’s what we would call sort of a ground-up push for change, but there have to be other changes as well. Faculty need to be provided with a kind of support to do their own transformations in their courses and their pedagogy. And that has to be a huge change. But that won’t happen unless they see the need for change. And so that’s why we think actually starting in the high school is a great place to do that. But it also has to come from the leadership of the universities themselves from the top down, so we have all three working together. And of course, the first change has to be on teacher preparation. But changes have to happen in other courses as well, and those will be slower and probably take longer to implement. But the teacher preparation implications for this kind of curriculum, this kind of way of pedagogy and his way of assessment… we haven’t talked about that yet… are really quite significant.

Joe: You’re not going to be able to redesign schools, like we’ve been talking about, without the participation of universities for a number of reasons. One is that people look to universities for permission to think differently, they trust the expertise of professors. So they want to make sure that if they do this, it’s not going to hurt their child’s chances of college. So you need a partnership, a very strong partnership with the University to make that happen. And you need leadership from the President and Provost to give it the heft to make it happen. And so universities have been and would be indispensable to making this thing happen.

John: And many of the discussions at universities are along the same lines, in terms of the purposes that you’ve mentioned at the start of this recording session, it just hasn’t quite made it into the curriculum to the extent which might be needed to truly affect that type of change.

Joe: So in part of my research, I came across the report on general education that was done at Harvard in 1945, after World War Two, and it was about what you needed to live in a free society. And it was a very strong statement about the need to prepare students both at the college and secondary levels to live in a free society and what that meant, and how central the humanities were in that process, but also, that you cannot just teach math or the sciences for their own sake, that there has to be a moral arrow to it as well, because we saw in Germany in World War Two, how engineers and scientists were used in nefarious purposes. And I think universities are caught in this dilemma that students want to come to universities because they want to get a better job. But universities have a social mission too, and it’s more than just private gain that you’re doing with a student, you really want to transform that student into something more than who they are already and to enculturate them with a grander sense of obligation and duty. That’s really the value of higher education and the institutions, as I see it. And I see them as indispensable. So we wrote this book in a way to give a history and a sense of permission and a way to begin talking about these issues with some common language. That’s why we wrote it, because we realize universities have the same problem. Your Gen Ed courses, what do you have in those? But also institutionally, where are you going? What are you about?

Rebecca: I definitely agree with all the things that you’re saying. And I’m appreciative that you’ve written this book. One thing that Deborah had mentioned was assessment. So I wanted to see if we could pick up on that thread very quickly. So when we’re thinking about a purpose-driven mission and curriculum, how does that change the way we assess things? Assessment has been a big subject on the podcast more recently. And so I think it ties nicely into a lot of the things that we’ve been exploring.

Deborah: Well, I’m really glad you picked up on that, because that was absolutely critical. But what we have to do is we first have to decide what is success? And we can’t answer that question until we know what the purpose is. And I was just in a group of educators last night, and they were talking about what are the challenges of grade inflation. I said, “Well, what happens if we change the way we grade so that there can’t be such a thing as grade inflation?” And so what we’re talking about is a way to so radically change things like through the kind of evaluation I was mentioning, where students do projects and they have outside evaluators, is it complicated? Yes. Is it challenging? Yes. Is it worth it? Absolutely. Can you have grade inflation when you use rubrics and do blind evaluation and so forth? Yeah, yeah, you can get rid of those things. But the bottom line question is: how do you define success? And that has to be driven by the purpose which has been, as we have mentioned, deconstructed into what are the skills and concepts necessary to achieve your aspirations?

Joe: In the book we talked about the tyranny of math scores, that and English language arts has been the sine qua non of achievement and value in schools, and if you look at New York’s test scores, for example, for grades three to grade eight, math and science scores, the gap between low income and non-low income people stays the same from third grade all the way to eighth. And that’s predominantly the black and brown, so the gap remains the same. And if you just focus on math, you’re always going to be in a situation where you see these kids as deficient in some way. Whereas if you have the idea that there’s many different things that you can measure, other than math scores, that are of value, you open up the possibility of trying to find assets within students. And you also, through your assessment, you have to allow students to fail, and then to recover from that failure by learning something new, and not feeling that they are labeled… you know, it’s the growth mindset idea. So there has to be assessments that do that. And I’ll just say one last thing, you know, whenever someone learns something, particularly if it’s a misconception they have, and then they come to a correct understanding, there is a moment of vulnerability, where they say, “If I’m going to learn, I have to open myself up to different ways of thinking.” That’s an extremely vulnerable state. So if you have an assessment system that is judgmental, kids are not going to be learning, that’s why it stays the same in terms of the achievement gap.

John: I think there’s growing recognition of that in higher ed, which is why there’s so much discussion of alternative grading systems that encourage students to recognize that making mistakes is part of learning, and not penalizing them by what they come in not knowing but evaluating them based on their achievement by the end of the term.

Joe: Well, we’ve submitted a proposal with UC Berkeley on elementary assessment schemes using learning progressions as a framework so that if you’re traveling from Oswego to Albany, for example, let’s say Erie, Pennsylvania, you have a roadmap of where you’re going. And if you’re not quite in Erie, you’re somewhere in the middle. Well, that’s where you are. It’s not that you’re a failure to go to Erie. That’s where you are. So having this sense of measuring progress, but not evaluating it in terms of whether you’re failing or not, but you’re informing the student, “Okay, you have achieved this much. Here’s where you need to go next.”

Rebecca: Well, I’ve loved our conversation today. There’s lots to think about at an individual level and a social level, institutional level, policy level, there’s a lot of levels. But we always wrap up by asking what’s next?

Deborah: Well, our book has just come out. And so we are really interested in trying to start these conversations, just as you have given us the opportunity to do today and get people to start to think about the ideas that we have. We wrote it with a general audience in mind. But although it’s an academic book, definitely in that respect, it’s being marketed by our publisher as an academic book, but we really want to try to reach a more general audience and get these conversations started. So we want to spread the word, we want people to start thinking about some of these ideas, and maybe talking amongst each other in small book groups or something as: “What do you think the purpose should be?” …and start it both as a grassroots and at all levels, get these conversations going?

Joe: Yeah, for me, it’s not about telling someone what their purpose should be, or giving them our idea of what the purpose should be. I mean, I have my private idea, but more it’s the process by which it’s arrived at in a sense that it can be done. So we look at this as a sharing of our ideas and opportunities. We’ve been struggling with this for many, many years. So when we’ve had this kind of success through this process, that we want to share it. We don’t want to own it, we’re giving it away. That’s what the whole idea of the academy is about. So we’re hoping that through these interviews that we’re doing here, that people will find value in these and will read our book and have discussions about the process and about how things are in their own places. And that there hopefully will be a critical mass developed. So that’s what we’re doing right now.

John: Well, thank you for joining us. You’ve raised so many interesting questions that need to be addressed for education to be more effective. Thank you.

Deborah: Thank you.

Joe: Thank you. You’ve been tremendous hosts.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

344. Failing Our Future

Posted in authentic learning, engagement, equity, feedback, first-generation, grading, higher education, inclusion, learning, mastery learning, motivation, teaching
June 5, 2024

The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, Josh Eyler joins us to discuss research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming  book, Failing Our Future: How Grades Harm Students, and What We Can Do About It.

Show Notes

  • Eyler, Joshua R. (2024, forthcoming). Failing Our FutureL: How Grades Harm Students, and What We Can Do about It. John Hopkins University Press (pre-order link)
  • Eyler, J. R. (2018). How humans learn: The science and stories behind effective college teaching. West Virginia University Press.
  • Brookhart, S. M., Guskey, T. R., Bowers, A. J., McMillan, J. H., Smith, J. K., Smith, L. F., … & Welsh, M. E. (2016). A century of grading research: Meaning and value in the most common educational measure. Review of educational research, 86(4), 803-848.

Transcript

John: The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, we explore research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Josh Eyler. Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming book, Failing Our Future: How Grades Harm Students, and What We Can Do About It. Welcome back, Josh.

Josh: Thank you. Thank you. Great to be here.

John: We’re happy to talk to you. And it’s been a while. Today’s teas are:… Josh, are you drinking tea with us today?

Josh: Well, you know how I roll, you guys, I don’t have tea, but I have some lovely water. [LAUGHTER]

Rebecca: So tasty. I have a Scottish afternoon today.

John: And I’m trying to cut back on caffeine, so I just have a peppermint tea today.

Rebecca: Well, if you wanted to cut back on caffeine, you could join Josh with water. [LAUGHTER]

Josh: That’s true. [LAUGHTER] Come join the club, John. [LAUGHTER]

John: I suppose that probably would not be an unhealthy thing to do. [LAUGHTER] And I have been drinking a lot more water recently.

Rebecca: So, we invited you here today to discuss Failing our Future. Could you tell us a little bit about how this book project came about?

Josh: Sure. Yes. So it has a couple of origin stories, actually. One is that when I was writing How Humans Learn a few years ago, I was working on the final chapter on failure. And I kept coming across research on grades and grading and the obstacles that they set up to learning. And certainly if you’re thinking about failure as a tool for learning, you immediately are confronted with the fact that in systems that prioritize grades, they push back on the natural cycle of learning, where we try something, we fail, we get feedback, we try it again. Grades are set up in the opposite direction, they arrest that process before it can really play out. And so that is what piqued my interest. And I don’t know that I could say honestly that I thought I was going to write a book about grades at that point. But I knew that it was an important element of what I was discussing. So I’d say a few years after that, I think what we were seeing was a lot of discussion about grades, a lot more experimentation with it. That led me to want to write a book for a broader audience, one that certainly included educators, but one that cut across the K-12, higher ed divide one that was also for parents and policymakers and students, one that really took a bird’s eye view of the larger conversation about grades, trying to pull all of what we know together so that we could move forward as a community to try and enact change. So that’s what got us here. And the process was certainly longer than I expected due to the pandemic, but it was also just a very illuminating process. And I heard some heartbreaking stories as I did interviews for this. But overall, I’m really proud of the work.

John: And you mentioned addressing this through the entire educational spectrum. And I think it’s important to address it through the whole system. Because by the time we see students in college, they’ve already been indoctrinated into a system of grades, and there’s a lot of resistance to change, and also a lot of damage, perhaps, may have been done by the use of grading systems early. Could you talk a little bit about how we ended up with this system of grading, which has been so much a part of the educational culture from kindergarten on up?

Josh: Sure, yeah. And you’re right about the fact that students are conditioned for at least 12 years before they get to us to think about grades as being the most important thing about education. So how did we end up here, we could have a podcast episode that would last five hours if we really wanted to dig into that. But I’ll focus really on the more recent history. And that is that the A through F letter grade scale that we have is really about 125 years old. Our first records of a full letter grading scale come from Mount Holyoke in the 1890s. So they were the first to implement it. Back then it was A through E. They later dropped the E because they were afraid that people would think that they were excellent when they were not, you know, God forbid that we allow people to think that they’re excellent. And so they dropped the E and added a letter where there could be no mistake about what it meant, and that was an F, for failure. So that’s when it began. But that system, the letter grade system, was not formalized, or standardized, really until the 1940s. That is when you see a majority, in fact, of school districts and colleges and universities adopt that system. And what I think is most important about that, actually two things. One is that none of this has ever been inevitable, that the 1940s is less than 100 years ago, so this is not something that was cast in stone at the minute the American educational system began; it has changed and can be changed. The second thing, though, is that all of these schools, districts, all of these colleges adopted that system, not because they thought it was the best way to document student learning, but simply because it was the easiest way to communicate across institutions. They felt that if they had a common system that they could communicate more effectively between themselves. So it was not about student learning. It was not about student progress or growth or development. It was simply about communicating. That’s how we got here. And now we’re trying to figure out where do we go next? How do we undo some of the damage that has happened since that time?

Rebecca: Speaking of damage, do you want to underscore what some of that damage is? [LAUGHTER]

Josh: Sure, and I know that there’s room for reasonable debate about the terms that we use here. But I honestly really do think we can use the framing of damage and harm when it comes to grades. And in the book, I have several categories. The first and most obvious category for that damage has to do with learning, that grades set up obstacles to learning. They inhibit curiosity, they inhibit creativity, they inhibit risk taking, they affect intrinsic motivation. This is painfully obvious, but grades are classic extrinsic motivators, which means that they are very good for getting people to do things that they would not otherwise want to do. Extrinsic motivators operate effectively, if your goal is compliance, you want to get people in seats, you want to get people to turn things in on time, you want to get people to say things in class, grades work for that. But if what you’re interested in is learning and quality, that’s what you need intrinsic motivation for. The grades can get people into learning spaces, but they in no way ever guarantee that a student will learn even though they are actually in the environment. So that’s really important. Environments that prioritize grades also incentivize cheating, and they just stand in the way of a lot of different things. But that’s just the academic part of it. Obviously, those are important, but to me, they’re less important than some of the effects of grades on students’ lives and wellbeing. Grades magnify and mirror inequities that have always been a part of the American educational system. For example, students who attend poorly resourced high schools have fewer educational opportunities, fewer textbooks, more teacher turnover, all the things that are tied to less-resourced schools. And what that means often when they get to college, they have experienced what we call opportunity gaps, which show up in their grades, especially in Gen Ed courses. So those grades that you see in the first couple of years of a student’s college experience, one who has experienced opportunity gaps, those are reflections of the past, they’re not indications of the student’s potential. They are indicators of systemic inequity. So that’s just one of many examples that we could rifle through. Grades are surveillance technologies. They’re often used for punishment, especially in high school, but kind of across the board. So that’s another category. inequities. And then the final one, the one that we’re getting more and more research on now, maybe than we’ve had in the past, and one that I personally care a lot about is that we have evidence now that grades are a contributing factor to the mental health crisis in teenagers and young adults, the academic stress caused by grades. We have a number of research studies pointing us toward the significant contribution of grades to that mental health crisis. So lots of areas here, I think, beyond the classroom. And that’s really our central idea of this book that grades have long afterlives, well beyond a semester, well beyond the classroom, that affect people’s lives, sometimes just in the short term, but often in the longer term as well.

Rebecca: Josh, you went from going from grades are no fun to grades are a super downer. [LAUGHTER]

Josh: Yeah, well, that’s the message. It’s a happy book, Rebecca. [LAUGHTER] It’s not going to win any awards for a happy smiley story, that’s for sure. [LAUGHTER]

John: One concern that some faculty have about moving to alternative grading systems is that, while intrinsic motivation is something we’d all really love to see in our students, there may be some tasks in our disciplines that students have to master which may not be as intrinsically rewarding as others, but they might still be foundational for students to be successful. If we move away from grading systems, what type of motivation can be provided to help students master those tasks that may seem tedious and may not be quite as intrinsically rewarding as other areas that students just find much more interesting?

Josh: Well, that’s a great question. There are a couple of things I’d like to address about this. So first of all, it’s not that we’re talking about either having traditional grades or having no grades, and nothing in between. What a lot of people are doing in the area of alternative grading practices is experimenting with a whole range of models. And so particularly for the types of courses and disciplines that you’re talking about in that question, John, I have found that many faculty gravitate toward a model called standards-based grading, where you can identify the skills, the content knowledge, the disciplinary norms and necessities that are important for students to develop. And the grade for the course is based, not on individual performance on exams, but how many standards they meet, over the course of the semester. And as a feature of that particular model, often students are given multiple attempts and multiple ways to meet the standards. So that’s part of releasing the pressure valve a little bit for students that gives them room to grow and honors the fact that individuals learn at different paces. So it doesn’t matter when in the semester that they hit the number of standards in order to get their A or B, because the process is a fundamental part of that grading model. So I think that’s really important. Another thing that often comes up in these conversations, though, is the fact that we do in fact, have gradeless colleges in America. The one I feature in the book is Evergreen State College in Washington. And they don’t have any grades at all. They have fully narrative transcripts. They give only feedback over the course of the semester. And they teach all the disciplines. So they have found ways to heighten the intrinsic motivation, and to use feedback to really help students navigate the path toward the goals that they have for themselves and for the class. So there are ways to do this. You don’t need grades to keep people in the seats or to have them do things that might draw on less intrinsic motivation. I think there are ways to structure our learning environments that allow us to do similar work without the pressure of traditional grades. And I think that’s the ultimate point here. I think what a lot of folks in this area who are interested in grading reform are trying to promote is not necessarily just ditching grades outright, although we could talk about that if you want, but reorienting students relationship with grades, the messages that grades send to students, so that it’s not necessarily an evaluation, but a tool helping them develop their skills and to push their learning forward.

Rebecca: It’s always been interesting to me that the A through F system was meant to provide some standardized ways of communicating between institutions yet, if you look at any single institutions’ body of syllabi, you can see that the grades mean entirely different things [LAUGHTER] depending on a class, like some might be about attendance, some might be about success on a test, some might be about achieving learning objectives. So it seems like if it was meant to communicate anything [LAUGHTER] across institutions, it certainly isn’t meeting that objective. I don’t know what my question is. [LAUGHTER]

Josh: I’m so glad this up, though.

Rebecca: I had a question, but I think I lost it. [LAUGHTER]

Josh: No, I’ll answer it. There’s a question in there, Rebecca.

Rebecca: Thanks for helping.

Josh: What you’re talking about there is what I call the measurement fallacy. And so there has been more and more work over the last few years, including one that I absolutely love, called A Century of Grading Research, where a team of experts dives into a massive amount of material. And the conclusion that they come up with is that a grade is nothing more than a subjective indicator by an individual instructor on a student’s progress toward that individual’s goals for a particular course. So I set the goals for my course, a grade that I would give is nothing more than an indicator of how much progress a student has made on my goals for my course. They call them learning intentions in this paper. It is not in any way, a kind of universal certification of learning in a particular course or discipline. Nor does it mean that the student who gets a certain grade in my course would get the same grade from my colleague down the hall teaching the same course. It is a truly subjective indicator of progress. And because of that, we cannot really say that grades measure what we have been told that they measure and what our educational systems have assumed that they measure, which is learning. They’re not universal measurements of learning, they’re subjective indicators of progress in the course.

Rebecca: And yet they’re used to make all kinds of decisions about students.

Josh: They are, and you’re absolutely right.

Rebecca: It’s very interesting. [LAUGHTER]

Josh: So I think there has been a lot of ill placed weight on grades without a subsequent amount of investigation into what grades have meant, and continue to mean. This is also what fuels the furor over supposed grade inflation. And there are lots of other tendrils of problems, when you begin to put so much emphasis on grades to do so much work in our educational system.

John: You mentioned Evergreen State College, but another college that had gone in that direction, very successfully was New College, and I teach in a program at Duke in the summer, and I had two TAs from New College, and they were two of the brightest people I had ever worked with, and their mastery of the subject matter was very much equivalent to that of other students I had who came from institutions with very traditional grading systems. So they certainly didn’t seem to be damaged by that. And yet, there seemed to have been a bit of a hostile takeover of New College. [LAUGHTER]

Josh: Yes, there has been but there is not evidence yet that the hostile takeover has affected that gradeless model. There was some discussion that I read in transcripts of board meetings about that grading system, but it has not changed yet. And I’m glad you brought up New College, John, because, and they have this on their website, as recently as last year, they were able to boast that they were producing students who went on to get the highest percentage of STEM PhDs in the country, relative to their size as an institution, which says a lot. If you are a gradeless institution, and your students are succeeding at that rate in STEM PhDs, it means that they’re learning not just content, but also the skills that are necessary to do that level of advanced work. So I think that that says a lot for what is possible, even within systems that do not have anything close to a traditional grading system.

John: And I should note that one of those students did go on to a PhD in economics. And the other student went on and picked up a master’s degree in data analytics. And they’ve succeeded very nicely, despite the absence of grades [LAUGHTER] in their college career. [LAUGHTER]

Josh: Imagine that.

John: When we began this discussion, one of the things you noted was that under traditional grading systems, the grades at the end of courses often reflect differences in the opportunities that students face before they enter our classes. Can you talk a little bit more about that?

Josh: Absolutely. So you have the reinforcing of those gaps, you have extensions of those gaps. So if you think about what happens to a student who comes into a college course with those opportunity gaps, and then is in a system, for example, where grades are curved, so not only then are they trying to bridge over those equity gaps, but their grades are really reflective of those gaps. But now they are also competing in a curved grade system with their other fellow students, many of whom have had more practice at the kinds of exams and the kinds of questions and the kinds of teaching strategies that they’ll see in those courses. So it just starts to pile on top of each other. And I think that this is something that we just don’t talk enough about, it’s just assumed, and many of these students are labeled unprepared or not ready for college, which places all the burden on them rather than this more systemic problem that I think we really need to face. Because we don’t talk enough about it, we assume that grades have to be a part of the landscape of education. And because of that, we, and I say we as the community in education, we’ve just been willing to tolerate all of the issues that grades have brought with them without really pulling back the curtain and looking at what the issues really are.

John: And we lose so many students in the first couple of years of college and many of the students we lose are those who come from low-income school districts, whose parents had less education, and who just come in with less preparation. And that’s not a very equitable way of providing an education.

Josh: Right, absolutely. And so this shows up on all kinds of metrics that colleges care about: retention, progress, graduation, all of which, by the way, are tied to grades and GPAs. That grades are the gatekeepers for keeping those students enrolled and moving through to graduation. So yes, that’s a major problem to address with respect to those students who are coming to us with fewer opportunities and from school districts with fewer resources. I think we need to really investigate the traditional ways of being In higher education, really look closely at them, reimagine them in order to create systems that have equity at the center and where we are actually, I think, invested in student success and helping those students from those different backgrounds to succeed.

Rebecca: I think the other thing to underscore related to equity issues is not only does an institution maybe have policies about grades and decisions are made at an institutional level, but at the federal level grades are used for financial aid. And so if you don’t hit a certain grade point average, and if you don’t have a certain number of classes that you complete successfully, so you’re not withdrawing from them, or having incompletes, etc, then you might not be eligible, you could lose financial aid. So whether or not a student has demonstrated growth in any kind of way doesn’t matter. It’s down to this idea of grades. And depending on the classes they’re enrolled in and the different things that grades mean, we’re essentially not allowing students to proceed because they can’t afford to.

Josh: That’s absolutely right. The more you really take the magnifying glass to this, the more you see how implanted grades are in all of these decisions, and all these key points in a student’s academic career, bringing in financial aid, I think, is a wonderful example of their gatekeeping function. And the idea that you’re raising there, that it doesn’t even give students a chance to try and bridge those equity gaps. It doesn’t honor their growth or development at all, it says you got less than a C average in your first year, therefore, we are not giving you any more money to move through the university, and so many leave, many dropout. And I think that there is a better way to do things.

John: And many of those students who leave end up with a fairly significant burden of student debt that they still have to pay, further increasing the inequity. What types of alternative grading systems do you discuss in your book, you mentioned standards based grading, but what are some of the other areas that you encourage faculty to consider?

Josh: Well, the book is divided into two parts. And the first half is all the problems and the second half is possible solutions. And so there’s a chapter for parents on what you can do in the home to help your children reorient their own approach to grades. And then the last chapter is about systemic change. But the one in between that is focused on what teachers in K-12 and faculty in higher ed are doing in their own individual classrooms to help with this. And so, there’s a whole range. There are the models that are related to standards-based grading, I call them the cousins: they’re standards-based grading, specifications grading, competency-based, mastery-based, proficiency-based, they’re all not very different from each other, they just have slightly different names, and maybe a few modestly different features to them. So that’s one bucket. Portfolio grading is one that I recommend for folks who just want to dip their toe in the world of alternative grading, because a portfolio model, you’re giving a lot of feedback throughout the semester, a lot of opportunity to revise and redo assignments. But ultimately, the portfolio that is turned in at the end of the semester gets a fairly traditional grade associated with it, it’s just that that grade honors the whole arc of the work for the semester. So that’s another one. Contract grading, and its various branches that we have recently seen. So classic contract model was developed in the middle of the 20th century. And it looks something like this: here’s a list of things to do for the course, if you do 15 of them at a satisfactory level… I’m just making up that number…if you do 15 at a satisfactory level, you get an A for the course, you do 12 you get a B, etc. That’s a contract model. More recently, there’s been a version of it called labor-based contract grading,…very prominent in writing studies… that tweaks it a little bit to emphasize more the amount of work, the amount of writing that students do over the course of the semester, and that is tied to fulfilling the particular contract. And then if we’re looking at a spectrum of grading models, with portfolio grading being one of the more conservative of the alternatives, on the other end of the spectrum you have what some people call ungrading and other people call collaborative grading where there are no grades throughout the semester, only feedback, lots of student self assessment, self evaluation, and then at the end of the semester, the students proposed their grade and the final grade is determined through a conference between the instructor and the student about that proposed grade. So there are lots of things that people are trying. I tried to send two messages about this, one is that my goal is hopefully to help faculty find models that will work best for them and their students, that it’s not a one-size-fits-all model. The same thing that works for me may not work for my colleague in the next classroom over. So that’s important. The other thing is that you do not need absolute fidelity to any one of these models. It’s not like Moses came down from the mountain, with tablets that you had to follow to the letter, lots of people are mixing and matching and taking elements from a host of different systems that they find works for them. So I’ve yet to find someone marching down the checklist for engraving or for contract grading, everyone’s kind of putting their own spin on it, and that’s good. Well, it’s actually not only good, it’s important since our contexts vary so significantly across higher ed.

Rebecca: As a lot of faculty start considering these different alternative models, collectively we can start to initiate culture change, kind of with a grassroots model. But systemic change requires policy change and other things as well. Can you talk a little bit about what role we might have in pushing forward some of those agendas as well?

Josh: Like I mentioned before, a major reason for writing this book was because my observation was that a lot of people are talking about grading reform, but they’re all coming at it, understandably, from their own perspective, the faculty perspective, the administrative perspective, the policymaker perspective, and the parent’s perspective. And I think that if we’re thinking seriously about systemic change, we need to have everyone on the same page collaborating to move change forward. All that is to say that I think about this question quite a lot. There’s certainly examples that we have of recent systemic change, some of the University of California schools have changed their grading systems in significant ways. Western Oregon just dropped the lowest grades on the scale, they’re no longer available to give. I just met someone a couple of weeks ago from Bryant University, they’re moving to collaborative grading at scale for their first-year seminar courses. So it is happening, what it requires is a network of people across an institution, all of whom believe that this is not only the right thing, but they understand how to use policy to make the reform happen, but also how to tell the story in a way that all the different constituencies can see themselves in it, and can agree that this is the direction that they need to head. So the networking is important, understanding how policy shapes grading habits and processes at a university. If you’re part of a public system, which I know you and I both are, you need to understand how the system policy and the state policy plays into this as well. So that requires having that network of different people at different positions in a university to really move that work forward. But what I really want folks to know is that you’re not alone in this, that there are lots of people thinking about this, and lots of people who are doing this work, sometimes under the radar, sometimes above the radar at institutions, and that one person taking a step is a step toward change. But multiple people taking the same step together begins to create a movement toward change. And so I want people to know that. I also want people to know that there are blueprints out there for how to do this work, that there are people who study grading reform, there are models out there for successful institutional reform. Given my work for the book, I think we have a lot to learn from K-12 school districts. There are so many of those each year that are transitioning to standards-based grading across the district. Last year, for example, the entire city of Santa Fe, New Mexico, all their school districts moved to standards-based grading. So this is happening. And the more we learn about process, the more we draw on our colleagues’ network and expertise in these areas, the more we work kind of as a collective in higher ed to push this forward, the more change you’re going to see.

John: We give a lot of workshops on campus and some of them are focused on alternative grading systems. We haven’t seen quite as much movement with that among faculty as I thought we perhaps might, because during the pandemic, a lot of people were open to trying some very different things. But there’s been some resistance. We’ve had a number of people trying it, but it hasn’t caught on quite as much as I had expected it to. Why are so many faculty resistant to moving away from traditional grading systems?

Josh: Well,I think this varies from place to place. But I will say that I think that a major thing that we all face as faculty is the pressures of time. Everyone wants to do their best for their students, I really do believe that, but when you’re faced with such a limited amount of time in which you can change elements of your course, you’re probably going to gravitate immediately toward the kinds of content that you’re teaching in the course in the assignments, the course design pieces of it, or you’re going to think about active learning strategy that John and Rebecca talked about a few months ago, rather than addressing the thing that is at the very foundation of our educational systems in America. That’s not going to be the place that you go first, [LAUGHTER] kind of an upheaval of everything that you have known about education. So I think some of it is just, in a system where we all have very limited time, and very few resources, where do you put your efforts, and I think that is driving some of that. I also think we’re kind of seeing this interesting bounce back that followed that period of innovation at the beginning of the pandemic. I think what you see is almost a defensive response to that period of innovation, that what we really need to do to bring students back and re-engage them in the coursework is to try and somehow recapture what our image was a traditional education pre-2020, that we need to move backward toward that rather than capitalize and extend the innovations that we were doing in the early stages of the pandemic. I don’t want to paint with too broad a brush on that, but I have seen some of that elasticity, that rubberband effect, in talking with folks across the country about some of these issues. So I think those two things are at the forefront of why some folks might resist that. And none of those really has to do yet with really trying to unpack people’s own philosophies of education, or how their grades and their grading models tie into their values that they have with respect to education. So that’s even kind of a deeper level that you can’t even get to until you address some of these other concerns.

Rebecca: And when we’re crunched on time, things that we’ve done before are easier to implement. And it’s also hard to find time to reflect on our practices and to do some of the things that you’re talking about, in thinking about how our values align with what we’re doing. We’re really good as institutions of talking about equity, but we don’t always fully analyze how our practices impact actual equity.

John: And we were all the products of a graded system. And we were somehow successful in that, or we wouldn’t be in these positions. And so I think there may be a little bit of psychological resistance to changing something that has been so fundamental to our own experience.

Josh: Yes. I think a huge capital Y-E-S [LAUGHTER] to both of what you’re saying, that Rebecca, your point, we are not very good at really interrogating how our practices may or may not be aiding our equity goals or advancing our equity goals. And John, yes, we have succeeded in graded systems. And I do think the psychological is a part of it. If we were able to do it, why shouldn’t other people be able to manage this? I think that is what some people think. And more than that, I often see a kind of defensive fallback. Part of this is natural, in that we love our disciplines and devoted our lives to our disciplines, but a fallback toward if I get rid of grades or if I change my grading system, what does that mean for the standards of my discipline, for how people will engage with my discipline? And so I think that’s a psychological element of this as well. And it comes out in arguments about rigor and standards, but what it’s really about is the individual psychology, I think, of the person making the argument.

John: Are there other topics from your book that you’d like to discuss that we haven’t?

Josh: I think the reform efforts that I have seen succeed, and those that have failed, always have hinged on communication, at every level. Why are we doing this? What is the purpose of it? How do we bring people into the process very early on at all levels. At the K-12. level, that means parents, teachers and administrators working together. I think at higher ed, it means utilizing the expertise of faculty right out of the gate, allowing them to shape the narrative and the conversation and the direction and draw on administrative and staff resources to help enact that vision, rather than having a top-down mandate that you had to change your grading. That’s never gonna work in higher ed. But I will say that the communication is what drives all of that work. And it is the hinge that either allows one of these efforts to succeed or fail.

Rebecca: Well, we have a lot to think about, but what’s next, Josh?

Josh: So, what’s next? Well, the book comes out at the end of August and so I’m really excited to launch that and to finally have it in people’s hands and have lots of conversations about what’s in there, hopefully, So, that’s what’s immediately next. I am thinking about the next project that has something to do with the students who are coming to college now are the students who their entire K-12 experience has been shaped by the Common Core, and the obsession with standardization in America, and so what does that mean for higher ed? And what does that mean for those students and their learning? So, I’m in the very early stages of trying to think about that a little bit, but that might be what the next project is.

Rebecca: You always have great projects, Josh, and get us thinking about really important topics. So thanks for your work. I’m looking forward to sharing your new book.

Josh: Well, thank you both very much. It’s always a pleasure to be on this podcast, one of my favorites, and I’d love talking to you both about all these issues, so hanks very much.

John: It’s great talking to you again, and I’m looking forward to receiving my copy of this book in the summer. I ordered it as soon as I saw the notification that pre-orders were available.

Josh: I appreciate that, John, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]