349. Growth Mindset Messaging

First-generation college students, on average, have lower GPAs and higher dropout rates than continuing-generation students. In this episode, Elizabeth Canning, Makita White, and William B. Davis join us to discuss a growth-mindset intervention that has eliminated this equity gap in a large STEM class.

Elizabeth is an Associate Professor in the Psychology Department at Washington State University. Makita is a graduate student at WSU’s Experimental Psychology Program, and William is a Professor of Molecular Biology and the Interim Vice Provost for Academic Excellence and Student Achievement at WSU.

Show Notes

Transcript

John: First-generation college students, on average, have lower GPAs and higher dropout rates than continuing-generation students. In this episode, we discuss a growth-mindset intervention that has eliminated this equity gap in a large STEM class.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Elizabeth Canning, Makita White, and William B. Davis. Elizabeth is an Associate Professor in the Psychology Department at Washington State University. Makita is a graduate student at WSU’s Experimental Psychology Program, and William is a Professor of Molecular Biology and the Interim Vice Provost for Academic Excellence and Student Achievement at WSU. Welcome back, Elizabeth and Makita, and welcome, William.

Bill: Thank you. It’s a pleasure to be here.

John: Today’s teas are:… Elizabeth, are you drinking tea?

Elizabeth: I am drinking tea. I’ve learned from past podcasts with you that I should bring some tea along. So I’m drinking some sweet and spicy black tea today.

Rebecca: Sweet and Spicy… [LAUGHTER] sounds interesting.

Elizabeth: It has some cinnamon and some orange. It’s very good.

John: Very nice. And Makita?

Makita: I’ve got classic Earl Grey today.

Rebecca: Can’t go wrong with a classic. [LAUGHTER]

John: And Bill?

Bill:I have a Raspberry Zinger herbal tea today.

Rebecca: I’ve moved on to Awake tea… this time of day. [LAUGHTER]

John: We recorded another podcast much, much earlier today. So that’s probably needed a bit more. [LAUGHTER] And so that I won’t be awake all night tonight, as I have been for much of the last couple of nights, I have just a pure peppermint tea.

Rebecca: Probably a better choice for this time of day. [LAUGHTER]

John: And for this time of the year.

Rebecca: So we’ve invited you here today to discuss your March 2024 study: “Growth Mindset Messages from Instructors Improve Academic Performance Among First-Generation College Students,” which is quite a mouthful. [LAUGHTER] But before we discuss the study, can you talk a little bit about the difference in outcomes between first-generation students and continuing-generation college students?

Makita: Sure. So when we say first generation, we usually mean someone who’s the first in their family to attend college. For this study, we specifically defined it as a student for whom neither parent has earned a four-year college degree. And these students typically have to face and overcome a lot of social and cultural barriers in order to be successful in college. And although almost a third of the people who go to college are first-generation, they don’t always seem to do as well as continuing-generation college students. On average, they tend to have a little more difficulty adapting to college, they typically earn lower grades and they have higher dropout rates. They also act a little bit differently, they’re less likely to do things like go to office hours, or communicate with their instructors, whether that’s by email or in person to seek clarification on course materials. And just in general, they tend to be less likely to try to access helpful academic resources when we compare them with continuing-generation college students, which would be students for whom at least one parent has earned a four-year college degree. And on top of all of that, the students are also usually… I mean, they have to be… dealing with having less familial guidance than their peers do when it comes to navigating higher education. So if you’re a continuing-generation student, you have a parent who might tell you something like, “Hey, it’s a good idea to go to office hours,” or “You should be asking questions” or “there’s a financial aid department,” important stuff like that. They also may be dealing with a mismatch between the values and culture that they grew up in, compared with the values and expectations of an American university, which is typically very individualistic and may not be as supportive as what they are used to. And of course, they’re also more likely to be working, living off of campus, and they usually get less financial support from their families. But there are other factors that may exacerbate the difficulties that they’re dealing with. And this study is about one of those.

Bill:And one of the things that, just for context at our local institution to keep in mind, is WSU has a lot of, for lack of a better term, brand loyalty. It was not uncommon for me in my class to have fourth generation Cougars. And if you think about that legacy of your great-grandparents had attended this institution, as had other people, that sort of familial knowledge that gets passed along, is a huge benefit for that population of students as compared to the first-generation students as Makita was talking about.

John: One of the things that you mentioned, Makita, was differences in help-seeking behavior. And one of the studies on that is one that we talked about with you and Elizabeth in an earlier podcast, and we’ll include a link to that in the show notes. Could you describe this current study?

Elizabeth: Yeah, so this study we conducted in 2021, I believe. It was the second semester online, so the pandemic year. I know we’re all sort of crossing out that memory from our minds… [LAUGHTER] …that whole year. As instructors, it was a little tumultuous. But Bill and I got together and we designed this study together in his class. He was teaching at the time the introductory biology class at WSU. And what we decided to do is test out some actionable techniques that instructors can do to communicate a growth mindset to their students. So a lot of the research that my lab does looks at how growth mindset messages impact different students from different backgrounds. And at this time, there was a lot of suggestions about how to implement growth mindset ideas in your class, how to change course material, how to integrate it throughout, but there wasn’t really any experimental evidence of specific strategies that instructors can do. So we decided that we would just test it out in Bill’s class. So what we did in this study is we randomly assigned everybody in the course to one of two conditions: a control condition or a growth-mindset message condition. And after the first two major exams in the course, students received an email from Bill, and in the control condition they received a message that is pretty typical of an instructor after an exam has been conducted in their class. It says things like: “exam grades are posted online, here’s how we calculated them,”
“here’s what it means,” and then “feel free to talk to me in office hours, or ask questions about it.“ In the growth mindset message condition, we included all of that information, but we also included a fairly lengthy paragraph… I’ve gotten comments on… but it’s a big chunk of a paragraph that basically says, in Bill’s words, “Here’s what I believe about student performance and exam performance and improving in the class.” And there we integrated growth mindset theory, we talked about how abilities can be improved, we talked about how academic struggles are normal to experience, and that this struggle is part of the process of learning, that is something that is controllable. That you can put in effort that you can use different strategies to make improvements. And I think we used, Bill, if correct me if I’m wrong, we started with a draft of this email that you had already been doing in your class. And we just sort of infused it a little bit more. So Bill really wrote like the first draft of it, and then we kind of went back and forth a little bit to edit it and infuse it a little bit more with growth mindset. But it’s really written in the voice of the instructor. After we implemented this email after the first exam, we did it again after the second exam, changing the language a little bit. We assessed their performance across the semester. So we looked at their exam grades, we looked at their final course grade, and we found that students who were in the growth mindset condition earned higher grades in the course. And this was predominantly driven by first-generation college students. So first-generation college students performed much better in the course, when they received those growth-mindset messages from the instructor.

Rebecca: Can you talk a little bit about the effects and how significant they were?

Elizabeth: Yeah, so when we look at the group differences in the control condition, what we find that’s very consistent across big STEM courses across the university, that continuing-generation students are getting better grades than first-generation students. So in this course, they’re earning about three-fourths of a letter grade higher than first-generation students. So this is a really, really big gap between the continuing-generation students and the first-generation students. It’s not specific to this one course. So we see this in a lot of different datasets. But what was really remarkable in this study is that that difference in their grades was completely eliminated in the growth-mindset condition. So first-generation college students earned about three-fourths of a letter grade higher in the growth-mindset condition compared to the control condition. So this is a pretty large effect on final course grades. And of course, it’s pretty significant in terms of the curriculum that they’re a part of. Bill, if you want to talk a little bit about the students who took this course and how it plays a role in their career trajectories.

Bill:Yeah, thank you for that. I was looking forward to jumping in a little bit. And this is a little bit about my journey too, in this course. So, at this point, just for context, this was the 16th semester that I had taught this course. Largely I had been either the sole instructor or I was the primary instructor of record for the course. It’s taken by about 450 to 550 students each semester, little bit of differential between fall and spring in size, over 50 different majors, or pre-major students take this class. It’s nominally a freshman majors biology course. But there’s a substantial number of students who are neither biology majors or life science majors. And there’s a sizable number of students who are not freshmen. So, in reality, the highest population of my course normally were sophomores, freshmen were the largest population in that spring semester. And that was pretty typical, because it is oftentimes taken as the second semester of an introductory biology sequence. So yeah, and to be honest, over the years before this study, I had been trying to narrow those opportunity gaps between first-generation and continuing-generation students. I knew they existed. I had done many things with the course experimentally over the years, and never had really closed that gap. This was one of the first times, if ever, I saw that gap completely close in anything that I tried, and I tried tinkering in the lecture in the lab, big changes, small changes, a lot had gone on before this.

John: You mentioned that the class is normally about 450 students. Could you tell us a little bit more about the specific course in which you implemented this? You noted that this was during the pandemic. Was this offered remotely? Was it synchronous? Was it asynchronous? And how many students were there in this particular sample?

Bill:So, the course was taught in a synchronous Zoom manner. So both the lecture and the lab were taught using zoom. I believe, if memory serves, I had the largest Zoom Room on campus that semester. I can’t remember the exact number, I want to say my enrollment that semester was somewhere close to 550, or 600

Makita: The final sample is 417, I believe.
Bill:417. That’s right. That’s how many students, so it must have been around probably 450, at the beginning of the semester, or something like that. Yeah, we had synchronous Zoom every day with students. In fact, I learned pretty quickly how to crash Zoom my first semester I taught remotely. [LAUGHTER] So by then, it was a little bit more of a pro, and at least knew how to keep it running for 50 minutes at a time.

Rebecca: And true badge of honor.

Bill:[LAUGHTER] Well, there are many lumps that we took in the pandemic and learning how to teach and Zoom was one of them. [LAUGHTER]

John: I had 350 students in the fall of 2020, and it was not my most enjoyable teaching experience. What are the factors you controlled for? Because you did control for some other variables to separate out the effect of the growth-mindset messaging?

Elizabeth: Yeah, so we controlled for three things in our analytic models. We controlled for students’ prior college GPA… so this was just their self report… GPA that they’ve had in college. Since this was in the spring semester, all the freshmen had had a semester under their belts at that point, to have an idea of what their college GPA was. We included this just because we were looking at performance. So we wanted to filter out or control for their prior performance, that’s a big predictor of their performance in the class. We also controlled for their self-reported personal mindset beliefs. So we asked the students themselves, what do you believe about the nature of intelligence? Do you think it’s something that can improve? Do you think it’s something that you’re just born with? So we measure their own self-reported mindset beliefs, and we control for that and all of the analyses, because we want to look at the effect of the intervention above and beyond student’s own beliefs about ability. And then the last thing we controlled for was their race or ethnic status. So what they self reported to us on a survey. Here, we controlled for that because there is some overlap between race, ethnicity, and first-generation status. We did look at the different groups separately, we don’t have enough power to really look at race ethnicity as a moderator at the intervention. So we control for it in our analyses to look at the effect of first-gen status above and beyond race and ethnicity,

Rebecca: I can imagine, as an instructor who’s fiddled quite a bit in their class to overcome these equity gaps, that it might have been a little surprising [LAUGHTER] how much of a change happened with such a relatively small intervention.

Bill:It was shocking. It’s funny, because one of the things that Elizabeth has really talked about is how we took materials that I actually already possessed in some ways, and modified them slightly in some cases, and more radically in others, but just that transformation was enough to start to really make an impact. And you could see it happening. And that was amazing to me. I remember when I halved my opportunity gaps in my class with an intervention. And I remember dancing and celebrating, [LAUGHTER] and then this one was just on that scale. This was huge and amazing to me.

John: One of the interesting things in this study is you were looking at the effect in terms of grades on subsequent exams, but you didn’t find much of an effect on the second exam from the first intervention. But when you had two of those messages coming out, that’s where you found the relatively large effect. Why do you think the second message was having more of an effect than just a single message?

Elizabeth: Yeah, so we have a couple of theories. We think that one reason could be that they just need the message more than one time. So they need two doses or more than two doses to sort of really get people’s attention and make them change their behavior. That c ould be one reason. Another reason is that it might be enough to just have one message, but it would take time for behavior to change in order to see a difference in their performance. So it could be that after the first exam, they started engaging with the material differently and then that then sort of snowballed to affect their third exam scores, and then again, their final course performance, but we don’t really know exactly what’s going on. Another reason could be that towards the end of the semester, students get a little antsy, and they start waking up a little bit to, “Oh, I should probably be doing something different if I want to have a grade that looks like this.” So it could be timing, it could be dosage, it could be the effects take longer to materialize in terms of performance changes. One of the interesting things that we found here in this dataset is we did try to look at behavior in the way that we had access to, so we pulled all of their activity on the course website. And this semester, our campus was using Blackboard as the course sites. And we pulled all of the Blackboard data. So we looked at whether students in the growth mindset condition were engaging with Blackboard differently. And we found that they were. Students, on average, who were in the growth-mindset condition engaged with the course material 40 instances more, on average, than students in the control group. This is around a 12% increase in webpage engagement. So after they get this message from the instructor saying that intelligence is something that can grow, abilities can improve over time, and here are some strategies that you might implement to realize those changes, they’re going to the course website, they’re clicking on things, they’re re-reading their notes, they’re looking over the course lectures, they’re engaging more with the resources that are posted there. And then not surprising to any instructor ever, [LAUGHTER] that when students are engaging with the course materials that then lead to greater performance in the class. So we found a mediation effect through their behavior with the course material that was posted online. So this is one example of the behavior change that I was talking about earlier, that when they get this message, it can be inspiring, it can inspire a different type of behavior that can lead to better performance in the course.

Rebecca: Those are really strong click rates generally, [LAUGHTER] especially when we’re often complaining that students don’t read our emails or messages. [LAUGHTER]

Elizabeth: Yes. So, we did look at differences in Blackboard in terms of clicking on course material versus going to the page where all their grades were, like the gradebook. And we didn’t find any differences on their engagement with the gradebook. So it wasn’t that they were just tracking it more or anything like that. But it seems to be the case that they’re actually engaging with the substantive course materials more, on average, in the growth- mindset condition.

Rebecca: It’s just incredible. [LAUGHTER] Can you talk a little bit about… and we’re hinting towards this, obviously… what does this study suggest that faculty and institutions can do to improve outcomes of first-gen students, or maybe even, Bill, what you’ve continued or started to do?

Bill:I think, for me, what I took away from it was being very intentional with my students in communicating what I believe about them. And I mean that in the spirit of the old NPR show, or the segment that they used to do, you know, “This I believe,” where people would have those small snippets. I felt like this intervention was something along those lines where I could be authentic in communicating what I truly believed about their abilities. And I did it in a way that, with some small tweaks, obviously had these much deeper and larger payoffs. And I think that’s one of the real nuggets of wisdom here is that when you want to transform, and have better student outcomes, you want to narrow these opportunity gaps for students from certain backgrounds or experiences when they come in. Oftentimes, you don’t have to throw away what you’re already doing. Oftentimes, you already have nuggets of really important things that you’ve built over time. And all you need to do is to work with somebody who sees things just a little differently and can tweak them with you. And I think that partnership with Elizabeth and Makita was really critical for me. They were able to take things that I already had, and show me a different way to just modify what I was already doing. That messaging would go out to every student after that time point. When you see an effect that size, it almost becomes ethical to in fact, it is ethical, in my opinion that you don’t preclude any student from experiencing it. So that should be the way that everyone should be communicating with their students, in my opinion. And you want to make sure that it’s authentic as well, because I started with a set of materials that I had already written and sort of came from my heart, my own experience, that made the authenticity, I think, real whereas if I would have tried to have said something to students that I really truly didn’t believe about their abilities, I think they would have sniffed that out in a heartbeat. And we probably wouldn’t have seen the effect that we did.

Rebecca: It’s amazing what a little bit of intention can do in the design of something, especially because we often don’t realize how big of an impact a simple communication might have. We almost think of it as like a throw away, or a quick thing you might be doing, but you add a little intention to it, and it starts having a really different impact.

Elizabeth: Yeah, I think you mentioned earlier that students might not read the email, you might think that they’re not going to engage with it. And we also thought that, [LAUGHTER] so we think that in designing this intervention, we were really drawing on a lot of the intervention research that had been done previously. So research on wise interventions, mindset theory, all of that was integrated into the design of the message itself. So we started with Bill’s original emails that he was sending out to students and the messages that he had already curated for his class, and timing it in a time in the class in which students are receptive to that information, I think, is also really key. So we don’t know this empirically. But I would guess that if we were to send that email a couple of weeks before the exam, it would not have the same kind of impact. We send it directly when that performance information was posted online. And that’s when students are questioning themselves. They’re saying, “What does this score mean? Does this mean that I’m not good at this? Does this mean that I can continue? Does this mean that I’m smart or not smart? …and that uncertainty, that is the right time to infuse your message as an instructor to counteract those uncertain messages about ability. So I think the timing really matters. I think authenticity, like Bill said, really matters. Makita and I have been working with other institutions to implement a similar type of intervention with instructors. And before we did that Makita did a bunch of focus groups with the students at these institutions and showed them the messages that we used in Bill’s class, and we ended up changing them quite dramatically. The students gave us feedback like “nobody here would ever say something like that,” or “here’s what you need to say to reach students here in my class,” or “here’s what I will be receptive to.” So students in different contexts are going to respond to different messages. And so you really do want it to fit your own class, how you would speak, how you would talk to your students, the needs of the students in that moment might be different from the class they’re in. So it really is not a copy and paste type of situation where you were gonna copy what we did here and put it in your class. I think you do need to kind of go through the process of generating it for yourself and your own students.

Rebecca: I could imagine there would be an element of needing to evolve it over time, too, as your student population changes, and as you change as a teacher.

Elizabeth: Yeah, absolutely.

John: One of the things that I think is particularly nice about this study is that we see higher DFW rates by first-gen students in STEM classes in particular, and so many students end up leaving STEM fields that they intended to major in in their first couple of years. And that’s where the rate of return to education is the highest. And in general, first-gen students come from lower socioeconomic groups and lower family income groups, and if we do want to see more equity in outcomes, getting more first-gen students into the STEM fields could do a lot to help reduce some of those equity gaps in terms of overall outcomes.

Elizabeth: You mentioned the long term nature of these effects. I think it’s a great question. I think it’s something that we’re currently looking at in a project that we’re doing that’s funded by the National Science Foundation, where we’re looking at how these interventions in an introductory course can change career trajectories, that when you get a B instead of a C in a major introductory course, that’s gonna mean something different to that student, that’s going to set them on a path of following that next course sequence, that next step. So that’s the question that we’re asking and the research that we’re doing now, and hopefully, we’ll have some answers in a few years.

John: Peter Arcidiacono at Duke had done a study on the decision to stay in STEM fields. And he did look at some of these questions in terms of the effects on major choice based on the grades that were received. We can share a link to that study in the show notes. In the IRB proposal, did you include the possibility of following up these students in terms of their outcomes a few years later, in terms of retention rates for the students?

Elizabeth: That’s a great question. I believe in terms of IRB, we had students consent to their academic records. So we should be able to go back and look at transcripts from the students who participated in the class. We could look at course taking, we could look at graduation rates, we could look at major. If these students were freshmen/sophomores in 2021, then we’re probably around the time where they would be finishing up so maybe in the next year we could.

Bill:I was just thinking the same. Now’s the perfect time to go back and take a look, because we’ve had enough time since the intervention to see what’s happened. And actually I was gonna go back to one of your earlier points. You were talking about statistics earlier. Basically, this is really the only way that we’re going to increase the number of STEM students going forward. If first-generation students at WSU are 40% of our population today, that means if we don’t improve their outcomes, then we’re going to continue to see the declines in STEM outcomes and STEM graduates that we are trying so desperately to fight against, and also, we’re not going to have the representation in our STEM graduates of the population of the US that we want to see.

Rebecca: I think in addition to the growth-mindset piece, I’m sure that the authentic nature of the messages also just showed deep care. And that can be really important for first-generation students who need that ally or advocate for them in their college journey. So I can imagine those messages coming from Bill in Bill’s voice that make them feel like “Yeah, you can do this, I care about you,” from that perspective is also really important.

John: And there was a study done by a couple of economists, entitled “My Professor Cares,” where they did follow up students four years later, and they did find a significant effect in retention rates four years later by a similar type of intervention, I don’t think it was quite as much growth-mindset focused, but just sending a signal that the professor cared about the students outcomes, a very simple message, made a significant difference in student’s grades as well as longer term outcomes. So it’s nice to see more studies of this, especially focused on issues such as growth mindset.

Elizabeth: Yeah.

Rebecca: So we always wrap up by asking: What’s next?.

Elizabeth: So Makita and I and our lab have been working for the past couple years on following up on these data in different institutions. We used this study as pilot data in a grant application. And now we’re sort of scaling up to other institutions. We have a project with a really large HBCU, a Hispanic-Serving Institution, WSU is continuing to be involved as well, an Asian and Pacific Islander Serving Institution to where we have a number of instructors who vary in different characteristics, different class compositions, and we’re looking at these instructor messages in an experiment in these different contexts to see what matters. Does it help students develop their own growth mindset? Does it change their behavior? Does it work in some contexts, but not others? Do the instructor characteristics matter? So far, in the past two years, we’ve collected data from around 10,000 students who have been involved in this intervention project, and we’re currently in data analysis phase. So stay tuned for future studies and papers that come out of that data set.

John: Makita?

Makita: Well, what’s next for me is Elizabeth and I are actually going to meet right after this to do a debrief and to discuss some of the analyses that I just ran on a study that will be involved in my dissertation.

Rebecca: And how about you, Bill?

Bill:For me, I think about: “How do I proselytize about this work?” Because I think about what little changes make a big difference in an institution. We talk a lot about student retention, we talk a lot about student outcomes, we talk about creating the global citizenry that we want to see emerge from higher education. And the one experience that every student has in common, whether they are a first-generation a continuing-gen, or from some other background, is they have to take classes, and they have to be in classroom environments every day of the week, for the most part, over a course of four years. And I think about what would it look like to have interventions like this, that have evidence that back them, that students get inoculated with repeatedly over multiple time points with different voices, and that faculty engagement in the process of helping our students become better, to find success and to move forward in their careers? To me, that’s the magic of this. I’ve been talking to other people about it and I think about what would a curriculum look like that had small things like this? And how could it transform what we do? And I think that’s a vision that I can buy into, and I hope others are gonna come along for that ride.

Rebecca: I think the scope of the interventions that you’re talking about, that have this kind of impact, are really easy to buy into. Because usually you’re thinking about some drastic, giant investment of time, is what most people are thinking about. And that often is what prevents action from happening. So I’m excited, excited to share this and to hear what comes of your bigger study.

John: And one thing we appreciate is when we see things like this posted on social media so we can hear about it [LAUGHTER] and we can share it out a little bit further. [LAUGHTER] And I hope you’re presenting this at various conferences to reach a larger audience.

Elizabeth: Yeah, we will be presenting some of this work at SABER this summer. So that is the Society for the Advancement of Biology education research, and we’ve presented it in some other psychology conferences throughout the past year as well.

John: Excellent.

Bill:And I gave a plenary talk at the summer educational meeting for the ASBMB, American Society for Biochemistry and Molecular Biology. So I gave a little preview of what we had found. And that generated a lot of discussion and excitement, actually people asking for additional resources and ideas and things. So I think there’s a receptive audience out there.

Rebecca: That’s wonderful.

John: Well, thank you for joining us. It’s always great talking to you and it’s great to hear about this newest study and to meet you, Bill. We’re looking forward to hearing more about how things go in the future.

Elizabeth: Thank you so much. Thanks for having us.

Rebecca: I hope you’ll reach out when your next study is done. [LAUGHTER]

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

348. Active Learning Initiative at UGA

While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, Megan Mittelstadt and Leah Carmichael join us to discuss the active learning initiative at the University of Georgia that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces. Meg is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia.

Show Notes

Transcript

xJohn: While there is compelling evidence that active learning results in increased student learning, these initiatives often face resistance from students and faculty. In this episode, we explore a large-scale active learning initiative that provides professional development for faculty, active learning training for students, and for the redesign of classroom spaces to better support this initiative.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Megan Mittelstadt and Leah Carmichael. Meg: is the Director of the Center for Teaching and Learning at the University of Georgia. Leah is the Director of Active Learning, also at the University of Georgia. Welcome Meg: and Leah.

Meg: Hi.

Leah: Good to be here.

Rebecca: Today’s teas are:… Meg, are you drinking tea?

Meg: I am, I’m drinking double chai. Tea is a big deal. In our center. All of the people who work here have favorites. And we provide tea service at every workshop and I checked this morning, the most popular tea at our workshops is lemon ginger.

Rebecca: Oh, that’s good to know. Another teaching center full of tea. [LAUGHTER]

Meg: Yeah, we even have the teas of folks who don’t work here anymore that we keep stashed, just in case. So, Leah, you know, Colleen. Colleen drank…

Leah: Yeah.

Meg: …caffeine free peppermint tea, and we still keep it stashed, even though she doesn’t work here anymore. [LAUGHTER] Just in case.

Rebecca: How about you, Leah?

Leah: Well, I am a diehard coffee drinker. And so I am enjoying a cup of coffee. But my best friend is a potter, so I have her beautiful mug here to enjoy it with.

Rebecca: Wonderful. How about you, John?

John: I’m drinking a black raspberry green tea today.

Rebecca: Oh, that’s nice. And I’m back to my blue sapphire black tea.

John: Black and blue?

Rebecca: It’s black and blue. [LAUGHTER] We’re recording this at the end of the semester. [LAUGHTER]

John: We read about the active learning initiative in a recent Chronicle article and we invited you here to talk a little bit about the active learning initiative at the University of Georgia. We were really impressed to see that the university had budgeted over a million dollars a year for five years as part of the university’s Quality Enhancement Plan. This is a really large investment in effective teaching practices, and it’s especially impressive that this is happening at an R1 university. How did this initiative come about?

Meg: Yeah, we’re really excited that this investment is happening. And it kind of stems from an initial pilot that happened in 2018. There was a presidential initiative on active learning that resulted from a faculty task force. And so in 2018, our president allocated a million dollars to active learning programs, as a pilot. I feel super lucky to have a president who cares so deeply about the teaching enterprise at UGA. It’s something that you don’t see at every institution. And so I really appreciate it, value it, and count my luck there. Our president allocated a million dollars in 2018 for an active learning course redesign institute and a pilot classroom infrastructure renovation, and also the creation of a one-credit course for our students called Becoming an Active Learner that’s deployed by our Student Success Center. And that first year, we thought, “Gosh, it might just be a one-year investment, so we really got to make this count.” And so all the folks in the center sort of got together and backwards designed the programming for our first Active Learning Summer Institute, which by the way, was one of the most gratifying and enjoyable experiences of my career to get five educational developers in a room and get to go totally meta and apply the backwards-design process to the creation of a Course Design Institute. We nerded out. [LAUGHTER] We had like the room where we had all of our notes on the walls, and it was sort of our Beautiful Mind room for the Course Redesign Institute. We loved it. And so we designed this institute for 2018, and it went quite well. The folks at our institution who applied were way more applications than we could accommodate in the Institute and the folks who got into the Institute had a good experience. And we were able to show a couple of courses that had been redesigned seeing some really significant positive outcomes for students, both in terms of their depth of understanding, but also in some of the metrics in courses like numbers of Ds, Fs and withdrawals from courses decreasing in a few really key courses like our introductory mathematics course sequence. And so we were able to go back to the President and the Provost every year after that and say, “Okay, here’s how the course redesign Institute went, could we get funding to do it again?” And so we continued to do it every year. And this continued for a couple more years. And when we got into the 2020 timeframe, UGA was preparing for our next reaffirmation through SACSCOC. And for SACSCOC reaffirmation, an important component of that reaccreditation proposal is the creation and proposal of a Quality Enhancement Plan (or QEP). And so a committee was assembled to come up with what that topic might be, and they looked at programs that existed on campus as well as the university strategic plan. And luckily, by this point, our strategic plan had been drafted with quite a few key performance indicators related to the adoption of active learning practices. And so they identified this opportunity to scale the Active Learning Initiative at UGA. And so they proposed this topic, it was selected and now the funding has been stabilized for some of the pilot programs that existed, but also some newer initiatives to0.

Rebecca: There’s a substantial body of work supporting the efficacy of active learning and increasing student learning and reducing equity gaps. And it’s so exciting to hear about such a big investment and the interest of faculty [LAUGHTER] having more interest than spots for example, But I know one of the things that sometimes happens when faculty decide to start implementing active learning in their classes is sometimes there’s some pushback from students, but I heard you say something about a one-credit course for students. Can you talk a little bit about that? And how that ties to preparing students for this endeavor?

Meg: Yeah, so because we had this course from the get go, there was already this idea that reaching out directly to students and building their capacities and dispositions as learners to amplify their success in an active learning environment is something we might ought to think about. Of course, we also think about it from the faculty side too. One of our most popular sessions in the Course Redesign Institute is a session on student resistance to active learning, and we rely heavily on Tolman and Kremling’s Why Students Resist Learning for that session and then talk about some of the instructor facilitation and explanation strategies that instructors can use to ensure that student resistance is not cropping up and to help prevent barriers to student engagement. I think what’s really cool about Tolman and Kremling’s book is that they reframe “student resistance” to “barriers to engaging” and so we work with faculty to reframe student resistance as “Where are the barriers to engaging that we need to address as instructors?” And then on the student side too, the philosophy is “Where are there barriers to engaging?” And then “When can we do some really important work to help students appreciate that engaging in active learning is a really actually durable practice for them?” It’s really going to benefit them long term. And since that idea was already in the mix, as we were coming up with the programming for the QEP, there was a really large committee, I think, a 33-person committee of faculty, students, and administrators who proposed the programming that would go into our Active Learning Initiative. And one of the things that they wanted to think about was the role that metacognition plays in active learning. That was identified really early on because we were already aware of the student piece, thinking about metacognition as a key component of active learning. The committee spent quite a bit of time unpacking: “What are the learning dispositions involved in student metacognition? And what are the learning dispositions involved in lifelong learning and other benefits of active learning? And where’s the overlap?” So we looked at things like the AAC&U value rubrics on lifelong learning. And I think the other one was, I want to say, it was like transferable learning, I’m getting the title wrong. But anyway, there were two AAC&U value rubrics that we looked at, as well as some of the literature on metacognition and literature on lifelong learning. And our assessment team identified four student learning dispositions that seemed to exist across all of the categories that we were looking at: students acting more curiously, students engaging with initiative, students becoming more reflective in their practice as learners, and students drawing connections between what they were learning in their active learning courses and developing other knowledge and skills. And so we use this framework of these four learning dispositions to really drive our student programming. And I’m going to pass it over to Leah, so she can share a little bit more about how we’ve expanded beyond this one credit hour course to a whole portfolio of student facing initiatives.

Leah: So, first of all, I would like to say with the instructor development piece, I was an instructor on campus, a lecturer, and was in that first cohort in 2018, an international affairs scholar. And it really was the first time that I had had such a professional development opportunity that felt like it truly revolutionized the way I taught, the way I thought about my students, the way I interacted with my students. And so whereas I was taught to transform one of my courses, I ended up transforming them all. And when I was able to be on the team for active learning, I signed up, it was really excited to do that. So it was an amazing experience. For the student piece, what I really love is that we have not just gone to the instructors and said, “Alright, it’s time to consider something different and then your job is to not only do all this hard work to change and rethink your courses, but then can you do us a favor and sell it to the students?” That’s just such a burden and very difficult. And so within CTL, they do a great job of training us in the Institute to go in first day and frame the course, to craft our syllabus language in a way that would potentially be able to introduce this course before even day one and to help decrease that student resistance. So that was a really important piece. But we also would have these teams of students, these just superstars on campus, who are also part of the programming for active learning. One of my absolute favorites is the peer learning assistants. This is an on-campus paid position for students. They apply and they are able to either return to a class or return to an instructor whom they have worked with before. And they attend a class where they facilitate active learning approaches and activities with the instructor. So the peer learner is responsible for helping with the planning of the course and preparation with the instructor implementing it and then also of course, following along with the course materials. And this is a key: this near-peer experience has been key in allowing the burden to be shared in introducing active learning within a classroom. And the students look at these peers, maybe a semester ahead of them sometimes, maybe years, and they start to see these mentors that are built in. And these students know and care about active learning. They see why it works in the class and they’re returning to the class. And we have found that those classrooms have been just the most wonderful place to see this cultural change among students. We also have active learning ambassadors. And they are trained to do outreach. They do demonstrations for faculty. One of my favorite moments is we have an active learning summit every year open to the public. But we have an active learning summit in which they run a showcase where the faculty walk in, and there are students behind tables with interactive material demonstrating different active learning techniques and how they see it in their class, how they use it for study skills, and how they would enjoy instructors being able to incorporate it in their classroom. The faculty love it. They stop, they ask questions, they think, “Well, what about this? Well, this would be difficult.” And then students give that experience from their perspective of how these techniques work for them. And we see that the faculty really crave that and really enjoy being the learner in that moment, and learning from a student about what techniques work best for them.

Meg: And Leah, I think one thing that’s really unique about our quality enhancement plan that I haven’t seen at other institutions yet, or if they have them, I haven’t seen them, so mea culpa. But one thing that’s really cool about our Quality Enhancement Plan is that we have a partnership with our Office of Student Affairs where they’ve designed some programming through our residential life curriculum to embed active learning in the co-curriculum too, so that students are experiencing and talking about the value of active learning not just in the classroom with each individual instructor, but experiencing active learning outside of the classroom in the co-curriculum. Students who are leaders on campus are trained in active learning facilitation techniques, so that when they go in to serve as leaders of student groups on campus, they have some of these active learning tools in their back pocket that they can deploy and experience on the other side as well. So we’re really seeing active learning proliferated, not just in the classroom, but outside of the classroom too.

John: It’s nice to see such a holistic approach. In so many cases where we’ve seen active learning initiatives, it’s usually just designed for faculty development, to go and fix all those issues and challenges. And by approaching it from all these angles, it seems to be a really effective way of addressing some of the challenges that faculty face when doing this. One of the challenges, though, that faculty sometimes face when they introduce active learning is they go into a classroom where the seats are either bolted down, or are very rigid. I think part of your plan involves some type of infrastructure changes to support that. Could you talk a little bit about how classrooms have been redesigned as part of this process?

Leah: It’s such a key aspect of it, I have the anecdotal experience where I graduated from the Active Learning Summer Institute, went into a classroom ready to be highly engaged and interactive and set up have a highly mobile class design only to find that I was in stadium seating, with fixed seats and I thought, “Oh, no, what am I possibly going to do?” And so we have three foci, within the initiative. And we’ve mentioned the instructor development piece. So key, but as you said, it can’t stand alone without support for the other two. We have the student piece, we also have the classroom piece. And so our goal is to every… usually it has to be in the summer, when it’s much quieter… every summer, we just push for classroom enhancements across campus. And we have three criteria in mind that we see as not necessarily perfect for every single classroom situation, but the baseline for which we want to capture for an active learning classroom. And so the first is: does this classroom have mobile furniture? Are we able to move students around? Could an English professor have a roundtable discussion? And then within the 20-minute transition time, could my students go into that same classroom, which happened last semester, in a highly mobile classroom and be able to set it up into a simulation of roleplay thing for my international politics course? So do we have classrooms with that mobile furniture is the first thing. The second is, do we have student collaboration tools? And these do not have to be high tech. In fact, we ran the survey, and students and faculty said just give us more dry erase boards. We don’t need the fancy, fanciest of smart screens. Please just give us the basics. But let us work on a project huddled over together and be implementing and creating something new. And then finally, it’s just general space, do we have enough space that we are able to not only move the furniture around, but move people around in an effective way. And so 25 square feet per student is kind of our ultimate goal. And so we looked around and I was actually just running the numbers recently. And I think on campus we have, depending on how you measure our classrooms, between 400 to 600 spaces, it depends on if you count labs, and these other learning spaces, about 82% have at least two of these criteria. And so we’re getting there. We’ve made a real push and we’re trying to say: “Yes, instructors, please change but also know that we’re going to meet you with an environment that facilitates this kind of approach to learning.

Meg: Well, I also have to interject and say that the stance of the CTL is that active learning can happen in any classroom no matter the features. And so of course, like a good backwards design course or re-design institute, we always start out with the situational factors of a course. And that might include what’s the likelihood that you’re going to be in a fixed-seat auditorium or fixed table, but moveable chairs, and think about the choreography of that, the active learning techniques that you’ll adopt based on that. But in Leah’s example, in particular, if I remember correctly, Leah, you had a student in a wheelchair in your course the first semester after you redesigned your course. And when you were in the tiered auditorium, that meant that there was a specific area where the student in the wheelchair could sit and they couldn’t move through all of the case study roles that you had available. And so you were able to advocate for a different classroom, you obtained one, thank goodness, and everyone could move through a flat classroom instead. And so sometimes things crop up, even if you have planned to be in a fixed-seat auditorium, that you have to be able to modulate on the fly based on the characteristics of your students each semester too.

Rebecca: Can you talk a little bit about whether or not your program has focused on specific programs or if it started with specific programs and its spread to different programs or what your strategy is for involving faculty across disciplines?

Leah: One of the most important parts that I really liked about this initiative is it’s not a mandate, we are not requiring any instructor to adopt active learning through a mandate. Instead, it really is a support, educate, encourage. And so we really have not been focused on okay, this unit by X year needs to have Y outcome in terms of an active learning goal. But in terms of strategy, what’s nice is we can start to see the people that walk in the door to find the active learning Summer Institute. And the high demand that first year was a nice little kind of bellwether that, okay, there is something going on here. When you have more people that can fit, and that’s consistently. The CTL needs to keep it kind of a quality curated experience with a cohort that goes through and has to do these really difficult re-education, understanding education differently, and then of course redesign. Such a dense set of tasks. So with this cohort like experience really requires so much of the faculty. What we started to notice, as we looked around and realized that instructors from different schools and colleges were coming to the Summer Institute, and we have seen that as the CTL has expanded beyond the Summer Institute and provided workshops, both standalone workshops, as well as a course redesign experience outside of the institute. We see hundreds of faculty each semester coming into the programming of active learning. And really we have, I believe, 19 schools and colleges that have undergraduate programs, and we have representatives from each that participate in this programming.

Rebecca: That’s wonderful.

John: What type of training do faculty receive as part of the institute and other training that you provide?

Meg: It was initially just that summer Course Redesign Institute. And what we started hearing from faculty as we would get these applications every year is the deadline would pass, and we’d hear from some faculty who would say things like, “Oh, I really, really want to do the institute. but summer is my field research season” or “I’m in the clinics in summer,” or “I have childcare duties that I can’t get away during that time of year.” And so there started to be some sort of advocacy for other on ramps into the programming, so that when we had the opportunity to stabilize the funding through the Quality Enhancement Plan for active learning initiatives, we started thinking about how can we make this program one that faculty could find the on ramp that is appropriate for their needs? Is it a faculty member who’s been using active learning for a long, long time and they really want to brush up on their skills in one particular area? And so that might look like engaging in a specific workshop on something like facilitating group discussions in a large lecture environment, or looking at student metacognition and how that plays into a particular student developmental level that they might teach in their courses? Or does somebody want to do the whole workshop series and Course Redesign Institute all the way through, but they can’t do it during the summer, or anywhere in between. And so what we ended up with is a workshop series, we have 12 standard workshops that we offer over time: four fundamental workshops, and eight special topics, workshops, and we rotate through those. We offer a couple of each every year. And then we have the active learning summer institute that happens every May. And then starting next academic year, over the course of the spring semester, we’ve designed a course redesign experience that takes the course redesign elements out of the active learning Summer Institute and decouples those elements from the active learning workshop components of the Course Redesign Institute that we currently have, so that faculty can upskill through the workshop series. And once they’ve taken eight of the 12 workshops, they can apply for the course redesign experience over the spring semester. Through that course redesign experience, they come and meet with us for a couple of hours every three weeks or so over the course of the spring semester. And the idea is that by the end of that course redesign experience and workshop series completion, they’ve recapitulated all of the experiences that you might have during the multi-week summer Course Redesign Institute. We also provide different communities for folks to engage with. We have several robust Faculty Learning Communities on campus. So we have multiple faculty learning communities related to active learning, one specifically coming up this year that Lea generously funded from the Active Learning Initiative budget for folks who are looking to engage in SOTL, Scholarship of Teaching and Learning, on their active learning endeavors, as well as specific faculty learning communities for the folks who have peer learning assistance in their courses, or who are engaged in the work after the course redesign experience, for the alumni of the Course Redesign Institute. We also have opportunities for faculty who are highly engaged and master teachers to serve as active learning mentors to their peer faculty. So we have three active learning mentors every year who work with the folks who’ve graduated from the course redesign institutes to give them opportunities for office hours with their faculty peers for micro mentoring sessions on specific topics of interest. They also create supportive resources. We have similar mentors for our PLA faculty, our peer learning assistant faculty. And for those who are serving in those mentoring roles, the Center for Teaching and Learning provides them with opportunities for additional coaching and support on instructional development, educational development, instructional coaching practices that they might bring to bear in those consultative sessions with their peer faculty, too. So the goal here is that we have some workshops that are entry level, some special topics, workshops, the course redesign experiences, some stuff for the master teachers, who are really wanting to be mentors and wanting to kind of serve as role models within their spheres of influence and for others who are moving through this programming, and we’re helping them build capacity in those areas, too. And then, of course, we have faculty on campus who have been working in active learning for years and years and years, and who published on active learning, who are real role models nationally. And so we really try our best to honor their expertise and bring them in to help frame the ways in which we deploy our programming, but also to serve as guest speakers and to engage with folks who are newer to active learning so that we can help provide opportunities for that cross-collaboration and multidisciplinary discussion.

Rebecca: I love the range of options you have available, and really how much you all have embraced the idea of flexibility for your teacher learners, just like we might want in our classrooms as well.

Meg: Yeah, yeah, that’s right.

Rebecca: Can you talk a little bit about some of the active learning techniques that faculty have adopted after going through the institute or other variations?

Meg: Yeah, so as you’ve heard now, our summer institute is three weeks long. And so not only are we giving sessions on active learning, but we’re modeling active learning throughout. And so folks see a lot of different techniques that they might not have seen before. And they’re trying to identify which things that they’re seeing, which instructor moves they see, that align with the needs of their students, of the needs of their course, and their teaching persona as well. So as you might imagine, folks to leave the institute, select different things that are authentic to their practice and their situational factors for their course. So we go into the Institute, which actually begins today for summer 2024. We’re recording on the first day of the Active Learning Summer Institute. So we’re getting the anxious energy of meeting the new faculty. We start out on the first day of helping faculty appreciate that we don’t see active learning as a homogenous set of teaching activities, we’re not going to give them the recipe. We want them to discover the recipe that works for their course. So our QEP is sort of predicated on the notion that the extent to which students are actively engaged in thinking and applying what they’re learning is of far greater importance than the particular active learning instructional strategies used. And so to that end, the course redesign process helps faculty first hone in on the key enduring understandings and learning outcomes at the course level they want their students to leave with and identifying key learning bottlenecks or threshold concepts that they’ve identified that students tend to either have a tough time with, or they’ve really had to work hard to help students grasp that concept in order for them to move from novice to expert in that field. And then we encourage faculty to design their active learning interventions in alignment with those key features of the course so that active learning is more like the trunk of the tree rather than the ornaments on the tree. And our philosophy there is that when pressed for time, the ornaments are easy to cut, but if the act of learning is baked into the backbone or the trunk of the course, that’s just inherent to the identity of the course and it’s not something that’s going to be cut for time. And also you’re putting your energy into where the biggest impact is going to be for your learners..

Leah: If I may add, this was the revolutionary part of being a participant in the Active Learning Summer Institute. I thought I was going for the tips and the tricks, I’d get my little suitcase of ideas and I would just kind of put it all in there through the Institute, and then graduate and use all those as those ornaments as Meg: described it. And instead, I found myself looking at my course and realizing that the backwards design principle had me throwing a lot of content at my students, and not really aligning the activities in the course towards the student learning outcomes. Once I was able to do that hard work, then the excitement of “Ooh, okay, well, which tips and tricks, which kind of active learning engaging styles would be most useful for this.” And that’s just a really exciting way to do it as an instructor.

John: Most instructors, as they come through their educational process, have not been exposed, particularly not in graduate school, to active learning techniques. How do you address issues of faculty resistance to completely changing the way in which they’re approaching the design of their classes.

Leah: It takes a while, [LAUGHTER] having been one of the resistant faculty, I’ll speak to it from this perspective, I think Meg: referenced the former colleague Colleen, and she was my mentor going through this institute. And I remember her asking me to write all the content that I felt was so essential to get the students to to learn throughout the semester, and I distinctly remember this moment where she said, “Okay, let’s put that in an envelope, all that stuff that you just wrote down, we’re just going to put it in an envelope, and we’re just going to set it over here, it’s still very much matters, don’t worry, it’s not going away. It’s all still important.” And then she said, “Let’s really think about the future student. Who are you teaching? What would be a success?” And I think that’s a moment where I could take a deep breath, as a faculty. At first, it was scary and fearful. But it’s a moment for the first time you take a deep breath and go, if I miss a case in international law, they are still going to thrive in this world. I’m focusing on what skills they actually need to thrive in this future career. And so it’s not about that specific case, it’s about the skills. And once I was able to realign that, and it’s so important to be sitting in a cohort, it’s not an isolating experience, you’re sitting in a cohort with other people doing the same reformulating of their whole profession, with mentors from the CTL, helping and encouraging them. Without that sort of ecosystem, it would be a very lonely and harder… just slow… process, maybe, even if you are willing to change. So I think the quality of that instructional development experience within the CTL cannot be replicated. And then I hope the other two kind of prongs of the initiative, once they leave the institute, they’ll still have the mentorship for the next year with CTL and with the faculty mentors. They then can leave and say: “We have the classroom engagement things.” We are also trying to change the environment to help you and we want your feedback. We are always asking those active learning instructors, what can we do differently within these classrooms? And then also the students… we’re engaging the students. So you’re going to walk into class, and they’re going to know what you’re trying to do, and that it’s recognized as a really good thing to do across campus. So it’s the ecosystem that will create more empathy and support and therefore reduce the resistance.

Meg: Yeah, so Leah mentioned, this idea of you have this mentor that exists for the year after the institute. And I don’t know that we explained that. That’s a pretty unique feature of our Course Redesign Institute. And it comes somewhat from our Center for Teaching and Learning has quite a bit of tradition around investment in longitudinal cohort-based programming. So we have a lot of one-year, two-year, multi-month cohorted faculty development experiences, because we find that that interdisciplinary conversation is so helpful for helping folks build connections and shared disciplinary content, and teaching practices with one another in ways that proliferate those great ideas across campus and seed innovation. And as we thought about the ways that our faculty fellows…we call them Faculty Fellows Programs at UGA… ways that our faculty fellows programs were successful, were in that sort of longitudinal support that was available. And we wanted to replicate that as best possible with our Course Redesign Institute. And when we thought about the outcomes of our institute, and how we wanted people to leave the institute, be prepared to deploy a redesigned course in the coming academic year. And we started thinking about what that experiences like as a teacher, and we thought, “Gosh, people are going to leave the institute, be all excited, go back to their department, where they might be one of few who’s engaging in this sort of robust active learning commitment, especially in the early years of the institute.” How can we support them in being successful when they go into their first class and leave the class and go: “Ooh, that didn’t feel like it usually felt I wonder if it went well?” or they do their first robust activity and maybe it doesn’t go quite as they were expecting? How can we be there for them in those moments? So as folks leave our Course Redesign Institute, they’re each paired with a mentor from the CTL. And that mentor is their touch point for the year following the institute, so that when they have that class session that throws them for a loop or they get some feedback from students, that they’re not quite sure how they’re To respond to it, they have a touch point that they can go to. And it’s someone who knows their course intimately, who’s worked with them through the redesign process, is aware of the ways in which their course has changed, and is there to provide them with that support. And also that mentor is the person who performs… we call them mid-semester formative evaluations… but at other CTLs they’re called small group instructional diagnoses. So at the midpoint of their course, someone from the CTL comes in and talks with their students to focus group them, to collect some feedback on how the course is going so far, and then works with the faculty member at closing the loop on that feedback to help students appreciate that they are such an important component of the redesign process too, because the instructor is engaging in this two-way exchange of information with the students both on the content and on the operational organization of the course and refining the course in response to that feedback. So we want the students to appreciate that their experience and the way that they’re learning and what they think could be happening to support their learning even better, how that could be incorporated into the course as it continues.

Rebecca: I really love the depth of the culture change components that are built into this initiative, from feeling like you’re part of something bigger, the fact that it’s voluntary, your support structures, it’s holistic, both Academic Affairs and Student Affairs, there’s opportunities for feedback loops. Change can be really difficult, but when you’re a part of that bigger network, it can be helpful. The timing of this was really interesting when it kicked off…so, a couple years before COVID, so you had some time to establish the roots, and then COVID erupts, AI erupts, there’s more first-gen students, we’ve got a lot going on, and there’s a lot of change happening. How do you help your faculty avoid burnout with so much change and maintain this momentum?

Meg: Yeah, great question. You’re taking me right back to the pandemic and that experience of everyone just doing the best emergency remote teaching you could do. And one of the things that I remember from that period is faculty who had gone through the Active Learning Summer Institute mentioning that they were so glad that they’d gone through the institute because it made pandemic teaching easier. They had recently conceptualized of their course learning outcomes that identified the most important content that they wanted to prioritize, they had a sense of some lesson plans for how to deploy these things. And even though the institute happens in person, because of the way we were talking about how you would make adjustment to different things in different settings, they could more easily conceptualize of how to engage online or asynchronously with students using active learning principles. So we had this early indication that the folks who’d experienced the course redesign process felt better prepared for emergency remote teaching and pivoting quickly. We also found active learning to be perfectly timed as an initiative as a response to post-pandemic disengagement. So when students were coming back into the classroom and members of our community were craving that deep intellectual engagement and community with one another, but then in some instances, were a little bit disappointed when it didn’t materialize in the ways that it might have before the pandemic, we found that faculty were coming to the institute, saying, “This is something that I’m hoping can meet this need that I have to engage students in rich conversation or to engage students in deep intellectual engagement in this learning community, and how can I address this through active learning?” And also, we’re lucky to have a large number of faculty champions for active learning on campus, folks who’ve been using active learning for a while, and in various departments across the institution, not just in specific pockets. And so they’re serving as role models and mentors to the faculty in their unit. And a message that those faculty champions were helping us reinforce is that teaching with active learning is a joyful way to teach. And so oftentimes, when these faculty champions, these mentors, were talking to folks who were saying things like “post-pandemic teaching is a little bit disappointing,” or “I’m feeling a little bit burned out because of everything that we’ve experienced through the pandemic, and through these various disruptions to higher education.” When their mentors were telling them, “Have you considered bringing some joy into your classroom and making teaching a more joyful experience for you and for your students? And guess what? You can do that through active learning. Let me tell you how I brought joy into my classroom through active learning.” We would see folks come to us and say, “I’m just really looking for some rejuvenation and some revitalization, and I’ve seen my colleagues just transition their course through active learning in a way that they’re just skipping down the hall after they leave their class, and I want that feeling. So I’m hoping that this will be that experience for me.” And so we had people coming to us asking for that, and we hoped that would be the case. And then I’ll just add that like many higher education institutions right now we’re in the midst of an evaluation of teaching culture change, too. And so we’re trying to adopt a more what we’re calling a three-voice culture of teaching evaluations that’s relying upon your individual reflection on your teaching some data from your students and the interpretation and contextualization of that data, as well as peer observation of teaching data and bringing that all together as sort of like triangulated data to provide some insights on our teaching, and then respond to that and have like a assessment loop on this. And so it’s this idea of a culture of continuous improvement with teaching where we’re able to proliferate this message of: “You’re not an award winning teacher necessarily on your first day of teaching, but this is a set of skills that you can develop over time. And also, you’re never done developing it.” And so this culture is helping to amplify the message that taking part in educational development is not just an expected part of being a teacher, but it’s an opportunity to continue your culture of continued mastery.

John: So one of the ways in which this is spread has been from faculty to faculty. One of the things I’ve been noticing a lot on my campus is that it also sometimes spreads from students to faculty. As more and more faculty move from faculty-centered teaching to student-centered teaching, students become a little bit more vocal sometimes about expressing what works for them and that will often result in faculty considering using new techniques that their students have recommended to them. Has that been happening as well at your institution?

Leah: That’s why we wanted to capitalize on this proactively and have channels of growth. And so again, we have little areas in which we have students really playing that intentional role. So rather than maybe a complaining, “I went into this class and I didn’t like it” kind of a customer satisfaction problem area, we really want to make sure it’s more that students, first of all, are trained in the active learning pedagogy. So my active learning ambassadors, as they are chosen, we then say, “Okay, here’s the wizard behind the curtain friends, come join us [LAUGHTER] and see what’s going on behind these intentional classrooms that you really enjoy and don’t quite know why you like it so much.” So we bring them into the understanding of what instructors are doing when they’re in an active learning classroom and why. And then we give them opportunities to then go and front face and tell other faculty, “this is what an active learning classroom feels like for a student.” This is why one of my favorite moments… we have, again, as I mentioned, an active learning summit every year in February, and there was an entire breakout session run by students who were peer learning assistants and active learning ambassadors. And they were talking about the study skills that they have seen through their training, or through their classes that really work in their discipline that they use outside of the classroom to kind of empower themselves to study and to learn. And what was wonderful is that the front of the room were students in a panel and most of the room or faculty and I found that the question and answer period, the instructors would raise their hand and say, “Okay, well, but I’m not in that discipline, but I’m thinking about bringing this into my class in X way, is this something that you would find effective?” And they were asking the students about designing their course to be student centered. And that’s when I think if we put those students that know what is going on, so it’s not a complaint section, it is more a reframing and a sharing out of we know what you’re trying to do and we really appreciate it, and here’s how it works really well, for us. It just starts to kind of change that conversation from being yet another instructor versus student dynamic, which I think we have plenty of those around campuses at any given time to becoming an instructor and student experience.

Rebecca: Having student ambassadors and student voices celebrating this work, is certainly a testament to the success of this work. And you’ve also mentioned some other moments of assessment. Can you talk a little bit about how you’re assessing this initiative?

Leah: There is a rich assessment team who is actually part of the larger Active Learning Initiative. And so I hope I do justice to their very nuanced framework that they have. We have macro level, mezzo level, and micro level indicators that we are trying to capture. At the most macro level demand is one of the earliest indicators of success for us. Are people applying to the Summer Institute? Are ambassadors applying for the program? How many peer learning assistants apply and how many instructors want them, just really kind of those basic quantifiable amounts for demand. And then we also are attempting to capture survey data across the university. We stop the students and say: “We don’t know if you’ve interacted with active learning in the classroom or not, we’re not selecting a given class, we’re just wondering what you would say if you observe a cultural change going on.” So we’re also trying to get these larger macro level indicators. Then at the mezzo level, we’re getting into those lifelong learning dispositions that Meg: referenced earlier, the curiosity, the initiative, the reflection and connection. We have artifacts that we pull from courses that have an instructor who’s either going through all the Active Learning Summer Institute, or who has gone through it already, so we do a pre- and post-analysis. And we have faculty scorers, who can go through these artifacts to see if we are capturing if the artifacts include a prompt about curiosity or a prompt about initiative. And then we are attempting to measure, and we’re working on different ways to do that, but measure if we’re seeing growth over time in these lifelong learning dispositions, and then finally, at kind of the micro level, we’re making sure that the disciplinary knowledge is being enhanced within this that are focused on active learning techniques. One thing we have yet to mention is yet another part of our holistic umbrella. This is why the programming is so large, but we have these active learning change grants. And this felt really good to hear for me as an instructor who went through the institute. And then it was about five years later that the next person in my department went through the institute. What these change grants do is it really allows those who go to the institute to go back to their department or their unit and say, “What if we change several classes at a time through some sort of infusion of deep learning and the CTO will support us, the assessment team will help us assess whether we’re doing what we’re saying we’re doing.” And so on this micro level, the assessment team is really starting to capture some pretty cool indicators, that when you kind of create a unit change, there’s a cascading norm it’s called in international affairs, where you change enough of the unit and enough of the actors in a given scenario, you can then really see that norm start to shift in a unit. And so at each of these levels we’re finding it’s still early. So we’re not saying “It’s a win, close up shop,” [LAUGHTER] but we are finding really robust indicators of early success.

John: Well, you’ve given us and our listeners a lot to think about, and it sounds like a model program that could be copied very productively at other institutions. But we always end by asking: “What’s next?”

Meg: We’ve started to hear from folks that are interested in proliferating initiatives. So, of course, we welcome conversations with folks who are interested in hearing about some of the elements of the program and brainstorming ways to deploy these on other campuses, lessons learned along the way, etc. I mentioned earlier that as folks crave these multiple on ramps, we’re deploying these additional on ramps over time. So starting in the fall, we’ve designed some of our foundational active learning workshops, we’ve taken those and created asynchronous online versions of those workshops that are going to be hosted in our learning management system. So they’ll be available on demand. So if a faculty member doesn’t happen to be available at the time that we’ve offered that workshop this semester, they can still engage, especially in those foundational workshops. And then the spring semester course redesign experience version of our Course Redesign Institute will be deploying for the first time in spring 2025. We’re really excited about that as well.

Leah: I would also say for your listeners, if there is an interest in kind of seeing what we’re up to and helping participate in engaging with this discussion, traditionally the active learning summit has been really focused on UGA community for the first two years, but in the third year, so February 2025, we are going to open up the summit to visitors and encourage them to be a part of these breakout sessions or propose different ideas that they would like to talk about. I know Rebecca mentioned earlier, the challenges of AI, we’re going to try to have a breakout session, though, that focuses on this: active learning best practices in the age of such change. We’ll send out a call for applications in the fall, and we’d love to have anybody who’s interested in learning more, I think it will be Thursday, February 27, and Friday, February 28, on UGA campus, and we welcome all those who are interested in attending.

Rebecca: That’s great. Thank you so much for joining us. There’s so much to think about and so much exciting work that you’ve all done.

Leah: Thank you.

Meg: Thank you so much for having us.

John: Thank you and we look forward to more conversations in the future.

Meg: Absolutely. Anytime.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

 

347. CATs and AI

Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, Todd Zakrajsek joins us to discuss how generative AI can enhance these techniques by providing more immediate feedback.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo.

Show Notes

Transcript

John: Classroom assessment techniques, initially developed at a time when chalk-and-talk instruction was the norm, helped to shift the focus from teacher-centered to learner-centered instruction. In this episode, we explore how generative AI can enhance these techniques by providing more immediate feedback.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Todd Zakrajsek. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published five books (so far) in the past five years. His most recent book is a 3rd edition of Classroom Assessment Techniques: A Handbook for College Teachers, with Thomas A. Angelo. Welcome back, Todd.

Todd: Thank you, Rebecca.

John: It’s nice talking to you again. It’s been a while. I think the last time was last summer.

Todd: Really? It seems like yesterday. Time goes by. It was last summer. You came to my office and recorded.

John: That’s right.

Todd: It was fun. You left me a really nice set of tea. Wooh.

John: And a mug.

Todd: Yes, I have a mug for the show. [LAUGHTER]

John: Speaking of teas, [LAUGHTER] today’s teas are:… Todd, are you drinking tea?

Todd: Every time I’ve been on this program I’ve been drinking tea and I’ve had some pretty exotic brands and blends but, I’m sorry, today I have water, the lightest of all teas.

Rebecca: It’s a very light tea. I have English Tea Time tea.

John: I have Lady Grey today.

Rebecca: That’s a good choice.

Todd: Alright, I think I came in third place.

John: We’ve invited you here today to discuss classroom assessment techniques. The first two editions to this book were written by Thomas de Angelo and K. Patricia Cross. In last week’s podcast, Tom joined us to discuss the origin of this book, where you joined as a contributor for the current third edition of this classic book. What was your reaction when you were first asked to join in this project?

Todd: Well, this is actually kind of a funny one, because it goes back to Linda Nilson and Teaching at its Best. And Linda was retiring, and the publisher, Jossey-Bass wanted to do a new edition of Teaching at its Best, and Linda says “Why don’t you see if you get Todd to do it and I will work with him on it and do it.” And we did it. And we got the book done a little bit early and on schedule, and it came out very, very well. So there’s a little plug for that book there. And it must have been about two months later, I had called them and said: “If you find another project like this, I love working with people like this.” And a couple of months later, I got a phone call and they said “We have another project we’d like to potentially have you work on” and I said: “Great, what is it?”… and they said: “Have you ever heard of the book Classroom Assessment Techniques?” And I held it together, and I said, “Well, yeah, yes, I have.” And they said, “Well, we thought it’d be helpful given what you do and what Tom does and everything, is maybe the two of you could work together on this book.” And so he talked a little bit about what there was and everything, and I hung up the phone and did a little happy dance because this book was one of the most influential books in my entire career. But the way the universe works is I ended up doing this one because of the work with Linda Nilson’s. And so that’s how I got started on this.

Rebecca: That’s amazing.

John: And, I agree, that is an excellent book as well. So you’ve got two classic books that you’ve worked on recently.

Todd: It’s really surreal, you know, a first-generation college student and every once awhile is kind of like I just can’t believe this is happening. But it was a really good project.

Rebecca: When did you first start using classroom assessment techniques in your classes? You mentioned that this book was influential.

Todd: I taught my very first course in 1987-1988. This book came out, second edition, 1993. And I was using it pretty much the moment it came out. Somebody had pointed it out to me and I didn’t know anything about classroom assessment techniques, and then I started reading through it and picking some out. I thought it was genius. Much like periodically, what you’ll do is you look at something and think this is ridiculous that nobody started doing this before. In some respects, it’s not that hard, right? You ask people questions, and you find out if they’re learning. It just took Tom and Pat to come along and say, “Why don’t we come up with strategies to find out if they’re actually learning.” So I think it was really influential for me because it kind of struck that chord with me that I always believed we should be finding out how much students were learning instead of focusing on teaching. But that’s not what was going on at the time, everybody was focused on teaching-centered educational practices.

John: What do you find most valuable about these classroom assessment techniques?

Todd: It’s interesting, because there’s a couple different levels. The one that’s really valuable, of course, is you find out if students are learning. Secondarily, you can find out if you’re teaching well, and you can change your teaching practices based on what they’re learning and based on what’s going on. But I’ll tell you the real reason I think this is really, really valuable is it addresses equity issues very, very cleanly. We walk into classrooms with all kinds of preconceived notions and stereotypes and implicit biases. And it’s really, really easy to think that a student’s struggling for a certain reason, unless you ask them. And so, for several of these strategies, I think it’s great because I’ve had students that have been practically sleeping through every class and then I started doing the classroom assessment techniques and he had some of the best responses, this one student did. And I would just totally blown away that this person was just processing it to such a level and I had a preconceived notion based on where the person sat and how they dressed and how they acted in class. I didn’t think they knew anything or they didn’t think they were learning and they really were. So I think it’s huge for equity and overcoming some of the biases we have in our classrooms.

Rebecca: That’s a really good example to underscore the importance of that. We had the opportunity to ask Tom what his favorite classroom assessment techniques are. So I’m really looking forward to asking you the same question: what are your favorites?

Todd: I’m one who says over and over again, in education, that we should always be careful about absolutes. Like, there’s never a strategy that always works. There’s never a type of student that’s always the best. Any situation can change. So there’s like 55 CATs in there. And I listened to the program with Tom and I heard him say that he didn’t have his favorites. And then he quickly picked out his favorites, and I’m sure it’s the same with his daughters. [LAUGHTER] So I have three girls, you know, they changed the favorites, I would definitely go with the standards of the muddiest point minute paper, those two are the big ones. Everybody’s done a minute paper, probably at some point. The muddiest point I like, which was the adaptation. And if you haven’t heard this show with Tom, make sure you listen to that, because he explains that very well. I think I like exam wrappers where you go in and kind of have the students talk about the exam experience. But for me, there’s several CATs that are based specifically on assessments. And it’s the thing that we almost never ask students for, and it’s the thing they can give us the best feedback on. So it’s like, when you have group activities, it’s an assessment on how well the students perceived that the activity worked for them. So instead of me just saying, we did a great group project today, you ask the students: “Was this a good group project?” There’s assessments in there about learning interests, what types of things you’re interested in, your study strategies, exam strategy. So you can ask students those things. And I think those make great classroom assessment techniques, to find out the processes that you’re doing, and how the students are receiving those.

Rebecca: Those are good opportunities to respond in the moment in your class, but also probably plan in the future too, to know what to do next time around, perhaps.

Todd: Yeah, I think specifically for exams. And one of the favorite questions I’ve always had on exams, because the students would get to a point where, and I got this, because of one of the assessment techniques I’d use early on, is students always were frustrated that there was something on the test that they studied really, really hard for, and it wasn’t on there. The problem was, it wasn’t always the same. So the students all had different things they studied. So because of collecting that information, and finding out from the students as they were frustrated, I started adding a new question at the end of every exam worth the same as some of the other questions there, was: “Please describe in detail something that you study that I didn’t ask you about on this test.” And students immediately felt better about the exams, they were happier with the test, because they didn’t spend four hours memorizing something that they perceived wasn’t valued on the test, even though the real reason for all of this stuff is life, not the test. But again, that was one from just asking the students: “What do you like and not like about the test?”

John: That’s a really great application of a technique that I hadn’t thought about, I can see how students would feel that their voices are being heard with that type of thing.

Rebecca: …or that their efforts were validated.

Todd: They got their points for their studying, and it just really seriously changed the affect of the class.

John: In the last podcast we did with you last summer, we talked a little bit about how teaching has changed from the time when you and I both started teaching, and this book was written back around the time when you and I were both relatively new at this. What role do you think this book has played in that evolution of instructional practices?

Todd: Well, first of all, I do want to acknowledge and help people just really understand the difference between 1993 and 2023. Some of the things we took out of the book was “display your results on an overhead projector,” there was no mention of the internet anywhere in that book, because the internet wasn’t here yet. Just stop and think about that for a second. So it was a very different time. A lot of heavy lectures. I mean, a lot of times people were lecturing, there were other activities, or other strategies, there was group work and role plays and those types of things. But primarily people are lecturing, because that’s just what people did, primarily. And so the role that this played, I would say, I don’t know if I’ve ever said this before. I think this book had a bigger influence on higher education than any other single book that’s out that I know of. There are some really, really good books out there. But this fundamentally changed the game. Because what it did was it started asking students about their learning, which means it started to really shift the focus from teaching to learning. This did happen to come along, by the way, if you really look at dates, the whole concept of really pushing active learning in higher education, at least in the United States, really took off in the mid- and late-1990s. So 1996 to 1998, we started seeing people like Richard Hake, do some huge studies about this. American Association of Higher Education was changing their themes from taking teaching seriously to taking learning seriously, all of those things happened about three to five years after this book came out. So I think this was the impetus for a huge, huge aspect of the whole push for moving from teaching to learning. I think it had that big of an impact.

Rebecca: In recent months ChatGPT has hit the news everywhere. We’ve heard all kinds of reports in higher education journals and things. accreditation organizations are talking about AI. And many faculty members are also talking about potential harms of AI or other see benefits. What do you think the impact of AI tools will be on higher education practices? …speaking of evolution… [LAUGHTER]

Todd: I don’t have the statistic right in front of me, but there was one I did a presentation just recently on how long it took to get to a million users. And they were talking about some things like Netflix and other things of taking two and a half years, and then six months and stuff. And I believe ChatGPT was five days. So it’s pretty incredible. So Rebecca, I love the question. I think this is a huge issue. The two biggest things that I heard out of the gate was: number one is how are we going to ever have students do their own work again? It was all academic integrity all over the place. And I couldn’t help but talk to a couple of people who were really freaking out about it and I said, “You should really read some of the research about academic integrity already and the proportion of students who are doing things that they shouldn’t be doing, so to speak, there.” And so that was it… academic integrity. And number two was like, “What kind of assignments should we do now? How do we structure assignments to be meaningful?” And I find this fascinating, because they weren’t talking about the learning process. And I still, to this very moment, I’m just starting to hear it now, but I have not heard much of anything about how can we use generative AI to influence the learning process in a positive way. And so I think that’s the huge thing we need to be looking at. So I guess the people who do know me, know me as the individual who writes a lot about helping students to learn and teaching tips in the classrooms kind of things. I think from a cognitive psychologist’s point of view, we can look at some of the foundational things like: we know that repetition is really good, we know that the process of explaining something really helps because it’s a chance for you to practice at retrieving information. We know spaced recall is really good. And there’s all these things out there that can kind of do that. But I’ll tell you, and I’m coming right back to the CATs, but ChatGPT and some of the other generative AI has some phenomenal opportunities. We know that teaching this is good. So I opened up by ChatGPT 4 the other day, and I was practicing and I typed in: I need to learn about metacognition, would it be okay if I taught you metacognition, and then you tell me how good of a teacher I am. And the chatGPT says, “Yeah, this will be fun. So let’s get started.” And I said, “Okay,” and so I explained metacognition. And ChatGPT asked me some questions like, “Well, what does this mean? And what does that mean? …and I acted just like I was teaching somebody. And then when I was all done, I said, “If I were a teacher, right now, how would you grade me?” And I did not get a good grade [LAUGHTER] which was kind of disconcerting there. But I said, “Why did I not get a good grade on this? And then it explained a section that I had completely forgotten, because I was doing this off the top of my head to try it. So I went back and explained that. And then ChatGPT says, “That’s much better.” I think that’s phenomenal. Another one, I will not get into details, but I typed in, or I gave it the prompt… it’s all about the prompts folks… “That I just learned in my introductory psychology class at a medium-sized, midwestern university. I’m a C-level student who’s worked really hard, and what I just learned were about persuasive techniques. Could we pretend we’re on a car lot, and I’m a salesperson, and you’re going to buy a new car, and you think you want to buy it today, but you’re not going to rush into anything.” And ChatGPT says, “This sounds like fun. Tell me about your best selling model.” [LAUGHTER] And I proceeded to explain all these things. And the scariest one is I said, “This Ford van right here has excellent ratings for safety.” And it said, “That’s particularly important for someone with a loved one and two furry friends.” So the learning possibilities are all over the place for this thing. And so for classroom assessment techniques, this is great, because we can actually teach students some strategies, how in the classroom, how to assess whether they’re learning. We can replicate these and have them do them with AI. So we could use AI to do things we couldn’t do before. But we can also do it to model things for students that they can do on their own. So I think there’s some really, really interesting possibilities coming along.

John: And one interesting one is if you ask ChatGPT for your biography, you may very well turn out to be a car dealer in the Midwest.

Todd: You know what, I’m just gonna jump in there and say that is totally true when ChatGPT was about six months old, because I did it. I said, “Write a letter of recommendation for Todd Zakrajsek… and you probably saw this, why you even mentioned it… and I think it had 19 things I counted in there and this term of hallucinations, but 17 of the 19 things were wrong, including where I went to school where I worked and everything else. That was when ChatGPT was a baby. I asked it the same thing just recently, and it was scary, how close it just nailed everything. But yeah, it was for a while.

John: It’s improved quite a bit, but one of the nice things about it is our students come in with really diverse backgrounds and we can provide supportive materials for students when they have some gaps in their prior learning. But, ChatGPT offers the possibility of having something that’s completely personalized to their needs, so that they can ask questions that are specific to the issues that they’re having trouble with, and not what we might have guessed they’d be having trouble with. So, it’s something I’ve encouraged my students to use in my intro economics course, and they appreciated the fact that they were encouraged to use it,

Todd: I think that individualized instruction is going to take on a whole new level. And so I think that’s going to go crazy. But I will say this, and I think it’s extremely important. I’ve been in higher education for 40 years, I have never seen a bigger possibility for an equity gap to be the widest chasm it could ever be. I think we got equity problems that are down the road. And if we don’t pay attention, it’s going to be awful, because I believe that generative AI is going to be a huge learning aid, and it’s going to help us with lots of things, for the people who have it available to them, they will skyrocket and be able to do stuff; the people who don’t have it available… reliable internet, a good computer, or some kind of a device, a safe place to sleep at night… those individuals are not going to be on that tract and they’re gonna get separated. Right now, there’s a couple of grade differences between people who are privileged and not in some classes. That could widen so much. So I think, everybody, we just got to really be careful about how to make sure that this was as equitable as possible for everybody.

Rebecca: Higher ed leadership at various colleges and universities are certainly dancing around the idea of AI and what kinds of policies and procedures to put into place. What would you advocate for? What would you encourage faculty members to advocate for to really make sure that equity is addressed?

Todd: I think one of the very first things we need to do is teach students how to use it properly. Because one thing we know about learning is, the more you know about something, the easier it is to learn something related to that topic, whatever the thing is, and what that means is that learning is not linear, it’s curvilinear. The more you know, the easier it is to learn something, which means you’re going to know more quickly, and as you know more quickly, you’re going to be able to learn more quickly. And this is why people struggle at the beginning with things and then all of a sudden they take off. So if some students know how to use the systems, and some don’t, that’s where we’re going to run into some huge issues. So I think one of the things we can do is teach students how to use it to help you, and when to be careful that it’s not going to help you. The movie Wall-E is a great thing to be keeping in mind. If you haven’t seen the movie Wall-E, it’s a plug for the movie Wall-E, it came out a long time ago, but that concept of the humans that had everything done for them ended up becoming just blobs that laid in chairs all the time, and I think our brains basically do that. If students use AI to do their work, they’re not going to develop critical thinking skills. If students use AI to help them to practice at retrieval and spaced out practice and do those things, they could become very good. So I think schools need to be careful about helping students to understand how this can really help them. That’s the first thing. Then there’s the obvious, helping people to have access to technology, we could have labs open, we could have laptops that we can loan to people, reliable internet, we could have rooms that we keep open 24 hours a day, keep them 24/7 there. Some libraries during exam time, they don’t close for like four or five days. We could have rooms in the library that people can come and have safe spaces to study and work, reliable transportation is going to be an issue, we’re just gonna have to work out how do we address those issues. It’s gonna be a challenge, but we need to be thinking about it.

John: The use of classroom assessment techniques has been growing steadily since they were first introduced. And each time there’s been new educational technology, it seems to have led to an increased use of those. During that time, we’ve seen the introduction of relatively low-cost computers, computer networks, the internet, mobile computers, and so forth. How do you think the availability of generative AI will affect the use and value of classroom assessment techniques.

Todd: I think there’s a couple things that’s going to really, really change. I think for classroom assessment techniques, number one is we’re going to be able to individualize a little bit more. So we can tailor… we don’t have to ask the same question of all the students, we don’t have to look for all the same type of responses. So we can think about it a little more creatively in how we can use it, almost like an individualized instruction toward classroom assessment. Probably the biggest thing, though, overall, is it’s going to allow us to just process data in numbers and levels we’ve never seen before, and particularly free responses. In the past, we’ve often ask closed-ended questions just because if you’ve got a class with 400 students, you can say: “On a scale of one to five, to what extent did today’s lecture help you to understand something?” And the students can pick a number, we could drop that into an excel sheet, or anything, and come up with a number very quickly. We can now say on a muddiest point, “What are you still struggling with?” Take 400 responses, dump them into a generative AI program and have it spit out five things within 30 seconds:” Here are the five things your students are struggling with.” So I think it’s going to allow us to do more qualitative types of things very quickly. And I totally get that this is not hardcore qualitative research with good analysis of the data. But for what I need in the classroom, I think we’re gonna be able to get those responses very quickly and in real time. So what that means is, we’re actually going to be able to ask something like a muddiest point at halfway through the class, and then I could have the students do a quick think-pair-share, while I analyze the information and a matter of three minutes later, they come out of the think-pair-share, to talk a little bit about what they talked about. And then I could say, I see that you’re still struggling with these concepts, so let’s revisit these things before they ever leave the room. That’s never been possible before.

John: When I’ve been teaching classes of two to 400 students, I’ve used the free response option and word cloud. But the word cloud is just highlighting individual words, the ability to do analysis in more detail is going to be incredible. And I think most providers of response software are working on introducing AI, and some of it is expected to be available fairly soon.

Todd: Oh, there’s some great stuff come check GPT and some terribly scary stuff at the same time. By the way before I forget this, because I have to tell you this because I have ADD, that’s one of my favorite phrases.ChatGPT 5,, one of the things I read about that’s going to be coming up fairly soon is the ability for it to launch his own AI’s as needed. That’s the one that we’ve been waiting for, and thinking, “Hmmm, that’ll be interesting.” But yeah, I think that the ability to read through and look through the information is just going to be a game changer. And online synchronous/asynchronous, face-to-face in all environments. We could look at stuff in asynchronous environments in ways we haven’t before as well. I could look at different ways that students are responding, I could ask classroom assessment techniques. In fact, they’re in the book. We have several of them… ways to use CATs in asynchronous environments, and AI is changing how we’re doing it.

Rebecca: Can you give an example of how that’s happening in asynchronous environments?

Todd: In terms of asynchronous environments, everybody likes to go to discussion boards, but quite frankly, discussion boards are boring. We could have students generate something like the script for a commercial. And after they do that, I could have the CAT that comes in to say, “To what extent was this helpful in the learning process for you?” And the student, as they’re developing this thing and as submitting it they could also submit, right along with it, their CAT that comes with it, and then I could be reading the CATs as they come in one by one right behind each one of them. And so the concept, there, would be kind of a real time CAT analysis that’s not waiting until there’s a whole group and then looking at the group. Because typically, when we think about these CATs, it’s like, the 400 students fill out a muddiest point and we analyze the 400 students. We don’t need to do that. In an asynchronous environment, we can have these CATs coming in and we can be analyzing them as they come in. And we could even add them to the previous ones, to group them if we wanted to. But the bigger one is, I can find out how the students are doing as they’re doing. So that’s probably one way to look at it.

Rebecca: One of the things that you mentioned in an in-person or a synchronous setting was being able to analyze results mid class. And that’s in part because a lot of our students have mobile devices and technology that they bring with them that they haven’t had before. Do you see other ways that these mobile devices are also changing the way instructors might implement CATs?

Todd: The way you just mentioned, though, I can’t think of many other than the concept of as an information gathering device. I will say, just because I happened to read this study just a little bit ago here, the devices are very interesting, in a sense that we’ve seen high schoolers tend to have the same amount of time they spent with their peers was very consistent until about 2012. And in 2012, the numbers started dropping off pretty quickly. And then COVID dropped it even faster, but the slope was already there. And the point here is that people are turning their attention to devices so much that they’re not talking to each other exhibited by when you see two people at a restaurant having dinner, and each one of them are on their device. There’s all kinds of psychology about how those things happen. But the overarching thing that’s the issue is that we’re having students at levels we’ve never seen before disengaged in the classroom, even when we’re trying to do CATs and engage. And part of that is because of the devices. So one of the things that’s kind of interesting about this is when you say: “Okay, everybody gets your devices out, and now we’re going to use them for a CAT.” And then we stick with that and how we use that data makes a big difference of whether they stick with it or go back to what they were doing. Strategically, it’s really helpful to use those devices to engage the students.

John: One concern with generative AI is many of the types of assessments we’ve used before can be answered very nicely by generative AI. What might we do to reduce the likelihood that students will use generative AI as a substitute for learning?

Todd: I think it’s really, really important to talk to students about the long-term implications. I know this is not going to be for everybody. I totally get that. But one thing I’d say to students is: “I don’t understand why anybody would run, why would anybody go jogging? You can just get in your car and get there faster.” So if you kind of pitch that to them, that idea is for most of the students, if there are any kind of health-related fields, or if they want to have a cardiac system later in life, exercise is just one of the most important things you can do for your body, by the way, in terms of that getting 150 minutes per week, and just higher respiration. If you take that away, it is bad for the human body. And so the response is the same thing is true of this. Just talking to the students. If you use generative AI to answer your homework problems, to develop the quizzes, to write a poem, and I’ve done all of these in workshops, where I’ve said to people, I can do this real quickly. I don’t care who’s in this room, I can ask you to write a short story. And I will crush you with generative AI. The problem is that, if I do that, I don’t learn how to think for myself. And so I think the biggest thing we can do for students, number one is to build community in the classroom, to number two, to tell them what this all means, and then number three, ask them, ask them, how it’s helping them, ask them how it’s hurting them, how they believe these things are working. And those are classroom assessment techniques, we can use these assessment techniques to find out to the extent that they’re doing this. While I’m on my little diatribe, we will never be at this spot again in our lives. We have just developed something that can totally short circuit cognitive processing and critical thinking. But we did it with people who have critical thinking skills and cognitive processing. So we developed something because earlier in our lives, we had to learn in the ways of joggers, we had to develop our systems. And now we can talk about the automobiles that would move as faster. The students coming along are going to step right into those automobiles, right into the AI, and if they’re never told that it’s important to go out jogging, they’re not going to develop the systems that are going to be needed later. And so again, just think about this for a second, we’ve developed a system using a system that may disappear if we’re not careful. And so I think that’s where we can change it. So it boils down to community, why we’re using it, and then assessment questions of how it can be used, and where it’s good and bad.

Rebecca: I think students really enjoy having conversations about AI and exploring how it’s useful and not. I have had activities in my class where we intentionally used AI to see what it was like and when it would be useful and also analyze where maybe they tried to use it and it was totally not helpful, and why it wasn’t. And they really appreciated those kinds of conversations and learned a lot from those.

Todd: I think the students do. And we could pick different spots in history. You go back to Socrates and his whole belief that if you write things down, it weakens the mind. It makes sense. If you have to learn it, you’re going to be much stronger than if you write it down. But I can’t imagine right now teaching without students writing things down. And Samuel Johnson came along several 100 years ago and said, “There’s ready availability of books. With books all around, why would we really need to teach this stuff, they can just go get a book.” I can’t imagine teaching without books right now. So these things that everybody got scared about, or thought that just going to change just became integrated. And when the internet came about, we talked about this a while back, in the sense that many of us were teaching before the internet actually showed up. But when it showed up the people who were teaching were freaked out. “How am I supposed to teach when the students can go and get anything they want off their computer?” And now I can’t imagine teaching without the internet. I believe five years from now people are going to say I can’t imagine teaching without writing things down, without using some kinds of print format, without using the internet, and without using generative AI. I don’t even know how I would teach without it. What that’s going to need is the same stuff we’ve done in the past, that’s how do you teach well with the Internet? How do you teach well with generative AI?

John: It’s an exciting time to be in the midst of all these changes. And it’s going to be interesting to see how we all answer those questions as we move forward.

Todd: It is, but before we wrap up, because I can tell that tone, [LAUGHTER] I want to put a general call out that there are several people who are saying: “I wish I could retire right now” or “This is a great time to retire.” The statement I really want to make is this is a hideous time to retire if you’re really good at what you’re doing. Because we have never needed humans with really good critical thinking skills as much as we do now, maybe we will later, but to date I don’t believe we ever have. And for some of the people who are saying “I just don’t want to teach anymore. This is awful…” we might need you more than we’ve ever needed you, so I don’t like this concept of the mass exodus of certain people.

Rebecca: It’s an important point.

John: One other thing I will add is Paul Samuelson once said, in describing the way in which models of the economy evolved, that “funeral by funeral, the science makes progress.” So there is the counter argument there too, that people who are very tied into the old ways of teaching do need to either adapt or perhaps they’ll be replaced by people who are more willing to try new alternatives.

Todd: Do I smell a little Kuhn in there? The revolution of science. Ah yes.

Rebecca: And we always wrap up by asking: “What’s next?”

Todd: I’m trying to keep busy. I’m kind of on a roll right now. And I want to tell you, I’ve been writing books and the only reason I’m unhappy about ChatGPT and gen-AI come along is people are going to assume that I’m using them so that I can crank stuff out. But thus far, I have not used any of this generative AI to write any of these last books that I’ve done, these five books in the last five years or something, and enjoyed working with Tom on this one. But I got another one coming out in about four months. I just turned in the final pages for that And it’s the Essentials of the New Science of Learning: the power of learning in harmony with your brain. That book should be out in a couple of months. And so the what’s next after that, I am working on a book right now for helping with neurodivergent learners. We’ve joked around in the past, I’ve got ADHD is about as bad as it gets. If it were a competition, I could come in probably close to first place, and also on the autism spectrum just a little bit too. And when I started sitting down with some of my colleagues who also have ADHD and autism spectrum, it occurred to me, we don’t talk to students enough about these things. And I think the challenge of that is that the students believe that they’re alone, and they’re not sure they can do things. And I was talking to a student fairly recently that said, “I now believe I have a shot, here you are with a PhD, you’ve written books, if you’ve got ADHD, as bad as you claim you do, maybe I can do something.” And then that’s when I really really was getting serious about we need to help folks out. And the topic I’ve been playing around with a little bit in a couple of workshops is The Ones Too Often Left Behind as the title I’m using. And it’s the students that aren’t built for the system that we developed. And so I think that we need to treat some folks out there a little differently. And I think we can really build up the pool of intelligent folks by helping to teach the people who just learn in a different way.

Rebecca: It’s really important work to have models and put models out in front of students because they need to see themselves in whatever discipline, field, etc., that they want to pursue. I’m excited to hear more about that.

John: And when will these books be out? You mentioned the timeframe for one of them.

Todd: The Essentials for the New Science of Learning should be out in September. And then the other one, if everything goes well, is probably looking at a February date that it would probably be available.

Rebecca: Well, it sounds like we’ll be talking to you soon then, Todd.

Todd: We may be. When you got ADHD pretty bad, you just can’t really predict when that books gonna be. But yeah, I’d love to chat with you when it does finally emerge.

John: And what are you going to do next week?

Todd: You know what? You gave me a great idea. I think we should do a book on procrastinating people with ADHD. [LAUGHTER]

John: It’s always great talking to you. And we’re looking forward to each of these projects coming to fruition and we will be talking to you about each of them, we’re hoping.

Todd: I appreciate the opportunity to come and chat with you. I think you two do a phenomenal job with your interviews and the programs that you pull together on really good topics and I just am honored to be one of the guests.

John: Thank you.

Rebecca: Yeah, thanks. We always enjoy talking to you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC] </p

346. Classroom Assessment Techniques

Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, Thomas A. Angelo joins us to discuss the origin of these techniques and evidence concerning their efficacy.

Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993.

Show Notes

  • Cross, K. P., & Angelo, T. A. (1988). Classroom Assessment Techniques. A Handbook for Faculty. Jossey Bass Wiley.
  • Angelo, T. A., & Cross, K. P. (1993). 2nd. Ed. Classroom assessment techniques. Jossey Bass Wiley.
  • Angelo, T.A. and Zakrajsek, T. (2024). 3rd. Ed. Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers. Jossey Bass Wiley.
  • Schwartz, Charles (1991). “An Academic Adventure.” March 4. – A description of the origin of the minute paper by it’s originator.
  • Chizmar, J. F., & Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. Journal of Economic Education, 29(1), 3–10. (the article that John referenced as his first reference to this topic)
  • Mazur, E. (1997). Peer instruction: A user’s manual. Pearson
  • Mazur, E. (2014). “Confessions of a Converted Lecturer” – a recording of a presentation by Eric Mazur at SUNY Oswego, 5/19/14
  • Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses. John Wiley & Sons.
  • Pam Hook and Josie Roberts (2018). “The who, what, when, where, and why of SOLO taxonomy.”
  • Angelo, T.A. (1993). “A Teacher’s Dozen—Fourteen General Research-Based Principles for Improving Higher Learning.
  • Pre-order for 3rd edition of Classroom Assessment Techniques: Formative Feedback Tools for College and University Teachers.

Transcript

John: Classroom assessment techniques can be used to shape instruction to the needs of our students. In this episode, we discuss the origin of these techniques and evidence concerning their efficacy.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Thomas A. Angelo. Tom is Clinical Professor Emeritus and Director Emeritus at the UNC Eshelman School of Pharmacy at the University of North Carolina at Chapel Hill. In addition to teaching for 40 years, he has been long involved in professional development and has served as faculty member and Director of teaching, learning and assessment centers at UNC, LaTrobe University in Melbourne, Australia, Victoria University of Wellington, New Zealand, the University of Akron, and Boston College. Tom is best known for his work with K. Patricia Cross on Classroom Assessment Techniques: A Handbook for College Teachers, which was first published in 1988, with a second edition in 1993. Even if you’ve never heard of these books, you may have included some Classroom Assessment Techniques in your teaching toolkit. Welcome, Tom.

Tom: Good to be here.

John: We’re very happy to talk to you. Today’s teas are:… Tom, are you drinking tea by any chance?

Tom: No, I was drinking black coffee.

Rebecca: It’s one of our most popular forms of tea [LAUGHTER] as regular listeners know. Today have some English tea time, John. How about yourself?

John: …and I have an Earl Grey tea today.

Tom: Excellent.

Rebecca: We’ve invited you here today, Tom, to discuss the third edition of Classroom Assessment Techniques, to be published shortly. These techniques described in earlier editions of this book have been widely adopted by faculty in all disciplines at all different types of institutions. And a lot has changed in higher education since 1993. But this book is still the go to reference on classroom assessment techniques, which is pretty impressive [LAUGHTER] for a book that was published 30 years ago. Can you talk a little bit about the origin of this book?

Tom: Well, I think the origin of the book goes back to the fact that when I was a young teacher, I recognized pretty early that many of the things that I was doing didn’t work and didn’t work for my students, and that I had to accept that things that had worked very well for me as a student weren’t working for many of them. So I began to experiment with asking questions in a variety of different ways and getting students to jot down answers anonymously. And through that, I really learned more about teaching and learning, I think, than from anything else that I did. And one of the things that I learned was not to assume. So, classroom assessment techniques are things that I’m sure almost every teacher has done. But what we did in the 1980s. K. Patricia Cross and I, was to really formalize those. And we did that in order to create a resource for teachers so that they could find simple tools that would help them ask students the kinds of questions that would help students learn better, and teachers teach better, as they went along. So it really began at that time, and it was kind of an accidental start, in many ways. We were doing it as part of a big project, and what was then the assessment movement, and it just kind of took off.

John: Most of our listeners are probably quite familiar with classroom assessment techniques and have seen them at lots of workshops and have probably, as Rebecca noted before, used them in the past. Could you define what a classroom assessment technique is, for those listeners who are new to the concept?

Tom: Well, in a formal way, a classroom assessment technique is a formative assessment technique. It’s a way of gathering data to use to improve and inform practice, rather than to make judgments. So we have too, most of us, to give students tests and exams and assign papers, and grade their work, evaluate their work, for grades and marks. Formative assessment techniques, on the other hand, are tools that we use to gather data along the way, before those moments in which we grade them, or mark them, so that we can help students get ready and improve and succeed when they’re going to be graded and marked and so that we can, at the same time, find out where they are and how they’re doing in order to help them improve along the way. So they’re really ways of diagnosing, in a way, students’ learning.

Rebecca: One of the most popular classroom assessment techniques described in your book is the one minute paper. There have been many studies of the effectiveness of this technique and it’s been widely adopted in many disciplines. Why was this technique, in particular, been used so extensively?

Tom: Because it’s simple and easy to use, probably, and it takes almost no time. But seriously, the one minute paper, the minute paper was invented, to the extent these things are invented by Charles Schwartz, in the 1970s, at UC Berkeley, and he used it as both a way to take attendance (in his case, he was asking students to jot their names down), but more importantly, to find out whether students were learning from his physics lectures, in his huge physics classes, what he hoped they were learning. And that idea spread through UC Berkeley really fast. And so soon, there were dozens and hundreds of people using these two questions. And the two questions, famously, were: what are the two or three most important things you learned from today’s lecture, in his case, and what questions do you still have the end of the lecture? So students would jot these down on those strips of paper, remember, there was no internet and no computers, really, in classrooms in the 1970s. And then he would collect them, and read through a sample of the 1000 or so that he had every lecture. And from that sample, he would gather ideas about what students understood and didn’t from the lecture, and also what questions were most common. And then he would begin his next class meeting, by going over some of those points that students had had trouble with, or not understood, or missed them completely. Sometimes it happened that a 1000 students didn’t get one point that he thought was critical, and also answering one or two of the most important questions. And what he found was, he could do all of that in five minutes at the end of class, and about five minutes at the beginning of class. And I think it’s that efficiency that has made it so popular.

John: My first exposure to this was a study that was published in the Journal of Economic Education way back around the time of your last edition. I did try to find it, but I couldn’t find the citation for it, but what struck me was that the instructor had used it in one section of the course, had not used it in another section, and found that there was a very substantial difference in learning outcomes between the two groups, which he attributed to the one minute paper. And it’s been incredibly commonly used. But one of the things that really struck me is that at the time when you came out with this book, including the one-minute paper, and so many other classroom assessment techniques, most instruction at the time was the chalk and talk type where people were at a blackboard, maybe a whiteboard, if they were a little bit more advanced in the technology, where most assessment was done with high-stakes assessments. So I think your book was a fairly important factor in helping people shift to more formative assessments. Why have these techniques become so widely adopted?

Tom: Well, I think that one of the reasons why people have found classroom assessment techniques and other formative assessment techniques, there are other kinds, so useful and so important, I think there are several reasons. One is that the student body in the United States and across the world has become ever more diverse in the last 30 or 40 years. And it turns out that no matter who we are, as teachers, most of our students are going to be pretty different in terms of their experiences and their interests and their skills than we are. And that level of difference has probably grown over the years, at least our awareness of it has grown. So I think most teachers now realize that it’s very important to check in on students and see how they’re doing, before we get to the midterm or before [LAUGHTER] we get to the end of the course, and to check in regularly and not to assume that students are understanding our brilliantly clear presentations or the material that we give them, or the readings, or the problem sets, and to really check early on to see if it’s making sense to them, and what kind of sense it’s making to them. So I think that there’s much more realization of the diversity of human learning, and of our students, than there was 40 years ago or 30 years ago. And I also think that as teaching has evolved, especially as it’s become online, people have recognized that we need data. We can’t just depend on students volunteering that data, they often don’t know that they don’t understand. And that that data, used well, can help us and can help them.

Rebecca: I think one of the things that you’re talking about in terms of assumptions is reminding me of some of the conversations that we had early on in the pandemic, where instructors we’re talking about not being able to see the sea of faces to know whether or not students are learning because maybe students aren’t nodding along or providing some visual feedback, but classroom assessment techniques are really important because they allow opportunities to gather that data, which is probably far more informative than the head nodding.

Tom: I think there are maybe a couple of other things that are important about this that people discover. One is that by stopping to ask students to reflect on their learning and respond, you give them time to think. And you prompt them to think about their learning. And from my point of view, forever, the most important thing about doing this has really been to give students tools to become more aware, we could say more metacognitively aware, of their own learning, and to better take responsibility and control or at least direction over their own learning. So what I found is I’ve had many, many, many very bright students who had never been taught any kinds of techniques to think, or to ask themselves questions, or to monitor their own attention, and having those sorts of tools, those sort of metacognitive tools, I think empowers students to make decisions about their own learning, and also to improve their own learning in those areas where it matters most to them, which may not be my course.

John: One of the things you mentioned was that the one minute paper was originally developed in a class with 1000 or so students. And one of the nice things about many of these techniques is that they can scale that way. Now, there was a time when the book came out when this was all done on paper. But now we have so many electronic tools that we can use to do this. Instructors could put up a QR code with a link to a Google form or something where students can give their feedback within a minute or so or we could use polling techniques, and so forth. And many polling systems have built in one-minute questions or some other classroom assessment technique as a default strategy. So this impact has been pretty substantial. And it’s really nice for instructors, not just for students, having that time for reflection, but also for instructors to get some feedback on what students have learned. Because if you just call on students, you may happen to get the only student who understands the material responding to a question or you might get the only student who hasn’t understood the material. But with these formative assessments, you can get a good feel for where students are. And they also get some feel for where they are in a way that they might not if they would just passively sitting there listening to a lecture. So these techniques have, I think, become increasingly more common from the time of introduction. So these techniques have held up pretty well. But you now have a new edition of this book, a third edition of this book. Could you describe some of the changes in the new addition?

Tom: Well, obviously, and you’ve touched on this, technology has changed. And that’s had an impact on teaching, and an impact on learning. So one of the things that I did, in going back to do this book after so many years, was to take a look at first what was out there, what had people published on this. And there are literally about 3000 published articles of various kinds and various quality, about classroom assessment techniques. There were about 300 that I thought were of sufficient kind of quality and rigor to be able to use. And in the chapter on the minute paper, I refer to 150 articles and studied them very carefully. There are many, many books that refer to the classroom assessment techniques. And I also read [LAUGHTER] all of those references. So I took a few years to work through this material and new things were coming in all the time. So one thing that differs is that we have an evidence base across many, many disciplines for what seems to work and what seems to work well. And also much useful information on how to use these with various technologies. So how best to use formative assessment with online, fully online technologies on totally asynchronous classes. And so that’s been really valuable. And those are experiences that I couldn’t have all of those experiences. So it’s terrific to draw in those from other teachers. The other thing is people have invented new techniques or developed new techniques over time. And so in the third edition 25%, or about 15 of the 55 techniques, are ones that didn’t exist in the second edition. And that’s about one quarter of all the techniques in the new book. So one of the reasons why we have different techniques in the 3rd edition is that some of the ones in the second edition weren’t much written about, weren’t much commented on, and they seem not to be much used as far as I could tell. So I’ve picked up the ones that were most used and most written about, and most referenced so that people in different disciplines can find those techniques. And I’ve also found another 15 techniques to kind of fill those gaps and extend it a bit.

Rebecca: Your book offers many, many assessment techniques. I’m curious what your favorite few are that you’d like to introduce instructors to?

Tom: Well, just like I can’t say which of my daughters are my favorite, I do have favorites. I’ll be honest. I have always used the minute paper ever since I learned about it all those years ago, and I learned about it really when I was at Berkeley, in the mid 1980s. And I learned about it, as everybody else there did, kind of by chance, and thought, “Wow, why didn’t I do this? Why didn’t I think of this?” So the minute paper, the muddiest point, which is a variant of the minute paper, which is developed by a really famous and eminent statistician at Harvard, named Frederick Mosteller. And Fred Mosteller heard Pat Cross talk about this, when we were both at Harvard, and being the kind of person that he was he immediately experimented with it. What he found was that it was more useful for him, in his courses, to just ask one question, and that question for him was “What was the muddiest point In today’s class? What was least clear to you?” Many other people have used that as well and written about that. That’s another favorite. A third one is one that I’ve used a lot, being a statistics teacher. And that’s predict, observe, explain, which is often called POE. And that’s one that I’ve found useful in many situations. And I’ve had colleagues who’ve used that even in literature courses, and other courses where they’ll say to students, “let’s stop at this point in the book or the film, and predict what’s going to happen next, write that down, then we’ll observe it, and then we’ll try to explain why that either met our predictions or didn’t meet our prediction.” And while it’s obvious how people do that in science, and it’s part of the scientific method, it can be used in all kinds of circumstances. And it’s one that I’ve used with teachers who I’ve worked with, about classroom assessment and about formative assessment. I’ve asked them to predict what they thought would happen, then observe it, and explain what did happen and whether or not that met their expectations. And that’s been a very useful tool for me, I think, and the teachers I’ve worked with. So those are three good ones. I’ll give you one more, if that’s helpful. And that’s something that we call the directed paraphrase. So three of those four were in the first and second versions of the book, predict, observe, explain is in the third edition, but directed paraphrase simply asks the questions that are sometimes called journalistic questions, who, what, when, where, why, who did what to whom, when, where, why, and how? And it asks students to answer those questions about some phenomenon that they’re observing, or to answer them about an essay that they want to write, answer them about any kind of question. So giving students a framework of questions that they can work through, to make sure that they really understood either a book that they’ve been reading or an experiment that they’ve been doing, has been really valuable to my students. And I’ve said to them, “Look, this is something that you can adapt and take with you anywhere to use to make sure that you understand what’s going on fully. And to find out which part of it maybe you really can’t answer, and need to do some more work on.

John: In terms of the predict, observe ,and explain thing. I took a MOOC that was put together by Dan Ariely on behavioral economics. And one of the things he talked about was how he would often present the results of an experiment where the results were not what most people would have expected. But then he’d asked people for their reactions only after he summarized the results of the experiment. And they all nodded and said, “Well, that’s exactly what they would have thought would have happened.” So he started revising his presentations, where he asked people to make a prediction, and then revealed the result. And it got people much more engaged. And the same type of thing is done by Eric Mazur with his polling technique, where he asked people to answer a question where the question is not obvious and where there’s going to be some disputes. And then once they’ve committed to an answer, they work with other people and he goes through it again, with a little bit of peer instruction. But the level of engagement when students have made some type of a prediction of what they think the answer is, and then they have an explanation of what the answer is either given by the instructor or by other students, the level of engagement and interest just seems to go up fairly dramatically. So I can see how that would be particularly effective.

Tom: Yes, and I’ve learned a lot from the work of Eric Mazur, as have many others. And I think that that’s a critical thing. Because one of the things that we need to help students with, and help ourselves with, is managing our inbuilt biases and that issue of when you hear something new, feeling that you’ve always known it, even though you haven’t, is a cognitive bias that’s common, I think, to all human beings, myself included. So doing something like that: predict, observe, explain routine gives us a chance to really check and see if we know the answer. And then to find the answer, or discover an answer. And then to realize, because we have a document, we have a record, we have data about what we knew in the first instance, to recognize that we’ve learned something and what we’ve learned and that we might have been misinformed or unclear in the beginning. And I’ve suggested to my colleagues, one of the reasons that students don’t value their education as much as sometimes we think they should, or value our course as much as they think they should, is because when they get to the end, they may think “I always knew that, that was obvious, I knew those things.” And so we can help students realize that there’s a start point. And then there are many points along the road to learning, and that they may have learned more than they thought they did in a given course or given program. And I think that’s important for us in terms of demonstrating the value of education. And I think it’s important for students in terms of understanding and developing some humility about how it is that we learn and how hard it is and how long it takes to learn anything that’s really valuable.

Rebecca: I think, in my experience, students really appreciate the opportunity to slow down and reflect on their learning, because they’re often not given that opportunity. And when you’re doing that throughout an entire semester, they really do get excited about how much they’re learning and they get energized by that and seeing that they can achieve things over and over and over again.

Tom: I agree, and what we found with most of the studies of classroom assessment techniques, the simple ones is: first, classroom assessment techniques can’t really be separated from everything else that’s going on in teaching, they’re just one little part of the whole teaching and learning spectrum. They, at best, seem to make about a 5- 8% difference in how much students learn, which is about half a grade. And I think that’s important, but I don’t want to overstate it. So they’re not a panacea and they’re not a silver bullet, but they’re something that can lift up, to some degree, students’ performance and their learning. And the last thing I’ll say is nothing works for everybody. And so there is no technique or no series of techniques that every single teacher or every single student that I’ve worked with found valuable. And so one of the points of having 55 techniques is to hope that there’s one or two or five in there that people will actually find that work for them.

Rebecca: So you just reminded us about there’s 55 techniques, how do you imagine an instructor using your book? Is it a read from the beginning to the end? Do you have it organized in ways that instructors can easily dip in and out?

Tom: I can’t imagine reading it from the beginning to the end, and I don’t think anyone ever has. That would surprise me, except maybe the proofreaders [LAUGHTER] and they weren’t worrying about what it meant. So it’s organized so that there are many ways, there are actually eight ways to find a technique that might be useful for a given situation or a given teacher. One of those ways is the simplest and that’s alphabetical. So if someone has mentioned a technique to you, you can easily find it that way. The others are a bit more complicated, but I think the most valuable for many people is an index of all the techniques that have been used by discipline and documented by discipline, and the examples by discipline in the book. And there are a couple of 100 examples in the book of how people use these in different disciplines, kind of from A to Z, from anthropology to zoology, and everything in between. So those are ways that many people use. There are also a couple of inventories in the book. One is something called the teaching goals inventory, which was in the first and second edition, and has been used in lots of ways by many people and written about and research has been done on the teaching goals inventory, and it is exactly what it sounds like, an inventory of what we think we’re trying to teach. For this book, I created what I think is kind of a more contemporary take on that, and that’s something that I call the course learning outcomes inventory, or CLOI, and it’s really flipping that and saying, okay, it’s important what we want to teach, but let’s look at it from the point of view of what do we want students to learn? And so the CLOI, the course learning outcomes inventory, the statements are statements of learning outcomes. There are 57, they’re divided up into kinds of learning outcomes, and people can use it as a self assessment. Those items both in the teaching goals inventory and the CLOI are all linked to classroom assessment techniques in an index. So if you pick number 47 goal or number 47 outcome, you can find techniques that are linked to that. Lastly, we’ve done indexing by Bloom’s Taxonomy, and by Fink’s taxonomy of significant learning outcomes, and also by the big solo taxonomy, which is one that’s much used in the Commonwealth world, in the British and Canadian and Australian and New Zealand systems. So there are eight ways to skin a CAT in this book, or to find a cat at least. And I hope those will be useful to folks. And I think different kinds of indexing work for different kinds of people.

John: You mentioned that when this book first came out these things were done on paper. But one of the things that’s grown quite a bit over the years, as you’ve mentioned, is online instruction. Could you tell us a little bit about how some of the techniques can be used in an asynchronous environment?

Tom: Well, I think there are huge advantages to the technology that we have now compared to having to do all this on paper, having to read all those papers after class, or all those index cards, in my case, after class and then waited until the next class, when I saw students again, the time lag was a real problem. So being able to respond almost immediately, or in relatively short time to students in asynchronous courses, to their questions, and to their confusions, and to also let them know that they’re learning some things really well… they really got that… and that they’ve come up with some ideas that I never would have thought of that are excellent. To do those things in real time, I think, is much more effective in terms of teaching and learning than waiting two or three days or a week. Also, as you mentioned, we have technologies for polling students and for analyzing their responses that are ever more sophisticated and powerful. And so those save teachers a lot of time and work. I was always going through things counting and deciding on the number of times people had mentioned things on paper. This can all be done and is all done for you if you want to do it. And that, again, allows teachers to focus on what really matters. And that is: what’s the meaning of this data? What are students telling me? …and to make decisions about when they’re going to respond and what they’re going to respond to, because you can’t respond to everything in data if you have more than two students. And most of us did, and do. So I think that asynchronous learning has huge advantages, it has some downsides, but it has huge advantages. And one of those probably is that it makes it more difficult to assume that by looking at students, you can tell that they’re learning or not, which I’ve always found an odd idea, but many of my colleagues at least used to seem to think that if students were nodding and looking at them, that they were just tracking with them and understood everything that they were saying. And I used to say to my colleagues, “I’ve known many smilers and nodders that probably were thinking of something else entirely, and classroom assessment proved that to me.

John: I’ve seen that so much. We’ve heard so many faculty during workshops over the years who say, “Well, I can tell whether they’re learning just by looking at the expressions on their faces.” And I’ve never found that. There are a lot of students who will be nodding and smiling. And then when you ask them a question, or they ask you a question, it’s really clear that they don’t have the basic idea. At an economics conference, I was presenting a paper and there was someone sitting in the first row taking furious notes while I was talking. So I went up to the person after and asked them what they were taking notes about, because I was curious. And they were actually preparing for a session later in the day. But they looked really engaged throughout the whole thing.

Tom: They were engaged, just not with your lecture. [LAUGHTER]

John: Yes.

Tom: So I think the converse is also true. And that is their students, and people, who don’t appear to be paying any attention at all, or who are frowning or who have had their head down, who are actually paying total attention and learning a lot. And that’s one of the things I learned as well, early in my career, was not to make assumptions based on those sort of superficial impressions. And it’s hard not to, but when you actually learn that some students who appear not to be engaged are engaged in their quiet way. And that’s a humbling lesson, or it was for me.

Rebecca: One of the things that I really like about your book, and also the way the collection of the CATS all in one place is the flexibility, the flexibility of navigate the book, the flexibility of options for the instructor, but also the flexibility of options for learners as well, in different contexts. So I just wanted to highlight that flexibility piece that we haven’t really talked about quite yet.

Tom: Well, that’s the hope. And if this is created as an ebook, and I expect it will be, then I think that will offer people even more flexibility and more sort of usability. It will be a more user friendly book. And again, there are huge advantages to the kinds of technologies that we have available now that we didn’t then. So I think the real key to this is one of the axioms in the book. And that is it’s really about adapting, not adopting. So there are many, many tools in there that people have adapted in a variety of ways, renamed, and that’s all fine. And that’s all exactly what should happen. You should find them, adapt them to their uses, and if they don’t like the names of the CATs, they can change the name.

John: Would this be a good subject for a faculty reading group where people from different disciplines could get together and talk about how either they’ve applied these things or how they might apply them in their classes?

Tom: Well, it has been from the beginning. And we began this whole process working with groups of teachers, initially in community colleges and state colleges, who were interested in assessing student learning. And, of course, the point being trying to improve student learning. And what we did basically was use the teaching goals inventory at that time, you could use the course learning outcomes inventory, with those teachers to get them to focus on what they really wanted to know. And in the book, I’ve written about, in one of the chapters, four different ways you can approach using classroom assessment techniques and figuring out which one to use. But everybody had ideas and theories and questions or problems that they wanted to try to solve. And so we started with that. And then we said, if that’s your question or problem or theory or outcome or goal, what might be a good tool to try to help you gather a reasonable amount, a manageable amount, of data from students that might help you learn, and might help them learn? And that’s really where we began, and those discussions and the collegiality that came from them, and the interactions that came from that, were always the most valuable part of that professional development. That when we ask people: “What did you take away from this?” The first thing that they said was not a bunch of techniques. The first thing that they said was the interactions with my colleagues, the relationships with my colleagues, learning from my colleagues, and then they talked about the actual content that they had learned and the things that they had done. So I think those interactions and those relationships that people can make, by having a group focused around this. This is an excuse to get together and talk about teaching and learning. I think from our experience, we found any excuse to get folks together to talk about teaching and learning, especially if it’s across disciplines, is always fruitful.

Tom: …particularly if there’s food. [LAUGHTER]

John: Yes. The third edition list you as author with Todd Zakrajsek, could you tell us how this collaboration came about and how it worked?

Tom: I think this was, in part, an insurance policy for the publishers. Given my advanced age, they wanted to make sure that they had someone involved in the project, who could take over if I just kicked the bucket. So, in the last six months of the book project, after I had finished writing the manuscript, Todd Zakrajsek joined with me and was really invaluable in finishing the process, in coming up with great ideas and suggesting great ideas, especially for the use of technology and AI with CATs, and in helping me manage the process with the publisher, which is much more complicated than it was 30 years ago. He also wrote the discussion questions for use by teachers and faculty developers, and I think those are going to be very helpful to folks.

John: We’re recording this in May, and the book is scheduled for release in June. We always end by asking, what’s next?

Tom: Well, I’ll answer that. But I’ll first say to teachers, what I always say, and that is, “Feel free to ask your librarian to order the book, so that you can use it, and your colleagues can use it.” Don’t tell Wiley and Jossey Bass that I said that, but I think that’s important. So what’s next for me is I’ve been working for a while with a group of people in, strangely enough, pharmacy education. I’m not a pharmacist, and I’m doing research with graduate students on what we call core concepts in developing what are the core concepts in pharmacotherapy? And that’s been very interesting for me. I’ve learned a lot from that. And so I’m ongoing with that. And we’re beginning to work on educational materials related to that. So that’s one of the things that I’m doing. And I’ve actually, I don’t want to commit to this too much, but I’m thinking about writing a book based on an article that I wrote that people have found useful, called “A Teacher’s Dozen.” And so I’ve updated that. I’ve been working on that actually for a few years, and something may come of that.

Rebecca: Well, we’ll be excited to find out what that is soon, maybe. But you haven’t committed, but maybe. [LAUGHTER]

John: Well, thank you. It’s been great talking to you. And my copy of the book is on preorder. And I am looking forward to receiving that.

Tom: So am I. [LAUGHTER]

Rebecca: Well, thank you so much for joining us. It was great to hear about the origin story of the book and how it’s evolved over time and I know a whole other generation of teachers will appreciate the new edition as well as teachers who have been practicing for a long time.

Tom: I hope it will be useful. Listen, it’s been a pleasure and a privilege to talk with you both. Thank you for giving me this opportunity

John: We should note that this is the first of two podcast episodes discussing this book. In next week’s podcast we’ll be joined by Todd Zakrajsek, who will be discussing his role in this work.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

345. New Era – New Urgency

Public confidence in the U.S. educational system has been declining while reports of student disengagement have been rising. In this episode, Deborah Pomeroy and F. Joseph Merlino join us to discuss the possibility of repurposing our educational system to better support the needs of our students and our society.

Deborah has over 50 years of education experience and is professor emeritus at Arcadia University. She has co-directed a Dewitt-Wallace grant, Students at the Center, for inner-city schools in Philadelphia and was actively engaged in the Bioko Biodiversity program in Equatorial Guinea. Joe Merlino has spent 39 years in education. He has been a principal or co-principal investigator and/or project director on numerous federal grants. He currently directs a seven-year USAID grant in Egypt where a team of US faculty are co-developing 180 new undergraduate STEM teacher-preparation courses for five large Egyptian universities.

Deborah and Joe are co-founders of The 21st Century Partnership for STEM Education where Joe has served as president since its founding in 2007. They are also the co-authors of New Era – New Urgency: The Case for Repurposing Education, which was recently released by Lexington Books.

Show Notes

Transcript

John: Public confidence in the U.S. educational system has been declining while reports of student disengagement have been rising. In this episode, we discuss the possibility of repurposing our educational system to better support the needs of our students and our society.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Deborah Pomeroy and F. Joseph Merlino. Deborah has over 50 years of education experience and is professor emeritus at Arcadia University. She has co-directed a Dewitt-Wallace grant, Students at the Center, for inner-city schools in Philadelphia and was actively engaged in the Bioko Biodiversity program in Equatorial Guinea. Joe Merlino has spent 39 years in education. He has been a principal or co-principal investigator and/or project director on numerous federal grants. He currently directs a seven-year USAID grant in Egypt where a team of US faculty are co-developing 180 new undergraduate STEM teacher-preparation courses for five large Egyptian universities.

Deborah and Joe are co-founders of The 21st Century Partnership for STEM Education where Joe has served as president since its founding in 2007. They are also the co-authors of New Era – New Urgency: The Case for Repurposing Education, which was recently released by Lexington Books.

Welcome, Deborah and Joe.

Deborah: It’s great to be here. Thank you so much.

Joe: Thank you.

Rebecca: Today’s teas are:… Deborah, are you drinking tea today?

Deborah: I am. I have a cup of chai right in front of me.

Rebecca: Perfect. How about you, Joe?

Joe: Yes, this is English breakfast tea.

Rebecca: Oh, yum.

John: And I have a black raspberry green tea today.

Rebecca: Nice. We got green tea in the office today. I have a raspberry Jasmine green tea today.

John: We’ve invited you here today to talk about your new book. Could you tell us a little bit about the origin of New Era – New Urgency: The case for repurposing education?

Deborah: Well, that’s an interesting story. We actually started the book over 14 years ago when we were working on the math-science partnership for Greater Philadelphia. And it was the culmination of work that both Joe and I had been doing, both independently and collaboratively, in education reform. And we were beginning to develop our ideas about education reform. And we’ll go into those in more detail later. But we started the book, and we actually got quite a bit of the book written. And then we were interrupted by a meeting with the Minister of Education from Egypt, who came to Philadelphia with a group of colleagues to study STEM schools in the US. And as a result of that meeting, we ended up working on this USAID project in Egypt, helping the ministry develop a series of model STEM schools. And when we got into the work there, which was basically 24/7, 365 days, and we were joking around that many times, it felt like we were drinking water from a firehose, it was so intensive. And we were never able to go back and finish our book. But as we were doing the work, we sort of looked at each other a number of times, sort of laughing and saying, “This is the last chapter of our book,” because in the last chapter, we hit envisioned what real transformation could be in education, and we were actually doing it and it was so exciting. And so we decided that we needed to recast the first part of the book, and then add this whole latter part of it, because we actually have a case where our ideas have been able to be put into place and have worked just amazingly well. And so the book is actually a total of 14 years in the making.

Rebecca: So I’m really curious about the word choice of repurposing because you could have probably picked many, many titles. Can you talk a little bit about why repurposing is your word choice in the title?

Joe: We felt, through our reform efforts that were difficult to implement, that the single most durable element that prevented reform from happening was the inability of teachers and students to articulate why they were teaching what they were teaching, as opposed to all the other possibilities of what you could teach per subject. So why are you teaching algebra? Why not statistics or probability? Why are you teaching it the way you’re teaching? So the idea of purpose, which is answering the why, was what we found to be lacking. That’s why we felt purpose was the central idea of the book.

Rebecca: I knew there was a key reason.

Deborah: When we started work in Egypt, the first questions that we asked them were: What are your aspirations for the future? And for the outcome of these schools? In other words, what do you dream these graduates could do and become, and when they were able to start to articulate that, and we’ll go into that in more detail later, then we built the entire curriculum and assessment system around that purpose. And so we found that that, in fact, was the linchpin for this transformative reform.

Joe: After our meeting in August of 2011. This is like a year after the Arab Spring in Egypt, that’s when the Egyptian delegation came to Philadelphia. Four months later, we found ourselves in Cairo, I did, and two other colleagues of mine. And so we’re sitting in the first school, which is outside of Giza, and it’s in the middle of nowhere. I mean, it’s 30 miles, it’s just dirt and dust in this building. And we’re sitting in the principal’s office, and looking around wondering what’s happening and in bursts the first deputy minister of education in Egypt, and he says, “Welcome to the dream.” [LAUGHTER] And we’re looking around, and we’re saying “there’s nothing here.” So we had the opportunity to rethink things, and for them to rethink things. And I’ll just say one other thing real quick, is that the purposes of education derives from your aspirations for your country or for your community, and that defines what your purpose is. And that’s what we did in Egypt.

John: And you begin the book by talking about how the purpose of education has evolved over time with a little bit of lag, because educational systems tend to adjust slowly. Could you describe how educational systems have responded to changing societal needs?

Joe: Okay, I’ll try to do this in 60 seconds, if I can, the 400 years. [LAUGHTER] For me, it was a tremendous learning experience. We have, in our time, the ability to have the internet, to have digitized libraries, and search engines. So you can go back and actually look at primary sources. So it became a real learning experience for me as I went through it. But we started with the idea that we’ve been in this business for a while now, from the early 80s to now, and we could see the development of the Nation at Risk archetype that has defined our era here. And we wondered about, are there other archetypes? Have there been other paradigms? Have there been other purposes of education other than, right now, preparing kids for the global economy, which was the idea of the Nation at Risk? So we started by looking at the earliest instances of purposes, and we found it was the religious schools in New England, the Calvinist idea of preparing kids to be a part of a religious community, salvation was the key. And then we looked at, “Well, why did that happen? Why was it salvation?” So we looked at the English history and why people came to the United States was to form a new Israel. That was the vision of the Puritans. And then we looked at how this may have changed over a single lifetime. We talk about in our lifetime, how much change there has been if you think about yourselves and your grandparents. So we looked at history through the lens of five lifetimes, laid end to end. We said, “Well, what happened during these lifetimes?” And you see a tremendous amount of change that happened, that’s what we described in each lifetime, but along with those changes have been associated changes in the purposes of education. So I’ll just give you a quick example. So from this idea of the Commonwealth of Massachusetts, from this idea of a religious community, came the idea that as more immigrants came in, there was different religious sects. And there was conflict between these sects, so much so that when William Penn came to Philadelphia in 1682, he brought the Quaker ethos, the idea of the Quaker schools which was based on religious tolerance, and peace. And so it became another purpose of education. And he founded the first Quaker school here in Philadelphia. So that’s an example. Another example is the industrial revolution. There couldn’t be industrial education if there wasn’t an industrial revolution. So that’s what we did.

Rebecca: It’s really done a good job of setting the stage of how change has occurred over time. Can you talk a little bit about how the current system aligns or does not align with our current societal needs?

Deborah: Right now, the paradigm that is pretty much driving education in the US is college and career success. And many schools that I’ve worked in, for instance, have that plus perhaps lifelong learning as part of their school mission and vision. But how does that define the curriculum as such, and there’s really a mismatch, and what would a curriculum look like if we’re really focused on lifelong learning? And while the college and career success is driven so much by standards and by test scores that really are quite empty when it comes to looking at what we need as a society going forward. And going forward, we need people who can collaborate together, we need people who are very well versed in assessing the truthfulness of information and resources, we need people who can listen, we need people who are critical thinkers in terms of being able to deconstruct problems and analyze them and look at their resources and so forth. We need people who know how to recover from a failure and actually see failure as a source of learning and from which positive things can come. Those qualities are not served by high test scores in math and reading and by a curriculum which is so focused on teaching a course to prepare students for the next course. Where does life come in? …students who understand citizenship, who understand the civics of our country, and the government, and our system of checks and balances within our nation and what it really means to be a responsible voter. And on top of that, our students are in a culture in which they are dealing with the effects of pandemics, they’re in a media culture, they’re in a culture of diversity, because our kids below the age of 18 are more brown than white, that demographic is increasing in the country and we’re going to become more and more diverse. We’re dealing already with major effects of climate change. On a more personal level, the kids are dealing with bullying and Infowars, drugs, gun violence, immigration, you name it, and where on earth does the current curriculum prepare students to deal with the culture that they are facing in their own lives and what we need as a country going forward?

John: And I think everything you’ve said are things that everyone would agree need to be addressed. And there have been lots of attempts to bring these things into the school system in terms of an emphasis on lifelong learning, critical thinking, an ability to interpret information, and yet we still see the same basic curriculum being used. Why have previous efforts at reform failed?

Joe: That was the basis of the book. That’s why we wrote the book, because we’ve been doing reform efforts for decades. But we’ve been operating within this bubble of this Nation at Risk archetype, and it’s not relevant to where students are and where our country is. So the curriculum has been set, and then there’s justifications for it, rather than starting at a blank slate and saying: What’s our aspirations? What do you want to do? and then working backwards and deconstructing the subjects to fit that purpose? Like, I’m a math teacher, or I used to be a math teacher. I’m a recovering math teacher, [LAUGHTER] but I also was a philosopher and a cognitive developmental psychology, etc. There are so many different subjects and areas of interest that could be taught, not just in math, but in everything. So why this? Why this curriculum? So when we redesigned the STEM teacher preparation program in Egypt, we had to design an entirely new program. So I went to Cal Poly who was one of our partners, and we had a design session with them. And so it was a group of people at Cal Poly. And I said to them, “we’re going to design a new teacher preparation program.” And I said, “Imagine a teacher preparation program with no courses. What would it look like?” And they were stunned at the question, but eventually, they came around and after three hours, they started to think about it. And they actually did come up with courses, but it was a whole different program than they would have done otherwise. And so I would challenge people to say, what if in high school, there was no college as there was in the first half of the 20th century, it was mostly just high school, there was no college. So what would high school look like? Why would it be there? So the impediment to reform when you get right down to it is the lack of a coordinated coherent philosophy of a school that can inform the curriculum choices that are made. And that this curriculum, that biology talks to chemistry and math talks to each, the humanities are included as a central idea of a school because one of the big problems of people is getting along with each other. So you need history, you need literature, you need to understand human behavior and what triggers you. This is absent, you’re not going to get that from precalculus. And I’m not saying that precalculus isn’t important. But Latin teachers would argue that Latin is important for different reasons. So this is why it comes down to what’s your vision for the kind of life you want to live and the kind of world you want to live in? And then let’s package that in a way that relates and is integral to a child’s life.

Deborah: So, in schools today, many teachers do try to include projects and problem solving with their kids. And we know that that’s very much a part of the pedagogy in many schools today. But the problem solving and the projects are, may be related to the subject matter, and may be of interest to the kids, and maybe not. But the way we did it in Egypt is the schools were designed around the premise that the graduates would be prepared to address the grand challenges of Egypt. And we have grand challenges like climate change, alternative energy, population growth, urban congestion, disease prevention, and so forth and so on. And so all of the courses are designed to provide students with the concepts and skills necessary to address these grand challenges. And so the projects that the students do are projects that are directly related to these grand challenges. And these challenges are things that the kids see every single day when they look out the windows or on their way to school. There are problems that are so pervasive that everyone in Egypt [LAUGHTER] looks around and says “Can’t we solve these problems.” And so what we’re doing is we told the kids, this is what we’re preparing you to do. And so when the kids undertake a project, or work on problems, they’re real, and they’re urgent, and the kids are empowered to think about the fact that they could become agents of change in their country. And that is so different from the kind of projects that we do in our schools today. And when I talk about all the challenges that we face in our society, and the kids are facing in their lives every day, those could be some of the kinds of challenges and problems that a curriculum is trying to address. And those are things that are very, very real and meaningful to the kids. And it sort of takes the kids back to when they were five and six years old and wanted to make lemonade stands to meet the needs of somebody in the community who needs money or something. That altruism, that wonderful altruism of smaller children that somehow gets taught out of them, or many of them, not all of them, certainly. But anyway, I’m trying to take what Joe is saying here and putting it into some very, very specific kinds of things that are so different between a school which is purpose driven, with a purpose which is meaningful, as opposed to a purpose which is really meaningless.

Rebecca: You described in Egypt kind of a context that was ripe for change, and ready to re-envision, repurpose, what would need to happen here to better align our schools for today’s needs in the US?

Joe: I’m glad you asked that. So we brought the process that we used back here to Philadelphia, and we convened a group of 120 people in eight design studios. We had a small grant to do this. And we asked them that question, we said, “What’s your vision for the greater Philadelphia area? What kind of society would you like to live in in the next 10 years?” And we spent time articulating that and then out of that came different purposes of education. It was, you might say, a mission statement… or more than a mission, it was a purpose statement that came out of that vision. And there was like eight elements of that purpose. I’ll give you two of those elements. One had to do with diversity, “how do you live in a diverse society?” And then the second one was, “What are the unifying elements for that diversity?” So if that is your top line theme, you might say then if you’re in biology, how would you exemplify that theme? Well, you talk about biodiversity. And then you talk about evolution as being the unifying principle of biology as one of those things. For chemistry, the diversity of the material world through the elements. And well, what’s the unifying principles to that? So you look at the natural world diversity, which is immense, and then you look at the human world of diversity. What unites us? We have tremendous diversities, not just around race and religion, but personalities and cognitive abilities. So it gets to then how do we live together among all of this kind of human and natural diversity? What unifies us? Kids want to know that. They have to deal with all of us, otherwise, you just get gangs. So that theme of unity and diversity can cut through all of the subjects, look at music, my God, and yet there’s a unifying principle. So those are the two elements, and so when you then sit a group of experts in a room that have diverse subjects, and you say, “Alright, here’s your theme: diversity, unity, you’re a biologist, tell us how would you structure the subjects along those two themes?” And we did the same. And out of that, then we create a curriculum. So it’s like a rug, warp and woof, the threads of a rug becomes an integrated curriculum from that, so that when a student comes into that school, they understand what the school is about, and why they’re learning what they’re learning.

John: How has that worked? How have the changes been received?

Joe: The people who participated were really amazed by the process, and they got a lot out of it. The next step in that is then developing the curriculum, and then implementing it. So we attempted to do that with the School District of Philadelphia. And they were not interested, even though we had this process. But why we were so thrilled and still are thrilled with our work in Egypt is that we talked directly to the Minister of Education, we were in meetings with him, we had his first deputy minister, who is still directing the project. So at the very highest level we had buy in. President El-Sisi visited one of our schools in Minya, Egypt and said, “My God, these students are amazing. I want 100 More of these schools.” And more recently, the minister and the head of the Parliament said we want to have this idea throughout all of our schools,17 million in Egypt, the elements of that idea. So what the biggest change has been in this process is, and the students will tell you, not me, the students will tell you is that they came in as individuals in a competitive environment. They came out of it, they had a personal transformation of seeing something greater than themselves, cooperating with each other, and thinking of it as “we” rather than “me.” And boy, wouldn’t that be great in this country if we had that spirit of let’s just work on this together?

Rebecca: I think your story really attests to the importance of some of the grassroots kind of components of reform as well as real support from leadership.

Joe: Exactly.

Rebecca: You also ventured into Bosnia Herzegovina. Can you talk a little bit about what that project was like there?

Joe: So a lot of this stuff happens through serendipity. We didn’t go out and apply for Egypt, they came to us, quite by accident. So the same thing happened in Bosnia Herzegovina which, as you know, it was a terrible situation back in 1991 with the breakup of Yugoslavia and tremendous genocide there. Save the Children had been working there on a USAID grant to work on a STEM school. This is in 2017. And it wasn’t working out too well. So they called on us, based upon our work in Egypt, and we went over there. And we did the same process with a group of university people and other officials of this design studio. We said, “What’s your aspiration for your country?” And their aspiration was: we want peace, and we have tremendous unemployment, and also people are leaving the country. So we want to be able to have peace, have people work together and not fight each other and be employed. So for them, they came up with the idea that the purpose of education is to be equipped to participate in a knowledge-based economy (KBE). I said fine. Well, what are they? So they listed 10 sectors of a knowledge-based economy? I said, “Fine, alright, so we have design studios, where we list these sectors and the elements of the sectors and then across this matrix was the subjects. And from that they developed a K to 13 curriculum that was integrated around these KBE sectors. So our role there was not to develop the curriculum, but to consult on the process by which they could do it themselves. And so we worked there for a couple of years. And we developed the curriculum app, an application, that was a relational database that had all of this curriculum in four different languages. So, it was interesting.

John: In your book, you advocate moving towards a more purpose-driven curriculum, but we’re living in a country where there’s a tremendous amount of political polarization and a lot of divisions, where there is a already a lot of pressure on schools to move in different directions tied to the local politics of people in that community. What can be done to move towards a consensus on the goals of the educational system in the US.

Deborah: There are a couple of principles that we need to deal with. One is that conversations have to happen across the sectors of community, whether we’re talking a local community, as a community of as large as a state, or national. One of the things of course, we don’t have a national education system as such here. The closest we get to a system is at the state level, but really, decisions are made much more at a local level. And we have tremendous diversity, even within the state of New York, for instance. I mean, you have one of the largest urban areas in the world, and then you have extremely rural areas, and with levels of affluence and poverty at the greatest extremes of the continuum. But conversations need to happen, but they need to be carefully facilitated, because you just don’t get in and say, “Well, what are we going to do?” You have to have very careful guidance and structure in this kind of design studio with everybody buying into the idea that things need to change. And we need to be able to really discuss freely and openly. And I think one of our beliefs is that when you have these discussions, I believe personally, and I’m a bit of an optimist in this respect, that we’ll find that many people, even from very diverse sectors, whether it’s politically or demographically or whatever, really have the same aspirations. I think the key is to identify the shared aspirations, and maybe where they aren’t shared, maybe there’s a way to forge unity between those. So I’m going to stop at this point, and then let Joe continue on with that.

Joe: I think that’s true. I think the key is to have a diversity of voices, and not just the loudest voices. And it needs to be, as Deborah said, very well facilitated in a process that doesn’t allow just the loudest voices to dominate. So that’s part of the process of developing that. But I think you also see that if you do a design studio in Oswego, for example, and then you also did it in Rochester, New York, and you did it in Hazleton, Pennsylvania, I would bet a bag of bagels that you’re going to find a lot more commonality [LAUGHTER] than you would usually think of it, because a lot of people just want to live, they want to have a family or they want to have success. There’s common human aspirations. But we live in a different time period than we did 1800s. So those aspirations have got to reflect the technology and the culture and the context of the times. And why that’s necessary is that you just can’t come in and impose a curriculum, you have to have buy in and acceptance… not to go back to Egypt, what we did was, we had to make sure that it was accepted. So if you have these aspirations, then you design the curriculum, you get affirmation from the curriculum, and everybody knows what the goal is. And once you decide you want to do it, and you have the authorization, if you will, from the decision makers at the policy level, financially, you can do it within a couple of years.

Rebecca: I love that you have great stories about success, and that it’s possible, because I think a lot of times it feels impossible when there’s a lot of division. One of the things that we’ve focused a lot on is the K-12 arena. What role does higher ed have in these conversations? Or what part of the conversation should we contribute to? Most of our audience are faculty and staff, administrators and things in higher ed institutions?

Deborah: I think higher ed has a huge role in several respects. Number one is if you change the way you’re teaching students so that they come out of their high schools with different sets of skills, and a different knowledge base and these kids go into their traditional university classes, there’s going to be a major disconnect. And again, I don’t want to beat the drum about Egypt, but very quickly a story that our students every semester, do a major semester-long, what we call a capstone project. And these are evaluated by outside evaluators. The very first time we did this, among the outside evaluators was a dean of engineering at the University of Cairo. And afterwards, she said, “Oh, my goodness, these kids are at the level of my master’s students or even above, and here they are just sophomores.” She said, “This is just amazing.” And I turned to her and I said, “These kids are going to be coming to your university in a couple of years. What are you going to do?” And she looks stunned. She was absolutely silent. She recognized the disconnect. And so that’s what we would call sort of a ground-up push for change, but there have to be other changes as well. Faculty need to be provided with a kind of support to do their own transformations in their courses and their pedagogy. And that has to be a huge change. But that won’t happen unless they see the need for change. And so that’s why we think actually starting in the high school is a great place to do that. But it also has to come from the leadership of the universities themselves from the top down, so we have all three working together. And of course, the first change has to be on teacher preparation. But changes have to happen in other courses as well, and those will be slower and probably take longer to implement. But the teacher preparation implications for this kind of curriculum, this kind of way of pedagogy and his way of assessment… we haven’t talked about that yet… are really quite significant.

Joe: You’re not going to be able to redesign schools, like we’ve been talking about, without the participation of universities for a number of reasons. One is that people look to universities for permission to think differently, they trust the expertise of professors. So they want to make sure that if they do this, it’s not going to hurt their child’s chances of college. So you need a partnership, a very strong partnership with the University to make that happen. And you need leadership from the President and Provost to give it the heft to make it happen. And so universities have been and would be indispensable to making this thing happen.

John: And many of the discussions at universities are along the same lines, in terms of the purposes that you’ve mentioned at the start of this recording session, it just hasn’t quite made it into the curriculum to the extent which might be needed to truly affect that type of change.

Joe: So in part of my research, I came across the report on general education that was done at Harvard in 1945, after World War Two, and it was about what you needed to live in a free society. And it was a very strong statement about the need to prepare students both at the college and secondary levels to live in a free society and what that meant, and how central the humanities were in that process, but also, that you cannot just teach math or the sciences for their own sake, that there has to be a moral arrow to it as well, because we saw in Germany in World War Two, how engineers and scientists were used in nefarious purposes. And I think universities are caught in this dilemma that students want to come to universities because they want to get a better job. But universities have a social mission too, and it’s more than just private gain that you’re doing with a student, you really want to transform that student into something more than who they are already and to enculturate them with a grander sense of obligation and duty. That’s really the value of higher education and the institutions, as I see it. And I see them as indispensable. So we wrote this book in a way to give a history and a sense of permission and a way to begin talking about these issues with some common language. That’s why we wrote it, because we realize universities have the same problem. Your Gen Ed courses, what do you have in those? But also institutionally, where are you going? What are you about?

Rebecca: I definitely agree with all the things that you’re saying. And I’m appreciative that you’ve written this book. One thing that Deborah had mentioned was assessment. So I wanted to see if we could pick up on that thread very quickly. So when we’re thinking about a purpose-driven mission and curriculum, how does that change the way we assess things? Assessment has been a big subject on the podcast more recently. And so I think it ties nicely into a lot of the things that we’ve been exploring.

Deborah: Well, I’m really glad you picked up on that, because that was absolutely critical. But what we have to do is we first have to decide what is success? And we can’t answer that question until we know what the purpose is. And I was just in a group of educators last night, and they were talking about what are the challenges of grade inflation. I said, “Well, what happens if we change the way we grade so that there can’t be such a thing as grade inflation?” And so what we’re talking about is a way to so radically change things like through the kind of evaluation I was mentioning, where students do projects and they have outside evaluators, is it complicated? Yes. Is it challenging? Yes. Is it worth it? Absolutely. Can you have grade inflation when you use rubrics and do blind evaluation and so forth? Yeah, yeah, you can get rid of those things. But the bottom line question is: how do you define success? And that has to be driven by the purpose which has been, as we have mentioned, deconstructed into what are the skills and concepts necessary to achieve your aspirations?

Joe: In the book we talked about the tyranny of math scores, that and English language arts has been the sine qua non of achievement and value in schools, and if you look at New York’s test scores, for example, for grades three to grade eight, math and science scores, the gap between low income and non-low income people stays the same from third grade all the way to eighth. And that’s predominantly the black and brown, so the gap remains the same. And if you just focus on math, you’re always going to be in a situation where you see these kids as deficient in some way. Whereas if you have the idea that there’s many different things that you can measure, other than math scores, that are of value, you open up the possibility of trying to find assets within students. And you also, through your assessment, you have to allow students to fail, and then to recover from that failure by learning something new, and not feeling that they are labeled… you know, it’s the growth mindset idea. So there has to be assessments that do that. And I’ll just say one last thing, you know, whenever someone learns something, particularly if it’s a misconception they have, and then they come to a correct understanding, there is a moment of vulnerability, where they say, “If I’m going to learn, I have to open myself up to different ways of thinking.” That’s an extremely vulnerable state. So if you have an assessment system that is judgmental, kids are not going to be learning, that’s why it stays the same in terms of the achievement gap.

John: I think there’s growing recognition of that in higher ed, which is why there’s so much discussion of alternative grading systems that encourage students to recognize that making mistakes is part of learning, and not penalizing them by what they come in not knowing but evaluating them based on their achievement by the end of the term.

Joe: Well, we’ve submitted a proposal with UC Berkeley on elementary assessment schemes using learning progressions as a framework so that if you’re traveling from Oswego to Albany, for example, let’s say Erie, Pennsylvania, you have a roadmap of where you’re going. And if you’re not quite in Erie, you’re somewhere in the middle. Well, that’s where you are. It’s not that you’re a failure to go to Erie. That’s where you are. So having this sense of measuring progress, but not evaluating it in terms of whether you’re failing or not, but you’re informing the student, “Okay, you have achieved this much. Here’s where you need to go next.”

Rebecca: Well, I’ve loved our conversation today. There’s lots to think about at an individual level and a social level, institutional level, policy level, there’s a lot of levels. But we always wrap up by asking what’s next?

Deborah: Well, our book has just come out. And so we are really interested in trying to start these conversations, just as you have given us the opportunity to do today and get people to start to think about the ideas that we have. We wrote it with a general audience in mind. But although it’s an academic book, definitely in that respect, it’s being marketed by our publisher as an academic book, but we really want to try to reach a more general audience and get these conversations started. So we want to spread the word, we want people to start thinking about some of these ideas, and maybe talking amongst each other in small book groups or something as: “What do you think the purpose should be?” …and start it both as a grassroots and at all levels, get these conversations going?

Joe: Yeah, for me, it’s not about telling someone what their purpose should be, or giving them our idea of what the purpose should be. I mean, I have my private idea, but more it’s the process by which it’s arrived at in a sense that it can be done. So we look at this as a sharing of our ideas and opportunities. We’ve been struggling with this for many, many years. So when we’ve had this kind of success through this process, that we want to share it. We don’t want to own it, we’re giving it away. That’s what the whole idea of the academy is about. So we’re hoping that through these interviews that we’re doing here, that people will find value in these and will read our book and have discussions about the process and about how things are in their own places. And that there hopefully will be a critical mass developed. So that’s what we’re doing right now.

John: Well, thank you for joining us. You’ve raised so many interesting questions that need to be addressed for education to be more effective. Thank you.

Deborah: Thank you.

Joe: Thank you. You’ve been tremendous hosts.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

344. Failing Our Future

The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, Josh Eyler joins us to discuss research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming  book, Failing Our Future: How Grades Harm Students, and What We Can Do About It.

Show Notes

  • Eyler, Joshua R. (2024, forthcoming). Failing Our FutureL: How Grades Harm Students, and What We Can Do about It. John Hopkins University Press (pre-order link)
  • Eyler, J. R. (2018). How humans learn: The science and stories behind effective college teaching. West Virginia University Press.
  • Brookhart, S. M., Guskey, T. R., Bowers, A. J., McMillan, J. H., Smith, J. K., Smith, L. F., … & Welsh, M. E. (2016). A century of grading research: Meaning and value in the most common educational measure. Review of educational research, 86(4), 803-848.

Transcript

John: The traditional grading system that we are all used to is of relatively recent historical origin. In this episode, we explore research on problems associated with traditional grading systems and possible solutions at different scales and in different educational contexts.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Josh Eyler. Josh is the Director of Faculty Development, the Director of the ThinkForward Quality Enhancement Plan, and a faculty member in the Department of Teacher Education at the University of Mississippi. He is the author of How Humans Learn: The Science and Stories behind Effective Teaching and a forthcoming book, Failing Our Future: How Grades Harm Students, and What We Can Do About It. Welcome back, Josh.

Josh: Thank you. Thank you. Great to be here.

John: We’re happy to talk to you. And it’s been a while. Today’s teas are:… Josh, are you drinking tea with us today?

Josh: Well, you know how I roll, you guys, I don’t have tea, but I have some lovely water. [LAUGHTER]

Rebecca: So tasty. I have a Scottish afternoon today.

John: And I’m trying to cut back on caffeine, so I just have a peppermint tea today.

Rebecca: Well, if you wanted to cut back on caffeine, you could join Josh with water. [LAUGHTER]

Josh: That’s true. [LAUGHTER] Come join the club, John. [LAUGHTER]

John: I suppose that probably would not be an unhealthy thing to do. [LAUGHTER] And I have been drinking a lot more water recently.

Rebecca: So, we invited you here today to discuss Failing our Future. Could you tell us a little bit about how this book project came about?

Josh: Sure. Yes. So it has a couple of origin stories, actually. One is that when I was writing How Humans Learn a few years ago, I was working on the final chapter on failure. And I kept coming across research on grades and grading and the obstacles that they set up to learning. And certainly if you’re thinking about failure as a tool for learning, you immediately are confronted with the fact that in systems that prioritize grades, they push back on the natural cycle of learning, where we try something, we fail, we get feedback, we try it again. Grades are set up in the opposite direction, they arrest that process before it can really play out. And so that is what piqued my interest. And I don’t know that I could say honestly that I thought I was going to write a book about grades at that point. But I knew that it was an important element of what I was discussing. So I’d say a few years after that, I think what we were seeing was a lot of discussion about grades, a lot more experimentation with it. That led me to want to write a book for a broader audience, one that certainly included educators, but one that cut across the K-12, higher ed divide one that was also for parents and policymakers and students, one that really took a bird’s eye view of the larger conversation about grades, trying to pull all of what we know together so that we could move forward as a community to try and enact change. So that’s what got us here. And the process was certainly longer than I expected due to the pandemic, but it was also just a very illuminating process. And I heard some heartbreaking stories as I did interviews for this. But overall, I’m really proud of the work.

John: And you mentioned addressing this through the entire educational spectrum. And I think it’s important to address it through the whole system. Because by the time we see students in college, they’ve already been indoctrinated into a system of grades, and there’s a lot of resistance to change, and also a lot of damage, perhaps, may have been done by the use of grading systems early. Could you talk a little bit about how we ended up with this system of grading, which has been so much a part of the educational culture from kindergarten on up?

Josh: Sure, yeah. And you’re right about the fact that students are conditioned for at least 12 years before they get to us to think about grades as being the most important thing about education. So how did we end up here, we could have a podcast episode that would last five hours if we really wanted to dig into that. But I’ll focus really on the more recent history. And that is that the A through F letter grade scale that we have is really about 125 years old. Our first records of a full letter grading scale come from Mount Holyoke in the 1890s. So they were the first to implement it. Back then it was A through E. They later dropped the E because they were afraid that people would think that they were excellent when they were not, you know, God forbid that we allow people to think that they’re excellent. And so they dropped the E and added a letter where there could be no mistake about what it meant, and that was an F, for failure. So that’s when it began. But that system, the letter grade system, was not formalized, or standardized, really until the 1940s. That is when you see a majority, in fact, of school districts and colleges and universities adopt that system. And what I think is most important about that, actually two things. One is that none of this has ever been inevitable, that the 1940s is less than 100 years ago, so this is not something that was cast in stone at the minute the American educational system began; it has changed and can be changed. The second thing, though, is that all of these schools, districts, all of these colleges adopted that system, not because they thought it was the best way to document student learning, but simply because it was the easiest way to communicate across institutions. They felt that if they had a common system that they could communicate more effectively between themselves. So it was not about student learning. It was not about student progress or growth or development. It was simply about communicating. That’s how we got here. And now we’re trying to figure out where do we go next? How do we undo some of the damage that has happened since that time?

Rebecca: Speaking of damage, do you want to underscore what some of that damage is? [LAUGHTER]

Josh: Sure, and I know that there’s room for reasonable debate about the terms that we use here. But I honestly really do think we can use the framing of damage and harm when it comes to grades. And in the book, I have several categories. The first and most obvious category for that damage has to do with learning, that grades set up obstacles to learning. They inhibit curiosity, they inhibit creativity, they inhibit risk taking, they affect intrinsic motivation. This is painfully obvious, but grades are classic extrinsic motivators, which means that they are very good for getting people to do things that they would not otherwise want to do. Extrinsic motivators operate effectively, if your goal is compliance, you want to get people in seats, you want to get people to turn things in on time, you want to get people to say things in class, grades work for that. But if what you’re interested in is learning and quality, that’s what you need intrinsic motivation for. The grades can get people into learning spaces, but they in no way ever guarantee that a student will learn even though they are actually in the environment. So that’s really important. Environments that prioritize grades also incentivize cheating, and they just stand in the way of a lot of different things. But that’s just the academic part of it. Obviously, those are important, but to me, they’re less important than some of the effects of grades on students’ lives and wellbeing. Grades magnify and mirror inequities that have always been a part of the American educational system. For example, students who attend poorly resourced high schools have fewer educational opportunities, fewer textbooks, more teacher turnover, all the things that are tied to less-resourced schools. And what that means often when they get to college, they have experienced what we call opportunity gaps, which show up in their grades, especially in Gen Ed courses. So those grades that you see in the first couple of years of a student’s college experience, one who has experienced opportunity gaps, those are reflections of the past, they’re not indications of the student’s potential. They are indicators of systemic inequity. So that’s just one of many examples that we could rifle through. Grades are surveillance technologies. They’re often used for punishment, especially in high school, but kind of across the board. So that’s another category. inequities. And then the final one, the one that we’re getting more and more research on now, maybe than we’ve had in the past, and one that I personally care a lot about is that we have evidence now that grades are a contributing factor to the mental health crisis in teenagers and young adults, the academic stress caused by grades. We have a number of research studies pointing us toward the significant contribution of grades to that mental health crisis. So lots of areas here, I think, beyond the classroom. And that’s really our central idea of this book that grades have long afterlives, well beyond a semester, well beyond the classroom, that affect people’s lives, sometimes just in the short term, but often in the longer term as well.

Rebecca: Josh, you went from going from grades are no fun to grades are a super downer. [LAUGHTER]

Josh: Yeah, well, that’s the message. It’s a happy book, Rebecca. [LAUGHTER] It’s not going to win any awards for a happy smiley story, that’s for sure. [LAUGHTER]

John: One concern that some faculty have about moving to alternative grading systems is that, while intrinsic motivation is something we’d all really love to see in our students, there may be some tasks in our disciplines that students have to master which may not be as intrinsically rewarding as others, but they might still be foundational for students to be successful. If we move away from grading systems, what type of motivation can be provided to help students master those tasks that may seem tedious and may not be quite as intrinsically rewarding as other areas that students just find much more interesting?

Josh: Well, that’s a great question. There are a couple of things I’d like to address about this. So first of all, it’s not that we’re talking about either having traditional grades or having no grades, and nothing in between. What a lot of people are doing in the area of alternative grading practices is experimenting with a whole range of models. And so particularly for the types of courses and disciplines that you’re talking about in that question, John, I have found that many faculty gravitate toward a model called standards-based grading, where you can identify the skills, the content knowledge, the disciplinary norms and necessities that are important for students to develop. And the grade for the course is based, not on individual performance on exams, but how many standards they meet, over the course of the semester. And as a feature of that particular model, often students are given multiple attempts and multiple ways to meet the standards. So that’s part of releasing the pressure valve a little bit for students that gives them room to grow and honors the fact that individuals learn at different paces. So it doesn’t matter when in the semester that they hit the number of standards in order to get their A or B, because the process is a fundamental part of that grading model. So I think that’s really important. Another thing that often comes up in these conversations, though, is the fact that we do in fact, have gradeless colleges in America. The one I feature in the book is Evergreen State College in Washington. And they don’t have any grades at all. They have fully narrative transcripts. They give only feedback over the course of the semester. And they teach all the disciplines. So they have found ways to heighten the intrinsic motivation, and to use feedback to really help students navigate the path toward the goals that they have for themselves and for the class. So there are ways to do this. You don’t need grades to keep people in the seats or to have them do things that might draw on less intrinsic motivation. I think there are ways to structure our learning environments that allow us to do similar work without the pressure of traditional grades. And I think that’s the ultimate point here. I think what a lot of folks in this area who are interested in grading reform are trying to promote is not necessarily just ditching grades outright, although we could talk about that if you want, but reorienting students relationship with grades, the messages that grades send to students, so that it’s not necessarily an evaluation, but a tool helping them develop their skills and to push their learning forward.

Rebecca: It’s always been interesting to me that the A through F system was meant to provide some standardized ways of communicating between institutions yet, if you look at any single institutions’ body of syllabi, you can see that the grades mean entirely different things [LAUGHTER] depending on a class, like some might be about attendance, some might be about success on a test, some might be about achieving learning objectives. So it seems like if it was meant to communicate anything [LAUGHTER] across institutions, it certainly isn’t meeting that objective. I don’t know what my question is. [LAUGHTER]

Josh: I’m so glad this up, though.

Rebecca: I had a question, but I think I lost it. [LAUGHTER]

Josh: No, I’ll answer it. There’s a question in there, Rebecca.

Rebecca: Thanks for helping.

Josh: What you’re talking about there is what I call the measurement fallacy. And so there has been more and more work over the last few years, including one that I absolutely love, called A Century of Grading Research, where a team of experts dives into a massive amount of material. And the conclusion that they come up with is that a grade is nothing more than a subjective indicator by an individual instructor on a student’s progress toward that individual’s goals for a particular course. So I set the goals for my course, a grade that I would give is nothing more than an indicator of how much progress a student has made on my goals for my course. They call them learning intentions in this paper. It is not in any way, a kind of universal certification of learning in a particular course or discipline. Nor does it mean that the student who gets a certain grade in my course would get the same grade from my colleague down the hall teaching the same course. It is a truly subjective indicator of progress. And because of that, we cannot really say that grades measure what we have been told that they measure and what our educational systems have assumed that they measure, which is learning. They’re not universal measurements of learning, they’re subjective indicators of progress in the course.

Rebecca: And yet they’re used to make all kinds of decisions about students.

Josh: They are, and you’re absolutely right.

Rebecca: It’s very interesting. [LAUGHTER]

Josh: So I think there has been a lot of ill placed weight on grades without a subsequent amount of investigation into what grades have meant, and continue to mean. This is also what fuels the furor over supposed grade inflation. And there are lots of other tendrils of problems, when you begin to put so much emphasis on grades to do so much work in our educational system.

John: You mentioned Evergreen State College, but another college that had gone in that direction, very successfully was New College, and I teach in a program at Duke in the summer, and I had two TAs from New College, and they were two of the brightest people I had ever worked with, and their mastery of the subject matter was very much equivalent to that of other students I had who came from institutions with very traditional grading systems. So they certainly didn’t seem to be damaged by that. And yet, there seemed to have been a bit of a hostile takeover of New College. [LAUGHTER]

Josh: Yes, there has been but there is not evidence yet that the hostile takeover has affected that gradeless model. There was some discussion that I read in transcripts of board meetings about that grading system, but it has not changed yet. And I’m glad you brought up New College, John, because, and they have this on their website, as recently as last year, they were able to boast that they were producing students who went on to get the highest percentage of STEM PhDs in the country, relative to their size as an institution, which says a lot. If you are a gradeless institution, and your students are succeeding at that rate in STEM PhDs, it means that they’re learning not just content, but also the skills that are necessary to do that level of advanced work. So I think that that says a lot for what is possible, even within systems that do not have anything close to a traditional grading system.

John: And I should note that one of those students did go on to a PhD in economics. And the other student went on and picked up a master’s degree in data analytics. And they’ve succeeded very nicely, despite the absence of grades [LAUGHTER] in their college career. [LAUGHTER]

Josh: Imagine that.

John: When we began this discussion, one of the things you noted was that under traditional grading systems, the grades at the end of courses often reflect differences in the opportunities that students face before they enter our classes. Can you talk a little bit more about that?

Josh: Absolutely. So you have the reinforcing of those gaps, you have extensions of those gaps. So if you think about what happens to a student who comes into a college course with those opportunity gaps, and then is in a system, for example, where grades are curved, so not only then are they trying to bridge over those equity gaps, but their grades are really reflective of those gaps. But now they are also competing in a curved grade system with their other fellow students, many of whom have had more practice at the kinds of exams and the kinds of questions and the kinds of teaching strategies that they’ll see in those courses. So it just starts to pile on top of each other. And I think that this is something that we just don’t talk enough about, it’s just assumed, and many of these students are labeled unprepared or not ready for college, which places all the burden on them rather than this more systemic problem that I think we really need to face. Because we don’t talk enough about it, we assume that grades have to be a part of the landscape of education. And because of that, we, and I say we as the community in education, we’ve just been willing to tolerate all of the issues that grades have brought with them without really pulling back the curtain and looking at what the issues really are.

John: And we lose so many students in the first couple of years of college and many of the students we lose are those who come from low-income school districts, whose parents had less education, and who just come in with less preparation. And that’s not a very equitable way of providing an education.

Josh: Right, absolutely. And so this shows up on all kinds of metrics that colleges care about: retention, progress, graduation, all of which, by the way, are tied to grades and GPAs. That grades are the gatekeepers for keeping those students enrolled and moving through to graduation. So yes, that’s a major problem to address with respect to those students who are coming to us with fewer opportunities and from school districts with fewer resources. I think we need to really investigate the traditional ways of being In higher education, really look closely at them, reimagine them in order to create systems that have equity at the center and where we are actually, I think, invested in student success and helping those students from those different backgrounds to succeed.

Rebecca: I think the other thing to underscore related to equity issues is not only does an institution maybe have policies about grades and decisions are made at an institutional level, but at the federal level grades are used for financial aid. And so if you don’t hit a certain grade point average, and if you don’t have a certain number of classes that you complete successfully, so you’re not withdrawing from them, or having incompletes, etc, then you might not be eligible, you could lose financial aid. So whether or not a student has demonstrated growth in any kind of way doesn’t matter. It’s down to this idea of grades. And depending on the classes they’re enrolled in and the different things that grades mean, we’re essentially not allowing students to proceed because they can’t afford to.

Josh: That’s absolutely right. The more you really take the magnifying glass to this, the more you see how implanted grades are in all of these decisions, and all these key points in a student’s academic career, bringing in financial aid, I think, is a wonderful example of their gatekeeping function. And the idea that you’re raising there, that it doesn’t even give students a chance to try and bridge those equity gaps. It doesn’t honor their growth or development at all, it says you got less than a C average in your first year, therefore, we are not giving you any more money to move through the university, and so many leave, many dropout. And I think that there is a better way to do things.

John: And many of those students who leave end up with a fairly significant burden of student debt that they still have to pay, further increasing the inequity. What types of alternative grading systems do you discuss in your book, you mentioned standards based grading, but what are some of the other areas that you encourage faculty to consider?

Josh: Well, the book is divided into two parts. And the first half is all the problems and the second half is possible solutions. And so there’s a chapter for parents on what you can do in the home to help your children reorient their own approach to grades. And then the last chapter is about systemic change. But the one in between that is focused on what teachers in K-12 and faculty in higher ed are doing in their own individual classrooms to help with this. And so, there’s a whole range. There are the models that are related to standards-based grading, I call them the cousins: they’re standards-based grading, specifications grading, competency-based, mastery-based, proficiency-based, they’re all not very different from each other, they just have slightly different names, and maybe a few modestly different features to them. So that’s one bucket. Portfolio grading is one that I recommend for folks who just want to dip their toe in the world of alternative grading, because a portfolio model, you’re giving a lot of feedback throughout the semester, a lot of opportunity to revise and redo assignments. But ultimately, the portfolio that is turned in at the end of the semester gets a fairly traditional grade associated with it, it’s just that that grade honors the whole arc of the work for the semester. So that’s another one. Contract grading, and its various branches that we have recently seen. So classic contract model was developed in the middle of the 20th century. And it looks something like this: here’s a list of things to do for the course, if you do 15 of them at a satisfactory level… I’m just making up that number…if you do 15 at a satisfactory level, you get an A for the course, you do 12 you get a B, etc. That’s a contract model. More recently, there’s been a version of it called labor-based contract grading,…very prominent in writing studies… that tweaks it a little bit to emphasize more the amount of work, the amount of writing that students do over the course of the semester, and that is tied to fulfilling the particular contract. And then if we’re looking at a spectrum of grading models, with portfolio grading being one of the more conservative of the alternatives, on the other end of the spectrum you have what some people call ungrading and other people call collaborative grading where there are no grades throughout the semester, only feedback, lots of student self assessment, self evaluation, and then at the end of the semester, the students proposed their grade and the final grade is determined through a conference between the instructor and the student about that proposed grade. So there are lots of things that people are trying. I tried to send two messages about this, one is that my goal is hopefully to help faculty find models that will work best for them and their students, that it’s not a one-size-fits-all model. The same thing that works for me may not work for my colleague in the next classroom over. So that’s important. The other thing is that you do not need absolute fidelity to any one of these models. It’s not like Moses came down from the mountain, with tablets that you had to follow to the letter, lots of people are mixing and matching and taking elements from a host of different systems that they find works for them. So I’ve yet to find someone marching down the checklist for engraving or for contract grading, everyone’s kind of putting their own spin on it, and that’s good. Well, it’s actually not only good, it’s important since our contexts vary so significantly across higher ed.

Rebecca: As a lot of faculty start considering these different alternative models, collectively we can start to initiate culture change, kind of with a grassroots model. But systemic change requires policy change and other things as well. Can you talk a little bit about what role we might have in pushing forward some of those agendas as well?

Josh: Like I mentioned before, a major reason for writing this book was because my observation was that a lot of people are talking about grading reform, but they’re all coming at it, understandably, from their own perspective, the faculty perspective, the administrative perspective, the policymaker perspective, and the parent’s perspective. And I think that if we’re thinking seriously about systemic change, we need to have everyone on the same page collaborating to move change forward. All that is to say that I think about this question quite a lot. There’s certainly examples that we have of recent systemic change, some of the University of California schools have changed their grading systems in significant ways. Western Oregon just dropped the lowest grades on the scale, they’re no longer available to give. I just met someone a couple of weeks ago from Bryant University, they’re moving to collaborative grading at scale for their first-year seminar courses. So it is happening, what it requires is a network of people across an institution, all of whom believe that this is not only the right thing, but they understand how to use policy to make the reform happen, but also how to tell the story in a way that all the different constituencies can see themselves in it, and can agree that this is the direction that they need to head. So the networking is important, understanding how policy shapes grading habits and processes at a university. If you’re part of a public system, which I know you and I both are, you need to understand how the system policy and the state policy plays into this as well. So that requires having that network of different people at different positions in a university to really move that work forward. But what I really want folks to know is that you’re not alone in this, that there are lots of people thinking about this, and lots of people who are doing this work, sometimes under the radar, sometimes above the radar at institutions, and that one person taking a step is a step toward change. But multiple people taking the same step together begins to create a movement toward change. And so I want people to know that. I also want people to know that there are blueprints out there for how to do this work, that there are people who study grading reform, there are models out there for successful institutional reform. Given my work for the book, I think we have a lot to learn from K-12 school districts. There are so many of those each year that are transitioning to standards-based grading across the district. Last year, for example, the entire city of Santa Fe, New Mexico, all their school districts moved to standards-based grading. So this is happening. And the more we learn about process, the more we draw on our colleagues’ network and expertise in these areas, the more we work kind of as a collective in higher ed to push this forward, the more change you’re going to see.

John: We give a lot of workshops on campus and some of them are focused on alternative grading systems. We haven’t seen quite as much movement with that among faculty as I thought we perhaps might, because during the pandemic, a lot of people were open to trying some very different things. But there’s been some resistance. We’ve had a number of people trying it, but it hasn’t caught on quite as much as I had expected it to. Why are so many faculty resistant to moving away from traditional grading systems?

Josh: Well,I think this varies from place to place. But I will say that I think that a major thing that we all face as faculty is the pressures of time. Everyone wants to do their best for their students, I really do believe that, but when you’re faced with such a limited amount of time in which you can change elements of your course, you’re probably going to gravitate immediately toward the kinds of content that you’re teaching in the course in the assignments, the course design pieces of it, or you’re going to think about active learning strategy that John and Rebecca talked about a few months ago, rather than addressing the thing that is at the very foundation of our educational systems in America. That’s not going to be the place that you go first, [LAUGHTER] kind of an upheaval of everything that you have known about education. So I think some of it is just, in a system where we all have very limited time, and very few resources, where do you put your efforts, and I think that is driving some of that. I also think we’re kind of seeing this interesting bounce back that followed that period of innovation at the beginning of the pandemic. I think what you see is almost a defensive response to that period of innovation, that what we really need to do to bring students back and re-engage them in the coursework is to try and somehow recapture what our image was a traditional education pre-2020, that we need to move backward toward that rather than capitalize and extend the innovations that we were doing in the early stages of the pandemic. I don’t want to paint with too broad a brush on that, but I have seen some of that elasticity, that rubberband effect, in talking with folks across the country about some of these issues. So I think those two things are at the forefront of why some folks might resist that. And none of those really has to do yet with really trying to unpack people’s own philosophies of education, or how their grades and their grading models tie into their values that they have with respect to education. So that’s even kind of a deeper level that you can’t even get to until you address some of these other concerns.

Rebecca: And when we’re crunched on time, things that we’ve done before are easier to implement. And it’s also hard to find time to reflect on our practices and to do some of the things that you’re talking about, in thinking about how our values align with what we’re doing. We’re really good as institutions of talking about equity, but we don’t always fully analyze how our practices impact actual equity.

John: And we were all the products of a graded system. And we were somehow successful in that, or we wouldn’t be in these positions. And so I think there may be a little bit of psychological resistance to changing something that has been so fundamental to our own experience.

Josh: Yes. I think a huge capital Y-E-S [LAUGHTER] to both of what you’re saying, that Rebecca, your point, we are not very good at really interrogating how our practices may or may not be aiding our equity goals or advancing our equity goals. And John, yes, we have succeeded in graded systems. And I do think the psychological is a part of it. If we were able to do it, why shouldn’t other people be able to manage this? I think that is what some people think. And more than that, I often see a kind of defensive fallback. Part of this is natural, in that we love our disciplines and devoted our lives to our disciplines, but a fallback toward if I get rid of grades or if I change my grading system, what does that mean for the standards of my discipline, for how people will engage with my discipline? And so I think that’s a psychological element of this as well. And it comes out in arguments about rigor and standards, but what it’s really about is the individual psychology, I think, of the person making the argument.

John: Are there other topics from your book that you’d like to discuss that we haven’t?

Josh: I think the reform efforts that I have seen succeed, and those that have failed, always have hinged on communication, at every level. Why are we doing this? What is the purpose of it? How do we bring people into the process very early on at all levels. At the K-12. level, that means parents, teachers and administrators working together. I think at higher ed, it means utilizing the expertise of faculty right out of the gate, allowing them to shape the narrative and the conversation and the direction and draw on administrative and staff resources to help enact that vision, rather than having a top-down mandate that you had to change your grading. That’s never gonna work in higher ed. But I will say that the communication is what drives all of that work. And it is the hinge that either allows one of these efforts to succeed or fail.

Rebecca: Well, we have a lot to think about, but what’s next, Josh?

Josh: So, what’s next? Well, the book comes out at the end of August and so I’m really excited to launch that and to finally have it in people’s hands and have lots of conversations about what’s in there, hopefully, So, that’s what’s immediately next. I am thinking about the next project that has something to do with the students who are coming to college now are the students who their entire K-12 experience has been shaped by the Common Core, and the obsession with standardization in America, and so what does that mean for higher ed? And what does that mean for those students and their learning? So, I’m in the very early stages of trying to think about that a little bit, but that might be what the next project is.

Rebecca: You always have great projects, Josh, and get us thinking about really important topics. So thanks for your work. I’m looking forward to sharing your new book.

Josh: Well, thank you both very much. It’s always a pleasure to be on this podcast, one of my favorites, and I’d love talking to you both about all these issues, so hanks very much.

John: It’s great talking to you again, and I’m looking forward to receiving my copy of this book in the summer. I ordered it as soon as I saw the notification that pre-orders were available.

Josh: I appreciate that, John, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

343. Writers’ Groups

Faculty writing groups can help motivate writing, provide peer feedback, and lead to higher quality writing products. In this episode, James Lang, Sarah Rose Cavanagh, and Mike Land join us to discuss their highly productive long-term writing group.

Jim is a Professor of Practice at the Kaneb Center for Teaching Excellence at the University of Notre Dame, the author of 6 superb books on teaching and learning and is the author of a regular column in the Chronicle of Higher Education. He was the founding editor of the West Virginia University Press series on teaching and learning, and is now a co-editor of a new series at Oklahoma University Press. Jim also was the founder and long-time Director of the teaching center at Assumption College.

Sarah Rose Cavanagh is a psychologist and the author of four books related to teaching and learning. She is the senior associate director for teaching and learning and associate professor of practice at Simmons University and is also a regular contributor to The Chronicle and many other publications. Jim and Sarah are regular keynote speakers and have both provided keynote addresses at SUNY-Oswego.

Mike Land’s early writing and editing experiences included 15 years of newspaper journalism, a masters and doctorate in Creative Writing at the University of Missouri-Columbia, and 23 years of teaching journalistic and creative nonfiction at Assumption, working for many years in the office next door to Jim Lang’s and a short walk from Sarah Cavanagh’s. He’s an Associate Professor of English and Director of the Community Service-Learning Program at Assumption University.

Show Notes

  • Cavanagh, S. R. (2016). The spark of learning: Energizing the college classroom with the science of emotion. West Virginia University Press.
  • Lang, J. M. (2016). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • Lang, J. M. (2021). Small teaching: Everyday lessons from the science of learning. 2nd ed. John Wiley & Sons.
  • Jensen, J. (2020). Write no matter what: Advice for academics. University of Chicago Press.
  • Scrivener
  • A General Education – Jim Lang’s substack account
  • Cavanagh, S. R. (2023). Mind over monsters: Supporting youth mental health with compassionate challenge. Beacon Press.
  • Reading groups at Oswego (in the last three we were joined by Jessamyn Neuhaus and colleagues from SUNY Plattsburgh):
    • Lang, J. M. (2016). Small teaching: Everyday lessons from the science of learning.
    • Lang, J. M. (2020). Distracted: Why students can’t focus and what you can do about it. Hachette UK.
    • Lang, J. M. (2021). Small teaching: Everyday lessons from the science of learning. 2nd ed.
    • Cavanagh, S. R. (2023). Mind over monsters: Supporting youth mental health with compassionate challenge.

Transcript

John: Faculty writing groups can help motivate writing, provide peer feedback, and lead to higher quality writing products. In this episode, we explore a highly productive long-term writing group.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are James Lang, Sarah Rose Cavanagh, and Mike Land. Jim is a Professor of Practice at the Kaneb Center for Teaching Excellence at the University of Notre Dame, the author of 6 superb books on teaching and learning and is the author of a regular column in the Chronicle of Higher Education. He was the founding editor of the West Virginia University Press series on teaching and learning, and is now a co-editor of a new series at Oklahoma University Press. Jim also was the founder and long-time Director of the teaching center at Assumption College. Sarah Rose Cavanagh is a psychologist and the author of four books related to teaching and learning. She is the senior associate director for teaching and learning and associate professor of practice at Simmons University and is also a regular contributor to The Chronicle and many other publications. Jim and Sarah are regular keynote speakers and have both provided keynote addresses at SUNY-Oswego. Mike Land’s early writing and editing experiences included 15 years of newspaper journalism, a masters and doctorate in Creative Writing at the University of Missouri-Columbia, and 23 years of teaching journalistic and creative nonfiction at Assumption, working for many years in the office next door to Jim Lang’s and a short walk from Sarah Cavanagh’s. He’s an Associate Professor of English and Director of the Community Service-Learning Program at Assumption University. Welcome back, Jim and Sarah and welcome, Mike.

Sarah: Thank you.

Jim: Thank you.

Mike: Thank you.

Rebecca: Today’s teas are: …Sarah, are you drinking tea today?

Sarah: Never. You know me. I’m drinking seltzer. It’s too late in the day for coffee. [LAUGHTER]

Rebecca: How about you, Jim?

Jim: I had several cups of tea already today, it’s orange pekoe? And then my last one was English Breakfast actually at about one o’clock.

Rebecca: Nice. How about you Mike?

Mike: I am sampling some Bengal spice which the box tells me is brimming cinnamon, ginger, cardamom, and cloves and it’s an adventurous bland.

Rebecca: Does it taste adventurous?

Mike: It does. Everything about this is an adventure to me. [LAUGHTER]

John: And I have a blueberry green tea today.

Rebecca: And I have not my favorite tea. [LAUGHTER]

John: And what do you mean by that?

Rebecca: It’s not a good tea. It’s the first time I’ve ever had it. It’s supposed to be gingerbread. And I think it’s kind of disgusting. [LAUGHTER]

John: Now you know why we never get any sponsorships from tea companies.

Rebecca: I didn’t say where it came from. [LAUGHTER]

John: That’s true, but I’m sure there’s probably not too many gingerbread teas out there.

Rebecca: There shouldn’t be. [LAUGHTER]

John: In past podcast discussions with Jim and Sarah, we had heard about the writing group that you’ve all participated in. Can you tell us how this group got started?

Sarah: I guess I can take that one. I think it was about 2014…2015. Mike, do you know the year? He’s giving me a big nod. 2014. And Jim had recently invited me to write the Spark of Learning and his first series at WVUP and Mike was working on a book…had just come off sabbatical doing a cross country trip. And Jim was working on the first edition of Small Teaching. Since all three of us were working on book projects, Jim said, “Hey, why don’t we form this writers’ group.” And we started meeting. And it was really such a beautiful process to write my first book with my editor [LAUGHTER] reading every chapter as it came out, was a very developmental process. And as well as forming some really great friendships.

Rebecca: Sounds like a great start to a long endeavor. Can you talk a little bit about how regularly this group meets and if you still continue to meet?

Jim: We do continue to meet, but not regularly, as you might expect, [LAUGHTER] especially just because we were initially together at Assumption and were all working in the same area, and I just mean sort of geographical, and now Sarah’s in Boston, I’m going back and forth between Worcester and South Bend. So it’s a little bit harder to get us together in the same place at the same time. So we kind of meet on a more ad hoc basis at this point. if one of us has a project that we want feedback on will sort of send an email around saying “I need help, [LAUGHTER] and so let’s get together and can we have a writers group coming up?” So we’re still probably doing that maybe six to eight times a year, I would say. Maybe we were trying to do it every month but now it’s kind of morphed into that kind of more occasional kind of schedule.

Sarah: So we usually squeeze in a Christmas one.

John: That’s true.”

Sarah: So usually a mid December writers’ group. Some of them are in Zoom just because of how we’re spread around but the Christmas one is always in person. It’s really nice.

Jim: Yes, we have holiday themed groups. [LAUGHTER]

Mike: …with teas.

Sarah: Well, for some of us.

Mike: …or beers, but yeah, we’ve tried to have a couple of year that are also just social and in person. We’ve got more used to Zoom,as everyone has, and since we’re in three different…well, Jim and I live two doors from each other, but it’s harder to get us all together physically in the same space, but we’ve try to do that two or three times a year.

John:. When you were meeting regularly at Assumption University, what happened during the meetings when you got together? How long were the meetings? And what did you do in the meetings?

Jim: That would also depend upon where everyone was, in terms of their writing projects. A typical meeting would last maybe 90 minutes. Mike’s especially a social guy, [LAUGHTER] he would sort of want us to start with some social time, we sort of built that in. And so we’d have a little social time initially, you might be around a meal, for example, for the first 15 to 30 minutes, just catching up on our lives. And then we kind of just go through the project, essentially. So we essentially will just pick one off the pile, and say, “Okay, so here’s my essay this time,” like, for example, I might be doing a Chronicle essay or a chapter of a book. You have to read it in advance. So that’s an important thing to note, that we don’t read out loud, which sometimes writers’ groups will do that. You read in advance, we send the stuff out at least 48 hours in advance. You’re expected to read that material, come into the group having already written comments, either online or actually some of us still use the paper versions and bring those in as well. So then, essentially, the person whose material that we’re critiquing is supposed to just sit and ask questions, and the people that have read will kind of go through, essentially page by page, there might be like an initial comment, “overall, this is what I think of the piece,” but then we kind of go through page by page. That doesn’t mean every page is commented on, because it depends on where the issues are. But say, each person’s piece could take 15 or 20 minutes or so to walk through it. And then we move to the next one. And so we move around the circle that way. That’s the basic process, but what am I leaving out?

Sarah: No, I think that’s most of it… Kind of the flow.

Rebecca: Can you talk a little bit about the benefits of having formed the writing group and what other people might think about in forming a group?

Sarah: It is so invaluable, I think, in so many different ways to have first readers who can read something. It’s so vulnerable writing and putting your thoughts out into the world, and to have a safe place where people can give you feedback on very early drafts, I think is so valuable. And it’s also a really good test of are you expressing your ideas well, and are your references sound. I’m always telling Jim, people who don’t know sports aren’t going to understand this reference, and he does the same for my sci-fi references, and so that kind of tells you what a broader audience might think of your ideas. And it’s just a place to bounce your ideas around as well, like sometimes we do a lot of line-by-line feedback, and Mike is awesome at that from his journalism background. But we also sometimes just ask advice and just say, Is this even working? Should I draft this project and go to something else? Kind of really broad ideas as well. How about you, Mike, what do you see as the benefits?

Mike: The things you said. I think it’s a lonely pursuit, and if you don’t have an audience of people who are intelligent readers to let you know whether you’re totally going in the wrong direction or not, you’ve been staring at what you’ve been working on for so long, you convince yourself that yeah, this makes perfect sense, or this was a good move, or they’ll laugh at this, whatever it is. And so at some point, you got to try it out on audience members you trust and who also understand the writing process, because they themselves are writing and have written a lot. So they differentiate some things, maybe what not to do or to do. There’s a temptation in fiction writing, when I was in the fiction writing program, we were warned about trying not to turn the short story of the person being critiqued just into the story you had felt like writing instead. [LAUGHTER] Well, I figured out what the person is trying to do, and help them do that, and stay locked in on that, to some extent. So yeah, there was a lot there. And it is an extra deadline and something to get you moving.

Jim: That’s an important thing to know, too, that it was oftentimes a way of saying, “Okay, I have this book coming out, so I want to make sure I go chapter by chapter, and so I would make the writing group deadlines, my internal deadlines for getting those chapters done.” That’s a really, really helpful structural enabler, when you’re trying to especially do like a long project. The other thing I mean has come up in actually both these comments from Mike and Sarah, about the audience. And that’s true in terms of your audience. Think about your ideal audience who you’re trying to really write for. It’s also really true for your editors and potential publishers. And we often talk about that, like, “Okay, who would really be the right place to submit this to?” And we might be able to share examples of the publishers that we’ve worked with or strategies for reaching out to an editor, maybe we have a connection, and we know somebody. So we can have also that as well. So especially if you have books in your writers group who have published other things, you can start building networks around a writer’s group and I think that’s really important, too. Again, I think what Mike and Sarah are saying about audience is a real big benefit of you’re sort of thinking about the readers that you’re trying to reach, and you can often refine those. The example that Sarah gave about references, like if I’m writing to a community of people who love science fiction, I don’t have to worry about those references. But if I am going to expand beyond that audience then I have to either explain them or make more universal ones, and I think that process of like hearing at least two other readers, helps you expand a little bit in the number of people that you can reach with your writing.

John: And I would think having that relatively timely feedback would be much better than writing a big chunk of a book, sending it off, having it go out to reviewers and then getting reviews back when it’s perhaps a little bit late to make some of those major changes, that getting that more immediate feedback, I would think, would be helpful in keeping the writing project going in the direction you intended.

Jim: Absolutely. For example, I had like a little mini readers group with my Chronicle editor. Now I’d submit my work to her, and she would really sort of hack it to bits, [LAUGHTER] in a nice way, she’s a great person. But over time, I’ve sort of recognized, this is what she wants, this is what she looks for, this is what she changes. And I kind of learned to sort of mold myself into that kind of form that she wants. And the same thing is true for your writers’ group, you can sort of start to anticipate the things that they’re going to point out to you, which are often little habits that we have, which are not always great ones. [LAUGHTER] I always remember that Sarah would always point out that I used the word “so” all the time as like, “and so this,” and I never would have noticed that unless she started to circle all my “so’s” and then “look how often you do this,” that was like a really helpful thing for me to know. So now I’m aware of it, I still use them, but I’m just more aware of it now. So it’s very helpful to see other people’s perspectives and then you can start to anticipate that.Sarah sometimes uses phrase, “I had Jim in my head when I was reading this sentence,” or “I had Mike in my head…

Sarah: Yep. [LAUGHTER]

Jim: …telling this anecdote” or something like that. So that’s a really helpful thing to have as well.

Mike: Yeah. And it’s interesting, you’ll learn those things about each other, and then we can decide when to accept or not accept that, right? [LAUGHTER] In my case, and I think it’s true of Sarah, we can anticipate that if we throw in one too many anecdotes, we know Jim’s going to be there waiting to pounce on our extraneous anecdote that we love. And I’m somewhere in the middle, from the South, so somewhat more elliptical in my storytelling and less linear. So I’m favor of the extra anecdote. But there are times when you can tell, when you looked at it, and you have that voice and you go, “Okay, this really is extraneous, get it out of the way, don’t even bother other people with it.” And that’s that kind of thing and that’s kind of why we assign undergraduate students to do peer editing of each other is to learn to be editors, and to hear those voices too. So it’s really useful that way.

Jim: Mike, you know about other readiness groups and this question, John asked us about, like, guidance for other groups, can you give some examples, Mike, if you know of other writers’ groups that have sort of different ways of working?

Mike: There have been some groups that seem to exist just for people to pat each other on the back. And they don’t really do much criticism at all. A friend of mine who’s dyslexic said that he was in a group where you could never correct anything ever, grammatically or stylistically, which would drive me crazy, because I would want people to note every time I slipped up on something. So those are a couple of examples, and I’ve been trying to come up with some more, but it seems like there are a lot of variations in the rules in how they do it.

Jim: Sarah, you have also one that’s more like keeping you accountable, right?

Sarah: So I have another writers’ group with some friends at Tufts University, and we call it “writing and hijinks.” And so we gather together in a coffee shop or somewhere and we just write quietly, although one of them, like Mike, is very social and has trouble with the quiet part. [LAUGHTER] And so we just write, it’s just dedicated writing time for a couple hours, and then we go out and have fun. Although we’ve been sliding more and more towards just the fun and not doing the writing [LAUGHTER] over the last few years. This is motivating me to say to them, maybe we should bring back the writing part.

Rebecca: We have a writing group at Oswego that functions like yours, Sarah, that you’re talking about, where it’s just kind of some dedicated time, accountable time, this is the time we’ve set aside to write…

Sarah: Right.

Rebecca: …which can be really helpful if you have a designated time and space to do that, if you aren’t good at scheduling it on your calendar otherwise. Julie

Jim: Joli Jensen has a book about writing from the University of Chicago Press, and she actually makes the argument that writing groups for academics should not be content critique groups, which is what we have, actually. And she makes the argument that it’s not helpful, for example, English professors to be working with a psychologist because they won’t get your references and all that kind of stuff, and it won’t be very helpful to you. But actually I would argue that what we’re doing here, because most of us are running for, like broader audiences, the content critique part is really important. If Mike and Sarah don’t understand what I’m saying, well, then I’m not gonna be able to reach a broader audience instead of just writing specific to my little expertise audience. So that’s another way to think about distinguishing content, whether you focus on the content or just the writing, the sort of grammar and the stylistic part of it. So that’s another way to think about what are you actually trying to do in your writers’ group? Are you trying to expand beyond your discipline and getting different kinds of feedback? Or are you just trying to focus on, as Mike said, maybe patting each other on the back or accountability or style and grammar? So there’s a lot of different ways you can go.

John: I think that would depend a lot on who your intended audience is. So that if you’re all psychologists, or all economists, or all mathematicians, if you’re going to be submitting things to journals, it might not be a bad audience. But for a general audience, having someone who is not necessarily an expert in the field, seeing it from a different perspective would lead to a lot of really productive comments in terms of what some of the hidden assumptions are, the implicit assumptions in the writing.

Jim: Absolutely, you can actually even think about it as a teaching thing too, I mean teaching to like a gen ed class, their first-year students, that’s almost like a content critique group in the sense you can’t take too much granted about background knowledge in discipline, like a really focused content critique group, all from a discipline, you’re essentially teaching a graduate seminar. And so you’re looking for different things from those experiences, and you’re being required to explain different levels of things to those two audiences.

Rebecca: So I’m curious what participating in this particular writing group with the three of you has done for you as writers? How has it shaped your writing? And why are the three of you a good group for each other?

Sarah: I think that one of the things that has really shaped my writing process, this is beyond all of the wonderful things that we have said already about audience and things like that, is being in a writers’ group with two English professors is really valuable. I tested out of intro writing in college, so I actually never took a writing course of any sort, and then decided I wanted to be a writer. And my writing is very organic and intuitive. And working with these two really helped me like the things that they would correct. Like I didn’t know periods went inside the quotes, until Mike corrected me. [LAUGHTER] But the first year, we were writing together, and you would think I would just absorb that from reading, but I just never noticed that. And so really tiny things like that, but then also just thinking about the structure of your writing and the organization of your writing, because I had never really done that for this kind of general audience writing. And so having this writers’ group with writing professors is really amazing, as Jim would sometimes send me to the whiteboard, when we would meet [LAUGHTER] at Assumption, because it’d be like, you’d have a lot of really interesting ideas here in the writers’ group, but they’re just in a big pile,[LAUGHTER] and he’d have me map out, do a literal map of my work. And that was very, very helpful.

Mike: That’s one of the fascinating things about being in a group is working with two people that have such different composing processes. And Jim, you’re fairly linear…

Sarah: fairly…

Mike: kind of have it in order [LAUGHTER] and smoothly.

Sarah: …the computer. [LAUGHTER]

Mike: So what was the computer program that you’re using that divided the screen into quadrants?

Sarah: Yes, Scrivener, you can put your bits of writing on literal digital index cards, and just move them all around.

Mike: Yeah, which is something I’ve seen poets do. I’ve seen poets tear their poems apart, line by line and move the lines around in the coffee shops. So it’s just two equally effective ways of doing that. And I guess I’m somewhat in the middle, but with the road trip narratives, obviously, there was someone’s this town and that town, but it’s fascinating to watch how people’s brains work differently.

Jim: Yeah, I think for me, Sarah and Mike are different kinds of writers. But as a result, I’m kind of benefiting from both the sort of storytelling advice that comes from Mike and from Sarah, again, more of the research, making sure that I’m supporting my points well, and thinking about other perspectives on the topics I’m addressing. And it’s helpful to get both those things, because I’m getting different things from others, like editors, for example. I like the idea of starting a piece of writing with a story or something like that, my editor at The Chronicle always wants me to lead with the thesis, essentially. And so I’m trying to put some of Mike’s ideas advices into that kind of storytelling opening, but also trying to satisfy my editor. So anyways, it kind of helps you develop new strategies and see how strategies might land with different people and Sarah’s especially good at ways that I should include more readers, more thinkers from different backgrounds, and essentially making my writing more inclusive. And I think that’s really valuable as well, and making sure that I engage with the research that I might not be aware of, in the teaching and learning fields. So that’s really important, too. So I can look at my own writing and see how it’s benefited from both their perspectives.

Rebecca: I think one thing that I’ve benefited from when I’ve participated in writing group opportunities is actually seeing early stages of other people’s writing, and just what that looks like and feels like and sometimes provides strategies to future me [LAUGHTER] when I’m stuck, [LAUGHTER] like, “oh, wait, this other person would have attempted it like this.” And when I felt stuck on something, sometimes I use the strategies that I’ve observed other people using, which can be really beneficial, that things that I wasn’t taught or hadn’t been exposed to otherwise.

Sarah: That’s great.

John: You mentioned future me. And one of the issues that we know from behavioral economics and psychology is that when given a choice between pursuing your long range objectives and immediate gratification, immediate gratification sometimes wins out, [LAUGHTER] because it’s always easier to start these things tomorrow or next week. And one of the things you mentioned earlier, Jim, was that having the regular meetings of the writing group provided a bit of a commitment device that you want to get to the meetings with some new material you want to share that material before it. Do you think that has helped increase the pace in which your work has taken place?

Jim: For me absolutely, yeah, having that structure there. I think structure is always a really helpful thing in teaching and writing. If you sign a book contract, the deadline might be two years away. So you know that two years, an editor might check in after a year, but like, you need to have more structure in that to kind of get yourself moving along. And I’m a pretty structured person, but I can also fritter time away. I’m pretty good at that, too. [LAUGHTER] But I’m self aware enough to know I need a little help in terms of structure. And so it definitely has pushed me to become more productive.

Sarah: Absolutely. I agree. This whole conversation is making me want us to do this more regularly, [LAUGHTER] instead of the as needed, because I think about especially the beginning of the writers group. And when I was writing Spark of Learning, my daughter was a toddler. And she was only in daycare three days a week, and I was on the tenure track and just doing a billion different things. And I do not think I would have written that book if it weren’t for writers’ group monthly meetings. I think at that point, we were really regularly meeting in person monthly. And just finding the time at four in the morning [LAUGHTER] or terrible times often, but I don’t know that I would have gotten it done if it were just like, “oh, I have a contract and I must do this giant thing.”

Mike: That definitely helped this begin…the structure. I don’t have a long-term project I’m working on currently so hard for me sometimes to come back around and really have anything that’s worth them looking at. But yeah, you mentioned those long period of the book contract, and there’s a long period of only doing shorter things, whereas so there’s not that continuity from chapter one to chapter two to chapter three, and you’re eager to write those next two pages that are the bridge to the next thing, so group can help with that as well. It seems like a lot, but what happens in writing group is when someone is working on a Chronicle column that needs to be turned in in 48 hours or letters and things like that. And I was wondering if you can give examples of that kind of stuff when the group comes into play. That’s not the big primary thing we’re working on, but it’s kind of the emergency stuff.

Jim: We’ve done a little bit of that, but it’s kind of hard because especially for an upcoming deadline, I think typically we’ve handled that when someone just needed something very quickly, we just handled that by email. But the writers’ group is available in those emergency situations, because we already have a quick sense of like, if Sarah sends me something, and it was for a quick turnaround, it’s for The Chronicle, I can be able to spot very quickly a couple of things that you could maybe do in that short window. And likewise for me. So it’s challenging, though, because you want to have time for people to have opportunity to read and think about that as well. But it’s available for those situations. So it’s good for that.

Rebecca: I can imagine that you’re strong relationship with one another, having written with each other for so long allows for that quick turnaround by email, because you can sense the tone someone’s providing some feedback in [LAUGHTER] and it…

Sarah: Help!

Rebecca: …has a different kind of context. [LAUGHTER] Yeah.

Jim: Absolutely.

Sarah: And it’s such a pleasure, like it’s hard to ask for emergency work from people, even people you’re close with, and maybe even especially people you’re close with, because you know, they have so much else going on. But it’s such a pleasure to read these gentlemen’s words, that it’s not a burden at all, even the emergency ones because it’s like, “Oh, I get to relax, not with a cup of tea, [LAUGHTER] and read some words from Jim or Mike.

Jim: That’s true. That part of it is good, because you know that you can help them in the emergency because they’ve helped you in the past, that same situation. Right now, you know, I’m sort of developing a substack account. And I haven’t really used the writers’ group for that. And as a result, I need to have my wife proofread for me,[LAUGHTER] because substack doesn’t have an editor, and I’m really annoying her by this process. So I need to start using my substack columns for writers group stuff as well.

Mike: Yeah, it is really helpful and we know that there’s just a lot less explaining to do amongst the three of us. If I were going to send something cold to another friend, I think I would have to explain a lot more about what was going on for them to get it. So that’s incredibly useful that way. But it’s funny how different it is from like growing up in a newspaper family, and I would interview someone in the morning and I would revise it four or five times in a period of like four hours and hand it to my editor who would then tell me to change three things. And the whole thing from beginning to end was less than a day. But there was no deep thought, there was no deep reflection or big structural things, and then my time sort of expanded magically as I became an academic and suddenly you can get trapped in a lot of whirlpools of uncertainty about which way to go or that way to go. Whereas in journalism, there wasn’t much choice a lot of the time.

Sarah: Writers note, I love that “whirlpools of uncertainty.” If I were in writers’ group, I would say underline it, put a little exclamation point next to it. [LAUGHTER]

Mike: Well, I was a really terrible student in an undergraduate poetry workshop at the same time my sports editor was trying to teach me to write in a linear way. And my teacher said at one point, Mike what you’re writing, as poetry is prose with line breaks in random places. And he said, “Let the language go where it wants to go.” And I was having a hard time imagining telling my sports editor that on [LAUGHTER] Friday night during high school football season, but it seriously took a long time for me to learn I could do both. I could loosen up and be this one kind of writer, and then be this very mechanical quick writer. And that took years as an undergraduate to figure out how can I do both of these things at the same time.

Rebecca: One thing I heard Sarah say is that currently you’re meeting as needed, but then I also heard in the conversation, that maybe we need these things, and we just don’t know it. So having it regularly on the calendar could be helpful. [LAUGHTER]

Sarah: Mm hmm.

Jim: Yes, we’re convinced now this conversation is very helpful to us. [LAUGHTER]

Mike: Yeah, and plus this almost the end of the semester, so suddenly, we’re gonna have a lot of time on our calendars.

Sarah: …well, those of us still teaching. [LAUGHTER]

Mike: Yeah.

Sarah: I’m at 12-month contract now.

MIKEL Oh, you are?

Sarah: Yeah.

Mike: For you, year round.

Rebecca: Me too, Sarah. [LAUGHTER] Maybe we need a separate writers’ group. [LAUGHTER]

Sarah: Or therapy. [LAUGHTER] …support group.

John: What sort of advice would you give to other people who are thinking about forming a writers’ group?

Jim: Actually, there have been been other people in the writers’ group over the years, I’ve been in other writers groups too. For me, the thing that I look for is people who I thought were sort of ambitious and were writing, because you don’t want to be the person who is trying to gather people together, when they’re just not writing, they don’t want to write, you don’t want to be like the taskmaster in a writers’ group. So find people that are writing and are interested in writing, and find people that you find interesting, you want to hear more of their thoughts. So that has always been a driver for me. And sometimes, if I’m looking for people that I want to hear their thoughts about my own writing, and I want to talk to them about their own writing, I’m gonna be more interested in going to that writers’ group and participating. And so those are sort of two general points I would make: look for people that are writing and interested in writing, and then people who are interesting, you want to see it develop and getting access to their ideas as they’re developing. And I would just say, most people, I think, are flattered, you would like to hear and read their work. So if you identify someone that and you might say, “Well, oh, you know, they’re probably too busy, or they probably already have other people to work with,” that might not be true. And so extend the invitation and see if they want to join you in pushing writing forward.

Sarah: And I think I would add, it’d be hard to figure this out, [LAUGHTER] but one thing that I appreciate about this group, and I would want in any writers’ group is having a similar level of like sensitivity and openness to feedback and criticism. So I think that all of us are really great at taking feedback and taking constructive criticism, without either taking it personally, or I think Mike referenced this, but sometimes the Jim in my head or the Jim on my page, I’m like, “Yeah, I’m not listening to you.” [LAUGHTER] But then a lot of times you do. And so t o people who are at that level of like, open to hearing criticism, but also confident enough to stick with their gut if they feel it’s right. And yeah, and so that’s a delicate thing. And it’s hard to figure out about someone without actually engaging in writers’ group. But I think if they were in your department meetings or on committees with you, kind of keep an eye and get a sense for how open they are to other people’s thoughts and how willing they are to give them.

Mike: Yeah, and along the line with open is something I always warn undergraduates about, but in grad school it would come up with separating the characters on the page, even the narrator, that that person I’m writing the first person about, it’s Mike, the narrator on that page, but it’s not necessarily all of what I am. So, ‘cause when you’re talking to someone, then you can point out stuff that you think is maybe sexist, or is whatever, but be able to talk about it in a separation where you’re not personally attacking the writer, but that you are holding that in a certain place. I think we’re really good about those sorts of things, we know the level each of us are trying to get to, we’re just trying to help each other get there and again, smooth out anything that the audience might misinterpret. That’s a mental discipline that takes a while to be learned. And I’ve had friends who have been in groups where things have devolved to name calling, and to really unfortunate situations where the writing group had to be broken up, and so I’ve even heard that happen in academic settings, like in actual seminars. Fortunately, I was not part of any of those, but where people went stomping out of the room. And so yeah, having the maturity in just the way you carry yourself in life, I guess, to know how to treat other people and work with them constructively, a really important thing. But otherwise, in terms of advice, let’s say that whatever kind of writing group you want to have, you want to sit and just right beside each other and then go have drinks that what this one’s gonna be, are you all aiming to publish and you’re trying to help each other get published? That can be a very different mentality, depending on what you’re trying to do. That’s my two cents.

John: Do any of you have any other thoughts about writing groups that you’d like to share with our listeners?

Jim: I will say just as an editor, you know, editing the book series, I can also sometimes tell people who had writers’ groups when they turn their manuscripts in who had prior readers. So just from my perspective as an editor, I absolutely recommend this, forming a writers’ group, especially if you’ve signed a contract, and you’re intimidated by it. The fact that you have this deadline, you have this big project coming up, A writers’ group can not only put some structure on to the process but also forestall a lot of the work that I will have to do as an editor. So you can make my life easier when you’ve had some prior readers who kind of work with you to iron out a lot of the kinks that have developed, as any writing project will have in the drafting.

Sarah: Yeah, my editor was very grateful for writers’ group. We met monthly, my editor for Mind Over Monsters and she was wonderful. But writers’ group was like a third character in the room. [LAUGHTER] Because we would meet monthly, I’d be like, “Alright, well, here’s the chapter, but this is what writers’ group says, and they think that I should reframe this.” So it was really like the three of us, or I guess, the four of us, melding you two into one being in the conversation and in the room. And that was an interesting process.

Mike: I think another thing that’s been interesting to me is that because Jim and Sarah have books out a lot of the time, and there’s a good bit of talk about the publishing aspect of it, there are times when say, Jim might know someone that one of us should hit up, or contact, or things like that, or how to interpret an editor’s reaction. And there’s been a good bit of conversation about those things where I kind of felt like I know, these editors I’ve never met. [LAUGHTER] I’m sure that’s a big stopping point for a lot of people is they’ve written this thing, and now they don’t know what to do. They don’t know what the path is, and how to get there, how to get someone’s attention. And so there’s a lot of talking back and forth about things like that.

Rebecca: Do you have wishes for your group moving forward?

Jim: I guess, more regular meetings.

Mike: …more regular meetings, yes. [LAUGHTER]

Sarah: Sounds like a way to save Jim’s marriage.[LAUGHTER]

Mike: I guess, I should sell them another road trip so I could write another road trip manuscript.

Jim: Yeah.

Sarah: There you go. I would read that.

Mike: As we sneak toward possible retirement…a travel book…that could be the plan.

Rebecca: So we always wrap up by asking: “what’s next?”

Sarah: Well, more regular writers’ groups, I guess. But I think as Mike just referenced, he’s retiring. So he’s going to have a lot more time to write coming up. And Jim and I are both, I think at the phase of thinking about next book projects. And so maybe it’s going to be a new day for writers’ group.

Jim: That’s true. I’m actually doing the final, final, final revisions of my current book manuscript, which are due on April 22nd. So that will be coming up shortly. And so I’m in that phase where like, I’m kind of finishing everything up, and also the editing and all that stuff, and copy editing, but still at least starting to open up in the sort of creative part of your brain about like, what might be next? And so that’s maybe one final thing I might say about writers’ group is sometimes we will come in with just like a one pager of ideas. And Sarah has done that several times. And we kind of give her feedback like, this is the one that seems like it might be the best path to pursue, for example. Anyways, I’m in that space right now. So I hope that for me going forward writers’ group might help me land on the right project next.

Mike: I should mention too that just the venting of frustration is good. And it just crossed my mind, one of Jim’s funnier Facebook posts of all time was when you were finishing up a book and what you posted about how you felt about the book. I don’t know if you recall this or not?

Jim: Yeah, absolutely.

Mike: So tell them. [LAUGHTER]

Jim: So I just remember talking about the fact that whenever you finish a book, I’m just so sick of it. At that point, I never want to think about it again. [LAUGHTER] And then of course, a year later it comes out and you have to think about it all the time. [LAUGHTER] I’m a person that has multiple interests, and they sort of change on a regular basis and like, by the time I finished a book, I’m like, “Okay, I’m done with this now, what’s the next thing?” But then, you know, hopefully a year goes by when it’s coming out, and then you get interested in it again during that time, hopefully.

Mike: Well, the specific line I remember was, “I hate, hate, hate, hate, hate you, stupid book.” [LAUGHTER]. It was that quote.

Jim: Yeah.

John: Well, I can see it’s probably good that you focus on the writing and not the marketing of these books. [LAUGHTER]

Jim: Yeah.

Mike: It wasn’t this book, it was a different book.

Rebecca: Well, thank you so much for joining us. It’s always a pleasure to talk to you about your work. And we’ve been hearing about this writers’ group [LAUGHTER] for a while. So we were glad to have the opportunity to dive in a little more deeply.

John: And we’re also grateful that we’ve been able to benefit from the writers’ group. I was just thinking back, four of our reading groups over the last few years were products of this writing group.

Jim: Wow.

John: So we appreciate the work that you’ve been doing.

Jim: That’s very cool.

Sarah: Thank you. It’s an honor.

Mike: Yeah, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

342. Infographics

When papers and projects are due at the end of the term, students often procrastinate even when the projects are carefully scaffolded. In this episode, Michelle Kukoleca Hammes joins us to discuss how a series of infographic assignments, combined with peer and instructor feedback, provide an engaging and productive learning experience. Michelle is an associate professor of political science and a CETL Fellow for the Center for Excellence in Teaching and Learning at St. Cloud State University.

Show Notes

Transcript

John: When papers and projects are due at the end of the term, students often procrastinate even when the projects are carefully scaffolded. In this episode, we discuss how a series of infographic assignments combined with peer and instructor feedback provide an engaging and productive learning experience.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Michelle Kukoleca Hammes. Michelle is an associate professor of political science and a CETL Fellow for the Center for Excellence in Teaching and Learning at St. Cloud State University. Welcome, Michelle.

Michelle: Hi. Nice to be here.

John: We’re looking forward to talking to you. Today’s teas are:… Michelle, are you drinking tea?

Michelle: I am. I don’t always drink tea, but I made sure this morning I did. And so I’m actually just drinking a green tea, a milk tea. It’s not quite as good as others I’ve had. I have a colleague from Wales who makes the best green tea, but it’s good this morning.

John: …and Rebecca?

Rebecca: I have a Hunan Noir Tea, which is very tasty.

John: You left out the Pinot.

Rebecca: I know. [LAUGHTER]

John: Okay. And my tea is a black raspberry green tea.

Michelle: Sounds nice.

Rebecca: So we’ve invited you here today, Michelle, to discuss how you’ve been using infographic assignments in your online comparative politics classes. When you first started teaching this course, what kinds of assignments did you use?

Michelle: I use a variety of assignments. I’ve been teaching this course for 27 years now. And then this is mainly for my intro level comparative course. But I’ve adapted it for upper levels too, so mainly traditional assays early on. As I moved to the online environment, I would still have them do a major paper of a country study. And so pretty traditional assignments for the most part, but I wanted to make some changes for the online environment especially.

John: So what made you decide to change to infographics?

Michelle: Well, a few things. One is I felt that students were doing these papers that were quite lengthy in their country studies, and they would turn them in at the end of the semester. And we never really had a chance to talk about the elements of them in depth prior to that. And so I really wanted to change that. So it wasn’t just the end of the semester. And they weren’t sharing them with each other, because of the format of them as just being a lengthy paper, it wasn’t the kind of thing that they would be interested in exchanging with each other. And then frankly, I was getting a little bored, [LAUGHTER] the format, not so much the content or their insights, but the format just became very boring. We all know that when you’re working with one individual paper, it can be really exciting to have that dialogue with the student. But when you have a stack of papers, it suddenly becomes too daunting. And so I was just looking for a way to make us all happier, actually.

Rebecca: That sounds like a good motivation. [LAUGHTER] Can you talk a little bit about what the infographic assignment is?

Michelle: So I will talk about in terms of that intro level comparative politics class. One of the things that there’s not time to do in any semester is to really cover every country in the world, obviously. First of all, I wanted to make sure that students could pick their own country that they have an interest in, and that would broaden the number of countries we would cover. And then I switched it to infographic for the online component because there were a few things I wanted to do with it. One is to retain the goals I had in mind for them learning about a country. But also then I wanted to tie in some best practices for online presence in the classroom, for example. My discussions, I was also unsatisfied at the beginning, we learn as we go along, and my discussions tended to also be maybe a little stale, or I’d throw out a question and I would get very similar answers from everybody. They were going back to the textbook and giving me something rather than being creative or energetic in their own answers. And so the way the infographic works is I have about 10 through the semester that they have to do. And so it’s basically breaking up the country study into smaller pieces, so that they have different topics as we go along the different topics in the class. And then before they submit them to me, they submit them into the discussion for the appropriate week. And that way, all the other students can see them, it gives them a lot of different skills. One is that they are able to critique other people’s work, they are gaining more knowledge about a variety of countries. It allows them to practice skills on their own in terms of working together. So I don’t see these always as only individual assignments, although that’s how they’re graded, but kind of a group work assignment that they can all enhance each other. And so there were a lot of ways in which I wanted to bring different elements into the same assignment, particularly because I didn’t want to have to think of how to bring these elements in and do different assignments for each thing I wanted to bring into the classroom. Having one assignment that was consistent really helped them.

Rebecca: It sounds like by having a consistent assignment, you do have to learn how to do the assignment over and over again, that part of it is covered. You do it one time and then it’s clear how to do the assignment and then you can focus on the content and the material.

Michelle: Yeah, exactly. And one of the other things is, unlike just having one country study at the end, there was a lot more formative assessment that could go into it. So exactly that, where we’re able to help build skills up early on. Some of the content early on is pretty simple. The first one is making an infographic about the demographics of the country. And so it’s pretty simple, but it allows them to get the first skills if they’ve never used, for example, Canva, or other programs that allow them to make an infographic. So they get their feet wet with that. It allows them to think about how to communicate in a different way other than a paper, because I also began to think about my intro level students who many would become political science majors, but many wouldn’t. And so in a course that they were using for general education, they’re not going to be writing political science research papers. And so allowing them in different ways to express themselves I felt was another skill that they could bring.

Rebecca: It was nice to see this scaffolding, separating out some of the technical skills from the content, because often we’re doing all of those things at once. And it sometimes can be unclear whether or not it’s a technical skill that is lacking or the lack of understanding of something. So separating those out of it really does seem like it would help have a clearer sense of where students are. And, as we know, [LAUGHTER] if you try to ask students to do too many things in one bigger assignment, then certain things end up edited out. And that might be the thing that was really critical. So it’s nice to have those separate assignments to focus in on those.

Michelle: Yeah, exactly. And I really enjoy the fact that, rather than focusing on grading a long paper, I’m able to go through that process with them. And then also in discussions, it gives me so many opportunities to re-teach something. So if I’m noticing in someone’s infographic that they’ve made an error, or they have a misunderstanding of a concept we’re working with, then all the students in the class can see me correcting that and doing it in a gentle way, that seems a little bit less high stakes for students that because it is just this one assignment, and I’m making this one little correction here, but I think it benefits everybody.

John: So for those students who haven’t done much work with infographics or with graphics in general, do you provide any instructions or recommendations to them? You mentioned Canva, do most students use that? Or is it a mix of tools?

Michelle: They usually do end up using Canva, because that is one of the tools that I put in the directions. And so I have quite lengthy directions in terms of guiding them through accessing technology, allowing them to know that they can use various different technologies, depending on their accessibility to them. There are some really great programs that of course, cost money or pro versions. But I let them know that for our purposes, they can certainly just download a free version of a program. They could really even do it in the Microsoft Office Suite in a variety of ways if that’s where their skills lie. And of course, that’s something that all the students have access to on campus. And so I do get them lengthy directions, they’re not all my directions. I don’t feel like I need to reinvent the wheel. So I just include several videos from YouTube that talk about how to use Canva. And those are sufficient and actually probably better than I can do. And so I put those out there with other directions, I give them alternatives. And then of course, if they’re still stuck, then they can always work through it with me. We can have a Zoom meeting where they pull up their materials, and we work on putting it together.

Rebecca: You mentioned that students were sharing infographics with each other, it sounded like maybe through a learning management system or something. Can you talk a little bit more about what that sharing looks like and the kind of feedback that students are giving one another.

Michelle: So the learning management system we use here is D2L Brightspace, and there is of course, like in most learning management systems, there’s that built in discussion feature, so I just use what’s already there. I don’t mind using third-party applications., but it just tends to make things trickier. So I just use the discussion. Have them post it right to the discussion, and have them use the discussion tools, whether it be just typing in or using some audio or other tools to give feedback to each other. And I find that the discussions are so much more lively. Even more than I expected. The first time I used it, it was for a summer class and I thought “okay, I’m gonna try this now because it’s summer, and no one wants to be sitting writing a 20-page paper in summer. What am I going to do?” So I put it out there for the summer class and I thought, Okay, well, maybe they’re busy with other things or they’ve got excitement in their own life over the summer or they’re working a lot of hours. I had never had such lengthy discussions go on in any class before that. They really enjoyed looking at each other’s work rather than just from me. Many of the students chose countries that reflect an interest they have, maybe it’s somewhere that they’re actually from, although I usually try to encourage them to broaden a little bit. But that also brings such a benefit to the rest of the students. So we do have students who do that. I have students who maybe have gone on vacation somewhere or have a family connection somewhere, I’ve had some that have been stationed in the military, and come back, and they want to talk about what they noticed when they were there, so they put a lot of that in the infographics. I found just use the basic tool, but allow it to be opened up for them. And I also try in discussion, and I do this even in other discussions that I have online, because I usually try to hang back for the first bit, quite a bit. I allow that to be organic. And then I’ll go in and make sometimes just a summary of everything that’s been said or things that I need to correct. So that it’s not simply another lesson from me. But I have that ability to correct when I need to.

John: You mentioned the first infographic assignment had them work with demographic data and display that, to what extent are these assignments tied to the specific course work that you’re doing during each of the 10 assignments?

Michelle: For most of those 10, they’re directly tied to the material for the week. So for example, the textbook will have a chapter where we’re talking about government structures, presidential systems, parliamentary systems, et cetera. And so the infographic for that week will be directly related. So they’re going to do an infographic laying out the government structure. And then another module will be on political parties and how they work in various parts of the world. And so it’ll be directly tied to: “So show me your country, and show me the main political parties in your country. Tell me what their main ideology is. Tell me who’s in power now… which party? And what does that mean for the policies that get put forward?” Another one would be one on current events, the last one that they do is: “Well, what’s going on there now? …and “With everything you know about your country now, what do you predict for the future for them? How do you think they’re gonna solve this?” So a little bit more policy oriented… also, “look at what they’re actually doing in terms of legislation and implementation.” And I also do one… I don’t use the textbook anymore, but it’s something I used years ago that I retained the content in a module, which is to look at the concerns of any country in terms of every country’s concern with prosperity, security, and stability. And so I asked them to just put out an infographic that asks them to assess threats to those things, and looking at their country. And so it’s a little bit on the current events side, but it’s focused for them in terms of every country needs to be aware of its security… what’s happening?

Rebecca: You’ve hinted that students are really engaged in these assignments because of discussion forums. But can you elaborate on just the general sense of what students have been able to accomplish by shifting to this format, and also their engagement with one another?

Michelle: I think, several things, some of which I’ve mentioned. So the formative assessment part, I think they gain a lot from not just getting comments at the end of the semester, oftentimes handed in that last week. And then, frankly, I don’t expect that every student has even read my comments at that point. So just having the opportunity to fully engage throughout the entire semester, and so we’re not also dealing with, “Oh, I have three days before this paper is due. So let me think about it today, make a plan to think about it more tomorrow, [LAUGHTER] and then maybe I’ll do it the next day before it’s actually due.” And not that that’s all students, but I know when I was an undergrad that happened to me. So I think that just slowing them down, allowing them space to actually think about their country in multiple ways, and not have it be “Okay, we’re all working to doing your own country study at the end of the semester,” but having them actually engaged the whole time, I think is of great benefit to them. And I think they learn a lot more. I think that taking it in these smaller chunks, actually has made them research a little bit deeper on each of the topics because they’re not overwhelmed with “Okay, I’ve got all of these topics now to cover in this country study.” And so they tend to just do it in a way that is most efficient for them, let’s say. And so I think this gives them that way to back up a little bit and actually enjoy the process of learning rather than the process of an assignment only.

John: I think you’ve made a really good point there about students’ tendency to procrastinate. I was thinking back to when I was an undergraduate, and actually a graduate student as well. And I can’t recall a single time when I didn’t start writing the paper the day before it was due. I spent a lot of time gathering materials, taking notes, and organizing and putting an outline together, but the writing generally took place in just one day. So, [LAUGHTER] this is forcing students to engage in a little bit more spaced practice, to engage with the material regularly, and I think there’s a lot of benefits over that rather than the traditional way of just rushing through and getting it done right before it’s due.

Michelle: Yeah, exactly. And I’ve adopted that in other classes too, the idea of doing a lot of prewriting throughout the semester. So I’m a big believer in the prewriting process, and it’s very valuable, and it is doing work. But when I mentioned earlier, sort of they spend the day thinking about it, I’m thinking more of myself as an undergraduate thinking about not the assignment, but thinking about the fact I have to do an assignment.[LAUGHTER] So it’s more anxiety, not so much productive work on the content of what I’m doing. And so yeah, the prewriting work for any assignment is really great. So I’ve been trying to shift that mindset. I also teach a research methods course in my department and I teach our senior seminar. And so the senior seminar, especially, just making it about an entire semester-long process, I think, has taken a lot of pressure off of them. They seem a little relieved by the end, it’s not as daunting,

Rebecca: It definitely seems like it deepens the study and the engagement, not just because it’s broken into smaller pieces, but because it’s a consistent country the whole time and because it’s a consistent kind of assignment the whole time, it really provides a structure to be able to do that. I’m curious whether or not any of your other colleagues have also shifted to doing assignments in this format, or other similar kinds of strategies.

Michelle: I can’t really say. I know that in my department, we’ve talked about them a lot. And so I think people are adapting parts of them. I don’t think anybody does it the full way that I’ve described in terms of the full semester, but we have talked about just different ways of communicating. I think that’s going to shift again, obviously, AI is something that everybody is grappling with how to best use. So I think there’ll be some shifts coming. But I can tell you that I was Interim Director of the Center for Excellence in Teaching and Learning, and now I’m a faculty fellow there. And I’ve talked to a lot of people across campus, and some of our off-campus partnerships about this assignment, because I get really excited about it. If you haven’t noticed, this is my favorite thing. And so I’ve talked to a lot of people, and I just can’t be sure how many have actually applied some of these things. But they certainly seem interested, and they’re certainly searching out the same types of things I was searching out: how to make things, first of all, more authentic for the students, how to engage the students more with themselves in an asynchronous class or with other students. Even in a synchronous class or on-the-ground class, you could still have the same assignments. You might want to introduce them differently, you might want the discussions to take place in the classroom, rather than obviously online, because you could engage in person that way. So you can do kind of a round robin around the room, maybe… any of those techniques. But I also let the students know that, for them, they will become a mini expert by the end on that particular country. So they should all understand the topic areas but. becoming an expert on the individual countries that’s all their own. And so I hope that colleagues, no matter what field they’re in, can find some way to utilize that. Otherwise, they’re probably just a little sick of hearing me talk about it. [LAUGHTER]

John: Speaking of fellow faculty, has anyone else shifted to using infographics at your institution as a result of your work with us?

Michelle: Yeah, I can’t be quite sure. But I do think that in talking to a lot of faculty that they are shifting modes of communication for students, and I have talked to people who are much more expert at creating infographics themselves. You can do it very, basically, but you can also get really creative and high level with it. And so I oftentimes collect infographics I see for other fields, so I share those with faculty. I’m not sure. It’s one of those things that I so appreciate a podcast like this because sometimes teaching can be a lonely endeavor. We’re a little bit maybe shy about sharing what we do, because we’re afraid it’s not measuring up necessarily to what others are doing. So it’s a good question, though, and I think that that’s something that maybe I want to engage… maybe in a workshop in the future on campus and bringing faculty in and seeing who’s doing this type of thing, or maybe another brand new idea that’s even better.

Rebecca: I think one question that often comes up from faculty when we’re talking about kind of alternative formats or unessays or other [LAUGHTER] ways of not necessarily doing a more traditional essay kind of assignment, is how you evaluate the work. So I wonder if you can talk a little bit about how you’re evaluating these assignments, just to give faculty a sense of what that might look like and what that feels like in comparison to a more traditional kind of assignment?

Michelle: Yes, that’s a great question. I’ve talked in many ways about why I did these assignments in terms of just making it a more engaging process. But in terms of measurable, assessable goals, I do use rubrics, and have laid out exactly what skills I’m assessing. So I do have a section of the rubric that is based on the skills of communication. I have a very low stakes, low number line for things just like visual engagement, because I don’t want it to be students worried about the visual engagement when they need to learn the political science concepts. but I want to let them know that that’s important. So I have in the rubric descriptions in each of the columns about visually engaging: Does it bring the reader in? Is it easy to read? So I have that skill section, I have a section specifically on the content itself: Do they understand the concepts? Are they using the language that we use in political science? And then I also have sections for pretty typical traditional looking at their writing and seeing if it’s appropriate writing levels, tell them take a look at other people’s infographics… go out in the world and look at the bottom, there will be the sources, you need to include those, you need to not leave those off. So even practice that they would get in papers about citing things, if they put a graph in, you still have to say where this graph came from. So the practice that they would normally get are the same skills. And so I just lay it out in a rubric for them, as I do with most of my assignments. Of course, the rubric is not secret to the students, that goes out with the very first assignment: this is what you should be looking for. I’m going to use it for grading, you can simply use it as a checklist. You can use it to evaluate other students and give them feedback and tell them where you think there could be improvement. And so I think that the assessment itself is pretty typical to what I would normally do when thinking about an assignment: What are my goals for the course? How does this assignment help to meet those goals? And then how do I convey to the students whether or not they’re meeting those goals? Or essentially where they’re at in the process of becoming expert in that goal.

John: Do you use one rubric for all the assignments? Or are there separate ones for each of the infographic assignments?

Michelle: So I’m a big believer in constructing the outline once and then using it, but having different content in it. And so I use the same basic rubric for each of the assignments. And then I give the feedback in that content area really is the main portion. In the content area, I will make some changes to what they need to do specific for that assignment. So if it’s political parties, I would have expectations about what types of things they’d understand about political parties. And so that will change but my basic outline of what that looks like. And I’m a big believer in that kind of consistency in everything. So I can tell you I do that also with just how I set up the modules. And so my modules always have the same outline. There’s an intro with the learning outcomes for that specific module: what can you expect to learn this week, then the section with textbook readings. So this is what you’re going to dig into and read in a very traditional way. Then there’s a whole separate content section, which might be a video I’ve created, it might be videos that I found online that they should be engaging in. And then I usually have a pre-test and post-test quiz, and then the assignment for the week. And in this case, always associated with that assignment discussion. And then always at the end, I have a section, another discussion, which is I just call “Ask the professor,” And it’s another space for it. I always have to tell them every time I put the same heading and I say, “Don’t ask personal questions here, [LAUGHTER] but if you have a question that everyone can benefit from on the topic this week.” So that’s just an example of how I feel that it just makes it easier for me to focus on what’s in each of those boxes, rather than recreating the format. And I think students really take a lot of comfort in knowing after a couple of week: this is the rhythm, this is what I’m expected to do. And so now they’re focused on different content, but they don’t have to relearn a different set of skills. So going back to your question about the rubric, and is it the same? I do that because then they get used to what skills they’re bringing, and then they can hone that same skill over time, they can go back and look at their rubric grading and see their improvement over time. And I’m also a liberal user of the comment section on rubrics. I know that a lot of people tout rubrics for the time-saving part of it. So oh, you can check these boxes, and it makes it easier. And there’s some of that. But I can’t stop myself from also than opening up that box and just writing a ton of things to them. So I think that, again, it’s one of those things where the rubric itself becomes a really good outline. And then it’s just a matter of what I choose to put into my grading when I do the rubric.

Rebecca: High structure is always helpful for folks, that’s for sure.

Michelle: That’s very reassuring.

Rebecca: I feel like there’s a story behind why infographics as opposed to something else. What led you to infographics as opposed to some other alternative format?

Michelle: So like I said, I’ve been at this 27 years, so maybe I’m just getting old, but I feel myself saying in my head: “Well, kids nowadays don’t have the attention span for writing a 20-page paper.” And that’s not necessarily true. But as I talked about before, maybe they just have to be taken through that process. It can’t be just one big thing. And that was long before the internet, people talking about decreased attention spans. But I do think there’s something to matching up with the way in which they use visuals in their daily life. So they utilize memes, they look at infographics in various formats, I find a lot of them in social media in different areas. So I try to structure discussions in a way that is very much like the way they’re already engaging on social media. So yeah, the story is really how could I find a way that it’s something that their mind is already used to, that they can then apply it in this way. And also the other story, my husband is the County Administrator here in our county. And so I was in his office one day, I was the Assistant County Administrator and I was in his office one day and from the League of Minnesota cities, they had an infographic, a stack of them on the front table about: what does county government do? And that was really the last push I needed because it was so well done. And I thought, “Okay, this is what we need, people need to understand what levels of government do.” People often come to the county with situations that have nothing to do with what counties are actually able to do for them, and so what does your county do? What can it do for you? And I just thought, “How brilliant is that? To put it in that way.” And so I just really was sold on on the idea that infographics can really be fairly deep teaching, if they’re done well. We get so used to the academy and everything has to be these long papers, but rather than bemoaning the fact that the attention span has gone away, I’m just finding a way to adapt to that. So how do students engage with each other? And let’s make that happen.

John: And I would imagine students are more likely in the future to be creating infographics than they are to be writing 20-page papers unless they’re going on to graduate study.

Michelle: Yes, absolutely. That was the other part of it is that this wasn’t really authentic… the 20 page paper… particularly in that intro, liberal education, general education course. That was not authentic to what they were going to be doing. And I felt that in political science, if they were going to be political science majors, they would get that deep research component in other courses and upper-level courses in their capstone. And so I felt that it wasn’t necessary to put them through a 20-page paper. And the reason I keep saying 20 pages is because I realized that the amount of content you need in the country study is pretty lengthy to really understand the country, and so it really was about 20 pages, I just felt that that wasn’t engaging to people, no one was engaging each other with a 20-page paper. And so it had to be something that you could look at more quickly, you have to have the skill of condensing the material, you really had to be really pithy with how you were presenting the material so that someone can get it in a very quick way. And so I feel that allowed them to engage. And then even if you assigned a student to engage with another student’s 20-page paper, pair them off, have them do peer evaluation, they weren’t going to do more than one at that length. Maybe if you cut it down to a 10-page paper, you could have them have a group where they exchange it among three or four people, but that’s still not as big as I’m able to get by having everybody in a class post their infographics so that you can all see huge variety of work in the span of scrolling through normal social media. And of course, I’m asking to do a little bit more, but in some ways, it’s almost that they don’t even notice that part of it because they’re just scrolling through and saying: “oh…” Comments that I get most on the first assignment have to do with “Oh, I’m so glad you chose to do that country, I’ve always wanted to go there. Have you ever been? …because it looks so interesting.” So it doesn’t start out as this deep political discussion. But because of those comments in the first week, I’m convinced of the engagement that starts to come from it. One part that I didn’t mention is I also put in the directions, that if they want to add something, like a fun fact, about the country on any of these infographics, they should, and I’m surprised at how many students then choose to do that on every single one of the infographics at the bottom. There’s a fun fact. And I’ve been amazed by what they’ve come up with. There are things that I don’t know about these countries. I thought, “this is fun.” So I do think that students take to it, they want to show other students their country, they want to say, “Hey, I learned this thing. Isn’t this cool?” And then I get to learn things I didn’t know.

Rebecca: Sounds like a great way to build community and to have a first interaction with a discipline, which sometimes can be off putting. And this sounds like a really positive way to get people interested in a discipline as well.

Michelle: Yeah, I think so, and I hope so. I hope that that does make students more likely to maybe take an upper level, look at us as a minor or major. And I will say, talking so positively about this assignment, because it’s worked so well. But I can also talk to you in a podcast about all the assignments that haven’t worked well, that I don’t get this excited about, that I can’t say were this level of success. And so when we were talking about coming on the podcast, I wanted to highlight this one, because of course, it’s the success. I wanted to show what has actually worked. But, of course, there’s a lot of ones that haven’t. So I do feel that I’ve hit on something here that can be very useful, that I’ve been really satisfied with, and my students seem to be really satisfied with.

John: We always end with a question. What’s next?

Michelle: So that is a great question. I’ve been thinking about that for a while. How do I take this to the next level? So I think one thing that’s next is sharing this out more. We’ve talked a few times about looking at other faculty and seeing if they would adopt something similar to this. So I think that’s one aspect. I think another aspect is to just continue to refine the assignment in a way that gets the most student engagement. I’d also like to see these assignments, if we can have them maybe printed for my asynchronous online classes. This particular class I teach mainly online now, I rarely teach it in the classroom another colleague has those sections. And so, right now, it’s simply archived either as the students own individual file, or in D2 L. And so I’d like to think about ways in which maybe we can archive them, maybe print out them in poster form and have them in the department so that people can maybe again, engage, get interested, a student walking by who’s not in political science who’s hanging out in the hallway waiting for their next class to begin, and starts reading the stuff on the walls and start saying, “Oh, that’s kind of interesting,” and maybe starts to get engaged. So looking at it outside of the particular classroom, I think would be a really exciting way to go with it.

John: Well, thank you. I’ve really enjoyed hearing more about these assignments. And I think it’s something that could work in many disciplines.

Michelle: Yeah, I hope so. And exciting for me would be now to sit back and kind of watch what other people do with it. And oh, okay. I hadn’t thought of that. That’s really cool. That’s always the most exciting part for me from the CETL angle is really just seeing how people take off with something in a way that I wouldn’t expect. And it’s been a pleasure talking with both of you.

Rebecca: I always love a good conversation about interesting assignments. So thanks for chatting with us and sharing in depth how the assignments work over the course of the semester.

Michelle: You’re welcome. My pleasure.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

341. Learning Losses

The transition to remote instruction during the COVID19 pandemic resulted in dramatic learning losses. In this episode, Peace Bransberger joins us to discuss a report that analyzes the extent and persistence of these learning losses. She is the Interim Director, Programs and Evidence, Policy Analysis and Research, and Programs and Services at WICHE, the Western Interstate Commission for Higher Education.

Show Notes

Transcript

John: The transition to remote instruction during the COVID19 pandemic resulted in dramatic learning losses. In this episode, we discuss a report that analyzes the extent and persistence of these learning losses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Peace Bransberger. She is the Interim Director, Programs and Evidence, Policy Analysis and Research, and Programs and Services at WICHE, the Western Interstate Commission for Higher Education. Welcome, Peace.

Peace: Thank you.

Rebecca: Today’s teas are:… Peace, are you drinking tea today?

Peace: I have drank tea and I ended my tea drinking portion of my day with a Trader Joe’s maple espresso tea.

Rebecca: That sounds interesting.

John: I’ve had some of their teas, but I’ve never had that one.

Rebecca: It sounds energizing.

Peace: Yeah, it’s a black tea, has a bit of a smokish coffee kind of impact from the espresso.

Rebecca: Interesting. That’s a new one. I don’t think we’ve had that one yet.

John: No, we haven’t. [LAUGHTER] Well, I have an old one here, a spring cherry green tea in the hopes that we will see spring here. We had a lot of snow in the past week. We’re recording this, we should note, in late March.

Rebecca: I have a Lady Grey today.

John: The Western Interstate Commission for Higher Education is focused primarily on post-secondary education, yet you issued a report which is titled “Navigating Learning Loss and Changing Demographics in Education” in February of this year. Could you tell us a little bit about why you had this focus on learning losses in elementary and secondary education?

Peace: Well, so my organization goes by the acronym, WICHE, you had it correct. We’re a Higher Education Commission. And one of the things we’re known for on the research side of things, because we do a whole bunch of other things to support the 15 Western states and then outlying Pacific territories, higher education systems, and one of the types of research that we’re known for our projections of the number of high school graduates. It goes under the long-standing title of “Knocking at the College Door,” and the projection of the number of high school graduates so that state and institutional planners can strategize around overall student flow trends. We last issued the update to that 40 year ongoing research in 2020, when COVID was first raging, and we issue these projections roughly every four to five years. So we’re in a prep year right now. As part of my job, and being the lead on the network, I am always monitoring K-12 trends data. So we’ll have a heads up about whether something would be monumentally different in the space of high school graduation. And as a result of what I was seeing, we decided in February to issue the report, kind of a summary brief, because the data were compounding and resoundingly indicating that yes, something is currently different, and might continue to be different about high schoolers, and younger K-12 populations as a result of COVID. We’ll go into detail on the learning loss side of things, but from the perspective that we really typically focus on,the numbers of high school graduates, the major demographic trends, the reason that we felt we needed get out in front of what we were seeing is because the learning loss, we see from that the potential that enough children and teens may have been so impacted by the learning disruptions and things that a quantitative impact on the actual high school graduation trend is possible. And that’s important to know, because it’s a second layer on something that I think we all kind of have on our minds, some people will call it demographic cliff, we don’t use that term, even though our data are used to depict the slope, the trend line, it’s the coming downturn in the number of high school graduates, because back in 2008, fewer babies started being born. And to this day, there has not been an uptick in the rate of fertility in the United States and across virtually all of the states. We know this, in fact, it could be next year’s high school graduating class that might begin to evidence some of that trend. And so people are front and center thinking about that demographic change, I would suggest, [LAUGHTER] at a minimum is a contraction in the youth population. I don’t know if what that means necessarily for higher education contraction, but we can talk about is learning loss going to do anything to help that? Are we gonna somehow see actually more high school graduates than we might have otherwise expected given C OVID? Or is it more like what I think we’ve probably all been waiting and worrying about, could it impact that trend and deepen it, amplify the downturn in high school graduates?

Rebecca: So you talked a little bit about the data suggesting losses in persistence of learning? Can you talk a little bit about what those losses look like?

Peace: Sure. And, just for our purposes here, I’ll speak to the national kind of overall results and trends. But I would strongly encourage your listeners, even just as a starting point, to go to the web page that I’m sure we’ll put in the show notes where the report resides. And within that page, I’ve been adding to the list where people can access more detailed data than the national trends. I’d go there because then you can poke around, based on your own institution, and get a sense of the kind of school districts that you might know, school districts that are kind of geographic areas that are really strongly important to your student populations. Because the detail is really important. It’s a really multifaceted, nuanced topic. So about the persistence of the learning losses, and this is in the K-12 pipeline. I mean, technically, we work in grades one through 12 data because kindergarten is not a universal requirement, so it can be hard to know what’s going on there, since the trend data could vary year to year. When I mentioned prior to COVID, or pre-COVID, in the K-12 school system, that would typically mean either the 2019-20 school year, or some folks go back as far as 2018-19, because the 2019 school year included the spring that was disrupted, but by then most learning and assessments had already occurred, and then the quote unquote, post-pandemic assessments that we have availability to summarize go through the spring of 2023. So that’s almost a year ago at this point. But by spring of 2023 students started, after a couple of years that we’ll discuss here about what really happened there, students started showing some resumption in the rate of annual learning and acquisition that was typical pre-COVID. So that’s just like, on average, in a given year, the assessments generally will say how close to on-grade-level and then was the amount of typical acquisition achieved. By spring of 2023, the good news was that annual rates, there was some evidence that students were a little bit more closer to back on track. The unfortunate problem, and otherwise not good news, is that students were definitely not on pace during the two previous years. So they lost total learning, and it sort of accumulated. Students would have needed to learn at really unprecedented rates in 2022- 2023, that year, where they are resuming sort of a typical rate, just to make up for two years of lost learning, if that’s even such a thing in learning, which is an accumulative kind of process. There are spring of 2023 results, there are some more recent results from several of the major assessment products for the fall of 2023, so getting into the current school year, beginning of the current school year, and they generally confirm what spring 2023 results showed, that students came into the year with the overall lost learning. At this point, it means that these K-12 students… and we’ll give some statistics by different grade levels and what have you, the nuance, but pretty much from the assessment results, you see the same trend virtually at every grade level, which is that students have been moving along, learning in the given grade that they’re at, but they’re still being bogged down by overall unremedied learning losses. And that’s for four years now, so that’s pretty substantial. One sort of point of reference for those students who were high school freshmen, as COVID was raging, their time is up. So four years, did they get back on pace? Were they able to stay on pace, such as I’ve just described? They’re gonna graduate now, and many of them will graduate, but not having been able to fully recoup what was lost. And so there’s not a lot of data that actually sort of compare, “Okay, so I’ve received my diploma, it was awarded to me. What amount am I behind?” And I think that’s part of the problem. In fact, there is increasing attention to the notion that grade inflation, that has always been something of investigation, but evidence that it may have really been at play during these past four years in a way that is really masking and complicating this issue, not making it clear whether students who are graduating from high school, would they be considered on par with previous cohorts that were emerging even prior to COVID?

John: So basically what’s happening is students seem to be learning at the same pace they were pre-COVID, but they’ve all been left behind fairly substantially as a result of that transition to remote learning. And they haven’t caught up to where they would have been had we not gone to that experience.

Peace: Yeah.

John: Were those losses roughly the same everywhere, or were they particularly bad in lower income communities?

Peace: Well, let me give you first even just a sense of the scale of losses, and some of this, you almost have to see it to comprehend it, but I think I can kind of show the scale. So this is high school, stick with high school students, and so they’re the ones most immediate to our faculty might see. It was reported in October, the high school students’ scores on the ACT College Admissions Test had dropped to their lowest point in more than three decades. And that was describing therefore the class of 2023, some of which are presumably in college right now. And they were in their first year of high school when COVID hit. The average ACT composite score for U.S. students was 19.5 out of 36, for that class of 2023, which was down from the prior year 19.8, which we’re talking already that those were some score levels that might not have been ideal to begin with. For as regards to SAT, total score declined for the class of 2023 as well down to 1028 compared to 1050, for the class of 2022 and compared to 1060 for the class of 2021. So really, those are just a couple of data points about this notion that schools are graduating students but what that means when they’ve graduated could appear very different by the time they arrive to college. Now, it’s hard. Sometimes people might poke holes on those type of data because ACT and SAT are not taken by every single student. So the other thing I’ll point out is that we’re focusing on COVID impacts here. But it’s important to point out that in those two assessment instruments, ACT and SAT, scores had been falling prior to the pandemic. And so the pandemic just accelerated those declines, accelerated and amplified them. Since we work in the demographic space, on demographics research, I’ve been talking to other researchers about possible reasons for that pre-COVID decline trend, and just, frankly, how hard it is to reconcile with graduation rates that had continued to increase over the same years, and then the grade inflation, so we don’t have conclusive answers on that. But it’s worth noting, we’re talking COVID. The question might be for faculty and instructors is, do things feel particularly difficult with incoming freshmen at this point, in some way, shape, or form, and probably in very nuanced ways, depending on discipline. But then it’s also like, are they feeling pretty good, prior to COVID, were they where you want them to be anyway? So assessment results from elementary and middle school grades, the one that provides some of the most normed results, and therefore can be compared over time and as a universal sort of indicator, is the National Assessment of Educational Progress, or NAEP for short. They’re the thing that really caught my attention. We tried to highlight where the score drops for 13 year olds, they’re no more or less important than any other grade level, or school age. But those are the next in line coming through high school. So we’ve just seen what was going on with the high school age students during that period. Well, the NAEP 50th percentile mathematics score in 2022, was 282. In 2023, it was 274. That was at the 50th percentile, so an eight point decrease at the middle, the decrease on the already relatively higher scores at a 90th percentile was six point decrease. So even the highest percentile, 90th percentile, had a pretty dramatic decrease between those years. And at the 25th percentile, it was a 12-point score drop between 20 and 2023. NAEP reading score declines ranged between three point drops at the 90th percentile to six point drops at the 25th percentile between those years 2020 to 2023. And so there, it gets to the topic that I’ll provide a few more thoughts on about the variability. Were all students affected by this? Which ones more or less so? Right there, that was a statement of a twice as large score drop for those at the 25th percentile, compared to those at the 90th percentile. And this is already on the 90th percentile, they already being, perhaps at or above a proficiency level compared to a 25th percentile, which might not actually be at a proficiency level. And on top of that, you have a steep drop. It’s also worth noting, just like with the SATs, and the ACT scores, that the NAEP scores in that 2023 school year, were just an amplified continuation, some declines that were already emerging pre-COVID, such that in the composite reading score it had by 2023, it was the total average, so at the 50th percentile, seven points lower in 2023 than a decade prior. So over the course of a decade, it was already reducing, and then 14 points in mathematics. So some of these things that we’re talking about is, I don’t know if you want to say learning loss or just what the best word to say it is, because it’s nuanced, but they were approaching prior to the COVID period. And so I really want to highlight that because kind of just in am analogy, prior to the pandemic, schools and students are already losing historical ground. So they were already maybe not in the best shape, so to speak. And now they have to also recover from COVID. And so that can definitely explain some of the lack of recovery and a slow recovery.

John: Going back to the issue of SATs and ACT scores, you mentioned some complications. I’m guessing the major complication is that fewer people are taking the SATs and ACT since at many colleges scores became optional, and the people who are most likely to avoid taking it are those who might on average, expect to do less well on the SATs or ACTs, so that may suggest that the losses would be even greater if we had the same proportion of students taking the SATs and ACT tests as pre-COVID when a larger proportion of schools required them for admissions.

Peace: Yes, that is true. However, the 2023 SAT scores, the participation was the highest ever. So maybe the 2022 that could have been a somewhat more appropriate consideration. It remains the case. We don’t know the underlying distribution of the students necessarily, and if they change year over year in consequential sorts of ways. But I think you’re right. I’m really looking for some good news on this topic. So I can be something other than a Debbie Downer. But the truth is, I don’t think we can look at this and pick away at the data. We went out because it was like if you download the PDF, and you’re not convinced just like access the 30 different year points or different reference tests, and what have you, and see if you’re convinced that this has happened, it really has. It’s pretty affirmative at this point, we’d love to see it turn around, but I don’t think we can just ignore it.

Rebecca: I think it’s really helpful too to point out that it’s not just COVID, the fact that you’ve underscored that, and it might be exacerbated by COVID is really important, because there’s a lot of blaming of COVID on many experiences that we have in the classroom, that may or may not actually be the cause.

Peace: I’d argue they are the cause of what’s seen with the past four years, but we can’t just pretend that that because it was so consequential over those time periods that even a return to normal would be where we want to be.

Rebecca: Oh, of course.

Peace: I think that’s the emphasis, normal wasn’t a good normal. And maybe we didn’t all look at it. It wasn’t so stark, somehow, different people might have been emphasizing it from an equity perspective. I would emphasize it from an economic perspective, because what I won’t highlight here is that there are researchers like Brookings, some of the think tanks, and then in other cases, some more consulting sorts of research organizations that are kind of putting out there, this has implications for the economy. I mean, we can talk about the implications for higher ed, and that part of the economy, but I mean, I just don’t think we can ignore past this, there might have been something brewing, that this was just a perverse sort of way to get our attention on some of it about what the youth of today need? Where do they stand? Are they getting prepared well enough in a way that we need to support the workforce of tomorrow. That’s not the only reason for higher education, but as you and I decide to go take our retirement, these are the kids who will be supporting the country’s economy, and there will be fewer of them. So their ability to do that is really important. So I think, if nothing else, it could be a wake up call. We should really wake up, but we don’t have to wake up screaming and yelling in the house like the fire alarm going off, we need to figure out what to do.

Rebecca: One of the other issues that your report underscored was some high rates of absenteeism. And that’s certainly something I’ve heard my colleagues talk about as well. Can you talk a little bit about some of the potential causes for that?

Peace: Yeah, I will. I also do then want to make a note about something else, but I’ll respond to the absenteeism. So the statistic, 30% of students nationwide were chronically absent in the 2021-2022 school year. So that’s two years into the quote unquote, pandemic period, which is double the pre-COVID19 average. And while not comprehensive, the preponderance of data, were suggesting only minor improvement in the most recent completed school year, 2022 to 2023. That’s huge. Now, chronic absenteeism is different than like average daily attendance and what have you. But still it was a doubling. Absenteeism is the question. I mean, the question is, is it a symptom or is it a cause? And it’s a little bit of both. Prior to COVID students with more day-to-day life or learning challenges were on average, more likely to be absent from school, and a real reason to be absent from school, especially given the kind of hysteria that was almost necessary. As a parent, I got this during COVID. I’m like I was sending my kid to school if they met the temperature well enough for them to go. But being sick is a valid reason for absence. And that was made so much more evident with COVID. And some households and students are just simply more likely to be sick or not recovered as solidly. Now, of course, some of them might already be vulnerable students by virtue of a health condition, but you’ve got students and families that due to their living conditions, or health care access, might be more likely to be absent. One of the phenomenon, if you will, or factors, is social prejudices, and hostile environments for some students more than others. So the Asian and Pacific Islander communities experienced more of that during COVID and that may be lingering in their attendance decisions and the Black Lives movement put in the spotlight the types of stresses that black students might face at school. It can be rational to avoid school, and at this point, some fatigue with that might have set in. So, but even let’s talk about the marginalized students, and is the average student’s disposition to attend school affected by this point in some way, because if it is, then it’s amplified for marginalized students. So school and education that we as adults just talk about and encourage our children to, try to guide them through, pull them through, whatever. They might have become synonymous with very easily influenced young children and their emotional memory with online learning and masking and fear of sneezes and coughs, let alone than what it felt like go back after being socially isolated. And my kid experienced, even in the higher achieving classes, just an unusual rate of disruptions from students transitioning back socially and stuff like that. So these are children. And that can be a far more formative experience. I’m not a psychologist, I can’t say trauma, what have you. But that can be something that that’s all you know, for some of them or a big part of your recent memory. And that can be all you can think of as school, so to ask me to go tomorrow, I might be relying on that recent memory. So I can understand any student in K-12 to some extent, also. Hopefully, as you mature into an adult, you’re able to sort of equip yourself to move past those things. But some of these young adults really having a lot of emotional memory that makes this a real sticky issue, the absenteeism or the lack of kind of bringing their best to the educational setting. I mean, if you want more factors, if that’s not enough, school transportation issues for the past two years and kids literally not having a way to get to school. You’ve got teacher fatigue, and we know how important instructors are in the classroom and what you can bring to it, your ability to do that. And if it couldn’t get worse, because I lived through this with my kid, you still got an unrelenting possibility of like school violence and mass shootings. So there’s a lot of reasons that school does not feel like running through the corridor anything, you know, that maybe we all might have felt as a more positive thing. Now, things hopefully, the last, maybe a year or so where kids are able to start washing some of that out of their memory, and it being replaced with a more normal environment. And hopefully, that’s a good thing for them.

John: One of the things that shows up in the data is while there were learning losses across the board, I believe the learning losses were a bit worse in the area of math. And those seem to be having a pretty significant effect, or at least from what I’ve seen in the classroom, that’s been having quite a bit of an effect on our incoming student body and may have a significant effect on their choice of majors. And we know the rate of return to education in the STEM fields is dramatically higher than it is in other areas. And in terms of the state of the economy, that’s something that could have a very negative impact unless we provide some ways of helping students get caught up in some way. What can colleges and universities do to try to bridge that gap, to take students who, on average are coming in at lower levels, and get them up to the level that they need to be at to be successful?

Peace: As you and I must have been reading some of the most recent coverage on this topic, even just this week. So I can dive into your second question through the lens of STEM. So yes, I had seen some stuff about it more from a HBCU perspective, Jill Barshay of The Hechinger Report, she often does deep data dives. And so earlier this week, she released some interpretations highlighting how the NAEP, those national scores for the K-12 population, those and similar results could conceivably suggest a narrowing of the STEM pipeline for what you’ve mentioned. And so it feels like there are a couple of really interesting points on that… math, at the very least, although clearly, reading comprehension is also very important for sciences and anything, obviously, but those more technical reading types of disciplines. Reading, it cannot be forgotten. So what’s maybe most worth thinking about from what she highlighted was the top students, the NAEP results, you’ve got it 75th percentile 90th percentile, what have you. Top students are staying on grade level, according to those data. And those top students, of course, are maybe definitely the front of the line on the STEM pipeline. But the eighth grade NAEP shows that far fewer of even those who are in the top portion are hitting an advanced performance level or even proficient. So just because you’re in the top, it’s a grading curve, so to speak. Top might not mean what we need for STEM types of disciplines. So, there were learning losses across the board, math scores among the top performers dropped as steeply as did the scores among lower percentile scores. Okay, they call out even the scores of students at Catholic schools. So we understand that that might be a place where some of the STEM students could be concentrated. Otherwise, the scores from those schools would indicate that they weathered the pandemic pretty well. But scores in eighth grade math plummeted. And I just think that that’s another one of these big wake up calls, and it’s in the microenvironment of STEM, but the importance of it to the economy, obviously to the colleges and universities for which that’s a key focus. Now, what might you do even just on this topic, because then I think we can get into a little broader discussion about what to do. I would just ask STEM faculty, if this is the case, if these are the facts, it’s the real fact of the matter. Let’s just free think a little bit and see, to what extent could you meet students at a lower proficiency level if needed. So if it can be quantified, they’re 10% off what you think should be the criteria for even beginning the STEM disciplines, then what have you? If that’s what you have, if that’s going to be sort of the circumstance of even your top applicants, can you do something differently to allow them to actually be admitted? …get on track? What can you do? What would you need to do to actually maybe bridge that distance? If it’s larger, I mean, then obviously, you have to go a little bit, quote, unquote, deeper into the pool or the topic. But if you haven’t, for a while, really look at some of those criteria, sort of indicators and stuff. Because otherwise, we go through the admission cycle, and you might for an entire year of students miss some who could really, with the right approach, potentially continue on their aspiration to STEM. I’d also say, and I don’t really know how this plays out for faculty, and maybe it’s a pie in the sky kind of idea. But we’ve kind of been hearing it from some of our state folks who run the boards of regents and these other sorts of things, which is recognizing that this has occurred, recognizing that it’s always been questionable or difficult necessarily to know, if you could get a guaranteed STEM pipeline, to what extent colleges start reaching backward a little further into the actual high schools and be part of their exposure to and understanding that I might be like, let’s say the eighth or ninth grader, I actually have really, really loved the idea of becoming a scientist or going into medicine or something like that. But I’m starting to waver with my math skills or something. And so these kids aren’t really quite aware. And they’re probably already thinking college and that kind of thing, even at that young age. And if they get a sense that they wouldn’t be admitted, or I’m going to be a failure that early, then they just lose the aspiration, among other things that kind of erode aspiration sometimes in the STEM disciplines. We’ve been hearing from some of the stakeholders, maybe just a real need to actually start acknowledging out loud, whether we can meet students where they are, could we meet them where they are, because they’re the ones who lived this, they know how hard it was. And so their sensitivity to maybe that kind of like, work really hard, and then not actually be able to get into the program they desire and stuff like that. So much of it’s the right thing. But it really comes down to the emotional reaction and decision, as much as quantitatively, could they be close enough to be accommodated, which would be in our better interest if that’s even a possibility? We have to be looking at, I can’t say up front, revise anything, but we should really be, in light of what’s occurred in COVID impacts in K-12, I think we should really be looking at what are some of the hard and fast admission criteria and stuff where they exist. Now, it’s true, like less selective institutions may just end up dealing with the majority of students who have experienced learning impacts, as they always have, but given the fact that even among top performers, there was learning loss evidence, I think we can’t, in any institution, sit back and say, well, it won’t be our problem. So what is within the realm of possibility to meet students where they are, which might be somewhat off of what we would hope. But if it’s not drastically off, for example, then at least we’re taking one step in the direction of meeting them. Because I don’t know how controversial that is. [LAUGHTER] I’m a data person, so from a data perspective, it is one of the few things that I see as a real possibility as you try to make those data points overlap.

Rebecca: So we focused a bit on STEM, if we broaden that a little bit to what universities can do. We talked about admissions criteria, are there other things that we should be thinking about to help our pipelines for all of our fields and disciplines and thinking about the future.

Peace: Yes. What can we do? There’s a sort of like current and then future tense implied by that question. What I would say is, we should have already been doing something. And the reason I say that is because “Okay, so we hear that there might have been some observable, pre-COVID developments on how prepared students were based on those pre-COVID assessment score declines.” So I would ask the question, “Was this not something that folks already were having to, in the margins, sort of deal with? And did they start doing something?” Because if so, do more. Look at the possibility that there should be a sustained, broader perspective. The other thing is that, prior to COVID, the rates of college enrollment, for example, among some of the previously lesser served student populations, Hispanic students, and two or more races, the rates of enrollment were really going up. So were colleges already having to address something about what had already been a changed student population, but for some reasons other than COVID. So again, if you hadn’t been, I think it’s really compelling, you’re going to have to now, but we can kind of maybe step back and say, “Well, what did we do?” Sometimes we forget, we were just dealing with stuff in the moment. And so if we look back, just go back to prior to 2020, and then start thinking about, “Oh, what have we tweaked over there or something,” if that was happening, that would give evidence of what you might consider doing now. Because you might have already been making some adjustments, You might have been piloting some things, you might have only been piloting them with the expectation that it wasn’t something that needed to go full scale, see if they’re there, and approach it from that perspective. As regards to helping students with academic preparation needs, it’s a different situation, but it overlaps with the topic of other types of student supports, because student attention, even if they have the aptitude, might be missing some of the content knowledge. But if they have the aptitude, if their attention is distracted from not only the perennial sorts of things, that some students have to deal with, work demands or lesser educational advantages of many different perspectives. Now, they also might have that academic learning loss, that if we do put some of that supportive environment in place, then could you, STEM or otherwise, meet students at a slightly lower bar, and then still get them successfully where they need to be. We’ve been trying to do that for different student populations for a while, but some of the data would suggest just do more of the same, and until something comes, for example, K-12 data about what to expect with incoming freshmen, expect to sustain it is what I would say. I listened to some of the podcasts that you pointed me to from previous episodes. And I will say one that really resonated with me was the one about relationship-rich education. And then there were several other ones, each of which really kind of touched on various aspects. But I think that relationship-rich education episode, it was specific sorts of interventions, a lot of which, as it suggests, are not specific academic intervention. But they’re the things that we are creating learning spaces, and whether we mean them to be punitive or not, they can be sensed as punitive for students who just really had a difficult four or five years, that about they’re sensitive, so to speak. But to be intentional about learning environments, that don’t take a lot of specific kind of empirically vetted interventions even but intentionality about airing some of this, I would say, with students, like if it can be a discussion point, and you have students who are really feeling the spotlight is going to be on them because it felt like it was really hard to get through my senior year and now I’m going to give this a go. And I know it’s supposed to be challenging and stuff, but maybe they know what part of their senior year or math class was the most difficult for them, if they’re given the space to kind of articulate these sorts of things, it doesn’t have to be in an open discussion forum in a classroom, maybe not, depending on what you need to talk about. But just making it clear to students that they can actually identify some of those needs, that they will be required to be the most responsible for their own learning, but even though we’re going to keep emphasizing standards, and do everything, that we actually do mean to support them, and that we see their success in our best interest, and therefore we’re open to listening, to hearing, to believing what they say. So I would say believe it. On a spectrum, there will always be some people who are maybe struggling the most or otherwise, we would maybe take a general approach that anyone who’s saying some of this is complaining. I would say the evidence would suggest believe [LAUGHTER] that whether we thought it should have been easier for the kids, it impacted them. And so believe it and open up the possibility that if they say what they need, they might be able to identify something for us to do, it could be far simpler and smaller. Some of it can be more time consuming, but maybe if it just becomes common practice to imbue our classrooms with this sense. Students themselves can support each other, that we can hear what they say they need to support them, and it might not be half of the things that we’re worried to mention out loud because we can’t promise it. It’s nothing brilliant, I apologize. And I wouldn’t be the person to speak to in instructional sorts of research, but it definitely resonates with these academic impacts as a result of something that was a societal experience, that we need to be in the space, not just of academics, but of what part of learning methods actually are important for learners. And I’m an adult educator. I have been an adult educator. I wonder, because when I listened to some things that are more about like instructional approaches for equity and for other sorts of things for adult learners, and just what I know about the science of adult learners is I wonder if some extent, this current, quote unquote, generation… I don’t know if it’s an entire generation, but certainly, maybe 10 years worth of students… have actually had adultified in ways that maybe bring them into a space where some of the methodologies that we use in our learning environments, we might learn something from actually thinking about those adult learning and education methods. I think to have to grapple with some of the things that some students, that the entire spectrum of students, it wasn’t just pockets of students have had to deal with, maybe they’ve adultified in some ways, and then yet are not showing it because they don’t have the skills they need from school, I might advise if people kind of consult with what are some things that would be different in an adult kind of focused classroom compared to some of the classrooms that are more typically going to be populated by younger students. There might be some methods there that they can be common sense, but they’re not obvious.

John: One other topic that has come up recently is that colleges were a little bit more flexible during the pandemic in terms of dealing with things like administrative holds, and so forth. Many colleges are starting to put those back in. And one of the implications of that is that students may not be able to register for classes at the start of the term and they may be coming into classes after other students have already been in a class for a while. And I’ve seen that myself this semester, in a way I haven’t seen it since COVID. Is that something that colleges perhaps should be a little bit more careful with?

Peace: For reasons that probably don’t have to do with COVID or anything… but yes, as most things go with COVID, it just amplified possibly. I’d say yes. So on an entirely different sort of research project that I’ve recently worked with, WICHE, my organization, led a action research study of sorts with 12 public colleges and universities. And they each did their own comprehensive data analysis on this topic of every single possible hold a student could get. I mean, it would be a registration hold, like limiting their registration, or maybe access to records or something, but it could be for academic probation, and it could be for paperwork of any sort, it could be for financial aid administration, advising holds that restrict registration. They went comprehensive, they dug deep. And in the first post-COVID academic year, which was 2021-2022, by which I mean, mostly removed from the funding supplements, more than 265,000 holds were placed on roughly 125,000 students across those 12 institutions. And there is wide variation across institutions on per student rates, and stuff like that, and the reasons for them and the observed outcomes that they appear to be having on even just the next-term persistence for students. But by and large, every single institution, we’re able to, by looking at the data, find certain segments of students, part time, certain colleges and disciplines that really needed some attention for what’s a wide-scale, maybe not problem, but certainly a wide-scale sort of thing that each of the institutions would say they didn’t know how to control it. One University, for example, found that a single academic department and this, of course relates to our faculty listeners, place the bulk of all the advising holds, and a significantly higher rate per student than any other academic department or college. And it turned out that these advising holes were well intended, had developed over time as a way to manage the major requirements. But the college never got a bird’s eye view of the whole picture about how they were just sort of being administered. So when the research team at that university brought the data to the Dean, the Dean immediately recognized the holds were not operating as they were intended and took action to substantially revise how they were used. That’s advising holds, and that’s one of the most frequently used type of holds, even though financial holds get a lot of attention for the right reasons. Institutions from those 12 are also reconsidering the exact timing of registration holds within a semester in case that they’re just being too preemptive. Like, it’s easy to just be like “Mark all XYZ-term students not to be able to register until advising” and what have you. I mean, it’s cost effective, it’s kind of solution at scale, if you will. One institution experimented giving a small cohort of students, those who are at risk of academic probation, the ability to actually register at the usual time during the semester with all other students. And then they only denied the registration at very end of the semester for the very small portion actually, even from those students who were still failing. And that small experiment for that university indicated that there wasn’t a difference and no benefit from preemptively limiting registration for the whole swath of students. So why not consider changing that? …especially if unknown, some of those students might have otherwise been impeded. And I want to really mention that this is for everyone, because you, in your seat, you don’t know what other holds are being used around the institution. So any given student is not only receiving the results of the one that you’re using for the right purposes, I’m sure, but they’re receiving any number of these. And if you dial back and see the full scope, often, at least at first glance, you got like low-hanging fruit all over the place that you can reduce them. There was repeated evidence in this study that some registration holds actually did the opposite of what they’re supposed to do, or intended to do, which was to advise students so that they would take the courses that they needed. So then we heard from students, and then some of the evidence that institutions started investigating, that the mere fact of not allowing the students to register during the peak registration period when other students were competing for classes, led them to register late and take classes they don’t need. I mean, they were going to take credits anyway. And could that actual hold be the reason that they end up taking and paying for these courses that they don’t need? Now, we know that’s not what we intended, we intended that they’re taking the right courses, so there are just different way to arrive at that. And then finally, on this topic, the FAFSA… well, I’m going to have to use the word that I see in the press, it’s not my opinion… mess, that I think we need to be ready, at least in this coming fall term, that there will be students and they would be students who have financial aid, who may experience untold, even if they’ve made it through the process they choose to enroll, they could face untold additional ripple effects, one of which is that holds on records and registration are part of the financial aid administration process, so if that has been delayed, and God knows what kind of other sorts of messes have accumulated in that space, if summer is the period during which most of this should be resolved, but students may not be engaging, definitely not as be accessible during summer, I think we need to be ready. And this includes even faculty being aware of what might be happening, or something you don’t know about a given student is struggling with just something like a paperwork mess that’s distracting from their studies. It’s distracting from their attendance and what have you. The more aware we are that could be happening, and it’s told even just through the data with holds, I think it’s just one of those things to be like, why not be that aware? Why not put it out there and be like, “If something other than your coursework, for example, even your financial aid administration after a kind of rough… you know, whatever, say the right words, to characterize it… please let me know so that I can be aware and you can move past that sort of thing.” These things that just add up and distract students who might otherwise be capable in the academic content. So just a lot of different things, honestly, about administrative processes, that really well intentioned, and in many cases can be proven to help support students in that, but just the administration of them, in this case it’s evidence through like holds, needs to be revised continually and kind of perpetually revised, because at the very least, each of the institutions and it wasn’t just these 12 I mean, research by the American Association of collegiate Registrars (AACRAO), and then also ITHAKA S+R had done some research on the topic, just revealing the scope of those administrative processes. Administrative processes should not be almost like a second admissions requirement for students, if they’ve been academically admitted, they shouldn’t be derailed by having to decipher something that literally just may not be well managed, because it’s accumulated over time.

John: And we think that would disproportionately affect first-generation students, where parents are not giving them as much guidance, perhaps, in terms of navigating all those little hoops that they have to jump through.

Peace: Yes, one example here, we’ve got financial aid recipients, which are not just first-generation students. I am a first-generation college graduate. And I only know now what I didn’t know, and thank God, that kind of thing did not stop me because I can look back and go like, “Wow, that was decades ago.” What I would say is the experience that I had, and I may be the case study of one having been exposed to some of these findings with the holds, I as a parent will not stand for that. If I get wind of it with my kid when she goes to college. Whether I would or not, since she’ll have to manage her own affair and stuff like that. But she actually knows about it now too. And just that’s no more than I would stand for a lot of things with my health insurance that were giving me hassles or my paycheck or anything else that’s really super important to me. When it comes to the administration, this is not again, it’s not like pointing fingers or anything. Things just develop over time, and they need to be revised and revisited. Not least of which because you have a new generation of people, but also just because the computer gets buggy and all of a sudden it’s really standing in the way.

Rebecca: You’’ve given us lots to think about. Thanks so much for all the work that you do, Peace, and deciphering it for us as well.

John: We always end with the question. What’s next?

Peace: Well on an organizational news level, we’ll be planning to really see update of the high school graduate projections in the last quarter of this year, 2024. That’s a big huge thing, especially given what I’ve just revealed to you. It’s hard to make projections or predictions about anything, I still don’t feel very much on solid ground there. And you know, I just want to mention on a personal level, me here sitting in my office, not a faculty office in the outside world, so to speak. I’m just really being intentional nowadays, like, I never have been before about compassion with myself and my colleagues. If I had students it would be compassion with my students, because it just really feels like things… zs we know, again, we just had the Debbie Downer discussion… things have been pretty frantic for years now, and it doesn’t feel like there’s any end in sight, because we’ve got some things looming on the horizon. But I’ve been really noticing that a few moments of silence and reflection, like literally just a couple, two to three moments, it goes a long way to getting me further than the two or three hours of just unrelenting pounding away at work that I also end up doing, so I mention that. We have to be kind with ourselves, no matter what we’re doing, and with our colleagues, and I would advocate for that, the compassion and kindness, bringing that to the learning environments can really go a long way, I think.

Rebecca: That’s a really good reminder, when it feels like there’s so much work to be done for sure.

Peace: We can do it.

John: …and having data on incoming students can help prepare us for what’s to come. So thank you for your work on this.

Peace: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

340. The Alan Alda Center

Graduate programs prepare students to communicate with other scholars in their discipline, but do not generally prepare them to communicate with public audiences. In this episode, Brenda Hoffman joins us to discuss a program designed to help scientists develop effective public communication skills. Brenda is Associate Dean for Academic Affairs and Graduate Program Director for the School of Communication and Journalism at Stony Brook University.

Show Notes

Transcript

John: Graduate programs prepare students to communicate with other scholars in their discipline, but do not generally prepare them to communicate with public audiences. In this episode, we discuss a program designed to help scientists develop effective public communication skills.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane , an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Brenda Hoffman. Brenda Hoffman is Associate Dean for Academic Affairs and Graduate Program Director for the School of Communication and Journalism at Stony Brook University. Welcome, Brenda.

Brenda: Thank you for having me.

John: Today’s teas are:… Brenda, are you drinking any tea?

Brenda: I have water. [LAUGHTER]

Rebecca: …as we always say, the fundamental ingredient of tea.

Brenda: If I start drinking tea, I’m probably going to cough and have to [LAUGHTER] put it aside and it’s going to burn my throat and I gotta stay safe with the water. Know your audience is always my number one. [LAUGHTER] So I’m knowing myself… [LAUGHTER]… sticking with the water, but usually my tea of choice is spearmint.

John: Nice.

Rebecca: Oh, yum.

Rebecca: John, today I have a Scottish afternoon tea.

John: And it is afternoon, so that’s very appropriate. And I have…

Rebecca: …for once…

John: … a Tea Forte Black Currant tea?

Rebecca: Cool. So we’ve invited you here today, Brenda, to discuss the Alan Alda Center for Communicating Science and the Science Communication Graduate Program at Stony Brook. Could you tell us a little bit about the origins of the Alan Alda Center and the Graduate Program in Science Communication?

Brenda: Yeah, so the Alan Alda Center started with Alan Alda and his idea about creating a training center for scientists. A lot of people know him from M.A.S.H., but what he’s also well known for is a TV show called Scientific American Frontiers, where he was interviewing scientists and learning about their work. And he found it so interesting as they were talking, just to have that conversation. And he always describes it as “They started very closed off, and then the more I talked to them and asked questions and they felt more comfortable, their guard came down, and they were more willing to talk.” So his idea was how can we train scientists to do that sort of thing, and talk in a more conversational way without having someone on the other end having to pull that out of them. So he shopped around this idea for a training center, and it landed here at Stony Brook University, and that was ages ago at this point, it feels like. And at this point, we’ve trained over 20,000 scientists, healthcare professionals all around the world.

John: And as part of that you also now have a graduate program in science communication, could you tell us a little bit about that, and how it connects to the Alan Alda Center.

Brenda: Yeah, so the School of Communication and Journalism is really the umbrella here at Stony Brook. And we offer a Master of Science in science communication. And we also offer an advanced graduate certificate in science communication for our graduate students here at Stony Brook that may be enrolled in science programs and want a four class add on to their program just to have that extra communication expertise and training and just some time to practice and get some feedback while they’re still in school. So those programs are really closely aligned, they’re very focused on training students to become professional science communicators. So that’s not to say that students couldn’t go on for a PhD after, but that’s not the focus of training students to become researchers and academic writers. The focus of these programs really is for students to either take their own science, their own social sciences included in that, their own research, or someone else’s, and help that person to translate that really complex information into ways that the lay public can understand, and ensuring the integrity of the science while doing that, and the accuracy of the science rather than just what we would say, dumbing it down, but really simplifying it and making it easier to understand.

Rebecca: There’s a lot of science discussions that happen in the media, on social media as well, from climate issues, to COVID treatments, and many other topics. But a lot of this is done with people with little or no training in science. So why aren’t scientists more engaged in public discussions on such issues?

Brenda: Well, you’d have to ask a lot of them because I’m sure their answers would differ. If I had to guess, I think a part of that would have to do with cancel culture, there’s this pressure to communicate with absolute certainty, or else. And so this idea of communicating uncertainty can be very intimidating. And so that’s one of the things that the Alda Center really focuses on in training scientists and medical professionals is how to communicate what science is and how it works and where it may not be 100%. And what does that margin of error look like? And what does it actually mean? I think. probably comes into play a lot. And so the center is really focused on helping scientists who are interested in engaging with different audiences really share their work and inspire others in ways that are meaningful for them.

John: In terms of the master’s program in science communication, what are the career objectives of most of the students? You mentioned that many of them are not scientists, per se, but they’re focused on some career in science communication, what type of careers are they going into?

Brenda: A lot of them are going into or already in these careers as professional science communicators. Which is not a term that you can plug into a job search engine and find. These careers go by lots of different names. Often it’s in the communications departments, but it could be within actual labs, people who are being those connector pieces that are working with the marketers and the communications teams, and they’re the middle person between those teams and the scientists or they could be going into careers where they’re content creators. A lot of students recently are interested in podcasting and webinars and that sort of realm. So I see that becoming a big career path as well. Social media, and again, those careers are going to go by lots of different names, depending on what company, what region you’re in. But I would say a lot of our students are are looking to fill those gaps, and they have a real passion for helping people communicate that science, I think that was one of the things that really was interesting for us is we were expecting a lot of scientists coming into our program saying, “I want to learn how to communicate my science.” And while that is a decent chunk of our students, we also have a lot of social scientists coming to the table saying, “I want to help. I want to take my communication skills or my psychology skills, and I want to advance them and I want to learn how to practice this. And I want to learn how to work with these people and do a lot of what Alan was doing in that Scientific American Frontier show.” How do you get the important pieces out of these people and put it into a coherent story and then work with them to get some feedback, to ensure that in that translation process, the information and the science is still upheld in that. I would say a lot of the students are looking for those kinds of careers, where they’re what we call boundary spanners, and again, that’s a newer term, this is an entirely newer field. So I think students are going to be really looking for those types of careers.

Rebecca: One thing that I think about is the importance of people in these roles in actually generating interest for science programs and to become scientists. Because if the science is inaccessible, then it’s not necessarily something people understand or know exactly what it means to be a scientist, or what the study of science is like. I know that at my household, I have a small kid, and we watch lots of videos in media about science. And when it’s delivered in a way that we all can understand, we all get much more engaged.

Brenda: People think about science as scary, oftentimes, and especially when you’re thinking about that K-12 demographic, where they’re really starting to shape those ideas. I think that’s where, if they don’t understand something, or the class is a little bit harder, they get scared of it. And if they’re actually able to engage with scientists, I think their viewpoints will actually change, because when you start working with these people, it’s like, “Oh, they’re just like me, they’re not all wearing white lab coats and have a Petri dish.” [LAUGHTER] So I think breaking some of those barriers, and really overcoming those challenges, and those norms and expectations can be very helpful.

Rebecca: Yeah, that exact example happened yesterday. My daughter came home so excited that one of our SUNY Oswego scientists was at her school talking about the eclipse, so that they could understand the history of that. And she came home so excited and told me all about getting to meet a scientist and how cool that was. [LAUGHTER]

Brenda: Aaahhh. That’s what we need to do more of.

Rebecca: Yeah. [LAUGHTER]

John: On the other side of that, in most graduate science training, and most social science graduate training, one thing we’ve noted in past podcasts is there’s not a lot of preparation in terms of teaching skills, but there’s also not a lot of preparation in terms of communicating to a public audience. Graduate programs are very effective in teaching people how to present papers at professional conferences when they’re only talking to other scientists, but, in general, there’s very little work in preparing students to talk to the public. So it sounds as if this program might be the type of thing that many of those people in graduate programs could benefit from.

Brenda: Yeah, I think so. You know, you think about different cultures and different graduate programs in different fields. The expectations are all over the map. I mean, if you look at health care, you really see that come full circle in terms of bringing communication to the forefront and bringing it to medical student training. Even pre-COVID, this was really becoming more important, you were starting to see schools adopted, even if it was just one class in communication training or bedside manner, what we call patient-centered communication, or provider-patient communication, that was really becoming part of the norm. And if you look now, almost all graduate schools have some sort of communication woven in, even if it’s through like a grand rounds or something like that. So I think we’re starting to see that shift with science as well, at least on our own campus. We have a lot of graduate programs coming to us saying, “Hey, can you come in and do some sort of workshop with us? Can our students join your classes?” Our foundational science communication class, we have a number of programs on our campus that actually require that one class as part of the graduate program, so I’m hoping that We’re gonna start to see that shift more broadly as well.

Rebecca: So we’ve talked a little bit about the graduate program and the certificate program and how that might support students who are in science programs. But also, you have the Alan Alda Center, can you talk a little bit about the difference between your graduate programs and the Center?

Brenda: Yeah, so the center is very external facing in that we go out to government labs, other universities, corporations, and we train people as groups in teams who are already in their careers all the way from graduate students up to senior scientists. And we have two-day modules in person, and then we also offer a number of online three-hour modules. The Graduate Program uses some of that, but it’s much more academic based, in that students go through a foundational science communication course where they learn the science of science communication as a field, and they read some of that literature. How is the field developed? Where is it now? Where is it heading? They take a research methods class, where they can just learn about basic social science research methods, so that they can really look at a study and be able to judge is this a good study to talk about or report on? Was it done ethically from what you can tell? How many people were in their sample? How generalizable are these results. And then they also take classes in diversity. And they can pick from a number of electives having to do with science policy, podcasting, things like that. And so they do get the benefit of the Alda Center. So we have this program where we integrate the Alda offerings into our coursework. So all the students when they come, in that foundational required class, they will learn the Alda method, the ins and outs of basically what would happen in the morning of the first day of our science communication program. And that introduces them to the method, understanding how is improv used with communication theory and practice and research, and how is that all intertwined. And then throughout their coursework, they will receive some of the actual online modules that we offer to external clients as somewhat of a guest lecture, so our Alda facilitators will come into a class period or two and go through that module, whichever makes sense. For example, there’s a class on communicating science and health risks to the public. And so our media interviews module fits really nicely into that as students are preparing for public briefings and dealing with questions. They get that media interviews experience as part of that class to prepare them for that final assignment. So there is a lot of integration, and I think it’s one of the benefits of our program that the students actually do get to experience that. The Alda Center does not offer programs for individuals. So if I’m over at this university on the west coast, and I decided, “You know what, I just want to go through a program.” At this point in time, that’s not offered. We do programming for groups, mostly, and anywhere from 16 to 32, so if you are going to get a master’s degree or go on for some professional development, I think that’s what sets our program apart is students do get that experience and they get to work with our facilitators, and they are experiencing that actual training. It’s not like they’re getting a different version of that.

Rebecca: So you mentioned the Alda method. Can you talk a little bit about what that is?

Brenda: Yeah. So, when I started at the Alda Center, I had communication training. I knew how to give a talk, I knew how to engage people, I knew how to do this didactic sort of training. And when I came to the Alda center, I had this idea of what it was going to be. And I think this is a lot of people’s experience of there’s going to be some sort of lecture portion and an activity and then I’m going to do something at the end. And it really flipped all of that on its head for me when I saw what it was, because it combines traditional improvisation theater training that actors would get to help them connect with a scene partner. So I love this line that Alan always says, “When I was on a show, or I’m reading a script or something, I’m acting out those lines, I’m saying those lines because they were meaningful to me, the other actor was drawing those lines out of me. It’s not because the lines were on the page, there was some sort of connection between those two actors that really made that experience feel more authentic.” So we put our scientists through that training, that improvisation training, with really no information upfront about necessarily what they’re going to experience. We tell them to bring comfy clothes and you’re going to be standing a lot, you’re going to be interacting with different people, you’re going to be making direct eye contact and a lot of those activities seem very abstract. It’s kind of like “What are we doing? Why are we doing this in the moment?” And then what we do is we debrief the exercise at the end and relate it back to communication theory, communication, research, evidence-based findings of what works and how to make connections with people. It’s very experiential in that way, which I think is what makes it different. It’s one thing for me to tell you: “make eye contact and smile and move around the stage.” It’s another thing to actually make the eye contact with someone and feel what that feels like and get over that hump of the awkwardness and then be able to really benefit from what it’s like to make that connection with someone, whether you’re in person or even whether you’re online. So I think that’s what makes it really unique and unexpected to some.

Rebecca: We recently had the actors from the London stage come and do a workshop with some of our graduate students using some improv techniques that we didn’t warn them about either. [LAUGHTER] And it was really powerful. Can you give an example of what a couple of those activities might be like, just to make it come alive for some folks that maybe have never experienced improv training?

Brenda: Yeah, of course. So probably the most fundamental one is something that we call “mirror.” And basically, we have partners stand across from each other, they start with their hands up, almost like they’re going to high five each other both hands, and one person is the leader, the other person is the follower, and there’s no talking in the exercise. So we say, “Alright, one person just start moving, do whatever feels comfortable to you, and the other person is going to mirror you exactly, almost like you were looking at your reflection in a mirror.” And so they’re like, “Okay, why am I doing this?” And you get over that awkward stage, and then it gets a little funny, some people start doing the macarena. They realize, oh, this is something that maybe the other person knows, right? And so that makes it a little bit easier, breaks the ice. But really what they’re doing is this idea of the norm of reciprocity, they’re being able to match somebody else’s movements. And we almost do that in conversation as well. If I’m telling you a certain level of information about myself, you would probably match that with the similar level of information about yourself. It also helps you to connect with an audience and to be able to really focus on them and recognize when they’re understanding something, when they’re liking something, when they’re not, when they might be confused. And that gives you a sense of when you may need to stop and shift your approach a little bit. I think a lot of times people are told, just look at the back wall and talk to the wall. And what that ends up doing is it disconnects you from your audience, because you’re not able to actually get that feedback in real time from them about what’s working and what’s not. If they have a question about something or they’re not understanding, or they just seem upset about something you said, that may be something that you want to stop and address, and in showing them that you’re a real person and you care and you want to be connected with your audience. That’s a really important step to take. So being able to experience that fully through this exercise, I think just gives them a different perspective to think about when they’re on that stage, and they’re feeling uncomfortable, and they’re trying to quell those nerves, they can think back to these exercises of “Oh, yeah, I did this here and that was the takeaway.”

Rebecca: Can you talk a little bit about what the debrief is like after one of those exercises?

Brenda: Yeah, so the debrief is actually a lot like a flipped classroom. So rather than us just lecturing at you, it’s really a lot of questions, almost like a podcast, where you’re asking someone questions, and they’re answering. And so we’re almost trying to draw it out of them. What was that experience like? We start very surface level. What did it feel like? What did you like? What did you not like? And then we start to get deeper? Well, what does this have to do with science? Why are we doing this abstract activity of follow the leader with your arms? And then maybe one person will have a slight idea, and then they’ll start to build on it, and we’ll filter in as well. And then we really get to that translational piece of “Okay, now, how can you take these concepts and apply it to the work that you’re doing?” And that’s where they’re really making those connections for themselves, whether that’s as a group or whether that’s at smaller teams within that larger group, or whether that’s just for them individually in the work that they’re doing? So it’s a lot like a conversation, but we’re drawing those inferences from them, and then adding in the information as well.

John: How do people respond to this training? How do they react afterwards?

Rebecca: Or how did you respond at first? [LAUGHTER]

John: Yeah.

Brenda: I’ve liked to think that I’m very open to new things. Even I was standing there like “Guys, what in the heck? What are we doing?” First of all, no one told me how to dress for the day. So I was in like a pencil skirt and blazer that didn’t allow me to really move my arms and heels. So I was already kind of like “What’s happening?” But by the end of the day, I almost had no words. It was just such a different experience from anything that I had experienced. And I think with a lot of people, that’s what we see. They’re kind of like “What is this?” They’re a little nervous. When people get nervous or anxious about something or uncomfortable, they tend to just be like, “I don’t like this.” And then most of them will get to this place where they’re like, “Wow, that was transformational and I’ve never experienced anything like this before.” And then you will have others that are just very standoffish and are uncomfortable. They don’t want to do the exercises, they want to know what’s going to happen before it happens. And that’s something you have to work with. And a lot of times you’ll see the people next to them, like, “Hey, it’s fine, just do it, just do it. It’ll be okay. And they’ll grit through it.” But you definitely have to be open to something new. And for me, it was a breath of fresh air, it was something completely different. And I think that’s probably the most common experience that we hear of.

John: You mentioned working with graduate students or with people who are already out there? What are some of the other connections you make with people on campus?

Brenda: Yes, so we really like to offer something back to the campus community, because so much of the work that the Alda Center does is off campus. And so a big part of that is the Graduate Certificate in Science Communication, and obviously, the master’s program as well. But we also do a number of talks around campus for faculty, for postdocs who aren’t necessarily enrolled in our program or teaching in the program, but who are just interested in taking a class or experiencing the Alda method. And we also use it as a ground for training our instructors as well. So we have a new program that is being developed and being piloted. Nothing we put out is just slapped together and sold to a client. Everything is pilot tested. We get feedback from the audience, we go back, we adjust. And so it’s a pretty lengthy process. And so a lot of times we have people on campus that are like, “Look, I can’t devote a whole two days to the science communication experience. But I would just like to experience a workshop or I’d like to be involved.” And so when we have those pilot programs, we do them right here on our campus. We have an open call. So we’ll have everyone from faculty, to postdocs, graduate students, and even some of the campus leadership that will participate. Word has really gotten out. So we’re starting to see, as new leaders come to campus, they’re like, “We’ve heard of this Alda Center, and we feel like we want to take a workshop.” So we’d like to integrate our own community in in that way as well.

Rebecca: Sounds like a really great opportunity for professional development for faculty and staff. It probably provide some interesting ideas for their own classrooms.

Brenda: It’s a lot of fun when you do it on a campus where everyone works, because when we travel out people will come in from different units that are located in different parts of the country, and they don’t necessarily work together. But when you’re on a campus where everyone works together, it’s really nice because you start to see during lunch, during the breaks, and even at the end of the session, they’ll start to be making connections and building working groups and talking about research ideas or grants they should write together. And so I think that’s a really nice outcome of the Alda Center as well is just building connections between scientists. We’ve heard some organizations that we were at 5, 6, 7, 8 years ago, are still having working sessions and working groups. They do this on their own. They’ll have the group of folks who went through the workshop together, come back once a semester, once a month even, sometimes, and just have it be a brainstorming brown bag session where they bounce ideas, or they say, “Hey, I have a presentation coming up, can I do this and get some feedback from all of you?” So it’s really nice to see those types of relationships evolve and maintain over time as well.

John: Do you think faculty should do more improv in their classes [LAUGHTER] to help break down some of the barriers with students? I mention that partly because we had a reading group on campus this past year in the fall of 2023 with Sarah Rose Cavanagh’s book, Mind over Monsters, and she has a chapter on using improv to help students make connections with each other and to help build a community. Do you think that type of activity might be useful in general?

Brenda: Yeah, I think anytime you can change up the expectation of what’s going to happen in a classroom, that’s an A+ plus for you as the instructor. I will say that we used to do discussions and activities and the students liked those well enough. And after COVID, when everyone went online, and the classes all became the same, it all became post to a discussion board and then do this and then do that. When we came back from COVID, we really had a flip everything on its head and start over because the students just weren’t as engaged. Those activities that once were super fun, and they loved doing, they just were burnt out and exhausted from doing those. So I think anytime you can have those conversations where students get to just let their hair down for a little bit. There’s so much pressure on our students now to find jobs and get the internships because there’s so much competition and the more you can have real discussions and show students that you’re a real person too, they can be really helpful. So again, you have to know your audience. You have to know what students are expecting and which improv exercise might take it a little too far and be a little too abstract. Something, maybe [LAUGHTER] that you would try in a communication class, you wouldn’t go that far in a chemistry class. But there are exercises, they run the gamut of how intense they’re going to be. So even just starting out with a little activity could really go a long way to having students break that ice and open up.

Rebecca: Is there a place where using the Alda Method would be particularly beneficial to embed? If you had a magic wand and you could embed it somewhere, where would it have the most impact?

Brenda: I would say federal funding. As part of federal funding, when people get grants, you’re starting to see the shift of what are the broader impacts of this project, we’re starting to see more interdisciplinary work come together and really be the requirement for some of these funding opportunities. And so actually coming in and having the Alda Method be part of an award, [LAUGHTER] a post-award requirement, have your research team and your community partners all go through this training together, learn how to not just do the research, but disseminate it out. Communicating it out is half the battle. If you’re doing all this research and you’re publishing in journals that the general public isn’t reading, how much impact does it really have? So being able to not just present it in the ways that we’re used to, but also to have conversations about it. I think a lot of people are not as comfortable with presentation in general, but then when you get into a conversation with interviews, or the media where it’s a little more uncertain, we’re just not trained in that, especially in a lot of the science fields, there’s just not a lot of training and expectations of that even in classes, so being able to weave it in. I would say in a perfect world, if I could put it anywhere, that’s probably where I’d put it to really give these teams who are being awarded all this money, an extra tool in their toolkit to really be able to go out and disseminate that amazing work once they’re finished and throughout the process.

John: And as you were saying that, I was thinking back to COVID, when I was reading any scientific article by epidemiologists, or by other researchers, but then I’d hop on Facebook, or I’d look at Twitter and the discussions there seem to be somewhat different than the ones that were showing up in the journals or in the working papers. And I think you make a really good point there about suggesting that when people are going to do publicly funded research, having them develop skills in disseminating that information to the public could be a really useful public service.

Brenda: Well, I think what that also shows is the importance of knowing your audience. I think a lot of times we get up and we give presentations, because we think that’s what people need to know, and what people want to know. And when you actually start talking to them, that’s not actually what they’re interested in. So being able to have those conversations and break those walls down could really give scientists a whole new way of knowing and a whole new way of understanding their audience and what’s meaningful for them.

Rebecca: I like that you’re also kind of pointing out that the audience might be curious about things. I think sometimes we’re so in our own discipline or our own lane, that we forget that people outside of that lane might be curious about what we’re doing, or about what we’re studying, but don’t always have the language or a way to ask those questions or have access to that information.

Brenda: I think we take a lot for granted too. At the Alda Center, we call it the curse of knowledge. You don’t know what it’s like to not know something, once you know it. [LAUGHTER] It’s training we give doctors a lot. Remember what it was like the first time you heard this information. And imagine that it’s the first time for your audience. So even in talking about academic programs, there was this one time I was sitting next to a guy on an airplane, of course. And I said “oh, I do communication research.” And he’s like, “Oh, well tell me more about it.” And so I wanted to talk about my research into teams and healthcare. And I started talking and he was like, “Okay, so health communication, what is that?” And so our whole conversation evolved into a discussion of just defining health communication, something that, for me, was just part of my everyday vocabulary. So it wasn’t jargon to me. But in talking to someone else, it was a good reminder that hey, not everyone knows what this means. This is great. Like, let’s talk about it.

Rebecca: There’s been lots of things for us to think about in this conversation. So thank you so much for all of the food for thought. And we always wrap up with one final question, which is: “What’s next?”

Brenda: Well, that’s the million dollar question. For our graduate programs. I think we’re hoping one day to be able to build out a PhD program in communication so that we would have a professional pathway but we’d also have an academic pathway for those that are interested in that. For the center, continuing to build new programming and listening to the audience’s needs. What was important 5-10 years ago is not necessarily the same thing that’s important today, and what’s important today won’t be the same thing that is top of mind 10 years from now. So think just continuing to be present and listening to our audience’s needs, whether that’s public, whether that’s potential students, and just being able to respond to that and staying true to our mission and staying true to the science, I think will make a better world for everybody.

John: Well, thank you. We’re looking forward to seeing more informed science communication out there in the world. And it’s great that you’re doing this and I hope this spreads.

Brenda: Yeah, thank you so much for having me.

Rebecca: Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]