A year ago, our campus announced that it was shutting down for a two-week pause so that the COVID-19 pandemic could be brought under control. To help faculty prepare for remote instruction, we released our first episode of many on March 19, 2020, with Flower Darby. We thought this would be a good moment to pause and reflect on this journey.
- Flower Darby (2020). “Pandemic Related Remote Learning.” Tea for Teaching Podcast. Episode 126. March 19.
- Todd, E. M., Watts, L. L., Mulhearn, T. J., Torrence, B. S., Turner, M. R., Connelly, S., & Mumford, M. D. (2017). A meta-analytic comparison of face-to-face and online delivery in ethics instruction: the case for a hybrid approach. Science and Engineering Ethics, 23(6), 1719-1754.
- Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies.
- Lang, J. M. (2020). Distracted: Why Students Can’t Focus and What You Can Do about It. Basic Books.
- Linda Nilson (2019). “Specifications Grading.” Tea for Teaching Podcast. Episode 86. August 21.
- Susan Blum (2020). “Peagogies of Care: Upgrading.” Tea for Teaching Podcast. Episode 145. July 22.
Rebecca: A year ago, our campus announced that it was shutting down for a two-week pause so that the COVID-19 pandemic could be brought under control. To help faculty prepare for remote instruction, we released our first episode of many on March 19, 2020, with Flower Darby We thought this would be a good moment to pause and reflect on this journey.
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.
John: Our teas today are:
Rebecca: I’m drinking English Afternoon for the first time in about a year. Because I’ve been home, and working from home, I’ve been drinking pots of loose leaf tea instead of bag teas. And so I’m bringing back the comfort of a year ago.
John: And we still have in the office several boxes of English A fternoon tea, but they are wrapped in plastic. So I’m hoping they’ll still be in good shape when we finally get back there …once this two week pause that we started about a year ago, ends.
Rebecca: Yeah, when we recorded that Flower Darby episode was the last time we saw each other in person.
John: Well, there was one other time…
Rebecca: Oh, when you dropped off equipment.
John: I dropped off a microphone and a mixer for you so that we could continue with this podcast. Actually, I think we saw each other from a distance because I left it on the porch because I had just come back from Long Island where infection rates were very high.
Rebecca: Are you drinking tea, John?
John: …and I am drinking Tea Forte black currant tea today.
Rebecca: A good favorite. So John, can you talk a little bit about where you were at mentally and just even conceptually, in terms of online teaching and things,when the pandemic started a year ago,
John: We were starting to hear about some school closings in other countries and in some cities in the US where COVID infection rates were starting to pick up and it started to look more and more likely that we’d be moving into a shutdown, in the week before we were to go to spring break. I was teaching at the time one fully asynchronous online class and two face-to-face classes. When it was looking more and more like we’d shut down I talked to my face-to-face classes about what options we’d have should we go online for some period of time. And I shared with them how we could use Zoom for this. And we had already used Zoom a few times for student presentations when students were out sick or had car trouble and couldn’t make it into class. Because they were actively using computers or mobile devices every day in class, anyway, they all had either computers or smartphones with them. And I had them download Zoom and test it out, asking them to mute their mics. And very quickly, they learned why I asked them to do that. I wasn’t very concerned because we’ve been doing workshops at our teaching center for many years now with remote participants. And we’ve been using Zoom for at least five years or so now. So I wasn’t really that concerned about the possibilities for this. And I thought the online class would go very much like it had and the face-to-face classes would work in a very similar way… for the short period that we were expecting to be shut down. I think even at the time, many of us thought that this would be somewhat longer, but I wasn’t terribly concerned at the time, because infection rates were still pretty low. And I think we were all hopeful that this would be a short-run experience.
Rebecca: And also maybe the fact that you’ve taught online before didn’t hurt.
John: Yeah, I’ve been teaching online since 1997, I believe. And so I was pretty comfortable with that and I wasn’t concerned at all about the fully online class, I was a little more concerned about the students who were used to the face-to-face experience adapting to a Zoom environment.
Rebecca: I had a really different experience because I was on sabbatical in the spring working on some research projects related to accessibility. Because of that, I was able to quickly adapt and be able to help some communities that I’m a part of, related to professional development. So I stepped in and helped a little bit with our center and did a couple workshops and helped on a couple of days with that. And I also helped with our SUNY-wide training too, and offered some workshops related to accessibility and inclusive teaching at that time. And the professional association for design locally, we had a couple of little support groups for design faculty.
John: I wasn’t too concerned about my classes, but I was a little bit more concerned about all the faculty that we had who had never taught online. And so, as you just said, we put together a series of workshops for about a week and a half over our spring break helping faculty to get ready for the transition to what we’re now calling remote instruction.
Rebecca: At that time, too. I had no experience teaching online, I’d used Blackboard and things like that before, but not to fully teach online. So for me, it was a really different experience. And I was helping and coaching faculty through some of those transitions too, not really having had much experience myself. So I had the benefit, perhaps, of seeing where people stumbled before I had to teach in the fall. But I also didn’t get any practice prior to fall like some people did with some forgiveness factors built into the emergency nature of the spring.
John: I think for most faculty, it was a very rapid learning process in the spring and instruction wasn’t quite at the level I think anyone was used to, but I think institutions throughout the country were encouraging faculty to do the best that they could, knowing that this was an emergency situation, and I’m amazed at how quickly faculty adapted to this environment overall.
Rebecca: One of the things that I thought was gonna be really interesting to ask you about today, John, was about online instruction, because you have such a rich history teaching online, and there are so many new faculty teaching online, although in a different format than perhaps online education research talks about. Many people taught asynchronously for the first time, but there’s also a lot of faculty teaching online in a synchronous fashion. There’s a lot less research around that. How do you see this experience impacting online education long term.
John: I don’t think this is going to have much of a dramatic impact on asynchronous online instruction in the long term. Online instruction is not new, it’s been going on for several decades now. There’s a very large body of literature on what works effectively in online instruction. And under normal circumstances, when students are online and faculty are online because they choose to be, online instruction works really well. And there’s a lot of research that suggests that when asynchronous courses are well designed, building on what we know about effective online teaching strategies, they’re just as effective as well designed face-to-face classes. However, a lot of people are trying to draw lessons from what we’re observing today. And what we’re observing today, for the most part, does not resemble what online education normally is, primarily because the students who are there, and many faculty who are there, are there not by choice, but by necessity. And one of the things that has come up in some recent Twitter conversations, as well as conversations that we’ve had earlier, is that many online students in asynchronous classes have been asking for synchronous meetings. In several decades of teaching online, I’ve never seen that happen before, and now it’s very routine. And I think a lot of the issue there is that, in the past, most online students were there for very specific reasons. So they may have had work schedules that would not allow them to sign up for synchronous classes. Some of them are in shift work, some of them were on rotating shifts where they couldn’t have fixed times of availability. Some of them would have large distances to commute and it just wasn’t feasible, or they were taking care of family members who were ill, or as part of their job, they were required to travel. In most of the online classes I’ve had in the past, there were some students who were out of state or out of the country. I had students during the Gulf War who were on a ship, the only time they missed a deadline was when their ship went on radio silence before some of the attacks down there. They simply would not have been able to participate in synchronous instruction in any way. And I think a lot of the people who are now taking asynchronous classes, strongly prefer a synchronous modality and are disappointed that they’re not in that. And I think a lot of what we’re seeing is a response to that and I think we shouldn’t ignore all the research that has come out about effective online techniques in light of the current pandemic, because this is not how online instruction normally has occurred. And people are in very different circumstances now in terms of their physical wellbeing in terms of their emotional well being and just general stress.
Rebecca: Yeah, during the pandemic, many more people are in isolation, and might really be craving some of that social interaction that they might not expect out of an online class traditionally, especially if it’s an asynchronous class. But if you’re just alone, and you’re not going out of your house, there might be more of a desire during this one moment of time …this one really long moment of time. [LAUGHTER]
John: During this two-week pause? [LAUGHTER]
Rebecca: Yeah. One other thing, I guess, is important to note as we’re talking about research and what evidence shows is that hybrid can be really effective with the combination of in-person instruction complementing some asynchronous online instruction. And of course, in that traditional research, hybrid really means this in- person and then asynchronous online, this synchronous online thing wasn’t really a thing prior to the pandemic. [LAUGHTER]
John: Right. And we can’t really draw too many conclusions about this giant worldwide experiment that’s being done in less than optimal conditions without really having a control of normal instruction to compare it to. And yeah, several meta-analyses have found that while face-to-face and asynchronous online instruction are equally effective, hybrid instruction often has come out ahead in terms of the learning gains that students have experienced. Certainly, we know a lot about hybrid instruction, face-to-face instruction, and asynchronous online, but not the modality that larger of our students are in. One other factor is that when people signed up for online classes before, they did it knowing that they had solid internet connections, they knew they had computers that were capable of supporting online instructional environments. They had good bandwidth and so forth. That’s not the situation In which many of our students and faculty are working right now, because faculty and students often do not have any of those things. And they’re often working in suboptimal environments that are crowded, where there’s other people in the household sharing the same space. And it makes it really difficult to engage in remote asynchronous or synchronous work as they might have when they chose to be in that modality.
Rebecca: I do think that, during this time, though, into kind of forced online instruction, although there are certainly people who don’t like that they’ve been forced to be online, and they prefer to be synchronous or in person, I think there’s a cohort of people who thought online education wasn’t for them, both faculty and students, who have discovered that it actually really does work for them. And even me, although I teach web design and do things online, you’d think online education would seem obvious to me. But in the past, it hadn’t really occurred to me. Our education tends to be in person, and you tend to replicate what you’ve experienced. [LAUGHTER] And although I have taken some online courses related to design and technology and coding in the past, it hadn’t really occurred to me to consider some options. And I think what we’ve discovered is some of our courses work well in this modality and some don’t. Some of our courses are better positioned to be potentially online or work well in that format, and could help with some collaboration pieces, or some other things that we might be doing. It might support the work that we were already trying to do in person.
John: And I think now, all faculty have gotten much more comfortable with a wider variety of teaching techniques and teaching tools than they would have experienced before. For many faculty, just having dropboxes in the learning management system was something new, moving away from paper assignments was something very new. And suddenly, faculty were asked to use a wide variety of instructional tools that they had been very careful to avoid doing in the past. And one of the things that struck me is how many of the people in our workshops who’ve said that they were perfectly comfortable teaching in a face-to-face environment, and they just didn’t see the need for, or they didn’t think that online instruction could work for them. And now that they’ve tried all these new tools and these new approaches, they’re never going to go back to the traditional way in which they were teaching. So I think there are going to be a lot of things that people have learned during this that they’ll take back into their future instruction, even if it is primarily in a face-to-face environment.
Rebecca: It may also be some changes in technology policies in the classroom as well related to just seeing how helpful technology can be for learning, but also where it can be distracting. So I think there’s some reconsideration of what that might mean.
John: While there haven’t been so many things that I’ve enjoyed during the pandemic, one of them is that this whole issue of technology bans have pretty much fallen to the wayside. I’m not hearing faculty complaining about students using computers during their class time now. And that’s a nice feature, and perhaps faculty can appreciate how mobile devices can be an effective learning tool. And yes, there will have to be more discussions such as one we’re having in our reading group this semester, where we’re reading Jim Lang’s Distracted: Why Students Can’t Focus and What We Can Do About It. There’s a lot of discussion about when technology is appropriate, and when it’s not in those meetings. But I think faculty have come to recognize how ed tech can be useful in some ways, at least in their instruction, whether it’s in person or whether it’s remote.
Rebecca: I think it’s also important to note that how some of the synchronous technology, video conferencing technology like Zoom, has some advantages, even if our class is not synchronous online. It could just be an in person class in the future. We’ve seen the power of being able to bring guests in easily without having to deal with logistics of traveling and the scheduling considerations that are often involved with that. We don’t have the disruptions and education related to snow days and illness, both on the faculty and student side. Obviously, that depends on how severe the illness is, right? [LAUGHTER] Professional development has worked out really well online, although we’ve done online or had a Zoom component where you can kind of Zoom and being all on the same platform at the same time has been really great, being able to take advantage of breakout rooms and things like that. We’ve seen record numbers attend, and then also with advisement and office hours. It can be really intimidating to have to find an advisor’s or a faculty member’s office and you have to physically go there. And then it’s kind of intimidating. What if the door’s shut? What if they’re look like they’re busy? [LAUGHTER] There’s all these things that can get in the way that online or Zoom calls can just remove some of those barriers and also allow for more flexibility because now you don’t have to plan for walking across campus which might take some time. Or you might be able to squeeze in something at a time you wouldn’t be able to otherwise.
John: And a lot of our commuting students are commuting from 30 to 60 miles away, and it was not terribly convenient for them to have to drive up to campus at a time that was convenient for their professors just for the chance of sitting there and talking to them for a few minutes. So, the access is much easier using Zoom or other remote tools.
Rebecca: We should also get real. Zoom fatigue is a real, real, thing. It’s about 4:30 right now that we’re recording. We’ve both been on Zoom calls since early this morning. And kind of constant. Our students have been as well. There’s no let up, there’s no breaks. We don’t get the little stroll across campus to the next meeting. [LAUGHTER] There’s none of that. One of the things that I am experiencing, as someone who’s definitely introverted, is this performative nature of being on camera all the time. And I know our students are too. And John and I were talking about this a little earlier today, that, in the fall, I had tons of students participating with their cameras on and their microphones on, and even in the beginning of the spring, but there’s something about the dead of winter in Oswego, that kind of Doomsday nature of it, it’s gray here. And then the black boxes just kind of emphasize it further. And they’re not as visible as they had been before. And I think it’s partly because it’s so performative, and you’re being watched all the time. And it’s not necessarily not wanting to participate or feel like you’re present. But really, it’s just a little much.
John: And neither of us pressure our students to turn their cameras on. We welcome that, we invite them to do that, but we know there are some really sound reasons not to, because people are often working in environments that they don’t want to share with their classmates or with their faculty members. And they may have bandwidth issues and so forth. But it is really tedious to be talking to those black boxes. And as Rebecca and I talked about earlier, both of us are also creating videos. So, we get to talk to our web cameras a lot, and then we go to class, and we talk to our students. Most of our students, I think, turn their microphones on. So we get to hear them one at a time. But it’s challenging to be talking to people you can’t see all day long.
Rebecca: I think it’s particularly challenging for faculty, because there’s more of an expectation for faculty to have their cameras on both in class and in meetings than students. So I think there’s an extra level of fatigue that’s happening with faculty and staff, because it’s more performance more of the time. Some days, I really feel like I wish I could be a student and I could just turn my camera off.
John: I have a night class that meets for about three hours. And typically when we met face-to-face, we’d take a 7 to 10 minute break in the middle of that. I asked the students if they wanted to do that the first two weeks, and each time they said “No.” I said, “Well, if you need to get up, use a restroom, or walk around, please do it. But what I wasn’t considering is the fact that, while they were doing that, I was still here interacting with them the whole time. And that three-hour session can be a bit challenging by the end of it, particularly if you’ve been drinking a lot of tea.
Rebecca: That’s actually important to note that, kind of unusually, John and I are both teaching three-hour classes, that’s probably not the norm for most faculty. I’m teaching studio classes. So for one class, it’s three hours of time, two times a week, and you’re teaching a seminar class, right, John, that’s three hours?
John: Yes, that meets once a week.
Rebecca: These longer sessions, we can break up by physically moving around the classroom and things when we’re in person, it becomes more of a challenge online. And I know that I’ve been thinking more about the orchestra of it all and changing it up in my classes. So we might do something in small groups then may do something as a big group, we participate in a whiteboard activity, then we might do something else, then we take a break, then we try to do something that’s off screen for a little bit and then come back. And so I’ve tried to build in some opportunities for myself as well to be able to turn my camera off at least for a few minutes during that three-hour time or take a little bit of that time during breakout sessions or whatever, because I need a break too. Our good friend Jessamyn Neuhaus has mentioned this to us many times before, that we’re not superheroes, and we should stop trying to be superheroes. And this seems like a good moment to remind ourselves of this as well. I know for me, it’s like I need a snack, I need to go to the bathroom, I need a drink. I would do that in a physical class. I take breaks then. So I’ve been making sure we build it in, and actually even padding it a little bit and giving people longer breaks than I would in person.
John: And our campus, recognizing the challenges that faculty faced with this last fall, put in two wellness days where no classes were held, and people were encouraged to engage in activities to give them that sort of break. I’m not sure about you, but I ended up spending about seven and a half hours of that day in meetings that were scheduled by various people on campus.
Rebecca: Yeah, and students also said that they ended up really needing that time to just catch up, because the workload in terms of student work hasn’t reduced, but being on screen has increased for most people, and you just need some time away. So, it ends up taking more hours of the day, just in terms of logistics, if you actually going to give your eyes a break and things. I did a little survey of my classes and they said they spent a lot of that time kind of catching up, although maybe the pace of the day was a little slower.
John: Going back to the issue of cameras being on, one of our colleagues on campus did a survey of the students in her class asking why they chose not to have their cameras on. And the response seemed to indicate that a lot of it was peer pressure, that as more and more students turn the cameras off, they became odd to leave them on. So I think many of us have experienced the gradual darkening of our screens from the fall to the spring,
Rebecca: I found that there’s some strategies to help with that as well. One of the things I did last week was invite students to participate in a whiteboard activity online indicating what they expected their peers to do so that they felt like they were engaged or part of a community. What should they do in a breakout? And what does participation look like in an online synchronous class? And they want all the things we wanted them too. They said, like, “Oh, I want people to engage.” And we talked about what that means, that it might mean participating in chat, it might mean having the cameras on, and things like that. And that day, right after that conversation, so many people during that conversation turn their cameras on. So in part, it’s about reminding, or just pointing out that it’s not very welcoming to have not even a picture up.
John: And this is something you’ve suggested in previous podcasts to that, while we’re not going to ask students to leave their cameras on to create a more inclusive environment, you could encourage students to put pictures up.
Rebecca: Yeah, we feel as humans more connected when we see human faces. So we feel much more connected than looking at black boxes. [LAUGHTER] So I’ve definitely encouraged my students. On the first day, I gave instructions to all the students about how to do that. And then when we had our conversation the other day, when I was starting to feel the darkening of the classroom and more cameras came on, I also just invited and encouraged everyone else. If you can’t have your camera on, or you have a tendency not to be able to put your camera on, that’s not a problem, but we would really welcome seeing your face or some representation of you as an image.
John: What are some of the positive takeaways faculty will take from this into the future?
Rebecca: It’s been interesting, because we’ve had far more faculty participating in professional development opportunities, initially out of complete necessity, like “I don’t know how to use Blackboard” and starting with digital tools and technologies, and then asking bigger and more complicated questions about quality instruction online as they gained some confidence in the technical skills. So there’s some competency there that I think is really great. And that’s leading to faculty wanting to use some of these tools in classes, it might mean just using Blackboard so that the assignments are there, and the due dates are more present, and just kind of some logistical things to help students keep organized. But also, there’s a lot of really great tools that, as we mentioned earlier, that faculty have discovered that they want to use in their classes. So maybe it’s polling and doing low-stakes testing in their classes during the class. I’ve discovered using these virtual whiteboards, which actually logistically work better than physical whiteboards in a lot of cases in the things that we’re doing, because everyone can see what their collaborators are doing better. So there’s a lot of tools that I think faculty are going to incorporate throughout the work that they’re doing. But also they’ve learned a lot more evidence-based practices. And maybe you want to talk a little bit about that, John,
John: At the start of the pandemic, the initial workshops, were mostly “How do I use Zoom?” But very quickly, even back in March, we also talked a little bit about how we can use evidence-based practices that build on what we know about teaching and learning. In the spring, there wasn’t much faculty could do in the last couple of months to change their courses. But we did encourage them to move from high-stakes exams to lower-stakes assessments to encourage students to engage more regularly with material, to space out their practice, and so forth. And at the start of the summer, we put together a mini workshop for faculty on how to redesign their courses for whatever was going to happen in the fall. And it was basically a course redevelopment workshop, where we focused primarily on what research shows about how we learn and how we can build our courses in ways that would foster an environment where students might learn more effectively. Our morning sessions were based primarily on pedagogy and then in the afternoon, we’d go over some sessions on how you can implement that in a remote or an asynchronous environment, giving people a choice of different ways of implementing it. By the start of the summer, people were starting to think about doing things like polling, about doing low-stakes testing, or mastery learning quizzing, and so forth. And people started to implement that in the fall. And then we had another series of workshops in January. We normally have really good participation, but we had, I believe, over 2000 attendees at sessions during our January sessions. And during those sessions, we had faculty presenting on all the things that they’d learned and how they were able to implement new teaching techniques. And it was one of the most productive set of workshops we’ve ever had here, I believe. And what really struck me is how smoothly faculty had transitioned to a remote environment. At the start of the pandemic and during spring break, we were encouraging people to attend remotely and yet faculty mostly wanted to sit in the classroom with us, and we wanted to stay as far away from those people as we could. But about half the people attended virtually. Butwhat’s been happening as people were getting more and more comfortable attending remotely and we’ve been offering the option of people attending virtually since I took over as the Director of the teaching center back in 2008, I believe. However, we rarely had more than a few people attending remotely. And it was always a challenge for people to be participating fully when they were remote while other people were in the same room, which gave us some concerns about how this was going to work in the reduced capacity classrooms that many colleges, including ours, were going to implement in the fall. And we knew we didn’t really have the microphones in the rooms that would allow remote participants to hear everyone in the room and vice versa. Once we switched entirely online, where all the participants in the workshops were in Zoom, it’s been much more effective to have everyone attending in the same way, so that we didn’t have some people participating in the classroom and others attending remotely. And I think that, combined with faculty becoming more comfortable with using Zoom, has allowed us to reach more faculty more effectively.
Rebecca: One of the things that I saw so powerful this January, in our experience on our campus, was all of the faculty who volunteered to do sessions and talk about their experiences and support other faculty experimenting with things. And I think it was just this jolt that caused us all to have to try something new, that was really, really powerful. We all get stuck. Even those of us that know evidence-based techniques, we get stuck in our routines, and sometimes just allow inertia to move us forward and replicate what we’ve done before because it’s easier, it saves time, and we have a lot on our plates. And it’s really about being efficient, because we just have too much to do. So it was nice, in a weird way, to have that jolt to try some new things. I heard some great things from faculty that I’ve never heard from before I learned some things from some other faculty. And it was really exciting. And the personal place in my heart that I get most excited about, of course, is how many faculty got really excited about things related to inclusive pedagogy, and equity, and accessibility. We offered, on our campus a 10-day accessibility challenge that we opened up to faculty, staff, and students as part of our winter conference sessions. And we had record accessibility attendance… never seen so many people interested in accessibility before. But that came out of the experience of the spring and the fall, and people really seeing equity issues and experiencing it with their students. They witnessed it in a way that it was easy to ignore previously. And so I think that faculty, throughout this whole time, have cared about the experience that students have and want students to have equity. They just didn’t realize the disparity that existed amongst our students. And the students saw the disparity that existed amongst students, which was a really powerful moment, really disturbing for some students who had to share that moment with other people, but also a really useful experience for faculty to really buy into some of these practices about building community, about making sure their materials were accessible. And all of that has resulted in a much higher quality education for our students.
John: It was really easy for faculty to ignore a lot of these inequities before, because the computer labs, the Wi Fi, the food services, and library services, and lending of equipment provided by institutions, compensated for a lot of those issues, so that disparities in income and wealth were somewhat hidden in the classroom. But once people moved home, many of those supports disappeared, despite the best efforts of campuses in providing students with WiFi access with hotspots or providing them with loaner computers. And those issues just became so much more visible. It’s going to be very hard for faculty to ignore those issues, I think, in the future, because it has impacted our ability to reach a lot of our students. And it has affected the ability of many of our students to fully participate in a remote environment. But going back to that point about people sharing, I also was really amazed by how willing people were to volunteer and share what they’ve learned in their experiences. Typically, when we put our January workshop schedule together, we call for workshop proposals from people. And we typically get 5 to 12 of those, and they’re often from our technical support people on campus. And it’s rare that we get faculty to volunteer. And normally we have to spend a few months getting faculty to volunteer so that we get maybe 20 or 30 faculty to talk about their experiences. We had about 50 people just volunteer without anything other than an initial request, and then a few more with a little nudging, so that we ended up with 107 workshops that were all very well attended. And there were some really great discussions there because, as you said, people were put in an environment where the old ways of doing things just didn’t work anymore, and it opened people up to change. We’ve been encouraging active learning and we’ve been encouraging changes in teaching practices. But this pretty much has reached just about everybody this time in ways that it would have been really difficult to reach all of our faculty before.
Rebecca: It’s easy during a time like a pandemic to just feel like the world’s tumbling down. And there’s no doubt about that. But it’s a time where I’ve also been really grateful to have such great colleagues. Because not only have we seen faculty supporting each other and using new technology, the advocacy that they’ve demonstrated on behalf of students who really had needs has been incredible. Likewise, for faculty, we’ve witnessed some really interesting conversations amongst faculty about ways to reduce their own repetitive stress injuries and other accessibility issues that faculty are also experiencing, equity issues that faculty are experiencing, caregiving responsibilities that are making things really challenging for faculty. But there’s a really strong network of support amongst each other to help everyone through and there’s no word to describe what that means other than being grateful for it, because people have been so supportive of each other. And that, to me, is pretty amazing.
John: Faculty have often existed in the silos of their departments. But this transition has broken down those silos. It’s built a sense of community in a lot of ways that we generally didn’t see extending as far beyond the department borders. There were always a lot of people who supported each other, but the extent to that is so much greater.
Rebecca: So we’ve been talking a lot about this faculty support. John, can you give a couple of examples of things that faculty have shared that have worked really well in their classes that they weren’t doing before?
John: One of the things that more and more faculty have been doing is introducing active learning activities and more group activities within their classes in either a synchronous or asynchronous environment. And that’s something that’s really helpful. And as we’ve encouraged faculty to move away from high-stakes assessment, and many faculty have worked much more carefully about scaffolding their assignments, so that large projects are broken up into smaller chunks that are more manageable, and students are getting more feedback regularly. Faculty, in general, I think, have been providing students with more support, because when in a classroom, you were just expecting students to ask any questions about something they didn’t understand. And sometimes they did and sometimes they didn’t. But I think faculty realize that in a remote environment, all those instructions have to be there for students. So in general, I think faculty are providing students with more support, more detailed instructions, and often creating videos to help explain some of the more challenging parts that they might normally have expected students to ask about during a face-to-face class meeting.
Rebecca: I think previously, although faculty want to be supportive, they may not have been aware of some of the mental and emotional health challenges that students face generally, but have been amplified during the pandemic. Students who might experience anxiety or depression and how that impacts their ability to focus, their ability to organize themselves and organize their time, all of those things have become much more visible, just like those equity issues. And so I think that faculty are becoming more aware of that emotional piece of education and making sure that people feel supported so that they can be successful. And even just that kind of warm language piece of it, and being welcoming, and just indicating, like, “Hey, how are you doing? I really do care about what’s going on with you.” And having those chit chat moments sometimes even in a synchronous online class, open up that discussion and help students feel like they’re part of the community and really help address some of those issues that students are facing.
John: And I think a lot of the discussion is how can we build this class community when we move away from a physical classroom. So there have been many discussions, and many productive discussions, on ways of building this class community and helping to maintain instructor presence in asynchronous classes, as well as helping to maintain human connections when we’re all distanced, somehow.
Rebecca: I think that also points out the nature of some of our in-person classes and the assumptions that we made, that there were human connections being made in class when maybe they weren’t, or maybe there wasn’t really a community being built, because students may also not know each other there. So I think some of the lessons of feeling isolated maybe themselves, or seeing their students feel isolated, has led faculty to develop and take the time to do more community-building activities. So that there is that support network in place sp that students are able to learn, the more supported they feel, the more confident they feel, the more willing or open they’re going to be to learning and having that growth mindset.
John: And we’re hoping that all these new skills that faculty have acquired, will transition very nicely when we move to a more traditional face-to-face environment in the fall.
Rebecca: …or sometime ever… [LAUGHTER]
John: At some point, yes. [LAUGHTER] But one thing we probably should talk about is something I know we both have experienced is the impact on faculty workloads.
Rebecca: It’s maybe grown just a little, John, I don’t know about you, but there’s some of it that has to do with just working in a different modality than you’re used to. So there’s some startup costs of just learning new techniques. Then there’s also the implementation of using certain kinds of technology that are a little more time consuming to set up than in person. So, the example I was giving to someone the other day was, I might do a whiteboard activity in person that requires me to grab some markers and some sticky notes. That’s my setup. But in an online environment, I need to have that organized and have designated areas for small groups. And I need to have prompts put up. And there’s a lot of structural things that need to be in place for that same activity to happen online, it can happen very seamlessly online, but there’s some time required to set it up. So there’s that. We’ve also all learned how low-stakes is so great, and how scaffolding is so great, but now there’s more grading. And somehow, I think there’s more meetings.
John: Yes, but in terms of that scaffolding, we’re assessing student work more regularly, we’re providing them with more feedback. And also going back to the issue of support materials, many of us are creating new videos. And when I first started teaching, it was very much the norm for people to lecture. And basically, my preparation was going into the cabinet and grabbing a couple of pieces of chalk and going down to the classroom and just discussing the topic, trying to keep it interactive by asking students questions, giving them problems on the board, having them work on them in groups. But I didn’t have to spend a lot of time creating graphs with all the images on my computer. I didn’t have to create these detailed videos and these transcripts and so forth, that I’d share with all my students now. And there’s a lot of fixed costs of moving to this environment, however, we’re doing it. That has taken its toll, I think, on all of us, as well as the emotional stress that we’re all going through during a pandemic.
Rebecca: I know one of the things that I’m concerned about is the ongoing expectation of time commitments that are not sustainable… period.
John: It’s one thing to deal with this during an emergency crisis. But this has been a really long emergency crisis.
Rebecca: And I think we’ve all seen the gains that students have had or felt like it’s worth the time and effort to support students. But it’s also time to think about how to support faculty and staff who have been doing all of that supporting and we need a reprieve… like, winter break wasn’t a break, summer break wasn’t a break, there isn’t a spring break, wellness days weren’t a break. Everybody just needs a vacation.
John: Yeah, I feel like I haven’t had a day off now since the middle of March of 2020.
Rebecca: I think one of the next things we need to be thinking about is: we created a lot of things that we could probably recycle and reuse in our classes, and so there were some costs over the course of the year. But perhaps they’re not costs in the future because we’ve learned some things. There may also be some strategizing that we need to do about when we give feedback or how detailed that feedback is with these scaffolded and smaller assignments so that we can be more efficient with grading. We’ve talked in the past on the podcast about specifications grading and some other strategies and ungrading. So maybe it’s time to think a little more or more deeply about some of these things now that we have them in place. How can we be more efficient with our time and work together to brainstorm ways to save ourselves time and effort and energy and still provide a really good learning environment?
John: Specifications grading is one way of doing it. But having students provide more peer feedback to each other is another really effective way of doing that. We’ve talked about that in several past podcasts, but that is one way of helping to leverage some of that feedback in a way that also enhances student learning. So it’s not just shifting the burden of assessing work to students, it’s actually providing them with really rich learning opportunities that tend to deepen their learning.
Rebecca: I know one strategy that I’ve implemented this semester, that definitely has saved time, although I just need to get more comfortable with my setup, but just I need to practice it, is doing light grading and the idea of having a shortlist of criteria. And then that criteria is either met, its approached or it doesn’t meet. And it’s a simple check box. And essentially, the basic rubric is what it looks like to meet it. And either you’ve met it or you haven’t. And that’s a much more efficient way of…
John:…either you’ve met it, you’ve almost met it, or you haven’t…
Rebecca: Yeah. And so that’s worked pretty well for me this semester. And I think it’s helping me be a little more efficient. And then I say like, “Okay, and ‘A’ is if you have met all of the criteria, ‘B’ is if you’ve met a certain percentage of the criteria, and approach the rest,” that kind of thing. The biggest thing for me is just getting used to my new rubrics and not having to like “Wait, what was that again?” when you go to grade it. But, I think, with practice, next time I go to use them, it’s gonna be a lot faster.
John: Going back to the point you made before, a lot of people have developed a whole series of videos that can be used to support their classes. Those can be used to support a flipped face-to-face class just as nicely as they do in a synchronous course, or a remote synchronous course. So a lot of the materials that faculty have developed, I think, while it won’t lighten the workload of faculty, can provide more support for students in the future without increasing f aculty workload as much as it has, during the sudden transition when people are switching all their classes at once to this new environment we’re facing. I know in the past, when I’ve normally done a major revision of my class, it’s normally one class that I’m doing a major revision on. And then the others will get major revisions at a later semester or a year. But when you try to dramatically change your instruction in all of your classes at once, it’s a tremendous amount of work.
Rebecca: I think another place where we’ve seen a lot of workload increase is also an advisement. There’s a lot of students that are struggling, many more students have questions about what to do if they’re close to failing, whether or not they could withdraw. what it means to leave school or come back to school, we’ve had the pass/fail option. So that raises a lot of questions. There’s a lot of those conversations that certainly we have, but they’re just more of them right now. And I would hope that as the pandemic eventually goes away, then some of that additional advisement will also start to fade away as well. We’re just drained. We imagine that you’re all drained too.
John: We always end these podcasts with the question, “What’s next?”
Rebecca: God, I hope there’s a vacation involved. Our household is dreaming about places we can go, even if it’s just to a different town nearby, as things start to lighten up, just to feel like we’re doing something… anything.
John: The vaccines look promising, and the rollout is accelerating. And we’re hoping that continues. And let’s hope that a year from now we can talk about all the things we’ve learned that has improved our instruction in a more traditional face-to-face environment.
Rebecca: The last thing I want to say is I hope everyone has, at some point, a restful moment in the summer, and we find the next academic year a little more revitalizing.
John: I think we could all use a restful and revitalizing summer to come back refreshed and energized for the fall semester.
John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
Tea for Teaching has been taken over this week by a couple of our favorite authors! Join our friends, Sarah Rose Cavanagh and Josh Eyler, as they interview each other about their current book projects.
Sarah is the author of The Spark of Learning: Energizing Education with the Science of Emotion and of Hivemind: Thinking Alike in a Divided World and numerous scholarly publications. She is the Associate Director for Grants and Research at the D’Amour Center for Teaching Excellence at Assumption College, the Co-Director of the Laboratory for Cognitive and Affective Science, and also Research Affiliate at the Emotion, Brain, and Behavior Laboratory at Tufts University. Josh is the Director of Faculty Development and a lecturer in Writing and Rhetoric at the University of Mississippi. Josh is the author of How Humans Learn: The Science and Stories behind Effective Teaching.
- The tweet containing the graph discussed in the podcast.
- Eyler, J. R. (2018). How Humans Learn: The Science and Stories behind Effective College Teaching. West Virginia University Press.
- Freire, P. (1970). Pedagogy of the oppressed, Herder & Herder.
- Hooks, B. (2014). Teaching to transgress. Routledge.
- Cavanagh, Sarah Rose (2019). The Best (and Worst) Ways to Respond to Student Anxiety. The Chronicle of Higher Education. May 5.
- Yeager, D., Walton, G., & Cohen, G. L. (2013). Addressing achievement gaps with psychological interventions. Phi Delta Kappan, 94(5), 62-65.
- Cavanagh, S. R. (2016). The Spark of Learning: Energizing the College Classroom with the Science of Emotion. West Virginia University Press.
- Cavanagh, S. R. (2019). Hivemind: The New Science of Tribalism in our Divided World. Grand Central Publishing.
John: Tea for Teaching has been taken over this week by a couple of our favorite authors! Join our friends, Sarah and Josh, as they interview each other about their current book projects.
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.
Sarah: Hello, Josh.
Josh: Hello, Sarah.
Sarah: So, I’m Sarah Rose Cavanagh. I work at Assumption University, where I am an Associate Professor of Psychology and also Associate Director of the D’Amour Center for Teaching Excellence. And since we’re here on Tea for Teaching, I will tell you that I am drinking water, because I have had an almost toxic amount of coffee already today.
Josh: And I am Josh Eyler, the Director of Faculty Development and Faculty in the Department of Writing and Rhetoric at the University of Mississippi. And I always feel as if I disappoint John and Rebecca on this podcast, because I’m drinking water as well. I never drink tea. [LAUGHTER] And we’re here for, I won’t say a special episode of the podcast, but it is a different episode, and we’re hoping not to crash it. So Sarah, do you want to tell everyone what we’re doing here?
Sarah: Absolutely. So, we both announced on Twitter that we have new book projects. And I wanted to interview Josh for his new book project. And he said maybe we could interview each other. And then John said, “Hey, you want a space to do that, publicly?” And so here we are interviewing each other about our new book projects on Tea for Teaching. We’ve taken over the podcast.
Josh: Yes, we have. And it’s my go round to ask the first question. But, I just want to say first, it’s always a lot of fun to talk with Sarah about her work, and so it’s a special honor today to hear about her new work. I’m working on two projects, a book on grades, which will be making up the bulk of the questions, but a smaller project on test anxiety and other kinds of academic anxieties. And so I have a question about that as well. And that’s the one that I’d actually like to start with. And so, Sarah, I’d like to ask you about the attributes of class climate that can lead to anxiousness. I know that you’re working on a project that has to do with our learning environments, and its connections to student wellbeing. And so I’m curious on your take, both from your own perspective, and from the research that you’re doing. What are some of the elements of class climates that can lead to anxiousness in students?
Sarah: Awesome, and thank you for that question. I think that our projects are very related, which isn’t surprising. We also are working together on a grant project that, in part, tackles the question of emotions and grading. And so, for me, anxiety is all about uncertainty. And when we’re unsure of our footing, when we think “what if this terrible thing happens?” when we can’t picture the future? these are conditions under which anxiety arises. So, in terms of classroom climate, the sorts of things that research suggests make students more anxious have to do with uncertainty and with a lack of clarity. And so when students don’t understand the material, when they feel like the material is presented in such a way that they cannot understand it, when there’s clarity and uncertainty there, that makes them feel anxious. When they’re not quite sure where they should be investing their time, and what they should be doing with their time and how their grade is going to be determined, that can also lead to anxiety. And when they are unsure of how they’ll be assessed, what a good result looks like, what the instructor is looking for… and every instructor grades differently, assesses differently, is looking for slightly different things. And I always felt as a student, that part of beginning a class was figuring out what does this instructor want? And what are they looking for? And so all of these things can increase anxiety. I think that climates that reduce anxiety are ones that help students feel safe, help them feel like they belong, and ones that are really transparent and clear. And one of the things I’m trying to tackle in my new project is that we need students to be able to grapple with uncertainty. And I think that we need learning environments in which students feel safe, and they feel like they belong, so that they can tackle uncertainty and so that they can be a little unsteady and a little unsure, without anxiety spiking.
Josh: Great, thank you so much.
Sarah: Alright. So you recently shared on Twitter a graph, and I’m going to try to describe it, [LAUGHTER] because we’re all audio, but it was about grading alternatives. And you had two axes crossing each other. So, if the audience can picture four different quadrants, one of the axes was liberation, from not liberating at all to very liberating… a grading scheme. And then the other axis had to do with logistics, fewer to more logistics. So, various grading schemes were in the different quadrants. We can think of specifications or mastery grading, which have lots of logistics. We can think of ungrading, which has fewer logistics. So, I have a couple questions about that. But, my first question is, what is your operational definition of liberation or liberating? What are its characteristics? Its defining attributes? How do we measure it? This is a psychology question. [LAUGHTER] What is ”liberation”?
Josh: Right, definitely. And thank you for that question. I should say that this book is kind of a first step in a new development for the way I’m taking my writing. It’s different from the last book, in that it doesn’t focus solely on higher ed, but looks at educational systems more broadly, from preschool to grad school, it’s the way I’ve kind of been describing it. And so that’s important for the kind of responses I’ll be giving, I think. Now, about the graph… Everyone loves when people in the humanities delves into graphs [LAUGHTER] and math-like things. So, this should be interesting. To answer your specific question, though, much of my philosophy about higher education is rooted in work on critical pedagogy, in particular, the work of Paulo Freire and bell hooks, both of whom talk about education as a practice of freedom. And for Freire, who was deeply invested in the political climate of Brazil, education as a practice of freedom means giving students agency, empowerment, and the tools, to be able to remake the world that both benefits them, but society at large, as well. So, the liberation there comes from freeing the classroom of the kind of controlling elements that prevent that kind of learning taking place. For hooks (her writing is, as far as I’m concerned, the best education writing that has been done over the last number of decades), she takes it in a slightly different direction. And for her, the liberatory practice, education as a practice of freedom means freeing the student to be able to remake themselves and carve out a meaningful life for themselves. In order to do that, a classroom has to break down some of the hierarchies, it has to give over control to the students equally, and importantly, for bell hooks, it also means that that is a very difficult process for students, and it requires them to be vulnerable. And if we’re going to ask students to be vulnerable, the faculty member has to be vulnerable as well, or be willing to be vulnerable. And for me, in the context of grading, what that means is wrestling with the traditions that we have been handed about what education means and what teaching means. And when I refer to some of these grading practices as being more liberatory, what I mean is that by removing the emphasis on evaluation, and placing the focus on feedback, it gives over some of that control, some of that agency and empowerment, to students. So, that’s the direction that I’m moving in, and how I conceptualize it in terms of mapping the practices. Now, when I posted that on social media, it was an honest and open invitation for discussion, because where they are on that grid for me is not set in stone. In fact, I’ve had really interesting conversations about some folks who do portfolio grading… has lots of logistics. And so that was an interesting conversation. Ungrading, for me to make sense of it in my head, take some logistics to help the students feel comfortable with the process. But then I know others who have a lot of success with fewer logistics with that. But, that aside, what I’m really looking at… how much agency does the grading scheme allow students to take over for themselves?
Sarah: Lovely. Thank you.
Josh: And now, the rest of the questions are going to be kind of rooted in that project, the book about grades. And so I’m really excited to hear what you had to say about these. And the first one is kind of more of a personal question, reflecting on some of your own responses to the grades that you got when you were a student, Sarah, and how that might have an impact on the grading models you use now, as a faculty member.
Sarah: Well, I don’t think you’re gonna like my answer, [LAUGHTER] unfortunately. But I loved being graded. It was my favorite thing. So I, in middle school, and in the beginning of high school, I was always intellectually curious, but I wasn’t super into school, I would rather be buried in my books reading. And I got kind of As and Bs and occasional C in junior high. And then I started in high school, and I started racking up A’s and I was like, “Oh, I’m good at this.” So, I started getting more and more invested in my studies and more invested in grading. And I actually ended up valedictorian of my high school class. I also particularly loved… and I’ll have a question related to this later on… I loved taking exams. It’s actually the thing that I miss the most about being in college, because it was a very mastery motivated sort of thing. It was like me versus this exam. People talk about flow, and they talk about the zone, and taking an exam is where I experienced that the most. And so I had this almost gamification approach to grades. I was going to figure things out. I was going to master this thing. And I had a lot of fun with it. And then when I got to college, I was sort of in this, you know, well, I’m not going to not get an A in the class, because now I’m someone who gets As and always gets As. And that, and I’ve written about this for the Chronicle of Higher Education and writing about this in my own book on student mental health, that was part of what helped me getting over my own personal struggles with anxiety, because I wasn’t anxious about grades, and I wasn’t anxious about exams, but I was anxious for social reasons. And I was anxious about participation. And I had a really hard time participating. And I managed to get through most of my college degree without participating in class and still getting As because even though professors would say you have to participate to get an A, they would still, based on exam scores and papers, give out As. But then I started picking up women’s studies classes and these women’s studies classes had a third of the grade was participation, because it was all about discussion and all about sharing and really grappling with lived experiences. And I was kind of confronted with this, you know, am I going to not get an A? Or am I going to try to do something about my social anxiety? So, the grades were actually the motivator that helped me, not just with my intellectual journey, but also with my mental health journey. And I’m not saying that that would happen to everyone, or that my anxiety was as severe as some people’s but for me, personally, grades were fun, and they were also this motivator that had this huge effect on my life.
Josh: Great. And what you said about, particularly your approach to grades in high school, really resonated with me. I was an athlete in high school and college and really approached it in much the same way, that it was just one more competitor, one more opponent on the wrestling path…
Josh: …to be conquered. But, reflecting back on it thinking, Okay, well, that obviously, there was an element of success there. But do I remember much about what I was learning in those classes? Thank you for that answer. I appreciate it.
Sarah: Well, you’re welcome. And I’m going to return to your graph, if that’s okay. [LAUGHTER]
Sarah: So, we talked about liberation. And thank you for that beautiful answer. And I love thinking about student agency and autonomy and motivation. But, my second question is about the logistics-the other axis- and this is, in my own thinking about my own grading, something that I struggle with. And so I absolutely am in agreement, that agency and autonomy, and giving students that control, and releasing your own status are all really important. But, logistics… I think of some of these grading schemes can get kind of Byzantine and convoluted, and I have experimented with all sorts of different schemes, and I know we’re going to talk about what we’ve done in the classroom ourselves. And I think students sometimes find those really confusing, and it reduces certainty, and it reduces predictability and transparency sometimes, and they’re trying to navigate it, and they’re really struggling. And I think that that can increase anxiety more than a traditional grade structure sometimes. But then, on the other hand, the ones that are low in logistics, I have experienced student feedback where they just feel like they’re left at sea, and they don’t know where they are in the grading scheme. And most of us are teaching in institutions where, at the end of the semester, no matter what you’ve done, you have to put a grade in the system. So, certainly, it seems like liberation is good to you – more liberation is better than less liberation. How about logistics? More better, less better? Sort of your thoughts and curve?
Josh: Yeah, I think, in the way, I was envisioning that, logistics are kind of neutral. If you think about that y-axis, the liberation as being the philosophical access, the logistics is the pragmatic axis. And so, what I’m really trying to do with that particular graphic or image is to help people make decisions about what they might want to try. And, as you’re saying, for both the students and the faculty, a scheme, a model that has more logistics could be intimidating, or it could be a deciding factor. Okay, yes, I want to move in the direction of having a more progressive grading model, but doing something that is going to completely change the way I’ve approached my work for the last 20 years, and pile on top of it all these other details to think about, may just be too much. So, maybe rather than going to mastery grading, maybe I’ll start with contract grading, which can have many fewer logistics, and see how that goes. See how it aligns with my own goals and values, and pair that with how the students respond to it. In trying a number of different models, one thing is consistent and that’s what you’re saying. The students uncomfortability with changing the way they interact with grades, the way they respond to grades. So, even in the portfolio-based grading system, which to me is the baby step out of the traditional grading model, where you’re giving mostly feedback but a traditional grade at the end. I mean, the grade happens to be based somewhat on improvement over time, but still fairly traditional. Even then, there’s a lot of groundwork with students, building up their level of tolerance and comfort for that. So, yes, any shift toward a grading model that privileges feedback over evaluation is going to increase nervousness and anxiety at some level with students. And so one thing I talked about with faculty all the time is there has to be a lot of work upfront, being very transparent with students and working through this particular process. And to be honest, I don’t blame the students, because they’ve had at least 12 years of being conditioned in an educational environment where grades have meant something very specific to them. So, logistics for me, neither good nor bad, but an important factor in making decisions, both for our own workload and the student response. And I think it’s worth taking that into consideration. Because, it really matters. If we choose something right out of the gate that has a ton of logistics, and we’re trying to navigate those and help our students, it may actually discourage some people from trying out or continuing down that avenue. And that’s not what we want. And so I have known folks who have experimented, gotten frustrated, and just gone back to the grading schemes that seemed easier for them. So, that, I think is where logistics really come into this equation, both on the faculty and the student side.
Sarah: Thank you.
Josh: So, we talked about your own response to grades as a student, but what about the models that you now use as a faculty member?
Sarah: So I’ve experimented with various parts of your quadrants. And so I’ve done something in one of my classes, not purely mastery, or specifications, grading, but getting there, where there’s a lot of options, and everything is evaluated, not on a numerical score or a letter score, but instead, developing competence, achieved competence, that sort of thing. And so I’ve done kind of more high logistics versions. And then, in one of my seminar classes, which is all seniors, 15 students, and three-hour class, we’re sitting around talking about peer-reviewed neuroscience articles, I’ve taken a more purely ungrading approach. And the thing that I think I’ve decided about all of this, for at least at my institution with the courses that I teach, is that some of these techniques don’t work as well with the lower-level classes that have more content, that are more introductory, and that have younger students in them. In those classes, the students have a much more negative reaction to these untraditional schemes. They have many questions, they get frustrated. And I also, and this will lead into a question that I have for you, our students, most of them work a lot. So they’re working a lot of hours. A lot of them are commuting, they have very complicated lives. And they are not super grade conscious compared to other places that I’ve taught. They’re more focused on getting through the degree program, getting their credits, they are not “I need that A,” for the most part. And they seem to, in particular, dislike some of these grading schemes, because they want to know where they’re at. I had a student come to me, I always give a mid-semester check in and say, “Here’s where I think you’re at, where do you think you’re at?” We talk about it… but we were right before that. And she was kind of lost. And she said, “I’m working 35 hours a week. I’m taking care of my little brother. I’m taking five classes, and I just want a B in all my classes. And I have this limited pool of time. And I need to know where I should be putting my time, and I don’t know where I’m at, do I have an A minus? Do I have a C plus, I want to know if I’m in that (upper) range, because if I am I’m going to dedicate more of my time elsewhere.” And so, that was a struggle. That said, my upper-level class, I absolutely love it. So, in that seminar class, I don’t grade anything. I give them constant feedback on everything that they submit. I have tons of rubrics, and I give them lots of feedback on all different aspects. So, we’re constantly back and forth, back and forth. But we check in once in mid-semester. I ask them where they think they’re at for a grade. And it almost always matches where I think they’re at. And it just takes grading completely off the table, and we’re just talking about neuroscience, and it’s really about the intellectual discovery, and it’s not at all about evaluation. Interestingly, these students because they’re neuroscience majors, for the most part, and their seniors, they are more grade focused, but they’re more motivated and we just chat and it’s lovely.
Josh: Wow, that’s exactly what you hope for.
Sarah: Yes, [LAUGHTER] yes. I will never grade in that class again. Ever.
Josh: That’s great.
Sarah: And so I guess that leads me into this question that I have for you. So, that story of the student who just wanted to know where to divert her time and her efforts, and then my own experience with grades being sort of fun, and then also a motivation that helps me tackle some personal challenges. And then also, I think, especially in the lower-level classes, there’s a lot that is sort of just boring, rote stuff that we have to get through. And that students need that information in order to get up to those upper-level classes, and to have the foundational knowledge where they can start thinking critically, and they can start being more creative. And I think for that sort of just getting the basics down, sometimes extrinsic motivators are really valuable. And I think that the data for motivation research shows that, for creativity and critical thinking and things, we need intrinsic motivation, but for just mechanical kind of rote things, extrinsic motivation goes a long way. And so my question for you is, given all of those elements – are grades always eeevvil? [LAUGHTER] Or can they sometimes key students in to what a professor values, you know, put more effort here less effort here, motivate some of this rote learning and encourage students to face some challenges?
Josh: That’s such a good question because, I think, too often, when we begin to have this discussion in any kind of group of faculty, it’s been my experience that some people hear this as a kind of confrontation or an accusation. You give grades, grades are evil, LAUGHTER] therefore, you must be evil, which is not at all what we’re saying. What we’re trying to find here are the best approaches to help our students to learn in a meaningful way. So, I want to not flip the question, but change it just a little bit in that, for those goals that we have that are more rooted in knowledge building and rote memorization, and I agree that having some element of an extrinsic motivator can get you out of bed and into the classroom and paying attention and focus on the reason why you’re there and why you need to do well. So, I agree at that level. The question for me, especially in those lower-level courses, is what does a grade communicate? And I’m drawn to the work on inclusive pedagogy and on opportunity gaps, and what grades communicate to students who are coming to our colleges and universities from under-resourced schools where they have not had the same kind of educational opportunities. For example, to have AP classes that would prepare them, to have teachers who were invested in moving beyond just what was on the page, to have the right books, to have the right materials. And so, certainly for a subset of students from well resourced educational backgrounds, a grade in an introductory level course, could be a communicator of this information as important, pay attention to it. The grade communicates that you have mastered it well. For another subset of students from less resourced schools. I think that what the research shows is that more often than not, the grade is penalizing them for what they didn’t have, rather than being able to demonstrate what they know and can do at that point in their career. Now, there’s time over four years to bridge that opportunity gap to get them to a different place. So, what I think about this question, ultimately, in thinking through what we were just talking about with grading models, is that there’s a happy medium, where we could take an approach in introductory level classes with a mastery-based grading scheme, where knowledge standards are an important component of that grade. So, you have a whole mastery based grading, you determine the standard and the points along the way that show that students have mastered that goal. And for an intro psych course, a significant subset of those standards could be focused on knowledge and the information. And if they don’t know the difference between extrinsic and intrinsic motivation, then they haven’t checked that box. They haven’t moved their way up to mastering that particular standard. So, for me, that’s a scheme and I was just addressing this question with folks in STEM, who were talking about this very thing. People need to know information before they go out and become doctors and pharmacists and other fields. [LAUGHTER] And that’s a scheme that allows you to accomplish that goal, and to account for the educational opportunities and backgrounds of students who are coming into those introductory level classes.
Sarah: And I think part of that response makes me think that, and I know we don’t have time to unpack all of this, but the relationship between assessment and grading, and so your response makes me think more about “What kind of assessments are you using?” And of course, it’s critically tied together with grading. But, I just finished intro psych this semester, and I taught it all online. And I had previously had exams, and they were in class. And so they were by nature timed, and students had to memorize things. And because we flipped to the online environment, and I was not about to use one of those proctoring softwares, they were weekly quizzes that they had the whole day to do… very similar material. But, because it was open book and untimed, I had higher expectations. And I just calculated my grades this morning, and I had the same grade point average as last fall. And so I think that we can probably still grade, but we can make the assessing process more equitable, I think, while still using traditional grading schemes at times.
Josh: That’s a really great point. And the mastery based schemes ultimately give what look like traditional grades. But, it’s the assessments… exactly what you’re talking about…that look very different from traditional courses. You asked about extrinsic motivation. And I’m going to flip it to intrinsic motivation. [LAUGHTER] And I have some amazing colleagues in the psychology department here, who came to a recent talk that I gave on this subject. And I kind of opened with the classic research that shows that grades are an extrinsic motivator, and will impede intrinsic motivation. And for the sake of time, I was kind of glossing over some important distinctions, maybe, but they gave me some really great feedback that I’ve taken to heart and sort of built into subsequent iterations of this, which is that, “okay, but just because you get rid of grades does not mean that magically students are now intrinsically motivated,” because, as you were just talking about, some of the work in introductory level courses is really, really difficult. So, for example, they said, “You might really want to learn how to play the piano, but at the beginning, it’s really difficult.” And so, the ultimate point here, is that in order to really capitalize on the opportunities presented by decreasing the emphasis on grades for increasing intrinsic motivation, you have to have good scaffolding in there, a teacher has to come in and cultivate the intrinsic motivation, it doesn’t magically appear. So, first I wondered if you agreed with that, and second, what are some ways that we can do that work? What are some ways that we can cultivate the intrinsic motivation?
Sarah: Absolutely. And I do agree. And I think that some of the ways that cultivate intrinsic motivation is to really demonstrate how these building blocks get you to really interesting places. LAUGHTER] And one example I had was in one of my first speaking engagements. And someone there shared the story that they were in a nursing program, and some of their introductory courses were very difficult and very rote. And the students were a long way from being nurses, and that they were having trouble motivating their students. And what they did, which I thought was wonderful, is they had their students who were in internship or placements. And in their senior year, or right after their senior year, they had them come back to their class and do a whole class on specifically what they were doing now, which was very interesting, working with patients and all the things that the students were looking forward to. But then they shared explicitly how they were using the information that they were learning in these introductory classes. And so this very real life demonstration of “Yes, this is kind of a slog right now, and I know it’s a struggle. But here are the ways that mastering this content is going to get you to this place.” And I think we talk a lot about various forms of representation. But, like having someone that you can visualize, “Oh, this is me in four years, and here’s how they are using this material in pursuing the goal that I want to pursue…” I thought it was a really beautiful example of doing some of that scaffolding. And even if you don’t have access to that sort of setup, I think that ways of not just staying in the simple foundational knowledge, but showing how “Oh, this is how this informs this current controversy, or this informs these decisions that are being made in the real world,” I think, are ways to demonstrate the intrinsically interesting aspects of this foundational knowledge.
Sarah: So, as you know, my new project is all focused on the best learning environments for student mental health, in particular. And you have one very successful book on how human beings learn and are working on these new projects on anxiety and grading. You direct your Center for Teaching Excellence and you get talks all over the country. So you are an ideal person to ask: “What do you see as the best learning environment to both help students learn and enhance their well being and mental health?”
Josh: I can’t wait to read this new book that you are working on. I think it’s gonna be great. So, what I keep coming back to the area of research that you and I both really love and keep coming back to, and that is emotion, and the environments that I see my own classrooms, my own university, but many places elsewhere, always come back to care and compassion. That over and over again, the learning environments that are most successful for students are those where they feel as if the instructor cares about their learner. And that can look wildly different depending on who the faculty member is. But the student response tends to be the same. I felt invested, because it was clear that the instructor really cared that I succeeded in this class. And you were just saying… Sarah gave a great talk at the University of Mississippi on Tuesday. And she was saying in that talk, and I thought it was a great point, that whenever you move into this discussion, an automatic reaction is “This is getting into touchy feely territory. Right?” [LAUGHTER] That’s a dominant response. And I understand that response. But that is not what we’re talking about. We’re simply talking about having an environment where students feel respected, where they feel valued, where they feel as if their success matters. And sadly, I think in a lot of learning environments, they don’t feel that. So it’s surprising to them when they are in a class that does allow them to feel those emotions. On the faculty side of it, and this is where I think this automatic response comes from, it doesn’t mean you have to develop friendships, although sometimes down the road that happens, and I still keep in touch with students I had 10 years ago, and they’re doing amazing things. But, that’s not what we’re talking about. You can keep all the professional boundaries in the world that you want, and still show students that you are there to help them succeed. Some of the research that I admire the most in the area of social belonging, Geoffrey Cohen and Gregory Walton have a wonderful paper that showed the amazing benefit of a simple comment at the top of a student’s paper. So one group of students only got targeted feedback, another group got targeted feedback plus a comment at the top that said, “I’m giving you this feedback, because I have high standards, and I have every reason to believe that you can meet those standards.” And that had ripple effects that they traced over years in the success of those students academically over time. And while that may seem extraordinary, the sentiment is not. In practice, that one simple way of communicating to a student that the faculty member is here to help you succeed. I’m not here to hear myself talk, I get tired of that, right? [LAUGHTER] I’m not here for me, I’m here for you. And that is the common denominator that, at least, I have observed, both in practice and in research as being the most beneficial thing for helping students in that way.
I have a question that I just thought of this morning. So, I hope you’re okay with some improvisation. It’s a general question, though. So no preparation needed. It’s in two parts. So, maybe if we take the first part, and then come back to the second one? The type of writing that you’re doing now, Sarah, is not… I wouldn’t say very different, but it isn’t necessarily traditional academic writing. It’s popular scientific writing. It’s for a general educated audience. And so I was wondering if you could just talk about how you made that transition as a writer and some of the strategies that you use, I think the audience may really value hearing about that process for you and the evolution of your time.
Sarah: Well, I had done quite a lot of academic writing. And then my first book… Jim Lang approached me about writing a book for his series, because I had been blogging. So, I guess that predated it. So I’d been doing some blogging, first for a Martha Stewart publication, and then for Psychology Today, and he was looking for people who could write accessibly about cognitive science. And he said to me that most social scientists write like robots. [LAUGHTER] And he said, “I don’t want my books to read like they’re written by robots. And so would you be willing to do this?” And so I think that part is just stopping writing like a robot and picturing your audience. You know, a lot of writing advice talks about picturing your audience and writing for an audience, and I think that that’s true. But I think what really is the answer is that I write in my teaching voice. And the voice that I have, I sometimes don’t like my writing voice, I think gets a little chatty, but I tap into the same part of myself when I teach. And so I think that it’s: how does this material relate to my own personal experiences? to things going on in the world? to real-life phenomena? What are metaphors that I can use? What are anecdotes that I can use to illustrate this? What are ways that I could do this in a way that’s kind of positive and hopeful? And all of these things are things that I try to do when I teach. And Hivemind, my second book, it had a purpose, but a lot of it was: here are the coolest things that I’ve been sharing in my classes for the last 10 years, my favorite neuroscience studies, my favorite psychology studies. And then there’s certainly an overarching framework. So, that was even more clearly my teaching voice in writing form.
Josh: I love one of the ways that Jim talks about writing is “the voice of the colleague down the hall.”
Sarah: Mm hmm.
Josh: And I think that you do that so well. He does that so well. And it’s really accessible. And I think an important development in books about teaching.
Sarah: Well, thank you.
Josh: The second part of this question, though, is a little bit more technical, but I think people might benefit from hearing about it. And that is, the sort of interviews that you and I are doing right now, that you did a lot in Hivemind, that you’re doing in your book, are different from the kinds of interviews that social scientists would do in peer-reviewed publications, because we’re not trying to use the interviews to make a research claim, we’re using them more in an illustrative sort of way. Here’s our point. Here’s someone who’s doing that thing, or thinking about that thing, and illustrating that. And so my kind of technical point that I think is worth thinking about, what is the role of your IRB in doing this kind of writing? Because I know, on my campus, they want me to submit about this project, and talk about the fact that the interviews look more like what a journalist would do than what a social scientist would do. So, that they can say this does not actually qualify as research. It’s just kind of a box checking, so that they have it on file, and they know what’s going on, and that makes a lot of sense to me. But, I’m wondering, kind of from a technical standpoint, what do you have to do at your campus to make this writing work for your school and your career?
Sarah: Well, until recently, I was chair of my IRB. [LAUGHTER] So, I don’t know if that taints my response, but it’s two different things. So, the interviews like this, I’m actually a little surprised to hear that your IRB had you submit. It sounds like an exempt, just checking that it’s exempt from IRB review, which is not IRB review. But, actually, the Common Rule states that journalism and oral history projects and a couple other sorts of categories of doing interviews, they’re not considered research. Research has this technical definition of data gathered in order to contribute to generalizable knowledge. And because things like oral histories and journalism are on specific topics and they’re opinions about things, they are not considered research. And so, on my campus, they don’t go through IRB review. The exception is, so in my new project, unlike my older projects, I am also doing student interviews, and that is actually not even exempt from IRB review. It’s gone under IRB review. And so that part of the project, I did submit for official review, and a number of committee members reviewed it and approved it. And so the student data, and that will be anonymous… The student interviews were considered research. were reviewed by the IRB, even though it’s for kind of a journalistic book project, but the expert interviews that I’m doing, I did not.
Josh: I just think it’s interesting, too, as we think about doing different types of writing, how our campuses see that, and what kind of role they play in that, as well. So thank you.
Sarah: So what’s next, Josh?
Josh: A lot of writing is next, I think. [LAUGHTER] And you know how this is, these are writing projects that require a lot of mapping out ahead of time and squeezing that into campus responsibilities, like workshops, and things like that. But, I’m looking forward to the process. It’s always fun to be kind of starting out on something new. Sarah, what’s next with you?
Sarah: Pretty much the same. [LAUGHTER] I’m going to be doing a lot of interviews. So, people listening to this may be getting little taps on their shoulders. And I am also launching the student interview portion, which I’m excited about, because I’m eager to hear what they think. This is my first qualitative study I’ve ever done. So, it’s going to feed into the book project, but it’s also hopefully going to be a peer-reviewed article. And I’m working on that with my honors student, Jasmin Veerapen, and she and I are presenting at POD together. So, people who go to POD, you can check that out. And I’m eager to see what students have to say about all of this, because we’re going to try to let the data speak to us, let the students speak to us. And so we have hypotheses in our head, but we want to see where they take the conversations.
Josh: That was an amazing conversation.
Sarah: This has been fun. Thank you.
John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.
Do you struggle to get students to complete readings or to deeply discuss readings in an online environment? In this episode, Margaret Schmuhl joins us to discuss how a social annotation tool can engage students in conversations with the text and with each other about the text. Maggie is an Assistant Professor in the Department of Criminal Justice at the State University of New York at Oswego. Maggie has also been working with us as the facilitator for our second cohort of faculty in the ACUE program here at Oswego.
John: Do you struggle to get students to complete readings or to deeply discuss readings in an online environment? In this episode, we discuss how a social annotation tool can engage students in conversations with the text and with each other about the text.
John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted byJohn Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.
Rebecca: Our guest today is Dr. Margaret Schmuhl, an Assistant Professor in the Department of Criminal Justice at the State University of New York at Oswego. Maggie has also been working with us as the facilitator for our second cohort of faculty in the ACUE program here at Oswego. Welcome back, Maggie.
Maggie: Hi. Good to be back.
John: Good to see you, Maggie. Our teas today are:
Maggie: Well, I’m having an orange spice herbal tea.
Rebecca: That sounds nice and warming.
Maggie: It is. It’s very cozy for a cooler fall day.
Rebecca: I have Scottish breakfast, which apparently is my new default tea.
John: I’m having a ginger peach green tea, which I’ve been having a lot recently, too. We’ve invited you here to talk a little bit about how you’ve introduced the social annotation tool, Hypothesis, in your classes this summer and this fall. Could you talk a little bit about what prompted you to adopt Hypothesis for your classes?
Maggie: Yeah, so a couple of reasons. First, in my spring semester classes, I like to think that my students are very open and honest with me, especially when I asked them if they’ve done the readings. And for the most part, I get a very resounding “Nope,” like, “…haven’t done them.” And I just take a deep breath and carry on with the class, knowing that none of them have done any of these readings. And so, after feeling a bit frustrated for a long time with my classes not actually completing the readings that I’ve carefully curated for the class, I was looking for something that could keep them accountable to those readings, I had had colleagues who would assign reading summaries and such, and that seemed great, but I wanted to be able to see something, I wanted to see how they were understanding the reading. And I think John, you actually inspired me to consider Hypothesis because you had used it in some of your classes. So, when we had been meeting with the ACUE cohort, it was super interesting to me. So, I took a workshop, I think through CELT, with the Hypothesis representative, and it seems like a super easy functional tool. And I really liked that it was embedded right into Blackboard. So, it wasn’t necessarily throwing a lot of new technology out at students. And they didn’t have to create accounts, they didn’t have to go to a third-party website to use the annotation tool. It was something that I could throw right into the module, and all they had to do was click on it and start writing. So, it’s simplicity was super accessible, I think, for my classes.
Rebecca: One of the things that I think is really interesting about Hypothesis as a tool is that, if you’re using it for accountability purposes, it ends up being more of a dialogue with the readings rather than what can be perceived as busy work of summaries, or some of these other things that either just feel annoying to do, or annoying to read as a faculty member. And the same thing can happen with quizzing, too. It’s another thing to grade or another thing to look at. Sometimes that can be really effective. But, it’s a nice different way of doing it. And I think it’s really enjoyable as a faculty member to see how students are looking at materials.
Maggie: Yes, absolutely. Because with reading summaries, there’s an easy way out for students just to like look for a summary on the reading. But when you really want them to start asking questions about the reading, this tool helps them be able to locate certain segments of the reading that they may not have understood or something they found particularly interesting, or were able to connect it back to other classes or other information that we’ve talked about in just a really fluid way. So, yeah, I absolutely agree. That’s one of the big benefits I’ve found with this tool.
John: Did this replace some earlier activity? Or was this a new activity that you introduced in your class?
Maggie: So, actually, I haven’t had it replace anything. This has been more of a tool I’ve used, in addition to discussion posts, and so forth. But now that I’ve had a couple of courses under my belt with this, I do think I’m going to move towards replacing discussion posts. The discussion posts, from students’ feedback, they see discussion posts as just answering the questions that their professors wants, whereas the annotation allows them to pull out things that are interesting to them. And they’re able to engage, they think, in a more natural way than it is on the discussion posts. So, they’re reading through each document and, along the way, they see what their classmates are writing and where’s discussion posts, you have to go back into each of their classmate’s forums to see what they have written. And it seems a little more, I guess, artificial in discussion posts to just kind of comment like,”Oh, I agree. Here’s what I also wrote.” And it seems like a much more casual way of interacting that’s more akin to what we have in the classroom when we are face to face.
John: Do you think it encourages deeper and closer reading of the texts?
Maggie: Oh, definitely. I think a lot of my students have given me the feedback that they’re not just skimming the text anymore. They’re not just looking for the main findings or the points to summarize, but they’re actually considering each part of the text. And as they’re considering each part of the text, they’re using this tool to communicate to me their interpretations of the readings, but also the ways it connects back to their own experiences. So yeah, I found it to be quite invaluable for that kind of engagement.
Rebecca: Do you see it as a way to facilitate or to encourage community building around the content?
Maggie: Yeah, so I think that when I do replace discussion posts, there will probably be a little bit more of that. But, I already see where students are using the reply function. So when they create an annotation, they have an option to reply to another classmate’s annotation. And so I see dialogues begin to unfold between three or four students, whereas in discussion posts, if I tell students: “Okay, engage with someone else on their work,” they’ll pick one person, and they’ll respond to them. But again, it almost becomes like a text message to each other. And in a way, it seems, I think, more natural for them to just quickly write back and forth in response to each other’s questions, as opposed to having something a little more drawn out in a discussion post.
John: So how have students reacted to the use of Hypothesis.
Maggie: For the most part, my students have really enjoyed Hypothesis. Of course, there are some students who find it to be a little tedious. But, for the most part, when I asked them, whether they prefer discussion posts or their annotations, most of them prefer the annotations. They felt like they wouldn’t have completed the readings in a systematic way if it weren’t for Hypothesis. So, they’ve pointed to this level of accountability that the tool gives them to those readings, they actually feel like they’re retaining more information from those readings because of the way that they’re engaging with it. When I have a synchronous session, and we are diving into some of the issues that these readings bring up, engagement in those discussions are much greater than they used to be. I used to feel like I had to tailor questions so if they did the reading, or they didn’t do the reading, they could still participate. But, now I feel like we can actually dive into some of the nuances of that text in a way that we just couldn’t do before, when they didn’t do the reading.
John: it’s a whole lot easier, I think, for students to actually read the text when they have to actively be in the text to do their comments. So, it’s a little more difficult for them to evade doing the readings.
Rebecca: One of the things along those same lines that came up in a reading group discussion that we were holding yesterday was the idea of accountability and faculty talking through the concerns that they had about students being held accountable for things and that they seemed less accountable, or that employers have said that recent graduates seem a little less accountable than they had previously. So, it’s interesting to be able to use some of these tools to encourage accountability. But also, I think, it mimics a more professional experience about how you might engage with materials professionally. And so maybe it just feels more authentic, and therefore it’s easier to be more accountable.
Maggie: Oh, I love that, because I do think that, at least in the context of our careers as academics, we use annotation tools like this all the time, whether it’s in Google Docs, and we’re making comments and we’re working with co-authors and other faculty members on different projects and such, I definitely see where we use those tools and those skills that it’s a good skill set to encourage students to build.
John: Since we have this integrated into Blackboard with an LTI, it’s possible to do grading in the LMS. Have you been grading students on their participation?
Maggie: Yeah, so when I first started using Hypothesis in the summer, I was grading them, but at the time, the grading wasn’t embedded right into the Hypothesis platform. And so I was grading on a separate rubric and grading them sort of apart from each other. But now that I’ve been able to use the grading function right within, it makes grading much easier, because I can simply click on the student’s name, all of their annotations, and all of the replies that they’ve given to other students will show up right there so I can review them, give them a grade, move on to the next students, and it automatically loads right into the grading center. And when I’ve talked to students, they actually, not so surprisingly, said that if it wasn’t graded, they probably wouldn’t have done some of those readings. So, it certainly made me feel better by including this as a graded portion of their final grade, because I think without that incentive, they may not have engaged with it. But, I will say that I require students to do a minimum of three annotations, and I’ve several students who are doing 7, 8, 10, 12, just depending on the reading and their topic of that reading. They seem to be willing to move above and beyond that minimum standard, which I think is pretty cool.
John: I’ve seen exactly the same thing, that even though I did have some minimum specified, most of the students were doing2 to 10 times as much as a minimum when they were using Hypothesis.
Rebecca: Perhaps that attests to, in both of your cases, of actually helping students establish a habit of how to read or you get in the habit of using that tool to read and then you’re reading the whole document anyway, so you just annotate the whole thing.
Maggie: Yeah. And I was afraid that, as students were going through the readings, they would basically stop at the first page and put all of their annotations right on that first page, but I haven’t looked at all of their submissions. We do annotations every week on a reading, and so I’d have to pull it all out and compile that data to see what kind of patterns emerged. But, it seems to me that they are doing these annotations throughout the entire reading, they’re not just going a couple pages in and then being finished with it. Of course, there are students who are like that, but when I’m scrolling through that document, and I get to page 17, there’s still annotations there, which I find encouraging.
Rebecca: Probably, once a couple of students do it, and start modeling that, that becomes the standard of behavior, then people realize, like, well, even I’m not gonna read the whole thing, you got to read parts of the thing. [LAUGHTER]
Maggie: Exactly, yeah. And I’ve had a lot of student feedback that they like seeing what their classmates are writing about, because it’s given them insight into their perspectives on the reading and how it connects to their lives and their experiences. And I think it allows for an engagement in an online platform that I typically tend to enjoy in a physical face-to-face classroom.
Rebecca: And reading can seem like a really lonely activity generally. And if it’s difficult reading, it can feel extra lonely, especially if it’s asynchronous. So it seems like a good way to connect people through reading, which is not a way we generally think about being social.
Maggie: Absolutely. I’m teaching a class on the death penalty this semester. And so there are some Supreme Court cases that they are reading, and they are 200 pages long. Now, I required them to read one opinion and one dissent from the respective justices that are writing those cases. But, with that, they’re not so scared with the 200 pages of reading. They’re not just like totally shutting down and not doing it. They’re still engaging with the material, which is more than I can say wa’s happening in the classroom when we were face to face.
Rebecca: Can you talk a little bit about your future plans in using Hypothesis moving forward? Yeah, so another way I’ve used Hypothesis is for peer review, and I know that John has used Hypothesis for peer review too, so I think he probably has some comments as well on this. But, in my classes that are writing intensive, I like to incorporate some kind of a peer review feedback, because not only are we requiring them to write their papers with peer-reviewed research articles, and so forth, but I want them to understand what that process means by engaging in that. And I think when some of that feedback comes from their peers, they start to feel like the feedback isn’t so over their head, that that feedback is something that they can accomplish, and something that they’re perhaps a little less afraid of, than when it comes directly from me. So, what I really like about Hypothesis is I can create all of their submissions into a PDF, I can assign each student to review a certain number of papers, I typically tend to assign each student a particular paper so that not one paper gets all of the annotations over another. And I give them a requirement of making at least 10 or 15 comments on the paper. And before I know it, it’s actually eased up a lot of work on my own feedback, because they’re catching the things that I now don’t have to spend a whole lot of time, telling them to capitalize a particular word or explaining how to use commas in a particular sentence. And so it’s been really nice, because they can simply highlight, they can make little comments about when a sentence doesn’t sound right to them. And then I can come back, overlay my feedback over atop of it. And then the students have all of that in one place when they go to work on their final drafts and incorporate that feedback. So yeah, as far as planning in the future, I do plan to continue using that as a peer review feedback as well as in my readings. I’m teaching online courses next spring as well, and so I plan to go get some good scans of my readings so that I can allow for them to become annotatable. Is that a word? annotatable?
Rebecca: It is now. [LAUGHTER]
John: I had a similar experience when my students did peer reviews, and they really liked that ability. They like that they could see the comments, they could react to each other. They could reply to each other’s comments and sometimes they’d disagree about whether a change should be made and there were some really good discussions embedded right in the text, right at the point where it was occurring. And then I’d come in, and sometimes I’d say, “Well, you know, I think maybe the original actually worked pretty well here” or something similar. And it did make my work quite a bit easier, because the students were doing a lot of the basic editing. Initially, a lot of the comments were primarily grammatical. But after the first time we did that, we talked about how it worked. And the students were saying it would be nicer if we could get more substantive comments, actually suggesting ways in which we could improve the substance of the paper. And I was going to suggest the same thing, but they brought that up themselves. And they seem to have much more of a sense of ownership of the review process. And that worked really well. Did your students have any concerns or negative reactions about the use of Hypothesis?
Maggie: Yeah, so I have found that the tool seems to be much better suited for my upper-division seminar style classes. I think that, even though I find it to be really useful for my introductory survey courses, the students did not like it as much in those introductory courses. But, it’s hard to know exactly why. Some of them pointed to not wanting to actually engage with other people, which I kind of have to laugh and move on from those comments, because that’s part of the process of these courses, is engaging with other people. But, I do wonder if it’s between discussion posts, and low-stakes quizzes, if adding annotations in a lower-division course becomes a bit overwhelming for them. But, I do think that the benefits of being accountable to those readings and having better discussions because of those readings probably outweighs some of that concern. But, I have had some other student feedback. They didn’t like that there wasn’t specific feedback available for the grading function. So, when you grade in Hypothesis, you just give a number grade, it doesn’t allow you to submit a rubric to indicate different levels of content or grammaticals or whatever it is you want to grade in a rubric form. So I did have some students who wished that there was some more specific feedback available for that. But, it did make me wonder, and it kind of reminded me of some of the reading we were doing in our reading group, the Small Teaching Online, when they were talking about specs grading, I thought that these annotations might be a really good place for that… to incorporate some all-or-nothing kind of grading. But again, with low-stakes grading, it’s not a significant portion of their grades. So I guess that’s just one thing to keep in mind, is that sometimes students want some of that detailed feedback. And that tool doesn’t necessarily give you a place to comment on their annotations, except within the annotations, you certainly can comment by replying to their annotations, which I do.
John: But you don’t want to make it public because of FERPA, and so forth. But you always have the option of not using the grading feature within Hypothesis and just adding a column to the gradebook, attaching a rubric to it, and then just evaluating each student… looking at their comments using the same technique, and then just going to the rubric and adding that to the gradebook. So, there are workarounds.
Maggie: Yeah, and I’ve done it that way as well. That just brought to mind like, maybe I need to go back to using that method for some of these classes. The other thing is that sometimes scans aren’t the best. I do think it’s really better to use articles that are already searchable. Sometimes when you’re scanning material, making them searchable and accessible, is difficult. There’s really good scanners and technology that can help us with that. But, sometimes the students are highlighting certain segments of the text, and it’s jumping to other areas of the paragraph. And so I think with that it takes a little bit of time to complete. I’ve also had some students saying that they don’t like to highlight over other students highlights, but I think that’s more of a personal preference. So I just encourage them to reply, then, to those students’ annotations so that it’s about the same material. And that pushes them to engage with each other a bit. But while there’s certainly some areas that students want different features and improvement on, they overall very much like using this tool in Blackboard… at least that’s been my experience.
John: I suppose one nice side effect of this is the more people who use this, the more it will encourage the creation of accessible PDFs because basically the issue is that you need a text layer that contains all the text where it’s supposed to be basically.
Rebecca: Yeah, and if it’s a fully tagged PDF, it works better in Hypothesis than just an OCR’d PDF, for sure.
Maggie: Yeah, that’s fair. I think it is a great tool for faculty because it really does push them to make all of their readings accessible. So, in terms of accessibility, it’s a good way to push everyone to make their materials accessible.
Rebecca: We always wrap up by asking what’s next?
Maggie: Well, so in terms of using Hypothesis, I’m teaching some upper-division seminar courses next semester online, and so I plan to keep using this for both peer review and for reading comprehension. I’m hoping also that one day, we’ll be able to use inclusive access texts with Hypothesis so that we can move through some of the main readings, especially if we have a textbook, where students are able to annotate together.
Rebecca: I would like to be able to annotate images.
Maggie: Right. Yeah.
Rebecca: Well, thanks so much, Maggie for joining us. It’s always a pleasure to chat.
Maggie: Yes. Thank you for having me.
John: Thank you, Maggie. It’s great talking to you.
John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, Dr. Susan Blum joins us to talk about ungrading as a method to support and motivate student learning. Susan is an anthropologist at the University of Notre Dame and the author of several books and articles on higher education. Her newest book, Ungrading: Why Grading Students Undermines Learning and What to do Instead, will be released as part of the West Virginia University Press series on teaching and learning in December, 2020.
- Blum, Susan (2020). Editor. Ungrading: Why Grading Students Undermines Learning (and What to do Instead). West Virginia University Press.
- Pedagogies of Care
- Blum, S. D. (2016). ” I Love Learning; I Hate School”: An Anthropology of College. Cornell University Press.
- Blum, S. D. (2017). “Ungrading.” Inside Higher Ed. November 14.
- Noddings, Nel (2010). Caring in Education. Infed
- Sackstein, S. (2015). Hacking assessment: 10 ways to go gradeless in a traditional grades school. Times 10 Publications.
- Arcidiacono, Peter (2020). Differential Grading Policies. Tea for Teaching podcast, February 26. (the podcast that John referred to that discussed women and underrepresented minoritized groups in STEM classes)
- Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6), 1121.
- A Theory of Public Higher Education
- Society for Values in Higher Education
- School Stories
John: This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, we talk about ungrading as a method to support and motivate student learning.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Our guest today is Dr. Susan Blum. She is an anthropologist at the University of Notre Dame and the author of several books and articles on higher education. Her newest book, Ungrading: Why Grading Students Undermines Learning and What to do Instead, will be released as part of the West Virginia University Press series on teaching and learning in December, 2020. Welcome, Susan.
Susan: Thank you so much. It’s a pleasure to be here.
John: Today’s teas are. Are you drinking tea?
Susan: I am drinking tea. I’m a tea drinker. I love the name of your podcast and I started my day with Mountain Rose Assam tea with milk and sugar. But now I’ve moved to Light of Day Organic Green Jasmine tea from Traverse City, Michigan.
Rebecca: It sounds like a lovely morning.
Susan: It’s as good as we can have during the pandemic.
Rebecca: It looked like you were drinking out of a lovely cup too, actually.
Susan: This is a Chinese made cup with lids that I’ve had for 35 maybe more years and I’m a China specialist by training and when I first went to China, and everybody was drinking out of covered tea cups, I came home and I thought I had to get some myself. So this is chipped and old, but it’s precious. So, thank you for noticing.
John: Very nice.
Rebecca: Yeah, thank you for describing it too. Sometimes we see things… We don’t always communicate all that to our listeners.
John: The visuals don’t translate well on a podcast.
Susan: It’s a white background porcelain mug with blue dragons and clouds and fish.
Rebecca: Yeah, it attracted my attention the second I saw it with your cup earlier. [LAUGHTER] I’m drinking Scottish breakfast tea and I haven’t quite decided what the difference between the breakfast and afternoon is. So I’ll have to report back next time.
John: That’s right. You were drinking Scottish afternoon before. I think the breakfast tea is supposed to be fairly strong. I’m not sure about their afternoon.
Rebecca: I’ll let you know if I can’t sleep. [LAUGHTER]
John: And I’m drinking ginger peach green tea today.
We’ve invited you here to talk about your contribution to the Pedagogies of Care project and your forthcoming book on ungrading. First, could you tell us a little bit about what prompted your interest in upgrading.
Susan: Well, for over a decade, almost two decades now, I’ve been investigating education. And I do that as an anthropologist. So, there have been a lot of dimensions of my inquiries. I began really thinking about plagiarism, which comes in part from work I had done previously on deception. And that comes in part from my own training as a linguistic and cultural and psychological anthropologist. So the plagiarism work made me really wonder what students were doing in school, what their purposes were, how they felt about it, what motivated them, and so forth. And that led to more research on student experience in college and what the purpose of college was. And that led me to really question what we were doing in the classroom and how we were actually meeting students, given what they need or what they want, or what we think they should want, which is a kind of strange conundrum, and how all of this fits into more general ways people grow up and become adults and are socialized into their societies. And so clearly it has to do with issues of social structure and social values and power. And when I think about power, I think about agency and I wonder who has the agency in learning? Is it the students? Is it the teacher? Where are the topics being generated? What is motivating the learning at all? What kinds of ways can we build on innate curiosity and desire to be competent and responsible people in social groups? And how do our pedagogical practices support or even contradict and prevent some of what we actually want? So my more recent book called I Love Learning, I Hate School: an Anthropology of College really explored a lot of the contradictory dimensions of what we claim we want and why those things don’t really work. And students are pretty aware of a lot of these things. So I really explore what I call and others call “the game of school” where if everybody’s going through the motions and the outcome is just a set of points and the learning…. it’s nice if you get it, but you don’t have to, you can sort of cram some thoughts into your head and do well on a multiple choice exam and get the points at the other side. Learning doesn’t happen. Coercion, fear, anxiety, lots of negative things happen. And that seemed to me to be tragic. It’s a waste of time, money, effort, and it doesn’t have to be that way. So I have been engaged for at least a decade in really rethinking my own pedagogical practices from top to bottom. You know, what do I teach? Why do I teach it? How do I teach it? What do I do? What do the students do? What do they do alone? What do they do together? And grading ended up being one of the kind of threads that connect to all these different dimensions of things. And I’ve also been part of a research project to study student learning in an internship. And there were no grades, but there were authentic outcomes of their practices. And so, I’ve been trying to make my classes as authentic as possible, rather than something people are doing simply for performance of competence, but to actually feel competent themselves. So, grades are thought to have three functions: sorting (which I reject), motivating (which we know doesn’t work), and communicating (which also doesn’t work). So I’ve tried to figure out how to make co-operative classrooms where everybody learns as much as they possibly can, for their own purposes, not for me, and I try to have students help generate their own goals so that they see this is not simply a task to be checked off, but as something that matters to them. I mean, I’m spending my life doing this work, and the thought that it’s just something to check off and get out of the way till they can get on to the real important stuff was very galling to me and actually, frankly, almost made me quit, which is kind of the topic of my next book. But the idea that I could actually change something that everybody thinks is central was so liberating to me. And it has really transformed the way I’ve been teaching. And so I’ve really been very pleased with the outcome. And since I published a short piece on Inside Higher Ed in 2017, I found that there are thousands of people at all levels of education, who are engaged in ungrading, throwing out grades, degrading, we call it different things, but we’re all engaged in the same enterprise. So editing this book… I want to be clear. I’m not the author of this book, I’m the editor. I have written the introduction and conclusion and a chapter but about 15 other people have also contributed to this book. And it’s been so gratifying and reassuring and stimulating and refreshing to know that all these other people are engaged in this too from all different directions. So I’m very excited about getting this out into the world so that we can provide some support and reassurance for people who might be interested in doing this but aren’t really sure how to make it happen.
Rebecca: Authentic learning is something that I’ve been really interested in for a long time. And so ungrading has always been really interesting to me, but I haven’t quite gone all the way there yet. And I’m certainly wanting to experiment in that space. Can you describe for folks what on grading look like and how that shifts the focus to learning?
Susan: Sure, and there are ways to do it partially or fully. So I’ve gone to total ungrading until the end of the semester when I am obliged to give a grade for my class. I wish I didn’t have to. I don’t actually think it’s meaningful or informative. But I’m required to do that. So that does happen, and I can tell you a little bit about how that happens, too. But ungrading really means you talk about what people are learning, maybe you have a conversation about what they’ve done well, what they haven’t done well, some things don’t actually have to be graded at all. We don’t have to assess everything. That doesn’t have to be the central activity of our teaching. And there can be what Nell Noddings refers to as free gifts. You can have people have experiences in the classroom, and the outcome is the enjoyment and the learning. And so that is its own reward. And if people perceive that they have been satisfied in their learning, then that’s an assessment. And you don’t have to translate that into some sort of numerical or letter reduction of what is, we hope, a fully human, rich multisensorial experience. I taught a class on food and culture last semester, which is a really fun class to teach. And students did activities, many of which they generated. I didn’t dictate everything. But one of the classes wanted to push one of the topics which was on technology and food. And they wanted to see what tools people use for cooking. So they had this idea that they would take pictures of what was in their kitchen drawer. This was before the pandemic. So take pictures of what’s in your drawer. So we talked about what was in the drawer. And then they had the idea that they would ask somebody older in their family to take a picture of what was in their drawer and talk about it. So then they had the idea that they could interview people about this. And anyway, it was wonderful as an experience. They interviewed their grandmothers and their mothers about what has changed and why do you have this tool? and what is the tool? …and we had so much fun talking about it, and everybody learned everything and it wasn’t graded. It just wasn’t graded. Because who wouldn’t want to do that? And so the motivation was purely intrinsic. And the assessment was when their classmates said, “Wow, that’s so interesting,” or when their grandmother said, “Wow, it sounds like you’re learning interesting things in school.” And so the measure of the outcome was part of the experience. And there was no need or use for anybody else to assess it. So that’s one type of grading is just not grading. You’re learning something, you’re enjoying it, you’re sharing it, and that’s what we’re here for. So there’s no point in doing more than that. But there are other kinds of assessments that are appropriate sometimes, and so for the assignments that are major assignments in my classes, I have the students include with their assignment, a self assessment. And these self assessments used to look a lot like grading, but they don’t anymore. They used to look like: “I did this right. And I did this right. And I had enough sources and I used the proper format. And I did this and that.” And then it was kind of a rubric where you could add things up. Now it’s much more: “What did you learn? What did you do well? What didn’t you do well? Why didn’t you? What do you need help with? What would you do differently? What are you taking with you?” So, it’s a reflection. So, it’s an assessment, but it’s much more of a reflection, which fosters metacognition, which we all say we want. And until this year, I had “adequate,” “not adequate,” and “exceeds expectations” or something which still kind of translates into like F,C and A. Now I just say, you know what you’re doing or you don’t know what you’re doing. And so sometimes in classes where things are new and hard, I teach a linguistic anthropology class where I have students do sometimes very difficult projects: ethnography, conversation, analysis, all kinds of stuff that they’ve never seen before and they admit is difficult. Sometimes they can say “I didn’t do this well.” And because it’s not a grade, there’s nothing at risk for them to admit that they actually haven’t quite felt secure about it. And that’s helpful information for me. It’s very honest, then we can say, “Well, actually, not that you know how to do this. And that’s okay. And we can work at it more, or I don’t expect you to because it takes two decades to master, or whatever it is.” So, then when I return their projects, I reflect on what they’ve done as their project, and sometimes I also reflect on their reflections. So there are a lot of layers here. So, that’s some of what ungrading looks like.
John: Since this relies on intrinsic motivation, what do you do to help build that? I imagine some classes students will come into them with a great deal of intrinsic motivation and in others that they see as just a gen ed requirement or something… a hoop that they perceive as a box they have to check off, which is something that, as you said, always bothers us. How can we perhaps help build that intrinsic motivation in classes when students are there when, as they perceive it, they’re just required to.
Susan: So I teach fundamentals of linguistic anthropology class, which counts as a social science requirement. So I get a lot of students in there for their gen ed requirement. It also counts as something among a set of choices for the major, but it might not be that they’re inherently interested in the topic. I personally think that everybody’s interested in language and communication. And everybody can become interested in anthropology, which is the study of people, but I don’t take for granted that they’re interested in it the way I’m interested in it. So, in that class, in particular, I have spent a lot of time really tweaking every dimension of the class, from the way they sit in the room, to who speaks first every day, to getting to know each other. I try to introduce play and fun. And I have teams and snack teams and students bring in interesting things for themselves. And anyway, this is really my laboratory where I work on how to create experiences that may allow students’ intrinsic motivation to flourish. Because I don’t think I produce intrinsic motivation. I just create conditions for it. So in that class, I now spend a whole week before we even get started just inviting them to ask big questions, to take charge of their learning, to think about what they’re curious about. Sometimes they work in groups that then they have a responsibility to each other. Also the social dimension… sociality, we know is part of it… I spent a lot of time thinking about the emotional and affective dimension of learning. I try to find really interesting things to do and read and try to connect it to their lives. Students are doing a lot of observations of things that are happening around them, which many of them have never done in an analytic or critical way; they’ve only done it in a reactive way. So, I think there are lots of ways to connect students’ own experiences beginning where they are, not with a deficit perspective, but with an asset perspective. You know, what do you know? What do you care about? …and then connect what we’re learning to something that you want to know more about? In that particular class. I have people write linguistic autobiography, and many of them say, “Well, I just speak English. I grew up in America.” And by the end of the semester, when they go back, and they look at that assignment again, they realize “No, there’s actually something to say because I speak this kind of variety…” and there are a lot of things to do that connect to students’ own lives that still get to the material. I’m not shirking my responsibility, but I also think there are lots and lots of ways to get there and they don’t all have to get to the same place. That’s perfectly fine with me. So, those are some ideas.
Rebecca: You talked a little bit at the very beginning about ungrading throughout the semester, but then, at some point, there is some authority that’s requiring a grade. Can you talk a little bit about how to negotiate that?
Susan: this was something I really worried about for years. And then in the summer of 2016, I came across Starr Sackstein’s book Hacking Assessment. She’s a high school teacher, and she has a chapter in our new ungrading book, and she talked about how to go grade free in a conventional school. So, that gave me confidence and ammunition to try to figure this out. So basically, I asked the students to suggest a grade for themselves. I have conferences, I actually try to do the mid semester and semester final to just say like, “If you were going to give yourself a grade, what would it be and why? What’s the evidence?” I’m not that fixated on the grades anymore. grades for me have become such uninformative flat measures of student experience that I find them very maddening. So if I had a student who came in who had never encountered the discipline before, and got very excited and tried some new things and didn’t do that well at those things, but learned a huge amount, to me, that’s a great accomplishment and a great gain, even if their paper wasn’t as good as somebody who’s a senior anthropology major whose paper it’s flawless. I want to say that both of them have had great learning experiences. And if they both say they earned an A, because they learned a lot. I’m actually okay with that. And I know one of the questions people always have is: “What about the engineers who design our bridges? What if the bridges fall down because one person learned a lot but they still don’t know it?” And one of the things we’re really excited about in our ungrading book is we have STEM teachers. So they are talking about what it’s like to teach computer science or math or chemistry and use an ungrading approach. So it can be done in slightly different ways. But for me, because I’m trying to get my students to see the world, reflect on it, analyze the world, that’s what anthropology does. If I get them there, then I am completely happy to give many of my students As. They don’t all ask for As; they don’t all think they’ve earned it. They come in with different standards and experiences about what grades mean anyway. and international students tend to have very, very high expectations for themselves. So they suggest well, modestly, “I only earned a B minus” but I might say that they really demonstrated great learning and accomplishment and it might be harder if they’re not a native speaker of English. So, I may bump it up. I can bump it up or down. They’re suggesting great, but usually, they’re pretty honest. And they learned a lot. They’ve worked hard. And I usually do accept the grade that they suggest.
Rebecca: How do you see the role of reflection and revision as part of the ungrading process? You mentioned handing in an assignment with a reflection, and then you reflect on all of that. What do they do next? Is revision or iteration a part of the practice?
Susan: It depends on the course. I’m teaching a writing course for graduate students again, and revision is obviously the heart of writing. Anybody can revise anything they want in my classes, and I’ve had students say, “I turned this in, but I procrastinated and I couldn’t really get it done, and I’m just not proud of it. And I’ll say, “Would you like to redo it?” And they’ll look at me like “What? What do you mean? I get to redo it. I’m not like branded as a failure my whole life?” No, if you want to redo it, I’m happy to read it again. I try, depending on the course again, to have things build on each other so that even if they’re not literally revising that assignment, they’re recognizing gaps or deficiencies or weaknesses or strengths that they can carry forward to future work that might rely on what we’ve already done. But I have not, at least in this laboratory class that I’m talking about, I haven’t really had one overall semester-long project. I have thought about doing that, and I haven’t done it yet. That could be something I do next spring. If we’re back in pandemic-ville, I may revise things completely again, just because why not?
Rebecca: I’m thinking about ungrading in the realm of, in the design world, doing sprints, so doing one project that builds on it, but having really distinct chunks that you get feedback on and can keep revising all semester. And so it definitely is in that same spirit. So I’ve been wrestling with how to completely implement that.
Susan: Well, I think in a skills-based discipline, there are certainly skills that you need to try and not be good at it first and then get better at. And that’s how we learn anything real. And it seems obvious to me now that punishing people for not knowing at the beginning is the wrong thing to do. So, having only the final product evaluated seems appropriate to me. But I know design… there are some things that people might all agree on, but there are other things that people don’t all agree on. And that’s true of real life. That’s one reason that a single scale of grading is such a distortion of how we really live our lives. People might make a movie and some critics love it, and some critics don’t love it. And to pretend that there’s a uniform single scale is to deny most of our actual experience outside school.
John: One of the things you mentioned in terms of international students is that they often underestimate the quality of their work. You also mentioned in your recording for the Pedagogies of Care project that some underrepresented groups in particular disciplines often experience the same problem. And we had a podcast recording related to that a while back that talked about how women and underrepresented minorities did as well in their introductory STEM classes, but they were more likely to drop out because they didn’t perceive the quality of the work as being sufficiently high. And that served as a major barrier. And I’m wondering how you address the issues of students who undervalue what they’ve learned or underestimate the amount of learning they’ve achieved. When you’re meeting with students and providing feedback and they undervalue their work, how do you address that with them?
Susan: Well, that’s where I’m grading is so perfect, because I can have a conversation. I have these short individual meetings with every student at least twice a semester. And I can say to them, especially if mid semester they say this isn’t very good, because I’m not smart or my grammar’s bad or I didn’t do this before or something, I can say to them face to face, or at least it used to be face to face, I can say, this was extraordinary. I loved what you did, this was such a contribution. So, I can just personally affirm their value and say, you might be focusing on this, but also notice this wonderful thing. And because also, students are constantly interacting in my classes, the students who may have fewer privileges coming in may get a lot of affirmation also from their classmates for their offerings. They may be quiet, they may not be willing to speak, but I try to make people comfortable, at least in small groups or pairs or something, so that they can make their contributions. So, I think it’s less of a problem when people can actually reflect and get comments back. Also, sometimes students exchange papers or exchange work, I tried to have an authentic audience, so that I’m not the sole audience so that people aren’t writing for me, but they’re writing maybe for real people. That’s something I’ve really tried to develop more. I think that when students read each other’s work, they tend, at my school anyway, to be very nice to each other. So they will get some kind of compliment. And I think then in that sense, there’s less of a potential for people to retain this idea that they are somehow deficient. But I also would like to say that schools can’t solve every problem. And teachers and classes, even the best, can’t solve every problem. And so we have broad racism and sexism, and ableism, and all kinds of other things in our society, and one particular teacher might make a difference, but these are really bigger questions that we need to address outside school also. As a professor, my realm is in my classroom, so I can try.
John: At the other side, what about the students who’ve come in who’ve read some material on the topic and have this fluency illusion where they perceive that they’ve learned it very well. I’m thinking of the Dunning-Kruger effect where the people with the lowest level of understanding often overestimate their competence the most. How do you address those issues?
Susan: I’m not so worried about that, really. I’ve had experiences like, that where students think that they’re kind of expert, and they’re not actually, but I don’t really see my role as like cutting down students’ confidence. I think there are enough forces out there trying to do that. So, I don’t really want to jump on the confidence destruction bandwagon, but I like to think that there will be some kind of real consequence where they will say something to somebody who knows more and that person will say,”Yeah, but X” or where they will interact with another student who will know more, and then the student who has this false sense of their own abilities will realize “Wow, I only noticed these three things and Julia noticed 25 things. I guess I have more room to grow as an ethnographer.”
John: This system, though, relies on intrinsic motivation. And you’ve mentioned using authentic assessments, also perhaps, to help build that. Could you describe some examples of authentic assessments that you use?
Susan: Sure. So in this linguistic anthropology class, one of the projects is in groups of three-ish, they have to create some kind of presentation about a particular language. So something they’ve heard of like Hindi, or something they haven’t heard of, like, I don’t know, Tzelta or something like that. And I give them some things they are supposed to include, but the form is completely open. So, I’ve had infographics, I’ve had lots of websites, I’ve had PowerPoint things. And one time, it worked really great… and I’m in this weird classroom that I like with a bunch of screens. The room is imperfect but they’re five screens around the room. And so I happen to have 10 groups that year. So the students plugged in their laptops and the other half of the students circulated and listened to the students as they were talking about their project. And then on these whiteboards that were next to the screens, they were writing praise, questions, suggestions and different kinds of questions. They had to figure out what kind of question it was: Was it a kind of application question? a factual question? or something like that. And the students really loved that project, because everybody saw what everybody did. And the assessment was basically peer generated. It was: “I liked this image.” “This was a really clear presentation,” or “I didn’t really understand what you meant here.” And so that’s assessment. It doesn’t look like assessment. It doesn’t say good and bad, but it’s getting feedback about what you’re doing that you can take with you. So if somebody says, “I couldn’t read the italic font,” next time, presumably they won’t read the italic font. But they’ve had 30, some people responding to their work, which is such excellent feedback, and so much more useful and meaningful than me just sitting there with something and writing a few things.
Rebecca: I like that poster session model idea. It’s a lot of fun and I think students really do respond to peer feedback in that way. I know I’ve been really successful when I’ve done class sessions that are like that poster session or fair-like atmosphere with those same kinds of categories to fill out, I think, is really super helpful. I’ve had really good experiences with that, too.
Susan: One of the things we’re all thinking about is how to translate all this physical stuff online. And you can. Like this past semester, that project ended up online. And so I had a Google Doc, where people were doing the same thing, and it worked fine. It wasn’t as fun as running around the room, but it was effective.
John: I’ve done the same thing the last couple of years in my econometrics class where students create posters, half of them present one day and half the next class day and I give a break, and a group of them can wander around and see the others on the days when they’re presenting, and I’ve invited members of my department. The Dean has come by in the past, and it’s been something they found so much more valuable than the PowerPoint presentations that they used to do, where they’d all be sitting there nervously, and then getting up and being glad to get through it, and then they’d sit there quietly waiting for the rest of them to be done. There’s so much more engagement when they’re up there presenting for the whole class period to anyone who happens to wander by. And it’s a form of a more authentic assessment, I think, that they value quite a bit.
Rebecca: …builds in more practice, too, because they’re talking about it multiple times. [LAUGHTER]
John: Yeah, having them talk for an hour about their project is much more effective than presenting to a silent audience much of the time. I liked it, they liked it, and they strongly encouraged that it continue.
Rebecca: So, we started talking a little bit about how to translate some of these things online. So, why is upgrading maybe particularly important to think about during this pandemic or in this transition to remote learning or the unknowns of the fall semester, as they currently stand? [LAUGHTER]
Susan: That’s a great question. We are in a very unpredictable moment. And every campus is trying to figure out what to do. The ones who are fully online have just made their claim, and so that’s a little bit more predictable at some level. The ones like mine that are committed to in-person except for exceptions or hybrid until we can’t do it anymore.
Rebecca: It’s a familiar story. [LAUGHTER]
Susan: Yeah. And I read everyday about what everybody around the country is doing. And we don’t really know. So anybody who is sticking to a rigid grading scheme is probably going to keep recalculating and recalibrating all semester long… if you’ve got participation, but then people lose their internet connection because they are stranded somewhere, then what do you do? Do you just have a different formula for that person? I think having precision in grading schemes has often been seen as equitable and comforting for students because it gives them security knowing what they’re going to do, but that presumes that you know what the conditions are going to be. I think, even in ordinary times, there are a lot of fallacies built into that and students’ conditions aren’t as uniform as we often assume they are. But, we know now during the pandemic, how widely and wildly variable people’s conditions are, and the New York Times has done stories about one student helping her family with a food truck and the other one is in the family second home in Maine, and there’s everything in between. There’s using the WiFi in the parking lot of the library or there’s using the WiFi in your beautiful six bedroom home. So the lack of uniformity just highlights all of the inequities and all of the unevenness of the conditions. So, if you’re sorting people, but they’re in wildly different conditions, you’re not actually doing a very good controlled experiment, and it’s certainly unequitable. Another dimension we should probably consider is that, in our current moment, everybody is experiencing some kind of stress and trauma. And the trauma-informed pedagogy is something that we all need to learn a lot more about. We know that one of the outcomes of grading in ordinary times (I don’t know what we’re going to end up calling this third condition) is that grades produce stress and pressure. Right now, with so many other stresses and pressures, we don’t really need grades to add to that. How we keep people accountable, how we keep them on track, how we keep them motivated, involved, connected to each other is really our challenge. And that’s what I think those of us who are really thinking about this are trying to spend every minute of the summer trying to figure out. But grading is not the best method for motivating people. So, I think that this is the perfect time to try ungrading.
Rebecca:So if we try ungrading, how would you recommend framing such things in our syllabi?
Susan: Well, I’ll tell you what I do. I have one sentence on my syllabus. My syllabus is not a contract. It’s not one of those punitive sort of legalistic syllabi. So, what we’re figuring out in this conversation is that everything is connected. But my syllabus has one sentence that says, “We will be practicing ungrading in this class, this will be explained.” And I begin most of the semester by having meaningful, enjoyable experiences where people are learning, and I don’t say it’s not graded, it’s just not graded. And then over weeks, I explain what oungrading is, and I show them this is what we’ve done, see how it works. And when I’m lucky, I have students who have had other classes with me and they can sort of support my claims that this is actually meaningful and they won’t just blow it off and they won’t just think it’s not important. I want to have an acknowledgement here before we end though that contingent faculty, graduate students, people of color, young women, people who are tenure track, people who are teaching lots and lots of classes, may not feel that they have the security to engage in something that’s unfamiliar. And it might be risky for some people. They may need to clear it with their chair or their Dean or somebody like that, who may say no, because it’s scary. That’s one of the reasons we’re trying to have this book so that a young contingent faculty member who really cares about pedagogy can say to the person who’s really holding their employment over their head, “Well, there’s research, too. Look at all these people who are doing it, they’ve done it, they’ve done it, okay, they’ve done it for years, and I would like to try it too. Can I try it in one class, maybe with a good outcome?” So, I don’t recommend starting from a completely conventional class last semester to a completely unconventional class online next semester. I think changing things bit by bit is probably the way to go.
Rebecca: I think that’s good solidI think that’s good solid advice, always: iterative practice with our classes. [LAUGHTER]
John: And you mean by that, perhaps, having some activities that are ungraded and then gradually expanding that as you become more comfortable and your department becomes more comfortable with that?.
Susan: I think that’s a great way of framing it. And it depends on the subject too. Some are much more amenable to ungraded, like writing or social science or something.
Rebecca: So we always end by asking: what’s next?
Susan: So, I’m part of a project called A Theory of Public Higher Education. It’s funded by Indiana University and the Society for Values in Higher Education and we are generating a theory of public higher education. We are going to be publishing our kind of manifesto. We’re finishing It this summer and fall, and it should be published next year. We’re very excited about that. It’s a group of six of us from all different institutions teaching all different subjects. It’s really led us to rethink what is higher education from the foundation up. Another project that I’m also really excited about is called School Stories. And I’d love it if your listeners would give it a look. You can find it at schoolstorieslab.com. And it’s basically crowdsourcing experience stories about being in school. So, it can be students, teachers, parents, administrators, it can be from any place in the world, from any level of school. Our only condition is that you have to be 18 to write the story, because otherwise, we get into problems. But we just launched last week, and we have worked on our web design, and we’ve worked on our IRB, and we’ve worked on every dimension of this and we’re really excited about it. There’ll be a new theme every week; this past week, the theme was racism. So what are people’s experiences of racism in school? We have a whole COVID sort of shell and context for what we’re doing now. So, please check that out. And then my next other project is a book I was writing really well until the pandemic hit. It’s about how your education, it’s called Progress Report about my own transformation in teaching, but it’s on hold right now, because I don’t know what to say, exactly. [LAUGHTER] I’m in a profound process of rethinking right now. So, I will write that but I don’t know what it’s going to be now.
Rebecca: It does seem like COVID-19 has transformed us all. We’re just not sure how yet. [LAUGHTER]
Susan: Right? I mean, we’re living through what we all perceive simultaneously as a huge transformation.
Rebecca: Well, thank you so much for sharing some insight into ungrading. It’s been an interesting conversation, and hopefully, it’ll provoke people to think a little bit differently about their plans for the fall and in the future.
John: Yes, thank you. We’ve really enjoyed talking to you. And this is a topic we wanted to get on the podcast for quite a while. So when we saw your contribution to the Pedagogies of Care project, it was an ideal match.
Susan: Well, thank you so much for your great questions and your welcoming demeanor and for your own little contributions to how to think about teaching, which I’ve kind of taken notes on, and to our listeners. Good luck to you and we’ll get through this.
Rebecca: We hope
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.
Is creativity something you value in the work that students produce? In this episode, Natasha Haugnes and Martin Springborg join us to discuss ways to spark, motivate, and support creativity.
Natasha has served in faculty and curriculum development at the Academy of Art University and as an adjunct professor at the California College of the Arts. Martin is the Director of Teaching and Learning at Inver Hills Community College and Dakota County Technical College, Natasha and Martin both contributed to the Pedagogies of Care project and are two co-authors (with Hoag Holmgren) of Meaningful Grading: A Guide for Faculty in the Arts.
- Haugnes, N., Holmgren, H., & Springborg, M. (2018). Meaningful Grading: A Guide for Faculty in the Arts. West Virginia University Press.
- Pedagogies of Care
- Haugnes, N., & Russell, J. L. (2016). Don’t Box Me In: Rubrics for Àrtists and Designers. To Improve the Academy, 35(2), 249-283.
- Haugnes, N., & Russell, J. L. (2008, 2014) “What do Students Think of Rubrics? Summary of survey results: Student Perceptions of Rubric Effectiveness”
- Sawyer, R. K. (2011). Explaining creativity: The science of human innovation. Oxford university press.
- Deci, E. L. (1972). Intrinsic motivation, extrinsic reinforcement, and inequity. Journal of personality and social psychology, 22(1), 113.
- Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education: Reconsidered once again. Review of educational research, 71(1), 1-27.
- Inoue, A. B. (2015). Antiracist writing assessment ecologies: Teaching and assessing writing for a socially just future. WAC Clearinghouse.
- Nilson,. Linda (2019). Specifications Grading. Tea for Teaching podcast. August 21.
- Tharp, Twyla (2006). The Creative Habit: Learn it and Use it for Life. Simon & Schuster
- Transparency in Learning and Teaching (TILT)
John: Is creativity something you value in the work that students produce? In this episode, we discuss ways to spark, motivate, and support creativity.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Our guests today are Natasha Haugnes and Martin Springborg. Natasha has served in faculty and curriculum development at the Academy of Art University and as an adjunct professor at the California College of the Arts. Martin is the Director of Teaching and Learning at Inver Hills Community College and Dakota County Technical College, Natasha and Martin both contributed to the Pedagogies of Care project and are two co-authors (with Hoag Holmgren) of Meaningful Grading: A Guide for Faculty in the Arts. Welcome Natasha and Martin.
Natasha: Good to see you. Yay.
Martin: Thanks for having us.
John: Our teas today are:
Martin: I’m drinking coffee this morning.
Rebecca: Always… Always the rebels.
Natasha: Well, I had my two cups of coffee and now I’m on to Wild Sweet Orange Tea…
Rebecca: That sounds good.
Natasha: … and it’s delicious. Yeah.
Rebecca: I have iced Scottish afternoon tea
Natasha: Afternoon? Huh…
John: And I am drinking Tea Forte Black Currant Tea. We’ve invited you here today to discuss Natasha’s contribution to the Pedagogies of Care project and your joint work on Meaningful Grading in the Arts. Natasha, could you start by telling us a little bit about your contribution to the project?
Natasha: Sure. “Nurturing the ‘aha moment’” is the topic of the video made. And it was based on one of the tips in the meaningful grading book that I co-authored with Martin and Hoag. This video focuses on the “aha moment,” or that moment of insight in the creative process, and how to really nurture students and invite them into that moment. I focused on the “aha moment,” which could also be called the moment of insight in the creative process because it really is associated with kind of joy and happiness and magic. And there are a lot of cultural myths around insight and creativity in general, but especially these magic moments. People think they come out of the blue, that they’re come down from God, that they’re somehow related to some innate ability. And research shows us, and people who are creative practitioners know, that this is not entirely true. So, I just decided to kind of hone in on that moment. In my work at the Academy of Art University, I have worked with a lot of students and a lot of instructors who are often drawn to creative fields because of the joy and they really want to engage in that, the joy of the creative process. But then when the students get to school, and when the new instructors come to teach, they often get really drained. And they find that there’s so much hard work and there’s so much stress in the classrooms, even in things like painting and graphic design and moviemaking classes, students seem to get really rundown, and they don’t connect with those moments of joy. So, this results in frustration. At my own school, we were seeing pretty high dropout rates of students at a certain point and I actually ended up working with at-risk students in my role as the Resource Center Director at the Academy of Art University many years ago, and that taught me a lot about working with students and engaging them in their creative process. A lot of the students I worked with, they were sent to me by an instructor who would say “This student is just not engaging. They’re really sloppy in their work. They’re really lazy. They’re not putting the time in.” And when I talk to those students, I would find patterns that really ultimately meant that they weren’t understanding their creative process. They were doing things like brainstorming a whole bunch of ideas, and then trying to finish one, but then getting distracted and thinking, “Ah, I’m going down the wrong path, I’m going to do this other project, I need to take this other approach…,” and they would go down another path, and then they would abandon that path, and they would take yet another approach and pretty soon it’s time to go to class and the project they’re presenting for critique looks like it was done at the last minute. Again, this is really frustrating for the student and the instructor. And I realized I needed to learn a lot more about the creative process in order to work with these students and help them connect to that joy, help them understand how the hard work connects with the joy, and help the teachers understand how the hard work connects with the joy. I think it’s really imperative that our faculty understand creative process and define it so we can teach it to our students. And this is especially important for students whose livelihood depends on creativity, like a game designer, a graphic designer, even an illustrator can’t just go to work and hope that insight comes, they need to learn how to have some control over that, not only for their own work, but just so that they continue to enjoy what they’re doing.
John: It sounds like part of the problem is that people think that creativity is just something that people either have or don’t have, and they don’t see that it involves a process that includes a lot of work. What types of things can we do to nurture students in making the connection between the work that they do and that aha moment to get them to that point, so that we don’t lose them on the way.
Martin: One thing that I talk about quite frequently with faculty, no matter their discipline, but especially in the creative fields, and one thing that we go back to quite a few times in Meaningful Grading, is rewarding failure and grading process versus grading that final product. If you value the development of a creative process and you value your students diving into the waters that they’re sort of murky, they cannot be afraid to do that. And at the same time, they should also be aware that you’re rewarding that effort and their engagement and what can be kind of a scary process for them, especially if they consider themselves non-artists or unable to do art because they don’t have some innate knowledge of it. So, as you develop grading systems, making sure to work into those grading systems those things that you truly value about that process and about your course.
Natasha: I think it’s really crucial. And something that I try to point out in the videos is breaking it down, scaffolding the process for them, breaking it down into small accomplishable steps and explaining to them: “No, this is not creativity, this is not your whole project. This is what you need to do now. And here’s what you need to do, and you need to put the work in to do it. And then you can move on to the next step.” I think that’s really important, and it’s just really important for the instructors to do that. We often have the overview, we understand the process, we have faith that they’re going to get there, but the students don’t, necessarily, and so that’s kind of what leads to those patterns of procrastination that we see with the students who aren’t doing so well. They put things off, they don’t understand the importance of that early hard work that you really have to just put in in order to get the payoff at the end.
Rebecca: What are some ways that you recommend building in experimentation or risk taking into the grading system? Because those are often things that we value in creative fields, but are harder things or things that we don’t always build into our evaluation systems. We might focus more on the principles of design or something technical, [LAUGHTER] because those are easy to measure.
Natasha: You’re a graphic designer, aren’t you, Rebecca?
Rebecca: I am. [LAUGHTER]
Natasha: I think graphic design is actually a really great example of a place where you can get really bogged down with rules, right? I mean, you can approach graphic design almost as a mathematician and just kind of go “ka-chink, ka-chink, ka-chink, ka-chink, ka-chink” and you can create stuff that follows the rules, but doesn’t really have a lot of creativity to it. And I guess one piece of advice, this goes to a recommendation that I’ve included in the video, but really simplifying criteria. Again, if you can break down the steps and have each step just be assessed on one or two criteria, that allows students to kind of say, “Okay, I’ve met the goal, now I can do what I want. I’ve done what that teacher needs to see, [LAUGHTER] and I’m going to pass, and now I can really play with it.” In some research that I did with a colleague of the Academy of Art University quite a while ago, we did this big study, twice actually, called “Student Perceptions of Rubric Effectiveness.” We found a common pattern in students’ responses, the students that really liked the rubrics said that they liked the rubrics because it told them exactly what they did have to do. And then once they checked off all those boxes, they could just run with it, and that was very freeing to them. We can talk later that a rubric is not always perceived that way, for some students, it kind of acts like a creative constraint. But, I think if we can keep the criteria to a minimum, that can allow students to know what they have to do and then have fun with it.
John: One of the things I noticed in reading through your book, and also in what you were just talking about in terms of giving stories scaffolding, is so much of the advice that you give could apply in pretty much any discipline. While your focus is on the arts, students don’t have the same expertise that we do. And the tasks that they’re facing are much more challenging and require much more processing. And they don’t always come in with that growth mindset. Much of what you’re talking about basically, is how to help students move from this binary view that they’re either good at it or they’re not to recognizing that learning is work, and that they can get better as they develop. And it was nice to see how closely this was aligned to the advice we try to give in so many disciplines.
Natasha: I totally agree, John, and actually I was in a conference at the University of Missouri where they actually viewed this video, and the person who was facilitating the workshop that I was lucky enough to be able to attend from the comfort of my own home office here, she’s a scientist, and she actually put up a map of the scientific method and said, this is the creative process and this is not the exclusive domain of artists and designers by any stretch of the imagination. So, I love having those cross-disciplinary conversations. I actually teach writing and ESL, and so I see some crossover there. I guess I’m just reluctant to offer a lot more advice to teachers of physics and math and economics and things like that, simply because I don’t have as much experience with those instructors. I’ve been exclusively art and design skills for a really long time. Martin, maybe you can speak to that. You have a lot more majors at your colleges.
Martin: Especially in those foundational courses, you’d certainly get students coming in at a variety of levels. So, they have past experiences, or they don’t, and those with past experiences sometimes come in with quite a bit of knowledge or experience in the arts. So, they’ve had a lot of high school experience, for example, that puts them at a different level than the other students in your class that are truly beginners and don’t have any prior experience and consider themselves very much non artists. So, one thing that’s important to do, just getting to the practical here, if you’re in an arts course teaching at that foundational level… or really going back to your comments about this crosses disciplines, no matter what discipline you’re in, if you’re teaching that foundational level course, getting everybody at that same base level at the beginning. Purely speaking from past experience here on this one point, I taught photography for about 20 years. And in my intro courses, I would frequently have students come in that had high school experience, and they had learned something and could demonstrate that thing. But, at the same time, they learned it in a, I’m not going to say the wrong way, but in a bad way. They picked up some poor practices from their previous education in that, and so you have to make an effort to untrain that a little bit and get them to that same process that you want everybody to engage in, at that very beginning level. So, that step and that effort also makes those students who are truly coming in as beginners and don’t have any previous experience realize that “Oh, yeah, this is something that I actually have to learn and that everybody has to learn and these students who come in with previous knowledge, it’s not just some inherent skill or ability that they have in the arts. Another thing that I found really helpful, in sort of leveling the playing field and making it apparent to those truly beginning students, is using my past beginning students who have come into my courses with no experience, using their products as exemplars when I’m talking about how I want somebody to do something. So, if I’m talking about an assignment, I’m using examples from, and I’m pointing out the fact that these students came in from like, say, they’re nursing students or their automotive students, or this student came in with zero knowledge, and this is the thing that they produced, and it’s actually an ideal example of what I want you to produce in this assignment. So, using that, and going back to those examples shows those students who come in as true non-native or true beginners, that that level of achievement is possible.
John: I think that was an interesting point, too, that also shows up in other fields. I know people teaching computer science often will note that it’s much easier to teach people who are true beginners than those who had been self taught or perhaps picked up something in a course, where perhaps not an optimal pathway was given to them. The importance of unlearning things, perhaps, or breaking down the structures that people have and replacing them with stronger structures, can be as much of a barrier as people who are struggling just to get to that initial level. And that I imagine is particularly true in the arts.
Martin: Yeah. And going back to what I mentioned earlier about valuing process, maybe they do produce a product, that’s roughly the same result, like if they come up with the same result, but the process that they engaged in to get to there is so much more complicated and convoluted than what you’re trying to get everybody to engage in. So, they do need to go back and learn process. They do need to be at that same level as everybody else in your course.
John: One of the issues that often comes up in discussing creative fields is the importance of intrinsic motivation. Could you elaborate a bit on how we can help develop intrinsic motivation for students in these fields?
Martin: So, another thing that we talk about or that we bring up in Meaningful Grading frequently is the building of a community in an arts classroom and how important that is. That community is the intrinsic motivator. For example, if you make that a primary goal of yours in a course, you would then grade heavy on participating in that community at the beginning, knowing full well that the goal you have is to make that a more intrinsic reward for students and to back off on the grading or drop it all together, that participation component. So, that they not only learned that after they leave your course and after they leave an arts program that an arts community is vital. Like you can’t develop work in some sort of vacuum. As an artist, you have to be engaging with others, but also within your course, it’s just showing them and it’s creating that intrinsic value. Like, what’s bringing me back to this class day after day is not the grades that I’m getting from my instructor, but the vast resource that I have in these 30 other classmates that are able to give me feedback and support. And that also show me what they’re working on… that give and take. So, that’s one example of building in that intrinsic value.
Natasha: Correct me if I’m wrong, Martin, but a huge part of that community is critique. It’s critique discussions, right?
Martin: Exactly, hours and hours of it.
NATASHSS: …and helping students to understand that just getting that conversation, it doesn’t even have to be feedback, but a conversation, and engaging people to talk about your work does build intrinsic motivation. That’s the big payoff that we’re working towards.
Martin: And if you don’t have that tight community in that class, when you get to the middle or the end of that class, when you really want students to be engaging honestly in critique, it’s going to be like pulling teeth. You have to foster that community so that students feel comfortable, that they can open up, they can give opinions about other’s work, and accept opinions about their own work.
Natasha: I kind of want to get into a little bit of that intrinsic/ extrinsic motivation research. And I guess one of the things that got me into this field, and my obsession with grading and creativity, which people kind of look at me and they say “You talk about grading in art school, shame on you.” But the thing that was so confusing for so many of these at-risk students that I worked with before was they were engaging in those conversations, or they thought they were, with their instructor and their instructor would say things like, “Yeah, you know, you’re doing great, keep going.” And that can mean “Keep going. You got to keep working. ] 3 handclaps] But you’re not there yet.” But the student was hearing it as “Yeah, I’ve done it. Good enough.” Right? And so that student would say, “I got a D+, I don’t understand. Like, what’s going on? The teacher likes me…” or “the teacher said I was doing great.” And so they weren’t able to suss out the actual evaluation in those conversations, especially these new students. So, this is where it is so important to actually have grading systems that align with those conversations and that reflect those conversations. Keith Sawyer, he is like the creativity guru who I follow. He’s amazing. He wrote this book called Explaining Creativity. And there are a couple of pages in this book, Explaining Creativity, where he does essentially a synopsis of all the research on the effects of reward and grading on creativity. And there’s some things that we can look at here that are kind of important… that yes, we can extinguish intrinsic motivation with grades, we can do it by giving As for everyone. We can do it by just throwing grades that are completely unconnected to the actual conversations we’re having in class. And we can do it when we grade students and use a whole lot of really judgmental language and convey that judgment. That will all really decrease intrinsic motivation and creativity. But a lot of that early research on intrinsic-extrinsic motivation goes back to the Edward Deci studies, I believe, and he actually did more work on this later. And there’s a more nuanced conclusion that he came to later that when grades and rewards are perceived as information, when these grades and rewards are based on the quality of work that students are turning in, that can actually enhance creativity, and it can really build intrinsic motivation. But even when you’re using grades well, they shouldn’t be emphasized too much. This is the conversation that I often had in faculty development when I was working with new teachers. Oh, come to class, you’ll get five points. Five Points, that’s not why you come to class. You should not be coming to class to get the five points; you should be coming to class because the conversations are important. That’s why we want you here… and just changing the script in how we talk about grades. You need to have a grading system that has a lot of integrity. But, we should not be banging that over our students’ heads all the time, it should be kind of in the background just running along in the background. And what we communicate to students is the intrinsic rewards of all the work that we’re having them do.
Martin: And that’s why your grading system has to transform a little bit over the course of a semester, going back to that grading heavily on participation at the beginning of the course, where you have to get the students to the course to participate in the beginning for them to realize that there’s value in those conversations. If nobody shows up, they aren’t going to have conversations, but then that can change and it can evolve over the 16 weeks or 10 weeks or whatever length your course is.
Natasha: Yeah, and there are those students who really do care about grades I find in art and design school, there’s a certain subset of students who really don’t care, and that’s fine. And so they’re kind of on their own path, and they’re often doing well. But there are those students who really care and there are the students who are on the verge of failing out of school so they have to care. And I find that just understanding that, instructors need to leverage that knowledge to convince students to do stuff that we want them to do… that we know will do them good anyway, right? So if I say, “Okay, you’re going to be really a grade grubber… you want an A do these things,” and they’re the things that they need to do anyway. It’s a way of kind of tricking them into doing what we want. If you’re grading what’s important in your course, it’s going to work out, it’s going to work out for the students who really care about the grades, for the ones who don’t as much, hopefully, they’ll just be intrinsically motivated to understand why they need to engage. But grading what’s important is really crucial in that, I think.
Rebecca: One of the things we’ve talked a bit about is scaffolding and helping provide structures. So if we were to provide structure for faculty who are thinking about the idea of building a grading system that has the values that we’re talking about, things that really they care about or are important to their class, what are some of the steps you would recommend they go through to actually develop that system so it actually does reflect the values that they want?
Natasha: Well, [LAUGHTER] my answer to that is it’s ultimately working towards a rubric. And again, that can be kind of a bad word. I’m the one who’s been walking around the art and design school for 25 years saying, “Let’s build rubrics. And let’s do normings.” And I had a photography teacher tell me one time “Natasha, you gotta understand when you say “norming” to an artist, I mean, that’s like death, you know?” So I’m like, “Ah, sorry.”
Martin: There’s a reason why we don’t have assessment in the title of our book.
Martin: That was on purpose.
Natasha: It was by design. Absolutely. For the really grade-averse instructors, I start with a conversation. And I usually start with grading because that’s a really good entryway. And I’ll just say, “What are you teaching? And what does that look like? And what does it look like when a student does it? And what does it look like when a student doesn’t do it?” And really, that’s where you start. And then I think the next step is really getting real student work in front of this instructor or this department or this cohort of instructors who are teaching the same course… different sections of the same course. They need to look at the student work and they need to say, “Well, yeah, that one meets the criteria for this course that doesn’t.” Why? Why not? Having those conversations, that’s like the best investment that I think any department or any instructor can make into really focusing their teaching and to improving assessment is just think about how you’re making what you teach visible. And then what does it look like when it’s acceptable and when it’s not acceptable, when it meets the goals and when it doesn’t meet the goals? And then it just moves on from there. And if what you think is important, the quality of the color print in your poster, or the resolution on your screen of your logo, or whatever the heck you’re talking about, it might be process. So again, what does that look like? Well, I want to know that they’re listening to the feedback and really taking it in. Well, how can we make that visible? Maybe I have them do a little recording or do a short paper saying here’s all the feedback I heard, and here’s how I responded. “Joey told me I should change the concept, but I didn’t like that idea because…, so I’m not going with that…” and actually have them make that thought process visible. So, it takes some, again, creativity on the part of the instructor in the field of the teaching and learning. But usually, if there’s something really important that you’re teaching, you can have a way to make it visible and figure out what you’re looking for. And what does it look like when there’s evidence that the student has done what you need them to do? And what does it look like when that evidence is not there yet?
John: I usually meet with new faculty and generally ask them what would they like to see workshops on and, about six or seven years ago, one thing that was requested was a workshop on evaluating creative work. So I reached out and we got four people from different departments. We had someone from art, someone from music, someone from theatre, and someone from English. And they put together a presentation of how they evaluate creative work. And one thing that was in common was they all used rubrics, and they all talked about how there are certain fundamental skills or processes that students have to follow. And that’s what they embed in the rubrics and it surprised a lot of people in STEM fields who were attending because they were much less clear about what they were expecting from students and They expected something that would be much less well defined. And so one of the things they also emphasized, and you’ve talked about is that it’s telling students exactly what they’re expected to do and what types of things they need to demonstrate in the work before they can embellish on that. And that was a really important feature in all of their discussions, the same arguments show up in your book. That surprised many people outside of the creative fields.
Natasha: Oh, those are my tribe. That warms my heart to hear that, John. That’s exciting, yeah.
Martin: One of the added benefits of using rubrics is that time saved as well. Faculty time is a precious commodity. And if you can convince them or just show them how much time will be saved by simply having that rubric available, and using it as a guide, as you’re going through the assignments that are piled on their desk, it’s a convincing argument.
Rebecca: So, we talked a lot about building in values into our evaluation system. Can you talk about some of the things we should avoid doing.
Martin: I can speak to that a little bit. So, one thing that I’ve seen a lot of arts faculty members do… from a student perspective. So, coming up through the arts, one thing I’ve seen a lot of, and heard stories about, is the instructors bringing their personal bias, their own career and background, and that subjectivity in general, to the process of evaluating student work. So I’ve heard some pretty bad horror stories about that. For example, I’ll just go into one story quickly because I think just every faculty member who’s hearing this should know that this is never something that you want to repeat. So all the work, as you can imagine, all the prints, lining the board during critique and the instructor just, without words, just going across the board, pulling work down and throwing it out the window. Like if he doesn’t like it, right… if it doesn’t meet his criteria, which are a mystery, by the way…
NATASHAS: I’ve been in those classrooms. I’ve seen that.
Martin: Tell non-arts people about these stories, and they’re like, “no.” Yeah, it really happened. So remembering that you got to check your personal bias and your personal preference for art at the door and rely a lot on, or more on, having students engage in self evaluation, like did they feel like, and how do they feel like, they have made this, or communicated this, through their work, this issue that they think is important through their work. And if it doesn’t, like if you’re not understanding, then engage in a conversation about it. Like how they feel they’re getting there and where you think they’re not getting there. So using that as a starting point instead of your own, “I am the authority on art, and this is why this does not work.” That’s a huge demotivator.
Rebecca: I think one of those biases that a lot of faculty might bring to the door, is the history of white art created by white individuals.
Martin: This is the history of art, it’s all white male.
Rebecca: If students are creating their work from different cultural perspectives, and the faculty member is not up to speed on other cultural perspectives, we’re enforcing essentially a white supremacist point of view and system. So how do we engage in those moments in a way that’s productive, especially if we don’t understand the cultural background that something is based on?
Martin: Yeah, if students can’t place themselves in the history that you’re talking about, you’re referring to, how are they to imagine themselves in that world in the future?
Natasha: I’m gonna offer just one little tip here because yes, I hear you, Rebecca, and we see it everywhere in the overwhelming influence and sort of self-perpetuation of the white colonialist culture, even in our art classes. Something that we found when we did our rubrics research was that students, in general, really love rubrics, it helps guide their work. But what they really loved… even more than the grid of language… was samples, examples of work, examples of work that span the quality. Here’s an example of something where somebody tried really hard but they didn’t quite hit the mark. Here’s some examples of passing work. Here’s some examples of work that really hits it out of the park. And it’s really important not to have one example, especially in a creative field, because what happens then? The students who are not very competent will copy. Here’s an opportunity to allow for many different interpretations and really show those to your students. Consider using student work from previous semesters from a diverse range of students with diverse content. And that gives students something to connect to, it helps them see themselves in the class, it helps them understand that you, as an instructor, see them and value them. And that even though you have these criteria, there are many ways to reach those goals and reach those marks, those criteria that you’re putting out.
John: And so, by including a range of examples too, from different genres or different approaches, so that it does not become just a Western culture, perhaps. In recent podcasts we’ve done with Kevin Gannon, for example, he talked about decolonizing your syllabus and just suggesting that when you’re putting together your syllabus or searching for examples or exemplars, you could just do a little Google search on decolonize your [insert subject matter here] syllabus, and you can often find some good discussions of that with some good resources that you can build in.
Natasha: This is incredibly important. In my work at California College of the Arts, there’s a very active group of instructors. They’re working on decolonizing the classroom, anti-racism, anti-racist pedagogies, and I’ve learned a lot since I’ve been in teaching there. I haven’t been there for a very long time. But I guess there’s a book called Antiracist Writing Assessment Ecologies: Teaching and Assessing Writing for a Socially Just Future by Asao Inoue. And he speaks quite a lot about assessment. And the point he makes about assessment is he says, in order to really decolonize your classroom, we need to be careful how we talk about quality, because quality so often is really culturally loaded. It’s so loaded that it is really hard for us to even untangle what we see and what we look for. And as a response to that, he really emphasizes grading on labor, grading on the work. And this, again, relates to some of the topics that are in this little video I put together although I don’t really call it this by telling students and taking all that quality judgment away from your rubric and from your assessment and just saying, create 50 of these things, [make 50 taglines, make 50 photographs, write five different thesis statements for your paper or write five different opening lines for your paper and just do that. And that’s the way of just asking for labor. You’re just saying do this work and it doesn’t have to look a certain way or be a certain way. But if you just put some effort into this, you will do well. This is a way of assessing work that actually pans out much better for students of color, students from cultures that are not traditionally represented very well in the faculty at colleges and university. So this is something I’ve been really taking to heart a lot. And in my writing class, I’ve actually, at CCA, where I teach freshmen composition to non-native English speakers, everything is graded on pass not passing yet. And so that really emphasizes the labor. If they’re not passing yet, the implication in that not passing yet grade is that you will do it again. Just do it again. Do it again. Nope, still not quite right, do it again. There have been a few students who have redone their essays four or five times, and it’s painful. But wow, they learn… they learn. And again, the trick is in not having five pages of criteria, but having a pretty narrow band of criteria that we’re looking for here that doesn’t get really niggly about the quality.
John: It sounds like it’s a specification grading system that you’re using. And it’s also building in something much more explicit than the “keep going” message that can be misinterpreted. So giving students the opportunity to try something to not quite get there, but to encourage them to continue working on it more explicitly than perhaps students always hear.
Natasha: And I’m glad you mentioned specifications grading, Linda Nilson has been a huge influence on the way I think about teaching and grading. She’s got a lot of really good thoughts out there for sure.
Rebecca: One of the things that’s really easy to evaluate is something that’s technical that has a right or wrong answer. How do we evaluate in a rubric format, things that are more qualitative, like the amount of experimentation or risk taking or other things that we might value in terms of creativity? Can you give us a concrete example?
Natasha: Actually, we have a a whole tip in our book about risk taking. There’s some really interesting ideas about ways you can really force students into making some mistakes and talking about them. There’s so much that comes up that seems, at first, like it’s going to be really hard to describe it in a rubric. But again, if we just get instructors and people who teach these disciplines together, talking about things, usually they can come up with something much more concrete, even if it’s not a cut and dry technical skill. Concept is one and I have some examples of like before and after for rubric wording. And often when we first write out a rubric, we might use some really sloppy language like “The concept is sloppy. It’s lazy. It just doesn’t work.” That just doesn’t work, right? [LAUGHTER] And so that might be the first draft. But then you start looking at some student work and talk with your colleagues. And you’ll find some more precise language will come out. Often when we talk about concept… I’m talking about the context of maybe an advertising campaign. But the concept is predictable. That’s a concept that is not acceptable is predictable. It’s the first thing that comes to mind when people think of this product. So, that is not a good concept. So there you go. Now we’ve made something a lot more understandable to the students and to the instructors when they’re using this rubric to grade later. And it can help you move forward in a way that that judgmental language won’t. It just makes the students feel bad. It makes the teachers frustrated, because we’re like, “Oh, it just doesn’t work.” But actually taking the time to look again at a range of work that doesn’t meet or that does meet the expectations for this thing that seems really nebulous at first usually you can manage to articulate it, and if you can’t, then maybe that’s not something you’re actually teaching in your class and maybe that’s outside of what you’re assessing. This is another tip that we come up with quite often. I think oftentimes instructors who fear grading, they think that they need to grade the art and you can’t grade art. No, you can’t grade art. You can’t say Picasso was better than Twyla Tharp. You can’t compare people and grade artists in a holistic way. Your grades should be based on what you’re teaching, and the objectives for your class. And we can communicate to our students, this is what we’re looking at here. You’ve also done this other stuff really well, but in our class, we’re really looking at this, so this is what your grade is based on. And that’s a really important factor in this whole endeavor, as well. One other little trap, I think, that faculty members can fall into when we talk about assessing grading or assessing creative work is that when we sit down to write our criteria out often the first thing we want to talk about is that incredible piece that that student two years ago did, it was amazing. It was mind blowing, it was so good and students need to see this and you get into those conversations. And that’s fun to talk about with your colleagues and you pull up that student’s work. And you talk about how great they were and what they’re doing now. Yes, that work should be shared with other students, that’s exciting. We have to celebrate those moments. But for the student in the middle of the pack in your class who’s kind of struggling, we need to think about what’s acceptable. That’s why it’s really important to really focus on that line between what meets expectations and what doesn’t meet expectations, because there are some students that just really need to work on that. [LAUGHTER] There are others that are going to blast through that and do really great things, but the ones that need our help are usually the ones that are hovering around that middle area.
Rebecca: So, we’ve talked a lot about rubrics and grading and evaluation, kind of assuming that we’re living in a perfect little world in some ways. But as we all know, right now, in this moment in time, there’s a lot of extra stress of COVID-19, protests related to Black Lives Matter, and any numerous other health things that are coming up because of COVID-19, remote learning. [LAUGHTER] All of these things, there’s lots going on. And so students are under more stress than normal. Students are often under a lot of stress, but this is like extra stress. So in these moments, what are ways that we can help promote creativity and also help our students really feel supported and being able to learn whether they’re on this point in the spectrum or they’re finding being creative really therapeutic and helpful, and all the way to students who just feel like they’re frozen because there’s so many things going on in the world, they feel like they can’t move forward.
Martin: I think now is a great time to be engaging students in creative process. It’s what gets us unfrozen. I’m speaking purely from my location at a Community and Technical College. If we can get students to engage in those often elective courses outside of their major or area of focus that allow them the opportunity to dive into those things that they are feeling a lot of stress about or anxiety about. It helps students be more successful in those courses that they do have to get through as a matter of course for their program of study.
Natasha: Oh, boy, these are hard times. I think, just most immediately from the video, the nurturing the aha moment, I think that it’s even more important than ever to break down our projects into small steps and help make those steps really kind of distinct from each other. I think that’s something that’s happening for students now, and for us, is we’re sitting and we’re staring at the screen all day long and it can become this big blob of existence where one thing bleeds into the other. And if we can really make the steps a little bit distinct, including a few steps where the students just disengage from all social media and anything online where they can actually be alone, without all of the electronic stimulation. I think those are things that can really help nurture their creativity. And also just I think there’s this funny paradox right now that we’re all alone. We’re all isolated. And yet, if you’re sitting there on your TikTok and Instagram and all day long you’re connected and that can be really, really stressful… and so convincing students to take a break from that, telling them we’re going through another step now. [LAUGHTER] And keeping things again really simple so that they can have that opportunity to use what we’re doing in our classes as a springboard to express themselves. Encourage them to incorporate what’s going on in their own life into the work that we’re doing, including examples and acknowledgments of what’s going on in the world. Really important. And it’s a fine line. I’ve just talked about this with my co teacher about how we’re going to be discussing Black Lives Matter, the latest George Floyd protests, and the Black Lives Matter protests, and the defund the police protests with our students who are mostly from Mainland China. Where do we even begin with that discussion? How do we do that without completely stressing them out, but also using it as an opportunity to feed their curiosity and acknowledge their own stress around these issues? So we need to let them know that we’re a safe space for everybody to engage and really help them break down things into small packages and celebrate their achievements. And again, let them keep working if they’re not quite there yet. Let them do it again. Let them do it again, let them do it again, I found myself being very forgiving on deadlines,
Martin: We also have to help faculty realize that they’re safe to engage in those redesigns and those conversations, and that comes from at that administrative level, engaging this at a college or institutional level. So that you aren’t leaving faculty to figure this out on their own. At my two colleges, for example, we have this new initiative that will run all the way through next year, and actually, for the next three years, probably called Equity by Design. And so we’re starting with a team comprised of administrators, directors, faculty, helping each other understand what this effort is going to be at a college level.
Rebecca: One of the things that you’ve both emphasize is kind of these small steps. And I think a small step for an expert might be different than a small step for a beginner. [LAUGHTER] Can we just take a minute or two to describe the differences between what an expert might think of as a small step and what might be in practice an actual a small step for a student.
Martin: One thing that we have been engaging in at my colleges is the TILT framework of Transparency in Learning and Teaching by Mary-Ann Winkelmes and her team. Mary-Ann came to one of our colleges in January and actually spoke and I’ve been facilitating communities of practice at both colleges on this topic this year. And in that work, there’s a realization as faculty review each other’s assignments and each other’s syllabi that you’re not starting at square one, you’re actually starting at square five, because we have to so often take a step outside of our disciplines to realize that, like you just said… So, what’s complex or complicated to one student is not for another and vice versa. So that transparency effort helps us to really outline the steps of an assignment, even those small steps. And so I’d encourage any family member struggling with whether or not to start at this point or that to review that transparency literature a little bit to engage with their colleagues, share assignments, and ask their colleagues whether or not they’re starting in the right place.
Natasha: That’s such a good question, Rebecca. The expert/novice thing is just something we grapple with all the time as instructors, especially if we’re teaching a new course… something that I’ve had to do in my own class… I was just thrown into a very new course for me a couple of years ago. And we did a new project on public service announcements this last semester, and I start something in class, I told the students “Now, choose a topic from this list of public service announcements that you’re going to create. And first thing you have to do is do some research. So let’s look at some websites.” And by having them do that in class and seeing what they come up with, I start to say, “Oh, right. [LAUGHTER]] They’re going to TikTok, you know, they’re going to these kind of places I didn’t even anticipate, and that allows me to then say, “Okay, I need to actually really scaffold this down.” I don’t want this to take two weeks of my time, I want them to find a credible source and then I ended up giving them a list of basically five places they should look. And you might say that is oversimplifying it, but again, this was just a step in the process of a larger PSA that they needed to make. So I needed to really like clamp that down. But I think if we can have students start in class and actually watch what they do, that gives us a lot of information about how big a step they’re willing to take on. And again, the little creative process chart that I put in the video that I created, I think a lot of creative practitioners, people who are really established, they’ve internalized this process, and they even don’t even want to put it on the line. They’re just like, “Oh, you bounce around, you know, you go back and forth and it’s not a linear thing.” And that’s not actually helpful to a new student who’s really nervous, who’s really stressed, who’s in school for the first time. They’re paying a ton of money to go to art school and their grandparents are really pissed because they should be an accountant. That’s intense. And so these students really need things broken down. And I think that just an awareness of our own expertise is a good starting point, and taking our cues from the students.
Rebecca: This has been really interesting. We always wrap up by asking what’s next? \
Natasha: What’s next, Martin? [LAUGHTER]
Martin: What’s next for me is to finish this book I’m working on with Cassandra Horii. We’ve been doing this project together for the past decade or so. I’ve been making photographs at colleges and universities across the country. We use those photographs that I make in classrooms in faculty teaching to help faculty think about their teaching practice. So we do this form of photo0based teaching consultation. So we’re putting those thousands of thousands of photographs together into a book. And we’re working with the same press that Natasha and I were with, West Virginia University Press, on that book. As far as my other life as an administrator in higher education, what’s next is figuring out what fall semester looks like. How are we engaging students? And in what space are we engaging them? Are courses going to be offered HyFlex, we don’t know? Are any courses going to be conducted face to face? Some of them have to be. You can’t teach arc welding at a distance. There’s some of that that has to be hands on. So figuring out exactly how we’re engaging students in this next phase is what’s next for me.
Natasha: I’m going really micro because these are really big questions. I’m going to keep working on the curriculum for my ESL class. I am now not in faculty development officially anymore at my university in an official role. My current role is that I coordinate and write the curriculum for one level of the English for non-native speakers at the Academy of Art University. And it’s exciting. So I’m working on actually integrating more of the anti-racist ecologies. I’m working on incorporating even more creative process readings and practices into my ESL course in the new zoom world, also really trying to figure out how to get students conversation practice in zoom. That’s the really tough one. So, I’m very much just kind of looking [LAUGHTER] about two feet in front of myself right now. And boy, as far as the bigger issues go, I don’t know. Let’s check in again in the fall. [LAUGHTER]
Rebecca: I think that’s fair.
Natasha: This afternoon, I’m going to make a creative genealogy for myself. I’m making a creative family tree, because I’m having my students do this next week when we start class and I’m going to do it for myself as a sample for them and also just to see what it’s like to go through that process. So that’s actually been really fun. That’s my fun thing that I’m doing.
Rebecca: It’s all about balance.
John: Well, thank you. This has been fascinating. I really enjoyed reading through your book, and I’ve enjoyed your contribution to the Pedagogies of Care, and it’s been really great talking to you. Thank you.
Rebecca: Yeah. Thank you so much.
Natasha: John and Rebecca, it’s been a really fun conversation. Thanks so much for inviting us.
This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, Dr. Jessamyn Neuhaus joins us to discuss the myth of the super teacher and the importance of focusing on self-efficacy, being human, and being reasonable with ourselves and each other. Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessamyn is a recipient of the State University of New York Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: A Guide for Intellectuals, Introverts and Nerds Who Want to be Effective Teachers.
- Neuhaus, Jessamyn (2019). GEEKY PEDAGOGY: A Guide for Intellectuals, Introverts, and Nerds who Want to be Effective Teachers. West Virginia University Press.
- Pedagogies of Care (Sneak Peek) – video trailer – website
- 82. Geeky Pedagogy – Tea for Teaching podcast with Jessmyn Neuhaus. May 22, 2019.
- Neuhaus, Jessamyn (2020). Remote Teaching While Introverted. The Chronicle of Higher Education. April 29.’
- Denial, Cate (2019). A Pedagogy of Kindness. Hybrid Pedagogy, Critical Digital Pedagogy. August 15.
John: This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, we discuss the myth of the super teacher and the importance of focusing on self-efficacy, being human, and being reasonable with ourselves and each other.
Rebecca: This podcast series is hosted by John Kane, an economist&hellip
John: &hellipand Rebecca Mushtare, a graphic designer.
John: Our guest today is Dr. Jessamyn Neuhaus. Jessamyn is the Interim Director of the SUNY Plattsburgh Center for Teaching Excellence and a Professor in the History Department at Plattsburgh. She specializes in the study of pop culture, gender studies, and teaching and learning. Jessamyn is a recipient of the State University of New York Chancellor’s Award for Teaching Excellence. She is also the author of Geeky Pedagogy: A Guide for Intellectuals, Introverts and Nerds Who Want to be Effective Teachers, which she talked about on one of our earlier podcasts. Welcome back, Jessamyn.
Jessamyn: Thank you for having me. I’m really happy to be here.
Rebecca: Today’s teas are:
Jessamyn: I am drinking anything and everything with caffeine all day long, every day since March. [LAUGHTER]
John: And I am drinking a ginger peach green tea which is, I think, my fifth or sixth cup of tea today.
Rebecca: I’ve got the Irish breakfast going today. You notice, my caffeine choices are definitely on the higher end lately, too. [LAUGHTER] The powerhouses of tea.
John: Caffeine has been extremely helpful in the last couple of months. [LAUGHTER]
Rebecca: What’s in your teaching tool belt? Some caffeine. [LAUGHTER]
John: We’ve invited you here today to talk about your contribution to the Pedagogies of Care Project, which we’ve talked about in the last couple of podcasts. Could you tell us a little bit about your contribution to this project?
Jessamyn: Sure. It’s called Pedagogy Nerds Assemble: Battling Big Teaching Myths during Troubled Times. And it’s really about encouraging faculty teaching self-efficacy in the face of so much uncertainty, and trauma, and loss, and struggle now and in the foreseeable future. It also takes the kind of little bit of the snarky tone that I enjoy adding to the scholarship of teaching and learning, kind of real talk to empower faculty to not buy into certain myths that can really interfere with our ability to appreciate our unique ability, our unique contributions to student learning and student success. And mine is in the form of a recorded PowerPoint presentation. I know the project has taken different kinds of format to try to be as accessible as possible. So, I’m very comfortable with PowerPoint, I definitely do not like recording a video of myself. I just did the first one yesterday, because I have a feeling I’m gonna need to do it more often in the semester to come and it was just as awful as I imagined it could be. [LAUGHTER] For the PowerPoint, I have a little picture of myself on the slide, but just my voice.
Rebecca: It sounds like something that we really need. Like self efficacy is something that, in a time when we’re really stressed, is something that we all need more, but also it’s hard to feel like you can empower people to feel like they can empower themselves. Do you have any tips that you can share with faculty about things that they can be thinking about?
Jessamyn: Well, I don’t want to give away all the myths, so I can build interest in the project. But, one of the myths of the three that I tackle in the presentation is the super teacher myth. And fighting that super teacher myth, the impossible ideals of the incredibly charismatic professor who magically helps students learn just by being entertaining. That myth is really, really persistent. And I think the more we can encourage people to recognize that that exists, even maybe at an unconscious level, but to really call it out and recognize it. And that goes a long way towards seeing: “Oh, so here are the ways I can help students keep learning even in these traumatic and troubled times.” I had a crisis pretty early in the shift to emergency remote instruction because I had not taught online before. And I was really struggling with being present to students and communicating to students because, as an introvert who had retreated to her house to replenish her teaching energy, I suddenly found myself needing to open up communication to students at home, while my beloved family (who I wanted to throttle) was humming and buzzing around me. And I had to be more accessible and communicate and present to students, all things I’d learned how to do in person pretty well as part of my teaching persona and to be effective, but I didn’t know how to do it online. And I was lamenting on Twitter: “I suck at this. I’m never going to be good,” and Flower Darby, a scholar of online teaching and learning, reminded me “It took you a long time, like it does for everybody, to learn how to teach effectively in person. The same is true for this new format, this new platform.” And it’s that super teacher myth, “I should be able to do it suddenly, even though I’d never done it before.” So, fighting the super teacher myth would be one of my top pieces of advice. I think.
Rebecca: I think one of the things that you’re pointing out that’s important to remind everybody, as we’re planning for fall in new platforms is there’s a learning curve to anything that’s new. And so if you’re having to learn the LMS, or a new piece of technology, or whatever, the faculty member needs to do that, but so do our students. So, [LAUGHTER] we need to build in some of the time and space to allow ourselves to do that as well as our students and they know when we’re not comfortable or we haven’t built up those skill sets to so being real with students about where we’re comfortable and when we’re not is also not a bad thing being human is important.
Jessamyn: No, and actually it can model for students having a growth mindset, and that learning takes time and it requires making mistakes and, as hard as it is, as difficult as learning is, especially in crisis conditions, especially in the context of trauma and loss, learning is also why we academic nerds and scholarly geeks got here in the first place. I know it’s helped me a lot this semester, in the midst of the struggle and this pain, to be able to look for things that I’m learning about teaching, and I’m learning about my students, and maintaining a growth mindset about my own pedagogical practices, remembering that it always takes a lot of practice, takes experience, takes reflection, but feeling like I was able to learn something… that always makes things better, that makes my nerdy heart happier. [LAUGHTER]
John: I think a lot of faculty have experienced learning in ways that many people hadn’t learned since grad school in terms of making an adjustment. Some people found it easy. The people who are ready teaching online generally found the transition at least smoother than it was for other people. But, for people who were used to only teaching in the classroom, this was a pretty traumatic experience, as it was for many of the students. I was just looking through some comments I got from my students this semester. And some of them said, “I didn’t sign up for an online class, because I really didn’t like it,” and they said the same thing in class right before we made that transition. So, it’s been a learning experience for all of us and maintaining that growth mindset, I think, is really helpful. How can we help students do that? I know you talked about that in your book as well as in your project.
Jessamyn: Well, kind of what I was just saying before, one thing I’ve found helpful is really the modeling portion, especially with the online aspect. And it was helpful with my students, first of all, to clarify this semester. This is not an online class. This is emergency remote instruction, and we’re looking to finish the semester the only ways we can in this crisis condition. And then, just liike we were mentioning before, I also was very clear and upfront about things I was learning how to do. And I’ve mentioned it a couple places now, so it’s getting a little less embarrassing, but I’ll admit it’s still embarrassing. One thing that I was forced to learn how to do was have students submit assignments electronically. I was still making them print out a hard copy of their paper and turn it in, even though for years, I knew I should not be doing that. I knew it made more sense to have them submitted online because I like to scaffold it. So, I always said, so I have to see my previous comments. It totally made sense. Plus, they didn’t have to pay for the printout, which was a real hardship for some of my students. So, I was finally forced to learn how to do it. And I told students, because I made a big deal at the beginning of the semester, I know this is old school and I am being an old Gen X lady here, but can you please print out your assignments? I’m really sorry about the extra step. And then halfway through I said, “Okay, Well, we’re all gonna do this together, and I’m gonna learn how to use the Moodle Dropbox” and I messed it up several times, the settings were wrong and students couldn’t submit. And they were so understanding. A couple of students said this to me, “I know you’re just learning how to do this.” So, it’s okay and it was kind of like modeling that and being clear about this was the technology that was new to me, and trying to be flexible with it. It kind of forced me to also rethink things like I have this really harsh and firm deadline. Well, yeah, except you messed up the Moodle dropbox parameters, so you can’t do that anymore.
Rebecca: I think one of the things that your story illustrates, to some extent, is that breaking down that one myth that you had just talked about, the super teacher, that by showing that we’re learning and that we make mistakes, it also shows students that the learning process includes making mistakes and so it’s not a terrible thing to have that occur. I know that when I’ve struggled with things in class before the students really respond to knowing like, “Oh, I don’t know the answer to that, like, let’s see if we can figure it out.” And the more you can indicate that you’re not some encyclopedia, [LAUGHTER] the more helpful it is.
Jessamyn: Yeah, I had a lot of students clearly very hesitant and fearful about going online. So, helping to demystify that a little bit, I think was helpful.
Rebecca: What are some of your takeaways from this semester moving into the fall?
Jessamyn: For me personally, definitely, kind of building on what I was just saying, being able to better utilize some of our learning management systems to achieve the pedagogical goals that I’ve always wanted to do. There are some very effective tools that I just had not utilized much before because I was doing face to face. One example I can think of is I live for discussion. That’s my favorite part of class and having students discuss, I’ve tried to keep my own piehole shut as much as possible and there are ways to structure, at least some discussion. Even if you’re doing a face-to-face class, you can also include some discussion in your learning management system that’s more inclusive, that will encourage what I hear from faculty lived experiences. And what I’m starting to read about is that there’s ways a good online discussion can increase student participation from people who, for whatever reason, are hesitant to participate in face-to-face discussions. Somebody I know who works with students with English as a second language said when they were forced to switch to online discussions, they started to hear so much more from students who had been hesitant about participating in face-topface discussions. So, my personal takeaway is definitely, when it helps me achieve the pedagogical goal that I would have in any format, I should be able to use some of the online learning tools that are out there. For faculty at large, I guess, I would say two things: One, I saw a lot of pain and struggle, as people were forced to give up things that had worked really effectively for them in the classroom. There’s a real loss there. That’s just one of many, many losses that faculty themselves were experiencing, and of course, in our personal lives during this crisis, but also as teachers, the switch was pretty traumatic in many ways. So, that kind of emotional component and being aware of what we lost and ways that the uncertainties that we’re facing really are going to take a toll day to day, class to class. And the other big takeaway, I think, I saw a lot of faculty and read about a lot of faculty really reflecting for the first time: What are we grading? How are we going to assess student learning? That really rose to the top among the faculty here. How can we possibly assess student learning? They’re just gonna cheat if they’re at home with their book and having it shoved in your face. Well, what do you want them to learn? What are they trying to learn? And how are you going to be able to assess that? So, really deep and difficult reflections on assessing student learning,
John: That type of reflection can result in improved practice, no matter how we’re teaching in the future, I think.
Jessamyn: Yeah, for sure. I just want to give one little shout out here to that term “Pedagogies of Care,” because I do think there’s some misunderstanding about it or assumptions that it means just being completely and utterly touchy feely and a lessening of academic rigor. And that’s not the case, as I talk about in Geeky Pedagogy and have talked about a lot in my own personal experience. You can express care for student learning and a wide range of ways you don’t have to be the extroverted, extra warm, motherly, fatherly professor. I am not that person. I’m very intellectual. And with students, I keep it really professional. But, I’m always getting feedback that she cares a lot, Professor Neuhaus cares so much, because I’m totally fascinated with their ideas and their learning and I do everything I can to help them learn. So, Pedagogy of Care, first for students, means clearly conveying to students that you want them to succeed. And that can take all kinds of different forms. The other thing in my contribution to the project that I emphasize about the Pedagogy of Care is that extends to faculty as well. And we really could all stand to be a little bit kinder and gentler to ourselves and to each other in these extraordinarily difficult times. The Pedagogy of Care extends to our own learning, and not “I flunked. I flunked the semester of teaching. I suck.” No, be as kind to yourself. I’ve repeated this to a number of people for the past four months, like just talk to yourself the way you would to a struggling student that you want to succeed, you know, you’re trying&hellip keep going&hellip you can do it. Don’t give up. This is an obstacle and it’s hard, but you’re learning. talk that way to yourselves too, and try to extend it to colleagues if you can.
Rebecca: I think that one thing that I heard a lot of faculty talking about in relationship to this idea is what they need for self care, and what they actually need and be able to kind of articulate it on a day-to-day basis beyond just the crisis, but there’s competing interests of like family and work and home space and workspace and what have you. And I think people are realizing what kind of time they need for different things to feel balanced, because everything got so out of balance, [LAUGHTER] going from one extreme to another.
Jessamyn: Oh my gosh, yes. I wrote about that. I had an essay in the Chronicle of Higher Education, and it was talking about being an introvert working from home and contrary to this kind of knee jerk: “Oh, introverts have it so great now, because they get to be at home.” Well, except that there could be other people there as well. [LAUGHTER] And demanding, finding, wresting out some solitude when you’re working from home was to me really vital, and it was not easy at all… it was stressful.
Rebecca: Yes, I remember reading that article and thinking, “Yes, all of these things.” [LAUGHTER] We have a system at my house now and that system is really helpful.
Jessamyn: That’s good. Yeah. structure. Yeah. And I live with two off-the-charts extroverts, like off the charts. And normally that works pretty well for us as a family. But, during this situation&hellip no&hellip social isolating. Like our needs were diametrically opposed. I need more time alone. I need more human contact. Yeah, it’s been rough. It’s been rough. [LAUGHTER]
John: We’ll include a link to your article in the show notes as well.
Jessamyn: Thank you.
John: So, you’ve picked a great time to be taking over a teaching center. [LAUGHTER].
Jessamyn: &hellip just the status quo, same old, same old, nothing really big going on.
John: So could you talk a little bit about what you’re planning to help prepare faculty for the uncertainty of the fall semester?
Jessamyn: Sure, It’s a great question. It’s definitely a uniquely challenging time to be trying to revitalize a teaching and learning center on a small rural campus with very limited resources and, like most state schools, facing some really severe financial and student enrollment problems, like maybe forever altering some structures. So, it’s tough. There’s a lot of managing of expectations and emotions. I think the advice that I’ve gotten by far the most, and makes the most sense as well, is the importance of building connections and building communities on campus and reaching out to a wide variety of stakeholders, including students, and really trying to foster pedagogical communities of practice on campus. So not trying to, again with the superhero theme, not trying to fly in and say I’m going to fix everything, but instead trying to encourage sharing of ideas, sharing of resources, support for each other, at this difficult time. We have a very small technology enhanced learning unit that has one instructional designer but we are collaborating on summer programming and working together and just trying to help everyone, I guess, really cultivate that growth mindset we were talking about, and try to approach this as an opportunity for learning. I won’t say silvered lining. That’s not how we want to frame it. But there is this opportunity because every campus has a small group of people who are bought into faculty development from day one, and they’re at every workshop and they want to take every offering and they’re your biggest fans. Then there’s a small group who are going to actively oppose faculty development in any way shape or form and will never ever come to your stupid pointless workshops for any reason, not for love or money. But, then there’s this whole middle population who you could maybe attract them, they could go to one workshop and find it useful and maybe go to another one. Well, that population, in the past three months, has just shot into a whole new world. And I have had, just in the past couple months I’ve been the Teaching Fellow for the CTE and then just starting this gig as the interim director. So I was doing some things with the CTE, and I saw more faces and heard from more people who had never darkened the door of the Center for Teaching Excellence appear and ask me questions and show up. Because, I think it’s not just personally “I don’t know what to do,” but suddenly everywhere, like literally everywhere are professors saying, “I could use some assistance with this. I’m not sure what to do.” Like for the first time in ever, there’s this like cultural acknowledgement that “I don’t know everything.” Like, that’s a major leap for academics to be like, “Well, yeah, maybe I don’t know everything here and I could use some assistance,” but everybody was saying it, everybody was doing it. So, there’s this opportunity to keep building on that and to offer assistance and encourage that growth mindset about their own pedagogical practices to a whole group of people who have never thought about it that way before. So it’s this precious opportunity. I hope I don’t blow it. [LAUGHTER]
John: I think we all share those thoughts about hoping we don’t blow it in getting ready for this.
Rebecca: Now, let’s not put unreasonable expectations upon ourselves.
Jessamyn: I know. [LAUGHTER]
John: But this is true not just for teaching. As human beings, we tend to do things as we’ve always done them unless there’s some compelling need to change. And when things don’t work the way they used to, it forces us to reevaluate how we’re doing anything. And then it’s a great growth opportunity. And it opens a lot of potential. It can be really difficult, as we’ve all been noticing.
Jessamyn: I do think it’s also, not to slam my beloved academic geeks, but I think it can be especially hard for scholarly experts. I mean, we were trained in graduate school: “You don’t reveal your vulnerable underbelly to the alpha academic or you’ll get your throat ripped out.” You always have to be the smartest person in the room. Like that’s the goal of getting your PhD and to back up and admit, “Well wait, I could use some help with this&hellip that’s a big leap for a smarty pants who’s used to their classroom kingdom where nobody ever questions their expertise and authority, which by the way, is not every professor,it depends on your embodied identity. That is a big caveat there. But, you do have this professorial authority and saying “I need help” or saying, “Wow, what worked before isn’t going to work here.” That’s a major leap. That’s a big ask for many academics.
John: And it can help break down that super teacher myth that you mentioned earlier.
Jessamyn: For sure. Yeah.
Rebecca: I think, along those same lines, too, it’s like that same group of people is recognizing all kinds of barriers that students face that weren’t so visible before.
Jessamyn: That’s very, very true.
Rebecca: So really, like transitioning the perception of the ivory tower to something a bit different, and I actually really hope it sticks.
Jessamyn: Yeah, me too. And that’s been amazing, actually, the way I’ve seen that on my campus as well. And it was interesting because I was doing a department-based needs assessment before the emergency pivot. So, I’d been talking to faculty about what they saw as teaching challenges and the student population. And then, within a few weeks after our shift, I saw some of those same professors saying very different things about their students and seeing them in a very different way. Like straight up saying, “My students lives are so hard…” like the obstacles and the lack of access to WiFi, for example, that’s a serious issue. Yeah. And it always has been. So, yeah, it really did. It humanized, I think is the term I hear some people saying is it humanized our teaching in new ways, for sure. And that could be a reach sometimes. Like I was saying, I am very intellectual, I don’t have a lot of personal discussions with my students. But, in these crisis situations, I was very clear about being worried about them and being concerned and hoping they were safe. And all my students appreciated it, but I knew some of them were like, “This is Professor Neuhaus saying, ‘Oh, I’m worried about you. Stay safe.’ Professor Neuhaus, really?” So, yeah, humanizing our interactions is important.
John: I hope that’s a message or a lesson that continues through into the future.
Jessamyn: Me too.
John: And I think it’s worked both ways, that I think a lot of students have seen some of the challenges their professors have faced, not just in terms of using the technology or teaching in a new format, but in terms of having children or pets or other things, or having technology issues, or having access issues themselves, where someone might be using a video game or something similar, cutting into the bandwidth. Many faculty have reported to us that their students have expressed concern, asking if they’re okay, and encouraging them to stay safe and so forth.
Rebecca: I think it’s also along those same lines brought to light some of the invisible barriers that contingent faculty have, being at multiple institutions or the incredible workload that they’ve been asked to bear without really any compensation for the time and effort and energy that’s gone into it.
Jessamyn: Yeah, for sure.
John: Our institution has provided loaner computers and other types of technology to both students and faculty. And an interesting phenomena was that there were more faculty who requested computing equipment and other tools than there were students even. They provided a good deal of it to both, but some of those barriers are not just for students, especially are adjuncts who often are struggling to get by.
Jessamyn: For sure. I was just going to reference and you can put this in the notes to Cate Denial, a historian who wrote a very well known essay, I’m hoping she’ll write a book, advocating a pedagogy of kindness. And I definitely saw how effective that can be this semester for me personally, but I also saw with a lot of other faculty for the first time really seeing what a little bit of flexibility and a little bit of kindness&hellip again, not lessening academic rigor, but bringing in, specifically, some of that humanizing kindness&helliphow effective that can be. And actually, on a similar note, the advocates of ungrading have gotten a big boost as well, because I’ve seen and read a lot of faculty saying, “Wow, you know, once I told my students pass-fail, for example&hellip Wow, their final projects were so great, like they actually did what I want them to do and learn what I want them to learn once the stress and anxiety and kind of false dichotomies, I guess, of grading were taken off the table.” So there’s some real possibilities there.
John: We’ve talked quite a bit about things that we should be focusing on in terms of getting ready for the fall. But what are some things we should probably avoid as either faculty or professional developers in preparing for the fall semester?
Jessamyn: I think a big one is to not ignore or try to just sort of skip over the trauma and the loss that people experienced and also not play like “Who had the worst trauma?” or “Who had the worst loss?” In all kinds of ways we experienced loss as we experienced trauma&hellip and the way trauma works, the weight loss and grieving works is even a small loss can be very difficult because your brain and your heart and your soul are trying to process all your losses and all previous grief and loss. I know I always love graduation. And even though I sit there&hellip it’s very long… it’s very hot… [LAUGHTER] and it can feel like a chore at times. But, the commencement ceremony is so meaningful for everybody, but especially for first-generation students and their families. And we tried to fill in a little bit with some online messages and some online rituals. And I started watching it and just started crying. And I’m like, “What is this? What is happening here?” &hellipand it’s a loss to not be able to engage in that ritual. It’s not the world-ending loss, but it’s a loss and it’s a trauma and people are going to arrive to classes in late August, whatever format it is, with all those things having really just happened to them&hellip faculty, students, administrators, I mean, everybody’s gotten a really raw deal this semester. And that’s not just going to be magically fixed, even if somehow we’re back to exactly where we were. And if all the face-to-face classes are in session as they were planned, and everything’s fine again, but what happened this semester is still gonna be there.
Rebecca: I think that’s a really good reminder. Our students are going to be changed and will be different. And I think in the moment of this semester, a lot of students weren’t able to process what was happening. So, you might really have a really different experience with students in the fall, when they’ve also had a little space to process what that experience was like and the things that they missed out on and are missing out on if they’re online in the fall.
Jessamyn: Yeah, the first chapter in Geeky Pedagogy, advocates for practicing awareness, and really just being as fully mindfully present to the reality of what’s happening around you. And I think that’s always important. But, I think it’s going to be especially important in the fall. And it doesn’t mean we shouldn’t try to change things that are bad, but to first really be cognizant and aware of what is happening, what is going on here. And the state of all our mental, emotional, and physical states is going to be something that we really have to pay attention to.
Rebecca: I think that’ll really shift what first days of classes look like in the fall.
Jessamyn: Yeah, for sure. The uncertainty remains. We don’t know what’s going to happen. And we can put plans in place, but we just don’t know. And that’s…
Jessamyn: Yeah. [LAUGHTER] For people who love to plan, and I had really fooled myself, you could see it in my book, too. I’ve attained a new enlightened state where I can roll with the changes and you got to be aware of stuff but then as soon as my world was severely disrupted, no, it was all gone. Just zip… gone. [LAUGHTER]
Rebecca: Emotions do have a lot to do with how we learn and process things.
Jessamyn: Yeah! [LAUGHTER] Come on, emotional learning… yeah.
John: As you know, we always end with the question: What’s next? A question that we’re all thinking about pretty much all the time.
Rebecca: Please tell us.
Jessamyn: Yeah. Wow, that’s crazy&hellip ‘cause I have been working on a project, an anthology of insights into effective teaching from women, marginalized, and underrepresented faculty. It hasn’t been suspended, but I extended the deadline, not having the bandwidth and assuming potential contributors also just utterly overwhelmed. And then I guess it’s like day by day&hellip What could maybe help a few people on campus teach effectively? And, of course, how am I going to prepare my own class to be as resilient and flexible as possible for the fall&hellip and just on a personal note, what about my child? He just finished his first year of college. It was not a overwhelming success. I mentioned last time I was here that he is a lackadaisical student. And he had many of the challenges that first-year college students face. And then, of course, this semester has been a disaster. He was one of those students who said, “I don’t want to do an online class.” He’s an extrovert. He’s very social. So, we’re not sure what’s going to happen for him in the fall. So, those are all the uncertainties awaiting us.
John: I first heard about the Pedagogies of Care project when one of the people participating posted a picture of the Zoom screen with all those people in it. [LAUGHTER] And I recognized all of them, and many of them had been guests on our podcast. So, the initial image didn’t talk about what it was for, it just said a gathering of present and future authors from West Virginia University Press, and it looked like a really impressive group of people. So, we’re very much looking forward to this project. We’ll include a link to that in the show notes.
Jessamyn: They’re really some of the smartest people I’ve ever met, and definitely the best collaborators I’ve ever had. It’s a unique experience being part of that series. I’ve never had a group of scholars who’ve kind of come together and really formed a supportive and encouraging community. It’s just amazing. I’ve never experienced anything like it. And this project, I think, is a good example of how the series at West Virginia University Press edited by Jim Lange is unique to not just the scholarship of teaching and learning but to scholarship period, because I’ve been in various series and journals and stuff, but there’s never been a sense like, this is a real pedagogical community of practice where ideas are debated and shared, and each scholar is really supported and I’m really incredibly grateful and proud to be part of it.
John: And that also shows up in the Twitter conversations that take place. For those who don’t follow the authors there, we strongly encourage that.
Jessamyn: Absolutely. Yes.
Rebecca: Definitely, that’s how we found out about this project.
Jessamyn: Thank you so much for having me.
John: Thank you.
The unexpected shift to remote instruction during the spring 2020 semester in response to a global pandemic disrupted established teaching patterns, forcing many faculty to rapidly learn new tools and techniques of engaging their students. In this episode, Dr. Betsy Barre joins us to discuss what we’ve learned from this sudden shift to remote instruction and how we can better prepare for the uncertainties of the fall semester.
Betsy is the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won, with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator.
- Course Workload Estimator
- Ask the CAT Archives, Wake Forest University Center for the Advancement of Teaching
- Meaningful, Moral and Manageable? The Grading Holy Grail
- Derek Bruff, Director of the Vanderbilt University Center for Teaching
- Beloit College, Curriculum Changes for Fall 2020
- Betsy Barre (2020). Let’s Talk about Grades. March 24.
- Anita McCauley (2020). “Assessing Student Learning: Alternatives to Closed-Book Exams.” April 9.
- National Public Radio interview related to Pandemic
- Linda Nilson (2019) 95. Specifications Grading. Tea for Teaching Podcast, August 21st.
- Tobin, T. J., & Behling, K. T. (2018). Reach Everyone, Teach Everyone: Universal design for Learning in Higher Education. West Virginia University Press.
- Newport, C. (2016). Deep work: Rules for focused success in a distracted world. Hachette UK.
John: The unexpected shift to remote instruction during the spring 2020 semester in response to a global pandemic disrupted established teaching patterns, forcing many faculty to rapidly learn new tools and techniques of engaging their students. In this episode, we discuss what we’ve learned from this sudden shift to remote instruction and how we can better prepare for the uncertainties of the fall semester.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Our guest today is Dr. Betsy Barre, the Executive Director of the Center for Advancement of Teaching at Wake Forest University. In 2017 she won, with Justin Esarey, the Professional and Organizational Development Network in Higher Education’s Innovation Award for their Course Workload Estimator. Welcome, Betsy.
Betsy: Thanks, I’m happy to be here.
John: Today’s teas are:
Betsy: I am not having tea, but I am having a raspberry lime Spindrift. I actually would love to have tea, but I just didn’t get downstairs in time, so I have my Spindrift here.
Rebecca: That sounds good. I have an English breakfast.
John: And I have oolong tea today.
Rebecca: Oh, you’re switching it up a little.
Betsy: Sounds exciting.
John: Amazon helps. [LAUGHTER]
Rebecca: Betsy, we invited you here today to talk a little bit about the work that you’ve been doing at Wake Forest to help faculty prepare for pandemic teaching. Can you talk a little bit about what the Center for Advancement of Teaching’s approach has been and what it will look like leading into summer and fall courses?
Betsy: Sure, I can talk more about what we’ve done, sort of what we’re planning for the future is still in process, as I’m sure it is for many institutions. One of the great things about Wake Forest is that our Center for the Advancement of Teaching is not the only office that has been working with faculty and faculty development and digital technology issues, academic technology, etc. So one of the first things that we did when we knew we were transitioning to online, or transitioning to remote teaching, let’s be specific there, is that we pulled together the offices that were adjacent to our office. So we pulled together the Office of Online Education, the Office of Academic Technology, which is an Information Systems RIT wing at Wake Forest, and also we had a number of librarians who did work on digital pedagogy. So we pulled all of us together and created a kind of super team that would support faculty, and that was really helpful to do that really quickly because it expanded our reach, the numbers of folks who could work with faculty and integrated it, so faculty didn’t have to go to a million different places, there was one place that they could go. We had about 850 faculty or so that were teaching that we had to work with and there were about 10 of us on our team. So, it’s a better ratio than some schools, but it’s still a pretty not ideal ratio, and so we tried to streamline things as quickly as possible. So, like many schools, we created a keep teaching website that had resources, but we also created a blog that had daily updates. So every day, they could subscribe to an email and get it in their inbox every morning that would have daily updates, but also resources, tips, things we’d heard from faculty, etc. That turned out to be really helpful and we’re still keeping that going, and it’s been helpful as they’ve been teaching. We also, though, really wanted to encourage them to share their expertise with each other. So, that week that we had off to help our faculty prepare, we did a series of open labs, where we were there to answer questions, but they could also share with each other what they were doing. And then sort of unexpectedly, a few things that we did that have gone really well is that those of us that are on social media saw some faculty talking on Facebook about this, we thought, “Hey, let’s just create a Facebook group,” and that group has been incredibly active. We have like over 300 faculty that are in that group now and some of our professional staff and it’s been a way of communicating. We’ve tried to communicate it outside of Facebook for those who don’t like Facebook, but certainly it’s been a wonderful way of building community that I think will live on after this, and so that has been nice. And then of course, our one-on-one consultations that we’ve always done, but we set up a easier streamlined system for requesting a consultation, and it would cycle through all 10 of us and sync up with our calendars, and so we found that to be really successful, and as successful as we could be in this trying situation. Summer and fall, a much more interesting wrinkle, that we’ve been working on. Once we got faculty up and ready to go, we now could transition to thinking about how are we going to support faculty in the summer and fall. And one of the things we’ve been saying all along, many institutions have, is that what we did in the spring where we had one week to transition is not really robust online teaching. At the same time, we don’t necessarily have the staff and resources to transition all of our courses online for the summer in a robust way, but we have more than a week. So we’re trying to hit some sort of middle sweet spot where it’s not exactly what we would ideally do with online education at Wake Forest, but it’s better and more intentional and takes more time than what we did for remote teaching. So currently, we’re planning for those who have volunteered to teach in the summer to run a three-week course for them to take asynchronously online to learn more about teaching online, and then we’re also gonna offer all 10 of us to do one-on-one consultations and some minimal instructional design work with them. Fall is still up in the air and we’re not really sure what’s gonna happen with the fall, but I think we’ll probably know in the next few weeks what we’re planning.
John: It’s interesting to see how similar the approaches of various institutions have become and a lot of it, I think, is social media made it easy to share some of those thoughts. We also have a Facebook group, we’ve also done lots of meetings and we’ve had a number of people working with us from our campus technology services in providing support and workshops, and it’s been nice to see everyone come together to help so many faculty make this surprise transition that they never expected and didn’t always entirely welcome, but they’ve been really positive in terms of how people have approached it.
Betsy: I agree.
Rebecca: One of the things that I saw you guys doing that I thought was really interesting was “Ask the CAT,” can you talk a little bit about that program and how it works?
Betsy: The name of our center is the Center for the Advancement of Teaching. We still haven’t decided at Wake Forest if we want to do cat or CAT, but we often joke a cat would be funny because then we could have all these funny cat jokes associated with that. But outside of the blog, we started getting some really simple questions that we realized would be helpful for everyone to hear the answer to. And so early on, a few people asked some questions, and we said, “Can we turn this into a sort of Dear Abby letter that we can then publish responses to, really quick responses on our blog?” and they were happy to do that, and then we turned it into a formal themed series in the blog where people can submit online, ask the CAT questions, and they can do it with a pseudonym, so there’s no stupid questions, any sort of challenges they have, and it’s gone pretty well. And we hope to continue to do that because I think we’ve seen on the Facebook pages, I’m sure you all have as well, is that often there are many similar questions, and so when they see us answering another question, faculty get ideas and say, “Oh, I could do that, now that makes sense.”
Rebecca: It seems like the ability to have a little bit of anonymity there in asking the question might allow for some questions that really need to be asked actually be asked.
Betsy: Yeah, sometimes they’ll just ask a simple tech question and we try to expand it a little bit beyond that to say “Okay, that’s great. Here, I’m going to give you your answer, but before I do, let’s talk a little bit about pedagogy and how you might think about universal design,” or something unrelated to the specific tech question.
John: Rebecca mentioned that you had won an award for your work on Rice’s Course Workload Estimator, which is something we recommend to our faculty regularly and people find it really helpful. How would you recommend people interact with that tool during situations like the pandemic, especially for people who are adjusting very rapidly from one mode of instruction to another?
Betsy: Yeah, so one of the things I shared with Rebecca before we started is this actually is really great timing for you to ask about this, because Justin Esarey is the co-author, co-creator with me and I… he’s my husband actually, we did it together… one of the things we’re thinking about doing in the next couple of weeks is actually revising it in a number of ways. We’ve had a long standing interest in doing it, just haven’t had occasion to do that, and there are some changes we’re going to make that aren’t specifically about online, but one of the changes we’re hoping to do is to actually create some categories that are related to traditional online assignments. And again, these are going to be guesstimates. I always tell people, this is an estimator, it’s not perfect. It’s just our best guesses. But to create some estimates of “How long would it take to have a discussion board if they have two posts, 500 words,” sort of things that we’re used to assigning in online education to hopefully help in that regard. But one of the things that I think is a reason this estimator is important is one of the things we’ve seen, and I’m sure you all have seen as well, after about the second week of remote teaching is that some students started to complain about workload, how much work these new remote courses were. And I think part of that is because faculty were incorporating more accountability measures into their courses, so they may have been expecting that work, but never were really holding the students to account to do that work. And so now students actually have to do and show their work, and so whereas before they might have been able to just show up in a lecture, study on their own time, or not study as the case may be, not do the reading as the case may be. Now, if they’re having weekly reading reflections, they actually have to do the reading and that significantly shifts how much work they feel they have to do. So that’s putting it on the students, but it’s certainly the case, and part of the reason we made the estimator, is that as faculty, we’re not really good at estimating how much time our work takes. And that’s true in a traditional setting, it’s true for me, that’s why I created the estimator. I am a humanist, and so I assign a lot of reading and I never really knew, like, how much time it would take them to read, and so that’s what motivated me to investigate the research on that. I think it’s particularly true that we’re not good at estimating how much time things will take when it’s a new assignment or activity that we’ve never assigned, and that’s what we see in this scenario, many faculty are introducing completely new activities and assignments that they’ve never done before. And they often might think, “Oh, yeah, I should give them discussions in a discussion board,” without taking into account how much time that will take, or “Oh, I really want them to make sure that they connect with me each week, in this way,” or “I need to make sure we have these office hours and they need to watch these videos, but since they’re watching the videos, now, we can have some discussion in class because the videos are no longer part of the class time.” And so we think we’re pretty good at sort of keeping track of that, but it turns out one of the things we found with our estimator is that when we asked faculty to play around with it, that we were often very wrong. Faculty were often very wrong about even their own estimates about how much time they thought they were expecting of students. So, I think it can be a valuable check. It’s not perfect, it’s not exact, but it can be a valuable check on our intuitions about how much time we’re expecting of students, particularly with some of these unique activities that we’re asking them to do online, and I also think there’s some really creative strategies by our friends in online education to help us think about a traditional assignment and how to make it a little bit more efficient, discussion board a little bit less time intensive, that we can talk to faculty about as well.
John: With the pandemic, I would think, some of those calculations based on online classes where people intentionally were in online classes might be a somewhat different situation when people are in households where there’s more people in the room perhaps, or where they’re sharing network access, or where there’s more distractions and noise than the people who had intentionally chosen the online environment.
Betsy: Yeah, I think that’s a really thoughtful insight. Absolutely. I think we’ll hopefully get to talk about this later in our conversation today, is that there are a variety of changes that take place here that are not just about the modality, but thinking about our students’ situation, how long it takes to learn the technology if they’ve never learned it as well. Like, “How do I upload this? How do I take an exam?” And so if we give them a certain amount of time for an exam, recognizing that they didn’t choose to do it, they also don’t know the technology as well, and so how do we account for those adjustments as well, for sure.
Rebecca: Yeah, I agree. I think all of those little extra things that now students have to do, including learning the technology or just getting used to a new system or a new rhythm, they all take time. In a semester, we think that’s what the first couple of weeks of the semester are, but then like this semester, we had two sets of those.
Betsy: Absolutely. And I mean, the fact that we are still sending out posts giving suggestions, means that some faculty are still changing things, they’re still adding new things, because they want to try something new or something didn’t work, and normally, we encourage that, but in this scenario, it’s particularly challenging for our students if new things keep getting piled on over five courses or four courses that they’re taking.
Rebecca: If we’re thinking in a traditional context where there’s in class and out of class work, and now everything is remote, how do we think about dividing up that time or what kind of time they should be spending on what kind of activities?
Betsy: I think this is a really good question. And again, my colleagues in online education who think about this question a lot have more subtle distinctions to make about this, but I actually was just having a conversation last week about what accreditors require and how to think about, quote unquote, contact hours in an online environment, and incidentally, one of the things we found actually, unexpectedly with our Course Workload Estimator… again, the motivation was for me as a humanist to basically answer the faculty question of “How much reading should I assign?” was a very narrow purpose, how much reading should I assign? But what we found is that the biggest usage were people who were instructional designers in online programs, who were interested in this question of “How much time is faculty contact hours, is it actually comparable to the face-to-face courses?” So it is connected, and so I’ve been talking about this a lot, and one of the things that, at least the federal guidance suggests, is that one credit hour is about 45 hours of work for students. So over 15 weeks, one credit hour, you do two hours out of class for every hour in class over 15 weeks, and so it’s about 45 hours. They don’t really enforce it, it’s a complicated question or a comparable amount of work, but that’s an easy way of thinking about it. It’s about 45 hours of work for a single credit hour and then 15 of that is expected to be in the presence of the professor. So traditionally, that would mean 15 of that you go to class, 30 of it’s at home. That’s the traditional model that we think of, but in online, of course, it’s different because everything is at home. So one thing, you could just say, “Well, everything’s at home. So then professor never needs to be engaged,” like, you can just say, “I’m going to record all my lectures, put them all up, and then I’ll grade your exam at the end.” Of course, we know that that’s not good pedagogy, online or otherwise. And so I think the way to think about this is, of the 45 hours of work your students are doing, are at least 15 of those hours, somehow engaging with the faculty member? But that could be, for example, a discussion board where the faculty member is in the discussion board engaging and providing feedback. It could be one-on-one sessions where you work on a paper together with the student in an office hour. There are a lot of ways you can imagine faculty presence and engagement that don’t have to be “Let’s have a synchronous video conference session.” But there are some good reasons for that too, particularly in the remote environment where students want some continuity to what they’ve already done. But I think that there should be more flexibility and I think there often is in good online program about what counts as those contact hours, but without just saying, “Oh, as long as we have a video, that counts as a contact hour.”
Rebecca: Along these lines, do you have any advice about designing learning activities and assessments when we have no idea what the modality might be in the fall?
Betsy: That’s a good question. I’m sure that many other people have been asking that question, I myself am teaching this semester, so it has been interesting in helping all the faculty but also teaching myself and figuring out what’s working and what’s not. And I think Derek Bruff at Vanderbilt had a, I actually liked this language that he shared initially about creating pivotable courses. He ended up changing it, he didn’t like that one as much, but I actually like that, like your course could easily pivot. And I think for me, one of the things that I saw was that my course, even though it was a face-to-face course, heavy discussion course seminar course, I had built in already some asynchronous activity outside of class, they were already annotating the text via Hypothesis, which is a really wonderful tool for those of you that don’t know about that in the humanities or any text heavy discipline, Hypothesis is wonderful and in that sense they were already used to and had learned how to annotate their text digitally in the face-to-face course. When we transitioned, it was easy. Okay, we’re going to be doing that. And that was already built into the course. I also think getting all of our courses so far as possible into a digital environment, whether that’s an LMS, or Google or whatever you prefer, can be an easy way too, because a lot of the time we spent with faculty was just getting them to like, “Oh, how do you collect assignments? Okay, let’s get you into the LMS. Here’s how you collect assignments. Here’s a way that you can think about sending a message to students that’s not just through email.” And so at the very least, if we all get in our LMS, or another digital environment, if you don’t like the LMS, and then think of some activities and engagement that our students can engage in at home with each other, or perhaps with you that’s outside of the regularly scheduled class time, you’re already making it easier to shift. But I also think one thing, and we may come to this when we talk about grading, one thing I’ve been thinking a lot about is I had to scrap one of my activities in my course when we transitioned to remote and I’ve been thinking about the particularly challenging situation for those faculty who had a semester-long assignment. So luckily, my assignment was at the second part and so they haven’t started it yet, we can just do something else, because that would be difficult. But if you have many semester-long assignments, that disruption can be really difficult, but if you could organize your course another way to make it pivotable is to organize it in modules, like really intentionally, not just in Canvas, but actually say, “Okay, we’re going to work on this unit as a self-contained assignment that will be done in two weeks. So that way, if we have to take off in week three, you’re already finished with that assignment in that module.” And then there’s one module that’s remote and then if we come back, hey, we get to start another module that might be face-to-face, and so it gives you some flexibility. If you design your course in a more modular way to prepare for disruptions rather than thinking about it as multiple whole semester-long assignments.
Rebecca: It’s interesting that you say that, Betsy, because I’m not teaching this semester, but I’m planning for my fall class, and I teach web design primarily and I was thinking about teaching agile design. So I decided that I would teach it in an agile fashion, which is really what you’re describing. [LAUGHTER]
Betsy: Yeah, that’s smart.
Rebecca: So, I started mapping out what that would look like in these little sprints to work on a larger project, and we would do maybe two projects, one that was collaborative and one that was individual, but in sprints that would rotate between the two projects. So I’ve been mapping out what that might look like, and my real reasoning for that was, specifically if something was going to be disrupted or if it was going to be online, I thought it would be a little easier to help students through the project if it had these clear checkpoints and finishes to things before starting something new.
Betsy: One of the things that made me start thinking this way, and this goes back to the question of how we’re preparing for fall and all the scenarios that all the institutions are thinking about, Beloit college just decided that they were going to actually teach their fall semester in two seven and a half week sessions, essentially. So basically, students will take two courses for the first seven and a half weeks, and then two courses for the second seven and a half weeks. Certainly it’s a lot of work on the part of faculty to transition their 15 week course to a seven and a half week course, but it also is creative because it means that we have to start late, only two classes are disrupted rather than all four and if you have to leave in the middle, only two classes are disrupted. So, there is a way in which it allows for some flexibility. You can even be as dramatic and radical as going to a block schedule like they have at Colorado College or other schools where they have one course at a time. That would be more work for our faculty and may not work as well, but I did like the idea of thinking, “Okay, let’s just prepare for our face-to-face courses to be seven and a half weeks as an institution.” And then it’s the opportunity to experiment with that kind of pedagogy anyway, because some schools have May terms and other things. And so we are not, at Wake Forest, certainly planning that, but it is an interesting fun thought experiment to think about.
John: One issue that we’re talking about on our campus is how faculty should administer final exams, and grading and assessment, and there’s a lot of concern over people trying to give timed exams and put other limits on students. What are your thoughts on how we should deal with assessing students as we move towards the end of the semester?
Betsy: Yeah, that’s a great question. So, there are a number of issues at work here. One is the challenges the students have at home and thinking about good universal design principles of giving students as much time as possible if it’s not one of your outcomes. If doing things quickly is not one of your outcomes, that’s an important thing to think about. Often, also, what’s in the mind of people, though, is academic integrity. And so part of the concern of a number of faculty is: “Well, I’m usually proctoring it in person. So, how do I give an exam in a way where I’m not going to be there in person?” And then that raises all sorts of interesting challenges associated with technology and the privacy concerns with those online proctoring systems, and so certainly we’ve been thinking a lot about this too, and how to give advice. One of the first easy answers that anybody who’s in pedagogy is going to say, is come up with different designs for your assessments. And I think, absolutely, we should start there. I don’t need to give a timed exam in my course, there are ways I can write the question where I’m not worried about academic integrity issues. So, there are certainly ways in which that’s possible, but I do want to be mindful of my colleagues in intro languages, or my colleagues in intro math, where there are some recall outcomes that are really important for them. And so I think I always want to just be careful to not say, like, “Oh, how dare you have any recall outcomes because that’s just not good pedagogy.” I don’t think that’s necessarily true, so for those colleagues who have recall outcomes, it becomes a more interesting question. On our blog, we have a post on this that we could share, if you’re interested, where my colleague, Anita McCauley, who’s amazing, posted a flowchart of ways of thinking about, “Okay, if you have an exam, what are ways that you can think through how to do something differently?” And one of the first parts of that flowchart that I really like is, if you’ve already assessed it before, you may not need to assess it again, and so particularly for my colleagues in Spanish and other intro languages, maybe they’ve already assessed their ability to conjugate verbs. Do you need to have it on the final in a cumulative way? That was just something that often has not been on the table and talked about and I think it’s worth saying. But, beyond that, they might be somewhat different outcomes where they have to recall but then explain why the verb was conjugated in that way, and so there are ways you can see whether they know it or not, that they can’t just get on the internet. And so being mindful of the challenges there but also saying that “Let’s try as hard as we can to come up with alternative assessments.” Then the questions of how much time to give them, again, always come back to say, like, “Is speed one of your outcomes?” and almost always it’s not; almost always the reason there are timed exams is because they’re in the timeframe of the class. So there’s 75 minutes for them to sit in the class and take the exam and that’s why there’s a limit. It’s not because speed is actually an outcome. So now they actually have some more flexibility where they could give them more time and the technological tools allow them to give them more time, and you can extend it as far as you want. I will often say, instead of giving accommodations to a student to get extra time, give the whole class extra time, especially as they’re learning new technology. If folks are still committed to traditional recall exam and worried about proctoring…. we, for example, at Wake Forest,have not bought proctoring software… and we’re not using it for a variety of reasons, and so one of the things I recommend is if you absolutely are still committed to that, then you can do a synchronous session just like you would normally where they’re taking the exam, and it’s you, not some outside vendor or AI etc, as you would in the classroom.
John: I’m not sure if the problem though, is just due to recall type exams because I can speak from my own experience. Last week I gave a test which were all applications in econometrics and copies of the questions (where there were many different variants for one problem, there were seven variants), most of those problems ended up on Chegg within about 15 minutes of the release of that, and answers were posted. Many of them were really bad answers, which helped make it really easy to find these things…
Betsy: [LAUGHTER] Yes, yes.
John: …within less than an hour of the time the exam was released. So even when people are doing some problems, there are some issues, or even when they’re asked to write essays, there are people out there who are willing to provide those responses for them.
Betsy: Oh, yes. And actually, that will be true in face-to-face classes, too, if you’re not doing in class essays. That’s the one level of academic integrity that you just are never going to be able to catch if you pay somebody to write your essays for you or take your online exam. My background, incidentally, is as an ethicist, so I think a lot about questions of academic integrity. I always get mad at my students when I give this lecture like, “This is an ethics class, you need to take this seriously.” But it is true that the empirical research on student behavior in this regard is not heartening. Let’s put it that way. So I really appreciate all the literature about “We need to trust our students,” and there’s a certain framework of what happens when we come into a course where we don’t trust our students, but the empirical literature about what students admit to have done is really not heartening, and so I do think it’s okay for us to think about these questions that you’re thinking about, John, which is, “Okay, we’re creating conditions where they’re tempted,” and that’s something also we don’t want to do either is to create the conditions where students might be tempted, particularly for students who do have academic integrity, because then they’re at a disadvantage if they choose not to engage in that kind of sharing of resources. What did you do, John, how did you address this?
John: I’m just dealing with it now, I was just grading those today.
Betsy: Yeah, it’s tough.
John: So right now I’m trying to identify the students and I’ll be having conversations with them. Because there were so many varieties of questions out there, it’s going to be pretty easy to identify which student did which. One interesting thing is, someone took one of the answers and ran it through a paraphrasing tool so that the “error terms” in the equation became “blunder terms” in the equation, which was a pretty obvious paraphrase. It was interesting.
Betsy: One of the things I’ve appreciated about this moment and having conversations like you and I are having right now is that it’s encouraged some faculty to think in different ways about assessment. They have a standard way of assessing “This is the kind of thing that I’ve done for years,” and now I have to think, “Oh, what could I possibly do differently?” So one thing I just keep coming back to, when I think about my own courses… there are challenges with this… there are problems with this, because it can be stressful for students… but I think oral exams are often some of the most effective ways to see whether a student knows something is that you, face-to-face, come to my office hour, and let’s talk about it. Tell me, and I’ll ask follow-up questions. That’s a way to really tell whether a student knows something, and so you can still do that virtually. Now that takes more time, especially if you have a big class, but thinking sort of outside of the box in that way of “How can I verify?” is important. I have a couple colleagues that are ethicists too, who have devoted their life to this issue of academic integrity and it consumes them. In some ways, I understand that, because it’s a real violation of trust, and it harms other students. But at the same time, too, I worry sometimes that it becomes so consuming for us that we lose track of all the other things that we should be thinking about with teaching, and so, in this scenario, where it’s as crazy as it is, this is why I think the pass-fail designation that many of our schools have done have made things easier, because we also know empirically that students are less likely to cheat when it’s a pass-fail environment. I think the fact that many of our schools did optional pass-fail means that we’re still in this wrinkle space where many of our students still want to get the good grade, and so they’re taking it for a grade and there’s still temptations. But thinking of ways to make it less high stakes can be another way as well to reduce the likelihood of academic integrity, but it is going to be a challenge that there’s no quick and easy solution for. I don’t have your solution, John.
John: Well, I don’t either, right now.
Betsy: Maybe somebody will… that they can tell us too, who listens to this podcast.
John: One thing I am also doing is I have scaffolded assignments where they have to develop things from the very beginning up to their final projects, and there it’s much more difficult for academic integrity problems to show up because they’ve been guided and getting feedback all the way through and that tends to reduce it, but when you’re trying to test some other things that they’re not using in their projects, but might need to know in the future, there are challenges there.
Rebecca: I think another question that’s come up quite a bit is how to grade fairly just over the course of the semester, either this semester or a future semester when there might be potential for another outbreak or something, when students are not in optimal work conditions, there’s distractions, they might be sick, they might be dealing with family members who are sick. So what do we do to make sure we’re fair?
Betsy: Again, coming back to me as an ethicist, I think a lot about academic integrity, but also about grades and what it means to be fair, and there’s some people who would make the argument that there’s certain notions of fairness… that it’s impossible to grade fairly, even in normal situations, especially if we’re taking into account differences in student background, etc., that they’re always going to be disadvantaged students in our classes. And so thinking about what a grade is, is really important. And again, I’ve been heartened by the fact that these challenges have led so many of our faculty to start thinking in new ways about “What the heck is a grade and how do I want to think about my grades?” And I do think that one way of thinking about fair grades is actually not the model of “Well, we need to take account of all these challenges the students have,” one way of thinking about their grades is that all the grade is, is a measure of their performance. Now, you could say that that’s unjust for other reasons, but that it’s at least I’m treating all the students the same. So this is maybe the difference between equality and equity. So like we’re treating them all equally, that’s a measure of performance and mastery, so it’s ensuring the integrity of the grade. But what’s interesting is that most of us don’t actually grade that way. Most of us have all sorts of other things in our grading scheme that are about behavior, rather than about outcomes. So like, “You have to show up, you have to turn these in by this due date, you have to make sure you participate in class,” and I have those in my typical grading scheme as well, and those we’ll refer to as behavioral grades. And there are some educational theorists, as you two probably know, that would argue that you should never grade on behavior, you should never have behavioral grades. I think we could have a much longer discussion about this. I sort of think there are some good reasons for doing it in the context of higher ed at least. But I think in this scenario, this is if there’s any scenario and this is what I wrote about in one of our first blog posts, if there’s any scenario where that would be unfair, the kind of behavioral grading, it would be this scenario because some of our students did not choose this, they’re in different time zones, they can’t make it to our class, they have to deal with things at home. They were already in the midst of the course too, so it’s not as if we say, “Well, wait a year and come back to us when you’re ready to take the class fully,” because they were ready, and we kicked them off campus. So there is all sorts of other complications here to the traditional model of like, “Well, wait until you’re ready to take a class.” They can’t, they were already enrolled, they already paid, we’re not giving them refunds. So in this context, being as accommodating as possible, and making our courses as accessible as possible, is really important. And some people have even argued, this is why we should give them all A’s like some people have argued, not just pass-fail, but actually all A’s would be a better approach. Because to say like, “Look, you’ve done some work this semester, let’s move on and give you all A’s,” of course that creates challenges for some of our colleagues, who are going to say “What about the integrity of the grades for future courses? Is that fair to students who take it a different time and don’t get the A?” So what I have argued for, but it’s again, not a perfect solution, is really dropping any behavioral grades that you have in these scenarios, at least for this context, and then really focusing on your mastery outcomes, but also being reasonable about the number of outcomes students can master in this scenario. So I actually dropped two outcomes from my course completely, completely dropped them. Now, that’s easier for me to do in an intro religion class than it is in an intro calc class where they’re prepared for the next course. So I always want to be mindful of the differences of my colleagues in different disciplines. But, if you are able to drop outcomes, you can drop them and still be rigorous with the outcomes you still have and being a little bit more compassionate and sensitive to your students. But doing mastery based grading also can be helpful in the sense that, for me, students get multiple shots at showing mastery, and so this would be like specifications grading if you want to read more for the fall, so they have multiple opportunities to show. So, if they have a bad week or an assignment doesn’t work well, they can try again, and as long as by the end of the semester they showed mastery, that’s enough. It’s not about averaging over the course of the semester, and so I already had a mastery based grading system in my course before I began this semester, so I wasn’t recommending to people in this transition, “Oh, completely revise your grading scheme,” that would be not helpful. But if people are thinking about the fall, you know, it might be worth considering thinking about that. There are downsides to mastery based grading too, so I don’t want to act as if it’s this, like, solution to everything, but it might be worth investigating a little bit and maybe incorporating some aspects of mastery based grading into your teaching.
John: And we did have an earlier podcast episode on specifications grading with Linda Nilson.
Betsy: Oh, wonderful.
John: So we can refer people back to that in our show notes as well.
Betsy: Yeah, it’s wonderful.
John: One of the issues with equity, as you mentioned, is that that problem became, I think, much more severe when students were suddenly sent home. On campuses, at least there’s some attempt to equalize that, that everyone gets access to high- speed internet. There’s computer labs in most campuses spread out across campus, and we also don’t have as much of an issue with food insecurity, at least for our on-campus students while they’re there. Suddenly when students are sent home, all those things disappear and the issues of inequity, I think, become a whole lot more severe and it’s something, as you said, we need to be much more mindful of.
Betsy: Yeah, one of the things that I really appreciated from Tom Tobin’s book on universal design, he has a distinction. I don’t actually know if it’s his… it might be his or it might just be generally in the literature on universal design… is distinguishing between access skills and target skills that you want your students to learn versus things they have to know or be able to do to access your material. What I really appreciate, as it helps us think about something as simple as like having a good internet connection, that should not influence their grade, because it’s not one of our target skills. That’s not what we want the grade to be reflecting, whether they had good internet connection. What we want the grade to be reflecting are the target skills that we’re interested in. So I think the way to think about equity here is to focus on any place where things that are irrelevant to your course outcomes are getting in the way of students being able to learn and demonstrate their mastery. That’s where you want to be lenient, that’s where you want to come up with solutions. So for example, in my first-year writing courses in English as a second language, if the thing that you’re assessing is not grammar, if the thing you’re assessing is the way they develop their ideas, the grammar can be a barrier. So there are ways in which you don’t want to grade on that, because your target skills are really about developing ideas. And so that’s a sort of inclusive teaching practice that’s really important in this scenario. What are the things that make it difficult for the students to show up in our Zoom session? And how am I going to create alternatives for them? One thing that we have suggested to our faculty is if you’re doing Zoom sessions, of course, they should be optional. But we also suggested recording it. So the students who couldn’t be there could watch it setting aside the problems with privacy, of course. We can talk about that too. But there’s another wrinkle there too, which is that then that means some students get the interactive, quote, unquote, face-to-face engagement, but the other students only get to watch recordings the whole time. So, one of the things we’ve also said is for equity is also to think of other ways you can engage with those students who can’t come to the Zoom sessions in a way that’s asynchronous or that perhaps at a separate time without overly burdening the faculty member as well.
John: One of the things I’ve done is I’ve shared my cell phone number because all the students have cell phones. [LAUGHTER] I’ve only done that once or twice before in senior-level classes. But this time I’ve done it with all my classes. I did get a phone call coming in right at the beginning when we started recording, and I sent back a text saying I’ll contact you later. But, that has helped because some students do have issues with being able to use Zoom.
Rebecca: And there are certainly tools that you can use to allow you to provide a number that’s not your actual cell phone number that students can still use your phone or texting to communicate.
John: You could use Google phone or you…
Betsy: There’s another one, though, that I’ve used in the past maybe five or six years ago, it’s used in K through 12 environments. Oh, Remind… Remind is the one. Yeah, so I actually used that when I started to realize my students weren’t checking email anymore. I was like, “Oh, this isn’t gonna work, emails not going to work anymore. So I need to find some other way to connect.” But that’s great to be as accessible as possible to your students, but recognizing also that equity issues for us, as faculty. Some of my colleagues can do that more easily than other colleagues who have three kids at home that they’re homeschooling. And so that’s a part of the challenge of this scenario as well, it’s not just what we know is good teaching practice, but also the labor implications for faculty too… that are significant.
Rebecca: Following up on that, that’s a really important consideration is the balance of fairness between both faculty and students because it’s certainly not a situation that any of us signed up for, but we’re all trying to manage. And it’s really possible that we might be in a similar situation in the fall, maybe not exactly the same in that we’ll have a little warning, but it still could happen. So, how do we think about balancing the ability to pivot and make sure that we’re thinking about the ability of teaching remotely without getting too much burden on faculty, but still have really good learning opportunities for students?
Betsy: Yeah, that’s a great question. So I think part of it is trying to think about efficient ways of development. So I was, and you, too, may feel the same way, that that week that we had to transition, I have never seen so much learning happening in a week and so much effort and work and those of us in faculty development probably would never have dreamed. I mean, I don’t know, maybe we would have dreamed that that would happen, but it was a really remarkable thing to see that faculty teaching other faculty can accelerate this in a way that often the model of one instructional designer with one faculty member for six months, that model like that’s how much we need, well, maybe not, now. Maybe we see that if you have to get it done, we’ll get it done, and we can have one-to-many trainings, we can have faculty training each other, we can accelerate that, in some ways, I think is important. But support is also important. So, making sure that we’re supporting faculty as they’re learning what things they can do, and also what we often do in faculty development is talk about efficiencies. So, it’s not just “We’re going to give you a million new pedagogies that we know work, but we’re going to give you a pedagogy that’s actually going to save you time,” and that is really powerful with our faculty and I think we can do the same thing here. So, if we know that there’s a faculty member who has children and has had a hard time with this transition, because they can’t do synchronous Zoom sessions, maybe we talk with them about other alternatives that might be easier for them that they can prep in advance, that will make that transition easier without having to show up at a set time for those synchronous sessions with their children at home. So, it doesn’t solve it, but I do think we should work really hard to come up with the most efficient ways of making the best outcomes possible given the resources that we have. And I think adjusting resources… so we’ve talked about at Wake Forest, outside of teaching and learning, some of our staff, their jobs are no longer really needed, so let’s transition them to other places where we need support. I think you could do the same thing with faculty as well. So, maybe those who have the capability of teaching more or have taught online before, maybe they do more in the fall, but then they get a leave in the spring. There are ways in which you can move things around. Again, I’m not a Dean making these decisions, but being creative about making sure to share the load equally. One question that has come up here, which is really interesting, is that for our faculty who teach more as part of their load, in some ways, this is certainly harder on them than those who have a more balanced teaching and research pipeline, because most of the effort here is in revising courses. Of course, if you have a lab that you have to shut down, certainly that’s a lot of effort. But, making sure we’re mindful of the differential impacts of this transition on our faculty and figuring out ways, not that we’re going to pay them for it, but figuring out ways that we might be able to balance the load moving forward once things go back to quote unquote normal, if they ever do go back to normal [KNOCKING SOUND] …knock on wood here.
Rebecca: I know one thing that I’m thinking about, having small children, is that I’m thinking about all the things that require a little less cognitive load that I’m doing right now while I have a toddler at home, and then when I think I’m going to have daycare again, I’m going to take advantage and do the things that actually require a lot more cognition. And I’m planning to do those at those times, including things like recordings or things like that, that I know I might need to do just to have it in the wings just in case something happens in the fall.
Betsy: I think this is an opportunity for all of us in higher ed to think creatively about how we distribute workload and how we think about the semester and timelines. So, even before this happened, our team read the book Deep Work, and we were just talking about how to create space in our daily work to do intensive deep work, and one of the stories he tells in the book is about a faculty member who stacked his courses so that they were all in one semester. So you know, you have a two-two load or three-three load and he decided to do six in one semester, and then none in the next which normally that sounds crazy, but there’s a way in which that could be really helpful in certain contexts and I think this is an opportunity to think about that. So, those who are doing really intensive work, building online courses, maybe they do a number of them, because it scales, economies of scale, like they do a number of them in the fall and then in the spring, they don’t have to teach… you know, ways of thinking about how to balance this, and then it also would allow us as faculty developers to work with a smaller cohort of faculty, rather than having to work with every single faculty member. Now, I don’t imagine we’ll do that, but it is an opportunity to think of these creative ways of making the workload more equitable as well.
John: And faculty, as human beings, tend to keep doing things the same way as they’ve always done them until there’s some sort of disruption. This certainly has been a fairly substantial disruption, and I think a lot of people, as you said, have learned how to use new tools and at least from what I’ve been hearing, many people now having discovered using Kahoot for quizzing, for example, or using Hypothesis. I’ve been giving workshops on Hypothesis for a while on campus, but not many people adopted it. All of a sudden, I’m getting all these questions about using Hypothesis where people are using it for peer review of documents, where they’re using it in the LMS or more broadly, and I’m hoping that this will continue in the fall. What sort of reactions have you been getting from faculty who are trying some new tools?
Betsy: Yeah, I mean, there’s so many who have said, “Oh, wow, I can totally use this in my face-to-face classes,” and that’s really exciting to hear, that they’re gonna keep it, they’re gonna keep the strategies in their face-to-face courses, or if it needs to go remote, of course, as well. As well as, “Oh, now I know how to use Canvas, so I’ll actually use the gradebook.” Things that are going to be nice for our students as well. Students have been asking for to have a place where they can see all their courses together. I think there was a kind of fear about these technologies in some ways and now that they were forced to do that, “Ah, it’s not so hard.” Now some things are difficult, some things are challenging associated with developing a really well designed online course, but some of these little tools that they have to use in this environment can be helpful in what they’re traditionally doing in their face-to-face courses and I’ve seen many of them say they’re going to do that which is such a wonderful thing to hear as well as pedagogical decisions they’ve had to make about assessment, about universal design, about academic integrity, grading, all the things they’re learning there can also translate back to their courses too, even if we don’t go remote.
Rebecca: And I think all those like crossover areas are ways that faculty can be more nimble. The word pivoting has been used a lot, but I think also being nimble, “I’m using this tool or method and it works both online and in person, so it doesn’t matter which modality I’m using” is something to think about. I did want to just ask one last question related to grading and evaluation, and that’s about motivating students to achieve our learning outcomes when there are so many other things in the world right now that we might be thinking about.
Betsy: Yeah, that’s a great question. I often like to quote the former Secretary of Education that said, “There’s only three things that matter in education; motivation, motivation, and motivation.” [LAUGHTER] So motivation is super, super important when we think about how students learn. We can design the coolest evidence-informed course and design, but if students aren’t motivated, it doesn’t matter. So thinking about our student contexts, and their motivation is really central to their learning, let alone how we’re going to grade them. And so there are a number of things we know that lead to motivation. Sort of important is the students have a choice and that they have some agency or ownership over what’s happening, and so I know a lot of my colleagues at Wake Forest did this and I did as well, is when we made this transition is to ask the students, “So what’s going on with you? What’s your preferences for how we restructure the course? How would you like to learn moving forward?” and to keep being in conversation with our students. And what I did, for example is, now again, I had the flexibility to do this in a religious studies course, but I basically threw out that project at the end of my semester, and so instead had time to say, “What do you want to read about? What things about religion do you want to know about?” And so we’ve been reading about religion in violence, religion in COVID-19, just things that they’re interested in, and that has allowed me to help a little bit with motivation is to just engage with the students a little bit more, but it’s tough. Typically, I think a lot of times we think of there’s carrots and sticks related to motivation, so you can certainly use sticks if you wanted to with grades, but that often has unintended negative consequences. So the more you can do carrots, which would mean thinking about what do they want to learn. I also think that my students, at least at Wake Forest, really miss each other. It’s a really communal place and they really miss each other, so creating opportunities for them to engage with each other, even if I’m not there. There are lots of little interesting activities I’ve seen people suggest where they get together and have video chats in groups, and then record them for the professor. Creating opportunities for them to spend time with one another… They will just want to spend time with each other, whether it’s about learning or not, but if you sneak in the learning, that can be something that will motivate them too, but the reality is some of our students, there are too many other more important things on their plate and we need to acknowledge that, and so I’ve tried to make my students feel that it’s okay to say that, that I’m not disappointed in them if they don’t do as well or if they choose to take it pass-fail that like, “Look, this is just a religious studies class. It’s one class among many. There are many other more important things happening right now. Yes, we want to help you learn if you want to learn, if you want to complete the course and get the credits you get, but we all know that there are other things that are taking our attention away right now, and that’s understandable,” and being sympathetic about that, I think, can also be motivating because they’re not demoralized if they don’t do well. “That’s okay. She understands. I’ll give it a try next week.”
Rebecca: I think that humanity piece is key, both for students and for faculty, and it makes people feel like they have a sense of belonging but also that belonging is often motivating.
John: We always end with the question. What’s next? Which is a question on everyone’s mind right now.
Betsy: I think for me, and maybe this is unique to me or my colleagues who are in teaching and learning centers and in faculty development. What’s next is I want to have some time to reflect back on what I’ve learned about faculty from this transition, and what I’ve learned about faculty development from this transition, and we talked a little bit about this in our earlier conversation, but I was really struck by what I saw on that week off that we had to learn how to improve. I mean, again, I need to spend more time thinking about this and what we’ve learned, but one of the things that was really striking to me was how important having a dedicated time to talk about teaching was, like, “This is a week where you’re going to work on your classes, faculty,” and often we talk about “How do we motivate faculty to do professional development?” We think about funding, we think about course releases or making it enticing in other ways, but my hypothesis that we learned through this transition is that time and dedicated time and a sort of cultural commitment to saying we’re going to take two days to focus on our teaching. What if we did that every year, and there are some schools that have a faculty development day, but what if we took three days every year where everybody got together and talked about their teaching. And I think that’s just one example of something that I would like to reflect on, but I think there are many other things that have happened in the past three weeks that can help inform the way we think about faculty development and I’m really excited to think about that as we, as a center, think about how we work with faculty going forward.
John: Things like that Facebook group that you mentioned, and we have a similar one, has been really helpful in building more of a community than I’ve ever seen before.
Betsy: Absolutely. Yep, I completely agree.
John: Well, thank you. This has been fascinating, and we wish you luck.
Betsy: Thanks for inviting me. It was great to talk with you all.
Rebecca: Yeah. Thank you so much.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Savannah Norton.
Grade change requests in college are relatively rare, but when they do occur, evidence suggests that male students make the request more often than female students. In this episode, Dr. Cher Li joins us to discuss these gender differences in grade change requests in college and why they might occur.
Cher is an assistant professor of economics at Colorado State University. Her research focuses on how public policies and social institutions affect the decisions of, and outcomes for, women. She is also a co-author of a January 2020 National Bureau of Economic Research working paper that investigates gender differences in grade changes.
- Li, C. H., & Zafar, B. (2020). Ask and You Shall Receive? Gender Differences in Regrades in College (No. w26703). National Bureau of Economic Research.
- Colorado State University
- Leibbrandt, A., & List, J. A. (2015). Do women avoid salary negotiations? Evidence from a large-scale natural field experiment. Management Science, 61(9), 2016-2024.
- John List, the Kenneth C. Griffin Distinguished Service Professor in Economics and the College at the University of Chicago.
John: Grade change requests in college are relatively rare, but when they do occur, evidence suggests that male students make the request more often than female students. In this episode, we discuss these gender differences in grade change requests in college and why they might occur.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
John: Our guest today is Dr. Cher Li. Cher is an assistant professor of economics at Colorado State University. Her research focuses on how public policies and social institutions affect the decisions of, and outcomes for, women. She is also a co-author of a January 2020 National Bureau of Economic Research working paper that investigates gender differences in grade changes. Welcome, Cher.
Cher: Thank you. It’s great to be here.
Rebecca: Today’s teas are:
Cher: I have Oolong tea from the Ali Mountain area in Taiwan. It is one of the best tea regions in Taiwan.
Rebecca: Sounds good.
John: Nice. And I have Cranberry Blood Orange black tea.
Rebecca: And I have a mango black tea today. John’s looking at me a little strangely because it’s very unusual for me. [LAUGHTER]
John: We’ve invited you here today to discuss your recent working paper with Basit Zafar on regrading requests and outcomes. Could you tell us a bit about the study?
Cher: Yes, this study is about gender differences in grade changes in college, and we look into the differences and why are the causes for the gender differences?
John: What prompted you to look into this?
Cher: So this research is actually motivated by my personal experience. In the last six years, I had many male students, but only one female student asking me to bump up their grade at the end of the semester. And I know from previous studies that men are more likely than women to negotiate for their salaries and promotions, and that made me wonder if these kinds of differences actually started earlier when they’re still in college. So, that is the motivation for this study.
Rebecca: Can you talk a little bit about how you conducted the study?
Cher: Sure. We tackled this research using three very different sources of data, including administrative records from a very large four-year university and we also use surveys and laboratory experiments. So, the first data source is a very unique administrative data from Colorado State University between 2010 and 2016. We have over 1.3 million observations of grade changes across all the departments, and this data set is very unique because it recorded not just the final grade, but it also recorded any changes associated with it. So we focus on the changes made by instructors, and our analysis showed that, although grade changes are pretty rare events, 95% of the time they will change to a better grade, and among those positive changes, men are about 18.6% more likely than women to get a better grade. And the difference in grade changes cannot be explained by the characteristics of students, instructors, and classes. However, the administrative record has several limitations. First of all, it doesn’t provide information on the cases when any regrade request was rejected by instructors, and it also doesn’t tell us about the grade changes during the midterm or any assignment throughout the semester. And finally, it doesn’t tell us “What is the scenario that caused the gender difference in grade changes?” So we can think about at least three different scenarios that all lead to the same data pattern. For instance, the first one is male students are simply more likely to ask, although instructors treat men and women equally… so, that is a possibility, given the previous literature evidence. And the second potential scenario is that male and female students are equally likely to ask, but somehow instructors are treating men more favorably, and if it is this scenario, we’re going to have something to worry about about discrimination. And the third scenario we thought about is, there may be a difference in terms of the timing of asking. So if men procrastinate while women are more organized throughout the semester in they ask their instructor to make changes while they find any errors during the semester, then they have no need to ask at the end of the semester. So, to examine the scenarios, we conducted two surveys, one on instructors and the other one on students, and we checked whether the reports are very similar across different surveys. And surprisingly to us, both surveys show pretty consistent patterns: that male students are indeed more likely to ask, and we find no evidence that instructors are treating males students more favourably. And we also didn’t find any support that there is a timing difference in terms of their asking.
John: That’s certainly consistent with what I’ve observed myself, that males have been more likely to ask both during the semester and after.
Rebecca: I don’t think I’ve gotten that many requests, generally.
John: Well, I haven’t either, I may get one a year or two a year out of like seven or eight hundred students, but…
Rebecca: I can’t think of a single one. I know that I have but I can’t think of a particular…
John: Well, your classes are smaller, too. So, that could be a factor.
Cher: Yeah. So I also typically only got like one to two students per semester at making that request. So it jumps out at me when there is only like one female student throughout the whole six years of my experience, that really makes a difference to me when I compare the frequency in terms of they approach me at the end of the semester.
John: One of the nice things I liked about it was your incentivized controlled experiment that looks at why there may be differences. Could you tell us a little bit about that aspect of the study?
Cher: Definitely. So focusing on the gender difference in asking, we did this lab experiment to answer that question, in particular in the lab setting. The participants basically take a quiz of 20 questions, and their payoff is dependent on the grade in the quiz. So we tell them we’re going to give them the grade. However, we also tell them the grade is noisy. So, we randomly picked three out from the 20 questions to randomly grade them. The grade we tell them could be their true grade, but it could also be higher or lower than the true grade. And then they have to decide under 10 different cost scenarios, ranging from paying up to $3.50 to being paid to $1, whether they are willing to pay the cost in each of the scenarios to get a regrade. If a participant chooses to pay the cost to ask for a regrade, then the true grade is revealed and their payoff is adjusted and we subtract the cost they have to pay, and if a participant chooses not to ask for a regrade, then the initial grade becomes final and they will be paid accordingly. And we find that, as long as the cost is positive, so basically the participants have to make a payment, men are always more likely than women to pay the cost to ask for regrades, and the gender difference in asking cannot be explained by their difference in risk aversion. We find that other majors, including their difference in confidence level, in their uncertainty about their performance, and personality traits explain nearly half of the gender differences in asking.
John: How accurate were the students’ metacognitions in terms of how appropriately they made a decision to request regrades? Did they gain by that on average, or did they lose?
Cher: So in general, they have pretty good guesses about their expectations in terms of how well they do in the quiz. However, their decision here really depends on their attitude in terms of pursuing more aggressively in the regrade or not. And we find that, in general, most students who decided to pursue the regrade were likely to get a better outcome, and we find that depends on the cost, basically. So when the cost is low, the more likely a student will get a better outcome, because they are facing a very low cost to offset the gain they got from a better grade. And the reason that they are more likely to get a better outcome is due to our lab experiment design, we make it asymmetric, basically. So, we are trying to mimic the real world scenario. So from the administrative data, we find that 95% of the time, students got their grade change to a better grade. Only 5% of the time it got changed to a worse grade. We intentionally designed our lab experiment to allow that asymmetry. So basically, when a student asks, they will have a higher chance to get a better grade than to get a worse grade.
John: How much more common was it for males and females at low, and then again, at high prices?
Cher: When there is a cost that really they have to pay out of their own pocket, around like half of male students are willing to pay the cost, while only about one third of women are willing to do so. However, when the cost is zero or they got paid to do it, it’s very similar. I think their percentage of requests is very high. It’s like nearly 80% when they were getting paid $1 to do so.
John: What happened when the cost was high? I believe you found some interesting effects for males at that.
Cher: So when the cost is high, we find that the gain from asking actually disappeared, because now the high cost offset the gain made by men. So although men are more likely to ask, they are not always made better off, so they are only gaining relative to women when the cost is relatively low. When the cost is above like $1.50, we see that men are also more likely to see their payoff decrease compared to women.
Rebecca: Can you talk a little bit about the gender difference in the regrade requests and what might explain those differences in general after looking at those three different approaches that you’ve taken to study it?
Cher: So we find that, it seems to be, there are some subjective beliefs that are different to begin with. So first of all, there are some evidence from other studies that men and women are different in terms of confidence level, and men tend to be more overconfident than women in a lot of contexts. So, we explore that too in our study, and we find that indeed, when we look into each of the questions they answer, we also ask them how confident they are on each of the questions. And we find that when they got the answer wrong, men and women are similarly over optimistic about their accuracy. However, when their answers are actually correct, women are much less assertive about it, so they are less confident about their correct answer. So that is part of the reason that is driving the gender difference in asking, and the other factor that we found are important in terms of the gender difference in asking is about their perception about the potential outcome, so basically the uncertainty they face when they make that decision. So we asked them about like, “What is the number of questions you think you got right?” And they will give us their best guess. And then we ask them then for that scenario, “What is the likelihood you think that’s gonna be happening?” So they will tell us like, oh, if they are very confident about “I’m getting 15 questions right out”, maybe assigning 100% as the probability for that outcome, if I’m super confident that is the outcome. But then if I’m not very confident about my guess I maybe say “Okay, 50% probably,” then I’m not really certain about that outcome. So we find that when men and women are giving us those guesses, women are much less certain about their guess. So they are likely to think it could be different outcomes, maybe it’s 14 or 13 questions they got it right, so they would also assign the probability of other outcomes that they anticipated. And that also explained a very large proportion of the gender gap, and we also find there are some differences in their personality traits. For instance, women are more agreeable than men, so they may be more agreeable to the grade they received and less likely to challenge it in terms of the context of asking.
Rebecca: When we start thinking about the impact this might have on GPAs or their academic standing, how does this really impact men and women differently?
Cher: So the overall impact on GPA depends on how we measure it. For those people who ask, we did a back of an envelope calculation and we find that for the female students, the accumulative effect from asking is up to .43 points in their overall GPA. But for male students, the number is a bit higher, they could receive up to .47 points as an improvement in their GPA, conditional on asking.
John: Which is a non-trivial difference in terms of that, especially if you were also to extrapolate and think about the possibility that this is also occurring during the class.
Cher: Yeah, we take that into account in extrapolating, and that is a cumulative kind of effect we anticipate. If a student reports a fraction of courses they asked for a regrade, that would be kind of cumulative effect overall.
Rebecca: Do you have any idea of what the not asking is doing, maybe to women, if they’re less likely to ask in the first place, how that might be negatively impacting their GPAs?
Cher: So in terms of not asking, you are comparing to those who ask, so that raw difference could be low depending on who we are comparing to, or we can compare it to those who actually are successful throughout their time, then that difference will be up to like .4 to .5 points in terms of the GPA.
John: At the start of this, you talked about how there’s a lot of evidence out there that women are less likely to bargain for higher wages for promotions and so forth. From those studies, is there a lot of evidence of similar types of causation?
Cher: So most of the studies we saw before look into their different tendency to ask, and one of the particular research I think is very interesting is in terms of the context. In the study made by other economists, including John List in University of Chicago. In their study, they did a field experiment, and they experiment the job applicants’ attitude in terms of negotiation for their salary. In their experiment, there are two settings, one is the salary is ambiguous if there is no hint or any information about it. And then they found under that context, men still negotiated for their salary, but women in that context tend not to negotiate and instead, they signal their willingness to accept a lower offer. In the separate setting, when they make it clear in their advertisement that the job salary is negotiable, then there is no gender difference in terms of asking; men and women are similarly likely to negotiate their salary when that kind of information is clear, that the social cue is you are able to negotiate. In that case, there is no gender disparity, and women are similarly likely to ask. So we find it relevant in our context as well because a lot of time professors do not put a clear policy about regrading, so students may have to take that into their own guess, whether this is an appropriate behavior to make and approach their instructors about it. So I found that kind of ambiguity is relevant in terms of the context when the grade change policy is ambiguous.
John: I’d be really hesitant to put in a procedure explicitly for negotiating grade change requests, but I could see how that could make a difference when it’s normalized as an appropriate policy. But are there any ways that you can see, based on your results, where we could try to compensate for this, where we could try to offset this difference in grade change requests between men and women?
Cher: So in addition to making a transparent regarding policy, I’ll be arguing, first of all, we are not trying to push women to over asking, basically. And a lot of time, we also find in some other literature, they find when you pressure women to ask, the results are not always favorable. So that is truly not our intention. The point we make about transparency in terms of regrading is to make that kind of policy more formal, and sometimes maybe also imposing some cost in terms of the regrading in terms of like, the professor may be able to regrade the entire exam or the assignment, typically, would be, like, not enforced, when students are just bringing your attention to one particular question they think they should get more points and argue for it. But if they have to come through a particular procedure, for instance, making those requests through like documentation, or they have to make some arguments for the points they think they should earn, it would make it clear to students in terms of what they have to jump through, and then other policy that we think may be helpful is in terms of providing better signals. In terms of the performance, for instance, “How is the grade assigned? What is the metric? What is the rubric that a professor uses to assign a higher grade when the student has performed well?” and just really to show them like these are the guidelines why a student would get an A instead of getting a B. What are the distinctions between the different grades, for instance, to make it clear to students why they’re getting their grade, especially when the grading is based on something that’s more ambiguous, like essays or something.
John: So if you grade essays, make sure you provide a rubric that provides a clear delineation and makes a grading process more transparent to everybody.
Cher: Exactly. So when the women are getting a grade and they see, for instance, “I’m getting a B, but I’m qualified for getting an A based on the rubric,” then I will be more confident that I am eligible to ask the professor in terms of whether my grade is accurate or not.
Rebecca: I think it’d be interesting then, the next step would be to see if it’s not just gender differences in asking, but then also, if it’s students that grew up in lower income families or other kinds of situations that might be less likely to ask in the first place.
Cher: Yeah, I think that would be interesting, but we do not have that much information about student income, so we are unable to really look into it. But we do look into some of the different demographics, including their race composition, and we do not find much differences when we account for those differences. And it doesn’t seem to be the main reason that’s driving those gender differences, but there might be some other interesting questions that could be asked in addition to the gender disparity.
Rebecca: One of the things that always comes to mind is if you start putting in policies or things that make it more transparent, it still means that a student would have to take action in some way, so they would have to feel empowered to be able to take those actions. So it’s always curious to kind of think about those power structures beyond just gender.
Cher: Right, I agree.
John: With providing something in a syllabus for challenging grades would probably help reduce those based on what you said.
Rebecca: Especially because it establishes that behavior as something that would be appropriate in that context.
Cher: Yeah, exactly. Because when we ask students in our survey, “Why don’t you ask?” because we start asking them whether they ever consider asking their instructor to reconsider their grade at some point, and if they do, we follow up in asking for the courses they consider to make that request and the reasons they decided not to pursue it, and we find that a non-trivial fraction of students actually just never thought of it. [LAUGHTER] So they don’t think that is appropriate or that is available, so that is one of the reasons we believe it’s good to have a transparent policy in place.
Rebecca: So we usually wrap up our conversations by asking “What’s next?”
Cher: Okay, so the future, so right now our current research is going through the review process, and we have thought about a few directions that we can follow up on the study. One direction is to look into the gender interaction between students and instructors, and we got this question a lot when we presented our work. And a lot of people are curious whether female instructors are getting more requests, and what are the results when you consider those gender interactions? And I think that could be one of a very interesting to look into. And one very interesting thing, I think, from our current study is, when we design our lab experiment, we intentionally take those mechanisms out. So, in our lab design, we shut down a lot of potential mechanisms that could be very important in real life. For instance, the fear of backlash when women ask, which I have found some evidence in current literature, that is totally shut down in our lab experiment. So that is not the reason that’s explaining the gender differences in terms of asking, based on our lab experiment. And we also shut down the gender interactions because the participants in our experiment do not have to interact with a real person. So they just have to make that decision whether they want a regrade or not, and their wish will be granted, basically. And that also shut down any potential gender differences in terms of negotiation skills, if men have been negotiating all along, since they were little, they may become better at it, and that may be contributing to the gender gap in that case.
Rebecca: Those sound like some really interesting follow-up studies.
Cher: Yeah. And we were also thinking about, one very cool thing that could be done, is to follow up with the students into their labor market experience and see how their negotiation behavior may actually lead to some differences in their labor outcomes. Are people who tend to negotiate their grade in school are getting better outcomes when they are working for others, for instance.
John: And I know many alumni programs collect data on first salaries out of college, and that might be an interesting place to do that linkage since you have the administrative data there.
John: Excellent. I really enjoyed your study, and I like the way you designed the experiment to elicit some detail on why this is happening, because the studies I’ve seen, mostly… just in terms of the labor market… just report that this is occurring, but really haven’t done as much, other than the study you mentioned with List and others, haven’t done that much to try to figure out why it’s happening. So I really enjoyed it.
Cher: Thank you so much. We enjoy the research ourselves, too.
Rebecca: Well, thank you so much for joining us.
Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.
Students generally receive lower grades in STEM classes than they receive in other disciplines. In this episode, Dr. Peter Arcidiacono joins us to discuss how these differences in grading policies across departments can help to explain the relatively low proportion of female students majoring in many STEM disciplines. Peter is a Professor of Economics at Duke University.
- Peter Arcidiacono, Professor of Economics at Duke University
- Ahn, T., Arcidiacono, P., Hopson, A., & Thomas, J. R. (2019). Equilibrium grade inflation with implications for female interest in stem majors (No. w26556). National Bureau of Economic Research.
- Johnson, V. E. (2003). Grade inflation: A crisis in college education. Springer Science & Business Media.
- Dowd, A. C. (2000). Collegiate grading practices and the gender pay gap. Education Policy Analysis Archives, 8, 10.
- Todd Stinebrickner, Professor of Economics at the University of Western Ontario
- Ralph Stinebrickner, Emeritus Professor of Mathematics and Computer Science at Berea College
- Stinebrickner, T., & Stinebrickner, R. (2012). Learning about academic ability and the college dropout decision. Journal of Labor Economics, 30(4), 707-748.
- Wellesley College
- Students for Fair Admissions v. Harvard College
John: Students generally receive lower grades in STEM classes than they receive in other disciplines. In this episode, we investigate how these differences in grading policies across departments can help to explain the relatively low proportion of female students majoring in many STEM disciplines.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Our guest today is Dr. Peter Arcidiacono. Peter is a Professor of Economics at Duke University. Welcome, Peter.
Peter: Thank you.
Peter: Thanks for having me.
John: Our teas today are…
Peter: Green tea. I got it specifically for this conversation.
Rebecca: Excellent. I have English afternoon.
John: I’m drinking Spring Cherry Green tea.
Rebecca: Oh, that’s a little switch for you.
John: I’m looking forward to the spring now that it’s 10 degrees outside here in Oswego in January when we’re recording this.
We’ve invited you here to talk about your December 2019 working paper co-authored with Thomas Ahn, Amy Hopson, and James Thomas, on Equilibrium Grade Inflation with Implications to Female Interest in STEM Majors. In this paper, you know that disciplines with harsher grading policies result in higher earnings than the earnings received by those majoring in disciplines with easier grading policies. Can you first describe what you mean by harsher grading policies?
Peter: Well, harsher grading policies… you look at average grades in STEM classes and economics, they tend to be a lot lower than in the other social sciences and in the humanities. In the school we look at, which is University of Kentucky, it’s a third of a grade lower. And that holds true across many other colleges I’ve looked at, be it the Duke University or Berea College, it just seems to be a general pattern that the STEM classes and economics tend to have lower grades. And in fact, there was a very nice book written about this with Duke data called Grade Inflation: a Crisis in Higher Education or something like that written by Valen Johnson a few years back, and it was making that exact point about the big grading differences across disciplines.
John: How does your model explain the grading differential across departments?
Peter: Yes, so that to me is where the big contribution of the paper lies, though. I don’t think our model’s particularly satisfactory in this regard. I think lots of people have noticed in the past these differences in grading standards. Where I think our contribution is, is thinking about how those grading standards impact different groups of individuals, but also in what you just asked about: What are the sources of this? And I think a portion of it is certainly demand for courses. So, at the school we look at, the STEM classes are twice as big as the non-STEM classes. An analogy from my own experience… when I first got to Duke, no one signed up for my grad labor class. And that was a bit of a disaster, because that would mean I’d have to do a new prep for some other course. And what I did is I bribed two students to take the class. And the way I did that was I said, “The class is going to be: we’re going to write a paper together.” And it was a great experience for them. They did get a paper, they had a revise and resubmit when they went out in the job market. And what do you know, they both got A’s in the class, but now I’m a little more popular. Not everyone gets A’s and I’m not co-authoring papers with everyone in the class. So I think part of it is these differences in demand for the classes affect things. You hear about weed-out classes, they’re weed-out classes because there’s plenty of students who are interested in taking that class, so they can afford to do that. If you’re in a discipline that has low demand, you’re not going to have a weed-out class because you’re not gonna have any students. So, I think that’s a portion of it. A portion of it can also be things like differences in preferences, maybe it is the case that some disciplines are just meaner than other disciplines. And a third component may be that it’s just easier to grade those classes, it might be easier to grade a math problem than an essay. I’m a little skeptical of the last one, in part because when I talked to high school students, it doesn’t appear to be the case that the math classes are that much easier than the humanities classes, for example, or grade that much harder than. And I think part of that is in high school, you have to take those humanities classes and when you get to college, you have a lot more choice and that influences things.
John: This also seems to relate to the academic labor markets. Because there’s so much demand for classes, there’s often more demand for faculty. So, faculty are not under as much risk of losing their job if they don’t fill their classes. While in markets in those disciplines where grades tend to be lower, labor markets tend to have a little more slack in them. And faculty perhaps may, using the analogy you used before, have to be a little more generous to make sure their classes fill so they can keep their jobs, particularly when there’s a large component of adjunct faculty and contingent faculty.
Peter: I think that’s exactly right. I’ve heard some economics professors brag about their low teaching evaluations. I just thought they can get away with that in other disciplines.
John: And we see that in many STEM fields as well, at least anecdotally… certainly not on our campus…[LAUGHTER] But certainly many STEM disciplines do have that reputation of bragging about keeping their grades low and having high standards and such things.
Peter: That’s right. You know, in my view, it’s when you change what you grade around, that doesn’t necessarily mean that the standards… how much work is required in the class may stay the same. And that’s where, I think what’s interesting about men and women, is that women actually study much more than men. So, if you change the workloads, that’s more of a deterrent for men than it is for women. So, you can think about different mixes of either giving high grades but with high workloads, or lower grades, with lower workloads and student life, on average being different between those things, but they won’t be indifferent between men and women. At least on average. Obviously, this is just an on average statement. There are some women who are not interested in studying hard, and some men who are. On average, that appears to be the case.
John: And you do know, though, that the STEM disciplines not only assigned lower grades, but they also require larger workloads based on the survey data you were looking at, I believe.
Peter: That’s right, substantially higher workloads. And that was my own experience… that where I went to college, I guess economics hadn’t quite crossed over to STEM at the time, but I started as a chemistry major, and there the formulas are a lot more complicated and you had to memorize them. In economics, I didn’t have to memorize the formulas and they were much simpler than those in the chemistry major and I had to work a lot harder in those chemistry classes.
Rebecca: Can you talk a little bit about how the difference in grading policies impacts underrepresented groups in these different disciplines? So in STEM, for example, where you’re saying there’s these stricter grading policies, there’s less women, there’s less underrepresented groups in general.
Peter: That’s right.
Rebecca: How do those things connect? Can you talk us through that a little bit?
Peter: Yeah, actually I had caused quite a stir here at Duke back in 2011, talking about underrepresented minorities, because you do see, for example, African Americans at Duke… They came in just as interested in the STEM classes in economics as white students, but they left at a much higher rate. And so the negative headline was something like, you know, “Potentially racist study says Black students are taking the easy way out,” and that was not the case. It was that they’re coming in with different levels of preparation, due to pre-college inequalities. We have serious differences in pre-college education. And that’s resulted in this coming in and then finding that it’s very hard to succeed in the STEM disciplines. For women, the story is different. And there, because women come in just as prepared as men and are interested in studying a lot more. For them, it’s not the workloads that are affecting this, but more the lower grades themselves, given that they’re working so hard, and then to see these low grades seems to have a bigger effect on women than men. I think one of my very interesting result from some of the work that Stinebrickner and Stinebrickner have done is actually asked men and women, “What grades do you expect you’re going to get?” And then they’ve showed them their grades later and said, “Now, that didn’t work out quite as you expected.” They got lower grades. Why do you think that was, and men were much more likely to say it was bad luck. And you could see that they actually believed it was bad luck, because they continue to take the same classes where they were getting low grades, and they didn’t change their study habits. Whereas women were much more likely to say that it was something about them, either that they needed to study more, or they weren’t as prepared as they had hoped to be for the class. And then you can see that they ended up being more likely to switch to different types of classes and to change their study habits.
John: I think you also found that women tended to not just equally prepare, but I think women came in with somewhat higher GPAs, if I remember correctly.
Peter: They do come in with higher high school GPAs, for sure, and they get better grades in college than the men, both in STEM classes and in non-STEM classes.
John: So the difference seems to be that women are more sensitive to the lower grades or they tend to blame themselves for the lower grades, while the men tend to be less cognizant of the fact that they may be the cause of the lower grades.
Rebecca: I feel like I’ve been socialized to blame myself for not being adequate.
Peter: Yeah, it’s been remarkable, you know I have five boys. And they’re not socialized that way, despite my best efforts to get them to really care about their grades and to think that studying even matters. Here I am, a researcher who knows that studying matters, and yet some of my boys are convinced that if they do poorly, it’s bad luck. It has nothing to do with their studying.
John: One of the things I assigned in my labor class is a reading from, I think it’s a sociologist, who noted that women seem to be more concerned with what she termed “local status,” while men tend to be more concerned with “global status.” Basically, the argument is that men tend to be more concerned with their income expectations and the impact that the courses in the majors have on their grades, while women tend to be much more concerned with the relative grades and seeing grades as a perception of their worth or their value in some sense, which makes them much more sensitive to that grading differential between majors. While men tend not to be as concerned about their local standing, their relative status, while they tend to be more concerned with how the degree will impact their future earnings.
Peter: And I think it’s compounded by the fact that I don’t think people know when they go to college how different the grades are. I think that they figure it out rather quickly, but at first they just don’t know that they’re graded so differently. I mean by the time they figure it out, they’ve already made the switches, which is too bad.
Rebecca: So what do we do so that there’s more representation in these disciplines?
Peter: Well I think at a minimum, that universities should at least have an obligation to tell people, “Here’s what the average grades are in these different classes,” so that students can make informed choices. I personally am in favor of doing something where, like they did at Wellesley, where they put in something that either capped the average GPA in the class, or having all the classes that are above a certain size give the same median grade so that people are not having their decisions distorted by the fact that one department is giving lower grades than another department. It’s not as though they’re learning less in these STEM classes, just because they’re getting these lower grades. They’re actually working very hard in these classes, so, it would be nice if that was somewhat remedied.
Rebecca: I wonder if even knowing the information across institutions and knowing that people get lower grades in these fields, no matter what school they’re at, would actually be useful to those groups.
Peter: Oh, I think it’d be very helpful. Yeah.
Rebecca: Not just within your own institution.
Peter: That’s right. Yeah, I think that’d be great. Because then you know what you’re getting into. And maybe when you get that B in the economics class, you’re not feeling like you failed, that that is a much more typical grade than what you expected.
John: I think a lot of people might object to the notion that there should be the same GPA on the grounds that you’re forcing students into this curve where if you happen to have a class that does really well you might be lowering their grades, which doesn’t always happen in the STEM field, but I think there might be some resistance to that. But the notion of transparency and making people more aware of that early on might help reduce that differential effect by gender.
Peter: That’s right. And I think also making departments aware… the implications that the level of their grading has on things like the gender composition, should matter. I don’t want the way I grade my class to be a deterrent for women taking economics classes. I think that that’s bad. And I’ve changed the way I grade a bit in response to this. If other disciplines are going to give much higher grades, and I want to have a good gender representation in my class, I’m still assigning the same amount of workload. So I’m not making the classes easier, but I am changing the level that I’m curving around, and I typically teach a class that has over 100 students in it.
John: And this is especially important because a large share of the gender differential in earnings is tied to major choice, because the fields that women are more likely to enter tend to offer much lower wages on average, while the STEM fields tend to offer much higher wages. So the differential impact on major choice has quite a bit of an effect on the gender wage gap.
Peter: That’s right. And this is potentially a very low- cost way, at least from a pure financial standpoint, to changing that. It’s not going to solve all of the gender issues in economics or in STEM fields. It’s not going to come close to solving it, but it can put a dent in it, where it’s often very costly to change something like that. Changing a climate is expensive and difficult.
Rebecca: I think I would have been one of those students that would have moved. I’m a person who does a lot of coding and things and could easily have been in computer science. But I’m not, I’m in graphic design, and it’s a field that is much more populated by women. I got much higher grades in the things that I did, and I was very sensitive to grades. So as a student, if it was something that I didn’t do well in, then I thought I couldn’t do well in. And I had a very fixed mindset when I was earlier on in my career. Obviously, it’s much different now. I do much different things and explore other things. But at that time, I think I was incredibly sensitive to that. If you came in as an A student and then it’s like, all of a sudden you’re not getting A’s, it’s like “Woah…”
Rebecca: “…maybe college isn’t for me, even.”
Peter: Yeah. And that’s just really unfortunate that decisions will get based on nominal differences in these grades, that don’t really have content outside of something that is relative, relative to whatever the professor’s curving around. Another advantage to doing something like this, too, is potentially getting spillovers. Because once you change the gender composition a little bit, then that makes the environment more comfortable. And then that can have multiplying effects.
John: So, why don’t departments want to do this to improve the gender composition of their classes and to get more students in their classes and persisting to graduation in their majors?
Peter: Well, this is sort of where the equilibrium aspect comes in. Because even if you want to do this, everybody ends up ratcheting up their grades, because some departments really need more students. And in order to attract students, you’re going to have to do something if the students don’t have an innate demand for your field. And people have been talking about this for some time, that demand for the humanities has been falling and what to do about it. So, I think that the pressure’s sort of on both sides, even if the STEM departments decided, “Look, we want to have our grades be a bit higher for this reason,” that’s going to lead to response by non-STEM professors as well. So I think it’s hard to get rid of the grading differences across fields, absent doing something that’s pretty draconian.
John: By imposing a grading standard that’s equal across departments.
Rebecca: I wonder if initiating conversations between the humanities and STEM fields about the interdependence there actually are on those two fields and that people in the humanities should learn things about STEM and people in STEM should learn things from humanities and that there could be more of a relationship there that would actually help start to equalize some of those needs, because there could be some referrals to certain classes or other things that might help boost some of those relationships. I don’t know.
Peter: That’s right. So right now we’re playing sort of a non-cooperative game and move it more towards a cooperative one. Yeah.
Rebecca: Yeah, it’s competitive rather than thinking about “Why are we at a liberal arts institution? And what does an environment like that or general education offer?”
John: Well think about what would happen, though, if grading standards were relaxed in the STEM disciplines. More students would major in them, enrollments would tend to fall in the non-STEM field, and that would tend to lead to fewer jobs for non-STEM faculty. And maybe that’s not something that departments would react well to, and they face the same incentive to just inflate grades.
Rebecca: I don’t think my suggestion was necessarily to do something about the grades directly, but rather to start helping to increase some demand for humanities and helping out some of those…
John: You mean, more interdisciplinary programs?
Rebecca: Yeah, I’ve been thinking a lot about, even for my students, the benefits of them exploring a lot of different disciplines to be more effective in their own, and understanding how other people problem solve, and think through problems, and explore the world.
Peter: I really like that too because if you end up providing these skills that are then rewarded in the marketplace through taking some of the STEM stuff, but combining it with the humanities, that will increase demand. And then that will naturally even things out a bit more.
John: And you could move more from STEM to STEAM.
Rebecca: Hey, I didn’t say it, but I’m totally a STEAM gal. [LAUGHTER]
John: Well I was expecting that at any moment. And just in case some of our listeners don’t know what STEAM is, would you like to…?
Rebecca: Yeah, included the A …it’s for the arts, [LAUGHTER] which I am a part of.
Peter: Yeah, they’re doing that at my kids’ high school, you know, having a STEAM program to try to get that integration happening a bit more.
Rebecca: There’s some research happening about the need for these creative outlets to help with innovation and things in STEM fields. So, there’s certainly some research suggesting this interdisciplinary approach to all of our disciplines could actually improve the work that we do.
Peter: Makes sense.
John: And many STEM students tend to specialize as much as they can in their field. So, that broader training could be useful in the future in making them more creative perhaps.
So, we always end with the question: “What are you doing next?”
Peter: Well, it’s been interesting, nothing quite related to this. I got derailed because I’m actually an expert witness in the Asian American discrimination case against Harvard, and that is taking up all of my time. We released a couple of papers on legacy and athlete preferences at Harvard, and we’re going to be writing some more out of my reports for that case. That’s what’s keeping me busy.
John: Very good.
Rebecca: Well, thank you so much for joining us, this was really interesting, and I hope that it launches more conversations across disciplines.
Peter: Thanks for having me, I really appreciate it.
John: Thank you.
John: Editing assistance provided by Savannah Norton.
There are a number of websites that market themselves as study tools and tutoring services that are used by students as tools for cheating. In this episode, Dr. Liz Schmitt joins us to discuss how these sites work and the steps faculty can take to protect their intellectual property and the academic integrity of their courses. Liz is an economics professor and Acting Chair in the Department of Economics at SUNY Oswego.
Rebecca: There are a number of websites that market themselves as study tools and tutoring services that are used by students as tools for cheating. In this episode, we’ll discuss how these sites work and the steps faculty can take to protect their intellectual property and the academic integrity of their courses.
Rebecca: This podcast series is hosted by John Kane, an economist…
John: …and Rebecca Mushtare, a graphic designer.
Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.
John: Our guest today is Dr. Liz Schmitt. Liz is an economics professor and Acting Department Chair in the Department of Economics at SUNY Oswego. Welcome, Liz.
Liz: Thank you.
Rebecca: Today’s teas are:
Liz: Cocoa and coconut.
Liz: It is. It’s my favorite. And I actually nagged John until he bought me more.
John: And I have ginger peach black tea.
Rebecca: I have Lady Grey today.
John: We’ve invited you here to talk a little bit about the growing problem that we’ve seen with the growth of online services that seemed to be designed primarily to facilitate academic dishonesty. I know you’ve had some issues with that in your courses recently. So, we thought this would be a good time to talk about that.
Liz: No, excellent. My post-traumatic stress hasn’t been maximized by the issue. So, let’s talk about it some more. But seriously, actually, it’s a really important issue. And I think our faculty are just not as well informed of this as they need to be. I think in the coming year, it’s going to become my mission to talk about this a lot more.
Rebecca: Can you talk a little bit about the incident that just recently happened to you to give some context and to help faculty understand how these kinds of things play out?
Liz: Certainly. So, really for years, even with the growth of the internet, if you’re using a publisher product… and it used to be sort of paper test banks and end-of-chapter questions… they show up on the internet. Some faculty just aren’t careful and were just posting them on non-password, open web pages. And then sites begin to grow where publisher content was widely published, by students often, with answers and answer keys to sort of pay it forward to future students who might be reusing it. I think what struck me this fall was the extent at which original content (because of this problem I had given up on publisher content)… but my original content in terms of my questions were being uploaded, essentially in real time. And within a few days, custom answers were being made by some of these sites. And this is really a new step in it, because it used to be the easiest way to get around this was to be writing your own content and guarding that content. And we find unfortunately, in September, that that was not enough.
John: The problem isn’t entirely new, though. For decades, maybe for centuries, fraternities and sororities and so forth, have kept files of old exams. But, the internet allows that to scale much more extensively and creates profit opportunities for people who facilitate those services. So could you tell us a little bit about some of the sites that do this?
Liz: Well, let me say this takes a step from the fraternity/sorority test file with this obvious broadening of the audience online, and no longer in geographic proximity. So, social affiliation and institution is no longer an indicator of what’s available to you. It’s searchable. So, it’s fraternity test file on steroids. It’s really one of the issues. But the other issue is these online proliferation of sites correspond to publishers emphasis on their digital content. As publishers really deal with essentially trying to get out of a used textbook market and kind of adapting for new ages of information, they really push their online content… which means faculty are relying more on that online content to operationalize many best practices such as retrieval practice for their students. So, it’s also more useful for students to have access to this. But, I think what really happened this fall is their ability to customize in short frames of time really eliminates the ability for me to just change up assignments in order to control the problem. So, really for low monthly fees, Chegg.com is what I ran into trouble with and for the low, low price of $15 a month they can upload questions. There’s a limit on how many questions, I think, they get a month but students potentially being strategic about it can upload questions and then get customized answers. Caveat emptor, some of the answers were great, some not so great and downright strange, which is really how I gleaned on to the problem to begin with. [LAUGHTER] And there are other sites. Chegg has some competitors I would say Chegg’s probably the market leader in this, but there are some competitors: Course Hero…
John: Course Hero has been out there for quite a while. My first exposure to this was about four or five years ago, when I had a student post in their online class that they were a member of Course Hero and they encouraged other people to join, mentioning that they could get their membership just by contributing a certain amount of graded work from the course to the site in return for that membership, and asking them to use her code so she could also get some credit for them joining. I had a little chat with that student about academic integrity and reminded her that if she posts anything on that site it would have to be taken down and she was going to be reported for this. This type of thing seems to be increasingly more common.
Liz: …much more common. And then you even have an issue like Quizlet. And one of the issues like Quizlet is students actually use it to create flashcards to study. So, I actually think there’s some honest intent for students that often come on to Quizlet. By making quizlets out of your questions and things you ask though, they actually create something for students that come after them. But, often Quizlet is really set up for students to actually create flashcards for different kinds of courses and study. But that’s really the leftover intent. And then, of course, there are online paper mills sites that actually offer existing papers… and again, narrowing topics can get around that and just not say “Write a paper on any broad topic…” that’s kind of really asking for trouble. But again, the customization that has come along makes that even more problematic as well.
Rebecca: And it’s somewhat affordable. It’s not incredibly expensive to be a member of these sites or to get a paper written for for you.
Liz: Yes, Chegg’s membership is $15 a month. So, that’s incredibly affordable. There have been sites… they come and go… kind of on a dark webish sketchy servers… but there are also sites offering to just take your online course. Those tend to be $1200 dollars, and up… Those sites get to be more pricier. But frankly, if you were a faculty member serving an institution with a more affluent student body, you should be very concerned about that.
John: Many of these sites also provide custom paper writing services. So, even if you’ve specified a narrow range of paper topics, students could still order a custom written paper.
Liz: Exactly. I would be fair to Chegg.com… that Chegg com responds within 24 hours in my experience to DMCA requests. So, they’re actually very responsive. And they take them down immediately. And they send warnings to the account that you posted things that you didn’t have permission to post. Chegg.com also cooperated with an honor code request, basically giving me upload dates and the emails of the members that uploaded content as well. So, they will cooperate, like I said, a pretty rapid fashion. I think it really, in the area of legality, it’s not clear if you correct yourself but become this conduit for things to happen, then, I don’t know what kind of legal responsibility is going to come there. It’s sort of beyond my expertise. But, I suspect it would take big players it takes like publisher lawyers and things like that, to really come up against this and demand that you really set up a situation that facilitates the repeated stealing of our content. Because it can be an intellectual property whack-a-mole game, because they take it down, but then another student post It from entirely different account and Chegg and these sites are kind of saying they’re not responsible for that. Legally, I’m not sure if you facilitate that, how responsible you are. There’s also paraphrasing tools. Paraphrasing tools… that’s very caveat emptor or buyer beware type of situation… and I actually have an example to show you what can happen when plagiarism tools go bad. Friends don’t let friends use plagiarism tools or this is going to happen. So, let me show you the answer from Chegg expert which was a very nice answer… is as follows: “Expectation of rise in inflation will lead to a drop in the prices of Treasury bonds. This is because the required yield by investors will increase in order to generate higher returns after beating the inflationary pressure.” Great answer, run it through a paraphraser and this is what you’re going to get: “Desire for ascend in the expansion will prompt drop in the cost of the Treasury bond. This is on the grounds that the required yield by the financial specialist will increment so as to produce higher returns subsequent to beating the inflationary wait.” [LAUGHTER] So, that’s not a thing… that is not a thing. I think the odd choice of vocabulary… Here’s words that college students almost never used. And here’s 10 of them right now, in the same paragraph. Obviously, that’s the warning sign. And the idea is, of course, context is important. Certain words that are used by convention or tradition in financial markets. Something that’s really a synonym, technically, in the English language is not in the context of financial markets.
John: A traditional way of catching students who were doing this often was that they were providing you with something that seemed a lot stronger than their other work.
John: And this sort of reduces that down to where, perhaps, it may not be as obvious in all cases that someone is doing that because the quality of the work will no longer look exceptional in the same way.
Liz: Yeah, the exceptional quality of work, you’re right… because the first answer actually does set up my radar because it uses a complexity in sentence structure that I might not expect in this answer. But, I think the other one was just immediately obvious because the word choice just doesn’t fit. So, even if you were actually looking around on Investopedia, or some other website to borrow language, it’s not language that would be used by someone writing for that.
Rebecca: Even as a non-expert in the field, I could tell you that that is not language of the field.
Liz: “Desire for ascend in the expansion?” Yeah.
Rebecca: No, no.
Liz: I think pretty much called that one out.
John: But if you have some foreign students in your class, that sort of thing might sometimes happen. I’ve certainly seen examples where students have used synonyms inappropriately. So, some of this may get by.
Rebecca: In some cases, though, if it was a student where English wasn’t the first language, you would probably have some assignments and know already the kinds of mistakes that that student would make.
Rebecca: So, you would have an idea of whether or not that would be consistent for that student or not.
John: We touched on the issue of hiring other people to do a student’s work, but could you talk a bit more about how the gig economy may play a role in academic integrity issues?
Liz: Well, the gig economy is basically the idea of you give me your specialized topic and I will write a paper custom for you: words, APA or MLA, the formatting etc., ready to go… Course completion, I will take your online course for you… paper writing services, again, is sort of that issue. And again, in ome of these paper writing services, it’s sent as a Word document, and they don’t actually strip the properties of who created that document.
John: And one of the things that has helped facilitate this, though, is the international reach of the internet. In countries such as India, where incomes and wages are a lot lower, you can pay people relatively small amounts to get reasonably high quality papers written in very strong, solid English
Liz: Right, exactly.
Rebecca: So, what can faculty do? This sounds very dismal.
Liz: It does sound dismal and there’s no solution that doesn’t involve some time on task here. And I think that’s a big problem. Because most of my faculty colleagues, I don’t really know anyone who sit around doing nothing… who has extra time to deal with this, and that’s got to give. Scaffolding is a common way to deal with this. So, it’s a recipe for disaster to assign a paper, be very unspecific about the topic: “Do any topic in post-World War II US history out…” That’s going to be bad. You want to focus the topic, then you actually scaffold. When do you want a thesis statement? When do you want an outline? an annotated bibliography? So, trying to have the pieces turned in really prevents their ability from coming out to get a paper or it’s really the red flag when a student several weeks into the process suddenly wants to change topic, having had struggled earlier with the topics and then they tell you when they want to change, you need to get ready for what that paper is going to look like. And you can scaffold in other ways, even on homeworks: you reference. I reference “Using the model from chapter seven,” “using the model we talked about in class in week six” or “using the discussion issues that were brought up in discussion three.” And so you can’t easily post those and get answers because then the students would have to provide the context, which defeats the purpose as well, of the questions. Algorithmic questions, and algorithmic questions can be posted and solved, but algorithmic questions are really after the fact, it makes it a lot easier to snag people. Because, if you just have a general problem…. So, I had a problem on the internal rate of return, which is a common time value of money concept in the field of finance. And while I think students really got their answers from Chegg, unless they were kind of lazy enough to exactly copy the wording of the answers, I couldn’t really get them because there really was a way that you solved this in the spreadsheet. So, the solution should look the same. So, you get around that by algorithmic questions where the numbers are unique, and then you know, exactly who uploaded that question. And in a learning management system, it’s easy enough to regenerate numbers every semester or year that you’re teaching the course. So, there’s a new set of unique numbers. So, that’s more about enforcement, I think. Questions that work from very specific data, maybe a data set, or quotations. So you can start with: “Mitt Romney, in a debate in 2012, called China a currency manipulator. What does that mean?” And so that quotation again, makes it harder to find general answers to that question. Questions that reference their own experiences, where they have to call up a specific experience themselves, and expound on it, and apply it in class. And current events. So, if you do actually have a very current event, then you can actually prevent going back in time and trying to find older questions, because it’s a current event as well. And so you’re constantly changing. So, you might be testing on the same topic, but you’re constantly changing the context. And your question has to be written so that you forced them to address the context as well.
John: I’ll often ask students to find an example of something in the last six months that illustrates some concept that we’ve just discussed in class. Because if I only offer the class once a year, they’re not going to be able to go back and find earlier examples from the class.
Liz: Right. And then I guess my least compromised questions are ones where they actually create a graph using the Federal Reserve Economic Database based on macro data or financial data. And again, that’s right, because it has to be current. And from year to year you change the time period, or what you want them to look at, or different measures of inflation, things like that. And then you can really grab that, and I have yet to see a FRED question appear on Chegg.
John:(…and FRED is an acronym for Federal Reserve Economic Data.).
Rebecca: What is your process for monitoring because clearly, you’re doing some monitoring.
Liz: Right. Well, when I found the extent of this problem in September, I basically went forward several weeks, and just copied and pasted questions into the internet search engines. I also ended up at least getting a subscription to Chegg so I could look, because the other issue is if someone screenshots an image of a question from their learning management system that might not show up in a Google search, but it’s going to show up on Chegg because Chegg actually has the text… kind of an alt text… that comes up, and that will show up as well. And so I started doing that. I opened a Google doc. And then every time I found one, I knew I had to change the questions in my graded activities. And then I would put it in the Google document. And I ended up with a 20-page DMCA document to Chegg about all of the questions I wanted to take down that take me through about week 10 my course. So… to be continued… as to how many I will find. But really, the monitoring really comes up at first when you get just a very unusual answer, or you start getting all the same answers. So, in a class of about 37, there’s about 10 students that were trying to tell me the same thing, especially if they illustrate a concept… I don’t ask for a numerical example… But they give me one. When you see the first person do that, you’re like “That’s great.” And then you think, “Well, why would the second, third and fourth person choose the exact same numbers to illustrate this concept?” There’s only one reason and it’s not a good one.
John: You mentioned the paraphrasing tools making it a little harder to find when people were trying to copy materials. There’s other tools out there too, though, which are online plagiarism detection sites where students can get a paper that they think fits the requirements of the assignment, upload it to that, see if it’s found, and if it is, just make minor changes and resubmit it until it ends up with a relatively low plagiarism score. And complicating this a bit, is that some of the major paper mills advertise that they use a TurnItIn service to check the papers that they sell for plagiarized content. Because one of the issues for the companies that are selling papers to students is that sometimes the people they hire to write papers simply plagiarize existing material. So, this makes it a little bit harder perhaps to detect that sort of work for hire. What might you do to detect papers that have already been run through the TurnItIn or similar systems and modified to come up with low plagiarism scores?
Liz: Well, I think you have to go backwards from that and recognize the limitations of those tools. I was never a big fan of TurnItIn. SafeAssign is just kind of a starting point. SafeAssign often gives a lot of false positives as well. Because if people are citing sometimes similar sources, which you would expect, then you’re going to see that pop up in SafeAssign. So, I think you got to go back and say, are you choosing the topic that is sort of paper mill proof, or things like that? Downloading when you have electronic submissions, which I do, and downloading them can also pick up issues of extra characters designed to trip it up. So, people put characters and they change the color to white to try and trip up that detector and that easily shows up when you highlight them. Those are some of the things that I would use. wWe haven’t had TurnItIn on our campus. We had SafeAssign, which I already knew the limitations, but the tell is you read it, and you’re like I’ve met five college students that write like that… in 23 years. So, I’m now deeply suspicious of that. Often the language doesn’t match, which is why scaffolding also becomes important… because the language doesn’t match what’s been done earlier in terms of: if you don’t think the student wrote the paper and you can’t find the source, you invite them in. And you ask them about where they found the source. Or I really love what you said about the impact of inflation on average households in the 1970s. Could you explain how you developed that? …and they won’t be able to and as a result, they’ll often say, “Well, I wrote this paper and I immediately forgot everything.” And faculty members get caught up into this. “Well, I have to prove a negative or I have to prove that that’s not possible.” And you don’t, I think you’d say: “On your face, that’s a really ridiculous argument. Either you need an MRI right now, because something wrong has happened. You’ve had a stroke and you need help, or you’re not being honest about where that paper came from. And you’re allowed a chance to defend yourself from these charges. But that defense isn’t reasonable. And I’m going to move it on.”
John: Pretty much every time I’ve run across that and brought the students in, they almost always confessed pretty quickly when they realize that they can’t explain what they had written.
Liz: Yeah, I think turn it in has these separate issues too. Is it ethical? We’re complaining about our intellectual property being posted. But, students that do write original papers, that’s their intellectual property. And we’re forcing them by virtue of getting their degree and meeting requirements of a course, to release some of that somewhat. So, I think there are issues of privacy and property, ethical issues that I know a lot of other faculty out there have just really pushed back against for years with turn it in. Others argue it’s about us policing students by requiring this prevention method, we’re almost taking the assumption that something’s going to go wrong, that it can create a hostility and an adversarial relationship that we don’t want in students. I think there’s some truth to that as well, which is why it becomes so important to think about the design of your course and your learning activities, because it’s so much better to prevent this and to make it very difficult than to deal with it afterwards. As I tell my students in my annual first day of class, don’t test me on this because I fail people at least once a year for this kind of thing. I actually say this is the worst part of my job, and I hate it, but I do it. So, that should tell you how important it is to me.
Rebecca: I think that it’s sometimes it’s a little more obvious to faculty how to prevent some of these things from happening in an face-to-face class, because you can be doing in class assignments, you can have them working on things in class and see their progress. But that doesn’t play out maybe as easily online because you don’t necessarily know what they’re doing. So, do you have some strategies of how you scaffold or do things a little bit differently online?
Liz: I would back up again, because some people say, “Well, this is why I don’t teach online, because I’m worried about ringers and things like that.” But, often people that tell me that do assign papers, and I say “Basically, anything that’s done outside of class is susceptible to these sites…anything.” And so really, again, there’s a lot of face-to-face classes, particularly in STEM that are using homework packages that students complete on their own. And then there are plenty of classes that assign papers that are done outside of class. The idea is, is that the in-class possibility of a face-t- face class does provide this check. But in an online class, you really compel them to actually talk about things in a discussion. And so you can use your online discussions to lay the groundwork of what they want to talk about, requiring them to reference discussions within the course as well. And so that’s one way to try and mimic that check that in-class does. And there’s proctoring software, which again, is potentially foolable as well, but there’s also design. So in an online class, the idea is no one thing should be worth a lot. Any one thing should be worth less than 20% in my opinion, because the idea here is if you want a ringer, you don’t want that ringer, by just taking a single exam, to move your letter grade significantly. You’d need a ringer for the whole course or a really, really good friend to do this for you. So, that’s another thing and in designing them exams, you can time them, not necessarily just time them but there’s one question at a time, no backtracking options. So, you want to think about a very structured way to require students to demonstrate understanding that just make it a lot more difficult to outsource.
John: You can do the same type of scaffolding online as you do in a face-to-face class. You just have different stages, as you said. In the first stage, you could have them submit a thesis statement. In the next stage, you could ask the students to submit a bibliography, followed by an annotated bibliography, and then a rough draft and a final draft of the project . And that’s not really much different than it would be in a face-to-face class.
Liz: And all in the same place are all of these writing samples because that’s how you’re communicating with students: via emails, course messaging, discussion forums, and then other graded work. So, in some sense, there’s a large body of written work to form a basis for your suspicions or concerns.
Rebecca: I’ll also add that we focused on lot on written work, test questions, and things like this. But, the same kind of plagiarism can happen with images, it can happen with code. I’ve had those same experiences in my classes as well…. sharing of digital files to make a particular design and boy does that look kind of similar to something I’ve seen before or an image that’s being claimed as their own or not documented where it came from. So those same things happen. They don’t play out in the same way in Chegg and some of these other sites, but those same practices happen through like a gig economy or just sharing amongst other students and when their digital files are a lot easier to share them when they were physical things.
Liz: Also in creative fields, there’s something worse in the way that’s sort of accepted, because if you look at fashion design, the ripping off of top designer early designs to then the knockoffs is astonishing. There’s a photo that has a Manolo Blahnik sandal, frankly next to an Ivanka Trump sandal, and they’re the identical red sandal. And it’s really just you slapped a different name on it and use different materials, perhaps. And obviously, they’re about six months apart in terms of product cycle. So, I think people in creative fields see that recycling, even in the music field, whether they get permission, you see music, that’s a remix of other older musicians, and you need permission for that, but you don’t necessarily know that when you’re listening to it and enjoying it. So, I think you actually see in some of the real-world situations how visual borrowing and frankly, stealing, is kind of accepted in some of these fields by some very successful people. Whereas, in the written world, when authors get found with plagiarism, it’s considered a big deal. And it’s kind of very embarrassing for them to get caught. Books can be recalled… things like that. It seems like written word… there seems to be more of a consequence, for now, when that happens, whereas in creative fields, that’s not always true.
Rebecca: There’s a lot of visual and audio copyright cases, though. And that’s where that tends to play out. The fashion example that you’re giving… those are considered functional things and not considered creative. So therefore, they’re not protected by the law in the same way.
Liz: Exactly. But, I think for what students observe, and behavior that they might emulate, when they look at professionals and their choices, that distinction isn’t going to come up.
Rebecca: One thing that I always argue is that when we talk about intellectual integrity, that copyright comes into the discussion in particular fields, because it is sometimes a common practice. And it’s there for a reason. we iterate on our culture, our culture creates new culture. It’s not a crazy concept, but you need to know where it comes from. And you need to provide attribution and in an appropriate way, which is no different than following MLA or APA or some other thing. So, I think that’s always something that people should be thinking of, and that when you’re having written papers, and there’s images and things in it, that you’re also thinking about that part of the content as well and not just the written word.
John: One other thing that I’d like to bring up is an earlier podcast that we did with Judith Boettcher, who talked about one way of avoiding this issue by having group projects that provide students with a lot more autonomy, but in a very structured fashion. And that’s perhaps a way of getting around this where students can take more ownership of the project and they create more of the project as a group, which would make it a little harder to engage in academic dishonesty. And we’ve also, in previous podcasts, talked about some open pedagogy projects, where the work that students do is posted publicly. And if they know that when their work is posted publicly, and they’re copying something from anyone else, it’s much more likely that they would be found and they’d get in some trouble later. So, those are two other things, perhaps, that might be ways of reducing the incentives for academic dishonesty.
Rebecca: Community based learning is another one… or service learning where you’re doing projects with the local community, because all of that context would be unique every time you’re doing something. So, that’s another opportunity for grading those assignments that really aren’t reproducible, and would be really, really hard to get an answer for… unless the person lived in the same community.
Liz: I agree. I think the biggest challenge is in some of these courses that are tool courses. In some courses you’re acquiring the tools that you would learn for projects and to consult, etc. And so when you teach these tool courses, it’s not always appropriate to have these kind of finished product things, because they’re in progress of assembling that toolbox that they’re going to use. And this is where reliance on sites like Chegg become a big problem.
Rebecca: At the beginning, we started with the idea that faculty aren’t always aware of these tools, or even the ways that students try to manipulate the system. Can you talk about ways that we can help increase this awareness with our colleagues?
Liz: I was just talking about this this morning with another colleague and we were bemoaning our Chegg purgatory this semester, and she says, “I just don’t think other faculty realizes… or how can we be the only ones that care about this.” So, I think some of them are honestly unaware because faculty aren’t always in sort of the student space and understand what those crazy kids are doing these days. So, I think, in some sense, faculty that have been more tuned in to creating learning community or kind of developing a relationship with their students are more likely to get ideas of what’s going on as well. But I also think there’s some willful ignorance here… this whole “Well, I didn’t know.” But I took steps to make sure I didn’t know because once you do know, it obligates you to do something, if you “don’t know,” and I’m using quotes, which is not helpful on a podcast, air quotes, but if you really don’t know that you’re MyLab component of the course is completely compromised, that doesn’t obligate you to think about changing up your course, and the weighting of activities and what activities. Once you know, it really obligates you to act as a faculty member. And so some people say, “Well, I wonder if my stuff is up there?” and like, “Oh, it is.” I don’t even have to look, I can say “Yes, it is.” And then academic dishonesty takes time. I’m really at 40 hours and counting with academic dishonesty documentation, DMCA documentation, and then reworking items in the course to deal with this issue so it wasn’t a lost semester. And I’m a full professor, think about an assistant professor not only trying to balance and develop their research agenda in conjunction with this, but also not wanting to rock the boat with unhappy students. I’m going to be getting a nice bottle of something sparkly, when I read my faculty evaluations in January. They’re going to be lit. Perhaps we’ll do a dramatic reading and have a tea party with tea and maybe something stronger. But, you know what? I can weather that. I can take that. And that’s going to be an issue. And frankly, I’m really proud of this institution, about how administrators really back faculty, enforcing the integrity of coursework and the degrees. And I know that doesn’t happen at other institutions, frankly other institutions that are maybe more tuition dependent and driven… that are unwilling to make steps that make students leave with financial implications. So, in some sense, this is really one of the best environments here at Oswego to actually try to enforce these policies as well. So, I think that’s one of the reasons that faculty don’t act, because there absolutely is some blowback to that.
John: And faculty might also see that each year they teach the same course, their students are doing better and they might be very content just to see that improvement in the scores so that all the work being submitted looks more like the best work from the year before. And it’s really easy to passively accept that.
Liz: Well, I would actually note that what’s more astonishing, is not the extent that Chegg has corrupted these issues, but there is a significant contingent of students that do not use it. Maybe they are unaware of it. Maybe they decide they don’t want to use it. But, when someone says, “Well, I can’t believe how many students cheat,” I would say, “Well, I can’t believe how many don’t, given the incentive structure.” And so that’s somewhat encouraging. But, also I had to decide if I was going to email the entire class and say, “Look, I’ve seen this happen. And you have to know that this is not allowed, that I’ve already made it clear in the syllabus. This is not allowed, and this is what will happen.” And it was a real-life decision about the Streisand effect, because you have to wonder if 20% of the class is saying “Wait, there’s a Chegg and we can get answers there? Alright!” You didn’t know if you wanted to clue them in, but I figured I had to act under full information.
Rebecca: I think it’s also really interesting when you start seeing students wanting to apply for jobs at those kinds of organizations, and then what kinds of conversations you’re going to have with those students, when they say, “Hey, I found this really awesome job that I want to apply for at Chegg.” It’s like, “Well, think about that. What does that mean? And what are those implications? And where are the ethics behind that?”
Liz: Well John, and I were talking earlier, there are Faculty Fellows at Course Hero, and some of them have made a name about teaching in the profession. John was probably more polite about that. One of them’s at a conference and I think I would stand up and say, “How do you reconcile partnering with a site that facilitates academic dishonesty and intellectual property theft every single day?” I would be curious as to what the answer is.
Rebecca: The pay is really, really good? [LAUGHTER] I don’t know.
Liz: I guess. The one I’m thinking of has a pretty sweet gig right now, so I’m not really sure.
John: this presentation was right after one that was riddled with references to learning styles, so I saved my powder for that one.
Rebecca: So we always wrap up by asking what’s next?
Liz: First, I’m already kind of brought this up in faculty assembly. And I’ve brought it up in academic meetings, because I’m acting chair this year. So I brought it up in leadership meetings. And I hope we can actually do a workshop through our Center for Excellence in Learning and Teaching. So, maybe a breakout workshop talking about our experience, and try and really broaden the understanding of the issues here. I’m also working with our faculty union, the United University Professors is also very interested in compiling violations of intellectual property rights, and trying to deal with that and push back against those sites. And I’m actually sharing my DMCA documents. So I made an editable form for some of the big sites, so you can easily go in and change them. I made a template for what you would ask the Associate Dean to fill out in order to ask for upload information from these sites as well. So trying to minimize the work involved for people who want to do this and take action. And then, finally, I’m just looking ahead about how I’m going to really redesign this course that I’ll teach a year from now, and to motivate and enforce original work.
John: And I should note, we’ve also been offering workshops for at least seven or eight years now.
Liz: You’ve been offering workshops on these sites and things like that….
John: …and attendance has been generally limited. We’re lucky if we get 15 people.
Liz: But you don’t have me. I’m a draw. I’m a draw… a star. [LAUGHTER]
Rebecca: …especially if we could do the dramatic reading of the examples.
Liz: Exactly. There will be a dramatic reading, and there will be sufficient supplies of snark. And I think it’ll work. But, I actually think maybe a case study. So, we come and we talk about these sites, but I can sit down with other faculty who’ve had this problem. And this is what I found… this is how I found out about it. And this is what I did about it, and this is what you should be doing about it.
Rebecca: This has been super informative. Thank you very much, Liz.
John: Thank you, Liz.
Liz: Well, thank you for having me.
John: Editing assistance provided by Brittany Jones and Kiara Montero.