128. Cultural Acclimation

International students enrolled in U.S. colleges and universities often face a multitude of challenges related to cultural differences and language barriers. These challenges can have an adverse impact on their academic performance during their adjustment process. In this episode, Don Donelsen joins us to discuss how the graduate business program at the University of Miami is working to ease this transition.

Don is a lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of a Spring 2016 University of Miami Excellence in Teaching Award.

Show Notes

Transcript

John: International students enrolled in U.S. colleges and universities often face a multitude of challenges related to cultural differences and language barriers. These challenges can have an adverse impact on their academic performance during their adjustment process. In this episode, we discuss how one graduate program is working to ease this transition.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Fiona: My name is Fiona Coll. I teach in the Department of English and Creative Writing here at SUNY Oswego and this is my turn to sit in as a guest host.

John: We should note that this podcast was recorded in the third week of February 2020. Many of the plans that are discussed here have been altered as a result of the nationwide shutdown of institutions of higher education since the onset of the global pandemic.

Our guest today is Don Donelson. Don is a lecturer in the Miami Herbert Business School at the University of Miami. He is a recipient of a Spring 2016 University of Miami Excellence in Teaching Award. Welcome, Don.

Don: Hi. Glad to be on.

Fiona: Today’s teas are… sweet cinnamon spice.

John: Are you drinking tea?

Don: I’m not drinking tea, but I do actually have a gift from a former student.

Fiona: Oh.

Don: I was told it was Chinese tea, but then other Chinese students said this is not Chinese.

Fiona: [LAUGHTER] Well, you can say we’re drinking tea and you’re looking at tea. I guess that counts.

Don: I am looking at Green tea.

John: And I am drinking a ginger tea. We’ve invited you here to discuss a program that you have proposed and are working on at your institution to help students from China adjust to cultural differences in how classes in the U.S. are taught. What prompted your interest in this issue?

Don: We’ve had a large influx of Chinese students at the University. That’s probably the main impetus on what prompted this. I asked our institutional people for some data, and just in the graduate business program, we had our Chinese population double just in the last year. So we’re up to about 400 and something in the graduate business program, and undergrads, we have about 1500. And we’re not a large school, that’s about 15% or so of the entire population. So I’ve seen noticeable increases in Chinese students in classrooms, especially in the STEM specialized master’s programs, which they’re very attracted to, for some visa reasons, and perhaps other reasons. And so I actually had a section, there were 16 students, 15 of whom were from China, not by design, just this is how many Chinese students we have and sometimes that’s how it worked out. And I started noticing some differences in how the Chinese students were interacting in that class when they were mostly surrounded by their peers from China than the Chinese students who in the past there was one or two Chinese students in a section of 19 or 20, but now there’s four or five. This semester, actually, I have more than half in every single section from China. And so I started noticing that in that one section, it was all Chinese students except for one from South America, the interactions were a bit different. And then discussions with colleagues, how to improve teaching… The courses that I teach are classes on critical thinking, and problem solving and communication. And so the class participation is an enormous component of the class, we teach critical thinking by forcing people to do critical thinking. And so the trope, I guess you might say, not just the Chinese students, but a lot of international students in general, is that their language barriers are most pronounced in a class like that, and they participate less in class discussion, and they have difficulties with communication. Most of them, in my experience, have been difficulties that only they perceive, difficulties that aren’t actually barriers. But all of those issues I found true with the Chinese students but amplified to a much greater degree, and that’s the general consensus among colleagues. And so, as the numbers started to grow, we actually created a course, a different numbered course from the core course that I teach in the graduate program, exclusively for non-native English speakers, because putting them on the same curve is really problematic to their grades when there’s a heavy written component. But at the same time, we have academic standards that matter, I can’t just give them a separate session, so we created a separate course. And so it’s different numbered, and it’s essentially the same course title except for non-native English speakers. And so as I had the accidental almost all Chinese student class last semester, I said, “Hey, I should probably take advantage of this opportunity to test run in the spring semester,” which is now, when I’m going to have by design a class of all Chinese students. So I started really trying to identify “Why would they participate when they did participate? Why didn’t they participate?” Trying to break down what it was that caused these issues that most faculty here observe with their reticence to participate and lack of comfort with speaking up.

Fiona: Can you talk a little more about the general differences between classroom interactions that you’ve observed in students from China as opposed to perhaps students who are mostly American?

Don: So it’s actually changing now, because of some of the stuff we’re doing. Where we were at to begin all this was they pretty much do not participate in class discussions. It’s very rare that they do. If you go full Socratic method and just start calling on people, some of them will participate when called on, but it’s clear that they’re uncomfortable with it, they don’t quite know what to do, and some will just refuse, even when called on. There will just be a minute or two of silence while they try to think of something or they kind of punt. And so with class participation in most of our courses in the graduate program is 20 to 30% of the grade, it’s a significant problem that inhibits their success.

Fiona: And I think you’re suggesting that there are multiple factors involved in this reticence. So, it’s not simply a language-based issue, but also a cultural issue that expectations around classroom culture differ so much that students really do feel unable to participate in a culture that feels so different from the one they’ve just come from.

Don: Exactly. So I started talking to students, one of the things I noted that was interesting was they’re very comfortable speaking after class, one-on-one, very frequent after class. And as I started having more Chinese students in my classes, it began to be a problem, actually, because I didn’t have enough time in between classes to field all of their questions. And so I thought that was interesting. And then in the few instances in which they had to give presentations, so an assigned presentation, like a stock pitch or something, they did remarkably well, mostly. And in fact, if I went and correlated my grades based on nationality, I strongly suspect it would be zero on the presentations, whereas on class participation it was a very strong correlation, so that got me curious. And I started asking them, “Why are you comfortable speaking to me outside of class?” which I’m very appreciative of. The Chinese students in particular here, some of them… I could put a cot in my office because of how frequently they would come to my office hours. And so it was clear to me that it wasn’t an unwillingness or a lack of care, which made me even more curious because some faculty misinterpret their classroom behavior as an unwillingness or lack of care. And so I started engaging and asking questions about participation, and the conversation just kind of grew, and I learned that the way that they do education in China, and there was some various experiences described, but pretty much all of them described an educational environment that they were brought up in, in which there is no mandatory participation as part of the grade. When participation is expected in class it is almost always in the form of a show of hands, and in no instance did any student describe a situation in which they had an open class environment where they would, without being prompted, speak up and make a point or something. In fact, the word that was consistently used by students, when I asked them what they think about speaking up in class, is “rude.” They think it’s rude. They look down, because looking up at someone is considered to be a sign of disrespect in many aspects of their culture. And so some faculty misinterpret it as they’re disengaged or they’re on their cell phone, but they’re actually fully engaged. They just think that that’s what the expectations are. So they think it’s rude to interrupt the class, they think it’s disrespectful to make eye contact. And so there’s a signaling problem, essentially, the normal ways that we would evaluate students to assess understanding, to assess engagement, don’t really work well without some explicit addressessing of these issues with the Chinese students. In addition, one major cause that we found was that their education is, perhaps not surprising to some people, but their education is designed around the idea that there are black and white answers to everything, there is very little gray area, their evaluation metrics appear to be almost exclusively objective, multiple choice, or true and false. Even in their English language classes it’s objective metrics… which of course, we all know the English language, for good or for bad, there are no objectively correct things, but they believe that there is a single correct way to make a statement in English, and so that causes a lot of hesitation in class because some students, they want to participate, but they spend time trying to figure out the right way for me to say this. Some other issues related to these issues with they’ve been taught that everything’s black and white… and even the English language is… they fear embarrassment if they mispronounce a word. They tend to be self conscious of their accent in ways that I don’t find as common with Latin American students or Indian students, who also are prone to having some self conscious issues with accents and like but not to the same degree as to Chinese students. I think it’s because they think there is one way to say it. And so there is a fear of embarrassment wherein it’s hard for them to grasp that they can’t be embarrassed because there’s really no bad answer when we’re having a Socratic discussion. And so fear of embarrassment was a contributor to these issues as well and a lack of specific directions. And so one thing that I found most startling, as I was going through these informal focus groups with Chinese students, is the number of them who could not articulate what class participation means in any way that aligns with what we know class participation to mean. Many of them thought that class participation simply meant showing up and that they would get their points from that. About half of them actually had no idea that their grade would be impacted by class participation, and the frequency and quality of that participation. I spoke with one second-year MBA student who I’d had in class last year for some insights from him, and he expressed that he was in that path, and he had no idea that class participation points mattered. And it pretty much put him in the bottom quarter of the class for his first semester, and then he eventually figured it out, and now he’s in the top quarter of the class, but that was just very upsetting to me. That was the point at which I said, “Okay, we are failing these students. It is not incumbent on them. We are not putting them in a position to succeed.” And that was the real fuel to the fire… to actually do some programming and create some initiatives to try and prepare these students for success better.

Fiona: There is so much about academic culture that feels straightforward and self explanatory until an experience just like this, when you realize how much of what we expect goes unexpressed, or unexplained, or is invisible in one way or another. So how did you begin tackling this enormous and multifaceted issue?

Don: I just made a checklist of all the things that we identified as causing these problems, and it was clear to me that we needed to be active. This was a significant enough problem with deep roots that it wasn’t as simple as just changing the way that we introduce our syllabus, and adding a five-minute spiel or something, it was much deeper than this. And so I proposed that we need to teach them how to be a student in an American classroom, especially in a program that requires Socratic discussion in most classes and is going to be 20 to 30% of their grade. So I proposed that we add a course to the orientation program that we have on how to be a student.

Fiona: And how detailed do you get in terms of approaching this from a metacognitive perspective? You explain to students the larger purpose of this kind of Socratic discussion, or do you simply dive in and have them start practicing? What approach are you imagining taking?

Don: My thoughts were kind of a two-half approach, wherein we first start off by teaching them what the expectations are. And so, exactly as you said, explaining to them things like “we value wrong answers.” John and I taught for many years at the Duke TIP program, and those students… very, very academically gifted… are younger, and so they can be very intimidated by Socratic discussion. And so I would always tell them that our jobs would actually be very boring if, every time we asked a question, the first student who answered gave a perfect and correct answer, and I don’t think that students would learn very much if that was the case. And so the idea is that we’re going to have discussions like that, kind of half teaching and half selling the importance and value to them in participating, accepting that there isn’t really wrong and right answers, we are moving a discussion forward. And when they successfully complete our program in a year or two, they’re going to be holders of a master’s degree in business and going to work in the business world, in which it will be expected that they put forward ideas that they don’t even think are necessarily going to work. Jeff Bezos at Amazon demands people put forward any idea they have, whether they think it’s going to work or not, and so we’re going to use case studies like that, a company they know, Amazon, a person they know, Jeff Bezos, and say, “This is who makes it to Vice President at Amazon, the person who’s willing to speak up in class and give a wrong answer.” So we’re going to educate and sell, really, participation and show them how to do it with some modeling. And so ideally, we will get some second-year grad students in their cohort who, through faculty recommendations, can be good role models and we’ll do some roleplay interacting with those students and demonstrate “Here’s what it looks like after extolling the virtues of it and demonstrating how we want to do it,” and then have them actually just do it, a mock class, and after that we would slowly morph from lecture into Socratic discussion.

John: And you’re planning to start this off with some type of bootcamp at the start of their year when they arrive, correct?

Don: So that’s actually complicated because we have lots of different programs. In business academia there’s been a seismic shift… really, like two years. It’s kind of startling, but there’s a major shift away from MBA degrees and a major shift towards specialized master’s programs. And so we went from having the MBA as our bread and butter, that was our main graduate degree program with I think we have like eight or nine specialized master’s programs now. And so there’s some logistics that have to be worked out because, you know, some of them have their own schedules, and there’s some departmental autonomy. Some of them are coming in July, some are coming in August. So they have a boot camp and then orientation, so those are separate. The idea is that as part of that boot camp, there will be a mandatory required course that is communicated to them, and it’s probably going to be two sessions. So I mentioned you haveb two halves, we’re thinking the best version of this would be overnight with a break in between those two halves with a case to go home with and prepare for this Socratic discussion.

Fiona: I’m wondering how you might incentivize an openness to failure or to wrong answers. Let’s say you’re not Jeff Bezos, don’t have someone’s employment in your hands. Have you experimented with, or thought about, or planned for ways to not just encourage students to take these sorts of risks in the classroom, but to actively acknowledge and perhaps even reward that sort of wrongful, rightful risk taking?

Don: Right, so, yes, and a couple of things I already tried out this semester with that all-Chinese section that I mentioned, I started off that class by saying “Nǐ hǎo, huānyíng lái dào wǒ bān,” which is a surely butchered way of saying “Hello, welcome to my class.” And then I asked them how many of them think I’m stupid because I butchered this phrase in Chinese, and, of course, none of them said that, and I said, “Well, that’s because you have context, and so the same thing is true for you all.” If there’s a student who’s born in Washington, DC, and has lived their entire life in the United States, and now they’re in a master’s program and they’re making grammar mistakes, I should rightfully judge them as having a lack of effort or some kind of problem. But for someone who is not a native English speaker, inductive reasoning does not allow the same kind of leap, and it would be illogical to assess someone on a personal level because they mispronounce a word or something. And so I said, “Just as you did not think lowly of me, or assess me to be incompetent or something because I butchered this Chinese greeting, you will receive the same benefits in your interactions with people.” And so they all laughed, there was a lot of laughter, and I think that kind of worked a decent amount, and then, and this might be unpalatable to some faculty members, but I have found with the Chinese students, they are extremely conscious of the opinions that their professors and peers have of them. And so it is very important to pat them on the back, especially early on. And so I started off with a low-stakes presentation, you know, one minute, because again, I found that when they’re provided with directions, and it’s required, and there’s a grade, they knock it out of the park, and then intentionally pointing out to good things that making them feel kind of safe, and that they’re not going to be embarrassed. Someone will mess something up, they’ll mispronounce a word or something and we’ll point out that it didn’t matter, and no one laughed at them, and that sort of thing. And so I found that doing that early on has helped in this section. And then in addition to that, the other main incentive that we’re playing with is changing class participation grades to more periodic updates, rather than what we typically do, which is just one of the last things that’s entered into our Excel spreadsheet probably after we’ve already gone on spring break, and because the updating of it in the feedback. And, John, I’m sure some of the other episodes I listened to that coincides with the importance of feedback on making better choices, and so not making it a surprise.

John: Letting students know that their lack of adjustment to American culture is actually harming their grades early on makes it much easier to adjust than when they find that out after the fact. So this will require some adjustments, not just from the students to adjust to American institutions, but also from faculty.

Don: Absolutely. If you’re a faculty member at an institution that wants to be a global institution, you have to think globally, and you can’t just expect every student to show up in your classroom Americanized. I think it’s kind of silly thinking really to just demand that the students Americanize themselves or westernize themselves and sink or swim because we’re not setting them up to have good outcomes, and that’s what we’re here to do.

John: We face some similar issues here. We’ve tried to focus on working on faculty to change the way they teach classes, but that only reaches the faculty who actually attend those workshops and students need to adjust to a wide range. And so I think there’s a lot of merit to this approach of working with students to help them get acclimated, especially perhaps in a business school where many of the students may want to go and work in firms where that type of participation and that type of activity is expected. So it’s also partly an introduction to American culture, as well as just American educational systems, which in a master’s program in business, would seem to be really appropriate.

Don: Absolutely. One of the drivers of the growth in specialized master’s programs for international students is STEM degrees get an automatic two-year work visa, which is very attractive. And so it’s pretty clear that, especially in those STEM credentialed degree programs, that their goal is to take advantage of that automatic to get employment, and it will impact them very quickly if they don’t acclimate to the classroom environment, which as you said, we do model business environments. And so I tell my students that we’re going to behave as though we’re in a consultancy meeting in class. You’re absolutely right. It’s beyond just a procedure for how to get grades, it is training for bigger things beyond the classroom.

Fiona: You mentioned that this program is focusing on masters level students, but the university also has a large number of undergraduate students. Do you think that this is a model that could be expanded to address that slightly different student population?

Don: I think so. Logistics would be an issue, of course, we’d probably have to recruit more help. But scaling aside, I don’t see much difference at all. I do teach a handful of undergrad sections every once in a while and the issues thinking back are identical.

Fiona: I have a slightly random question. Do students from China come to America with a pre-existing idea of what college is going to be like, what graduate study might be like? Is there any access they have to set up any horizon of expectations for them?

Don: This is not information that I got from our focus groups, but my observations, I strongly suspect that it’s something they don’t even think about. It’s kind of like you don’t know what you don’t know, and they’re startled to find that it’s not just what they have been doing for the last 12-15 years of education.

John: When they’ve been adjusting to a school system for over a decade, it’s pretty easy to base your expectations on what you’ve observed in the past.

Don: Right.

Fiona: I, too, interact with international students in a slightly different context, in an English literature department.

Don: Oh, that’s got to be tough.

Fiona: And many of the issues, especially that black and white thinking you’re describing, are amplified in my discipline, but I always do ask them partially by way of getting them to think about their own expectations, you know, “What were you imagining this class might look like?” And often they do have some sort of pop cultural version of what school might be like that emerges in a fascinating way. They have particular reasons for wanting to come to America in the first place to study, so I was just curious as to whether or not you had any insight into that version of things for Masters of Business students?

Don: No, I think they just value an American degree in business. Some of them do come from American undergrad institutions, and those students generally have already acclimated, but most of them are coming straight from China. Many of them, the day before orientation was their first day ever in the United States. That’s really scary, and so they show up at the airport and realize that they’re not nearly as good at speaking English as they were led to believe. And so I think there’s a lot of just very understandable ignorance of what they don’t know. We’re using a lot of social media, and we’re encouraging a lot of social media use, our Chinese students to communicate with their peers back in China, both as we find it’s a very good recruitment tool, and also to help with that kind of expectation.

Fiona: One of the other things I hear from the students I interact with who have come from China is their shock at how quickly Americans speak English, how fast professors speak in the classroom, and that’s a tougher one to handle in a boot camp class, I suppose. And I find myself simply trying to reassure them that they will be surprised at how quickly they adjust and they just need to give it some time and some practice, but it’s a very real source of anxiety for them in those first weeks of the semester.

Don: Yes, absolutely. And I’m glad you mentioned that because that is such a common… I mentioned before that students frequently will come up to me after class to engage and the mode interaction is asking for clarification on something that was said an hour prior in the middle of class that they didn’t understand. And in some cases, that’s fine. I can clarify, and it works, but in some cases, it’s like, “Okay, well, everything after that for the last hour, you didn’t get either if you didn’t understand that, and that’s really disappointing as well.” And so I make it a point, whether it’s an all Chinese class or not, I tell them that I want them to stop me if I’m speaking too fast, or I say something they don’t understand, that I want them to stop me and let me know that, and that I don’t think it’s rude. And in fact, I would be very upset to later learn that I wasn’t communicating effectively enough for them to gain understanding, and so that has helped a lot. It usually takes more than once of saying that for it to set in, but by the second or third class this semester, I’ve had pretty much every class someone stopped me and said, “I didn’t understand that” or, “my translator is not registering that word. What is that?” But yes, that’s an enormous issue.

John: Is lecture capture used there at all?

Don: I don’t use it.

John: Many of us use it. And I know in my econometrics class, before I flipped the class, I used to do some interactive lectures, and one of the things that mostly Korean students and occasionally Japanese or Chinese students noted was that if there were parts they didn’t understand that well, they could go back and play it back at half speed to make it easier to understand things, and that’s a nice accessibility feature for anyone who’s not a native English speaker. Another option might be to let students record things too, and then they could go back and play it back at a reduced speed until their understanding was able to keep up with the actual rate at which we speak.

Don: Interesting that you mention that because one thing that came from the Chinese students themselves in these focus groups was a pretty surprising number of them who said that we should ban electronic devices completely, because they said that they’re… so much more than American culture… they’re wed to their electronic devices. And they pretty much admitted that you are going to have some engagement issues unless you just forbid the electronic devices.

Fiona: It’s interesting that they can recognize [LAUGHTER] a certain sort of problem and are asking for help, I suppose.

Don: Well, they asked for help after the semester was done, and they made me promise to not name them if I implemented a ban. So there was some strategy involved, so that’s critical thinking.

John: But there is also the capability of a recorder which would not be that much fun to interact with. So…

Don: Right.

John: …there are devices that could work that would not be distracting.

Fiona: I’m really struck by the way in which you’re paying attention to a very specific cultural group here and you’re adjusting to very specific problems that that group has. You mentioned your checklist of things you know are happening and your desire to find the source of those things, but I can’t help but think that the adjustments you’re making are, in fact, adjustments that might benefit all sorts of students, all sorts of students for whom academic norms are a little bit hard to penetrate, or to understand, students who might have cognitive differences and struggle in discussion situations in particular. And so your particular intervention here seems to open out into a larger issue of what inclusive teaching might look like.

Don: Absolutely, and I’ve actually had that same thought. We’re singling out Chinese students, but boot camp, is there a problem with that? And is there some perception bias on our end that we don’t recognize these same issues with some other groups? But I think it’s very institutional, it’s not just culture by culture, it’s institution by institution. What we ended up finding is, because of our very, very heavy South American roots, our Latin American students are non-native English speakers who are native Spanish speakers, I think they feel more comfortable here than they might in many other institutions, and so you could have these same exact kinds of problems rearing their head, even maybe 100 miles north of here at another institution where there’s not a critical mass and those kinds of deep connections. We even offer some of our core courses in Spanish, and so I think they feel very comfortable. But it struck us that it wouldn’t take many changes in our circumstances for them to be in the same boat, perhaps, as we found the Chinese students. So you’re absolutely right.

Fiona: We do usually finish up by asking “What’s next?”

Don: What’s next is interesting, because the virus issues and so we are probably facing deferment of admissions to Spring 2020. And so what’s immediately next is adjusting on the fly as those things develop. But as far as the specific issue with helping students acclimate, where I would like to go next is just keep learning, keep having these discussions with students. So I had 20 something odd students participate in a pretty lengthy focus group session with me, that’s not enough, and so keep learning, start implementing some evaluation methods on ourselves. As John mentioned, sometimes it’s difficult to get faculty to cooperate with things but ideally, we would mandate standard language on class participation in all graduate syllabi, we would mandate periodic updating of their score on that. And we would even add in our reporting metrics, how the scores are changing in those classes that are doing that. Are they improving after we give them the first update, and so learning more about the students and the issues that they face so that we can better serve them I think is what’s next and really probably what always should be next.

Fiona: That’s great.

John: Sounds really good.

Fiona: Good luck with all of it.

Don: Thank you.

Fiona: Sounds like a very worthy intervention.

John: Thank you, Don. It’s always good talking to you.

Don: Thanks for having me, so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

127. Gender and Grade Changes

Grade change requests in college are relatively rare, but when they do occur, evidence suggests that male students make the request more often than female students. In this episode, Dr. Cher Li joins us to discuss these gender differences in grade change requests in college and why they might occur.

Cher is an assistant professor of economics at Colorado State University. Her research focuses on how public policies and social institutions affect the decisions of, and outcomes for, women. She is also a co-author of a January 2020 National Bureau of Economic Research working paper that investigates gender differences in grade changes.

Show Notes

Transcript

John: Grade change requests in college are relatively rare, but when they do occur, evidence suggests that male students make the request more often than female students. In this episode, we discuss these gender differences in grade change requests in college and why they might occur.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Cher Li. Cher is an assistant professor of economics at Colorado State University. Her research focuses on how public policies and social institutions affect the decisions of, and outcomes for, women. She is also a co-author of a January 2020 National Bureau of Economic Research working paper that investigates gender differences in grade changes. Welcome, Cher.

Cher: Thank you. It’s great to be here.

Rebecca: Today’s teas are:

Cher: I have Oolong tea from the Ali Mountain area in Taiwan. It is one of the best tea regions in Taiwan.

Rebecca: Sounds good.

John: Nice. And I have Cranberry Blood Orange black tea.

Rebecca: And I have a mango black tea today. John’s looking at me a little strangely because it’s very unusual for me. [LAUGHTER]

John: We’ve invited you here today to discuss your recent working paper with Basit Zafar on regrading requests and outcomes. Could you tell us a bit about the study?

Cher: Yes, this study is about gender differences in grade changes in college, and we look into the differences and why are the causes for the gender differences?

John: What prompted you to look into this?

Cher: So this research is actually motivated by my personal experience. In the last six years, I had many male students, but only one female student asking me to bump up their grade at the end of the semester. And I know from previous studies that men are more likely than women to negotiate for their salaries and promotions, and that made me wonder if these kinds of differences actually started earlier when they’re still in college. So, that is the motivation for this study.

Rebecca: Can you talk a little bit about how you conducted the study?

Cher: Sure. We tackled this research using three very different sources of data, including administrative records from a very large four-year university and we also use surveys and laboratory experiments. So, the first data source is a very unique administrative data from Colorado State University between 2010 and 2016. We have over 1.3 million observations of grade changes across all the departments, and this data set is very unique because it recorded not just the final grade, but it also recorded any changes associated with it. So we focus on the changes made by instructors, and our analysis showed that, although grade changes are pretty rare events, 95% of the time they will change to a better grade, and among those positive changes, men are about 18.6% more likely than women to get a better grade. And the difference in grade changes cannot be explained by the characteristics of students, instructors, and classes. However, the administrative record has several limitations. First of all, it doesn’t provide information on the cases when any regrade request was rejected by instructors, and it also doesn’t tell us about the grade changes during the midterm or any assignment throughout the semester. And finally, it doesn’t tell us “What is the scenario that caused the gender difference in grade changes?” So we can think about at least three different scenarios that all lead to the same data pattern. For instance, the first one is male students are simply more likely to ask, although instructors treat men and women equally… so, that is a possibility, given the previous literature evidence. And the second potential scenario is that male and female students are equally likely to ask, but somehow instructors are treating men more favorably, and if it is this scenario, we’re going to have something to worry about about discrimination. And the third scenario we thought about is, there may be a difference in terms of the timing of asking. So if men procrastinate while women are more organized throughout the semester in they ask their instructor to make changes while they find any errors during the semester, then they have no need to ask at the end of the semester. So, to examine the scenarios, we conducted two surveys, one on instructors and the other one on students, and we checked whether the reports are very similar across different surveys. And surprisingly to us, both surveys show pretty consistent patterns: that male students are indeed more likely to ask, and we find no evidence that instructors are treating males students more favourably. And we also didn’t find any support that there is a timing difference in terms of their asking.

John: That’s certainly consistent with what I’ve observed myself, that males have been more likely to ask both during the semester and after.

Rebecca: I don’t think I’ve gotten that many requests, generally.

John: Well, I haven’t either, I may get one a year or two a year out of like seven or eight hundred students, but…

Rebecca: I can’t think of a single one. I know that I have but I can’t think of a particular…

John: Well, your classes are smaller, too. So, that could be a factor.

Cher: Yeah. So I also typically only got like one to two students per semester at making that request. So it jumps out at me when there is only like one female student throughout the whole six years of my experience, that really makes a difference to me when I compare the frequency in terms of they approach me at the end of the semester.

John: One of the nice things I liked about it was your incentivized controlled experiment that looks at why there may be differences. Could you tell us a little bit about that aspect of the study?

Cher: Definitely. So focusing on the gender difference in asking, we did this lab experiment to answer that question, in particular in the lab setting. The participants basically take a quiz of 20 questions, and their payoff is dependent on the grade in the quiz. So we tell them we’re going to give them the grade. However, we also tell them the grade is noisy. So, we randomly picked three out from the 20 questions to randomly grade them. The grade we tell them could be their true grade, but it could also be higher or lower than the true grade. And then they have to decide under 10 different cost scenarios, ranging from paying up to $3.50 to being paid to $1, whether they are willing to pay the cost in each of the scenarios to get a regrade. If a participant chooses to pay the cost to ask for a regrade, then the true grade is revealed and their payoff is adjusted and we subtract the cost they have to pay, and if a participant chooses not to ask for a regrade, then the initial grade becomes final and they will be paid accordingly. And we find that, as long as the cost is positive, so basically the participants have to make a payment, men are always more likely than women to pay the cost to ask for regrades, and the gender difference in asking cannot be explained by their difference in risk aversion. We find that other majors, including their difference in confidence level, in their uncertainty about their performance, and personality traits explain nearly half of the gender differences in asking.

John: How accurate were the students’ metacognitions in terms of how appropriately they made a decision to request regrades? Did they gain by that on average, or did they lose?

Cher: So in general, they have pretty good guesses about their expectations in terms of how well they do in the quiz. However, their decision here really depends on their attitude in terms of pursuing more aggressively in the regrade or not. And we find that, in general, most students who decided to pursue the regrade were likely to get a better outcome, and we find that depends on the cost, basically. So when the cost is low, the more likely a student will get a better outcome, because they are facing a very low cost to offset the gain they got from a better grade. And the reason that they are more likely to get a better outcome is due to our lab experiment design, we make it asymmetric, basically. So, we are trying to mimic the real world scenario. So from the administrative data, we find that 95% of the time, students got their grade change to a better grade. Only 5% of the time it got changed to a worse grade. We intentionally designed our lab experiment to allow that asymmetry. So basically, when a student asks, they will have a higher chance to get a better grade than to get a worse grade.

John: How much more common was it for males and females at low, and then again, at high prices?

Cher: When there is a cost that really they have to pay out of their own pocket, around like half of male students are willing to pay the cost, while only about one third of women are willing to do so. However, when the cost is zero or they got paid to do it, it’s very similar. I think their percentage of requests is very high. It’s like nearly 80% when they were getting paid $1 to do so.

John: What happened when the cost was high? I believe you found some interesting effects for males at that.

Cher: So when the cost is high, we find that the gain from asking actually disappeared, because now the high cost offset the gain made by men. So although men are more likely to ask, they are not always made better off, so they are only gaining relative to women when the cost is relatively low. When the cost is above like $1.50, we see that men are also more likely to see their payoff decrease compared to women.

Rebecca: Can you talk a little bit about the gender difference in the regrade requests and what might explain those differences in general after looking at those three different approaches that you’ve taken to study it?

Cher: So we find that, it seems to be, there are some subjective beliefs that are different to begin with. So first of all, there are some evidence from other studies that men and women are different in terms of confidence level, and men tend to be more overconfident than women in a lot of contexts. So, we explore that too in our study, and we find that indeed, when we look into each of the questions they answer, we also ask them how confident they are on each of the questions. And we find that when they got the answer wrong, men and women are similarly over optimistic about their accuracy. However, when their answers are actually correct, women are much less assertive about it, so they are less confident about their correct answer. So that is part of the reason that is driving the gender difference in asking, and the other factor that we found are important in terms of the gender difference in asking is about their perception about the potential outcome, so basically the uncertainty they face when they make that decision. So we asked them about like, “What is the number of questions you think you got right?” And they will give us their best guess. And then we ask them then for that scenario, “What is the likelihood you think that’s gonna be happening?” So they will tell us like, oh, if they are very confident about “I’m getting 15 questions right out”, maybe assigning 100% as the probability for that outcome, if I’m super confident that is the outcome. But then if I’m not very confident about my guess I maybe say “Okay, 50% probably,” then I’m not really certain about that outcome. So we find that when men and women are giving us those guesses, women are much less certain about their guess. So they are likely to think it could be different outcomes, maybe it’s 14 or 13 questions they got it right, so they would also assign the probability of other outcomes that they anticipated. And that also explained a very large proportion of the gender gap, and we also find there are some differences in their personality traits. For instance, women are more agreeable than men, so they may be more agreeable to the grade they received and less likely to challenge it in terms of the context of asking.

Rebecca: When we start thinking about the impact this might have on GPAs or their academic standing, how does this really impact men and women differently?

Cher: So the overall impact on GPA depends on how we measure it. For those people who ask, we did a back of an envelope calculation and we find that for the female students, the accumulative effect from asking is up to .43 points in their overall GPA. But for male students, the number is a bit higher, they could receive up to .47 points as an improvement in their GPA, conditional on asking.

John: Which is a non-trivial difference in terms of that, especially if you were also to extrapolate and think about the possibility that this is also occurring during the class.

Cher: Yeah, we take that into account in extrapolating, and that is a cumulative kind of effect we anticipate. If a student reports a fraction of courses they asked for a regrade, that would be kind of cumulative effect overall.

Rebecca: Do you have any idea of what the not asking is doing, maybe to women, if they’re less likely to ask in the first place, how that might be negatively impacting their GPAs?

Cher: So in terms of not asking, you are comparing to those who ask, so that raw difference could be low depending on who we are comparing to, or we can compare it to those who actually are successful throughout their time, then that difference will be up to like .4 to .5 points in terms of the GPA.

John: At the start of this, you talked about how there’s a lot of evidence out there that women are less likely to bargain for higher wages for promotions and so forth. From those studies, is there a lot of evidence of similar types of causation?

Cher: So most of the studies we saw before look into their different tendency to ask, and one of the particular research I think is very interesting is in terms of the context. In the study made by other economists, including John List in University of Chicago. In their study, they did a field experiment, and they experiment the job applicants’ attitude in terms of negotiation for their salary. In their experiment, there are two settings, one is the salary is ambiguous if there is no hint or any information about it. And then they found under that context, men still negotiated for their salary, but women in that context tend not to negotiate and instead, they signal their willingness to accept a lower offer. In the separate setting, when they make it clear in their advertisement that the job salary is negotiable, then there is no gender difference in terms of asking; men and women are similarly likely to negotiate their salary when that kind of information is clear, that the social cue is you are able to negotiate. In that case, there is no gender disparity, and women are similarly likely to ask. So we find it relevant in our context as well because a lot of time professors do not put a clear policy about regrading, so students may have to take that into their own guess, whether this is an appropriate behavior to make and approach their instructors about it. So I found that kind of ambiguity is relevant in terms of the context when the grade change policy is ambiguous.

John: I’d be really hesitant to put in a procedure explicitly for negotiating grade change requests, but I could see how that could make a difference when it’s normalized as an appropriate policy. But are there any ways that you can see, based on your results, where we could try to compensate for this, where we could try to offset this difference in grade change requests between men and women?

Cher: So in addition to making a transparent regarding policy, I’ll be arguing, first of all, we are not trying to push women to over asking, basically. And a lot of time, we also find in some other literature, they find when you pressure women to ask, the results are not always favorable. So that is truly not our intention. The point we make about transparency in terms of regrading is to make that kind of policy more formal, and sometimes maybe also imposing some cost in terms of the regrading in terms of like, the professor may be able to regrade the entire exam or the assignment, typically, would be, like, not enforced, when students are just bringing your attention to one particular question they think they should get more points and argue for it. But if they have to come through a particular procedure, for instance, making those requests through like documentation, or they have to make some arguments for the points they think they should earn, it would make it clear to students in terms of what they have to jump through, and then other policy that we think may be helpful is in terms of providing better signals. In terms of the performance, for instance, “How is the grade assigned? What is the metric? What is the rubric that a professor uses to assign a higher grade when the student has performed well?” and just really to show them like these are the guidelines why a student would get an A instead of getting a B. What are the distinctions between the different grades, for instance, to make it clear to students why they’re getting their grade, especially when the grading is based on something that’s more ambiguous, like essays or something.

John: So if you grade essays, make sure you provide a rubric that provides a clear delineation and makes a grading process more transparent to everybody.

Cher: Exactly. So when the women are getting a grade and they see, for instance, “I’m getting a B, but I’m qualified for getting an A based on the rubric,” then I will be more confident that I am eligible to ask the professor in terms of whether my grade is accurate or not.

Rebecca: I think it’d be interesting then, the next step would be to see if it’s not just gender differences in asking, but then also, if it’s students that grew up in lower income families or other kinds of situations that might be less likely to ask in the first place.

Cher: Yeah, I think that would be interesting, but we do not have that much information about student income, so we are unable to really look into it. But we do look into some of the different demographics, including their race composition, and we do not find much differences when we account for those differences. And it doesn’t seem to be the main reason that’s driving those gender differences, but there might be some other interesting questions that could be asked in addition to the gender disparity.

Rebecca: One of the things that always comes to mind is if you start putting in policies or things that make it more transparent, it still means that a student would have to take action in some way, so they would have to feel empowered to be able to take those actions. So it’s always curious to kind of think about those power structures beyond just gender.

Cher: Right, I agree.

John: With providing something in a syllabus for challenging grades would probably help reduce those based on what you said.

Rebecca: Especially because it establishes that behavior as something that would be appropriate in that context.

Cher: Yeah, exactly. Because when we ask students in our survey, “Why don’t you ask?” because we start asking them whether they ever consider asking their instructor to reconsider their grade at some point, and if they do, we follow up in asking for the courses they consider to make that request and the reasons they decided not to pursue it, and we find that a non-trivial fraction of students actually just never thought of it. [LAUGHTER] So they don’t think that is appropriate or that is available, so that is one of the reasons we believe it’s good to have a transparent policy in place.

Rebecca: So we usually wrap up our conversations by asking “What’s next?”

Cher: Okay, so the future, so right now our current research is going through the review process, and we have thought about a few directions that we can follow up on the study. One direction is to look into the gender interaction between students and instructors, and we got this question a lot when we presented our work. And a lot of people are curious whether female instructors are getting more requests, and what are the results when you consider those gender interactions? And I think that could be one of a very interesting to look into. And one very interesting thing, I think, from our current study is, when we design our lab experiment, we intentionally take those mechanisms out. So, in our lab design, we shut down a lot of potential mechanisms that could be very important in real life. For instance, the fear of backlash when women ask, which I have found some evidence in current literature, that is totally shut down in our lab experiment. So that is not the reason that’s explaining the gender differences in terms of asking, based on our lab experiment. And we also shut down the gender interactions because the participants in our experiment do not have to interact with a real person. So they just have to make that decision whether they want a regrade or not, and their wish will be granted, basically. And that also shut down any potential gender differences in terms of negotiation skills, if men have been negotiating all along, since they were little, they may become better at it, and that may be contributing to the gender gap in that case.

Rebecca: Those sound like some really interesting follow-up studies.

Cher: Yeah. And we were also thinking about, one very cool thing that could be done, is to follow up with the students into their labor market experience and see how their negotiation behavior may actually lead to some differences in their labor outcomes. Are people who tend to negotiate their grade in school are getting better outcomes when they are working for others, for instance.

John: And I know many alumni programs collect data on first salaries out of college, and that might be an interesting place to do that linkage since you have the administrative data there.

Cher: Exactly.

John: Excellent. I really enjoyed your study, and I like the way you designed the experiment to elicit some detail on why this is happening, because the studies I’ve seen, mostly… just in terms of the labor market… just report that this is occurring, but really haven’t done as much, other than the study you mentioned with List and others, haven’t done that much to try to figure out why it’s happening. So I really enjoyed it.

Cher: Thank you so much. We enjoy the research ourselves, too.

Rebecca: Well, thank you so much for joining us.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

126. Pandemic-Related Remote Learning

Over the last two weeks colleges across the U.S. have made the decision to shift all classes from face-to-face to remote instruction in an attempt to reduce the spread of the COVID-19 virus. In this episode, Flower Darby joins us explore the challenges and the opportunities associated with this transition.  Flower Darby is the Director of Teaching for Student Success, an adjunct instructor in several disciplines, and the author, with James Lang, of Small Teaching Online. She is also one of the developers of the Online Teaching Toolkit created by the Association of College and University educators (or ACUE).

Show Notes

Transcript

John: Colleges across the U.S. have recently made the decision to shift all classes from face-to-face to remote instruction in an attempt to reduce the spread of the COVID-19 virus. In this episode, we explore the challenges and the opportunities associated with this transition.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Flower Darby, Director of Teaching for Student Success, an adjunct instructor in several disciplines, and the author, with James Lang, of Small Teaching Online. She is also one of the developers of the Online Teaching Toolkit created by the Association of College and University educators (or ACUE). Welcome, Flower.

Flower: Hi, John. Hi, Rebecca.

Rebecca: Hi.

Flower: Thanks for having me.

Rebecca: Thanks for joining us quickly.

Flower: Absolutely.

Rebecca: Today’s teas are

Flower: I’m drinking a tall iced tea.

Rebecca: It can never be tall enough these days, right?

[LAUGHTER]

Flower: That’s right.

John: I’m drinking Ginger Peach Black tea.

Rebecca: I’m drinking English afternoon. Sometimes you just got to go with comfort.

John: This would be one of those times.

Rebecca: Yeah, I think so.

John: We’ve invited you here to talk about the transition that faculty are having to go through throughout the country, and probably throughout the world, on fairly short notice to migrating classes from face-to-face to remote instruction. And a lot of faculty are really anxious about that. What advice do you give faculty in terms of where
they should focus their attention, especially if they haven’t done much work with online instruction?

Flower: I think the most important thing to start with is frequent communication with your students. Students are also very anxious, and so even if all you’re saying to your students is that you don’t know yet how it’s going to go, I think that that really puts students at ease, and the transparency will really serve everyone well. In addition to that, I know that centers like the one that you have there are offering all kinds of support workshops, tutorials, self-help, articles, all kinds of resources that faculty can avail themselves of. I’m not sure that the word is consistently getting out to faculty members. So, I would encourage you to turn to your teaching and learning center or your learning management system support area in order to find out what they’re offering. Schools across the country have broken down: Here’s the basics that you need to know to get up to speed with your LMS, especially for people who aren’t familiar with or don’t typically use it.

John: One of the nice things is how widely those resources are being shared. Look online and there’s lots of places, and your teaching and learning center is likely to be sharing them with you fairly regularly as long as you open their emails. [LAUGHTER]

Flower: I was working one-on-one with several faculty members yesterday in our drop-in support sessions, and my experience was that faculty didn’t know about all the websites that we launched last week and all the resources. So, as you said, John, one of the heartwarming aspects of this current situation is to see how institutions are sharing resources with each other very openly. So, as you said, do a little exploring, see what’s available generally online and what your own institution is offering as well.

Rebecca: I think along those lines, though, there’s also information overload because there is actually so much being shared right now. So if we’re looking for specific topics or subject matter related to this, what are maybe a couple of things that faculty should focus on to just get going?

Flower: That’s a great point, Rebecca, because I myself have felt that I’ve been in a blizzard of emails and resources, and distilling the awesome information into usable and organized material has been a real challenge this past week. So, again, I think the main thing at this point is to communicate. So, learning how to use the announcement tool in your LMS, deciding what is going to be your primary communication strategy, it might be that
you’re going to use email. Then the next thing that we recommend is getting your syllabus and a course schedule into your LMS if you haven’t already done that, and then third, understanding how to use the grade center to allow students to track for themselves how they’re doing in your course. Those are communications, uploading basic files or putting basic, like I said, course syllabus type of information and then beginning to understand how to use the grade center. That would be my recommendation.

John: And in terms of the syllabus, perhaps an updated or some type of addendum for the syllabus might be useful to let students know how things might be modified given the shift in instruction, right?

Flower: Great point, John. Faculty, while we don’t want to let go of or change our learning outcomes for the course, it is absolutely the case that many of us might be modifying what the original plan was, we might be changing the structure of tests and quizzes or creating new and different assignments. So yes, I love that idea of post an updated or an addendum to syllabus. And of course, maybe you also include a prominent statement that says “Subject to change with appropriate notice” to the students. One of the things that I’ve always communicated to my students is that if I make any changes in the syllabus or the course schedule, it will always be to your advantage. And I think students appreciate that sort of sense of security, that knowing things could change, but it will be done to help them if needed.

Rebecca: I think one of the other things that faculty are feeling a little overwhelmed by are all the possible tools and technologies that they can use, right, and sometimes this is opening wide doors of possibilities that they didn’t know existed, but then also, there are so many possibilities… “And my colleague here is using this tool and
my colleague over here is doing this tool, should I be doing that, too?” ends up being this common question, and I know my response has been “Don’t introduce too many new tools because information overload or that now there’s a whole learning curve there.” What are your thoughts on this? There’s so many tool possibilities.

Flower: Sure, Rebecca, I think we’re exactly on the same page, and I’ve been doing this kind of support work for several years now. And my philosophy really does align, as you might imagine, with the book, Small Teaching Online, and James Lang’s original , which is to start small. In terms of technology, I always recommend faculty to choose
something that they themselves are comfortable with, and to not make the mistake of trying, as you said, either several new tools, or possibly trying a tool, a technology, that is so sophisticated and complex that it’s outside of faculty’s comfort zone. So, I’m a big fan of deciding something that you feel comfortable with… starting small. One of the things that I’ve been thinking about is we’re kind of in triage mode, I mean, pretty clearly right now,
but maybe four weeks from now, we may have settled into a better rhythm and you may be able to add or layer on additional approaches or technologies or different ways of engaging with your students. Again, as long as you’re communicating with your students, this is what we’re going to start with. And then later on, if you have the bandwidth personally to learn something new, or maybe after surveying what your colleagues are doing, you identify and isolate the one thing that you really want to bring in. Definitely keep it simple and understand that if you as
faculty are not comfortable using a tool, it’s going to create additional challenge for yourself and for your students as well. So stick with what you’re comfortable with.

Rebecca: I think that’s really good advice, and sometimes faculty just need a little reassurance that what they know is okay.

Flower: Absolutely. For sure.

John: Might you also recommend that they check with the students to see how it’s working and ask them if they have any barriers to whatever they’re doing? We have some people in the region who live in areas without broadband, or there’s some people who are living in households where they can’t afford wireless.

Flower: Yeah, there’s a really robust national conversation going on, which is helpful to really think about the student experience. Once again, I think it’s really important to remember that we’re all people and worldwide this is a weird situation, and everybody is under unusual stress and anxiety. And so another one of my basic rules of advice or guidelines here is to practice empathy and flexibility as much as you can. And so that said, I’m seeing contradicting opinions or different approaches, I should say, “Should we be asking students what technology they have available? Or should we not? Is that too intrusive?” I think, again, being transparent with your students and saying, “Hey, we’re going to try this, and if it’s not going well we will figure it out, we’ll change the plan,” can be a really helpful way to go. And then the other guideline that I’ve been sharing with faculty is to go low tech as
much as possible. So I know many faculty and many institutions are suggesting synchronous, live lecturing, audio and video options. Those are actually the hardest and most complicated, and the most prone to fail or challenge. That’s the peak of what we could be doing. In order to ensure the greatest accessibility, including both student access to technology and also any students who might be using things like screen readers or other tools, going lower tech and
using the tools and functions that are within the learning management systems, such as PDF readings and online discussion forums, and quizzes and assignments, those are actually the most possible to create success. Again, for the moment, it may be that later, after you’ve taken the pulse of how your students are doing, you might add more. Or you might host optional synchronous sessions, maybe a virtual office hour or a review session. But for the most success, I recommend going low tech, aiming for the lowest common denominator.

John: And that would certainly satisfy lower bandwidth requirements for people who might be on slower connections.

Flower: Absolutely.

Rebecca: What are some things faculty might want to think about if they know that a lot of their students are using mobile technology rather than desktops and laptops? So in addition to this lower tech approach, are there other things that you would recommend when you know screens might be small that our students are relying on?

Flower: Great question, and I’ve long thought that higher ed is way behind the curve on mobile learning. If you look at industry or corporate training and professional development, there’s some really great mobile apps now. I’m not
saying that now is the time to go out and find a new mobile learning app, let me be clear about that. But I think higher ed has some work to do here, just a couple of simple strategies to consider. First of all, keep in mind the powerful computer that the smartphone is, and again, you have to be careful not to assume that everybody has a smartphone, but it can be a really interesting tool. Maybe students will record video reflections on their smartphone camera, the tablet camera, and upload those or maybe instead of a long, robust written assignment, maybe
you’re going to be okay with little blurbs of text that students can type with their thumbs on their device. It’s a time for flexibility, for creativity, for rethinking the way that you normally do things, and just embracing the adventure, really.

Rebecca: One of the things that I’ve thought a lot about too is making sure that you’re not putting too much emphasis on things that have nothing to do with your learning objectives. So if grammar and spelling really isn’t part of your learning objective, then some forgiveness over mistyping,[LAUGHTER] and maybe using voice commands and
things like that and using voice to text maybe is appropriate in this case.

Flower: Yeah, Rebecca, I think this is a time to rethink everything, honestly, in higher ed. And I’ve been thinking hard about “What are we going to reflect on looking back when we’re through this immediate crisis situation?” Absolutely, I’m a fan of: if the scholarly citation isn’t really needed for this particular demonstration of student knowledge, then maybe you don’t need to require that. So, I would invite faculty to really critically examine all of their usual practices because it’s not the use right now.

Rebecca: What are some of the questions that you’re getting hammered with?

Flower: Faculty have different levels of experience and comfort, and so some faculty are like, “Okay, I already use the Learning Management System, help me think about additional ways to engage my students online.” The necessity of building and creating and maintaining community cannot be overstated. Again, especially in precarious times such as
these, so many faculty want to know how to engage and interact meaningfully with students. Then we also have, of course, the very predictable question about “How do we do what we do in person such as a lab, or a studio or performance class or field work experience? How do we do that in an online setting?” That’s complicated and challenging, but faculty are resourceful and creative people, and I know that they’ll figure it out. The main takeaway for that question is really identify what the learning goal is for that activity, and then think creatively about how students can achieve that learning in an online or remote setting. Now, keeping in mind that it may be the case that you start to have students do some kitchen sink science labs, or some living room dance moves, or whatever it might be. Students don’t have to stay in the learning management systems, again, with their devices, they can capture video, they can take pictures and upload the evidence of what they did. Just a matter of really focusing on that learning goal and then thinking about the activities that will help students… and again, we have to be careful
not to assume that all students have all the things, that providing options for students to achieve that learning no matter where they are.

Rebecca: Can we circle back to this community piece? You’ve mentioned facilitating community is really important. You’ve talked a lot about communication. What are some ways to get students to come together and feel like they’re still a cohesive whole, rather than disparate people who have been dispersed across the world or across the nation?

Flower: So, before all of this happened, if you’re familiar with my work, you may know that I’ve really focused a lot on increasing the social connections in online classes because there is an inherent distance. It is most often the case that students doing online classwork are by themselves sitting at their home desk or at a coffee shop. It
is unusual for students who are doing online classwork to be sitting with another student or with others. And so just really thinking about that physical isolation, and then thinking about how we can’t use the non-verbal cues that we use when we’re in the classroom. So if we’re explaining something as we’re presenting a mini lecture, and we see a whole bunch of furrowed brows or we see that students are clearly off daydreaming about something else, we can
adjust our approach, we can stop, slow down, re-explain, ask the students what their questions are, and we don’t have that real-time feedback in an online environment. So it’s just very important to be really intentional to cultivate that. It can absolutely be done, you think about how we interact in social media settings. We can engage with other people in online situations, but it takes a little bit more intentionality. So, be visibly present for your students, post those announcements, return assignments, timely answer emails… students still say all the time
that they just wish their online instructors would answer their emails. But those are just ways that you want to be visibly present, posting in an online discussion forum, those kinds of things. And then encourage students. You know, I wouldn’t be surprised if it might be helpful to just create a discussion forum just to say “What’s on your mind right now? How can we help? What are you dealing with? What are the challenges?” and just encourage people in
the class to interact with each other as people.

Rebecca: Wait, we’re all people? [LAUGHTER]

John: Students don’t always have that perception of their faculty face-to-face. This is a nice opportunity to open up in ways that perhaps you haven’t done in the past.

Flower: For sure.

Rebecca: I think a lot of faculty will be teaching remotely from their homes with their own levels of distraction and pets and kids and relatives.

Flower: Yes.

Rebecca: Other habitats, right, to their households, just like students. And I think the more that we can share that and that we’re also trying to manage, or even strategies that we’re using for managing that, could actually be useful as a model for students as well.

Flower: I read a really funny, the beginning of what I think will be a series in The Chronicle of Higher Ed this morning, and it was basically, I think The Chronicle has now dispersed to all working from home. And this one reporter was just describing the challenges of sharing a house with three other working adults and two pets that
don’t get along with each other, and trying to be professional, and be on video conference calls or whatever when the cat’s rear end is brushing against the monitor. [LAUGHTER] So I do think, Rebecca, your point about being really authentic about the challenges that we’re all facing, and again, just practicing flexibility, both for your own approach, and then encouraging your students to do the best they can and you’re there to support them. I don’t think
we can message that frequently enough.

Rebecca: I certainly had cars getting drawn across my keyboard yesterday. So… [LAUGHTER] And up my arm and on my head. [LAUGHTER]

Flower: It’s a challenging situation, and this is just one aspect. You think about the potential financial impact that students and their families might be facing as so many businesses are shutting down right now. You think about if a student becomes sick themselves or somebody in their family becomes sick, there’s so many potential challenges
and barriers. I really think this is a moment for humanity to shine without overstating that, and just supporting each other and being willing to be as flexible as we possibly can, helping students achieve the learning, holding them accountable, but being really willing to flex and empathize as needed. I guess I would just reiterate that we should be kind to ourselves, we should not expect to be online teaching rockstars, we should remember that this is not online teaching as we traditionally think of it, this is a triage mode remote delivery of instruction. And we can’t become really well developed online teachers on the spin of a dime. So be kind to yourself, be patient, take it slow, do what you know how to do. It may be the case that in coming weeks, you can add more, you can become more educated as you avail yourself of the resources that your center and others are providing. But ,just kindness is all that I can really recommend to yourself and to your students.

Rebecca: I think that’s a perfect note to end on, and a good reminder that that flexibility goes both to yourself as well as to your students.

Flower: Absolutely. These are unusual times. We’re all freaking out about lots of different things. And so we have a job to do, and students have a job to do, and we can band together and support each other. I’m just thinking about what movie will be made by Hollywood. [LAUGHTER] I mean, there’s got to be tons of movies that will come out after
this, but specifically about higher ed, that would be interesting. [LAUGHTER]

Rebecca: I think the term you used earlier was weird times, and I would agree, we can all be together in these weird times. [LAUGHTER]

Flower: Just practicing basic humanity and consideration for your fellow human beings, I think, is gonna go a long way.

John: It was just so much nicer reading Stephen King novels than it is to live in one of them.

Flower: That’s true. This is bizarre. Let’s just admit that and determine how we can best move on.

Rebecca: Well, thank you so much for the tips, and also just allowing us all to realize that we’re not alone.

Flower: That’s a great reminder, Rebecca, we are not alone. Let’s help our students feel like they are not alone, and we’ll get through this. We’re resourceful people.

John: We always end with a question. What are you doing next?

Flower: So, John, just this morning I offered and was encouraged to write a new column for The Chronicle. Now, I haven’t written it yet. But the tentative title of my new column, which I hope to develop in the next couple of days, is “Okay, So You’ve Pivoted Online. Now What?” and I really want to talk to people about, there was all this adrenaline, and all this frenzy and excitement, and then after we get through this rush, I can imagine that people might begin to deflate, they might become discouraged, the prospect of finishing this semester, it might not be really energizing to people. So I would like to write out some suggestions for regaining your energy and your enthusiasm and discovering the benefits of teaching remotely in this particular situation. We’ll see, I haven’t written that yet. We’ll see.

Rebecca: Yeah, I know that some of the conversation about graduations, and in my department it’s senior exhibitions and things, those kind of capstone moments that are really special and how we can make them special remotely, maybe they’ll be extra special.

Flower: So much opportunity for creative thinking right now. I’ve been wondering if this is going to be the demise of the higher ed conference or other industries as well, if we’ll ever get back together in person or if we’ll find so many other ways to interact virtually that things might be really different from here on out.

John: I know in our workshops, we’re seeing a lot of people coming in over Zoom that we’ve never seen before in workshops, and we’re hoping to see a lot more of them in the future. So, it’s opening up this type of remote access to people who have never tried it before, and that’s a really positive aspect.

Flower: Yeah, I love the focus on the opportunities that this situation is affording us. And then let’s think carefully, when we get to a point that we can kind of look back on the situation, I think higher ed leaders really need to be thinking critically about what needs to change to support effective teaching with technology, because if
there’s one thing we’re learning here, it’s a staple, it’s a support that we can’t do without. And yet many institutions don’t really support the effective use of technology in our teaching in a really sort of simple and sustainable way. So I’m, I’m encouraging, again, specifically leadership in higher ed to think critically about centers such as the ones that we live in, and about the role of instructional designers and “How do we make this much more of a core function in support of our institutions?”

Rebecca: I hope those conversations start.

Flower: I’m gonna do everything I can, talk to everybody I can about it. [LAUGHTER]

Rebecca: Agreed.

John: And I think those types of support are more valued than they ever have been before. I think a lot of people have discovered the instructional designers and the teaching centers across the country.

Flower: 100%. Again, that’s an opportunity that we’re being afforded right now is to help people see what we can do and access those people who haven’t come to our workshops before and demonstrate our value, a real opportunity to do that right now.

Rebecca: Well, thank you so much.

John: Thank you. It’s great talking to you again.

Rebecca: Stay well.

Flower: Thank you, you too. Thanks for having me.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

125. Peer-Led Team Learning

Many studies have found that peer-led team learning is effective in helping students learn. In this episode, Dr. Christina Winterton joins us to discuss her study of the factors that result in more productive relationships between peer leaders and the students they work with. Christina has returned to SUNY Oswego as a full time visiting professor in the Department of Biological Sciences, and was previously the Associate Director of the Collegiate Science and Technology Entry Program at Lemoyne College.

Show Notes

  • Julia Snyder, Research Assistant Professor in the Department of Science Teaching in the College of Arts and Sciences at Syracuse University
  • Jason R. Wiles, Associate Professor of Biology and Science Teaching in the College of Arts and Sciences at Syracuse University
  • Peter Arcidiacono (2020) 122. Differential Grading Policies. Tea for Teaching Podcast, February 26th.
  • Student Assessment of their Learning Gains (SALG)
  • Michelle Miller, Director of the First Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University.
  • Winterton, C. I., Dunk, R. D., & Wiles, J. R. (2020). Peer-led team learning for introductory biology: relationships between peer-leader relatability, perceived role model status, and the potential influences of these variables on student learning gains. Disciplinary and Interdisciplinary Science Education Research, 2(1), 1-9.
    https://doi.org/10.1186/s43031-020-00020-9
  • Ryan Dunk, Graduate Student in Biology in the College of Arts and Sciences at Syracuse University.

Transcript

John: Many studies have found that peer-led team learning is effective in helping students learn. In this episode, we discuss a study of the factors that result in more productive relationships between peer leaders and the students they work with.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Fiona: My name is Fiona Coll. I teach in the Department of English and Creative Writing here at SUNY Oswego and this is my turn to sit in as a guest host.

John: Our guest today is Dr. Christina Winterton. Christina has returned to SUNY Oswego as a full-time visiting professor in the Department of Biological Sciences, and was previously the Associate Director of the Collegiate Science and Technology Entry Program at Lemoyne College. Welcome, Christina.

Christina: Hi, thank you for having me.

Fiona: Our teas today are:

Christina: I am drinking green tea infused with raspberry, pomegranate, and strawberry.

Fiona: Amazing. I am drinking Tazo’s Refresh Mint tea with a wink.

John: And I am drinking Republic of Tea’s Spring Cherry Green tea. We’ve invited you here today to discuss the dissertation research that you’ve done on peer-led team learning in an introductory biology class. Can you first describe what is meant by peer-led team learning?

Christina: Sure. Peer-led team learning is an active learning technique that typically gets implemented in courses where there’s going to be a high enrollment of students, so a lot of the introductory sciences. I specifically helped with the peer-led team learning at Syracuse University. So the intro to bio course there had between 600 and 700 students for the first semester, and then it went to about 500/400 for the second semester. So with all those students, we still had two lecture periods. So all those students were in one of two lectures. So to supplement the lecture, we found a bunch of research on PLTL, and they implemented peer-led team learning at Syracuse University. And what it is, is it takes students who have recently and successfully completed that introductory biology course, they become peer leaders. So, they need to get a B plus or higher in the course, but then they also have to enroll in a one-credit course with a learning specialist. And our learning specialist at SU is Dr. Julia Snyder, who actually was a huge factor in implementing PLTL at all with Dr. Jason Wiles, who is the professor of intro to bio. So, the premise of peer-led learning or PLTL is you take the student who just recently and successfully completed the course, you give them a course about how to be a leader. So, it covers pedagogy, Bloom’s Taxonomy, different teaching styles. And then after they meet with the learning specialists, they are on their own to lead a 55-minute section of the supplemental peer-led team learning course. So, they get modules that they work through with the students. And it’s different than a teaching assistant or a faculty because, first of all, there are no grades involved. The peer leader’s not grading the students, they’re just there to help, and a big difference, a big key, is that the peer leaders are not provided answer keys to these modules. So, they’re truly acting like a fellow learner, a fellow student. They’re relying on what they remember, what their notes are, and they’re just facilitating the conversation. And as a group, the current students are working with this peer leader to construct new original answers and in that they become friends, and they talk more about their feelings and what they really think of certain aspects and how to work through it, and the peer leader is able to relate and share stories because they just took the course.

Fiona: Could I ask some very practical questions about how this works?

Christina: Yeah.

Fiona: So how big are the groups, the peer groups?

Christina: So each group will have six to eight current students with a peer leader.

Fiona: And these are sections that are scheduled along with the massive lecture courses.

Christina: Yeah. Yep, they’re supplemental. So it’s a co-enrollment.

Fiona: And they’re required?

Christina: They are not required. The literature breaks it down with different incentives for different universities to have the peer leaders and the students both choose to be in peer-led team learning. So, for the leaders, typically it’s a learning credit or a leadership credit, and it’s a resume builder. It’s a way for some of the students who are going to maybe take an MCAT to refresh themselves on the material. And then for the students, a lot of the times it’s connected to maybe a portion of extra credit. So that’s the different carrots.

Fiona: Yeah, that’s fascinating. And I want to talk very, very much about the way you assessed this program, but could you give us maybe an example of a teaching technique or something that the peer leader might learn in the one-credit course you were mentioning they’re a part of?

Christina: Sure. I was lucky enough to be able to fill in for Dr. Snyder sometimes when she was out. One of my favorite lessons that we did was as the instructor for the day, I stood up in front of the classroom and all the leaders all had different pieces of paper. And I said, “Okay, listen to what I say and do it as I say it.” And then I said aloud instructions, “Fold the top of the paper to the right towards the center, make a triangle on the side, do it again with the other side,” and I verbally gave them instructions on how to fold a paper airplane. Well, obviously, that didn’t go so great. So I said, “Okay, keep that to the side.” I didn’t even tell them what they were making. They were like “What am I doing?” So they put it to the side. The next thing, I gave them a handout that it said “Step one, fold this” and it was just written instructions. And then for the final round, I put it on the doc cam and folded it with them. And then we looked at the three different outcomes, and we talked about different strategies used to teach or explain. “Did everybody do it the best the last time? Did maybe somebody do it better the second time? Is that right? Is that wrong?” And we talked about different ways to convey the same information to different groups of students when they’re actually leading a session.

Fiona: And do the peer leaders who are forming a cohort of their own essentially, as they’re going through this…

Christina: Yeah.

Fiona: Do they report back through the semester with questions or problems, does that fold into this?

Christina: Yes, I know that Dr. Snyder, as part of the course, she had them keep journals. And so they would reflect on the session. And each leader really only had one group of students. So they knew their names, they knew their habits, they were able to write really good reflective journals and keep notes on “They liked this module or they didn’t like this module. You know, they really liked the debate that we did this week.” Or next week they’d say, you know, “Those concept mappings?…the students struggled, but I think I personally saw them learning even though they didn’t.” So we were able to get really good insight on the models just from those journals.

John: There’s a lot of research that shows that peer instruction can be really effective. What makes it so effective?

Christina: What makes it so effective is actually what my research honed in on, because I was obviously a member of this Biology Education Research Group at SU and I saw Dr. Snyder and Dr. Wiles doing all these really cool things with PLTL. And I know it got results in the form of the research coming out was saying the students enrolled in PLTL had higher grades than their non-PLTL peers, because like you asked, it’s not mandatory. So, some students were in it, and some students weren’t. And you’re able to see the difference in final grades. And I just wanted to know “Why,” right? “What is it about the session? Is it the environment? Is it that there’s no faculty? That it wasn’t in a lecture hall? (It was in a small library room.)” So I just wanted to figure out what about the interaction made it so effective, and I thought like you’d ask that it would have to have something to do with the dynamic between a student-to-student rather than a student-to-faculty. So I was reading up on some literature trying to find out in particular for the STEM fields, what makes peers help each other more. And that led me to some research on role models and relatability and seeing someone you can relate to be successful or do something you want to do. So that led me to develop my question of “Is the PLTL setting enough to leave a student calling their peer leader a role model by the end of this? And what makes up a role model? And did it have an effect?”

Fiona: Don’t keep us in suspense.

John: Although maybe we want to save the answer until a little bit later.

Christina: That’s another question. Yeah.

John: So make sure you listen to the end of the podcast to find out what happened. [LAUGHTER] So you talk about, in your dissertation, the importance of education in the STEM fields. Why is it important to focus our attention on STEM programs?

Christina: So it’s important to focus on STEM programs in particular during the first year or so of college when the students are choosing their major and deciding if they want to pursue STEM after they get a sample of it because the highest rate of students leaving STEM comes between the freshman and sophomore year. So a cause could be what they perceive to be an unwelcoming, large intro course. They come from high school where it’s a smaller class, students know each other, the teachers know them, then you go to the lecture hall and nobody sees you and attendance doesn’t matter, you know. So we can make it feel welcoming and inclusive and that your voice is being heard. And if you have a question, don’t be afraid to ask it and other people struggle as well, then maybe we can keep them and say, “Okay, I still want to pursue this. I got through first year and now I’ll be able to choose an elective that hones in on my specific interests.” So, we particularly don’t want to scare them away with large intro courses, because the overall number of degrees conferred in STEM in the United States is lower than China and Russia and other countries comparable to us.

Fiona: And we also know that the struggle to complete let’s say, STEM is not evenly distributed across different sorts of student groups. That attrition that you described…

Christina: Absolutely.

Fiona: is exaggerated in certain underrepresented groups.

Christina: Absolutely, actually, again, we’ll speak to some of the amazing people I was able to work with while I was at SU in that Biology Education Research Group, one of their studies specifically showed that peer-led team learning closes the gap between underrepresented minorities and their peers in the course. So peer-led team learning can provide this avenue to level out the playing field.

Fiona: Did you have any experience with peer-led learning yourself in your own education?

Christina: I didn’t actually, no.

Fiona: Everything you’ve described makes so much sense. Motivating students, making learning spaces accessible to students.

Christina: Yeah.

Fiona: And I can’t help but think what would be different about my educational path if I had had peers involved in the process in some way?

Christina: Yeah. And I actually went to SU myself, and this opportunity wasn’t available. So it’s fairly new.

John: I started doing it after quite a few years ago, maybe 12/13 years ago, I went to a conference. And I wish I knew who presented this. But there was an experiment, and I believe it was a chemistry class, where they introduced your instructors and they measured the students’ performance for everyone, including the TAs. There was a larger group, and they had larger sections, but they had graduate TAs and they were using undergraduate TAs for the first time. And they measured the increase in performance of students from the beginning to the end, including the TAs, and what they found is among the undergraduate students in the class, the graduate student TAs, and the undergraduate TAs, the undergraduate TAs actually learned the most, and the students who work with the undergraduate TAs had larger learning gains than those who work with the graduate TAs, which is very consistent with the types of things that are being found here. There have been quite a few studies, and I wish I knew some of the citations off-hand, that have found similar effects. It’s interesting. I should also note, in a podcast that we released on February 26, we talked a little bit more about the dropout rate where we specifically looked at the gender difference in dropout rates, and that women are disproportionately likely to leave the STEM fields in response to the relatively lower grades that they receive in the STEM field compared to other disciplines, even though their grades were higher than males in the classes, at least in that study. So the lower performance that many students experience in the STEM fields does seem to be a pretty significant deterrent for, as you said, underrepresented groups. Also, for first-gen students, it seems to be pretty significant in a number of studies. So we’re losing a lot of people who could be really successful in these fields, and as you noted, we’re falling behind other countries. And as we close our borders to the people coming from other countries that could seriously hurt us in the STEM disciplines if we don’t bring more people through here or eliminate some of the barriers to foreign students entering.

Christina: Actually as part of my research, when I was delving into some of the psychology research that came up because I was looking at perception and role models, so we went past biology and education into a little bit of the psych realm, and there was one study that just laid it out really clearly like exactly what you’re saying. And it just got me so interested because their study specifically aimed to put students in a situation and give them exposure to things that would specifically counteract the two prevalent STEM stereotypes and identified the two prevalent STEM stereotypes as: 1. STEM is for white European males, and 2. STEM is only for those who are innately gifted at it. So, that really caught my attention because as an advisor, and as a professor, the students have come up to me a lot of times in STEM courses and said, “I got a bad grade on this. I don’t think I’m good enough at this. I studied for 50 hours that week.” And they always put an emphasis on how hard they had to work to get their grade as if it’s a bad thing. And it really just speaks to that that they truly think that people who are successful in the fields are just born with it and that even your professors never had to put in work. So now after reading that article, I always try to directly counteract those stereotypes for them as well and say, you know, “I have to put in a lot of work to maintain too, you know, it’s normal. And it’s not just for those people and don’t give up, everybody has to put in work, even though they don’t maybe show it or maybe you don’t see that.”

John: And that’s where the peers can be really useful because they just went through it. And serving as you said, as that role model, can be really effective in letting them know that this is just normal. This is how we learn.

Christina: Exactly.

Fiona: And I could see how hearing “This is normal, this is how we learn” might resonate very differently, if that comes from a peer than it does from a professor, even if you’re being…

Christina: Definitely

Fiona: entirely sincere and open.

Christina: Definitely, and especially from someone who has no power over their grades, or what’s going to be on the quiz, you know, so if they can just hear it from a peer.

John: And who not only had just been through it, but had been through it successfully and was able to do well in the course.

Christina: Yeah, and if they can share with them, how many hou rs a week they put in and how they prioritized it as well. That’s gonna all lead to part of the finding.

Fiona: You mentioned that you didn’t have any experience with peer-led team learning in your own education.

Christina: Right.

Fiona: Can you tell us a little bit about how you discovered this as a topic? And what made you decide that this is what you wanted to research?

Christina: Sure. So, I actually discovered this because I went to Syracuse University myself, and I was a biology undergraduate. But, then I decided to get a master’s degree in forensic science because the pieces of biology that I enjoyed most were DNA and genetics. So I pursued a Master’s of Forensic Science, and as a graduate student, I was assigned to be a teaching assistant in intro to bio. So I began doing labs, and as I mentioned before, Jason Wiles is the professor of Intro to Bio, so he was my supervisor then and he became my PhD advisor as him and I started agreeing on some teaching strategies and techniques. So, I saw the students as a teaching assistant in the lab. And the lab by itself is such a different feeling than the lecture. I saw 24 students at a time, not 600, so I was able to see that perspective when I was a teaching assistant, and I was giving them weekly quizzes and grades, and they could come to my office hours, and it was like a tutoring type thing. So, I saw that perspective of them. And then they implemented the peer-led team learning. In addition to the lecture in the lab, this became an option. And they started doing research on it, and Dr. Wiles was leading that research. And then when I chose to get my PhD, he still had active projects going on. And I was looking at those results of how come this increases their grades, and this does increase their grades. And then it just made me think: Why?

John: So that brings us to your study, how did you design a study to investigate these issues?

Christina: Alright, I went back and forth on the design for this a lot because a lot of the research was focusing on assessments, quiz grades, test grades throughout the course and final course averages. So, I wanted to include that because I knew that there had been good results as far as a positive benefit for PLTL and you could find this in the grades, in the difference between the grades. So, I knew I want to collect data on the final course grades, which is easy enough. But then I also wanted to hear from the students what they felt PLTL gave them in terms of their learning gains. So I used an instrument called the Student Assessment of Learning Gains. And I had the students fill that out. And I was able to use that to see their learning gains, and you calculate the total learning gains by just adding up all of their different categories. So, you ask them questions on a Likert scale, and then you get the summation, but I knew I wanted to do learning gains rather than just course grade because there’s a lot of different ways to get a good course grade: you could memorize you could just remember from the slides, you could just be a very good test taker and good at multiple choice. So I just wanted to see if they were truly learning and I really was drawn to the SALG because that tested the things that they were able to carry on to upper-level courses like their ability to connect concepts in biology to one another is one of the questions your ability to connect concepts in biology to other courses you’re taking. “How strong do you feel you could explain a biology concept to a family member? How strong do you feel that you could create your own position after reading a graph of results and defend your position of how you interpreted the graph?” So this was testing actual skills and true learning gains, not just their end course grades. So I want to look at both of those factors. So, I gathered up their final course grades, and I got the assessment of their learning gains as well.

Fiona: So, they provided assessments of their own learning gains.

Christina: Yeah.

Fiona: And the peer leader?

Christina: So, as I got thinking, like a year into the project, I was thinking, all right, my whole study is testing interactions between the students and student-to-student interactions and what makes a good peer leader. So first, I gave out at the end of the semester, there was a qualitative questionnaire one of the questions asked “Do you consider your peer leader to be a role model in any way? Please explain, yes or no.” So they gave me qualitative data there for why they saw the peer leader to be a role model. But then, as I got thinking that I wanted to do this interaction between them, I thought, why not ask the peer leaders, because this is all the student perception so far, Let’s ask the peer leaders, “Do you consider yourself to be a role model in any way? Yes or no? What makes you a role model to your students? Why do you think you’re not a role model?” And we’re able to look at those two qualitative sets of results and find the discrepancy so that if there is a difference between what the student’s looking for and what the peer leader thinks the student’s looking for, but they’re not providing, we can address that in the leadership course, and say, try to implement these strategies, try to display X, Y, and Z. So, we collected that qualitative data. And then in addition to that, I was also wondering, okay, maybe the student just likes that peer leader, like maybe they’re just a funny person. Maybe they’re just nice. So I thought it would be interesting to see how well the peer leader gets to know the student because as we discussed one of these benefits, this is a very small group of six to eight students, you can spend one-on-one time you see this person every week you talk outside of class. So, I also asked the peer leaders to fill out the Student Assessment of Learning Gains, but ask them to do it in regard to their students. So I said, “Alright, Jane, you had Joe as a student, fill out what you think Joe’s learning gains are.” And then I took Jane’s assessment and compared it to Joe’s assessment and found that the student and peer leader pairings who were closest together had the least amount of differences between their assessments had higher course grades and saw the peer leader to be a role model and relatable and higher learning gains.

Fiona: So very elegantly designed approach to this incredibly thorny, difficult, complex process of learning. And so you’ve essentially got two perspectives on a single student’s experience.

Christina: Yes.

Fiona: From their sense of perception, which is also a fascinating idea. There could be a difference between learning and a student’s perception of learning.

Christina: Right.

Fiona: But by having this very elegant combination…

Christina: And part of what led me to saying, you know, there have been errors in students’ self-reported data before, but perception is valuable, the student perception is valuable. And there are other studies that show if a student perceives themselves to be learning and doing well and happy in a course, they’re going to stay in the discipline despite whatever their final course grade is, and that only helps our retention.

John: But you also said that the people who had the better match that way also had higher course grades as well. So, that seems to validate at least the general nature of that instrument.

Christina: Right.

Fiona: Could we talk about these very interesting concepts, so the idea of being a role model, but also the idea of being relatable?

Christina: Yes. So, originally, the design of the project was on role models because there’s so much literature on role models to support it. So the role model question, the qualitative portion, was asked to the students and the peer leaders. But then, as I was analyzing the results of role model, relatability got mentioned so often that it almost seemed like it was a stepping stone to role model status, that to become a role model, you had to be relatable. So, I didn’t have those students anymore, so I did a follow up question to the peer leaders because we still had access to the peer leaders, their course was a little bit longer. So I was able to ask the peer leaders to write about relatability and how relatability came into context with their relationship with the students. So, for the relatability factor, we have the qualitative answers from the peer leaders. And for the role model factor, we have it from the students and the peers, but students were mentioning relatability in a lot of the role model answers.

Fiona: And what does relatability mean in this context?

Christina: So relatability, I was able to categorize into a few different codes and categories. So the students were calling the peer leader relatable. Sometimes it was very clear, they would say the phrase “In my shoes, you know, they were just in my shoes, they know exactly what I’m going through, they’re in my shoes.” Other times, they attributed it to a common career goal. So these are all bio majors. They’re all “I want to be a doctor and my peer leader is taking the MCAT. That’s what I’m going to do in the next year or two. So, I can relate to her because I see her doing this thing that I’m about to do.” What was interesting is that if there was a commonality there, it would always be a shared attainable goal. Like a “In the near future, I’m gonna do that.” It was neat to see that and it’s definitely an attainable goal of the students. Sometimes when you think role model, you think, “Oh, they need to be a famous celebrity and they’re rich,” so it was neat to see the students just saying “They’re a role model because they’re succeeding in what I see to be a hard path that I also want to do.”

Fiona: They’re a role model because “I want to be there next year.”

Christina: Yes. And I relate to them.

John: So we have all that evidence about the effectiveness of your instruction. But this is discovering why it’s working or what specifically allows it to work.

Christina: Right.

John: So, what does this suggest that in terms of developing your lead team leader program, what should that training focus on?

Christina: I would recommend that we share the results with the peer leaders and show them that in a lot of the student responses, when they were saying what they thought made the peer leader a role model, or the peer leader relatable, a chunk of them did say that they got a good grade in this course, and that’s what I want to do, and that’s my goal is just to pass. But, there was a lot more in depth answers than I think maybe I was even expecting, and it went into things like “They were just kind, they were nice. I could tell that they cared if I was learning, they stopped and adjusted the course to my pace,” and that’s a huge benefit that we don’t get in the lecture halls when you just have to keep going, and you’re not able to assess “Are all 600 students with me? Do they all understand?” And a lot of the responses just came up that there was a caring peer leader who truly cared if they were learning, and talked to them and asked them. So, I think that showing the peer leaders that would be a little bit of a difference, because I think a lot of the times it just gets focused on “Well, I did really well in this course.” So, here’s what I think, that there’s so many personality attributes that go into it.

Fiona: I keep thinking of Michelle Miller’s line, that teachers are professional motivators, fundamentally the role that teachers play in drawing out the learning potential in a student in some way. And this seems to point to that in some sense, that what matters the most to the students… yes, the content and skill development matter… but that, in fact, there was some other piece to the puzzle.

Christina: Yeah.

Fiona: That might be effective or emotional.

Christina: And motivate is a word that was frequently used, so much so that I used it as a code for the qualitative piece.

John: And I understand you just heard that a paper from this study is going to be published. We’ll include a link to that in the show notes, and we’ll list it as forthcoming until the paper becomes available.

Christina: Just got accepted. Yes, thank you.

John: Congratulations.

Christina: Thank you. The first paper shows the quantitative aspect, that students had a difference in either perceived learning gains or final course grade, if they saw their peer leader to be relatable and a role model. And then part two is coming up with the qualitative research that explains what those definitions were in the eyes of the students.

Fiona: I have a question that I think goes a little beyond the scope of what you were focused on. But it comes back to this idea of what being relatable means. And I’m thinking about that gap, that the attrition rate or the achievement gap or however you want to talk about that for certain sorts of underrepresented groups. Does your research suggest that the identity of the peer leaders might matter? We know representation matters. We know that this is part of envisioning yourself in a discipline or a field process that your research is getting at. And could you imagine the approach to setting up a peer-led team learning approach could be useful in like specific interventions?

Christina: Actually, that’s a really interesting question that you brought up, because a lot of the research points to that. And then I was expecting to see that, but it was the only time that that was really brought up about the identity of the peer leader was out of all the responses I had, I believe six times the student just said, “Because they’re my age” and interestingly, two times when the identity of the peer leader was brought up for the “No, they’re not a role model,” the answer was “Because they’re my age.” So, I thought that was a really neat finding, that some of the students are viewing age as this positive thing. And other students are using it to say, “No, they can’t be a role model to me. They’re my age,” and other students are saying, “Yes, they’re a role model, because they’re so successful, and they’re just my age.” So, I thought that was really interesting, and I’m not sure, there wasn’t any answers about ethnicity or even gender, and I’m just wondering if the students maybe felt a little like restricted because it was a written out response, and if I just looked at the data of the ethnicities of each group and did a comparison there, if maybe I would find something. But I didn’t go that far in this experiment, but it would be really neat to do so in the future.

John: That could be an interesting follow-up, just to see if the gender and race of the team leader affected the performance of people within the group if they’ve matched or if they diverged.

Christina: Yeah, I agree. I did look at if STEM major and non-STEM major, because in semester one of peer-led team learning, it could be used for any student. So, even if you’re not a bio major, you could be using it to satisfy your science elective. So, I was trying to see if there was a trend with the students who were STEM majors getting more out of it than the non-STEM majors, and we didn’t see a significant difference.

John: Interesting.

Fiona: You mentioned the fascinating finding that when the peer leaders’ assessment of the students’ learning and the students’ assessment of the students’ learning matched, that there was a definite correlation with the overall performance. Did you pull out any information about when there was a mismatch? Or was there anything interesting or useful about the opposite scenario?

Christina: Yes. The peer leader responses, in general, were always a little bit more informative and longer than the student responses. A lot of the time when there was a mismatch, the peer leader would leave a note at the end or something that would say, “This is what I think. But so and so didn’t come to class very much.” In those cases, there was a big discrepancy where interestingly, the student would say, “Oh, I learned a lot, high learning gains,” and the peer leader would put them at a low learning gain, and then there would be a big discrepancy, and I would try to see why and the peer leaders would say, “I never saw them, so I don’t know if they learned or not.” My research group at SU, one of my good friends, Ryan Dunk, who was actually on this paper that just got published, he helped me with some statistics for this. Another question that came up was, “Could it possibly just be that some peer leaders care more and know their students more?” But, we were able to find that there was no significant difference between the peer leaders or the section, so time of day didn’t matter and peer leader didn’t matter, so that was neat to find out too.

Fiona: Fascinating.

John: You mention that the participation in the peer-led team leader program was optional for the students.

Christina: Yes.

John: What proportion of the students in the classes participated in that?

Christina: I can answer that for the semester of the data set that I’m working with, it was after you eliminated students who either didn’t consent to research, or were under 18 and couldn’t be included into research. Out of around 650 of them, 241 participated in my study. So, that means that at least 241 were in peer-led team learning.

John: and it could have been more…

Christina: And it could have been a few more.

John: …if they chosenot to participate or were under eighteen.

Fiona: Are you currently, are you still crunching the numbers and figuring things out from all the data you currently have?

Christina: There’s a good three more publications that are all in progress in various stages. Review, sending it to a partner, and working with my advisor still.

Fiona: Yeah.

Christina: He’s still doing different projects too, though they’re still being studied.

Fiona: Do you think peer-led team learning can work across disciplines?

Christina: I do, and a lot of research was shown to back that. And it’s actually pretty interesting because it started in a chemistry class, and then other sciences picked it up, biology and physics. But, it’s expanded into mathematics, and computer science has had success. And other studies also supported the finding that it would work in smaller classes too. So, it could work at universities, it could work at colleges, it could work at private institutions. So, it just looks like it can work across the board.

Fiona: There’s still room to find out, room to experiment.

Christina: Yeah.

Fiona: We’d like to finish by asking you “What comes next?”

Christina: I think if I don’t answer that I’ll get those three publications out that my advisor is gonna… [LAUGHTER] I’m gonna hear about it. So all the publications.

Fiona: We very, very much look forward to the results from those various publications.

Christina: Thank you.

John: This was fascinating.

Christina: Thank you. I’m so happy to be here. It’s such a privilege.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

124. The Missing Course

Graduate programs provide very strong training in how to be an effective researcher, but generally provide grad students with little preparation for teaching careers. In this episode, Dr. David Gooblar joins us to discuss what all faculty should know to enable us to create a productive learning environment for all of our students.

David is the Associate Director of the Center for the Advancement of Teaching at Temple University, a regular contributor to The Chronicle of Higher Education, and the creator of Pedagogy Unbound. He is also the author of The Missing Course: Everything They Never Taught You about College Teaching.

Show Notes

Transcript

John: Graduate programs provide very strong training in how to be an effective researcher, but generally provide grad students with little preparation for teaching careers. In this episode, we explore what all faculty should know to enable us to create a productive learning environment for all of our students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Fiona: My name is Fiona Coll. I teach in the Department of English and Creative Writing here at SUNY Oswego and this is my turn to sit in as a guest host.

John: Our guest today is Dr. David Gooblar. David is the Associate Director of the Center for the Advancement of Teaching at Temple University, a regular contributor to The Chronicle of Higher Education, and the creator of Pedagogy Unbound. He is also the author of The Missing Course: Everything They Never Taught You about College Teaching.

John: Welcome.

David: Thank you so much. I’m happy to be here.

Fiona: Today’s teas are…

David: I’m drinking green tea.

Fiona: That sounds delicious. I have a cup of wild orange something that’s quite delicious, but not as memorably titled as one might hope.

John: And I have a pumpkin spice Chai today.

David: Wow. Okay.

John: A little out of season but I wanted to try it. We invited you here to talk about The Missing Course. This book is intended for those of us that never received formal training on how to teach, which applies to most college faculty, and maybe we could start with this question of why don’t faculty receive more formal training while they’re in grad school, since most people do end up in teaching careers?

David: It’s a great question. And one that’s not that easy to answer, I think. To me, our graduate programs, almost across the board, almost across all disciplines, still sort of operate with an outdated model of the industry that they’re training students for. And I’m not sure that the model was ever actually something that applied to all graduate students, or even most, but there’s this idea of an academic career as being a tenure-track career where you teach two courses a semester, and the rest of your time is for research, and you are evaluated on your research and scholarship. And so our graduate programs still, for the most part, train students for that career. And what is clearer and clearer to more and more people is that that career is for a very slim minority of people who get PhDs. And as you mentioned, the job for most PhDs if they managed to stay in academia is, for the most part, teacher. And so I do think as well, there’s this notion that teaching is something that content knowledge is enough preparation for, so that if you know something really well, obviously, you’ll be able to teach it, that is not something that needs training in and of itself, that it can be a sort of byproduct of scholarship. And really, anyone who’s had a teacher [LAUGHTER] should be able to see the holes in that logic, that there are better teachers and worse teachers, and that doesn’t depend on how well they know their subject, that depends on how well they know how to teach. So this struck me as a big hole in our education system, and something that I thought would be, at the very least, a good frame for a book about teaching.

Fiona: That hole you describe, the missing part of graduate education, is something that produces a phenomenon you describe early in the book, which is your sense of surprise at discovering a community, a community of people interested in teaching as itself an intellectual and technique oriented discipline, I suppose. And it’s an amazing feeling to happen upon a group of people who are thinking the way you are. And I do wonder if you could maybe expand on what you talk about early in The Missing Course, which is a discussion of how your own view of teaching evolved from that initial focus on content coverage that you’ve just described. Could you tell us about your own transformation in that respect?

David: Sure. I don’t think unlike most people, in that I wasn’t trained to teach at all, I was thrown into the classroom. And to some extent, that’s not the worst way to begin to learn something, is to be thrown into it. But I of course did what most people do, which is that I sort of mimicked what my professors had done. And so my background’s in English, I’m not sure I would have been conscious of this, but my idea of what an English professor does in the classroom is to go and be brilliant about books in front of students… to go talk really smartly about the book in question. And I guess the idea is that you’re modeling interpretation, but you’re not really showing students how you arrived at your interpretation, you’re just showing them the fruits of it. So I would prepare, I was a very devoted teacher as a grad student. All my preparation went to uncovering smart things to say about texts. And what the students would be doing never entered my mind at all, for sure. And so this is kind of how I taught and it seemed to be successful. I don’t know how I was measuring success, but it seemed to go okay. Until I, as an adjunct at Augustana College in Rock Island, Illinois, was given (as adjuncts are often given), a course in which there was no real content to teach. It was a skills-based course, the course was called Rhetoric in the Liberal Arts. And I was meant to teach students how to write academic papers, how to critically read, critical thinking was a big part of it. I was supposed to promote an understanding of the value of the liberal arts sort of broadly. And there were a couple of essays that I was given to teach, but for the most part, there wasn’t really a central text to teach. And so my old method of being brilliant about texts really quickly showed itself to be insufficient. I really scrambled. I had no idea what I was doing. I remember that semester so clearly, because I worked very hard to figure out day to day, there was such panic, “How am I going to fill the class period?” And I guess what I came upon is that it was the students that were my material, rather than material that I needed to explicate for students. Actually, it was these human beings in the room that I was tasked with improving, or helping to develop, or helping them to see things. And that’s just a completely different idea of what teaching is… and it changed my life. It changed how I see the job and the job became increasingly important to me. Because, and you’ll forgive me for rambling, but if the job is saying something interesting about books, well, that’s something that I think can be mastered pretty easily. If the job is help a new set of human beings to reach their potential or grow in ways that they didn’t think were possible, that’s an endlessly fascinating, endlessly challenging, and mostly difficult task. And that, to me, is something worth devoting yourself to. So that really, really sort of turned me on to teaching as a discipline in itself.

John: When you started to change your focus to students, what were some of the first things you did to focus on the students you had, instead of your earlier approach?

David: I think a lot of it comes down to, maybe strangely, trying to find out more about the students themselves. And I often tell faculty now in my role at Temple that the beginning of the semester is really a time to learn about the students in your room and to win their trust. If you’re going to be asking them to do things during the semester, you can’t assume that they’re going to trust that you’re doing it for the right reasons, you’re doing it in their best interest. So, forging that relationship is really the most important thing we can do at the beginning of a semester. So that starts, I think, by turning the tables a little bit. Usually students come into a class and they know the deal, which is that they’re there to find out the professor’s rules. They’re there to read the syllabus, “What’s this class going to be all about?” And the professor sets the course… and the students, if they’re good students, they sort of follow. And so I really like to advise faculty to try to switch that around, and from the very beginning, try to signal at every turn that the course is for the students’ benefit, and this should be their course. And therefore the opening weeks should be about finding out about them, and what they want to get out of it, and why they’re taking this course. And even if the answer to that is “it was required,” that’s good knowledge for you to have too. So, sort of going from there, from trying to find out who these people are, what their goals are, what challenges they foresee, how they want you to help them. That to me is the starting point and signals to them and to you that the point is the students, not the stuff you came up with beforehand.

John: Are you doing that through class discussions, or are there specific activities that you use to try to elicit information from the students?

David: Yes, certainly, I like to do a lot of class discussion and that’s going to depend upon your class size, for sure. A lot of class discussion. I also like to do a sort of initial, no- stakes writing assignment where I ask students for their goals for the course. And this helps me, of course, see more individually, and in a way that doesn’t make them say things out loud, which can be a problem for shy students or for students who maybe need a little more time to formulate an answer. To see what they’re going for, I ask them sort of what things have helped them in the past, you know, what their best teachers have been like, what challenges they foresee. I have to teach writing, it also helps me see their writing level really early on, so I can see where to sort of aim my writing instruction, what kinds of things they might need help with. So, a sort of diagnostic, low- or no-stakes writing assignment can help. But, a lot of it is trying to just be human in the classroom, and be actually curious about these people, and make that a subject of the early class periods in a way that, I hope at least, that the students will get to know each other as well. Is that gonna help as well with motivation down the line?

Fiona: You focus on teaching as this drawing out process from students as opposed to the fill the cup model, which is perhaps the older lecture-based content delivery model… but you do, you focus on this drawing out and you pay some attention to the verb “elicit.”

David: Yes, I do love that verb.

Fiona: It’s a very good verb. But part of the way you use that verb is to turn your attention towards active learning techniques…

David: Yes.

Fiona: … in the classroom. And could you give a brief sense of what you feel are the most important evidence based nuggets to know about active learning as a technique?

David: Sure. I’m increasingly convinced that more than any particular strategy or technique, the most important thing for teachers is to have a solid understanding of how people learn. And what we know about how people learn is that it doesn’t work by pouring knowledge into people’s heads. You don’t learn by someone giving you knowledge. We learn by actively revising our preconceptions. And so even just to have that as your starting point, that students aren’t blank slates to be filled up. But students come in with all sorts of ideas, usually wrong ideas, but all sorts of ideas about our subjects. The job is then to help them hold those conceptions up to the light, see that they’re lacking in some way, and revise them. So I try to let that guide whatever strategies I use in the classroom, so active learning strategies is a really vague term to refer to anything you do in the class where students are doing stuff. So, there’s nothing wrong with lecture per se, but lecture that assumes that just telling students information is enough is going to fail. So if we lecture we want to lecture in a way that elicits… there’s that word… that elicits students’ attention, that intrigues them, that sort of draws them in, that can work really great. We’ve all had really good lecturers in the past. And usually they’re the ones that get our attention, that present us with a puzzle that we try to solve. It’s us, as students, who do the learning, we, we can’t do the work of learning for our students, students have to do it. So, if that’s our guide, then that can sort of help us chart class periods. Active learning strategies are often a combination of trying to get students to see what their intuition is, get to see that the intuition is in some way insufficient, and try to solve that problem. If what I initially thought was wrong, how can I be right? That’s often what I’m trying to do with an active learning strategy is help students do that, if that makes sense.

Fiona: Most definitely. And one of my absolute favorite parts of The Missing Course is the sheer number of practical, applicable ideas for this nurturing, motivating, eliciting process you describe. It’s wonderful just as a resource for ideas, the book is fantastic. Do you currently have a strategy you’re trying or that you’re feeling very excited about? Do you have an anecdote from your recent teaching to put something into focus a little?

David: One thing that I really love that I’m kind of endlessly fascinated by is this idea of naive tasks. I do mention that in the book, but it’s something that I often talk with faculty about. It’s particularly helpful if you do need to lecture if you have large classes, or if you teach in a discipline that is very content heavy, and you have a lot that you need to tell students. The idea of a naive task is you give students a puzzle, or a challenge, or a question that they’re not yet equipped to solve. And you give that to them before you give them the information that would equip them to solve it. The example that I always use comes from a guy named Donald Finkel who wrote a great book called Teaching with Your Mouth Shut, and he writes of a physics professor who gives his students before a lecture, this challenge, which is that he wants them to picture a canary in a very large sealed jar. And the jar with the canary inside is on a scale. And the problem is this “if inside the jar, the canary takes flight inside the sealed jar, does the reading on the scale change?” So, I put the challenge to you two, what do you think? Does the reading on the scale change if the canary takes flight?

Fiona: I’m feeling very elicited at the moment.

David: Yeah.

John: I read that, and I wondered that, because I don’t think you answer the question in your book, do you?

David: Oh, good. I’m glad I didn’t. I don’t remember. You got to make the reader do the work. So, what’s your intuition? Does the weight change when the canary is flying in the air inside a sealed jar?

John: My intuition would be no.

David: The weight stays the same, there’s still a bird in a jar, even if it’s in the air?

JOHN and Fiona: Yeah.

David: And does your intuition change if the jar is open, does that change things?

John: My intuition would change, then, I would say no in that case.

David: That the scale would change?

John: My intuition was that the scale would change if the bird took flight.

David: Okay, that is correct. In a sealed jar, the amount of force pushed down by the bird’s wings is exactly equal to the weight of the bird. That’s how flight works.

Fiona: Of course, right?

David: And so that would keep the reading on the scale the same. If the jar is open, the air can escape.

Fiona: Yeah, right.

David: So, this is a great example of a naive task for students who have not yet learned the laws of weight and mass and force. They have to struggle with this, and it’s kind of an intriguing puzzle. It’s a challenge. And in trying to answer it, and of course, you can sort of prod them along and help them consider the various factors, they’re engaging with the material in a way that they’ll want to know how this works after working on the subject. You’re priming them for your lecture. So I often like to work with faculty and I give them this example. This is always my example, because it’s one I remember. I actually don’t know very much about physics, but I can tell you this. And then I ask them to try to come up with naive tasks for their own discipline. And that’s sort of like “What’s a puzzle? What’s something intriguing that you can give students that they won’t be able to do yet, they won’t be able to solve yet or you think they might not be sure about?” and try to give that to them before you give them the information that would help. So I do like that idea of that the timing of when we give students activities can matter, and can make it more effective when we do tell them things. That’s been something that increasingly I talk to faculty about.

John: And the nice thing about that, too, is not only does it make them curious and want to know the answer, activating that curiosity, it’s also forcing them to activate whatever prior knowledge…

David: That’s right.

John:…whether it’s actually correct knowledge or any misconceptions they may have, but it makes them consciously aware of what they know and what they don’t know. So, they’re not only primed to find out more, but they’re also ready to make more connections and try to integrate the materials.

David: Yeah, ideally, it will leave them in a place of sort of dissatisfaction with their current model of how things work, which is where you want them.

Fiona: You do describe the learning process as a process of revision… revising, exactly as you’ve described, preconceived notions or prior knowledge, updating it, expanding it, refining it. But, as you suggest, this doesn’t feel easy to students. There’s a lot of pushing of comfort zones…

David: Right.

Fiona: …There’s a lot of truly difficult risks sometimes involved for students.

David: Yes.

Fiona: And so how do you approach that piece, the effective piece, or the willingness of students to maybe trust your methods or to actually engage in these essentially revising oriented activities? What do you do about student motivation?

David: It’s a very important factor. I mean, what we know about student motivation and a lot of this I take from the great book How Learning Works from a lot of folks at Carnegie Mellon that came out maybe 10,12 years ago. Their chapter on motivation sketched three main factors from the literature that govern motivation in educational context. And the first two are probably maybe the more apparent. One is value. The more students value a goal, the more they’ll be motivated to pursue it. That seems obvious. The second is efficacy. If students feel like they’ll succeed, they’re more likely to be motivated to pursue something. That makes sense to me too. The third one, which is maybe less obvious, is the extent to which students perceive that their environment is supportive. It’s a very important factor. If, as you say, we’re asking students to take risks and significant learning does involve taking risks, they’ll be more motivated to pursue those risks if they feel like they’ll be supported if and when they fail. We want them to be comfortable with failure. We want to give them the sense that they’ll be supported. So, how do we do that? That to me is again, part of this opening weeks of a semester trying to cultivate trust. And part of that is through signaling to them at every turn that your intentions are for their benefit to help them, not because of your agenda. But part of that, I think, is also trying to cultivate a supportive community between students, among students, kind of apart from you. I mentioned in the book a number of times research by the sociologist Polly Fassinger who tried to sort of pin down what are the most important factors that govern why students participate in class. And what she found, maybe surprisingly, is that more important than anything that the professor does is the students’ sense of their peers as being a supportive group. And that makes sense to me, that students wouldn’t feel comfortable opening their mouths if they felt like they might be the object of fun or they felt like maybe they weren’t in the mainstream of the class. Again, a lot of this work is best done early in a semester. Cultivate community. Often that means trying to encourage students to bring in informal parts of their lives to class, get them to share bits of themselves, get them to, if not become friends, become comfortable with each other as people they can talk to and share an experience with. If I’ve got that, if I’ve got a class that seems to genuinely like each other, respects each other, wants to find out what other people are thinking might want to learn from each other, that makes my job so much easier. And I think most professors can think back to classes they’ve had where the students really gelled. And those to me are the best classes, and not just because it’s pleasant, but it seems to magnify the possibilities for learning if students want to be there, and usually what makes students want to be there is not that they like you so much, but that they like each other.

Fiona: Could you give us an example of a relatively community building activity you can do early in a semester?

David: Yeah, what I really like to do, and it does take a little bit of time, but I think it’s worth it. This is an idea that came from a colleague and friend of mine at the University of Iowa, Ben Hassman, he calls it question roll, and it’s a way to take attendance at the beginning of each class. And it’s certainly easier with a smaller class, but you can do it with larger classes with a little bit of finagling. Instead of just calling out people’s names, you ask a question, and you go around the room and everyone has to answer the question as their way to signal that they’re there. And the best way to do it is for these questions to be absolutely easy to answer. They should be like “What is your favorite movie?” or “What’s something that your parents cooked for you when you were a kid that you have fond memories of?” And so they’re often informal. I like to make them have nothing to do with class content. You can, of course, make them be lead-ins to discussion, but I like to give students an opportunity to speak up that is so easy that they’ll speak up. Basically, I think there’s something that happens after someone speaks up for the first time in a class period that makes it more easy for subsequent times. So, that’s one reason to do it. The other thing is that if you do it regularly, students get to know each other, they remember that so and so likes spaghetti. They remember that so and so likes comic book movies. And so that’s a kind of easy ritual that helps students bring a little bit of their outside lives into the class. And again, signals to them that this class might actually matter to those outside lives, class might be something that they want to do not just because they have to, but because it’s something central to their goals and ambitions.

Fiona: And I’d imagine that’s an approach that also includes you in that learning community, right? You learn?

David: Of course, I have to answer the question.

Fiona: One of the chapters in your book is devoted to what you call teaching the students in the room. Could you give us a little bit of an overview of that approach?

David: Sure. I do think that if there’s anything that teaching is about, it’s about helping particular people develop and learn. And so that is going to change. This is what I was talking about earlier, that’s going to change depending on the human beings in the room. And so I do think that as much as most professors are preparers and were good students themselves and like to do lots of preparation to make sure that they have every “T” crossed and “I” dotted before entering the classroom, there’s a limit to what you can do without meeting and understanding the people you’re going to teach. And again, it’s that idea of thinking of the students as the material rather than the course content or the text. So I do think it’s important that we do more than just tweak our teaching, depending on who’s in the room, but we try to leave a lot to the particular people we have in our classes.

John: One of the things you described is something I’ve been doing recently which is starting with a shell of a syllabus and letting the students pick some of the content on that. And I’ve done that to a varying degree, but I found a dramatic increase in the amount of student engagement when they have some say over the course concepts. Could you talk about some examples where you’ve used or heard good examples of that type of tactic?

David: Yeah, I do love that tactic as well. I recommend to faculty to make a list of things that they absolutely have to have in their courses that are non-negotiable that have to be in their classes, and things that aren’t on that list give to students to decide. This is, again, a way to get as you say, greater engagement, have students have a sense of ownership in the class. I’ve had real success when I taught a survey of American Literature. And I gave students as the assigned text one of these big Norton Anthologies of American literature, these huge 1200 page, very thin paged, big books of poems, and short stories, and excerpts from novels. And what I did was I left slots throughout the semester, throughout the calendar for poems, but I didn’t choose any poems, and I had each student assigned to a slot on the calendar and I said “On your slot, we are going to read the poem you choose. You’re going to introduce it for the class, talk maybe five minutes about it, but then we’re going to talk about it, you’re going to put it on the calendar.” You can do this with bigger classes through voting, through putting them in groups to choose, there are ways to do it. And then I had as the first writing assignment, the first essay for students to justify why they chose the poem. “Why should your classmates read this poem? What’s so good about it?” And the reason I loved how this worked in that particular class was one, of course, that it had that engagement that students owned part of the class, they had their little patch of land, but it also made them read the book. They had to look through this anthology that otherwise they would have just flipped to what I assigned, this made them browse and look for a poem that they could pick and I think it made them think about, “Well, what makes a good poem? What kinds of poems do I like?” Certainly, some students just picked very, very short poems, and we were grateful for that too. But I think students surprised themselves. They discovered things they wouldn’t have found otherwise. And they thought a little bit about, “Well, what’s my taste in poetry?” which is a wonderful thing to encourage, at least in my students. I’ve also had success, I do this all the time now, with letting students determine policies for the class. I do it all the time with technology policy, using technology in class. This is easy for me to give to students because I don’t care at all about what students do with their phones or laptops. I’m not a stickler, other professors feel differently, and that’s fine, of course. But that was an easy thing for me to offload to students because I don’t care. And what I found is that inevitably, students come up with a technology policy that is more strict than anything I would have come up with. I asked them to think about what really annoys them about people using phones in class and come up with regulations, and they tend to regulate pretty heavily when left to their own devices. And it works better if the rules are coming from them than from me. I just don’t like to play policeman, that’s not a role I’m suited for. So policy is a good thing to give to students, whether that’s a technology policy or late policies some professors do, they leave it to students to come up with. And that’s another thing where you’d be surprised that students are not that laissez faire. Some tend to be quite strict when they suggest late policies. But sort of going through your syllabus and looking for “What can I give to students to come up with?” And I think that you’ll see that there are sort of outsized effects from letting students decide. Student ownership is, I think, a really helpful concept for professors.

Fiona: And a totally practical question. So is this an activity you do on the first day, first week kind of thing? And then do you codify it in a formalized syllabus in some way?

David: Yes. So, usually, if I do a technology policy activity, it’ll be on the second day, it’ll be sometime in the first class periods. And what I usually do is I put them into small groups and have them brainstorm possible challenges with using technology in class and possible rules that they want. And I tell them I want them to think about their experiences as students, that they are veteran students, they’ll know probably better than I what comes up from a student’s point of view. And then I open up a document on the screen and I say, “Call out your suggestions.” And I type them as they call them out on their transcriber. And sometimes I have them vote if there are some things that are different, but often they kind of find consensus. I sort of try to guide a discussion, we find consensus. It’s a sort of constitutional document that I add to our learning management system. It lives there for everyone to see and I almost never need to refer to it. But there have been times where I do refer to and I say, “Let’s remember the rules we put in place.” And again, it helps that they’re not my rules. It helps enforcement if I can say “This is what you came up with, do we want to revise them? Maybe this doesn’t work anymore, that’s fine.” I would much rather be their learning coach, for lack of a better term, than to be Nurse Ratched in enforcing rules.

John: The only problem I’ve ever had with this strategy is the secretary keeps nagging me for a copy of the syllabus a week or two before the semester and I say “We won’t have a syllabus until after we meet” and I just open up a document which has the learning objectives and broad categories, and then we spend that first class period just working it all out.

David: I tend to make a distinction between the syllabus I give to the people upstairs and the sort of living document that governs what we do in the classroom. Some people make appendices to their syllabus. I think there’s things that the college needs to know, and I think that’s appropriate. And there’s things that the college doesn’t necessarily need to know.

Fiona: I wonder if I might turn the direction of our conversation towards the way that you talk about faculty and teachers themselves in this book, because many of the ideas: that learning is a process of revision, of challenging your preconceived notions about how something might work, you actually suggest that teachers themselves do the same sorts of processes in terms of their own experience as learners of teaching.

David: Yeah.

Fiona: Could you talk a little bit about how you ask people to revise their teaching?

David: Sure. I think you’re right that how I think about students learning influences how I think about giving advice to faculty on improving their teaching. And I think it’s important to remember that most of us did not have a rich education on how to teach. And so we sort of have to teach ourselves as we go, which does mean that it’s a process of revision. It’s also the nature of teaching that it’s such a complex pursuit. The product in question, the thing that is produced by our teaching, is spread out over a whole semester and diffused by how many students we have. And it’s very difficult to assess how well we did. And you can come out of a classroom thinking “That was a great class, I did wonderfully.” And you can come out of a class period thinking “Oh, that didn’t work well,” and equally have no idea why it worked or it didn’t. So, I do think it’s important for faculty to know that it’s a difficult process and to give attention to trying to assess, trying to evaluate, trying to stay on top of what they do that’s healthy and works, and what they do that doesn’t work. So, I do try to encourage what we might think of as good writing advice for teaching. And that’s thinking of a drafting process when coming up with a lesson plan, just as I tell my writing students to write a really bad draft as their first draft just to get something down, knowing that you’ll come back to it. I do the same thing with teachers I work with, I say, “First just fill the class period. Come up with a class period and then try to be critical in looking at what you came up with. Does that meet your goals? Does that do what you think students should be doing? Does that engage students?” And keep going back to and going back to it. The other thing that I think is really important is after a class period, or after a unit, or certainly after a semester, writing down how you think it went before you forget. I find that, especially in August for a fall semester, but even in January, if I’m teaching something I taught before, I am always kicking myself for not having taken good notes on how I changed things from what I wrote down, because our lesson plans don’t often survive a class period intact. And so trying to keep good notes, trying to see teaching as a sort of career long pursuit that you’ll benefit from having an archive of notes to self. Just sort of developing those habits I think are really important. I try to be aware of this whenever I talk teaching, but certainly when I was writing the book, that most faculty members are, without a doubt, overworked and underpaid, and almost without exception, not given enough credit or time for their teaching. And so I think it’s all the more important that we try to come up with strategies that help us to be better teachers that don’t keep us up all night grading, and that don’t add to our considerable anxiety and stress. I think we need to sort of navigate within what’s still a pretty unfair system for most faculty members when giving advice. So I really try to be aware of that.

John: Talking about that issue of the effort required for grading, you do recommend giving students the right kind of feedback. Could you talk a little bit about how we can do that? We know that students benefit from having feedback. But the more feedback we give students, the more time it requires. What type of feedback would be most helpful for students without overburdening faculty with the work of grading?

David: That’s a really good question. I think the most effective feedback comes at a time when students can still use it. So, too often I think we give students feedback after the fact, we give students feedback on their essays or on their assignments when we give them their grade, and then we’re already moving on to another unit. Often, because of time constraints, it might take us a month to grade a paper. And we’ve already moved on for a month to another unit when we’re giving them feedback on something they did last month. And so that can tell them a little bit. But most students, it doesn’t help them that much to find out that a month ago, they made an error with a too long paragraph or something. So I really try to give students feedback while they’re still working on something. And so the way to do this without doubling your work is to give them much less feedback after the fact. So I think it’s entirely appropriate to give just a grade and very minimal feedback at the end, if you give lots of feedback in the middle. So when I teach writing, I have all of my students, for their essays, they have to turn in a rough draft, and this is after a number of checkpoints where they’re working on components, but they turn in a rough draft at a midway point. And I meet with them one-on-one. Again, this is something that can be adjusted for bigger classes. But I try to give them feedback at that point, like really extensive feedback. That’s where they hear from me the most. And why that’s effective is, well one, because they still haven’t gotten a grade, there’s still something they can do with the feedback, but two, it allows me to be much more of a coach rather than evaluator. And that’s where, at least personally, where my strengths as a teacher lie. I’m much worse at justifying a grade than I am at giving advice on how to improve work. So I never answer in those conferences, questions like “What grade do you think this will be?” Or if they ask me that, I say, “Well, it’s not done yet. This is a draft, I don’t grade rough drafts. Here’s what I think you need to do for this to be better.” I try to sort of strenuously resist being evaluator there. But I do think looking for ways that you can shift when you give feedback to earlier in the process can help you keep that feedback formative. I think it’s useful to have this distinction between formative feedback, which is feedback designed to teach, designed to help students grow and learn, and summative feedback, which is feedback designed to deliver a verdict on their performance. Summative feedback is important. Certainly an engaged student will want to know whether their work met the grade. Certainly we have institutional pressures that want us to give them a summative grade at the end of the semester. But to me, formative feedback is so much more valuable, it is teaching rather than just grading. And so looking for ways whenever possible to make our grading formative, I think, is good practice.

John: Do you recommend giving some sort of light grading though, even just “They completed this or not” just to provide incentives for those who may not be as intrinsically motivated?

David: For sure. 34:21 Yeah, I think that can help. I think that it’s worth considering the ways that our grading policy, like how much weight we give to each element of the course itself, is a rhetorical signal that tells students what we value and so I think, unless you’re able to do away with grades altogether, and most of us are not because of our institutions, it’s worth looking for ways to use that to help us teach, for sure. And extrinsic motivation is not worthless, even as most people, I think depend on it too much. But if you’ve got the grades yeah, I try to use them strategically.

Fiona: Could I ask about the chapter which you’ve entitled Teaching in Tumultuous Times, which in many ways pulls together all of the threads we’ve been discussing. What it means to meet students where they are, what it means to work as a learning coach in order to develop long-term, perhaps civic oriented ways of being in the world, but also opens up into issues of inclusivity in teaching and what it might mean for teaching to be a space in which we can actively address some of the more alarming directions that the world is tending at the moment. Could you talk about teaching in tumultuous times?

David: Yes, I believe very strongly that teaching matters. And I believe that because of my experience in the classroom. I really think that the transformations that can happen through working closely with young people are real, and they are incredibly meaningful to me. And once I realized that teaching is the pursuit of helping people develop, or change, or learn, it then became inevitable that I’d have to be committed to helping every single student develop, or change, or learn. And I recognize that I am not going to be successful in reaching every single student, but my practice needs to be committed to helping every single student. So what that means on the one hand is student centered teaching, as we’ve been talking about what that means, on the other hand, is looking closely at our status quo of teaching and realizing that it doesn’t serve every student equally well. I think it is decidedly the case that our normal way of teaching rewards students who can navigate the school environment, it rewards students who have better preparation, it rewards students who are mentally healthy. It rewards students who are physically capable and we want to ask ourselves, “Is this what we want to promote in our classrooms?” It’s not what I want to promote. And so that sort of led me into thinking about inclusive teaching practices, which I talk about not nearly long enough in that chapter, but as part of that chapter. I do think if we’re committed to helping every student learn, we need to think about making our classrooms more accessible. We need to think about practices that unwittingly, unwittingly, privileged students who have had better schooling, we need to think about ways that we are reproducing inequality. The other part of that chapter, or one other part of that chapter, has to deal with talking about politics in the classroom. And to me these things go together in a sense, because they are, as you implied with your question, they’re kind of about taking responsibility, or seeing that teaching does produce changes in our students and trying to be responsible for creating citizens. And I think that’s kind of a difficult pill for some professors to swallow, who say, “Well I teach chemistry. I don’t produce political thinkers,” but I do think that we have more influence than we know. And it’s worth thinking about what kinds of thinking we’re encouraging in our students. For those times when politics do come up, and they do come up more frequently than you think, it’s worth thinking through what we’re trying to achieve and what we’re trying to avoid. What really helped me in writing about politics in the classroom, and you’ll know this if you’ve read the book, is coming across some material from philosophy of education on the concept of indoctrination. And I really like this because indoctrination is one thing that everyone can agree we don’t want to do. And so it helped me to clarify my thinking to get clear on what we’re trying to avoid. So there are a number of philosophers who have worked to define more closely this concept of indoctrination, and what they’ve come up with is that indoctrination requires us to use our authority to inculcate an uncritically adopted belief. And so keeping that in mind that we have authority that can be used for ill, and that we want to avoid as students adopting beliefs without examining evidence without being critical. That helps me think about how I do want to talk about politics. So I want to privilege open mindedness rather than closed mindedness. I want to foster students to have more control rather than me having the control, I want to work against the authority that I might have. So things like that, I think, came out in that chapter. It’s a chapter that I get asked about probably most of all, I think these are subjects that are on a lot of teachers minds. And I think for good reason.

Fiona: The other piece of this puzzle is, of course, how the authority you just described that inheres in the figure of the teacher in some way in the classroom is itself not evenly distributed. So different sorts of teachers do and don’t match students’ ideas of what an authority figure is, what a professor looks like, what a teacher looks like.

David: Absolutely. Right.

Fiona: And so I wonder if you could talk a little bit about how these techniques you suggest also are risky?

David: Yes.

Fiona: For professors or for teachers who are already themselves in some way at risk just by trying to be in a classroom in the role that we call teacher.

David: Yeah, I think that there’s a real problem in advice given to teachers that ignores the many kinds of teachers that there are and ignores the, the huge differences in power that belong to different kinds of teachers. I learned this very vividly when I worked with grad instructors, all novice instructors, particularly young women struggle with authority in the classroom with students not respecting them, with students openly challenging them. And it’s something that I think we need to take very seriously. So it’s worth remembering that there’s no one size fits all teaching advice. You need to take a teaching approach that works for you particularly as a teacher. But it’s also worth remembering that no matter how powerless any one faculty member feels, institutional realities ensure that students have less power, at least within the institution. There are many other kinds of power, and those are real as well. But as long as we have the power to grade students and the power to fail students, we have to remember that giving students more control, giving students more power where we can, is almost always going to benefit us, no matter our position. And so that’s a difficult thing to get across, particularly for me as a white man, relatively healthy, relatively young, though that’s less and less each day. So I do try to be aware of that, that what works for me is not going to work for every teacher. And that’s pretty clear to me. And so no teaching approach is going to work if it makes the faculty member uncomfortable, if it makes the faculty member feel like they’re going to lose the room, as it were. So this is things that anyone who gives advice, which I guess it’s something that I’m doing professionally now needs to keep in mind, for sure.

Fiona: It also points to the importance of collaboration and community building among teachers, which is something that all of your work points towards, and your suggestion that we take co-teaching opportunities if they come along, or at the very least sit in on each other’s classes.

David: Yes.

Fiona: Which is something that I don’t think we think about as an actual pedagogical strategy.

David: Yes.

Fiona: But which seems so, so important, both in terms of what you described, which is seeing the classroom from the students’ perspective, but also seeing another teacher…

David: Absolutely.

Fiona:… In their environment dealing with what they’re dealing.

David: Yeah, I often think that so many of the problems that afflict teaching as a profession, so many of the reasons why teaching isn’t taken seriously as a discipline. So many of the reasons why students and learning is not understood as well as it should be, has to do with how private connectivity teaching is, how thick our classroom walls are. And actually the traditional independence of tenured faculty, which tells everyone else to stay away and “don’t tell me how to teach.” I do think that we do ourselves a disservice when we don’t pay attention to community, where we don’t try to learn from other people who are doing the same thing we do. Almost every other profession tries to take advantage of this. And so it’s a really strange and terrible tradition that we don’t get in the habit of seeing other teachers. It’s really amazing actually that you can go your whole career as a professional teacher and never, unless you’ve been required to evaluate someone for promotion, never see the inside of someone else’s classroom.

John: We’re starting to introduce an activity this semester where faculty will sit in on other people’s classes and then get together informally, and I think more and more colleges are doing that, but it is a relatively recent phenomenon. And I think it’s one we can all benefit from. One of the things you suggest, and we just talked about a little bit, is breaking down some of the barriers caused by that authority figure in the room. One of the ways you recommend doing that is by letting people know that you don’t know things.

David: Yes.

John: Could you talk a little bit about that strategy?

David: Yeah. And this is something that I often get pushback from faculty of color, from women faculty, but I get a lot of mileage out of self deprecation as a teacher. And again, this isn’t gonna work for everybody. But I do think that in that pursuit of students feeling like they own the class, part of that has to be me, giving up some of my ownership. It doesn’t need to mean inviting disrespect, but it does, to me at least, mean showing students one that I don’t know everything, which is pretty easy, for me at least, but two, that learning is a process and that you’re never done. And it’s not just that I’m not so smart, but that anyone, no matter how much they’ve learned, they’ve got more to go, and they’re going to make mistakes, and they want to keep learning. So I’m trying to encourage this to my students. A lot of it comes from being a parent actually and trying to encourage this in my children, this idea that I can always learn more and that curiosity is the best engine. So I do try to model uncertainty. This is quite common advice, but I do tell faculty all the time that it’s very powerful to tell students “I don’t know, but I’ll find out.” When students ask something, to not feel the need to pretend like you know everything, I think, is really powerful and can take away, in fact, some of that anxiety that young instructors have about being found out, which is something I certainly felt very strongly when I was starting out. If you take away from your concept of teacher this idea that you have to be the supreme authority, it can actually, maybe paradoxically, increase your authority a little bit because you’re no longer aiming so high. You don’t need to know quite so much. You can be a sort of fellow searcher with your students. And that’s a great place to be, where you’re part of that community of looking for answers of learning. You can model along the way for them.

John: And you could also work with them perhaps in trying to explore how you would find an answer to model that more explicitly and get them involved in finding the answers themselves.

Fiona: That sounds great.

John: One of the problems we had in preparing for this is in reading through the book, there were just so many wonderful things that we wanted to discuss that it could have taken days to go through all of this.

David: That is wonderful to hear. Thanks for saying so.

Fiona: We do end by asking “What’s next?”

David: I’m very excited about what I’ve been working on. I’m working on what I see as my next book. We’ll see if it gets there. But I’ve been thinking more and more about what I think of as one of the most important and most persistent problems in higher education, which is inequality of outcomes. We still have quite shocking completion rates in America. These numbers can vary depending on how you measure or what you’re looking at. But typically 70% of all White students in universities graduate within six years. That number for Black students is 40%, which is shockingly low, I think. For Hispanic students it’s at 50%. We see similar gaps for first generation students, we see similar gaps for students with disabilities. We see similar gaps for academic achievement for women in STEM still, we see similar gaps for low income students. So it’s something that I’m looking at in my research. Most universities are aware of this problem and are tackling it, or trying to tackle it, in one way or another with diversity and inclusion initiatives. And what’s really interesting to me is that for the most part, these initiatives don’t tackle teaching. They very often target students support networks. There are financial incentives, of course, but they don’t really tackle teaching. And I guess my intuition is that there’s a lot we can do in the classroom to affect these outcomes. My initial research so far has borne that out, that actually there’s been quite a lot of research in the scholarship of teaching and learning in the past decade on narrowing academic achievement gaps within particular classes. And I guess my running thesis is that by changing teaching approaches, we can do something about these broader completion gaps. So that’s the big picture project. I do think it’ll take a number of years to turn into a book, but that’s where I’m working right now. And I’m finding it very exciting, I think it’s something that’s really important.

Fiona: Your work is exciting and inspiring to us, we look forward to hearing about the developments in this newest direction.

David: Right, thanks so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

123. New Trends in Science Instruction

Science instruction in K-12 education has long been provided as if science consisted of a body of facts to be memorized. The Next Generation Science Standards, however, rely on an inquiry-based approach in which students learn about science by engaging in scientific exploration. In this episode, Dr. Kristina Mitchell joins us to discuss this approach and its implications for college instruction.

After six years as a director of online education at Texas Tech University, Kristina now works for a science curriculum publishing company and teaches part time at San Jose State University.

Show Notes

Transcript

John: Science instruction in K-12 education has long been provided as if science consisted of a body of facts to be memorized. The Next Generation Science Standards, however, rely on an inquiry-based approach in which students learn about science by engaging in scientific exploration. In this episode, we discuss this approach and its implications for college instruction.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Kristina Mitchell. After six years as a director of online education at Texas Tech University, Kristina now works for a science curriculum publishing company and teaches part-time at San Jose State University. Welcome back, Kristina.

Kristina: Thank you. It’s good to be back.

Rebecca: Today our teas are…

Kristina: Diet Dr. Pepper.

John: The same as last time. [LAUGHTER] I have a green tea today.

Rebecca: And I have a nice pot of brewed English Breakfast tea.

John: We’ve invited you here today to talk about the Next Generation Science Standards. How did you get involved with the Next Generation Science Standards?

Kristina: During my work at Texas Tech University in online education, I was doing a lot of consulting for publishing companies on their online and curriculum offerings, which led to a full-time job offer from a curriculum company that works specifically in K-12 and some higher ed science curriculum, and I learned very quickly that one of the newest trends in science education at the K-12 level are these Next Generation Science Standards that were created by college-level science professors in order to change the way we teach science to K-12 students.

John: How did these standards come about?

Kristina: I have to admit that because I’m relatively new to the science curriculum world, I am not an expert on their creation. But I do know that there were several panels of scientists from various higher education institutions that got together to try and figure out from the perspectives of different disciplines, “What do we want our students who come to university to know about science once they get there?” When I was in middle school, I don’t know how much you remember your middle school science lessons, but I remember dissecting frogs and doing our Punnett squares about genetics, but what I remember most is that we would learn those principles of science like Newton’s laws throughout the week, and then Friday, we would do a lab to prove that they were true. And unfortunately, that’s not how any of us do science. Being a social scientist myself, that’s not how I did science at the college level, as a researcher. So, the panel of experts from various scientific disciplines sought to change the way students think about science to think about it more as a process of investigation, rather than something you sort of memorize and then confirm.

Rebecca: The Next Generation Science Standards have three dimensions. Can you talk about those a little bit?

Kristina: Sure. So, these sort of three dimensions, or the three pillars of the way students are learning in K-12 about science, the first is they’re learning disciplinary core ideas. So these disciplinary core ideas are the core features of disciplines like physics or chemistry and the things that they really have to know about that specific discipline in order to build on that knowledge. The second one is science and engineering practices, so these are the ways we study science. And the great thing about the ways that we study science is that it doesn’t change across disciplines. Whether you’re a physicist or a chemist or a sociologist, you’re going to use the same types of scientific practices to answer your research questions. And then the final dimension, the third dimension, are cross cutting concepts. So, these are things like observing patterns or observing cause and effect, concepts that also span across all disciplines, not necessarily limited only to science. So it’s concepts that students can use to understand the world around them, regardless of what questions they’re asking.

Rebecca: Can you give us an example of how these three dimensions play out in a particular grade level?

Kristina: Sure. So when we’re thinking about what students are doing in science classrooms these days that might be different from the way we learned science when we were in seventh grade, students are doing a lot more investigation centered science. So, you walked into your classroom, your college classroom, and somebody walks in and says to one of your students, “What are you guys doing?” And your student might say, “Oh, we’re reading this chapter on…” Rebecca, I think you’re an art person, right?

Rebecca: Yeah, design.

Kristina: “So we’re reading this chapter on Photoshop,” and you ask them, “Okay, well, why are you doing that?” And they’d say “Well, because Professor Mushtare told us to.” In trying to get them away from that instead, the way they’re trying to implement science in the classrooms now would be to get students, you know, if they came into your classroom to say, “Well what are you guys working on?” And they might say, “Well, we’re trying to make a flyer for this project.” And you’d say “Why?” And they’d say, “Well, because we’re trying to figure out how we can make people more excited in this project that we’re working on.” So, getting students to start from the point of “What are we trying to figure out and why do we have to do this activity in order to figure that out?” …rather than “What are we memorizing, what are we learning because the teacher told us to?”

John: So it’s more of a project-based learning or inquiry-based approach to the discipline.

Kristina: Absolutely. And we’ve seen that a lot emerging in higher ed. So, it’s really great to see it reflected in the K-12 space too.

John: But it’s important probably for faculty to be aware of this, because if students are starting to use this approach, as they move forward, they might expect to see more of that in higher ed, where they might see it as a step backwards if they have to move to something that involves more rote learning, for example.

Kristina: Exactly. We’re seeing a lot less “sage on the stage”-type activities, both in K-12 and in higher ed. So it’s really important, I think, that we all kind of know what both sides of education are doing so we can at least know where our students are coming from.

Rebecca: How do you see this new approach helping students be better prepared for college?

Kristina: Well I definitely think that the idea of empowering students to be in charge of their own knowledge is really important. Sometimes I find that a struggle that my students have is doing this sort of figuring things out on their own. If I don’t give explicit instructions with explicit rubrics, they sometimes feel very lost. And so empowering them to recognize themselves as people who can ask questions and figure out the answers to those questions, that goes far beyond just physics class. That goes into just being a responsible and productive member of society. So, I think hopefully we can see students taking a little bit more ownership in their learning because of these trends at K-12.

John: Does it also perhaps give students a little more motivation when they have a goal in mind, and they know why they’re doing things rather than just doing it because their instructors told them to?

Kristina: Yeah, I would definitely think so. I definitely remember my seventh grade year not being passionate about the science, other than “Wow, cool. It’s a dead frog.” So thinking about empowering students to ask these questions might get them more excited about what they’re learning about.

John: In one of our earlier podcasts, we talked to Josh Eyler, who emphasized the importance of curiosity and nurturing curiosity and this approach seems to be very nicely tied into that.

Kristina: Absolutely. And I think about my own political science classroom. One of the fun activities I’ve done and I swear the first time I went through it, it was one of those days where I thought, “Ah… I’m tired. I don’t have anything that I’m interested in lecturing about. We’re almost through this semester, what am I going to do today?” And so I decided, I’m just going to have the students write questions on index cards related to political science and hand them all in and I’ll answer all of the ones that I can. And that was one of the best days of class. I’ve recreated that many times in semesters. And so seeing that these Next Generation Science Standards where we’re trying to get students to think there are some things you learn about each discipline, but there are some practices and concepts that apply everywhere and you just learn how to ask the right questions and you use these concepts and practices to answer them. It’s something that I’m using in my political science class, even really, before I realized that I was specifically doing so.

Rebecca: Do you have any idea how students are responding to this curricula?

Kristina: So every individual curriculum and program is going to be a little bit different, but they are all going to tie into these main ideas. And one thing that we’ve seen in student performance and student preferences is that the students really like doing things in class, having activities and hands on specifically to science. Of course, that often means labs. But one thing that I often talk to teachers about and get a lot of really good feedback on is the idea that we kind of think of doing science looks like it’s in lab coats with safety goggles, but we all know as researchers ourselves that science looks like talking sometimes. And sometimes it looks like asking questions, and sometimes it looks like taking notes or revising models, and getting the teachers and students to realize that that’s all part of what doing science is. They get really excited at the idea of doing science themselves. Of course, they like to see things explode in class, but sometimes just getting them the opportunity to have discussions with each other. It’s amazing to me what the middle school students are capable of, and our elementary age students, what they’re capable of. I have kids myself and I didn’t know that they were learning such higher level concepts of science in class.

Rebecca: There’s big movements to move STEM to STEAM. Is the humanities represented in this new model?

Kristina: So, when we think about STEM, that’s kind of what everyone’s prioritizing these days: the science, the technology, engineering and mathematics… and the A to add it to STEAM would be the arts. So the idea that the arts are also important, but I think there are some key pieces missing here. When we focus exclusively on STEM and on creation and innovation, that’s really good. We need creation and innovation. But as I feel like sometimes we’re learning right now, without a good background knowledge of history, we might be creating and innovating things that either already exist, or have already been tried, or that don’t have a good basis in what we’ve experienced as a society. It’s like they said on Jurassic Park, you spend so much time wondering whether you can and you don’t stop to think about whether you should.

Rebecca: So you were just raising that question about ethics and things like this. So,how do you see this curriculum evolving so that the humanities are better represented or integrate more into our integrated learning in K through 12?

Kristina: Well, one thing that I think is really great about the NGSS is that it definitely starts to sort of de-silo the disciplines of science. So, when we think about physics and chemistry and earth science and life science, when I was in school, those were all completely separate years, both in middle school and in high school. And the NGSS standards definitely start to recognize that we can’t silo our disciplines like that because they have so much to speak to each other. Some questions you need many disciplines of science to answer. And so I’m hoping that slowly, more and more curriculum writers will apply that same standard to the humanities, to the social sciences, and to the arts, the idea that nothing can truly be siloed because social science speaks to even questions like “What scientific research do we do? What research questions do we ask?” That can’t be explained using natural sciences alone. Because yes, natural sciences might seem really unbiased, but we’re asking certain questions and we’re not asking others. So using social science and humanities to help answer those bigger questions. I’m hoping that we see less siloing across disciplines over time. And even at universities, we’re starting to see increases in things like cluster hires, where multiple people from different disciplines all come at a sort of similar question, kind of like what we’re doing here. We’re three people from very different disciplines. But we’re all interested in the question of, you know, what makes teaching good or bad.

Rebecca: One of the questions that’s risen a lot in higher ed, and I know as a popular conversation on our campus, is about fake news and debunking pseudoscience. Do you see that this curriculum helps with that,? …perpetuates that?

Kristina: I think that’s a really good question, and kind of a difficult one, because I think it’s really important to teach students how to interpret the world around them, and what science is and how it’s done, because that can help us determine the validity of the information that we get. But I also think, like we were just talking about, there is a move away from rote learning, from the idea that an expert in a field tells us what is true about that field and we learn it. The fact that we’re moving toward anyone can be an expert, anyone can do the research themselves, anyone can ask questions. Sometimes, I also worry that that pendulum might swing too far, because that’s sometimes where we get people who go online and find whatever Google can tell them about vaccines not being safe or about climate change not being real. And because they feel empowered to be their own expert, they’re treating themselves and their research as equivalent to people who have devoted their lives to getting advanced degrees and who know the answers to these questions. And so I worry that maybe we’ll see the pendulum swing a little too far in the other direction, but I’m not sure.

Rebecca: Based on knowing what you know about higher ed and what you know about this science curriculum, what do you think faculty should be thinking about in terms of what their students might know or might not know, or how they might know differently than they did before?

Kristina: I think that it’s important to teach students what expertise is to begin with, when we think about what it means to be an expert and know something. So, when we go back to those next generation science standards, and think about those three dimensions, the disciplinary core ideas, to me, those are the things that experts already know and that we can accept as the core ideas of each discipline. So, starting from that point and saying the question of vaccine safety has already been established, we know that the vaccines that are here are safe. That’s a disciplinary core idea. But now let’s use our science and engineering practices and our cross-cutting concepts to think about other types of preventative medicine that maybe don’t have the same consensus or new types of vaccines? How would we know that those are safe or how would we know that those are effective? So, I think it’s important not to forget about those disciplinary core ideas that are things that experts have already figured out questions that have already been settled, and then allowing people to start from that point and ask their own questions and do their own research using the scientific method.

John: Changing the topic just a little bit, we get a fair number of people who are interested in the STEM fields when they’re in middle school and even high school, but then they get to college, and all of a sudden, they have to start taking calculus classes, and Matrix Algebra, and Organic Chemistry. Is there any way that perhaps lessons learned from this approach could be used to make college classes more effective and not having so many people get lost along the way once they hit these courses that have been traditionally a bit of a barrier to progress in those areas?

Kristina: That’s a really great question, and we even see that in political science. I teach our research methods class in political science and it has just the tiniest bit of statistics in it. And students are like, “Why am I doing math in a political science class?” And then of course, in graduate school, we’re seeing increasing amounts of quantitative knowledge needed to finish graduate programs and then to compete on the job market. So, I definitely think that the rise of math as something important is a deterrent to some students who might otherwise be interested in entering STEM or social science fields, but I think that also comes down to the pendulum again. I think that it’s important to realize that quantitative methodology is a method of understanding scientific questions, and you don’t have to use quantitative methodology to answer every scientific question. Some of them can be answered using qualitative methods, and even discussing political theory or other types of theories in other sciences or in the humanities. I think it would be good if we could open up the doors a little bit to people who have really high level intelligence and interest and curiosity, but maybe don’t have an interest in math. I think both by making math more accessible and less required, we might be able to get some really interesting new perspectives in our college level courses. I don’t like the idea of using hard math classes as a way to weed students out. And I think that’s sometimes what they’re used for, because I took Matrix Algebra in grad school, and it was really hard. And I’ve never used it since that class… ever again. So it’s not as though it was a skill I needed to be a PhD in political science. It felt more like a class that was used to weed out students who weren’t as interested or weren’t as good at memorizing how to do Matrix Algebra.

John: The point I was trying to get to, I think, is that perhaps if more of these inquiry-based methods were used to motivate the material, it could help get past that. I actually had a similar experience with Matrix Algebra. And when I was learning, I was able to do the proofs and work through it, but I didn’t really understand it until I started using it in econometrics and other disciplines, when all of a sudden it just made so much sense and why all those theorems were there and useful. But we lost a lot of people along the way there. And perhaps that’s something we could build into all of our classes to help motivate students and not just hit them with a lot of theorems and not necessarily even just rote learning, but a lot of inquiry for which the motivation may not be obvious to people when they take it.

Kristina: Definitely. And I think that when I teach my Statistics and Research Methods class, it always is easier for the students, and they always do better, when I tie the math we’re doing to a real question. So even something as simple as “We’re trying to figure out whether the average American is conservative or liberal,” and this is our question that we’re trying to figure out. Now we need to do a difference of means test to figure it out. And so that would be, going back to what we were saying earlier, the idea of you walking into my class and asking my students what they’re doing. Well, they could answer “We’re memorizing how to do a difference of means test from chapter 12 because Dr. Mitchell told us to,” versus saying, “Well, we’re trying to figure out what the ideology is of Americans from different regions, so we have to use this mathematical test so we can answer that question.” So really about framing it in terms of “What are you doing and why are you doing it?” and making sure that we know that the students would answer in a way that gets them excited about the material.

Rebecca: I think context is everything. The real world problems that you’re talking about in the new science curriculum is not unlike the kinds of things that you’re talking about in your classes or even when you have to use math in design class. So, a lot of visual arts students think math is completely irrelevant to them, but when it’s put in context, when you need to do certain things, all of the sudden it makes more sense. I teach a creative code class and we do fairly complicated geometries there, actually. And if you can see the results, sometimes it makes a lot more sense to some of our students.

Kristina: Exactly. So the fact that they would know why they’re doing this and it’s to accomplish a specific goal, rather than just “We’re memorizing this because the professor told us we had to, it was going to be on the exam.”

Rebecca: Yeah. [LAUGHTER]

Kristina: Lord, help us for how many times we hear “Is this going to be on the exam?” That seems like the motivation for trying to learn something. And it would be great if we could move it away from that being the motivation, toward the motivation being “Well, we’re trying to learn this because it helps us figure out this question or this design problem.”

Rebecca: Or the motivation, like, “Hey, I really need to understand this mathematical principle, because I really want to know the answer to this question.”

Kristina: Exactly.

Rebecca: And that leads to where they actually start asking for the things that we’ve prescribed previously. [LAUGHTER]

Kristina: Yeah, and it becomes just a tool to figure things out. And if they want to figure the answer out, then they’re going to need all the tools.

John: Could you tell us what this would look like if we were to stop by a seventh grade class, and this was being put in practice?

Kristina: Absolutely. So if we think about what a seventh grader might be doing, so if we start from that point of the goal is to figure something out, and maybe they’re learning about Space science. They’re learning about Earth and Space and trying to figure out “Why do some objects in Space orbit other objects?” …like “What does that mean? How does that happen?” So, one really fun activity that I’ve seen middle schoolers doing is they have these embroidery hoops and these marbles. And so what they do is they put the marble in the embroidery hoop, and they sort of spin it around on the table. And so the marble is swirling in circles and then they lift the hoop and marbles start shooting everywhere. So it’s very funny in a seventh grade classroom, because you got kids running everywhere trying to chase their marbles. And so at first, they’re just having fun, tracking which way the marble goes and trying to figure out when they lift the embroidery hoop which direction will the marble head, and then eventually they start realizing, “Oh, well, the marble kind of wants to go straight, but the embroidery hoop is keeping it going in a circle.” So, if we think about the sun, with the planets orbiting it, if the sun suddenly disappeared, Earth would kind of want to go straight out in whatever direction it was headed. But because the sun’s gravity’s in the middle, it’s keeping Earth going in a circle around the sun. And so they start putting these pieces together and eventually they come up with this idea of the relationship between inertia, which keeps the earth going, and gravity, which is what keeps it in that same orbit. When they think about what is the core idea they’re learning about, well, they’re learning about orbit, but the way they’re figuring it out is by looking at this small model of what is actually happening at the planetary level. And that’s really reflective of the kinds of things that scientists do in their labs. We’re running models that try to simulate what’s happening in space, or what’s happening in the political world, or whatever discipline we’re doing. We’re trying to create these models to help us understand it. And so that’s what the students are doing, too.

John: That reminds me of a talk that Eric Mazur has given in several places, including at Oswego, where he talked about the Force Concept Inventory, and how, while students were able to plug numbers into formulas and solve problems very well, when they were asked about questions like that, specifically, if I recall, “Suppose that you’re swinging a rock on a sling, and you release it, what would the motion look like?” And basically it ends up flying in essentially a straight line… well, other than the effect of gravity. But when people were asked that, they would bring up things that they had seen in cartoons where they expected it to loop through the air, perhaps as it’s moving. That’s actually one of those things where people in really good institutions who had taken a college class had some really poor intuition. And this type of practice can help prepare them with better intuition on which to build later.

Kristina: There was also a video at, I think it was Harvard’s graduation, where they asked a bunch of STEM majors who had just graduated if they thought they could make a little circuit that would light up a light bulb. And so they handed them everything they would need to do, and everyone’s like, “Oh, yeah, yeah, I got this no problem.” And they couldn’t, they couldn’t figure out how to light the light bulb up, because as much as they had done the math in class, they just had never really done the practical aspect. And so getting students hands-on to do the things with those science and engineering practices in whatever discipline we’re in. It’s really important to get our students in the driver’s seat to start practicing what they’re going to be doing when they leave the classroom.

Rebecca: It also seems like having multiple pathways to the same information is a good way to play with our memory, and make sure that it sticks and it stays there. So, something that’s embodied, something that you memorize, something that’s a couple different kinds of examples.

Kristina: Exactly. Maybe it can move it from our short-term back to the long-term memory to help us keep that knowledge.

Rebecca: So we always wrap up by asking “What’s next?”

Kristina: So I think I’d really like to start exploring what this does look like in a classroom setting for myself. So rather than just focusing on what it looks like for students in K-12, I’d like to see if I can use a similar style of teaching in my political science classroom, and just see what kinds of evaluations I get and how the students like it. I think the idea of getting rid of these silos and recognizing that we all use the same practices and concepts could help us with keeping our college experience for our students interesting and consistent. And we start realizing that they’re going to learn in each of our disciplinary core ideas, but there are a lot of things that we all have in common. So, I’m really going to try this semester, maybe my student evaluations will suffer greatly, but we all know from my past talks that they’re biased anyway. But I just kind of want to see what it looks like if I let my students take a little bit more of a driver’s seat in asking the question, “What are we going to figure out? What do we want to know about the world?”

Rebecca: Do you have a piece of this semester that you’re most excited about that you already have planned in this domain?

Kristina: Well, I’m doing the research methods class this semester, which of course, involves some math. And I think that I would like to use their ideas about what kind of data we should be looking at and what kind of questions we should be answering. Because I think that’ll make it a lot more meaningful for them, rather than if I say, “We’re going to look at the American national election study,” letting them figure out “What question do you want to answer?” And then I’ll teach them whatever method it is that they need to know in order to answer that question. So, we’re going to try it. We’ll see how it goes.

John: In a past podcast, Doug Mckee talked about a similar situation in his econometrics class where he was using a time-for-telling approach. Where basically you face students with problems and let them wrestle with it a bit before giving them some assistance and helping them resolve things, and there’s a lot of evidence that those techniques can be really effective.

Rebecca: So, it sounds like well wishes on your new adventures. [LAUGHTER]

Kristina: Thank you.

Rebecca: Thank you so much for joining us.

Kristina: Of course, this is great. It’s always great. I think eventually I’ll have to come visit y’all.

John: That would be great. Well, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

122. Differential Grading Policies

Students generally receive lower grades in STEM classes than they receive in other disciplines. In this episode, Dr. Peter Arcidiacono joins us to discuss how these differences in grading policies across departments can help to explain the relatively low proportion of female students majoring in many STEM disciplines. Peter is a Professor of Economics at Duke University.

Show Notes

Transcript

John: Students generally receive lower grades in STEM classes than they receive in other disciplines. In this episode, we investigate how these differences in grading policies across departments can help to explain the relatively low proportion of female students majoring in many STEM disciplines.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Dr. Peter Arcidiacono. Peter is a Professor of Economics at Duke University. Welcome, Peter.

Peter: Thank you.

John: Welcome.

Peter: Thanks for having me.

John: Our teas today are…

Peter: Green tea. I got it specifically for this conversation.

Rebecca: Excellent. I have English afternoon.

John: I’m drinking Spring Cherry Green tea.

Rebecca: Oh, that’s a little switch for you.

John: I’m looking forward to the spring now that it’s 10 degrees outside here in Oswego in January when we’re recording this.

We’ve invited you here to talk about your December 2019 working paper co-authored with Thomas Ahn, Amy Hopson, and James Thomas, on Equilibrium Grade Inflation with Implications to Female Interest in STEM Majors. In this paper, you know that disciplines with harsher grading policies result in higher earnings than the earnings received by those majoring in disciplines with easier grading policies. Can you first describe what you mean by harsher grading policies?

Peter: Well, harsher grading policies… you look at average grades in STEM classes and economics, they tend to be a lot lower than in the other social sciences and in the humanities. In the school we look at, which is University of Kentucky, it’s a third of a grade lower. And that holds true across many other colleges I’ve looked at, be it the Duke University or Berea College, it just seems to be a general pattern that the STEM classes and economics tend to have lower grades. And in fact, there was a very nice book written about this with Duke data called Grade Inflation: a Crisis in Higher Education or something like that written by Valen Johnson a few years back, and it was making that exact point about the big grading differences across disciplines.

John: How does your model explain the grading differential across departments?

Peter: Yes, so that to me is where the big contribution of the paper lies, though. I don’t think our model’s particularly satisfactory in this regard. I think lots of people have noticed in the past these differences in grading standards. Where I think our contribution is, is thinking about how those grading standards impact different groups of individuals, but also in what you just asked about: What are the sources of this? And I think a portion of it is certainly demand for courses. So, at the school we look at, the STEM classes are twice as big as the non-STEM classes. An analogy from my own experience… when I first got to Duke, no one signed up for my grad labor class. And that was a bit of a disaster, because that would mean I’d have to do a new prep for some other course. And what I did is I bribed two students to take the class. And the way I did that was I said, “The class is going to be: we’re going to write a paper together.” And it was a great experience for them. They did get a paper, they had a revise and resubmit when they went out in the job market. And what do you know, they both got A’s in the class, but now I’m a little more popular. Not everyone gets A’s and I’m not co-authoring papers with everyone in the class. So I think part of it is these differences in demand for the classes affect things. You hear about weed-out classes, they’re weed-out classes because there’s plenty of students who are interested in taking that class, so they can afford to do that. If you’re in a discipline that has low demand, you’re not going to have a weed-out class because you’re not gonna have any students. So, I think that’s a portion of it. A portion of it can also be things like differences in preferences, maybe it is the case that some disciplines are just meaner than other disciplines. And a third component may be that it’s just easier to grade those classes, it might be easier to grade a math problem than an essay. I’m a little skeptical of the last one, in part because when I talked to high school students, it doesn’t appear to be the case that the math classes are that much easier than the humanities classes, for example, or grade that much harder than. And I think part of that is in high school, you have to take those humanities classes and when you get to college, you have a lot more choice and that influences things.

John: This also seems to relate to the academic labor markets. Because there’s so much demand for classes, there’s often more demand for faculty. So, faculty are not under as much risk of losing their job if they don’t fill their classes. While in markets in those disciplines where grades tend to be lower, labor markets tend to have a little more slack in them. And faculty perhaps may, using the analogy you used before, have to be a little more generous to make sure their classes fill so they can keep their jobs, particularly when there’s a large component of adjunct faculty and contingent faculty.

Peter: I think that’s exactly right. I’ve heard some economics professors brag about their low teaching evaluations. I just thought they can get away with that in other disciplines.

John: And we see that in many STEM fields as well, at least anecdotally… certainly not on our campus…[LAUGHTER] But certainly many STEM disciplines do have that reputation of bragging about keeping their grades low and having high standards and such things.

Peter: That’s right. You know, in my view, it’s when you change what you grade around, that doesn’t necessarily mean that the standards… how much work is required in the class may stay the same. And that’s where, I think what’s interesting about men and women, is that women actually study much more than men. So, if you change the workloads, that’s more of a deterrent for men than it is for women. So, you can think about different mixes of either giving high grades but with high workloads, or lower grades, with lower workloads and student life, on average being different between those things, but they won’t be indifferent between men and women. At least on average. Obviously, this is just an on average statement. There are some women who are not interested in studying hard, and some men who are. On average, that appears to be the case.

John: And you do know, though, that the STEM disciplines not only assigned lower grades, but they also require larger workloads based on the survey data you were looking at, I believe.

Peter: That’s right, substantially higher workloads. And that was my own experience… that where I went to college, I guess economics hadn’t quite crossed over to STEM at the time, but I started as a chemistry major, and there the formulas are a lot more complicated and you had to memorize them. In economics, I didn’t have to memorize the formulas and they were much simpler than those in the chemistry major and I had to work a lot harder in those chemistry classes.

Rebecca: Can you talk a little bit about how the difference in grading policies impacts underrepresented groups in these different disciplines? So in STEM, for example, where you’re saying there’s these stricter grading policies, there’s less women, there’s less underrepresented groups in general.

Peter: That’s right.

Rebecca: How do those things connect? Can you talk us through that a little bit?

Peter: Yeah, actually I had caused quite a stir here at Duke back in 2011, talking about underrepresented minorities, because you do see, for example, African Americans at Duke… They came in just as interested in the STEM classes in economics as white students, but they left at a much higher rate. And so the negative headline was something like, you know, “Potentially racist study says Black students are taking the easy way out,” and that was not the case. It was that they’re coming in with different levels of preparation, due to pre-college inequalities. We have serious differences in pre-college education. And that’s resulted in this coming in and then finding that it’s very hard to succeed in the STEM disciplines. For women, the story is different. And there, because women come in just as prepared as men and are interested in studying a lot more. For them, it’s not the workloads that are affecting this, but more the lower grades themselves, given that they’re working so hard, and then to see these low grades seems to have a bigger effect on women than men. I think one of my very interesting result from some of the work that Stinebrickner and Stinebrickner have done is actually asked men and women, “What grades do you expect you’re going to get?” And then they’ve showed them their grades later and said, “Now, that didn’t work out quite as you expected.” They got lower grades. Why do you think that was, and men were much more likely to say it was bad luck. And you could see that they actually believed it was bad luck, because they continue to take the same classes where they were getting low grades, and they didn’t change their study habits. Whereas women were much more likely to say that it was something about them, either that they needed to study more, or they weren’t as prepared as they had hoped to be for the class. And then you can see that they ended up being more likely to switch to different types of classes and to change their study habits.

John: I think you also found that women tended to not just equally prepare, but I think women came in with somewhat higher GPAs, if I remember correctly.

Peter: They do come in with higher high school GPAs, for sure, and they get better grades in college than the men, both in STEM classes and in non-STEM classes.

John: So the difference seems to be that women are more sensitive to the lower grades or they tend to blame themselves for the lower grades, while the men tend to be less cognizant of the fact that they may be the cause of the lower grades.

Rebecca: I feel like I’ve been socialized to blame myself for not being adequate.

Peter: Yeah, it’s been remarkable, you know I have five boys. And they’re not socialized that way, despite my best efforts to get them to really care about their grades and to think that studying even matters. Here I am, a researcher who knows that studying matters, and yet some of my boys are convinced that if they do poorly, it’s bad luck. It has nothing to do with their studying.

John: One of the things I assigned in my labor class is a reading from, I think it’s a sociologist, who noted that women seem to be more concerned with what she termed “local status,” while men tend to be more concerned with “global status.” Basically, the argument is that men tend to be more concerned with their income expectations and the impact that the courses in the majors have on their grades, while women tend to be much more concerned with the relative grades and seeing grades as a perception of their worth or their value in some sense, which makes them much more sensitive to that grading differential between majors. While men tend not to be as concerned about their local standing, their relative status, while they tend to be more concerned with how the degree will impact their future earnings.

Peter: And I think it’s compounded by the fact that I don’t think people know when they go to college how different the grades are. I think that they figure it out rather quickly, but at first they just don’t know that they’re graded so differently. I mean by the time they figure it out, they’ve already made the switches, which is too bad.

Rebecca: So what do we do so that there’s more representation in these disciplines?

Peter: Well I think at a minimum, that universities should at least have an obligation to tell people, “Here’s what the average grades are in these different classes,” so that students can make informed choices. I personally am in favor of doing something where, like they did at Wellesley, where they put in something that either capped the average GPA in the class, or having all the classes that are above a certain size give the same median grade so that people are not having their decisions distorted by the fact that one department is giving lower grades than another department. It’s not as though they’re learning less in these STEM classes, just because they’re getting these lower grades. They’re actually working very hard in these classes, so, it would be nice if that was somewhat remedied.

Rebecca: I wonder if even knowing the information across institutions and knowing that people get lower grades in these fields, no matter what school they’re at, would actually be useful to those groups.

Peter: Oh, I think it’d be very helpful. Yeah.

Rebecca: Not just within your own institution.

Peter: That’s right. Yeah, I think that’d be great. Because then you know what you’re getting into. And maybe when you get that B in the economics class, you’re not feeling like you failed, that that is a much more typical grade than what you expected.

John: I think a lot of people might object to the notion that there should be the same GPA on the grounds that you’re forcing students into this curve where if you happen to have a class that does really well you might be lowering their grades, which doesn’t always happen in the STEM field, but I think there might be some resistance to that. But the notion of transparency and making people more aware of that early on might help reduce that differential effect by gender.

Peter: That’s right. And I think also making departments aware… the implications that the level of their grading has on things like the gender composition, should matter. I don’t want the way I grade my class to be a deterrent for women taking economics classes. I think that that’s bad. And I’ve changed the way I grade a bit in response to this. If other disciplines are going to give much higher grades, and I want to have a good gender representation in my class, I’m still assigning the same amount of workload. So I’m not making the classes easier, but I am changing the level that I’m curving around, and I typically teach a class that has over 100 students in it.

John: And this is especially important because a large share of the gender differential in earnings is tied to major choice, because the fields that women are more likely to enter tend to offer much lower wages on average, while the STEM fields tend to offer much higher wages. So the differential impact on major choice has quite a bit of an effect on the gender wage gap.

Peter: That’s right. And this is potentially a very low- cost way, at least from a pure financial standpoint, to changing that. It’s not going to solve all of the gender issues in economics or in STEM fields. It’s not going to come close to solving it, but it can put a dent in it, where it’s often very costly to change something like that. Changing a climate is expensive and difficult.

Rebecca: I think I would have been one of those students that would have moved. I’m a person who does a lot of coding and things and could easily have been in computer science. But I’m not, I’m in graphic design, and it’s a field that is much more populated by women. I got much higher grades in the things that I did, and I was very sensitive to grades. So as a student, if it was something that I didn’t do well in, then I thought I couldn’t do well in. And I had a very fixed mindset when I was earlier on in my career. Obviously, it’s much different now. I do much different things and explore other things. But at that time, I think I was incredibly sensitive to that. If you came in as an A student and then it’s like, all of a sudden you’re not getting A’s, it’s like “Woah…”

Peter: Yeah.

Rebecca: “…maybe college isn’t for me, even.”

Peter: Yeah. And that’s just really unfortunate that decisions will get based on nominal differences in these grades, that don’t really have content outside of something that is relative, relative to whatever the professor’s curving around. Another advantage to doing something like this, too, is potentially getting spillovers. Because once you change the gender composition a little bit, then that makes the environment more comfortable. And then that can have multiplying effects.

John: So, why don’t departments want to do this to improve the gender composition of their classes and to get more students in their classes and persisting to graduation in their majors?

Peter: Well, this is sort of where the equilibrium aspect comes in. Because even if you want to do this, everybody ends up ratcheting up their grades, because some departments really need more students. And in order to attract students, you’re going to have to do something if the students don’t have an innate demand for your field. And people have been talking about this for some time, that demand for the humanities has been falling and what to do about it. So, I think that the pressure’s sort of on both sides, even if the STEM departments decided, “Look, we want to have our grades be a bit higher for this reason,” that’s going to lead to response by non-STEM professors as well. So I think it’s hard to get rid of the grading differences across fields, absent doing something that’s pretty draconian.

John: By imposing a grading standard that’s equal across departments.

Peter: Exactly.

Rebecca: I wonder if initiating conversations between the humanities and STEM fields about the interdependence there actually are on those two fields and that people in the humanities should learn things about STEM and people in STEM should learn things from humanities and that there could be more of a relationship there that would actually help start to equalize some of those needs, because there could be some referrals to certain classes or other things that might help boost some of those relationships. I don’t know.

Peter: That’s right. So right now we’re playing sort of a non-cooperative game and move it more towards a cooperative one. Yeah.

Rebecca: Yeah, it’s competitive rather than thinking about “Why are we at a liberal arts institution? And what does an environment like that or general education offer?”

John: Well think about what would happen, though, if grading standards were relaxed in the STEM disciplines. More students would major in them, enrollments would tend to fall in the non-STEM field, and that would tend to lead to fewer jobs for non-STEM faculty. And maybe that’s not something that departments would react well to, and they face the same incentive to just inflate grades.

Rebecca: I don’t think my suggestion was necessarily to do something about the grades directly, but rather to start helping to increase some demand for humanities and helping out some of those…

John: You mean, more interdisciplinary programs?

Rebecca: Yeah, I’ve been thinking a lot about, even for my students, the benefits of them exploring a lot of different disciplines to be more effective in their own, and understanding how other people problem solve, and think through problems, and explore the world.

Peter: I really like that too because if you end up providing these skills that are then rewarded in the marketplace through taking some of the STEM stuff, but combining it with the humanities, that will increase demand. And then that will naturally even things out a bit more.

John: And you could move more from STEM to STEAM.

Rebecca: Hey, I didn’t say it, but I’m totally a STEAM gal. [LAUGHTER]

John: Well I was expecting that at any moment. And just in case some of our listeners don’t know what STEAM is, would you like to…?

Rebecca: Yeah, included the A …it’s for the arts, [LAUGHTER] which I am a part of.

Peter: Yeah, they’re doing that at my kids’ high school, you know, having a STEAM program to try to get that integration happening a bit more.

Rebecca: There’s some research happening about the need for these creative outlets to help with innovation and things in STEM fields. So, there’s certainly some research suggesting this interdisciplinary approach to all of our disciplines could actually improve the work that we do.

Peter: Makes sense.

John: And many STEM students tend to specialize as much as they can in their field. So, that broader training could be useful in the future in making them more creative perhaps.

So, we always end with the question: “What are you doing next?”

Peter: Well, it’s been interesting, nothing quite related to this. I got derailed because I’m actually an expert witness in the Asian American discrimination case against Harvard, and that is taking up all of my time. We released a couple of papers on legacy and athlete preferences at Harvard, and we’re going to be writing some more out of my reports for that case. That’s what’s keeping me busy.

John: Very good.

Rebecca: Well, thank you so much for joining us, this was really interesting, and I hope that it launches more conversations across disciplines.

Peter: Thanks for having me, I really appreciate it.

John: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Savannah Norton.

[MUSIC]

121. Persistence Scholars

A college degree generally results in higher incomes, more pleasant and more stable jobs, greater life satisfaction, and lower unemployment probabilities. Many students that enter college, though, leave without a degree, but with high levels of student debt. In this episode, Dr. Michelle Miller joins us to discuss an innovative program she helped develop at Northern Arizona University in which faculty members work together to discover ways of helping their students successfully complete their educational goals.

Michelle is the Director of the First Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of  Minds Online: Teaching Effectively with Technology, and has written about evidence based pedagogy and scholarly as well as general interest publications. She has been working with a Persistence Scholars program at NAU for the past two years.

Show Notes

Transcript

John: A college degree generally results in higher incomes, more pleasant and more stable jobs, greater life satisfaction, and lower unemployment probabilities. Many students that enter college, though, leave without a degree, but with high levels of student debt. In this episode, we discuss an innovative program in which faculty work together to discover ways of helping their students successfully complete their educational goals.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Michelle Miller. Michelle is the Director of the First Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of Minds Online: Teaching Effectively with Technology, and has written about evidence based pedagogy and scholarly as well as general interest publications. She has been working with a Persistence Scholars program at NAU for the past two years. Welcome back, Michelle.

Michelle: Hi. Thank you. It’s great to be here.

Rebecca: Today’s teas are… Michelle, are you drinking tea?

Michelle: I sure am. I’ve got a… I think it’s some type of green tea. It’s actually imported from China, so I can’t read the label, but it tastes great. [LAUGHTER]

John: I have Forest Fruits green tea, which I picked up at the OLC conference in Orlando.

Rebecca: And I have vanilla coconut tea.

John: We invited you here to talk about the Persistence Scholars program at Northern Arizona University. Could you tell us about this program?

Michelle: Yeah. So this is a faculty professional development experience that works very much as a blended course, and it’s run for the past few years. And it was something that we developed and designed right in our institution to address, as you can probably guess from the name, student persistence, and broadly speaking, student success. And to do so in ways that would complement programs that we already had in the works, as well as some other more traditional kinds of faculty professional development programs and courses that focused more exclusively on teaching or course design. So, that’s how we brought this into this space. And it’s been a really exciting experience to get to build this from the ground up and to run it with a number of cohorts of our faculty at Northern Arizona.

John: Could you tell us a bit about how the program was structured?

Michelle: Maybe I should back up a little bit and tell you a little bit more about some of the roots of this program and why there was such great support for it from the beginning. This came out of some real brainstorming. I was in a group a few years ago that was charged with just really open ended brainstorming about this topic of student persistence. And as you can imagine that, from an institutional standpoint, that fits into some very important questions, such as retention, especially retention from the first to second year which, those who are in this arena know is a really critical area for ensuring that we keep the students who we recruit to our institution and ensure that students can accomplish the goals they set out to when they sign up with us. So, I was in this group and as I said, we had this very open ended charge of saying “What else could we do to support student persistence?” And because of my background with the First Year Learning Initiative, which is another kind of student success initiative at Northern Arizona, my perspective is always “What about the faculty? What about the academic side of student persistence and engaging faculty in advancing that, and getting excited about that question?” So, together with some of the other folks I was working with, notably John Doherty, who I’ve collaborated with a number of times on student success initiatives, we got to thinking, “Well, how could we reach out?” I had seen quite a number of programs or appeals to faculty, which really came at it from a very emotional, or sort of heart perspective, saying, “You know, really think about your students, have compassion for the backgrounds that many of them come from and the challenges they’re facing.” And I think that’s wonderful. That’s great and conventional wisdom about how to recruit people and get them excited about something. They say “Speak to the emotions, get to why.” Well, I think that’s true. But, faculty are a bit of a special case. I think that we’re wired a little bit differently [LAUGHTER] in some ways. And I think that we have to come at this intellectually as well. So I said, “What if we had some kind of a program that would bring people in and really engage them in this very rich scholarship that’s around, not just teaching and learning, but also everything we’ve come to know about the factors… institutional factors… psychological… social factors… all these things that play into students persisting until they do attain that degree?” So, that was the idea. Now it sort of went down on paper and sort of stayed on ice, stayed in a file drawer for a few years. But then my leadership came back to me and said, in the context of some other things we were doing, they said “Wait a minute, what about this program that we had thought up?” And at that point, we were able to really put it together and make it happen.

Rebecca: What are some of the things that you covered as part of this program that would be different than the first- year program that you’ve ran, or other things that are focused on pedagogy?

Michelle: We tell faculty when we recruit them in that this is not the place to start if you do want that traditional, like, “How can I improve my classroom?” teaching, or “How can I brush up on these skills?” We have lots in place for that. So, what is different about this is that it does focus on the scholarship of persistence. And you know, from my background, I’m a psychologist, I’m a research psychologist by training. So I actually didn’t know, and I started to get into the course design and student success game. I really wasn’t aware of just how much really good quality scholarship has gone into this and how people have thought about and really committed to many books and articles, all this knowledge that they’ve come up with, or what impacts student persistence and what institutions can do, what faculty can do. So, it does have that flavor of a slightly different content area that, again, many of us are just not aware of, even if we care a lot about teaching in our own discipline. And I think what’s also different about it is that it doesn’t take a traditional kind of workshop or book group kind of approach. I think those are really, really great. We all see great examples of those in faculty professional development, but this was structured as a blended course, specifically. So it’s designed with a kickoff workshop that lasts about a day. And then we go online and do just some very structured weekly modules, largely focusing on some readings and discussions and one culminating project. So, I think that, as well, is something that faculty rarely have the opportunity to engage in. I think there’s some national programs out there, for example, ACUE’s program… that’s online, but that’s also a full year. And this is a little bit more compact, and I think it’s designed in a way that’s a little bit more manageable with a typical teaching and research load that faculty have.

John: And you also had people do some visits to various places on campus too, as part of that, I believe.

Michelle: Right. This is the culminating project which we tried in the first few iterations to kind of refine this and I think we ended up with something that’s really a standout and here I have to credit my leadership K. Laurie Dickson. Dr. Dickson is a colleague of mine and part of the upper leadership at Northern Arizona University. This was her idea and she really encouraged me to develop this. We didn’t want to have, as a culminating project, kind of a very typical five paragraph essay or research project or something like that. We wanted to push faculty out into some areas that were particularly new. And we wanted to have them engage in some perspective taking on angles and aspects of the students experience and the university experience that they just normally would never do. So we called this the field project… so, a very generic name, but here’s how this played out. It was up to them to design an experience. It didn’t have to be lengthy… didn’t have to be some gigantic multi-day thing, but just something that they could go and do and experience, and then write about it from a very first person, very subjective perspective. And also, we did ask them to kind of tie it back to some of the readings that we had done and some of the concepts that we had seen over the course of the experience. The examples of what faculty came up with were just… it’s mind boggling, the creativity that people brought to this. Now one of the popular ones was to simply go on a campus tour. Now, how many times do we as faculty ever do that? And I mean, I work in a building where the campus tours originate. So, I see them every single day going and coming, the parents, the students and everybody, the student tour guides. And it’s just never occurred to me to ask, “What are they saying? What’s the little back conversation? What’s the mood like among people who are on these tours? What do we tell students and their parents, as they’re coming into our campuses?” So people could opt to go on one or more of these tours, you could also go on a department specific tour, which is also a fairly popular twist, and then reflect back again on “What does this tell us about what it’s like to be a student here?” and to start taking that perspective as a student and thinking about “What would affect my likelihood of persistence?” So that was one, but we’ve also seen many other options on this as well. One very creative faculty member decided to go out physically to these different student support spaces and organizations. And we all read about those, I know I do, I get the email that says, “Oh, here’s the center that we have for veterans. Here’s the center that we have for Native American students. Here’s where you go, if you need help with writing.” Well, we see those, but what do they look like? What do they feel like? Are students there when you visit, and what sorts of activities are taking place there? And she actually put her reflections together as a photo essay. So, she took pictures of the spaces, she thought about the look and feel of the spaces, and through that she demonstrated that she was taking this new perspective. And this was not an art or design professor, by the way, her specialization is in foundational math, so you can see they’re crossing out into other disciplines. So, even something like observing a class that’s not yours outside of your discipline, you can make that work as well. If you come at it from this perspective, not as like “I’m here to critique the teaching and get ideas for my own teaching,” but “What’s going on in the back row? What’s more clear, what’s less clear, how might the mood or the feel of the classroom change if I come over a couple of different weeks of the semester, and how does that seem to me?” So those are some of the things that faculty actually did to experience some of these things from the other side.

John: How many faculty were part of this program?

Michelle: We usually have run cohorts between about 12 and 20 faculty per semester. And I think we’re about four semesters in, so it’s not an enormous program. But you could see over time with a concerted effort and continued dedication to the program, continued support for it, that we’ve now directly engaged quite a few faculty from around the university. And I should say as well, here’s another little twist that I was not anticipating when we sat down to design this program, is that it’s not entirely all faculty either. We’ve also reached out to staff members, for example, people who work within our advising center or our academic support centers, which function as our tutoring centers on campus. In the first cohort or so I just received a request of somebody’s saying, “Hey, my staff would really benefit from this, do you mind if we have a person or two participate in it?” First I said “Well, okay, I wasn’t planning on that. But I can’t see why not?” Well, I soon learned that having that mix of individuals in the cohort is part of the power of it. Because you think academic disciplines are siloed, we are tremendously siloed in terms of units of student support across campus. To see the interplay in discussions and in meetings between people who work in these more direct student support roles, and people in more traditional faculty roles is really amazing. It really cuts across several of those silos as well just in the participation.

Rebecca: Can you talk a little bit about some of the content that participants were surprised by that was counter to what their assumptions were?

Michelle: One of the challenges in pulling the content for this was that I did have to be really, really selective. Being excited about this, of course, I’ve got 100 articles and things that I want to share, and all these concepts to lay on them, and I went with just a very, very few that I felt were the most powerful and the most backed by research. I would say that one of the ones that surprised them, maybe pleasantly so, is some of the academic work around lay theories and belongingness, which is related to mindset. And probably a lot of your listeners are familiar in some way with those. But, in particular, the work of David Yeager, he’s one of the major researchers in this area. His work focuses on how you can communicate to students that things like intelligence and the potential for academic success are not fixed, they’re something that can be built up through effort. A piece of this is normalizing struggle in a way that, just because you get to campus and you feel lonely, and you feel overwhelmed, that a lot of successful people start out that way, so don’t quit. And what’s I think surprising to faculty and definitely was to me as well when I started reading the work is just how powerful some relatively small interventions can be. Just going through, say an online module that exposes students to some of these mindset concepts can result in statistically significant changes to the likelihood of persistence, retention, and things like that later on down the line. So, I think I was surprised, and I think many faculty are surprised by that as well. And that that work is really high quality in terms of the scholarship behind it, the statistical analysis, how the studies are set up. That’s another kind of pleasant surprise too.

John: A while back, we had Angela Bauer on the podcast, who’s now at High Point University. And she had an intervention in the chemistry department there, where just growth mindset messaging that was delivered by slides that were used by all the people in the department eliminated the achievement gap there. So it was a remarkably powerful effect, which is very consistent with what you’re describing there.

Rebecca: Can you talk about a couple of other small interventions that faculty can implement that are really powerful?

Michelle: Another theme that’s come out of the work on this has looked at the effect of structure… increasing course structure so that, for example, instead of the two midterms and a final, we have those distributed smaller assignments over the course of the semester. And that’s one of those things that there’s got to be a dozen good reasons, from the memory research all the way down to mindset, why this is a really good and powerful thing to do. Now, whether that’s a small intervention or not, that could be a matter of perspective, because for some people, if their course is designed in a completely different direction, that could be some major overhaul there. However, I should say that many of the faculty, in fact, most of the faculty who participate in this, are part of our First-Year Learning Initiative already. In fact, that’s kind of why we decided to develop the program as strongly as we did, is we felt it was a really good complement to those courses that were already part of this initiative we have to ensure really best practices in design for key first-year courses. So, many of those courses are already supposed to have that type of design. But this is a way to continue to engage faculty, particularly those who maybe weren’t on the scene when that course was first designed, they show up and they’re saying, “Why do we have all these grading quizzes?” or “How come it’s set up this way?” Well, this gives them some of the backing behind doing that. I think as well, some of the things that we can look at are simply the communications we have with students. So, that’s another area where I think it may be a little bit under the radar, just how important this stuff is for student persistence, that it’s not even the course design or how the course is taught, just the words that get exchanged in, say, office hours, or the tone of the email that you send to a student to respond to them when they write to you with a question. I think that an experience like this gets us to stop and think and say, “How can I tweak my phrasing or bring in some of that good perspective taking to make those communications either more compassionate or gentle?” or to communicate something like a growth mindset that, “Hey, it’s not a matter of whether you got it if you don’t, we’re just going to jump in where you’re at. And with effort, you can succeed at this.” So, I think those are some of the key things that we can bring in as faculty to affect this very big issue of persistence.

Rebecca: Can you talk a little bit about things that you do in the kickoff workshop? Can you talk a little bit about some of the things that faculty come in knowing or not knowing or mis-knowing?

John: Mis-knowing, is that a word?

Rebecca: I don’t know, I just invented it. [LAUGHTER]

John: It is now.

Michelle: Yeah. I love that term. So, there’s that kickoff workshop where we most directly start to query people’s assumptions, knowledge, and misconceptions about persistence, and to introduce them to this idea that yeah, this is a serious area of academic inquiry that’s interdisciplinary, and we can all access it for the benefit of our students. And in a blended course, it’s generally a good practice to offer face-to-face bonding and group cohesion experience first, before we all go off to our separate online corners. At this kickoff workshop, there are elements of it that are recognizable to anybody who comes to faculty development workshops, but I think there were some novel components too. I mean, one of the things that we do is, it’s simple, but it’s a really effective kickoff exercise. So, we would have either a sticky wall where people can put ideas, or we’ve sometimes bedded rooms that have whiteboard walls, and we have pens, and I say, “Okay, what are some things you’ve heard about why students don’t persist? Just give me reasons. You don’t have to endorse them. They could also be the things that we do here in the faculty meetings.” So, that’s a nice kind of permission giving, kind of opening, I think, to let people say things that they know that are maybe not backed up or they’re not sure, or they don’t agree with them, but they think they’re important to put and they’re also encouraged to put those ideas in groupings. So, there’ll always be some around economic factors, or psychological and social emotional factors, family support. So, we all write on the walls and have these things in front of us for the rest of the day, statements about what barriers there are… to keep that in the front of our minds of what really is affecting our students. As we engage with this work, there is a presentation component, but I really center that around five key claims. So, I think too, it’s important to come with this not just like, “Well, here’s some tips that you can have and some things that some people believe.” I say, “I am not neutral on this. And here’s my five beliefs and these underlie everything that we’re going to do. And you can agree or disagree with these, but I can back them all up, that this is what drives us” and you know, as faculty I think that’s appealing. We want to know what are those assumptions and just to list them off real quick, there are academic persistence matters, so this is important. There are disparities that both reflect and perpetuate inequalities, ethnic class, economic, that we do know a lot about how persistence works. So, that knowledge base does exist. That there are effective strategies for addressing those disparities, although they’re not easy or cheap, I’m not there to sell faculty on magic bullets or “Hey, if you just tweak with this one thing, everything will be fine,” because we all know that’s not the case. And then lastly that faculty do have the ability to positively affect persistence through their teaching, but also through those interactions that they have informally advocating for certain kinds of policies with the institution. So, I really present that. And then lastly, we have a hands-on data exercise. Now one of the things that I think can be a barrier for faculty as they want to get involved with this is we think, or we really don’t have, access to the information that is specific to our campus. So we also have an exercise where I bring in librarians, this is really great. They’ve supported me a great deal in this and we get people on laptops and say, “Alright, here are some sites to explore, national sites about student persistence, databases, article databases you can look at. Use these to uncover solutions, facts about student persistence right here, right now, just do this right now.” And we also get them access doing some basic working knowledge of our institutional dashboard for looking at things like pass rates or grade breakdowns which you can do, you can do it by course, you can do it by semester, you can do even more fine grained by student characteristics. This is all out there, but the vast majority of faculty just do not either know that or they don’t have that working knowledge. So, what I envision is okay, a faculty member can, if it comes up in their department, “Oh, hey, what can we do about this course that’s maybe a bottleneck or we think we’re ready to redesign this one over here?” They can pull the data for themselves and say, “Well, here’s how things changed when we brought in, say, a courseware system, or here are the students who are having the most difficulty, or if a student passes this course, here’s their likelihood of succeeding in this one down the line.” Faculty love that. And once that power is in their hands, I think that they really can carry that out. That’s all the stuff we do, and the kickoff that we have right there and how we establish that grounding for them.

John: It’s great that you have that data. Many institutions are very protective of data, even though it could be really useful in helping us learn about what works.

Michelle: Then to turn around and say, “Well, faculty are kind of in the way here, faculty are this or that?” Well, yeah, we do have to look at what have we empowered faculty to be able to do reasonably and in ways that are appropriate to their own discipline?

John: What are some of the myths that people come into this with in terms of what leads to students dropping out, or failing, or withdrawing?

Michelle: I don’t know if I’m ready to quite call it a myth, but there is perhaps a sort of counterproductive concept, which is the old “If we would just admit better students” who are, and I’m going to use a terrible phrase, “college material.” I mean, that phrase is awful on many different levels as we look at our students, who are these complex human beings, who’ve come to us willing to step up and try to do these incredibly challenging things to accomplish goals that benefit them and benefit our whole society. There is that. And I think an associated belief is, all of this should just be addressed in K-12. And aside from the practical issues there, especially if you teach at a public institution, which we are, I don’t think that’s right to just say “This has to be sort of repaired as a problem by the time it gets to me, or I can’t… or shouldn’t… do anything.” So that whole complex of beliefs about something didn’t happen before this student graduated from high school therefore kind of what’s the point and if the school wants to retain more students, we need to admit the more academically skilled students from the beginning. I say I’m not ready to call that a myth, that is because, yes, absolutely, things like the accomplishments and achievements, academic experiences you’ve had before you come to college. Yeah, those are all great predictors of retention. It’s not that that doesn’t matter at all. But a great deal of other things do matter. And I think that those are maybe where we want to redirect students. And I think as well among faculty who still have themselves a form of fixed mindset, that is really problematic too. And, you know, this really hit home for me. There’s a recent article by Elizabeth Canning and her colleagues at Indiana University Bloomington, titled “STEM Faculty who Believe Ability is Fixed Have Larger Racial Achievement Gaps.” Wow, just think about that for a second. They were looking at the beliefs that are in the heads of the faculty, not even their teaching or what they selected, or what they said to students overtly, but the beliefs they have about who achieves and why, and whether that capacity, that potential is fixed. That plays out in accentuating the exact types of gaps and disparities that we are here to shrink and get rid of. That is surprising and disturbing. And they also find there’s less student motivation overall in those courses. So, I think that that’s maybe another constellation of very counterproductive problematic beliefs that, “Oh, the students aren’t motivated. They don’t want to do the work, but maybe they can’t do the work, maybe they aren’t cut out for this.” If that’s in my head, that is going to leak out and infuse the teaching that I do. And then we have more of these gaps at the end of the day. Those are some of the beliefs that I think are more of an issue. I think less frequently, we’ll see some version of “Well, we’re maybe trying to come from a place of compassion and look at things like oh, family issues, caregiving responsibilities, jobs that students have to hold down in order to be able to support themselves and their families as they go through their education.” It’s great to acknowledge that, but then I think that sometimes faculty can then have this very kind of dead end view of it and say, “Wow, I don’t know if there’s any way this could work.” And yeah, there are only so many hours in the day and we can’t just say, “Oh, education can happen on the margin, no big deal.” But I think too, what we need to step back and look at those beliefs and say, “Well, what are some institutional policies? or “What are even some things written into my syllabus that accentuate that barriers, or put barriers up for students who have those responsibilities? Do they all have to be there? What can I take away that doesn’t get in the way of what students are accomplishing or what’s expected of them, but simply make some of these much more possible?” So that’s kind of a set of those ideas too.

John: One other point there is that students who are most at risk often end up leaving with a large amount of debt and have the most struggle trying to pay for it, putting them at further disadvantage. So, the more we can help these students to be successful, the better off they’ll be.

Michelle: Right. And so many faculty, I mean the faculty who I’ve talked to, and I’ve talked to many at this point, I really believe that they care. They do care about that issue… that deeply disturbs them and deeply bothers them, the idea of somebody leaving with tremendous amount of debt that’s going to limit their lives, and what if they leave with that debt and without the degree that they came for? That’s a tragedy. And I think that we can take that intention and that reaction and channel that into positive action.

John: What types of incentives were there for faculty to participate in this program?

Michelle: For those of us who do work in this space of faculty professional development, we know that that’s an issue. There’s so many demands on faculty time, and so it’s important to have that. So, quite simply, we have a small honorarium. And in fact, it’s small enough to where at first I thought, “Well, do we really need this?” but the feedback I got from my staff and also from faculty was that yes, this is important, if only as a gesture, that we realize this takes your time. So that was $150, and they came in the form of professional development funds. So it’s just enough to plug in, maybe get some books or help make up a gap in some funding for a conference. Those are some of the typical things that faculty use that for. So we have that, and as a kind of a less tangible, but still very important incentive was, as I said, this is part of the First-Year Learning Initiative. And so courses that want to maintain their presence in that program and kind of stay in good standing have to demonstrate this ongoing engagement. So, especially after the first semester or two, we started to say, “Yeah, this is a powerful program, and we really want to make this First-Year Learning Initiative participation contingent on doing this.” So many of the faculty who are there, they do come in because it’s really required for their participation in this bigger program. But, then we have some who come because they’re simply interested and they’ve heard good things about the program as well. So there’s a spectrum of those incentives, both tangible and intangible.

Rebecca: What are some of the students that are the most at risk for persistence? What students are we really helping by engaging in this literature and these methodologies?

Michelle: I think that for people who have some familiarity with the area of student persistence, no surprises here. It’s students who are first generation, as a large proportion of our students at Northern Arizona are. So there is that, there is kind of a constellation of socio-economic factors which can play out in everything from just simply the financial resources one has to attend college, all the way down to the quality of the schools, and the preparation, the pre-college preparation that you were able to get as part of the education you were provided in K-12. So, there is that. Students of color, students of color definitely are going to have a number of barriers and challenges that are going to play out in terms of persistence. And then there are, within particular disciplines, as many of us are familiar with… in particular disciplines where the gender representation or representation of women is relatively low, there can be some persistence issues there as well. So, in the more traditionally male STEM fields, engineering, mathematics and so on, but really, largely these issues of class, of race, and economic opportunity are what all are coming to a crux when students are in these crucial early semesters of college participation. That’s what we’re seeing.

John: It’s fairly early. You’ve only been doing this program for two years, but do you have any evidence of its success in terms of impacts on students?

Michelle: This is a very faculty- and staff-oriented initiative. And there are so many different factors that impact retention and that all go on at once. And by the way, that’s something that I’ve definitely learned as… when I got into this as well… is that there are just this enormous number of options, and even outside of the classroom. Then you have things like learning communities, residential communities, bridge programs, mentorship opportunities, all of these things are kind of getting into the mix sat once, which is probably not a bad thing to have all of these, but it does make it difficult to tease that apart when you look at something like overall retention rates or persistence rates for an institution. However, we have gathered some really systematic assessment data through our participants specifically. So what we did over the past few semesters is we brought in a kind of a pre-assessment so we could capture some very key things about participants’ knowledge and commitment to and ability to advocate for student persistence at the beginning… at the outset of this… before we did anything, and then at the end, after they’d done this about six to eight week program, and so there we do see some pretty dramatic changes and some really dramatic improvements. So, one in particular that stands out is that we asked participants how capable they feel to discuss and apply concepts from the research literature on persistence. And that is very, very low at the beginning. It’s about two and a half on a scale of one to five. And that went up to a little bit over an average of four on that same scale of five after the program. So, that’s something where faculty said, “Yeah, I feel like I can come into this as an informed advocate.” Knowledge about student persistence, that’s another area where the self-rated capability goes way, way up. And also, another thing we asked them is how capable they feel to identify and dispel some of the major misconceptions about attrition and persistence. So there too, the numbers are very, very similar. So we get positive comments, but I also feel like those quantitative ratings have really targeted what I wanted to change as a function of this program.

Rebecca: We talked a little bit about institutional concerns about retention and persistence. Why should faculty be engaged in this piece? We often think, “Well, that’s not our responsibility.” But, why should it be a faculty responsibility, in part?

Michelle: So here’s the thing. I think that this really fits with my experience over about 10 years of working on this at the institution. I think that so many of the initiatives that institutions spend all this money and their political and social capital on setting up, those live or die in faculty meetings. And I think that there’s very limited realization of that on the part of leadership. And it’s understandable because that’s one place where they don’t get to go. But I’ve sat in many, many, many such a meeting over my career. And here’s the thing, in my experience, it can just take one person who thinks that this initiative is misguided, or they think we ought to just admit better students that that should be fine, or they only care about retention for financial reasons. It only takes one highly vocal person to shut that down in that department and there may be other people who are sitting there who are interested in this… they’re saying, “You know what, I care. I think that social inequality is perpetuated when students don’t persist. I see real disparities, and I’m not comfortable with that. And I think this is a social justice issue.” Well, especially if that person is more junior or is not tenured, and the person who’s highly vocal is senior and is tenured, that initiative is not going anywhere. And I don’t care how much money you put into it, or what kind of big stipend is attached to it, it’s not happening. So that’s where I really had this vision as a designer of this program that I wanted people to be able to kind of raise their hand and say, “Well, actually, there’s some research that shows this”, or “I learned about this one concept,” or “Have you thought about how inequality is perpetuated, and maybe we should care for those reasons.” So, to equip and emboldened people to do that… Now that’s always up to them. They can take persistent scholars and come away with whatever conclusions that they want. I honestly come at it that way, that it is up to them to draw their own conclusions, but I do feel, especially given those things they tell us on our assessments, that we’ve done the best we can to equip them to go in to be those advocates. And it isn’t just teaching too. Don’t forget faculty, even though we can’t always affect things like financial aid or how drop/add policies are handled or any of that, we do have faculty senates, and sometimes we can weigh in on those issues. So, if we can bring pressure to bear in a positive way on our administrations, we usually think about it as “Oh, the administration is kind of leaning on us to support student success,” well that runs the other direction, too. And it can. And how does that happen? When we have the information because, again, faculty, we run on evidence… that’s baked into our culture, and that is who we are. So if you are the person at the meeting, you can say, “Well, I read this entire book by Vincent Tinto, who’s the most respected researcher in this area, I’ve actually read that book. And here’s what I took away from it. And so here’s why we should maybe give this initiative a second thought.” That’s what I think can be very, very powerful for creating change.

John: Faculty are well intentioned, but they don’t always know what they can do to be effective, and it’s really easy to blame the students when students aren’t successful. And we see that in lots of departments and lots of people. Providing them with information, I think, could start to make a big difference.

Rebecca: I think a lot of faculty are overwhelmed. They might be interested in these topics, but don’t have time to dig around and find the research and sort through it. So, having a curated opportunity like this is a good way to engage deeply with some key materials and come out of it with that perspective, which I think is really valuable. And we see that in other areas too, where we want to learn more about memory, or we want to learn more about learning strategies or whatever. If we can curate those things, then it’s often easier for faculty to engage and think about how they can individually commit to those ideas because they don’t have to sort through all of the information. It’s collated for them.

Michelle: And that’s just such a perfectly articulated way of describing what our design philosophy really was. And yeah, to say you can make a website or a giant compendium of “here’s a lot of suggested resources,” but it’s a different challenge to say, “Okay, you can assign three things. You can select three things for us to read over this three-week period. That’s it, what are those three things going to be?” And I did, it really did force me to really focus on quality and what was powerful. Yeah, that belongingness mindset lay theory piece was one, transparency was another that I selected. And really the last iteration to it, I also selected an excerpt from Lisa Nunn’s book, 33 Simple Strategies for Faculty. It is oriented towards first generation, but it really crossed over into so many practical applications of the research we were reading about. So that was a huge hit with the last cohort of participants as well. So being selective, having one targeted experience that you can simply share in a very informal way, rather than sitting down to write the giant literature review, I think that’s the sort of thing that we do need. And we did design it with that blended approach with that idea of maximum flexibility. Every week was its own modular piece where we did the same thing, so there wasn’t a whole lot of thrashing around about “what are the expectations” and so on. Even things like designing it so that it starts up about three weeks or four weeks into the semester and wraps up, like in the fall, we wrap up before Thanksgiving. That’s a big, big deal to faculty. If you coordinate it with the students’ semester, that’s just going to be too much. And you’re going to hit people with way too many demands right at their busiest time. So, that was also really appreciated as a factor that promoted faculty participation.

John: In an email exchange prior to this conversation, you mentioned something about the AR program at NAU that you’ve been working with and some results that were relevant to this discussion. Could you tell us a little bit about that?

Michelle: It’s so funny, this faculty professional development program ended up intersecting with a completely separate piece of my research agenda right now. I’ve been working for the last about two years with our amazing cross disciplinary group here, the Immersive Virtual Reality Laboratory at NAU with Professor Norman Medoff and Professor Giovanni Castillo. They had designed, already, this program for organic chemistry so students get to actually manipulate with molecules and they worked with a chemistry professor to make actual meaningful assignments that would use that program in this really cool way. We even set up kind of a almost experimental study where we did the classic flip a coin and one section has access to the VR and the other section does not… it has a substitute assignment instead. Of course I’m interested in looking at the impact on student success. Well, I got into the data, and I looked at the overall impact and there is, there’s a reasonably consistent trend towards better grades in O-Chem. And also better final exam scores if you have access to this particular technology and way of interacting with the material. But then I started doing the subsidiary analyses and I was really surprised. We broke it out by first-generation status and first-generation college students, which was about half of our participants, in this case, experience improvements, positive impacts of this intervention that were larger and more consistently they were significant. All the measurements that we looked at were consistent in terms of the advantage that they got. And we’re working on writing this up for publication right now, and we did present them at a conference over last summer. And it’s really stretching my mind as well to try to say, “Well, why is that? What does that maybe communicate to students when we offer them this? How might it actually maybe shore up the experiences of students who have not had access to as good of a chemistry education, most likely, before they got to our university, compared to students who come from continuing generation families?” I was so surprised. And now there’s something that once again is telling me persistence has a lot to do with these other factors. Can we control them? Can we address them? Of course we can’t, as faculty, but we can look to discover ways that extend what we’re doing in the classroom or take particular approaches, and like so many of the interventions that we do in course design, this is one that doesn’t bring anybody down. I mean, if I’m from an advantaged background, I’m from a majority group, I’ve had this great background when I come in, I can benefit too, that’s fine, but somebody else is going to experience disproportionate benefits. And it’s maybe in a way, replicating a pattern that we’ve seen time and again with other ways of approaching these challenging foundational level courses.

Rebecca: That sounds really exciting.

Michelle: Thank you.

John: I’m looking forward to reading that.

We always end by asking, “What are you doing next?”

Michelle: Well, I have handed off the Persistence Scholars program. So while I’m still very proud of the work and feel very engaged with it, I have stepped away from the First-Year Learning Initiative, and as part of that the Persistence Scholars program is going to be led by a colleague of mine, Cody Canning at NAU, and I’ve handed off that program before as part of sabbatical and so on. So it is neat to build a program from the beginning that can be taken on and have it structured in depth enough to where you could take it on and then bring your own expertise and particular perspective to it. I’m still very engaged nationally though with spreading out these ideas about student persistence, learning and success in the first year, and looking at how we can take those and develop those in other places and really spread those efforts out, since I know so many of us nationally are just really fired up about this. So that’s where that stands right now. I’m working on a book right now with West Virginia University Press, with a very dynamic editor and a group of writers who are all working right now on writing about different issues in pedagogy in higher education. So that’s an honor, and I’m having a lot of fun with that book. So, memory and technology is what I’m writing about, and that’s something that springboards off a lot of the teaching that I do and some other writing as well. And that is something that I think is an issue that we see recurring now as being a very timely issue for people who are teaching. So that is taking a lot of my intellectual effort right now, and I’m looking at ways to keep engaging people in Minds Online, which, although it does have that specific technology angle, I think does pick up on many of these issues of promoting student success, and reducing disparities, and finding sometimes very surprising things that happen when we start to teach in new ways. So, that book came out around five years ago, it’s hard to believe, but I’m also looking at all the ideas and research that’s come out since then, and new applications that faculty have come up with. So, I’m looking at some new ways to keep that percolating along and kind of harness some of that energy we all have around that topic. So, I would say with that, just stay tuned or contact me to learn more, and we’ll see how that develops over the next year or so.

John: And when is this new book coming out?

Michelle: Oh…

John: Tentatively?

Michelle: It’s coming out after I write it. Let’s just say 2021. So it is well, well underway. We’re in striking distance of having that out in 2021.

John: And that’ll be part of the West Virginia University Press series edited by James Lang.

Rebecca: Well, thank you so much for joining us. It’s always a pleasure.

Michelle: Likewise, it’s always great to talk about these issues with both of you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

120. Scaling Accessibility

Adopting a culture of accessibility at an institution can seem both daunting and full of barriers, but movement forward can happen with the right strategies in place. In this episode, Dr. Sherri Restauri joins us to discuss how institutions can progress from providing accommodations for individual students to an institutional commitment to building accessibility into the course design process.

Sherri is the Director of Coastal’s Office of Online Learning and also serves as a teaching associate at the Department of Psychology at Coastal Carolina University. Sherri has served for a number of years on the steering committee for the OLC Innovate and Accelerate Annual Conferences, including serving at the 2020 OLC Innovate Conference in Chicago in the role of Co-Chair for Equity and Inclusion.

Show Notes

Transcript

John: Adopting a culture of accessibility at an institution can seem both daunting and full of barriers, but movement forward can happen with the right strategies in place. In this episode, we discuss how institutions can progress from providing accommodations for individual students to an institutional commitment to building accessibility into the course design process.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Sherri Restauri. Sherri is the Director of Coastal’s Office of Online Learning and also serves as a teaching associate at the Department of Psychology at Coastal Carolina University. Sherri has served for a number of years on the steering committee for the OLC Innovate and Accelerate Annual Conferences, including serving at the upcoming Innovate Conference in Chicago in the role of Co-Chair for Equity and Inclusion. Welcome, Sherri.

John: Welcome.

Sherri: Thank you. Thank you for having me.

John: Our teas today are:

Sherri: So, I actually am drinking my favorite iced coffee. I have a salted caramel mocha with me.

Rebecca: That sounds good. Sounds a little chilly for us to have that around here today… it’s a little cold, but… I have a vanilla coconut tea.

John: That one’s new.

Rebecca: Yeah.

John: I have a Forest Fruits tea from Epcot… from the Twinings pavilion there, which I picked up at the OLC conference.

Rebecca: How appropriate. [LAUGHTER].

So, digital accessibility is becoming increasingly important, valued, and a topic of focus at many institutions, but really requires a cultural shift to fully implement. You’ve been really successful at directing these efforts at Coastal. Can you tell us a little bit about your efforts and strategy for becoming a more accessible campus?

Sherri: I would be happy to share. One of the things that I feel like we have really been most successful at here at Coastal was trying to change the language that we use, specifically around the concept of accessibility. And my work specifically began when I got here about three and a half years ago, and originally, there wasn’t any kind of workshop or training on our campus specific to accessibility or digital access. And I built the first workshop, which was actually called Digital Accessibility in Online and Hybrid Learning, around the concept of increasing student access to learning. And so the premise of the workshop, in the first five minutes actually, introduces the faculty, staff, and students who attend that workshop to the concept of increasing access to education and educational material, rather than to the idea of accessibility. And so they’re taught the scope of access is something that’s much, much larger than maybe the way that we were taught when I started in the field of education teaching 21 years ago. It’s not about accommodating one student out of 400 in an entire academic year. Instead, it’s about designing from the ground up in a way to make content accessible for all users. And so it is truly taught under the umbrella of a universal design for learning and teaching approach. And honestly, this is many times the first time that faculty have heard that approach. In the past, a lot of times they were instead under the assumption that we were making accommodations to address one specific student. And so instead, we’re actually designing so that all content is accessible and is readily available to all students, even prior to the first day of class, if possible. So, I think our work in the culture shift here on campus started with modifying the language and the vocabulary. So, we don’t actually say accessibility as much as we do instead use the term digital access, because for us, that includes things such as affordable learning, and OER, all of that actually falls into the scope of access on my campus.

Rebecca: How do you recommend allies and advocates nudge or pull or push others to join them in accessibility efforts to actually make for true cultural shift in the organization and institutions that they’re a part of? Sometimes it’s really easy if you’re the person that’s like really excited about it, to make some efforts, but it’s hard to figure out how to get others engaged and really feel as committed as you are.

Sherri: I totally agree with that, and I think that I’ve seen it be successful, as well as unsuccessful in my own campuses at different institutions. And one of the things that I found that was most instrumental, looking at the three campuses now that I’ve served on, the two that it was most successful on, were ones in which I could launch a digital accessibility initiative, not as a champion myself, but with a committee backing me. And so on the two institutions where I found the most success, it actually specifically was ones in which there was already a designated body that I could bring on board who had perhaps a faculty representative from each academic college. So, when I began initially launching the digital accessibility initiative on the Coastal campus, I immediately presented that my Digital Learning Committee, which is actually what used to be called the Distance Learning Committee, and presented to them initially the idea that “This is coming, I need you to come on board as my college representative advocate, and I need you to be a champion as well within your academic unit.” In doing that I brought allies on board. It’s advantageous that the main technology tool that we were to later adopt is actually called Ally. But, one of the tools that we were to later adopt, it was actually in conjunction with my Digital Learning Committee, where they were able to be an advocate through our budgeting, our purchasing, and instructional needs in order to provide support for me. If I had tried to do this type of an initiative without having a standing committee to support me that I don’t think it would have been quite as successful. So being able to form allies with maybe committees that already are in existence, and maybe even also to strategically reach out to certain committees and/or departments on campus that look like they may be a benefit to use. So we made an outreach directly to accessibility and disability services, to diversity and inclusion, to the university library, to our provost’s office, and then also to freshman orientation, in order to find a way to make sure that access to all students was truly viewed as something much broader than just accessibility.

John: How did you get faculty to buy into the program? And what sort of training did you provide for faculty?

Sherri: I was honestly extremely excited when I took this position to know that there was already what we actually like to call COOL grant. COOL grants are a faculty monetary incentive grant that has required as well as optional elective training opportunities built in with the ultimate outcome that’s twofold. The twofold piece of that is faculty will learn information that will help them to build and administer better, stronger, high quality digital learning courses. But, the ultimate outcome is also specifically that the courses that they are actually building for digital learning purposes will be submitted to my unit for a quality review with evaluative feedback on how to improve those. And so from that particular perspective, there was already an existent grant program that existed. When I came on board, I decided to modify that grant program and incorporate accessibility into it. When I got here, it was focused on quality, but accessibility had not yet been incorporated. And so when I got here in 2016, that was one of our first components that we actually added into it. And we were able to incorporate that as the ninth of our 10 core components. And within our second semester of arrival here, we began offering the digital accessibility as a required component of that training. We do offer additional, I would call them optional electives, for faculty as well, so they just can’t get enough about digital accessibility. Then part of the grant is that they can take up to five elective trainings to learn more and they can take as much about digital accessibility as they want. I think one of the things that really became advantageous to our faculty is, the more they learn… even if they submitted with the intention of one specific class, let’s say, Psychology 425 was a grant program… they learned so much that they initially might have intended to only apply it to that first course, but then they turned around and applied it to every course from that point forward. So, that’s where that cultural shift started was once they learned how advantageous these principles were to every single one of students in every class, they started modifying the way that they taught. And they started modifying the way that they uploaded materials and that they built the course items into their course as well. It was amazing to actually watch that type of a shift for our faculty. It’s like watching light bulbs come on, because they suddenly understand there’s technologies that already exist here that can make it so that my students who have an hour-long commute can learn. It’s not just about a student who may or may not have a disability, it’s about a student who has a long commute, or a student who doesn’t own the software, or a student who can’t afford my textbook. So, their accommodations were for all kinds of students that met all kinds of needs that I don’t think that they had anticipated, but they learned very quickly it could help the students in many, many different ways.

Rebecca: I think it’s one thing to learn about some of the technology related or the strategies you can use to make things more accessible to increase access, but it’s different from making that part of your regular workflow. Do you have any tips for faculty about how to incorporate these practices in the way that they generally approach their classes and prepping for classes and preparing their classes each semester?

Sherri: I do, and one of the things that I think is important to understand is every single university, campus, or school environment is configured and structured in such a very different way. Some of us have centralized units that support faculty and some campuses do not. I’ve served on both types of campuses. And so it’s important to recognize that faculty may be coming at this from a very different perspective. Some have more support than others. We have a very small unit. We have two instructional designers for a campus of over 10,000 students. And so with that in mind, to make this successful, we actually invested as a unit all of our time in building very well structured templates, both course templates as well as material templates… and by material templates I mean accessible syllabi, accessible course modules, everything that we have is already compliant with WCAG standards, so that all they have to do is make modifications and adapt it. In addition to the template materials and the template courses that we generated, I think one of the things that we also found most exciting for faculty is we started looking at ways to expedite the administrative struggles that they were running into. And a good example is, when a faculty member submits a course for a grant review, we end up looking at it for every single component. And so one of the biggest red flags that we would see is our faculty were spending hours and hours and hours creating wonderful lectures, but none of these had closed captions. They couldn’t pass a grant without having accessible courses. And so one of the very first things that we were able to do was implement a component by which faculty simply alert us of all multimedia, and my unit, in my department of online learning, actually complete the closed captions for all of them for everybody on the entire campus. So, we’ve taken the workload off of them. We actually, through a training program, have trained our graduate assistants and student workers to do closed captioning to standards. We utilize a third party program called Echo 360 for our lecture capture. And starting in the fall of 2019, we enabled what is called ASR for automatic speech recognition, so that there is a component in place to begin the captioning to standard, and then our student workers and graduate assistants pop in and make the modifications. The faculty actually have no administrative responsibilities for that, unless they choose to caption themselves. So, one of the pieces that’s heaviest lifting for compliance, specifically, was just simply the time to make those modifications. And so I immediately started looking at what are the heaviest lifting pieces and what can we do as a unit to lift that for the faculty so that they don’t have to make those accommodations. One of the things I think that we got such positive feedback as well about during the 2018-2019 academic year was when our campus learning management system integrated the Ally accessibility tool. I was looking for ways for faculty to best understand how to use that tool to convert their files. And the tool itself is fairly simplistic, but it can be implemented in different ways. So, the technique that I actually created, and I’ll be happy to share with your listeners, is we actually created what we call an Ally drop spot. And in that particular drop spot, faculty simply mass or batch upload all files at one time, check all files at one time, and then move content around once it’s corrected. That’s not something that’s actually taught by the vendor. It was just a creative idea that I came up with because I too am a faculty member, and was trying to figure out a way to save time. And so that seems to be something to celebrate. And so we use the Ally drop spot, not just for academic classes, because the culture has changed so much on our campus that administrative documents, things like announcements about upcoming movies on campus that are going out via email, everything that is now distributed digitally must be WCAG compliant, which makes my heart very happy. And so from that perspective, having an Ally drop spot, having a centralized technique in which faculty can batch upload or staff can batch upload content to be checked and corrected immediately, again, with administrative heavy lifting that we built into simplify the process for them.

John: So, the focus has been on new development and redesign. Have faculty started to go back and redesign or re-tool some of their older materials as well?

Sherri: Yes, is the short answer to that. I think that one of the things that, and I’m going to point at Ally but it wasn’t necessarily because of Ally, that having that particular type of tool on our campus helped us realize is: Ally, for example, scans all documents that may exist in your learning management system and inadvertently one of the after effects of that is that happens to alert you to the age of your original file creation. So for example, if I’m teaching a spring 2020 psychology class, and Ally scans my file, it may alert me that that file was originally developed
in Microsoft Word 1997. And so it will alert faculty, through happenstance, that it might be time to update the version that they have been using to create their files. And so I think one of the things that’s been really interesting to watch is even though the intention was “let’s make content more accessible,” the answer, John, is that actually the tools that we’ve given them, whether it be Echo 360, or whether it be Ally, they started using these to improve all courses all the time. I had a conversation just this morning after our “welcome back to our campus” presentation with a faculty member. And we were discussing how when we started teaching more than two decades ago, we used to be taught that the best way to make files accessible for everybody, because everybody doesn’t have the same software, is just save it as a PDF and everybody can access it… and actually, that backwards now…

Rebecca:: Yup. [LAUGHTER]

Sherri: …and we were talking about how important it is to understand that the times have changed. And if you started teaching like I did two decades ago, the things we were originally taught have changed so drastically, and I think faculty coming to a 90-minute digital accessibility workshop, that sounds so short. But, that one 90-minute workshop gives them enough information to understand times have changed. “Wow, here’s these 18 tools that are available. And here’s these five processes that exist. And I thought that I couldn’t do these things because I don’t have an extra 100 hours available this semester to add captions, but I have services that are provided by my institution to allow for that.” Our campus is not one that has a significant technology budget. And so most of the items that I’ve mentioned today in your episode are things that we developed internally. We don’t send off for closed captioning for third parties, we simply don’t have the funds for that. Most of the types of things that we’ve done have all been in-built. The quality metrics that we use for evaluating our courses to make sure that they’re compliant and that they meet standards are all in-built. We are not using a third-party for quality evaluation, we instead built our own. And so many of these are actually using the resources you already have. They’re using the personnel you already have. Again, we retrained and retooled our graduate assistants and student workers to make sure that they can assist that with doing a ton of heavy lifting. And I feel like one of the things that I think, if I could communicate to other campuses who are looking to implement something like a digital accessibility initiative, is this isn’t really about having to have an extra $300,000 per year or having to have an extra 10 staff. We don’t. [LAUGHTER] We didn’t actually purchase really anything additionally, minus one extra product. Past that it was just finding ways to make this fit into and enhance our current processes so that everybody was compliant, and really bringing the whole campus forward alongside of us.

Rebecca: Can you talk a little bit about your student staff team and their role? You mentioned it a little bit, but can you talk a little bit more about the scope, the kind of training that they received, and what tasks they actually help your team with?

Sherri: So, as a student worker myself 20 plus years ago, one of the things that I remember feeling is that I never really felt like part of the team. I was given small tasks and I never really understood the mission, or the vision, or how me putting computer parts together in a computer lab ultimately helped the mission of the university. And so, for my office in digital learning, when we bring our staff on board part-time or full-time I tell them during the orientation it’s the last time they’re going to hear me call them a student. After that, they’re simply referred to as staff or part-time staff. With that in mind, they are asked to, expected to, attend all formal meetings, all informal meetings, all staff meetings about visioning and planning and they go through all levels of training, including they are required to go through the same level of digital accessibility training as our faculty. They spend about the first two to two and a half weeks learning what digital accessibility is and why that’s important… the mission of not just my department, but also to the campus. After that, we have a standardized set of trainings they go through focused on quality enhancement, and also on best practices for digital learning. And then they go through a set of training relating specifically to closed captioning and what we typically call WCAG compliance for the web accessibility. So, they’ll go through those processes. They’re also trained on very specific technologies specific to my campus, like Echo 360, Moodle, and all the other tools that we use here internally. By the time they end up graduating and leaving us, they are truly experts. So they learn a tremendous amount about the technologies, but what we actually ask them to do because we only get to have students for no more than 20 hours a week. I don’t know if any of you have ever had to sit down and try to do closed-captioning or closed-captioning editing…

John: Every day.

Sherri: …but nobody can do that for 20 hours a week. So, I would never ask anybody to do it for more than about five hours a week. So, out of their 20 hours, only about five of that is actually closed-captioning. They will do the edits. And the edits are actually super easy to do through the vendor that we actually have as well. They allot about five of their 20 hours to closed captioning. The other 15 are spent in doing other types of work such as conversions within the Ally system, helping our faculty to make modifications to improve the digital accessibility of their courses. Two of our student assistants, the part-time workers, also directly assist us with the open inclusive education process so that they help our faculty to locate OERs that can be implemented into their courses. We only launched our OER initiative about two and a half years ago, and we already have over 60 faculty. So, we’re very proud of that particular access work. And so we’ve been able to make a tremendous amount of headway super fast, partially because of the assistance of our part-time staff. We’ve been doing a lot of the legwork to locate really high quality, open and inclusive materials for our faculty.

One of the things that we found with our particular students is once they have learned about all the different principles that they’re being taught to use to assist faculty, they’ll turn around and start using those to improve their own learning in their own courses. So that’s actually something that again, wasn’t planned. But students start learning study strategies based on alternative formats. So, a good example is they learn about the Ally tool which provides alternative formats to a PDF, which may have MP3s, and all of a sudden, our student workers may start using that as a study technique, which changes the entire course of how they progress in their own programs, too. We had one particular student who served as a part-time staff member specifically focused on digital accessibility. And she enjoyed it so much, she built an entire website for pre-service elementary education teachers who would benefit from learning more about digital accessibility. So, I feel like bringing students into this couldn’t have been a better choice. It’s actually directly impacting not just our student workers in the way that they actually study, but it’s impacting their careers as well because we recruit students and graduate assistants from every academic college.

John: That’s great training for those who might be doing some of this work in the future. And everyone’s going to be doing more of this as we move forward, so I think it’s a great program.

Rebecca: How many students are on your team?

Sherri: We just had two graduate, which is, of course, what they’re here for. But it’s always so sad that we have to have some of them leave. So currently, we have two student workers and two graduate assistants so that’s four, they’re 80 hours a week of part-time staff.

Rebecca: Great, that’s exciting. Can you talk a little bit more about the faculty grant recipients and their role in onboarding some other faculty to get them excited about accessibility as well?

Sherri: Absolutely. So, you’ve heard me say the term COOL a couple of times, like our COOL grants, that COOL is the acronym for my unit the Coastal Office of Online Learning. These grants which are called COOL grants, there’s actually two different formats of those. One is a year-long, and the year-long course development grant is focused on building brand new courses that have never been placed into an online or a hybrid program. And we give them an entire year to do that. A lot of times what I found is as they’re going through a series of trainings, the thing that they benefit most from, in addition to the 10 trainings that they take along the way in a year-long grant program, is physically being in the same space with other faculty here also going through that grant program. So, we don’t necessarily tell them that they’re going to be put together into a peer group, but they are. We call it a cohort. And so for example, cohort eight ran throughout the 2019 year, they’ve just submitted their grant for review on January 6th. We have about 40 who participated in that program, and they have truly formed a cohort. What will happen is once their courses are reviewed, evaluated, modified and completed, those courses become what are called COOL certified and so those certified individuals from cohort eight all stay on an individual listeserv. And then they also get grouped in with previous cohorts. We have recently formed what we call a Digital and Open Learning Faculty Learning Community. And that particular community pulls from these faculty who have previously completed COOL certified courses. And we’re actually taking this full circle so that not only “Are they learning the material, are they building a course? Are they offering the course?” But the faculty learning community is pulling all of those individuals together, so that we can do research and write manuscripts together. So, that’s our final step: “Wow, we’ve learned so much together. What if we collaborate as a group who successfully completed a certification and a grant and actually pull this together into a journal, or a web article, or a blog, or whatever it might be?” So we’ve already been blogging, our very next step is this faculty learning community is going to start doing some of the outreach regarding manuscripts and presentations as well.

Rebecca: You mentioned universal design for learning a little bit earlier as your approach to accessibility rather than using the accessibility term. Can you talk a little bit about pedagogy and how accessibility and universal design for learning overlap but also where they diverge? And what role you see these initiatives playing in our approach to student learning?

Sherri: That may be my favorite question yet, actually. [LAUGHTER] So, interestingly enough… universal design. I like to ask that question when I begin teaching my digital accessibility workshop, that’s actually one of the first things that I say, Rebecca, is I ask them “How many of you have ever heard this term?” And it shocked me. I am a former instructional designer, so I honestly thought, “How would a history professor know this? How would a math professor know this?” Most of them know. And I don’t know how that has occurred, but most of them have heard it. But, in practice, they’ve never been taught it. And so once I realized about three workshops in that they had conceptual knowledge of what universal design was, I started actually flipping the script a bit on that and saying, “Explain to me, what kinds of workshops fit you’re learning needs better. Let’s start talking about how, if instead of me teaching this workshop face to face, what format would be better for you and explain to me why.” And once I began actually putting into practice for them, from their own learning needs, because my COOL grant recipients are required to take 10 trainings, and with children and with work duties, it’s very difficult for them to fit that in. So I like to give them the opportunity to explain to me what they perceive universal design to be, as it fits their learning needs. And once they grasp that and they personally apply it, then we actually start working on “Let’s talk about what universal design actually means when we build content. Let’s talk about what universal design means when you’re teaching a hybrid class that only meets four times a semester.” And one of those meetings is for an international trip with the class. So, let’s discuss all of these different pieces and how interaction and how accessibility all intersect. I tend to find that, because I do distinguish for them the difference between Universal Design for Instruction and Universal Design for Learning, that seems to actually create a pathway for them. And the way that I tend to distinguish that is UDI, Universal Design for Instruction is what we do. That’s us. That’s the faculty. We provide instruction. Universal Design for Learning, that’s what our students do. That’s how they learn. And so I feel like giving them the two pathways of “Let’s see what you do, and let’s see what they need to do and let’s focus on both of those pathways,” that actually feels like a wave that helps our faculty to best understand how to make adaptations. When I do teach the digital accessibility workshop to our faculty, I think one of the most challenging concepts that we faced prior to the year 2018 was having to individually go through one software program by the next. “Let’s check your document in accessibility for Excel. Let’s check your document for accessibility in Word.” After 2018, once we introduced our newest software purchase, and that’s the Ally tool, We had like a one-stop-shop, and I feel like helping them to understand you don’t have to do all of the legwork, but you can rely on a product that we’ve been able to bring on board for you to help teach you how to do those things. I think that actually relieves a lot of the stress. We have over 10,000 students here, and I truly believe all of our faculty are here for the right reasons. Their hearts are in the right place, their motivations are in the right place. A lot of this is “I don’t understand the steps. I don’t understand this technology, and it’s going to take me too long to learn it. And I don’t have that time available to me.” So, I view my role and my team’s role as being advocates for finding the right technologies and the right techniques. And then again, like I said earlier, doing that heavy lifting. Universal design, then, I think, interestingly enough, ended up applying not just to how our faculty develop their classes for our students, but it also applies to how I teach those classes to our faculty. I have to design my training, my support, in a way that is also universal. I made modifications by the second year I was here so that many of my face-to-face workshops were converted into an online webinar instead to accommodate the learning needs of our faculty. And that, to me, is the epitome of universal design. I didn’t just change my own classes, I changed my training classes for our faculty.

Rebecca: We’ve heard you say a couple of times about the heavy lifting done, in part, by your staff. Not all institutions or campuses have a staff equivalent to yours to do some of the heavy lifting or the student groups set up. How do you suggest individuals get started when they might not have the time resources or the funding resources or the staff resources, and it just seems, like, impossible?

Sherri: So, that particular question is one that I hear at most presentations that I’ve had the opportunity to share our successes with digital accessibility. I think one of the things that stands out for me the most is, both at this campus and another campus that I’ve served at, is we are not a heavily funded campus, most of the resources that we utilize are free. The development that we did with both our quality metric that we use for evaluating our courses is free. And I’d be happy to share that with the listeners as well afterwards that they’d have access to evaluate their own courses to determine the accessibility. We also have many online webinars that we share with our faculty are actually those that are offered for free because we don’t have the budget to actually purchase many of the higher cost ones. And so we utilize, a good example would be many of the free webinars and free downloads, specifically related digital accessibility come from the R&D sector of 3Play Media. And a lot of times we share those with our faculty members as an opportunity for them to self learn. Many of the templates that are available that we actually share with our faculty came from state level consortia. So a good example of that is, in my state, in the state of South Carolina, we have a new initiative called SCALE, which is South Carolina Affordable Learning Initiative. And South Carolina was actually one of the last states to come on board with that type of an initiative. So most states within the U.S. actually do have a state-funded initiative, but many faculty don’t know that it’s there. So one of the things I would definitely encourage them to do is maybe reach out to their library and ask what the state level initiative is. And I also would be happy to share information specific to SCALE because though it is specific to the state of South Carolina, all resources are freely available to anybody outside of the state as well. And it’s one that we like to promote here at Coastal. I would definitely say also, Rebecca, one of the things that I found, because I was one of those faculty members at one campus that had to serve independently without a tremendous amount of support, and I did a lot of my own research. And I found most of mine, interestingly enough initially through some searching, through MERLOT. MERLOT, a lot of times people think is just specifically to OERs. But a lot of times what I actually found was, in my discipline of psychology, I actually did just a direct outreach to other faculty who were members of MERLOT, but in the psychology discipline, and many of them freely share all of their available materials, and that includes things like accessible syllabi. So, it’s really interesting to see where things like OERs and accessibility have intersected in academic disciplines.

Rebecca: Those sound really great and helpful. Can you talk a little bit about the training that you’ve been doing for your faculty? You mentioned a little bit that there are webinars or what have you. Are they structured as full-fledged courses or one-off training opportunities?

Sherri: Yeah, so we actually do offer a number of different training opportunities, some of them do fall directly under the COOL grant initiatives. With the COOL grant initiatives, faculty do have a set of both required and elective courses, and examples of those would be quality assurance. The quality assurance teaches the faculty 10 different core principles and again, I’d be happy to share that information with your listeners. The other one that I think is probably my personal favorite, of course, is the digital accessibility. That one is actually taught in a face-to-face, 90-minute format and we usually offer it anywhere between six and eight times a semester. It usually is part one for our faculty. They learn a lot in that 90-minute workshop but because they were so interested in learning it we had to build a second part. And so they typically take digital accessibility face-to-face and then part two is what we call Ally intensive… and so it’s a hands on work specifically with the Ally tool. We also specifically, of course, teach face-to-face as well as self paced workshops specific to our learning management system, which we use Moodle here on our campus. We also teach both OER I and OER II. OER I, I specifically teach on our campus. And again, it’s taught actually as open and inclusive education. And so I teach the face-to-face one so that faculty learn how to include things that might already have been paid for out of our students tuition and incorporate those into their classes so that there’s no additive cost. And the OER II is a self paced that faculty progress through on their own in order to complete that one. Probably one of my favorite ones though, Rebecca, is actually a little bit different. This is one I get probably the most positive feedback from faculty about is what’s called a hybrid blended workshop institute. That particular workshop is 10 weeks long. And because hybrid is something that maybe, is just a different thing for faculty, is something that sometimes they maybe have already been teaching hybrid but they didn’t understand really, the functionality of what that is. What we decided to do, my co-instructor and I actually built the hybrid blended so that the faculty complete between 60 and 70% of that online, and the other workshops are face-to-face. So, by the time they finish, they have successfully taken as a student, their first hybrid class. I think that has the most impact on them. Being able to actually go through 10 weeks as your own student is so impactful. And that particular course has had a tremendous impact in a positive way. We also teach courses specific to academic integrity. We teach them about not just academic integrity principles for building content, but also about different types of tools that they might employ in order to enhance academic integrity in their courses. And then multimedia… that we teach about like personal lecture capture and utilizing multimedia for learning and those types of things. And my unit actually works hand in hand with a sister unit on our campus, which is the Center for Teaching and Learning. And they offer many of the offshoot classes that have to do with pedagogy as well. So, we’re not all by ourselves, we actually have a sister unit who helps to supplement a lot of what we do by teaching specific pedagogy classes.

John: That’s a nice, rich mix of workshops that you’re providing.

Sherri: Thank you. Yeah, we work very hard. We actually have just done a complete overhaul and have modified all workshops during the fall break before we came back for spring. So, we’re excited about the upcoming offering.

Rebecca: That’s great. So we usually wrap up by asking “What’s next?” You’ve already said so many things that you’ve been doing. But what’s next?

Sherri: So, part of what we’re doing is we’ve been working, we’re trying to figure out a way to bring all this full circle, and so the development of the Open and Digital Learning Faculty Learning Community is brand new. It has just been formed. It hasn’t even formally been announced. I’ve been talking about it for about two months. But, it hasn’t even been formally announced to our campus yet,… with the rollout of the faculty learning community, so that all of these previous cohorts can come together. And for those that are interested, we can conduct research. We can write manuscripts and here’s my most exciting part that I’m actually looking forward to, I can actually collaborate with faculty members on grant proposals. And we have not been able to do that together yet. So being able to sit down and work together and bring all this to fruition in a way that will actually move our campus even further forward by being able to write grant proposals and publish about some of our positive outcomes. I think that will be fantastic. Up to this point, we’ve offered what we call three different exemplary showcases. And the way that those work is we evaluate our COOL grant faculty recipients and to even graduate you have to become COOL certified, their courses have to hit or exceed at least 80% in quality criteria. Some of our faculty who are incredible superstars end up hitting or exceeding at 100%. And so those faculty get nominated as what we call exemplary showcase presenters. We’ve now hosted three of those, and our fourth one will be actually hosted this year in 2020. So, the exemplary showcase will be an opportunity for this next round of faculty to continue to present about the best practices. It will be our first showcase where we have individuals presenting who have implemented OERs in classes that fall outside of the scope of just digital and online learning. So, these will be classes that were taught in a traditional face-to-face environment that have converted to OER for affordability and inclusion reasons. So, I think that’s important because you mentioned, Rebecca, at the very beginning of the interview, “How is this a cultural shift?” I began here with only really being able to do my outreach to just online classes. By year two, I rolled that out to hybrid. And so in year four, which starts very soon, it will have been rolled out to all formats, all classes, all faculty, all students. And so that’s a cultural shift. Being able to find a way to show people that access is for all people, and affordability is for all people, all course formats is important. So that, to me, is the biggest thing that is coming is this cultural shift is going to continue to expand, because we’re opening up so many of these grant opportunities and these faculty support initiatives to faculty here teaching face-to-face, or hybrid, or flex, or flipped, or any variety of format that you might want to teach because access is supposed to be for all learners. And so I think that was our ultimate goal. It’s just taken us a few years to get there. And this will be our first year to actually see it open up and be totally inclusive to all formats as well, and having a faculty learning community as a target goal for how to showcase all of these best practices and all of this incredible hard work and dedication that our faculty have had in converting their courses and making them available, that will be fantastic. In the 2019 academic year, my team first started tracking the success of students based on their GPAs and their drop, fail, withdraw rates in order to see if there was any kind of correlation between those and the courses that had successfully completed quality course review. And I’m so pleased to actually tell you that, for the first time, we were able to actually see decreased drop, fail, withdrawal rates and increased GPAs in classes that had complied with and excelled in all of these quality initiatives there. So, for the 2020 year, we’re going to do a much larger research sample on that and start continuing to investigate quantitatively, “Are we able to track and even predict students success relating to GPAs and drop, fail, withdrawal rates when the faculty have successfully implemented this?” At this point we’ve seen it in undergraduate, graduate, and across all academic college disciplines. So, I’m hopeful we will continue to see it in the 2020 year as well, so we’re excited and this is one of the things that the faculty learning community is going to be helping me with, is tracking the student success metrics. And hopefully seeing some improvements across all the disciplines as well.

John: Were the results statistically significant?

Sherri: They are significant. However, one of the things, John, is the sample is still quite small. So yes, the answer is they are statistically significant. I am not comfortable publishing them yet until we have a much larger sample. So in the 2020 year, we will be working towards a much larger sample, so that I will feel more comfortable in promoting that.

John: Excellent.

Rebecca: Well, thanks so much for sharing your insights and strategies today.

Sherri: You’re very welcome.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

119. Faculty Incentives

If faculty were paid more when their students learned more, would student learning increase? In this episode, Sally Sadoff and Andy Brownback join us to discuss their recent study that provides some interesting results on this issue. Sally is an Associate Professor of Economics and Strategic Management in the Rady School of Management at the University of California at San Diego. Andy’s an Assistant Professor of Economics in the Sam M. Walton College of Business at the University of Arkansas.

Show Notes

Transcript

John: If faculty were paid more when their students learned more, would student learning increase? In this episode, we discuss a recent study that provides some interesting results on this issue.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guests today are Sally Sadoff and Andy Brownback. Sally is an Associate Professor of Economics and Strategic Management in the Rady School of Management at the University of California at San Diego. Andy’s an Assistant Professor of Economics in the Sam M. Walton College of Business at the University of Arkansas. Welcome.

Andy: Thank you.

Sally: Thanks. Great to be here.

John: Our teas today are:

Andy: I wanted to represent Fayetteville, so I went to the tea shop and I got what I have been told is the world’s greatest cup of Earl Grey tea. [LAUGHTER] It’s an award winning cup. They promised me this. [LAUGHTER]

Rebecca: Does it taste award winning?

Andy: I haven’t had enough of it yet. [LAUGHTER]

Rebecca: Reserve judgment?

Andy: I don’t give these awards out lightly.

Rebecca: And a nice lineup of mugs on your desk too.

Andy: Yes, many, too many. So this is just a way I avoid doing dishes. [LAUGHTER]

John: And Sally?

Sally: I’m drinking coffee but I’m on California time, so I’m excused.

Rebecca: And I’m drinking Spice of Life today, a white tea, John.

John: Pretty good.

Rebecca: Unusual, right?

John: And I’m drinking Oolong tea

Rebecca: You’re drinking nothing cause you forgot the cup of tea. [LAUGHTER]

John: . If I remember where I put it, I think I may have left it in the office before I came over here. But I did make a cup of Oolong tea and I did have a sip of it before and I will have it right after this.

Rebecca: I intended to drink tea. [LAUGHTER]

John: We invited you here to talk about your forthcoming article on improving college instruction through incentives. Could you start by giving us a general overview of this study?

Andy: Our study, we partnered with a large community college in Indiana called Ivy Tech. And what Ivy Tech wanted to do was incentivize instructors based on student performance. At the same time, they were rolling out a new set of large end-of-semester comprehensive, and importantly, objective exams. And so we were able to partner with them to use those exams to incentivize instructors based on the outcomes of students. So, that’s kind of the high level overview of what we were doing. I know we’ll get into more detail in a bit.

Rebecca: Can you talk a little bit about what motivated the study in the first place?

Andy: Yeah, absolutely. So, community colleges are obviously really important. It’s thought of as a sort of pathway to the middle class. At the same time, the rates of success at the community college level have been relatively low. And so if we think of community colleges as a particularly good tool for upward mobility, then it needs to be the case that they achieve better outcomes. And with the low current rates of success, it also leads to long times of accruing debt without receiving the benefits of these higher incomes from having that college education. So, there’s a whole host of factors that are kind of coming into play to make these both important and potentially underachieving tools for upward mobility. And then the other side of the equation is also that the faculty at community colleges are predominantly or at least, there’s a large percentage of adjunct faculty with really low pay and sort of what could be seen as an unsustainable business model where you’re relying on people to work in short-term, non-guaranteed contracts regularly and teach these classes. So, we wanted to address both sides, both the student achievement side, as well as the sort of personnel side of the community college setting.

John: And in terms of student success, specifically, I think you’re referring to the proportion of students that move through to a four-year degree program as being lower than what students intended. Is that the primary metric?

Andy: Yes, that’s one of the primary metrics. You can think of the community colleges as having two goals: one being graduating students with associate degrees and another being transferring students to four-year degrees. Now, Sally will know the exact number, but a large percentage of students attending community colleges, I forget what the number is, but their ultimate goal is to eventually transfer and graduate from a four year-college with a bachelor’s degree. So, there’s kind of two ultimate goals. In the process of achieving those goals there’s also gains from simply taking additional classes or receiving accreditation in certain skills, and that’s something that a lot of people go to community college to do. But, our primary long-term concerns are graduation rates and transfer rates.

Sally: Yeah, I think it’s really fascinating. Most of my work up until now has been at the K-12 level. And I think most economists, if you look at education economists, there’s a lot of focus on the K-12 level and looking at teacher quality at the K-12 level and how can we improve teacher quality at the K-12 level? When we came to the college level, there’s been work showing how important it is who your instructor is. Instructor quality matters a lot. But we couldn’t find any work looking at how can we improve instructor quality at the college level? I think it’s really interesting because community colleges are getting a lot of attention from policymakers because they’re low cost, they expand access to underrepresented populations that normally don’t have as much access to college: minority students, students who are first generation college goers, students who are working and so they can’t travel necessarily to go to a college. And so we think that community colleges provide amazing opportunities to students, but as Andy was saying, they really struggle with success rates. And so 80% of students entering a community college say they intend to transfer to a four-year school and fewer than 30% end up doing so. Fewer than 40% of students graduate with any kind of degree within six years. And so these colleges, and we see this working with Ivy Tech, they are incredibly dedicated. The administrators and the teachers there are incredibly dedicated, but they’re working with students who are struggling, nd so there’s a lot of room for improvement. And what we found actually that’s interesting, I think, at community colleges, is that there’s actually more room to think about how to structure employment contracts than there is at the K-12 level. Because often, the instructors aren’t unionized, as Andy was saying they work under these short-term, flexible contracts. And so there’s a lot of flexibility. And really, people haven’t thought much about how to structure these contracts in a way that can improve performance and motivate both instructors and students.

John: It’s a fascinating study. For those of our listeners who aren’t familiar with field experiments, could you tell us a little bit about what a field experiment is?

Andy: Yeah, absolutely. So a field experiment is, in our case, a test of policy. And the way it’s experimentally designed is through what would be known as a randomized controlled trial, meaning that you take a sample of people from a population and you split that sample into a treatment and a control group, and you do this randomly… and that’s the really important part. Because if you test a policy with an assignment that’s anything but random, then you can’t guarantee that these two groups are otherwise equal. But in our case, we’re going to randomly assign people to be in the treatment group or the control group. So, the treatment group will receive the policy, the control group will continue in the current status quo. And then what we will do is look at outcomes and how they differ between the two groups. Now, since the assignment to the two groups is random, again, there’s no mechanical correlation between treatment assignment and any of the characteristics of the groups themselves. Then we can know that any differences subsequent to the assignment are results of the treatment itself and not any sort of spurious correlations or selection biases.

Sally: Yes, I think listeners are probably familiar with this kind of experiment when you think about testing a drug or a vaccine, those kinds of clinical trials. And more and more economists have brought those models in for testing policies. And I think they gained a lot of attention recently because of the recent Nobel Prize, which highlighted how powerful these experiments can be for evaluating policies. And so I think that they gained a lot of attention from economists, they’re growing in their use, and it’s really thanks to partners like Ivy Tech that are willing to let us come in and test things in this way. Because, I think although people are very comfortable with the idea of testing a drug in a clinical trial, sometimes there’s discomfort with testing policies in this randomized way. And so we’re really grateful when we have partners who are willing to let us come in and try these new policies and implement them in this randomized way where some instructors receive incentives and some won’t.

John: And in a sense, we’re always testing things. It’s just, we don’t always measure the effect of it. When you something new in your class, you are doing an experiment. But unless you have a control group to compare it to, you can’t really assess whether the gain is due to that particular intervention or something else that was happening.

Sally: That’s exactly right and we really try to emphasize to people exactly that, that you’re always trying things, rolling out new policies or stopping one thing and doing it differently. And if you’re going to be making these changes, do it in a way where you can learn from them instead of just trying something, trying to step back and try to understand whether it worked or not. How do you know whether something is working or not unless you can compare it to a proper control group?

Andy: And just to emphasize the importance of this methodology, there’s a lot of policy that gets rolled out based on bad data and bad evidence. And so if you’re using a poorly designed experiment, or simply looking at correlational data and rolling out policy, what you could be doing might not be effective, it might be actively detrimental to students. But once you have this clear causal evidence, we can be really confident in the policies we roll out and understand the cost-benefit analysis of the policies prior to implementation.

Rebecca: Can you talk a little bit about the policy that you were testing in this particular experiment?

Andy: Yeah, so as we talked about, we wanted to roll out incentives for instructors based on student performance. And we base these incentives on objective, comprehensive exams for a variety of courses in a variety of departments. The exams are designed outside of the classroom in the sense that it was designed by deans and department heads and represented the types of material that they wanted the students to master by the end of the semester. So, those form the basis of our incentives that we would be giving to instructors. Now, we didn’t just want to offer incentives based on outcomes. We wanted these to be potentially as powerful as possible. So, we leveraged an approach that Sally’s researched in the past in a paper with Roland Fryer, John List, and Steve Levitt, where they looked at loss contracts.

Our incentives were actually such that every instructor would receive $50 for every student who passed the exam, and passing the exam is defined as receiving a 70% or higher on the exam.
So, we framed these as losses. And we delivered incentives at the beginning of the semester, as if half of the students in an instructor’s course had passed the exam. Now, this established it as sort of a target, but it also allowed us to leverage this idea of loss aversion, that instructors would value keeping money potentially more than they value gaining an equivalent amount of money. So, as the students progressed through the semester, at the end of the semester they would take this exam, we would have these objective evaluations for how many students passed the exam, and then we calculate their final payments. If their final payments exceeded this initial payment, they would receive additional payments. If their final payment was less than this initial payment, we would clawback some of that payment. And this was all explained at the outset of the experiment. And again, this sort of loss framing is leveraging a long line of research in behavioral economics, about how much more motivating it can be to face potential losses than equivalent gains.

Sally: Yeah, so just to give an example, if you have 20 students, and you get $50 per student who passes, half of your students passing, that would be $500. So we would send you a check for $500 at the beginning of the year, the beginning of the semester. At the end of the semester, if fewer than 10 of your students pass the exam, say only eight students pass the exam, you have to write us a check back for $100. If more than 10 of your students pass the exam, say 12 of your students pass the exam, then we send you a check for an additional hundred dollars. And we found in previous work that having this money in your bank account and knowing that you potentially could lose it if your students don’t pass the exam can be very motivating, compared with rewards that you only receive at the end of the semester.

Andy: Yeah. And one point about the logistics real quick is that these initial targets were based on enrollment as of what they call the census date. It’s not the drop deadline in the sense that you can’t drop afterwards, but it’s the deadline at which point dropping a course is no longer costless. All the students at this point in the course are enrolled sort of formally, and instructors will receive the upfront incentives based on that number of students. So, there’s multiple margins at which the instructors can influence student outcomes.

John: One thing I think that’s probably worth noting is that one advantage of doing it in a community college is that it’s much easier to have that standardized testing. I know in a lot of four-year colleges, faculty would object to having to assign an externally designrf exam at the end of the term, while in community colleges that type of standardization is much more common, which makes it a bit easier to design a study like this, I would think.

Sally: Yeah, that may be the case. Interestingly, I think, even for accreditation, for example, often you need to show that the test has certain questions on it. I know in large classes with many sections, they often write the exam together. The goal at Ivy Tech was to sort of create this bank of questions that every year tests would be drawn from, and I think moving classes over to that model is interesting. And there’s more openness to it than I thought. So, for example, when we started this study, I thought, “Oh, the only courses we’re going to get in this study are going to be math and maybe some science courses.” And what’s really interesting to me about this study, is unlike at the K-12 level, where it’s primarily focused on math and reading, we have a really wide range of courses. We have anatomy and physiology, art history, nursing, psychology, criminology, sociology, psychology. And so what it showed to me was that you can really get a wide range of courses into this kind of framework. And it doesn’t cover every element of the course. But, for example, in the English courses, one thing they were moving toward was evaluating the essays in a more objective way where you’d have two readers that would both rate the essays and compare ratings. And as colleges move toward those models, I think that this kind of framework will be more and more implementable.

John: It’s certainly good for assessment, and it’s certainly good for evaluating the effectiveness of innovations in instruction. There’s a lot to be said for it. I’m just thinking, at my college I know in many departments there’d be some objections to this. We used to have a standardized common final in the economics department where I teach and people objected to that for a long time, and we eventually moved away from it, but we are talking about doing something similar with at least some subset of questions that would be standard, for that sort of purpose.

Sally: Right. And I think always a concern about these kinds of studies is if the incentive is based on the objective part of the exam that can be tested and assessed in that way, does it take away from the other parts of the course that are more qualitative or more specific to each instructor? And so one thing we were really careful about in this study was to look at not just performance on the test, but how did students do in the class overall, how did they do on the other courses they were taking at the same time? How did they do in future coursework? And I think that’s really important that it’s not just all about teaching to this one assessment that’s going to be used for the incentive.

John: Given the strong findings on loss aversion in terms of how people find losses much more painful than gains of equivalent value, how did faculty react to that incentive structure? I believe you surveyed them on that early on, and then again later.

Andy: Yes, at the outset or at the baseline, the faculty did not like the idea of these incentives. This is both evidence-based where we have survey information and people were willing to sacrifice a rather large amount of money to have these contracts converted into contracts that were gain-based contracts that wouldn’t be paid out until the end of the semester. Anecdotally, this fits with my experiences, I went to explain these contracts. There was quite a bit of pushback in asking why these were framed in this way, and some people potentially wanting to approach them differently. Interestingly, this was very heterogeneous across departments. The accountants were like, Okay, well, I know what to do with this, [LAUGHTER] and put it away, and the psychologists were particularly upset because they knew exactly what we were doing. But, the data show that with experience, our treatment group, on average, has no preference between a loss contract and a gain contract, meaning that a large amount of this distrust of the contract could be attributable to just a lack of experience with this style of contract. And that as instructors gained more experience, they also gained a comfort level with the contracts as well.

John: I still wouldn’t rule out loss aversion as being a factor, but it is interesting that it gets reduced after they’ve experienced it.

Andy: Oh, absolutely. So, that’s not to say that loss aversion isn’t still a factor. But, as you gain experience with these contracts, maybe you start to appreciate the motivating qualities of loss aversion. So, maybe you understand that although these contracts cause you to work harder, or cause you to exert more effort around a certain goal, that by increasing that effort, you’re actually achieving greater outcomes for yourself. And if that’s the case, then they’re still motivating you through loss aversion, but you may not be as averse to the contracts as you were ex ante.

Sally: Yeah, so it may be that people are using them as a type of commitment contract where they know that yes, it will be painful while I’m in the contract, but it’s a way to motivate me to work harder, and I’ll walk home with more money than I would otherwise.

John: Just a couple of months ago, we did a podcast on commitment devices with Dean Karlan…

Sally: Oh nice.

John: …and we talked a little bit about that, and StickK.com, the site he created for that. Now, we’ve talked a little bit about the incentives for faculty, but you also introduce an incentive for students. Could you talk a little bit about that as well?

Andy: Yeah. So, on the student side, this was only in the spring semester. We rolled it out in the fall semester, where we had a pure control group and instructor incentives only. As we moved to the spring, we then cross randomized those two groups with student incentives. The students were incentivized with the following possibility. If they pass the exam, that is receive a 70% or higher, they would get a voucher for free tuition for a summer course. And this could be worth up to about $400 worth of tuition. So, now students are incentivized alongside the faculty. And we wanted to test whether 1. student incentives were effective and 2. if they made the instructor incentives even more effective.

Sally: Yes, we were interested in whether there’s complementarities between student incentives and instructor incentives. We knew from prior work that offering student incentives alone has, at best, modest effects. But, we thought that maybe if we put them in combination with instructor incentives, we could imagine the instructor saying to the students, “Look, guys, you guys have something at stake here too…” and it could create this positive cycle.

Rebecca: So can you tell us a little bit about the results?

Andy: That’s on page 22. [LAUGHTER] We found that the instructor incentives were really effective. They increased student outcomes by about 0.2 standard deviations on those exams. It’s a really nice effect in this literature. What’s also exciting is, suppose you don’t believe our tests or don’t like our tests, they also reduce course dropouts by 3.7 percentage points, which is about a 17% decline in the course dropout rate. They raised grades in the course by over a 10th of a standard deviation. And that’s even if you take out the effect of the exam itself, the course grades still go up by about a 10th of a standard deviation. And these positive results spill over into other courses. They complete other courses at higher rates, they accumulate more credits, and they even go on to transfer at higher rates. So, that’s in the faculty incentives or the instructor incentives branch of the study. When we look at the student incentives by themselves, we see essentially no effects on any key outcomes that we care about. When we look at them in combination, they actually don’t improve the impact of instructor incentives. If anything, we see a pretty small negative effect that wouldn’t be any significant difference at all. But, there simply doesn’t seem to be any impact of the student incentives. Now, this could be attributable to our specific student incentives. But, you’d have to believe essentially that they have either no value or very limited value to say that it’s just the fact that we’re incentivizing students in a very specific way.

John: When you first were talking about it, one of the things that struck me as… I think it was W.C. Fields who was talking about a contest where he said the first prize was a week in Philadelphia. Second prize was two weeks in Philadelphia. [LAUGHTER]

Sally: So, Andy and I are doing a separate study on summer school. And we do find that students do not want to attend school in the summer. But, interestingly, if we can get them to attend school in the summer, it has a really big impact on helping them graduate sooner. So, we’re really fascinated with understanding how we can address this aversion to summer school. But, that may be for another podcast. But,we agree that, I think that the incentive for students may not have been very motivating. I think just to return to the results about the instructor incentives, I think there’s some really interesting results there. First, something that’s unique to the college setting that you don’t find in the K-12 setting, is this really large problem with students enrolling in a course, paying for the course, and then not completing the course. So, about a quarter of students fail to complete courses that they’ve enrolled in and paid for. And this is a big struggle at community colleges. So, just increasing these rates of persistence in the course we think has a really large impact. And what it seems like is happening is instructor incentives get students to keep coming to their course, and so students go to their other classes as well. And so it has this really positive reinforcement effect on students completing all of their courses that they’re taking that semester. I think another really exciting result is that a year after our program ends, when we’ve stopped giving anybody incentives, you see these really large impacts on transfers to four-year schools… about a 20% increase in the rate of transferring into a four-year school, which we think is really exciting, which is the primary goal… as we talked about the primary goal of community college is to get these students to transfer to four-year schools. They really struggle with that. And so we see that this could have a really large impact.

John: And education is costly. And if we get more people finishing, the private and social returns, both go up significantly. And the cost of doing this is relatively low. It’s substantially less costly than the student intervention.

Sally: Yeah, it’s incredibly low, about $25 per student. One thing that’s interesting, again, about community colleges, because adjunct faculty are not paid very well, you can offer relatively cheap incentives that represent a significant bonus. So for these adjunct instructors, the average bonus represented a 20% increase on their baseline salary. Our adjuncts are making about $1,700 for a 16-week course. So, you can get a lot of bang for your buck with adjunct instructors, and we see the largest impact among adjunct instructors. Those are the instructors that really responded to the incentives. And adjunct instructors are increasingly becoming the model for schools, not just community colleges, but four-year schools as well. So, they represent about 50 to 80% of instructors at four-year and two-year schools, respectively. And that’s on the rise. So we expect that to increase in the future.

John: And that’s another topic we actually address in a podcast that was released on December 18.

Andy: So, I think the adjunct effect is also one that’s worth emphasizing, just because of the model of using adjunct faculty or increasingly using adjunct faculty is unsustainable at the current pay rates. So, if we think about these contracts as being more flexible as these adjunct instructors are more used to working on temporary contracts, if it turns out to be the case that you can’t continue to pay people such small amounts for so much work, then how do you design contracts in the future that can maximize student outcomes? So, if we’re in a world where we know we have to redesign these contracts, what we wanted to be able to do with this study is say, “This is a way you can redesign the contracts and achieve the outcomes that you hope to achieve.”

John: That works well, when the test is administered or designed externally. There would be some incentive issues, though, if the instructors had more control over the test or that assessment of how well their students did, I would think.

Andy: Yeah, absolutely. And that was at the front of our minds while we were designing the study was, “Are we not simply motivating people to either teach to the test, or to lie to us outright,” and based on the way the exams were designed, these are both objective and for the most part, externally graded. So, it’s still possible, for example, for a teacher to just erase answers and write in the correct answer if they wanted. But, there’s a certain point at which you have to start trusting your subjects, that they’re not attempting to deceive you. And so we kept that sort of in mind as we were thinking about how to design the study.

Rebecca: Did you have any feedback from faculty at the end of the study, when they discovered that your incentive worked, for example?

Andy: So, we have been in touch with our partner in the administration, we haven’t been in touch with the faculty themselves with our working paper or now the forthcoming paper. So, we hadn’t gotten feedback at that point. We did get feedback in the process of the study that is like at the end of the fall semester, and at the end of the spring semester, and just like the preferences for these contracts, the feedback was, of course, not universally positive. But for the most part, the majority of people appreciated the extra money. And I guess this is something that we haven’t emphasized yet, but we didn’t really change anyone’s contract, they were still operating under the existing contracts. And these served as a bonus on top of those contracts. So, there was very little room to think of these as sort of a really detrimental change toward your contract. Because the worst-case scenario is that you were under the exact same contract as you were previously.

John: If everybody failed, or if everybody came in below the threshold.

Andy: If literally zero percent of your students were able to pass this exam, you were in the same world you were previously.

Sally: We had high rates of sign up in the fall, and then even in the spring semester, there were people in the fall who hadn’t signed up that chose to sign up when they had a chance again, and all but one instructor continued the study from the fall to the spring. So, I think that instructors did like participating and we generally got positive feedback.

John: So, you got really strong results for the incentives for instructors with larger results for the lower-paid instructors… for adjuncts. Was there any evidence of the mechanism by which this affected student outcomes?

Andy: So, we look into mechanisms in two ways. One, we look at self reports of time use. And we really don’t see any significant differences between the treatment and control groups. So nothing that would clearly identify a change in behavior. Now we have one caveat to this, and that’s that when we put the time-use survey out, we limited each activity to 16 hours, not thinking how many of our instructors might spend more than 16 hours on a given activity. And that was made pretty obvious with the outside-work option. And so it is possible that we are top coded there and unable to differentiate between the two. And we also look at student evaluations, and we don’t see any significant differences between the way students evaluate instructors that were in the treatment versus the control group. So, we don’t really see a specific mechanism that’s driving these differences in student outcomes. And if we really wanted to try to isolate these things, we would need to maybe have some better or more objective data about instructor practices or a more fine-grained approach to looking at time use, I think.

John: That could be an interesting follow-up study.

Sally: Yeah, I think now that we’ve shown that these incentives work and can be very powerful, getting inside the black box of the mechanisms is our next step. And we’re currently working with an online university where everything instructors do and everything students do is passively recorded because they’re interacting online. And we think that will give us more fine-grained data. If you think about it… If I asked you last week, “How many hours did you spend on email? How many hours did you spend prepping your course?” It’s really hard to recall that without a lot of noise in there. And I think the other thing we discovered after presenting the results, talking to instructors, talking to administrators, talking to other people who work in this area, is that a lot of it might not be captured by time spent. Some of it might be… you learn the names of the students in your class… when you saw a student in your class who was on their phone, instead of letting them be on their phone, you said, “Please put your phone away, please close your laptop.” And so it might be much more subtle practices that we need to either observe classrooms or do focus groups or really get more qualitative data. And that’s something we’re really interested in doing.

John: Because it could be motivational, it could be that instructors who know that they’re going to get paid more might put a little more effort into those things that may not be captured by those measures. One hypothesis I was thinking is that it could also be that the existence of the incentives might perhaps encourage people to develop a growth mindset. And there’s a lot of evidence that faculty that have a growth mindset tend to have students that do better, or at least that have narrower performance gaps.

Sally: That would be really interesting, I think, for evaluating. We’re already surveying instructors at baseline and throughout and so we could see if the characteristics of the instructors change or their attitudes. We do ask them their attitudes about teaching and their view of students. For instance, questions like “most of my students achievement is determined by background” or “I’m able, with enough effort, to change how my students achieve.” And so we can look more closely at those questions. We use them mainly as baseline questions to characterize teachers about their attitudes. I don’t think we’ve looked to see whether their attitudes change. So that might be an interesting approach, we should take a look at those data.

Andy: One other mechanism that’s opened up by our incentives is that what we’re doing is essentially giving people a big influx of cash at the beginning of the semester. And so this could also just open up resources or capacity constraints that they had without these incentives. So for example, you could imagine someone who’s also working part time, who now gets a check at the beginning of the semester based on all of these potential student gains and doesn’t have to spend as much time working in their other job. Things like that could be potential mechanisms and could also explain why adjunct faculty have this really large differential effect. But again, we don’t have that hard data. And so it’s something that’s really interesting to us. But, unfortunately, not cleanly identified by our data.

Sally: One thing that we received is an unsolicited text message exchange between an instructor and their student, which I thought was interesting, because my students don’t have my cell phone number. But, things like that, giving out your number, exchanging text messages, the sort of individual support that I think, especially for community college students who may be less connected to campus, less connected to the community, could be really important. And so we want to think more about that sort of sense of connection to the community, to your instructor, to your fellow students.

Rebecca: I’m really excited to find out what your next round of studies reveals, because you have interesting directions that you can go in right now. And then really valuable information that you’ve already discovered.

Sally: Yeah, I think another interesting direction that we’re very interested in… is Andy’s talked about this model of being sustainable, especially as schools move more and more over to this adjunct model. So, another thing we want to understand is if a school offers these kinds of incentives, what kinds of people do you attract? Are you better able to retain your high quality instructors? Do you recruit higher quality instructors? So, that’s another question we’d really like to answer in future studies.

John: Because you’re offering higher pay to the faculty that are more effective, which could have an interesting self-selection effect on the faculty composition.

Sally: Exactly.

Andy: Yeah. And if anything, our results suggest that it takes a little bit of experience with these contracts to really appreciate them. So, moving to a model where you have these types of contracts, there might be a transition period where it was challenging before it became something that people understood as beneficial to themselves.

Rebecca: And not just to themselves, to the bigger educational community. Yeah.
So, we always wrap up by asking, what’s next?

Andy: I can talk about a project Sally and I are working on right now, as we talked about earlier, summer enrollment was seen as this potential mechanism to drive student success. And so we did a really simple experiment where we just randomly assigned people to receive a free summer course and then tracked their outcomes for the two years subsequent to that summer course. So, we’re wrapping up a working paper on that. And it looks like summer has this really nice long-term effect that would be kind of hidden in the short-term data because of the fact that you don’t see impacts on retention between spring and fall. But, you do see these impacts on credit accumulation in the short run and then graduation and transfers over these shorter windows as well.

Sally: So, I think as behavioral economists, something that Andy and I are really interested in is the intersection between preferences for contracts, preferences to attend in the summer, and the impact of those kinds of contracts on your future outcome. For example, we find that instructors don’t really like these loss contracts, but they perform really well under them. We find that students don’t really want a summer scholarship, but it has a really big impact on their future outcomes. And so trying to understand this intersection of your preferences for the here and now, and how these things may or may not translate into your future outcomes, is something that I think will be really interesting for future research.

John: This is a topic we keep coming back to in other contexts, that in terms of student metacognition, that the approaches that we know are most effective for learning are the things that students tend to value the least, and tend to perceive as being less important. So this is a pretty general problem, I think.

Andy: And isn’t there data showing how students give worse evaluations to teachers that cause greater amounts of learning?

John: There was that Harvard study a few months ago in a physics program there, where they found that students believed active learning to be less effective in terms of their learning. And yet the students who were exposed to active learning techniques ended up with larger learning gains. And that was also a randomized control trial.

Andy: Yeah.

Rebecca: People just don’t know what’s good for them.

Sally: But it’s hard, because I was trained at the University of Chicago. I am a behavioral economist, but I’m also a University of Chicago economist. And I believe in respecting people’s preferences and their choices. And so we have to be very careful about how to sort of take these complex and think about how to translate them into policy.

John: In terms of gentle nudges that work well.

Rebecca: Well thank you so much for joining us, it’s been really interesting.

John: It’s always better when there’s economists on.

Rebecca: I’m always outnumbered.

John: This has been fascinating, thank you.

Andy: Thank you.

Sally: Thank you so much for having us.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]