318. Reducing Equity Gaps

Gender and racial equity gaps exist in economics and other STEM fields. In this episode, Tisha Emerson joins us to discuss research on strategies to reduce these inequities. Tisha is the chair of the economics department and the James E. and Constance Paul Distinguished Professor at East Carolina University and is the incoming Chair of the American Economic Association’s Committee on Economic Education.

Show Notes

Transcript

John: Gender and racial equity gaps exist in economics and other STEM fields. In this episode, we discuss research on strategies to reduce these inequities.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Tisha Emerson. Tisha is the chair of the economics department and the James E. and Constance Paul Distinguished Professor at East Carolina University and is the incoming Chair of the American Economic Association’s Committee on Economic Education. Welcome, Tisha.

Tisha: Thanks for having me. I’m so glad to be here.

John: I’m very happy to be talking to you. And our teas today are:… Tisha, are you drinking any tea?

Tisha: Yes, my favorite tea, I found a couple years ago, it’s a Golden Moon brand of Tippy Earl Grey.

Rebecca: That sounds nice.

Tisha: It’s delicious.

Rebecca:I have Harvest Memories today.

John: I don’t remember that one.

Rebecca: Well, it is from my little favorite tea shop in Canandaigua, New York, and it’s autumn flavors.

John: Autumn flavor?

Rebecca: Autumn flavors.

John: Do they have leaves dumped in there?

Rebecca: I mean, it almost looks like that. [LAUGHTER]

John: Ok. [LAUGHTER] Well, that’s one way of getting rid of all the leaves that have been falling. And I have English breakfast tea today. So we’ve invited you here today to talk about some of the research you’ve done, and also your new role and your past role with the AEA’s Committee on Economic Education. One of the things that we’ve observed is that there’s some significant equity gaps in economics and in STEM disciplines in general in terms of race and gender. And in one of the studies that you’ve done with this, in a 2023 paper in the Southern Economic Journal, you and KimMarie McGoldrick examined retake behavior for students who are not successful in their initial attempt at completing an introductory microeconomics class. Could you give us just a general overview of this study?

Tisha: First, let me say that this started a long time ago. And we got access to this great data, the MIDFIELD dataset, that was actually originally funded by the NSF to study the gap in engineering education, but it’s student transcript data. So we said, well, you could use that for economics too. And we have. And so this particular study looked at almost 180,000 students who take Principles of Microeconomics for the first time. And what we ultimately wanted to do is to think about the likelihood of success, and actually more so, failure, because a lot of papers already look at success and they stop there, because you have too small of a sample, fortunately, of the failures to continue on to look at them. But when you start with 180,000, you have enough to continue. So we find, as many others do, that, of course, aptitude matters, but that, unfortunately, females tend to be much more likely to be unsuccessful in their Principles of Microeconomics class, as are underrepresented minorities, or URM. And then we follow those students and we say, okay, so if you were, in fact, unsuccessful, and we define that as a grade of a D or an F, so grades that wouldn’t really let you continue in your study of economics, or you withdraw. And we say, if you’re in that category, what do students tend to do?… or first of all, for those students that are unsuccessful, what helps predict that? So given that you are in the unsuccessful category, what is more likely to cause that? And we saw that students who were carrying more concurrent credit hours were more likely to withdraw. Which makes sense, because if you have constraints that lead you to want to be full time, if you start with 12 hours, and the course is not going well… any of them… and you were to drop, then you would not be full time anymore. And so it gives them sort of this flexibility, they have more concurrent credit hours. But we found that students who got D or Fs, given that they were unsuccessful, they tended to have taken more related courses. So things like accounting (financial accounting in particular), calculus, and macro principles. And so we didn’t actually see a lot of gender or racial differences there. But then those students who were unsuccessful (D, F, or W), on retake decisions, we found that women were much less likely to retake if they were unsuccessful. But we did find that women who were of higher aptitude, who were unsuccessful initially, they were more likely to retake. So that was a positive. And we did find that underrepresented minorities were more likely to retake the course, but then more likely to be unsuccessful again, on their second attempt. So there’s some good news, but also, a lot of not so good news.

John: So you were able to follow students over time, how long a time period was there in the data set that you were analyzing?

Tisha: We had over 20 year period of data, but that doesn’t mean we followed any particular student for that length of time. Once we saw that they were unsuccessful, we made sure that they had at least one more semester at the institution so they did in fact have a chance to choose to retake the course or not and then we gave them three years post their initial unsuccessful attempt. And if they didn’t retake it in that period, then we said they chose not to retake.

Rebecca: In a 2019 study in the Journal of Economic Education, you and KimMarie McGoldrick use the same dataset to examine the decision to switch into an out of the economic major. In this study, you find that very few students selected economics as a major when entering college and that 83% of economics majors actually selected the major after taking their first economics course. What did you find in terms of the gender and racial composition of those that selected the economic major?

Tisha: So what we found was that, for the few students who started out as economics majors, so going into their principles class they already had said they were majoring in economics, we found that females were as likely to persist or not. That’s good. And we found something similar for underrepresented minorities, that is the females and URM students who went into the principles class planning to major in economics, they were as likely to persist as not.

John: What happened to those students who were not majoring in economics.

Tisha: Well, unfortunately, females were not as likely to persist and decide to major in economics. There was a larger proportion of females who opted to remain in their original major. So, they come into the principles class, they have some major other than economics, they were much more likely to stay in their other major and not switch to economics. Men were considerably more likely to switch, on average, than females were. But on a more positive note, we did find that students from traditionally underrepresented racial and ethnic minorities and Asian students were more likely to switch to economics.

John: And one thing we should note, and you mentioned this in your study, is that because this datset was focused on engineering, the schools in this sample have relatively large engineering programs and some students switched to economics because economics was perceived as being easier than their original major. So some of this may not apply as broadly to liberal arts institutions that do not have engineering programs.

Tisha: That’s possibly true. And in fact, business majors were even more likely to switch to econ. So it was about 10% of those switching in were switching from engineering and 27% were switching from business. So at the liberal arts schools, to the extent that they don’t have business or engineering, there wouldn’t be that source of majors coming in.

Rebecca: Both of these studies are really interesting. But, how do we think about some strategies to create a more inclusive environment in economics and STEM classes? So what did these studies tell us? What should we be doing?

Tisha: Well, I don’t know that they tell us what we should be doing. They tell us that what we’re doing is not really working.

Rebecca: Fair.

Tisha: …which is unfortunate, because that’s what we’ve been doing for a really long time and we’ve been really unsuccessful at attracting women and underrepresented minorities. So there’s a new sort of strain in the literature that’s really trying to address some of this. And there’s a study, for example, by Porter and Serra, and that was published in 2020. They did a randomized control trial at Southern Methodist University, where they, just by happenstance, picked a couple of very charismatic female alums and they randomly selected which principles classes they would speak to. And this was just like a 15- or 20-minute exposure, where they talked about majoring in economics, and how that helped them in their career. And they found that this significantly increased the number of women from those courses who decided to major in economics and additionally, not necessarily just major, but take more economics courses. And I thought it was really fascinating that there was no effect on men in those treated classes, it was just the women. So that suggests possibly that there’s some room for role models. Other work that looks at role models from other directions, like some of my own work with KimMarie and John Siegfried, didn’t find any evidence that supports the idea of role models. So I think, still mixed, and the exact sort of interaction that students may or may not have with these people I think is going to be important.

John: You’ve done a number of studies on these types of issues, and one of them was looking at the effect of classroom experiments. Could you talk a little bit about what types of classroom experiments have you looked at and what’s the impact of those on students?

Tisha: Sure, I talk about that a lot, because I really love the classroom experiments. I think that they’re a great pedagogical technique. And so since your audience is more general, maybe I’ll explain just what we mean by classroom experiments. And that’s basically the idea that we’re going, in some cases, simulate markets or other decision environments for the students that will mirror the types of environments that we’re talking about in class. So for example, if we’re talking about a market, then we will have students participate in an experiment where some are buyers and some are sellers and they negotiate and trade. In my classes, we always do the experiment first, I don’t talk about any of the concepts. And then we collect the data as we’re going through the experiment on the decisions that they’re making: prices, quantities that are traded, then I’m able to talk to the students about “Well, look, this is what actually happened, this is your actual data that you generated, and now let’s talk about the theory.” And this is what the theory would predict the outcome to be. And they can see how well they match up. And they are able to discuss in class, this is why I chose to behave in this way. And they see how it fits in to the economic theory. And I did a study back in 2001, which was published in ‘04 in the Southern Economic Journal, where we were looking just at the effect of experiments and we looked to see if there was a differential gender effect. And in fact, there was. So you often see that women underperform men in these principles classes. But what the experiment did is it closed that gender performance gap. So it raised sort of everybody’s performance, but it raised the females disproportionately, so that they were closer to the same outcome as the men if they were in a classroom with experiments. So I do think that there is definitely room through pedagogy to improve women’s outcomes, and make them more attracted to economics, but also make economics more attractive to them. There’s a large literature that suggests that women are much more grade sensitive than are men, various people have gone about showing this in different ways. And if you can bring up their performance so that it’s on par with men. And it’s not really even that comparison, but you’re bringing up the females’ performance, then they’re getting better grades, and they’re going to be less likely to want to leave the study of economics.

John: And we’ll throw in a reference to an earlier podcast we did with Peter Arcidiacono, who had looked at that very impact in STEM fields in terms of the decisions of people to move from one major to another. And in particular, in that study, they found that women did better on many of the tests but still left because the grades were relatively lower than in other classes. So one issue I think might be worth addressing is the issue of grade differentials, that if we’re going to continue to assign grades that are lower than most other departments, there’s a good chance we’re going to lose majors in general, but disproportionately more female majors, based on what the research literature tells us.

Tisha: There’s this really cool study, McEwan, Rogers, and Weerapana, 2021. They’re at Wellesley, and so it’s all females, but they were trying to address grade inflation. And what that meant was, of course, there are some disciplines that have higher grades, they had to bring their average grades down, and economics is treated in the opposite direction, and that it actually drew more students into these harder grading disciplines, like economics and the other STEM fields. And so yeah, that’s certainly an issue. I don’t think we’re gonna across the board be trying to treat grade inflation. I don’t think this is published yet. But I was reading another study out of Wellesley, where they were allowing students… I think it was maybe for their gen ed classes… to take them all pass fail. And again, you saw students flowing into these harder grading disciplines.

John: I know a number of schools during a pandemic made all of their first-year classes pass fail, which is a good way of letting students get into a discipline without worrying so much about the grades so they can explore things without worrying about the impact on their GPAs. That might be another useful strategy.

Tisha: Yes, I agree, and then this is really anecdotal, but I’ve heard other people observe the same thing, that women, they’ll be in your principles class, and they’re doing well, but maybe they’re going to get a B plus. And they say, “Oh, either I’m not gonna take another economics class,” or I’ve even had students withdraw at that point, and I’m like, “but you’re getting a B plus, I don’t understand,” but they’re so sensitive to grades. And then there will be male students who are getting a C, barely, and they’ll say, “I’m going to major in economics.” and I say, “Okay, are you sure?” It’s hard to understand sometimes.

Rebecca: It’s interesting to think about withdrawal policies that have really kind of loosened up due to COVID. So our institution changed the policy more recently so that students can withdraw through the last day of classes without documentation. And so students who are doing really poorly might take that option, but as you’re discussing students with higher grades withdrawing, that sounds really concerning that policies like that could be kind of counterintuitive, or really have some other negative impacts. Classes all cost money, [LAUGHTER] and there’s financial impacts to withdrawing.

Tisha: It’s hard to really predict how this is all going to play out. And I have to say, I’ve never heard of such a generous withdrawal policy. At my previous institution, I think they could withdraw up to sort of two weeks out and not even have a W necessarily on their transcript. So this all seems like a lot. There’s a lot of moving parts. But on the positive side, so let me just say, first of all, part of me feels like it’s a little bit too generous. But part of me also thinks back to the work that KimMarie and I did. And if you don’t have that D or F, that you have to overcome, it’s a lot easier to persist, not just in your major, but towards graduation, whether you switch majors or stay. And part of me feels like that’s helpful to the student. So in some ways, I like a generous withdrawal policy, I definitely want students to know what the withdrawal policy is, what the deadlines are, so that they can make a good decision.

John: Yeah, the only concern I have with the policy is that it can lead to students making slow progress towards a degree and they could run out of funding before they complete their degrees. But on the other hand, it does provide a bit of a safety net for students who are adjusting. And as our student body has become more diverse, and we have more first-gen students, and we have more students coming from schools that did not prepare people particularly well for a college environment, it provides a safety net, which has allowed students to take some chances early on and not be harmed. And I’ve seen some students struggle in their first year or so, drop classes, and come back and be very successful, when before they might have been deterred from further study in the discipline.

Tisha: I agree, I can see that.

Rebecca: So, definitely one of the things that I’m always thinking about when students are talking about withdrawing is making sure that they know all of the ramifications. So what does it do to your GPA? What does it look like on a transcript? What are the financial implications? …so they can make a well informed decision. And depending on who they’re talking to, they are not necessarily always getting all of the information, and that can be problematic, too.

Tisha: Yes. So I think that points us to the issue of how important advising is, so that there are well informed advisors and that students have access to them, not just at the end when it’s time to register for next semester, but they have a relationship with their advisor, and feel that they can go in and ask those questions, because as John said, a lot of those students can’t go to their parents, they’re first gen, their parents don’t have advice for them. And even if your parents went to college, they don’t necessarily know all of the rules and regulations of the institution that you’re at. It’s really hard, I think, for students to have all the information that’s necessary.

John: I’ve had trouble keeping up to all the changing information [LAUGHTER] over the last few years. In terms of those requirements. You’ve done some work with cooperative learning. Could you talk a little bit about that?

Tisha: Sure. So that really came out of the fact that I had done a lot of efficacy work around classroom experiments. And KimMarie and I are really good friends. She is a cooperative learning expert. So I should say that she is the expert on CL. And she wanted to do an efficacy study, so we decided to team up and do that. So what we did is unlike a lot of some of the other work in efficacy, is that a lot of the work is comparing lecture. So you don’t do anything to this active learning technique. And with cooperative learning, the students are working on exercises. And when we talked about it, we said you know, it’s really not fair, and not even interesting to compare a student who is in a lecture-based class where they don’t get to see any of these problems to students in a cooperative learning class that are working on these problems in class in teams. So we thought a lot about the collaborative learning approach. And we’re trying to isolate what we thought could be the mechanism that might be driving different outcomes. And what we decided was we were going to compare the active team-based learning to individual learning, working on problems. So in our control, the students were exposed to the same problems, they had the same amount of time in class to work the problems as in the treatment, which was the collaborative learning. It’s just that in the treatment, they were working in teams in this think-pair-share share framework, and we didn’t find a significant difference. So at the end, we said that the cooperative component didn’t seem to be driving any difference. Although there are other people who’ve done work comparing collaborative learning to lecture based and they did find significant positive effects from cooperative learning.

John: One type of thing that I’ve done that seems to have been fairly successful is using clicker questions. Following the methodology of Eric Mazur, students are given a challenging question where typically half or so them will get it right the first time when they’re asked individually. But then they get to talk it over with the people around them, preferably someone with a different answer, they get to debate and argue it and I normally see between a 10 and 20 percentage point increase in the correct answer. And the really absurd answers tend to disappear down to about nothing. So that’s not a very formal study, but that second stage of that process where students are engaged in some peer instruction seems to have had a pretty significant impact. And I point that out to students, that when they talk to each other about it and explain it to each other, they do much better than when they’re working on their own. And I encourage them to try doing that outside of class, to work with other students, because there are a variety of studies… I haven’t seen too many in economics, but there are quite a few studies that find that that type of peer instruction, either in the classroom or outside, can be fairly effective.

Tisha: Yeah, so I’ve observed that too. I haven’t used clicker questions, but just going in and observing my colleagues when they teach, and they are using clicker questions. And I have to say, I’m stunned, because I don’t know how the right answer bubbles up as opposed to the person who had the wrong answer convincing the other of the wrong answer. I don’t know how that doesn’t happen. It’s like magic to me, almost. So part of me wants to think a bit more about what is the mechanism and how is that working? That’d be really cool to do a study on that.

Rebecca: One of the things that’s always curious to me about that particular dynamic is sometimes students discover, as they try to explain something to someone else, [LAUGHTER] that they really don’t know what they’re talking about. [LAUGHTER] They realize, huh, I can’t actually explain this concept, so maybe I’m the one that’s not correct. [LAUGHTER]

Tisha: Maybe that is what’s happening. Because everyone always says, “You have to really know your stuff to teach it,” and so if you are trying to explain it to your partner, you do have to know the material.

John: And when they’re raising objections and you don’t have a counter argument, that can help break down your misconceptions, because getting rid of those incorrect perceptions can be as important as trying to build new ones.

Tisha: Yeah, I could see that and maybe the people who are initially giving the wrong answe are just guessing. So maybe they’re not trying to explain and convince their partner of their answer, I guess. I don’t know. So maybe it’s not magic.

Rebecca: [LAUGHTER] Feels like magic.

John: It does. And it works.

Tisha: When I see it happen, it does seem like magic. In general, I think that active learning is helpful, because I think that women are going to be more interested when you show them the applications, and that when you have active learning, there’s some community building that happens. So you’re illustrating relevance, you’re building this community, giving them a sense of belonging in the classroom, which is a lot of what I think actually happens with the experiments. And then actually also with the experiments, they’re kind of discovering some of the theory for themselves, which is a growth mindset. And there is research that shows that if you can show relevance, belonging, and growth mindset, and develop these in your classroom, that students are going to do better, but that it also is more appealing to females and underrepresented minorities.

Rebecca: There’s lots of things for all of us to think about, even if we’re not necessarily in economics classrooms. Definitely good food for thought. Switching gears a little bit. Can you tell us a little bit about your work on the Committee on Economic Education?

Tisha: So I’m not officially on it at the moment, I’m the incoming chair. But I have served on it in the past. And I’ve been shadowing KimMarie McGoldrick, who is the current chair for the last year. So the committee is doing a lot of really great work. I think. When I first served on the committee, we basically just organized sessions for the American Economic Association meetings. And when I say “just” I don’t mean “just…” Compared to now, it’s just but we have, I think, seven sessions that we organize for the meetings every January. But while I was first on the committee, then chair Mike Watts, who was at Purdue, he said,”I have an idea. We should have a conference, so an all economic education conference,” and we started it while he was chair, and it’s CTREE, so the Conference on Teaching and Research in Economic Education. So the committee is in charge of that. It’s every end of May, beginning of June, depending how the calendar falls. So our next CTREE will be in Atlanta, not that far. It’s on the East Coast, maybe you can join us. And it will be May 29 through the 31st of 2024. We have many other things that we also are overseeing: an educate program, which is professional development for college faculty, and that includes two-year college faculty, in course design, different active learning pedagogies, and we talk about how to bring diversity and inclusion into the classroom. KimMarie actually started this this year, we have a newsletter called EconEdNews. And that features different pedagogies, it features the winner of the AEA Distinguished Economic Educator Award, which the committee also oversees, and some centers that we have for Econ Ed across the country… just a lot of information that’s in there, we have two issues a year. So we’ve had our first two, and our next one will come out in March of ‘24. So just a lot of great work, I think, that the committee is doing and I think really thanks to Mike who started us with our own conference. And then Sam Allgood succeeded Mike and added and then KimMarie is an overachiever. And she added a lot more in her six years as the chair.

John: And going back earlier, early surveys of economic instruction suggested that economists were using much more chalk and talk than many other disciplines were using. So these efforts have been fairly important in shifting people away from that. It’s a somewhat overdue change, but it’s nice to see that happening. And I would love to go to CTREE. But there’s two barriers that I’ve run into one is we run a series of workshops here, right around the same time, and the other is I teach at Duke in the summers. So I either am doing workshops here, or I am in North Carolina teaching econometrics now down there, so I haven’t been able to get there. I would love to go to a CTREE meeting.

Rebecca: I think I would have no clue what’s going on if I went. [LAUGHTER]

John: Oh I think you would.

Tisha: I think you would, I think you’d have a nice time. Everyone there is just so lovely and very engaged in teaching, and just really cares about the enterprise.

Rebecca: It’s nice to see education groups out of these professional organizations to really formulate communities of practice around teaching.

Tisha: Yes, I agree. I think economics, unfortunately, was late to the game. But we’re increasingly appreciating the importance and the importance not just of focusing on four-year colleges, but two-year colleges. So going back to the point of diversity, and the lack thereof in the discipline, the students in four-year colleges that I’ve worked with have predominantly still been very much what economists already looked like. And if we really want to improve the diversity in economics, I think we need to reach down to the high schools to the two-year colleges, and make economics more interesting and accessible.

Rebecca: We always wrap up by asking: “What’s next?”

Tisha: There’s lots of things. I guess, with regard to the committee, I’m hoping to just kind of not start anything new in my first term, and keep everything running as well as KimMarie has it now. But then for my own research, I am really interested now in looking at comparing pedagogies. So it’s been a lot of efficacy work looking at a particular pedagogy compared to sort of the standard chalk and talk. But now, how do they compare to each other? Is there a better pedagogy for students? And my first study, and I have the data, so I just have to start working on it will compare experiments to cooperative learning. And I told KimMarie, if experiments lose, then she won’t hear any more about this. I won’t write it up. I won’t talk about it. Because I don’t want to say that experiments lose, but no, I mean, seriously, that’s the next thing that’s first on the agenda. But also, I have a deep interest in diversity questions. KimMarie and I with Scott Simkins have a paper looking at HBCUs compared to PWIs and the study of economics there. And that was inspired by some work that showed that HBCUs contributed disproportionately to the STEM pipeline. And since economics is part of STEM, we thought, “Well, that should be true for economics as well.” So we did some work in that area. And we show that there are some positive contributions from HBCUs to the economic pipeline. But I think there’s a lot more to do there. So I would like to dig down in that. I’ve heard a lot of people at HBCUs talk about secret sauce, and I want to learn what that is. And not just at HBCUs but look at MSIs as well, more generally. And I do want to look at some questions at two-year colleges, so that’s for the next five to 10 years…

Rebecca: You’re going to be busy. [LAUGHTER]

John: For those who aren’t familiar with the terms PWI are predominantly white institutions and MSI are minority serving institutions.

Tisha: That’s correct.

John: Those terms are becoming more and more common, but just to make sure all of our listeners are familiar. Okay, well, thank you. It’s great talking to you, and it’s nice meeting you. And we will keep trying to get KimMarie McGoldrick on the podcast. I’ve been asking her for about four years. She’s been a little resistant, but we’ll see if we can get her here in the future. Well, thank you.

Tisha: Thank you so much. I hope you have a good afternoon.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

282. Moving the Needle

The study techniques that most college students adopt do not align with what research tells us about how we learn. In this episode, Sheela Vermu and Adrienne Williams join us to discuss what happens when an instructor in a community college biology class attempts to encourage students to adopt evidence-based study methods. Sheela is a biologist at Waubonsee Community College in Sugar Grove. Illinois. Adrienne is a biologist at the University of California, Irvine.

Show Notes

Transcript

John: The study techniques that most college students adopt do not align with what research tells us about how we learn. In this episode, we discuss what happened when an instructor in a community college biology class attempts to encourage students to adopt evidence-based study methods.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Sheela Vermu and Adrienne Williams. Sheela is a biologist at Waubonsee Community College in Sugar Grove. Illinois. Adrienne is a biologist at the University of California, Irvine. They are co-authors of a study entitled “Moving the Needle: Evidence of an Effective Study Strategy Intervention in a Community College Biology Course.” Welcome, Sheela and Adrienne.

Sheela: Thank you.

Adrienne: Thank you.

John: Today’s teas are:… Are you either of you drinking tea?

Sheela: Yes, I am.

Adrienne: We heard that was a thing.

John: And what type of tea?

Sheela: I’m actually drinking my favorite tea. It’s called A Sama tea. It’s a calm relaxed, lavender, rose, chamomile and cardamom.

Rebecca: You just describing it just took my blood pressure down. It sounds very relaxing. [LAUGHTER]

Sheela: I was doing it for myself. [LAUGHTER]

Rebecca: Well, you helped me too. [LAUGHTER]

Adrienne: I can’t compete with that. But I have a classic Trader Joe’s pomegranate white tea.

John: Very nice.

Rebecca: I think, John, today was my closest call of not having tea. Because when I got home to record about an hour ago, we had no water at my house. The water was off, but it’s on now. And I have a Scottish afternoon tea.

John: And I have a black raspberry green tea, which I haven’t had for a while but it’s nice.

Rebecca: Yeah, that is one that you haven’t had in a while.

John: I had it last time when I was in quarantine at home, [LAUGHTER] which is where I am now with COVID.

Rebecca: It’s like you’ve established quarantine routines. [LAUGHTER]

John: Unfortunately, yes, this is my second time. We’ve invited you here today to discuss the study that you both worked on. One of the things that you noted in the study itself is that there’s relatively few studies of student study strategies at the community college level. Even though community colleges provide an introductory college experience to over half of all the students that graduate with a bachelor’s degree in STEM fields. Could you give us a brief overview of your study?

Sheela: Thank you, John, for asking that question. Because it sort of made me think about what really brought about my interest in studying this topic. It stemmed from me becoming a community college Bio Insights Fellow, and Insights is a network of community college instructors who are actually interested in investigating teaching and learning in their classroom. So when I became a fellow in that network, one of the goals was for us to think about what are some teaching practices that could inform the scholarship of teaching and learning in a community college setting. And I teach microbiology and anatomy and physiology at Waubonsee Community College. And while I was starting to teach, I realized that students really struggled to study effectively. And community colleges occupy a very important position in higher ed, especially in STEM because they provide low cost training and education for workforce training, preparation for transfer, and also recently an opportunity to reskill for many of our underemployed and our underserved students and population. So keeping that in mind and our classroom structure, we noticed that in the biology education field, papers or authorships for community college faculty, in a CC context was very few, only 1- 3% of all the biology education research articles had a community college context question or a community college. So this network sort of enriched me to think about what do I need to ask a question in the classroom that would encourage my students to use better study strategies? So in some ways, I wanted to ask this question, but it was the Insights that helped me think about this from the scholarship of teaching purposes. So the brief overview of what we see as the basic study of this paper, is we were really wanting to ask this question, what kind of study strategies students are using in a community college context right now in a biology classroom that I teach, that was one… and can an intervention of some kind from an instructor really intentionally encourage students to reflect upon their current study strategies and guide them in some ways to change their strategies to ones that have been shown through research to have high impact. So it was to just gauge the field, but also to see if we can gently intentionally guide that providing guiding practices. And that’s sort of the big picture of what we did. But we administered a pre- and a post- survey. The pre- and post- survey was taken from a previously published work. The survey actually asked questions about the study habits and the study strategies that students actually used. The intervention was administered. And one of the things that is really important is I was able to get approval from our college institutional effectiveness team, and those are the ones that serve as the institutional review board and they look at the studies research paper and also helped assist with gathering institutional data, because institutional data was very meaningful to our study. And Adrienne helped arrange the statistical power. That is what made a big difference in the study. And she also helped me wrap it up and write it up and get it published in the CBE Life Science Education journal.

Rebecca: So a lot of students enter college and plan to acquire a STEM degree but often change their mind or change their plans. Which students are disproportionately likely to give up on their planned STEM degree.

Adrienne: In general, regardless of institutions, students have a difficult time with a STEM degree. They tend to give lower grades and that can be very discouraging for students, even if they’ve done well in high school… when they get to college, and they start to not get A’s can be very discouraging. So there is a history for all undergraduates, when they enter even a four-year or a two-year program that they start to leave STEM. They find it less rewarding than they had thought, I would say community colleges have additional hurdles to overcome, because they are an open access system. And so they get students with a wide variety of past experiences. Some are people who’ve just come out of high school and are used to studying and are pretty on top of things… they remember their math, they’re accustomed to memorizing things. And then there are students that are adult learners, or perhaps have families, been in the military, perhaps they get a GED and have worked for awhile. And so there’s a lot of habits to relearn. That can cause problems, particularly in STEM where the grading is just historically rather harsh.

Sheela: I would agree with what you said about historically STEM attrition is pretty problematic. But the problem is also more exaggerated in marginalized and minoritized students who come from backgrounds that could have been a first generation, some kinds of a financial issues, students who probably did not have a whole lot of high school curriculum that prepared them well for a STEM field. And there’s been recent work in 2016, that talked about how the National Academies needed to look at improving underrepresented minority students persistence in STEM, not just entry into STEM, but for them to be able to stay in the pipeline, and successfully move on and build a career trajectory for themselves.

John: We saw similar results in an earlier podcast interview with Peter Arcidiacono from Duke. He was a co-author of a study that looked at the determinants of students’ continuation in STEM fields and found something very similar: that, holding other factors constant, females and students from historically minoritized groups were much more likely to change out of the STEM fields than students who were white males, even when they were doing relatively better in these classes, then the students who chose to remain in the discipline,

Adrienne: STEM classes, particularly, are not welcoming to many students, because the exams are difficult. Now, there’s just the culture that we’ve developed, not necessarily for good reasons but it does cause many students who are doing fine, who are scoring above the mean in the class, to feel like they’re not succeeding, because they’re only getting 70-80%. And that’s just kind of an unfortunate reality that we’re working to change. But in the meantime, we would like students to be as successful as possible on their exams.

Sheela: And recently, there’s been some studies that talk about some concrete steps to diversify the scientific workforce. And that came out last year in Science, and that talked about how students in the STEM field sometimes do get discouraged, and often feel really compartmentalized by the climate and some of the teaching methods and assessment, exactly what Adrienne was talking about in the STEM classroom. And there’s been an exodus of students, specifically from the underrepresented minority communities.

Rebecca: Can you talk a little bit about some of the study techniques that students were using before your intervention as part of your study?

Sheela: Yeah, so when I walked into the classroom, what I noticed were that students were typically cramming the night before the exam, or perhaps two days before the exam, and they would use one study session to just get all the notes that they could for that particular unit. The other thing that I noticed students doing were, they would be using a lot of flashcards, they would walk into the classroom with a whole bag of flashcards and sometimes it would all drop and they would have color-coded flashcards, different colored highlighters. And as the classes were going on, as I would see them studying, I would see them using most of these methods which was underlining key words in the textbook or terms or in the PowerPoint and in In some ways, creating some flashcards, and massing all of this study as they thought it was in one particular session, and then just go to the exams. And I saw them a little bit frustrated thinking that they had put in so many hours into it, and not having the results that they would have expected out of those exams.

John: Could you tell us a little bit about what study techniques you recommended to students and how you convey that information to students?

Sheela: So one of the key things that I did for myself was just to get familiarized with what were some high-impact study strategies, because this was not something that I was very familiar with. And one of the things I figured out was it was distributed spacing. And in the STEM fields, especially in the subjects that I teach, like microbiology, anatomy, and physiology, we wanted students to know one concept well, before they moved on to the second concept. So if they tried to do all of this studying the night before the exam, it’s very possible that they missed out one of the pieces in the Jenga, and the whole Jenga fell apart the day of the exam. So, something that I talked about was spacing, but also making sure that spacing is not just happening two days before, but it is distributed throughout the unit that we are actually covering. And the second high-impact practice that I asked students to think about was self testing. Self testing sometimes can be difficult for students to understand, especially at our community college setting, because the kinds of self testing that they’ve been exposed to in high school, is a teacher giving them a set of practice problems, or they having 50-60 questions in a test bank, and just going over those, and in some ways, just memorizing or flash carding and making it into a Quizlet. And they would think that is self testing. So I had to go back and walk them around and say that self testing would be to self test a concept. And to double check and see, do you really understand that concept by looking at a problem or a question, which is at the end of the chapter, before going on into the next particular concept? The other piece that is also very important that high impact practices that I talked about to the students was diagramming. And this was something that I wanted them to get away from, just trying to just reread chapters, condensing chapters, and summarizing chapters. I wanted them to get away from those kinds of practices and move on to something called diagramming.

John: So what did you find in terms of the use of self testing and spaced practice, as well as the drawing applications? Was there a significant change in the student use of these techniques over the course of the semester?

Sheela: So, in the beginning, there was some resistance from the students. Because this was something difficult for them to do, they had never heard of these kinds of strategies. And as we proceeded with the intervention, which was periodic, we found that students, over the period of the entire semester, improved their spacing, so they went from high levels of cramming to high levels of spacing. We also found that self testing, which was very low in the beginning, also improved at the end of the course. Students were not creating their own questions, but instead, they were using the questions that were at the end of the textbook that were desirable for comprehension. They were using questions that were problem-solving oriented, which were part of the assessment to test themselves and diagramming also improved by the time the semester finished.

Adrienne: Let me also say something that we noticed was students were willing to give up some strategies very easily. I do similar work at my large four-year R1 institution, and students also arrive in their first quarter at UC Irvine and they’re very fond of flashcards, and tend to also be fond of underlining and highlighting. And those are two study strategies that students see kind of quickly just don’t work. The exam questions that they get in college are just not associated with things that they did on flashcards and so they’re like, “Okay, boom,” and they drop them with very little resistance. Every year I check, every year, they come in high flashcards and they leave low flashcards. They’re like, “Nope, that didn’t work.” And they’re happy to drop that. Other study strategies like rereading their notes: they come in high rereading their notes, and we tell them, that won’t help. Yes, you have to understand the concepts, but stop rereading your notes, that is not an effective study strategy. And that one doesn’t move. They’re like, “Nope, still gotta reread my notes.” And so that’s been interesting for us. I talked to Sheela about that. I talked to students about that. And so some of these strategies are easier to move than others. And I think what’s going on… we’re going to foreshadow some metacognition here in students choosing what study strategies to use. They feel like when they start their study session, they have to decide what is it that I’m going to work on and so they reread their notes as a way to set themselves up with what needs to happen in the study strategy time they have coming up, but it just takes so long to reread all their notes that they know half their times gone by the time they finished doing the setup activity. And so they’re unable to do the more useful things of self testing, explaining to others, doing things from memory, painful things. It’s also just more encouraging and soothing to look at your notes and go “Yep, yep, I remember that. Yep. Okay, yep, I remember that.” And that feels like studying, even though not really.

John: That’s sometimes referred to as fluency illusion, that the notes are familiar, you see them, you remember the organization on the page, and it’s reassuring, even though it adds virtually nothing to your ability to either recall concepts or to transfer those concepts into different applications. But it’s hard for people to give that up, especially when they’ve been told to reread their notes and reread the materials over and over again, throughout all their prior educational experience.

Adrienne: Yeah. And it worked great for him in high school. So why would they believe us until they try it and find out that it doesn’t work? So some strategies are easy.

Rebecca: So much easier when your answers are in your notes, and not from learning?

Adrienne: Well, the ironic thing is, in my classes, I’m now all open note, open Google. And so they still think the notes will be the answer to all their problems. Now, it’s all application problems.and so you can’t find all that. So it’s hard, it’s difficult for students to change, even if they know like, one of the new things I’ve started to ask is, what do you think researchers say is more effective: spacing your studying or cramming? And all the students know that spacing is more effective. And then you ask, and what did you do for this test? And they’re like, “Well, I had to cram.” So a lot of it is not lack of understanding of what should happen, but just the difficulties associated with it, just like eating right and exercise, the difficulties with actually doing the difficult work.

Rebecca: I think often students mentioned that it’s difficult to plan their time, or to have the time to do the thing that they know is good for them.

Adrienne: Which is why we’re going to again, come back to metacognition, I think, that this talk about how we’re kind of moving forward out of just asking students, how are they studying, but thinking more broadly about metacognition.

John: Before we get to that, though, could faculty reduce the incentive to cram by using more low-stakes activities so that students don’t have that incentive to cram before a high-stakes exam and ignore studying the rest of the time.

Sheela: So John, in the courses that I am teaching, we have not just one high-stakes exam, we have many small unit exams. And these unit exams come every two to three weeks. But the material in STEM, as we speak, is so dense for the students that they have to move from lower order Bloom’s… just remembering terminology… all the way to concept analysis to an explanation very quickly, in a very brief period of time. And I think even having those low-stakes assignments is not enough for them. Because those assignments, they may not choose to use the high impact study strategies, they may get away by looking at a summary, or maybe looking at the notes in the low stakes. But when they come to these unit exams, even if it’s just two chapters, I found students in my college, in my classroom, really struggling, even if the low-stakes assignments were done at a 95% completion, which is what led me to think of this study and say, what is it that I can do as an instructor, I would love to change all the dynamics of higher ed, and move things seamlessly to make it a beautiful world for everyone. But I can only control what I can do in the classroom, which is the intervention. And I think a persuasive intervention through modular use over a period of time, which is consistent, and short and brief, is a possibility that faculty could use in order to shift students’ practices of study strategies. And for them to be cognitively aware that this is a good study strategy. I’m actually aware of it. And that’s the knowledge I have from the literature, which is what Adrienne was talking about. But then how do you implement it in the moment and modify just a little bit so that you can actually get good grades in that unit exam that’s coming? Because there are five of them, you just can’t afford to blow each one of them. You have to just get gooder and gooder if there is a word like that. [LAUGHTER]

John: You mentioned working to improve students’ metacognitive skills. Could you talk a little bit about how you built in metacognition into this approach.

Sheela: So when we did the intervention, my intervention was very simple. It was a PowerPoint presentation, and it was a PowerPoint presentation before and after each exam. And we have five big exams in that course. And we had an exam wrapper as well. So the PowerPoint presentation was not just this is what you need to do. It was things from the literature of high-impact study strategies, and also being aware of what is a low-impact study strategy. And the PowerPoint presentation was 15 minutes long, it had about 15 to 16 slides. But there were some examples of how to use those strategies and how not to use them. For example, if they were looking at muscle contraction, and if they were looking at how the skeletal muscles, students know that it’s a bicep, how does the bicep actually contract instead of just making 100 flashcards of every piece of that information, which is in that unit? How can we translate that into concept, and learn each concept and see how that moves into the next concept. For example, when we think about this, you think of a motor neuron that actually stimulates a skeletal muscle like a bicep, and there are multiple segments in this piece, the neuron has to send a signal, which is an action potential, so they need to know what’s an action potential. And they also need to know the structure of the neuron. So there are two big pieces there, then it sort of travels into a terminal, and it causes some channels to open, they need to know the kind of ions that actually travel through those channels, and what causes those ions to travel from A to B. And then through that influx of those ions, they need to know there is a release mechanism. And that release mechanism causes another set of ions to open postsynaptically in the muscle. And from there, they move on into understanding how it contracts the muscle tissue or cell. So there’s a sequential activity that goes on. And for students to be able to compartmentalize that and get good at understanding each concept before they move on to the next is something that my intervention was part of. And for them not to just make all of that into a highlighting flashcard, summarizing it into three sentences and say, the neuron travels and the bicep contracts. And in the middle, there is a gap. And that’s called a synapse. And that’s what they would do when they summarize. And that’s not effective, because the questions are not summary driven. The questions are application driven. What would you do when a drug blocks that particular channel? What would happen upstream? What would happen downstream? And now they’re like, “I never thought about this much detail.” So that was the intervention. But then there was also an exam wrapper. The exam wrapper, it was more of a reflective piece. I just thought it would give students an opportunity after every exam to self reflect and see what study strategies they used. How many hours did they spend studying? What would they like to change for the next exam? And I thought it was reflective, I gathered a lot of paper and I gathered a lot of data, it’s sort of gave a quiet moment for our students to reflect on exams. But talking with Adrienne and working on this for a little while, realized that the research on exam wrappers does not show… it’s not efficacious enough to change students on how they learn. So now I’m doing something a little bit different where I’m not just doing an exam wrapper and calling it as a check in point on Canvas for them to sort of reflect after the exam, but not just using exam as the main tool, which is what shifted us to think about the next steps, which is the metacognition, which Adrienne was talking about.

Rebecca: Adrienne, do you want to share some of the details about the metacognition side of it here?

Adrienne: Yeah, I do research on my students. And it’s just been fascinating how students seemed to choose to do strategies which didn’t seem helpful. And so in additional reading that we’ve done, Sheela and I have learned more about metacognition. And you can kind of break down metacognition, in knowing what good strategies are… like, do students know that spacing and retrieval practice and interleaving? And diagramming? Do they know that those are considered the good strategies? Second, do they have the metacognitive skills to use them at the right places, or even though they know spacing is good, they never actually set up time to do the studying days in advance. And then thirdly, metacognitive judgment, do they have a sense for yes, they now know this information, can they judge their learning? Is their appropriate judgment of learning that goes on? And so all of these are steps that we realized, all we knew was which strategy students chose. We didn’t know if they had the knowledge about which ones were good ones. And we didn’t know whether or not they recognize whether or not they were successful. And so we’ve expanded our questions to students: do you know which strategies are successful? …and pretty much they do. That doesn’t seem to be the missing link. It’s the scraping together, the organization time to actually apply them early enough that they can spend the time appropriately. And we’re also trying to determine can we help students? Give them regular feedback? The exam wrappers didn’t seem to work really well. There wasn’t a lot of evidence that they were really doing a lot of changing of student behavior. So, something I’m trying this quarter, particularly in my anatomy class, is to ask students: “If I said different study strategies were worth different amounts. I said, rereading your notes is only worth 0.2 points per hour and I said, explaining concepts to a neighbor or drawing diagrams by hand of the different systems and that’s worth 1.5 points per hour, will you change how you study in order to maximize your studying points? And so we’re in the middle of that right now to see is there anything that can motivate students to attempt to try new, difficult, painful, complicated things, other than the comforting things of rereading and rewatching videos? That’s a bigger metacog picture I’m working on, Sheela’s also doing that, kind of increasing our focus away from just single study strategies to this metacognitive view, do students have the metacognitive chops? And is there anything that we can do as instructors to help them with these applications? Because once you’ve told them that spacing is important, they’re like, ”Alright, spacing is important.” But that’s not solving the problem much. It’s an important initial first step, but are they actually going to figure out how can I schedule an hour a night on my anatomy, so that I have some chances to forget and relearning, and so that relearning really kind of nails down those mental pathways, and so it’s easier for them to do the work for application problems when they get to an exam? So we’re enjoying just trying different things as we move forward, to see can we expand our understanding of metacognition. help students understand? Can they tell us what they understand and don’t understand? And can they tell us what they really struggle with in the application so that we can help reward them for doing good studying? Happy to do that.

Rebecca: Have you tried sharing a study plan for the week just to see if they followed it to see if it worked?

Sheela: Yes. And I think that’s sort of the third part of metacognition, that Adrienne was just talking about. The knowledge is one, which is the strategies that they know, and whether they can actually apply those strategies when they are thinking or listening or reading difficult tasks. That’s very important, because you can apply these strategies when it’s in your comfort zone. It is the application of these high-impact strategies when you’re out of your comfort zone, and that is really important. And that’s where the planning comes in: having a calendar and making sure you plan it, and you’re putting in the hours. But what I found in my explanations with my students is that even though they planned the time, I had students who had devoted an X number of hours, they just didn’t know when they got to the muddiest point that they were even muddled. [LAUGHTER] And that is, to me, metacognitive judgment, right? You know, that you don’t know, and you know, that you need to do something that you don’t know, and students were like, they didn’t even know they were muddled. And so they didn’t go ahead and use the appropriate strategies, or change directions, or make some adjustments one week before the exam so that they can actually monitor their learning, which is where we are working on building the set of skills, and just making sure we ask our students and see if we can shift their practices to more of a judgment outside of just planning the time. Planning the time is very important, but it’s not just planning the time. What do you do with that time? And we couldn’t be with them all the time, to sort of shift it.

Adrienne: You’re probably familiar with the idea of high-structure courses, having many assessments. And I think that is really important. It’s a lot more work for the instructor to be building all of these assessments and managing them. But the more cases where you can help students get feedback on whether or not they’re learning successfully, the more likely they’re going to realize that the thing they thought would be fine is not actually working.

John: You talked a bit about ways in which we might be able to better improve student’s understanding of effective study strategies. But did you find any impacts of the techniques that were used during this experiment? Did it make a difference for some of the students?

Sheela: The drawing did. To me, the way I was thinking of drawing was, I was thinking of it as a visual representation of a science process. That’s how I was thinking of it. But the drawing was very meaningful to the students, because students before in my class, were sketching typically, or taking a diagram or a figure from a textbook. Because most science textbooks have beautiful colored large figures, they will just take that figure and translate it into a sketch and just draw some diagrams and draw some arrows and point to some facts. But drawing the process was very important. And not only drawing the process as a flowchart, but actually organizing the conceptual information and connecting the dots. So in some ways, the way I was using drawing representation was more like a concept mapping. But I also realized that students without a lot of encouragement on how to do these concept maps or how to do these representational drawings, were not getting a lot of feedback, because they were just drawing and maybe they’re drawing it beautifully. But they were not really using that drawing to really understand and self test themselves on some of the key concepts. So this semester, what I’m actually doing is asking for our students to show some of their drawings and upload it on Canvas. And I have two or three criteria before it’s accepted as a drawing. And one would be to make sure that they have some basic notes that they have talked about, some key concepts. And then they have also asked themselves a question, which would be like a feedback, like a retrieval question using that drawing. So they’re using the drawing as a practice. And they’re using the drawing as a self-retrieval practice to ask a question. And I’m hoping that it would have some change. But the literature in the drawing area is not very clear. And I would like to use it as an if and then statement, maybe, like, you know, if this happens to the sodium, and what happens to the action potential, or maybe like a causal effect kind of statement, or sometimes maybe even say, why and how, why does this happen? And how do you think you can make it better?

Adrienne: It really does take time and scaffolding for students to be effective at something like this. You can’t just assign them a drawing or tell students you should draw more when you study. But taking the time in class to assign something, give them appropriate scaffolding, give them feedback, show them what you were imagining it would look like, asking them a test question that should have been easy to answer if the diagram had been appropriately written. And just training students, we tend to do a lot of assumption that somebody before us did the training, and that’s just not appropriate in many cases. It is really helpful to make space during class time, either by flipping the class or by flipping 10 minutes of the class, moving that outside to pre-class work so that you have time in class to train students how to think carefully and study effectively so they can be more successful going forward.

Rebecca: I think one of the themes that I’m hearing both of you point to is helping students prioritize things, there might be a sequence to knowing or what might be most important versus kind of an extraneous detail that’s not as important until you have the big thing figured out. And a lot of times when you’re new to something, these are not obvious things. But when we model how to make a diagram and verbalize how we made a decision, or how we chose what might go in a diagram, it can be really helpful and enlightening to students because they’re seeing how that thought process might work. If they’ve not experienced it for themselves.

Adrienne: It’s not like students aren’t working hard. I think we tend to think that somehow students that are doing badly in the class are slacking, and that is often not the case, they’re working very hard. But if you’re spending hours making flashcards, that’s just not as useful a thing to spend time on

Rebecca: …for hours going down a rabbit hole that’s like not actually important… [LAUGHTER] which I sometimes have had conversations with my students like “You spent how long on that? Yeah, that’s not something to worry about. Maybe you should do this other thing instead. [LAUGHTER]”

Sheela: Yeah, and color coding. And our textbooks in the sciences have a lot of colors. And the textbooks also have a lot of highlights. So in some ways, I feel that some of these high-impact study strategies are not very, very clearly explained in our textbooks. So when they see the highlighted word in a textbook, or they see the color-coded diagrams, the students often believe that that’s the secret. In one of our exams recently, I asked them to bring that color-coded diagram and see if that helps them answer the questions. And they were surprised that they could only answer 50% of the questions on the exam with that color-coded diagram. The rest had to be some kind of retrieval practice, some kind of higher-order thinking which they had not spent the time doing, because there was spending more time drawing that thing out, like a sketch.

Rebecca: …and probably essentially just copying whatever they saw. [LAUGHTER]

Sheela: and just making it look prettier. That’s it.

Rebecca: Yeah. [LAUGHTER]

Adrienne: The good strategies are painful, they’re difficult and painful, and expose everything you don’t know. And they feel like you don’t understand things if you’re doing the good work. And that’s a difficult thing to fight through. Nobody likes to feel like they don’t understand.

John: I think we all face that challenge with our students.

Adrienne: Um hmm.

John: They like strategies where they could sit and passively listen to someone explain concepts to them. But that doesn’t really seem to help them remember it, but it feels good. It feels like they’re understanding every step of it. But when you’re trying something in active learning, and you’re getting feedback that tells you that “No, you don’t quite have that,” it’s certainly not as pleasant. And there’s no real way around that, other than perhaps reassuring students that that’s an important part of learning. I’ve been trying to explain that to students for years without complete success, because they much prefer to listen to a lecture or to read a book and assume that they know things until the professor tricks them with these questions that clearly are not a fair measure of their learning, and it’s a difficult cycle to break.

Adrienne: True for all humans. We like to minimize the calories we use in everything.

Rebecca: Is this a line of research you’re hoping to continue on?

Sheela: Yes, we are, as we speak, to fine tune what we learned from the study last year. By the time the study finishes and the time the paper comes out a whole long time has gone by. So that was a great time to sort of think about what are some areas and it was the exam wrappers that sort of prompted me to think about this metacognition, especially metacognitive judgment, because exam wrappers are self reported, they’re sort of reflective, and I found that students were reflecting the same thing, like I spent only 5%, doing X, it hasn’t changed from unit exam to the next unit exam. Then the question became, “Why is it that they were not able to monitor their studying, or make those adjustments as they moved forward?” So maybe for an exam on tissues, they spent 10 hours studying a certain way, but probably when they come to the nervous system, they probably need to modify that or adjust it, but students were not able to do that, based on the difficulty of the task. They were just steamrolling it. They were just doing the same thing over and over again, they said, Oh, “You told me to study X number of blocks, and you told me to study 12 hours and I’m just going to do those 12.” And that’s prompted us to think about the next steps. And we are asking this question in a community college classroom and see what kind of metacognitive skills students have. And we are dividing those skills into three parts, which is knowledge of the skill, are you aware of it, can you actually apply it, judgment would be can you apply it and monitor and change it when needed based on what’s happening in the moment, and then sort of plan and have a control on your self-regulated strategies? I think this is an uphill task, because the data that you get could be a little messy, just like most education research, but I think we’ll just have to continue and plod through it just like how we did the other one, before we get some kind of a baseline that suggests that whatever we are doing, whatever intervention we are planning to have in the classroom, has an effect. So I think of myself as a practitioner, Adrienne is also a practitioner, but she’s also a researcher. So for me, if it doesn’t make sense in the classroom, I’m very happy that it did make awesome sense in the world of research, but I would like it to really make sense in the classroom because I want to see our students benefit, move forward and have great STEM careers, however it may be.

Adrienne: I’m also doing projects. I’m continuing to work with Sheela on adding some metacognitive aspects to her class. A couple of things that I am trying to work on is an implementation strategy where I try to have students, each week, think back on the week that they’ve just completed, and how did they study on that week and so just regularly get feedback and to overtly tell them each week: these are high-impact strategies, these are low-impact, which ones did you use and what do you plan to do next? And a lot of hitting over the head, perhaps. But it takes a lot to change this for them. And secondly, I’m particularly interested in students who are studying with friends. I saw an interesting effect in my intro bio class a year ago, where students who really valued studying with friends were doing worse in the class. And so I’m attempting to figure out what’s going on there, because that was kind of non-intuitive to me. And I think it has something to do with some students really value studying with friends, because it’s an opportunity to have their friends explain the material to them. So what I’ve been asking students this past year was what do you do when you study with friends, they’re like, figure out what they’re actually spending their time doing. And some students are spending a lot of times explaining and others are spending time getting explained to and there’s different relationships in how they do in the course. That is pretty clear, though students that are in study groups in order to learn the material, that’s an indication they’ve got other struggles going on. They need help. So things like that, I’m still figuring out the best way to ask students and to figure out where the pain points are so that I, as an instructor, can say, “Alright, if you find that you like study groups because you really need somebody to help explain things to you, that is a sign that we need to help you understand the material more effectively, and get you extra help.” So both research and being a good practitioner as Sheela says.

Rebecca: I think one of the things that we don’t always recognize when we’re introducing students to strategies that are really new to them is that they’re learning that in addition to the course content at the same time, so there’s an extra layer of things that need to be learned and we need to do spaced practice and things on that, too. [LAUGHTER] Start building some habits and remember that that’s a thing that they’re also learning and remind them that that’s also a thing that they’re learning and that accounts for some time. It takes time to adopt a new set of practices. It takes time to plan [LAUGHTER] or whatever.

Adrienne: And I like to give students a small amount of points are doing it, because it’s part of the work of learning and I want to reward that, and frankly, they need the points.

Sheela: sAnd some study trategies are also discipline based. So it’s very difficult for our students, especially when they’re freshmen and sophomore, and they are in three or four different types of classes. Some are STEM-based, some are non-STEM-based, maybe their non-STEM faculties guiding them to read and create some kind of a graph or some kind of a writing narrative. And here in our STEM class, I’m saying, don’t just spend your time reading and rereading, while the other faculty is sort of giving them a different point of view.

Adrienne: They have a lot of bosses.

Sheela: Yes. And that can be very difficult for them, because they’re like, “What do I do? My other teachers telling me this, and this teacher is telling me this, and I don’t know, and I’ll just do what I know to do, which is flashcarding,”

John: We always end with the question: “What’s next?”

Sheela: To me, what I would like to do is continue this kind of education research, which is classroom based, that also has an efficacy for student improvement. These three are very important to me. But I also have an overall goal because of the CC Bio Insights network that started me to think about biology education research, is to make sure that these kinds of questions that are community college centric, asking questions that are based from a community college classroom, are also being part of the education research and part of the biology education journals published so that people can actually see what’s going on in these classrooms, and perhaps build some credibility to the work that’s done in this area. So that would be sort of my big picture, giving back to the overall community of community college and how it affects higher ed.

Adrienne: That’s great, Sheela. Yeah, what’s next is always iterations on improving my teaching, new projects for research. But I thought working with Sheela on a project like this was super helpful, both for my career, because it benefits me to publish… that’s a really important part of my job… and I have a community of people with statistical skills and experience publishing, that if I can bring that to a partnership with a community college faculty person who has access to community college students, that’s a important connection that the community college students can benefit from the research finds for them, the faculty member in the community college isn’t overwhelmed attempting to learn a whole bunch of additional publishing skills that they don’t need for their career advancement, but they still want the message to get out that I can carry some of that burden in a way that benefits me. So it’s a win-win for me, for Sheela

Rebecca: Well, thank you so much for joining us. And we look forward to hearing what else you learn.

Adrienne: Thank you so much for having us.

John: And this is really important work that you’re doing because we lose so many students from the STEM fields, and the students we’re losing are the students who could gain the most if they were to be successful in the STEM fields because the rate of return to a degree in the STEM fields is so much higher than it is everywhere else and we face some serious shortages in these areas. So it’s an area in which the research could be really beneficial to a lot of people.

Adrienne:: We’ll do what we can.

Sheela: Yes, keep marching along and carry more with us.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

212. Faculty Mindset

Research on the impact of mindset has often centered on the mindset of the student. In this episode, Elizabeth Canning joins us to discuss the impact that faculty mindset has on student achievement. Elizabeth is an Assistant Professor in the Department of Psychology at Washington State University. Her research focuses on how to create equitable and inclusive instructional environments.

Shownotes

Transcript

John: Research on the impact of mindset has often centered on the mindset of the student. In this episode, we discuss the impact faculty mindset has on student achievement.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners..

[MUSIC]

John: Our guest today is Elizabeth Canning. Elizabeth is an Assistant Professor in the Department of Psychology at Washington State University. Her research focuses on how to create equitable and inclusive instructional environments. Welcome, Elizabeth.

Elizabeth: Thank you.

Rebecca: Today’s teas are… Elizabeth, are you drinking tea?

Elizabeth: I’m drinking water today.

Rebecca: Alright, still a good choice. The base of tea, of course.

Elizabeth: Yeah. [LAUGHTER]

John: And I have a restricted set of tea because I brought most of my tea back up to the office, but I’ve been sent home with COVID. So I do have a ginger peach black tea still here though.

Rebecca: That sounds like a pretty standard fare.

John: It is.

Rebecca: I made it out to that tea shop I’ve discovered. And so I have a new one called Yunnan Jig…

Elizabeth: Ooh!

Rebecca: …and it’s a golden-tipped black tea.

Elizabeth: That sounds delicious.

Rebecca: It is very delicious.

Elizabeth: Exotic teas.

John: We’ve invited you here today to talk about your 2019 study that examined the effect of instructor mindset on racial achievement gaps in STEM disciplines. Could you tell us a little bit about this study?

Elizabeth: Yeah. So this study was a study that we did while I was a postdoc at Indiana. It was in collaboration with Dr. Mary Murphy, Katie Muenks, and Dorraine Green. We were really interested in instructor beliefs about intelligence… so whether they believe that intelligence is something that is innate, something that you’re just born with, you can’t change it very much, we call that a fixed mindset. And what we also call a growth mindset is this belief that intelligence can grow over time and change with effort, strategies, help-seeking, things like that. And we were interested in whether instructors’ beliefs about intelligence then predicted the experiences that students have in their classes, and then how well they do in their classes. So we sent a survey out to the whole university, and we didn’t expect a whole lot of responses because faculty are busy and all of that. But we actually got a lot of responses. We had 150 STEM faculty respond to our survey. And so with that, we were able to link their survey responses to the grades in their courses over a two-year period. And we ended up with this fantastic rich data set that we were able to look at students’ grades in their courses.

Rebecca: How prevalent was the fixed mindset amongst the faculty that you surveyed?

Elizabeth: Yeah. So it was pretty normally distributed, shockingly so I think. A lot of people think that fixed mindsets might be more rare. Other people think fixed mindset might be extremely prevalent, but it was pretty normally distributed. It’s a continuous scale so if you graph it, it’s a pretty normal curve. So we’ve got the most people in the middle. And then we’ve got some people at each extreme. But what’s interesting when you look at it, though, is especially on this scale, most people, they won’t go extreme, like the very, very tip of the fixed mindset scale. But they’ll do the next level over. It’s like, “I’m not gonna be that person, but I’ll be right next to it.” So it’s interesting to think about that when you think about the profession of being an instructor, and part of your job is to educate people. But yet we’re finding a significant amount of people have this mindset that intelligence or abilities can’t change.

Rebecca: So you talked about it being a normal distribution, was there any variation amongst race or gender, ethnicity, age, or STEM discipline?

Elizabeth: Yeah. So we get that question a lot. Like, can we predict who it is that has the fixed mindset? Well, this is the only table I’ve ever published that had every single line in it non-significant. [LAUGHTER] So we tested pretty much everything in our data set—whether mindset differed by gender, by faculty race, by age, by teaching experience, tenure status, anything that we had in our data set—and we found no differences by mindset. So it seems like having a fixed mindset or having a growth mindset is prevalent among all faculty regardless of these characteristics. We also looked within departments because a lot of people want to think that it’s, “Oh, it’s those economists or…” [LAUGHTER] looking at you, John. Or, “It’s those computer scientists, or the physicists, or the mathematicians.” But we didn’t find any differences by discipline. So faculty in any kind of discipline can endorse a fixed mindset or a growth mindset. And this suggests that these mindsets are not going to resolve itself on its own. Like if this was a generational thing, then we would expect it to resolve over time, or we could go into a specific discipline and educate faculty, but it seems to be pretty widespread, regardless of these characteristics.

John: That was one of the most surprising things in your study, to me. I was expecting that this would vary, particularly with age, but also perhaps with gender as well. And I was thinking that maybe this would be better over time. One of my favorite quotes from Paul Samuelson, an economist who died a while back, was, “funeral by funeral, the science makes progress,” and I was kind of hoping that that might occur with growth mindsets here, too. But it doesn’t look like that’s going to happen.

Rebecca: So we’re left with saying, “Great…” [LAUGHTER]

John: When you were looking at the effect of instructor mindset on student grades, were you also controlling for the characteristics of the students as well?

Elizabeth: Yes. So we controlled for a number of things in our analyses, we wanted to put anything in the model that might predict student performance. So we controlled for whether the student was the first in their family to go to college, their first-generation status. We controlled for their prior achievement level, so their prior GPA before they enrolled in that class. We also controlled for a number of course characteristics. So whether the course was, at this university, a 100 level, 200, 300. So, like, an entry-level class versus more of an upper-division class or upper-level class. We controlled for a number of faculty characteristics as well, like whether they were tenured, their age, how long they had been teaching. And so all of that was in the model to control for those variations in terms of what predicts their grades. Because a number of things we know predict grades, like class size. If it’s a bigger class versus a smaller class, we know that’s a pretty robust finding. And so we controlled for that in all of the analyses.

John: What was the overall effect, controlling for all the other student and instructor characteristics, of instructor mindset on student grades?

Elizabeth: So students, on average, received a higher grade in faculty’s classes where they endorsed more of a growth mindset. And this was, again, controlling for all of those things, regardless of student characteristics, class characteristics, and faculty characteristics. We also looked at this by student race, so we found an interaction with student race. So it’s not just that everybody on average is receiving or earning higher grades in the growth-mindset courses. This is particularly true for students with racial- ethnic minority status. So Black, Hispanic, Native American students performed better in the courses that had faculty who endorsed more of a growth mindset. When we look at the achievement gap between White and Asian students compared to Black, Hispanic, and Native American students, we see that this racial achievement gap is twice as large when the faculty endorsed more of a fixed mindset, compared to when they endorsed more of a growth mindset.

Rebecca: Can we talk a little bit about differences in assessments or the way that courses are structured between the fixed-mindset faculty courses versus the growth-mindset courses, because I think your paper talked a little bit about that as well, right, Elizabeth?

Elizabeth: We couldn’t dig into the specifics of it just because of the scale of this project, but we’ve done a lot of follow-up to see: What is it that faculty are doing in these classes? Because students are obviously picking up on it. It’s not just the belief that you hold near and dear to your heart, this is something that is being communicated to students in some way. And what we know from our other research is that it’s communicated in a lot of ways depending on the instructor, depending on the class. So it can be communicated in your course policies, like how you design your syllabus, how many assessments you give in your class. So, fixed-mindset professors are more likely to have a midterm and a final, and that’s your only opportunity to display your abilities in that class. Whereas the growth-mindset professors are more likely to have weekly quizzes where you can improve over time and see that improvement over time. And mistakes are less deadly in those classes, so to speak. So it’s in the way they design their courses, but it’s also in subtle ways. So what they say in class, how they talk to students who are struggling in their office hours, it’s in their attitude, it’s in a lot of different behavior. And students are pretty perceptive, they can pick up on it pretty quickly.

John: So we can significantly reduce racial achievement gaps if instructors have growth mindsets. Is instructor mindset something that’s changeable?

Elizabeth: Yeah, I think so. We know a lot from trying to change students’ mindset beliefs, it’s actually pretty malleable. You can teach people the science behind how our abilities grow over time, the changes that occur in your brain in terms of neuroplasticity, and faculty are pretty receptive to that. Most faculty want to do things that are going to benefit their students, most faculty want to be good instructors. And so it might just be finding ways to communicate that. In some of our research, we’ve found a disconnect between what faculty think their mindset is and how they’re communicating that, to what students are actually perceiving. And so it might just be communication, making sure that you’re very explicit about what you believe in your class, standing up on the first day and saying, “Here’s what I believe about abilities and intelligence, you don’t have to be, quote-unquote, ‘smart’ to do well in this class. Here are other ways to do well in this class, like learning and improving and using different strategies and things like that.” We also know that there are very critical times where these messages matter more. So there are times during the semester where students are searching for information about their abilities. So when you hand back that first exam grade, or that first assessment that you give in your class, that’s the perfect time to communicate your mindset beliefs, particularly if they’re a growth mindset. Because students are searching, “What does this mean? Does this mean that I’m good at this? Does this mean that I’m not good at this? Am I going to do well, should I drop this course?” It’s a time of uncertainty for students. And so, for faculty to communicate those beliefs during that critical time, it can set forth sort of the snowball effect for how they should view their learning and improvement throughout the semester. There’s a number of ways that faculty can do this in their classes. But back to your original question about, “Can we change faculties’ mindset?” I think so. I think there’s a lot of literature suggesting that we can do that.

Rebecca: And there’s been a lot of money pumped into making these STEM pipelines in the first place. We want them to be effective. So investing in this education around mindset and learning might be a really good use of funds.

Elizabeth: Yeah, one of the things that I think was really shocking about this paper is the faculty that reported their mindset beliefs, there were 150 of them. But when you look at all of the students that they touch, over a two-year period, how many people they teach, it becomes a really big number pretty quickly. So in this sample it was around 15,000 students that these 150 faculty taught over a two-year period. And so, instead of intervening with 15,000 students, you might intervene with 150 people and see similar or maybe even greater effect.

Rebecca: I know from our experience, John, with working with faculty around mindset and around helping students learn how they learn, that faculty who demonstrate a growth mindset are often very willing to share what they’ve learned about learning with their students. And so having an intervention with 150 people then reaches many of those students, because that information ends up being communicated out in a more distributed way.

John: One thing I’m wondering is whether you can separate out, in these results, the impacts of the way in which people teach from the messaging that’s coming up indirectly in other ways? It strikes me that that may not be possible, in that the instructors with a growth mindset provide lots of opportunities for students to make mistakes and learn from those mistakes, while those who have a fixed mindset are more likely just, as you said, to use a small number of exams as high-stakes assessments. Is it possible that some of the effect is just from better teaching practices and using more evidence-based teaching approaches that give students these opportunities for more retrieval practice, more spaced practice, and so forth?

Elizabeth: I think that’s very possible. However, we’ve done some follow-up research where we randomized faculty messages within one course. So it’s the same professor, it’s the same structure, some students are getting growth-mindset messages from that instructor, some students are just getting control messages, and we’re seeing really great effects at that level. And so it has to be more than just the way their course is structured. It’s more about the messages that they’re giving students and how to frame mistakes, how to frame ability. We’re providing it at a specific time like I mentioned earlier, and so it’s probably a combination of both. But with this new experimental evidence at the student level, or at the classroom level, we’re seeing that it’s more than just their teaching ability, or the way that their course is structured.

John: What you just described reminds me of a podcast we had done a while back with Angela Bauer at High Point University in Episode 49. In that episode, she talked about trying to reduce some of the achievement gap in their introductory biology classes. And they first introduced some active-learning activities, but there was still a non-trivial achievement gap remaining. So they introduced some growth mindset messaging, and that seemed to remove the remaining racial achievement gaps. So that provides a little bit further evidence that growth mindset messaging can play a significant role in helping to reduce these achievement gaps.

Rebecca: One thing that’s really powerful about that idea, though, is that it may not really take a lot to make a change.

Elizabeth: Yeah.

Rebecca: It doesn’t really seem like that much of a financial investment, or even a time investment to make a difference, or at least chip away at the problem a little.

Elizabeth: I think that’s what makes this so appealing to people is because it’s subtle, and it’s also something that can be done pretty easily. So you’re not changing the curriculum, you’re not flipping your classroom that requires extra work and time. Faculty are already overloaded with the expectations of what they’re supposed to be doing with teaching in terms of all their other responsibilities. We have implemented mindset messages at a pretty really basic level. So putting messages in a syllabus, putting messages in an email, maybe a couple videos in the class, for instance. And that’s really it. As long as it’s done in a way where it’s at a critical time, it’s more meaningful for students, and it’s done sincerely, then it’s not a whole lot of extra work.

John: From a faculty member’s side though, for those who may have a fixed mindset who believe that students’ ability is fixed, a conversion to a growth mindset may very well, and that’s consistent with your results, change the way in which they structure their courses. Because if you believe that students can learn by making mistakes and practice, you’re probably going to redesign your courses to build more of that in and that’s, again, very consistent with what you found. And it will be a bit more work typically for instructors unless they can do it in a way in which there’s some degree of automation.

Elizabeth: Yeah, I think there’s a lot of discussion around rigor, and if you have a growth mindset that it means you’re too soft, and you’re not having a difficult class. So it brings up all these questions about course difficulty and rigor. And I think our perspective and some of the follow-up work that we’ve done in this paper, and in other papers, that the perspective of what is difficult or what is rigorous really depends on who you ask. So if you ask a faculty member how difficult their class is, or what it means to have a difficult class, that’s going to vary quite dramatically from the students’ perspective. So what we’ve seen is that students actually find the growth-mindset professors to be a little bit more difficult or challenging than the fixed-mindset professors, and it’s for that very reason that they have more work to do in the class. [LAUGHTER] They have to make improvements, and they have to redo assignments, and the workload is maybe even a little bit higher, versus a fixed mindset class that might have a midterm or a final, there’s less, quote-unquote, “work” to do in that class. And so it really is in the eye of the beholder what class is difficult and what that means in terms of student achievement.

Rebecca: There’s a difference in regular accountability…

Elizabeth: Yeah. [LAUGHTER]

Rebecca: …and therefore feels like a lot more work when you’re being held accountable on a regular basis. [LAUGHTER]

Elizabeth: Yeah, I’ve done some follow-up work, because a lot of people will also think that it just comes down to being nice. If you’re nice and warm and friendly, then people are going to do well in your class, and if you’re not, then people will not do well in your class. And so I’ve done an experiment where we have manipulated that. There’s a laboratory study that we’ve done, where we manipulate whether the message is a fixed-mindset or a growth-mindset message. And then we manipulate whether that message is delivered in a way that’s warm and friendly and positive, versus cold and unfriendly. So this results in four different ways of messaging. The best is always going to be sort of this warm, friendly growth mindset, and the worst is always going to be the cold, unfriendly fixed-mindset. But what’s really interesting is what happens to the two in the middle. If you have the growth mindset but you’re cold and unapproachable, versus a fixed mindset and warm and approachable, How do those two shake out? And what we’re finding is that, very consistently, it’s the mindset message that matters more than the warmth or approachability. So they do interact, of course, it’s good to be warm and friendly. But the message itself also matters. So we can tease these things apart. They are different things, and they have different effects.

John: Since we’ve got you here, we were looking through some of your other research, and you’ve done a number of studies that have looked at the impact of utility-value interventions. Could you talk about what those are, and what you’ve found involving those?

Elizabeth: Yeah. So a lot of the research I’ve done on utility-value interventions was conducted in collaboration with Dr. Judy Harackiewicz at Wisconsin, Madison. And so, utility-value intervention is an intervention that’s directed at students. The way that we implement it is we have students write a short essay. They pick a topic that they’ve covered in class, and they write about how that topic is useful or relevant to them personally. And the way that we’ve done this, we’ve had them do it a couple different times throughout the semester. So they’re reflecting on what they’re learning and they’re connecting it to their own personal goals, their life, and it makes the material that they’re learning more relevant to them. And what we’ve found is that this intervention is particularly beneficial for underrepresented racial minority students who are also the first in their family to go to college. So this group is doubly disadvantaged, they’re disadvantaged due to race, they’re also disadvantaged due to social class. These students have the largest achievement gap in most STEM courses, and they really connect with this intervention. What we found in this research, these students have a particular motivation for going into science. They want to give back to their communities, they want to help their friends and family out after they’re done with college. They have specific goals that may not be met in science courses, or may not have this direct connection to what they’re learning in science. And so, by providing them with this opportunity to reflect on that and connect the material to those goals, we’re seeing that they make marked improvement in terms of their grades in the class. Whenever you do an intervention where you have students write something, and you get really rich data because you can look at what they’re writing about. So we analyzed over 1,000 different essays that these students wrote and we said, “What are they writing about?” We ran them through the linguistic analysis, and they’re really connecting it to those goals. These students are more engaged in the assignment, they’re writing longer essays, they’re more specific in their writing. And that then contributes to learning more in the class, which results in higher grades.

Rebecca: Sounds like, again, a very easy intervention to make or to build into classes. I know that I’ve been doing more of that in the classes that I’m teaching and seeing really good results and having really great conversations as a result with students as well.

Elizabeth: Yeah, I think there’s a difference between simplicity and ease of this type of intervention, because it does require the students to put in some effort, they have to write an essay. It also requires some grading on the part of the instructor if that’s not already in your class. So it is a little bit extra, quote-unquote, “work”. But I think it can be done in automated ways to benefit students.

John: And in these studies, you’ve also looked at the effect of student-identified value versus when it’s communicated from the instructor instead. Could you talk a little bit about the relative impact of directly-communicated utility-value interventions from the instructor or those that come from the students?

Elizabeth: Yeah. So from a really practical perspective, we wanted to know, “Do students have to write the essay? [LAUGHTER] Do we have to grade all these essays? Or can we just stand up in front of the students and sort of give them this information, kind of summarize how what they’re learning is relevant to a number of domains?” Because that would ultimately, in a practical sense, take less time. So we did an experiment where we manipulated this, and what we found is that students benefit the most when they get both. So they have a little bit of scaffolding from the faculty member, where they’re given some ideas of how the information might be relevant or useful to them. And then they write about it in their own words and get really into specifics. So a professor can stand up there and say a bunch of ways that it might be relevant, but every student is unique, every student has different goals, every student has different interests. And so, it really needs to be personalized to them. And the process of putting it in your own words and reflecting on it is also useful, right? That’s part of the learning process. You get into the specifics of it and write about it. So, we ended up concluding that it’s both. There’s some scaffolding involved from the faculty member, but then the students really need to generate something for themselves too.

John: I think you also looked at this in terms of the differential effect in two-year and four-year institutions, and you found somewhat different results between a community college and a four-year institution. Could you talk a little bit about that difference?

Elizabeth: Yeah. So a lot of this research has been done, or the data have been collected from four-year institutions, particularly four-year research-intensive institutions. And so, I wanted to see how this could translate to other types of colleges, particularly in the two-year context, because a lot of first-generation students go to two-year colleges. That’s a gateway to a lot of different career paths. And so what we did is we went around to a whole bunch of different two-year colleges, we connected with the instructors there, we tried to tailor it for their students and their context. But ultimately, what we found is that there needed to be a lot more scaffolding in terms of the writing process. So because the intervention was done with students at four-year colleges, we kind of made assumptions about how ready students were to think about utility and think about relevance. And a writing intervention in those contexts just wasn’t appropriate. So we didn’t find the same findings, we actually found that the control essays were more beneficial than the utility-value condition, in these interventions that control is summarizing course material. And that was actually really beneficial for students in this context because they weren’t already doing that in these courses. Whereas in the four-year college, that was sort of the status quo, and they were able to take that next step to make that course material relevant to them. So in working with their instructors, what we concluded is that you can do this intervention in a different way. It doesn’t necessarily have to be writing. It could be done in small group discussions, it can be done in presentations, it can be done in a lot of different formats that might not present a barrier of writing ability or writing practice. A lot of students in two-year colleges take concurrent writing courses in addition to their science courses. And so removing that barrier of writing, I think, would have been necessary in that context.

Rebecca: It’s a nice helpful reminder, I think, for faculty to be thinking about ways to have reflective practice that doesn’t always involve a lot of reading too. [LAUGHTER] Like, if we’re doing presentations, or if we’re reflecting in a video, or reflecting in conversation. These are all other places that provide some variety, too, so that we’re not always grading the same things or having to intake the same kinds of information. That can also be overwhelming to faculty too. So mixing it up is helpful I think. [LAUGHTER]

Elizabeth: Yeah, I think it also speaks to, you know, one-size-fits-all interventions just are not appropriate. So we publish these papers and we say, “Wait, we found these really amazing findings.” But that’s in one context, maybe with one instructor, maybe at one institution, and every student body is different, every class is different. And so, you really need to figure out the needs of your students and meet them where they are, and also take a step back and look at the purpose of the intervention. Maybe it can be implemented differently. Maybe if you take the philosophy of it and customize it for your context, that’s going to be the more appropriate approach.

John: We always end with the question, “What’s next?”

Elizabeth: Well, right now I’m still working on mindset messages. We just got word that we might potentially be getting a nice grant to look at this. And so what we’re going to be looking at is growth-mindset interventions directed at students crossed with growth mindset messages from faculty. The idea is that when you try to get students to believe in a growth mindset, it’s going to be most effective when the environment is supportive of that message so that it’s supported by the faculty member in that class. So we’re going to be looking at that over the next few years in a bunch of different contexts, in a bunch of different institutions.

Rebecca: Sounds like more beneficial, useful, and exciting information.

John: And again, as Rebecca had said earlier, these are really relatively simple and easy-to-use interventions that I think could be much more widely adopted. Well, thank you.

Elizabeth: Yes, thank you for having me. It was great to talk with you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Anna Croyle.

[MUSIC]

211. What Inclusive Instructors Do

Our students bring a rich diversity in their life experiences, skills, and prior knowledge to our classrooms. In this episode, Tracie Marcella Addy, Derek Dube, Khadijah A. Mitchell, and Mallory E. SoRelle join us to discuss how we can create inclusive classroom communities in which student diversity is treated as an asset and where all students feel a sense of belonging. Tracie, Derek, Khadijah, and Mallory are the authors of What Inclusive Do: Principles and Practices for Excellence in College Teaching.

Shownotes

Transcript

John: Our students bring a rich diversity in their life experiences, skills, and prior knowledge to our classrooms. In this episode, we discuss how we can create inclusive classroom communities in which student diversity is treated as an asset and where all students feel a sense of belonging.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners..

[MUSIC]

John: Our guests today are the authors of What Inclusive Instructors Do: Principles and Practices for Excellence in College Teaching. Could you each introduce yourselves to our listeners?

Tracie: Absolutely, my name is Tracie Addy and I’m the Associate Dean of Teaching and Learning at Lafayette College in Easton, Pennsylvania.

Derek: Hello, I’m Derek Dube. I’m an Associate Professor of Biology and the Director for the Center for Student Research and Creative Activity at the University of St. Joseph in Connecticut.

Mallory: I’m Mallory SoRelle, and I’m an Assistant Professor of Public Policy at the Sanford School of Public Policy at Duke University.

Khadijah: Hello, my name is Khadijah Mitchell. I am the Peter d’Aubermont Scholar of Health and Life Sciences and Assistant Professor of Biology at Lafayette College.

Rebecca: Today’s teas are… Tracie, are you drinking any tea today?

Tracie: Yes, I’m actually drinking Twinings peppermint tea. I like peppermint.

Rebecca: Yum!

Derek: Today I am just drinking your standard run-of-the-mill tap water.

Rebecca: Always a good option.

Mallory: I’ve got some green tea with lemongrass and mint today because I needed a little kick of caffeine

Rebecca: A mint team around here.

Khadijah: Well, I really don’t need to be drinking caffeine. [LAUGHTER] So I am drinking AHA sparkling water. It’s orange and grapefruit.

John: And I am drinking Twinings mixed berry black tea, because I need a bigger kick of caffeine.

Rebecca: I got here late and didn’t have time to make tea, and it’s really hot, and so I have a glass of water. And this is the first time I’ve ever not had tea for Tea for Teaching. But this is a very inclusive crowd, so I know it’s going to be okay.

John: Could you tell us a little bit about the origin of this book project?

Tracie: Yes, I can share about that. So, we were very interested in a lot of different research questions around inclusive teaching, for example: What predicts whether instructors adopt inclusive teaching? What are the barriers that they face? As well as, what can we do to kind of move this forward at institutions? So initially, we were very kind of research-minded, and we noticed that there were other questions that we could explore. Also, in our study, that I know later one of my co-authors will talk about in more depth. And those questions were, “What do inclusive instructors do?” So we ended up collecting a lot of really interesting information about the practices of inclusive instructors. And so that led us to think… Wow, wouldn’t it be wonderful to put this all together into a beautiful story that included the voices of instructors, that included instructors across disciplines, across institution types, across ranks, etc., and put it together in a guide that would really be practical, that would help instructors really think about inclusive teaching in a very practical way? So that essentially initiated this project. And I invited my co-authors who are joining today to partake with me in this project to write the book, and I thought of each of them for very specific reasons. And I value, very much so, their contributions and what they did around inclusion. And we kind of put it all together, and we worked together on this great work. Now, this is also coupled with more studies, some of which have been published as well, that kind of get into this big picture, thinking about inclusive teaching, thinking about… What do we do? How do we do it? And then even further, How do we actually enact it? What are the barriers we face? And how do we overcome or address those barriers?

Rebecca: Can you talk a little bit about what inclusive teaching is, and why it’s important, to kick off our conversation today?

Tracie: Yes and I think it’s so important to define our terms here so that everybody starts off on the same page. So when we talk about inclusive teaching, and especially in the book What Inclusive Instructors Do, we’re talking about teaching that is creating a classroom environment that’s welcoming, so students feel a sense of belonging to the actual classroom setting. And we’re also talking about, that it’s equitable, and it’s thinking about the diversity of learners, and it’s very responsive to that diversity in the classroom. So we’re kind of joining here, this idea of belonging, as well as this idea of equity together, and all the practices, which are many, that we can actually use in our classrooms to be inclusive. And with regards to, “Why is it important?”… inclusive teaching has always been important. Inclusive teaching is excellence in teaching. We publish this book now, but this has historically always been a critical area to think about in teaching and learning. And some of the reasons why… well first, there’s a history of exclusion at institutions of higher education, some are able to be educated and have these experiences, and some are not. And there’s also a lot of good research around thinking about that and belonging. There’s clear research that ties belonging to academic achievement, it ties it to wellbeing for students, and many other important things that we know are really important for students’ success in college. Also, we teach diverse students in many of our institutions. So it really behooves us to really think about that and that diversity. And so it’s important now, it’s always been important. I know, with all of the things happening in our nation, there have been more calls and more attention towards inclusion and equity. But I will say, as I’ve said already, that it’s always been important to actually have environments in our classrooms that students feel as if they belong. We know that that’s a place where students can feel excluded.

John: You also conducted a survey of faculty about inclusive teaching practices. Could you tell us a little bit about the survey that you used?

Derek: Sure. So I’m happy to share a bit about that. Now, as Tracie had mentioned, there’s four of us that worked as co-authors on this book, and we all have different experiences, and backgrounds, and expertises, and roles at our institution. But we didn’t want this book to be just our voices and our four experiences, we wanted it to be much more than that. So with that in mind, we dove into the literature around inclusive teaching—what’s published, what’s the research out there—but really to figure out what’s going on right now and what are inclusive instructors doing, we wanted to have as broad a swath as possible. So working together we created a national survey on inclusive teaching, an inclusive teaching questionnaire, and we shared this both directly to various institutions of various different rank and style, master’s institutions, doctoral institutions, community college liberal arts institutions. We also connected with listservs, and social media, and directly with instructors as ways to share this out. This survey was given for about a month to two months in early 2019. And in the end, we ended up having about 566 participants that had started the survey, over 300 of which reached the end of the survey, and over 200 of which responded to all of the questions that we asked. And it was really interesting, because when we looked at the demographics and the backgrounds of those who responded, we saw a wide range of individuals from different types of institutions, male and female, various backgrounds, various disciplines, whether they were tenure track or not, and also the fields that they worked in. So we really felt like we got a good feel of a variety of different instructors being able to speak to what inclusive teaching means to them, what they’re doing, and how they see it at their institution. Also, geographically, we had respondents from the northeast, from the southeast, from the northwest, from the southwest, and everywhere in between, which was really nice to see as well. So when we did this survey, as Tracie mentioned, we had a few different things that we wanted to know some of them were… What are instructive pedagogies? What are inclusive instructors doing right now? But also, What are barriers they’re facing? What experiences for training have they had? How confident do they feel in their own ability to teach in equitable and inclusive ways? So all of these things were pieces of information that we were able to get from this broader swath and bring in and really pull in and really allow, in a lot of ways, those voices to be the voice of the book.

John: So a very inclusive approach to developing a book on inclusive teaching.

Derek: That was the idea, yeah.

Rebecca: In your first chapter of the book, you suggest that faculty should treat student diversity as an asset rather than employing deficit models that we definitely have experienced in our own educations and perhaps in our institutions. Can you describe ways that faculty can convey this message to students through their instructional practices and actually take advantage of these assets?

Mallory: Sure thing, that’s a great question. So, the idea that we should approach differences in background, experience, personality, skill sets, as an asset to the learning environment, something that improves the learning environment instead of a challenge to be overcome or an obstacle we have to deal with, was one of the most, I think, significant themes that comes out both in the scholarship around inclusive teaching, but also in the words of the folks in our survey. And a lot of examples came out in people’s responses about how they go about doing this in practice. And that begins with course design and syllabus with things like incorporating diverse perspectives in the material you’re assigning in class to demonstrate the value of these different perspectives. It comes from incorporating welcoming statements in a syllabus that explicitly state the value of multiple perspectives in the classroom and devising participatory strategies that are designed to bring those out. It also includes trying to build assignments that take an asset-based approach. I’ll give you an example of one, in a group project where you ask students to identify some of their different strengths: Are you good at researching? Are you good at writing? Are you good at editing? Are you good at presenting? And putting groups together that assemble students who identified different strengths and having them talk about those. The idea that not all students have to be good at every one thing, that we all bring these different strengths to the table. And one of the things that I think is really important for this asset-based approach is knowing something about what those assets are in your classroom. And that requires knowing something about who is in your classroom. So one of the things we also talk about in the book that I think is a good tool for helping to treat diversity as an asset set in the classroom is what we call a “Who’s in Class?” form, which is a form that can be given anonymously to students at the beginning of the semester, to just help and identify what are some of the social identities in the classroom, some of the skills people bring to the classroom, some of the different perspectives that students are bringing to the classroom, to give instructors more of a sense of what that diversity is, and how that can be used over the course of the semester to really improve the learning experience for everyone.

Rebecca: So I’m curious, with a survey like that to learn who’s in the classroom, are those results something that we should be sharing out to students and having a conversation about?

Mallory: Yeah, so I’ll take that and also open it up to Tracie, because she’s done a lot of work in the development of this form. I think the goal is to distribute this, allow anyone who wants to participate anonymously to participate, and then, yes, to share back the aggregate takeaways to the class, because it lets other students know who else is in this class with them. And particularly, I think, for students who might feel like there’s something they’re bringing to the table that maybe they don’t know other students are also bringing to the table. It’s a way of saying, like, “Look, there are lots of folks who are both like you and lots of folks who are not like you. And that’s going to be something that’s going to help us throughout the class this semester.”

Tracie: And I guess I’ll piggyback on that a bit. And I will say, you definitely can share it with your class. I think the important thing is letting your students also know that in aggregate, we will be sharing this. And also, if there’s certain things on there that really does it make sense to share with everybody? …having that discretion too. Because students will share lots of different things on that form, and some of it can be used to introduce this conversation, like Mallory said, and to really think about the diversity of the class. And I know also Khadijah has done things of that nature, she’s actually used the form in her class and done things like that, and has had a lot of positive feedback from students, with that regard. Derek might have done too, I’m not sure, but… [LAUGHTER] I know Khadijah has voiced that to me as well. So I think it’s a good opportunity to really think about who’s in class and a safer way for students… students will often feel more comfortable sharing in that type of format than just asking them without that kind of anonymity tied to it.

Derek: And I can actually just chime in a little bit here too. One of the ideas that Mallory brought up, and then Tracie added to, was getting to know your students. It’s really hard to teach your students in a meaningful and inclusive way if you don’t know who your students are. So finding ways to do that, especially early on in a course—really early, the earlier the better—was really important to us. And that’s where the “Who’s in Class?” form was born. It was born as a way to instead of waiting for, “Okay, I’m going to meet and learn my students throughout the semester, maybe get to know them more at the end with evaluations and things like that,” …what can we do right away? And because the students may not necessarily know us right away, or what our intentions are, we thought that the “Who’s in Class?” form could be most powerful as an anonymous and aggregated way of collecting data. Where students could feel safe, that their privacy was protected, that they could share that information that they wanted the instructor to know, but maybe didn’t want them to know about them in specific. So that’s why we moved that way. Now, in thinking about getting to know your students and being able to really, in a directed way, be inclusive and equitable and support different students with different needs, we do believe that moving from anonymous to a more non-anonymous way of getting that information can be important in a lot of situations. But we think that it’s best when it’s student-directed, when the students decide that they’re comfortable to share that information with the instructor, that’s the time when it’s most likely most appropriate. The “Who’s in Class?” form can be a way to ease into sharing information in a safe way. And then you come, you talk to your class about, “Why did we do the ‘Who’s in Class?’ form? What did we learn in aggregate?” And then you open up and say, “I’m here to extend these conversations, to continue these discussions. I have office hours that are open that you’re welcome to come to and talk to me if there’s any specific thing here that you want me to know that directly relates to you.” I know that Khadijah, at least, has, in some of her courses, used situations where there’s essentially mandatory office hours, I think right in the beginning, like little meet-and-greets where it’s only 5 minutes or 10 minutes, but you’re going to come in and you’re going to meet and you’re going to have an opportunity to talk. And you can share what you want to during that time, but you’re going to get that face-to-face time. And maybe she can talk about that more in a moment or two. But other things that I’ve done, if you have large classes where maybe there’s not a ton of time to have individual meetings with every student, in a lot of my classes, one of the first assignments is an online discussion board using our learning management system, which in my case is Blackboard, where students make a post about themselves and some information about not only them academically, but also their hobbies or interests. They post a picture either of themselves or something that represents themselves. And then there’s an opportunity and encouragement for students to reply in meaningful ways to each other, to get to know each other, because it’s not just about the instructor knowing who’s in the class, but it’s about the class knowing who’s in the class too, for it to be the most positive experience. So that’s been really beneficial. And I as an instructor then take time, and I can do it at nine o’clock after my kids are in bed, to make sure that I respond to each student in a meaningful way and try and make connections where I can, “Oh, you like science fiction, well I’m currently reading this series, we should talk about that sometime,” or things along that line. So I think that starting in a safe, anonymous way like the “Who’s in Class?” form can be a great way to get that ball rolling and, if the students feel comfortable and feel like it would be meaningful, allow them to break that anonymity border by offering opportunity.

John: We’ve been running a reading group along with SUNY Plattsburgh, and this was a topic that was discussed really extensively in one of our meetings, where there was pretty much a consensus that there’s a purpose for both an anonymous form to let people express things that they might not be comfortable revealing, but then also giving students the opportunity to share either with just the instructor, perhaps through meetings, or if it’s a larger class, a discussion forum, or Flipgrid, or VoiceThread, or some other way where they can share their identities with other people. And I think the consensus was, there’s a good purpose for each of these, and some combination might be really helpful.

Khadijah: One thing I just want to add on to what everyone is saying is that the “Who’s in Class?” form has been transformative for my classroom spaces. And I know Derek brought up something about large class size and thinking about large classes, it even can help with that. But I think we also need to think about the other end of the bell curve, very small classes, because even though someone may be not identified, there’s some aspects of their identity that could then disclose who they are. So I think that we also need to be mindful of that. For example, clearly there are visible aspects of our identity that would be able to disclose what a particular student was in a small setting, that would not be as much of an issue with a large setting. But I do think that there is so much power in that. And speaking to what Derek mentioned about the essential office hours, so for every class that I teach, I do use the “Who’s in Class?” form and these essential office hours. And even though the “Who’s in Class?” form is anonymous, people do share with me during these essential office hours, and it really fosters a greater classroom environment in that way.

Rebecca: I love the name essential office hours, I love the emphasis on the “essential.”

Tracie: Absolutely. And I was going to share that the development of the “Who’s in Class?” form was with collaboration with students too. So I asked a number of students about this form, as we were going through the process of creating it, from questions like, “Are these questions that we should ask? How should we implement it or administer this? Would they answer these questions?” And so that was also very helpful. But I will say that working with a number of instructors on the “Who’s in Class?” form in my center, there are a number that actually do have a separate form as well that’s course specific, that’s not anonymous, they add additional questions on that. And then we have all these wonderful variations that, like Khadijah said, the essential office hours and other ways to get to know students, which I think, John, well you mentioned, I think is obviously fabulous. There’s all these different avenues for students to be able to share aspects of themselves with not only the instructor, but as Derek mentioned, with the class. What a wonderful thing that is for building a more inclusive classroom.

John: Once you have this data on who’s in your class, how can you use that to convince students that the diversity of the class is actually an asset to the class? What sort of methods could you use to help convey that message? In particular, how can you avoid issues such as stereotype threat?

Khadijah: Well, I can speak to the first part of your question, John, I think about: what can you do with this data? So I actually summarize the data, and we have a little PowerPoint presentation, and I share that back out to the class so that we appreciate this diversity. I also then go tweak and tailor my classroom to the students that are in the room. So if there are particular issues that may be salient to that group and that population, then we address that as a learning community together. Thinking about stereotype threat, so this is really important, particularly in the discipline that I’m in, in STEM disciplines. So when we think about stereotype threat, we normally think about negative stereotype threat. And that’s the perceived risk of confirming negative stereotypes about a particular social group that a student may be assigned. And what it leads to is this imbalance of how the student’s sense of self, which is typically positive, versus this inconsistent expectation of whatever group that they fall into. And so this is really, really pronounced when we think about various academic disciplines, and notably people who’ve done work in STEM. And what happens is this leads to worse academic performance or a threatened or less of a sense of belonging. So things like the “Who’s in Class?” form that can help with that sense of belonging. I think that there are several evidence-based approaches that we use to mitigate this impact in effect. And the first is really thinking about self-affirmation. So there are a lot of the instructors in our study, and we see the voices in the book. We talk about reinforcing the students’ feelings of integrity and self-worth and that this self-affirmation dramatically reduces the effects of negative stereotype threat. And we know that this can change achievement gaps and bolster this sense of belonging along with initiatives like using this “Who’s in Class?” form. I think one thing to keep in mind is, although we often talk about negative stereotype threat, there also is positive stereotype threat. And so, one way, as instructors, we can combat that, is thinking about the stereotype content model, because this allows for both types of stereotypes. And what happens is this model is a psychological model that’s based on perceptions of warmth and competence. And thinking about particular stereotypes as high or low warmth and competence. And, in particular, we know that inclusive instructors realize that harm can arise from either one of these and depending on visible and invisible identities. So what happens is you can use this stereotype content model across different types of courses, levels, times to acknowledge and reflect on the individual’s own stereotypes, to offer apologies for students that may have resulted in harm, and to carry out actions that would re-establish welcoming spaces. So we like to think about the stereotype content model can be coupled with these three As: acknowledge, apologize, and act. And so that would just be examining your own background and experiences, and apologizing if there’s been any type of misspoken or things that weren’t addressed, and thinking about how to act and take action in the face of some of these stereotypes.

Rebecca: So as we start thinking about some of these ideas, how do we start building these inclusive principles into our course designs? We’ve talked a little bit about the openers, considering some of these ways of acknowledging and recognizing who’s in our spaces, and who’s in our classes, and who’s in our community. But how do we make sure that we continue that thread of inclusivity throughout the entire semester?

Mallory: So I think course design is a really critical tool for inclusive teaching, and particularly the way that manifests in a syllabus. So I’m a political scientist by training, I like to think of a syllabus as a little bit of a constitution. It’s kind of the founding document of your class. It tells us what our common purpose is, it tells us who’s part of this community, it tells us how we act within that community, what we owe to one another, how we participate in that community, and really what we’re doing. And all of those are really integral questions if we’re thinking about inclusive teaching. So in the survey, I would say there are three really broad themes that came out of people’s responses to how they try and enact inclusive practices in syllabus design. And so the first one was really trying to demonstrate that everyone has a place in the field. We think back to what Tracie was saying earlier about belonging, being important, this is an obvious tie-in to that. And so perhaps the most frequent comment that got made in the survey was, probably unsurprisingly, “We should incorporate diverse perspectives on the syllabus,” and also in other course artifacts throughout the semester, but particularly on the syllabus. So that’s one way to demonstrate that everyone has a place in the field. The next big theme that came out was encouraging everyone to play a role in the learning process. And so we could think about that as another form of fostering belonging, but I would also say that’s part of the equity piece as well, providing space for everyone to be an active part of this particular learning community. And so there were a few different ways that came out in people’s responses. So one idea was encouraging everyone to play a role in the learning process by essentially just setting a tone that this will be an inclusive classroom in the syllabus language. So that could incorporate something like having a welcoming or a diversity statement directly in your syllabus. It could also just be the tone of the language you use. Is the language hierarchical? “The professor will do this and the student will do this,” or is it more inclusive? Is it, “Hey, we are doing this, [LAUGHTER] we will talk about these things, we will tackle these assignments.” Another piece of that puzzle was about setting citizenship expectations. If we want everyone to play a role in the learning process, we want to set some expectations for how we’re going to treat one another while we’re doing that. And I think a lot of syllabi are good at setting expectations for what students owe to their faculty. But, one of the things that we talked about a little bit in the chapter that addresses this is also that a syllabus is a good place to set expectations for how students treat one another, but also what faculty owe to students. And so, again, sort of leveling that playing field and establishing we are all in this community, we all play a really important role, we will all have give and take and here’s the responsibilities we have to one another. And then the third theme that came out, in thinking about inclusive course design, was essentially promoting the conditions for everyone to be successful in the course. So that really nails that equity piece. And so, one of the one of the big-picture ways that people implement this is to think about a syllabus as an opportunity to explain to students, not only what you’re doing, which I think most syllabi do a pretty good job of, but also how you can go about doing that successfully, and critically why we’re doing this. So the “what” is sort of setting clear expectations, so that everyone is on the same page about what we’re all trying to accomplish. The “how” is potentially providing resources to help students accomplish those goals. So directing them to the library, directing them to a writing center, if such a center exists. That could also include things like mental health resources to help students navigate the semester, particularly in the past two years that we’ve been having, those can be especially critical. And then also that last one, the “why,” giving a rationale. We all have reasons, hopefully we have reasons, for designing courses in the way that we do. But we often don’t explain those to students. And I think we often forget that students aren’t inside our heads and don’t really know why we’re asking them to do things in a particular way. And so part of setting the conditions for people to be successful is to explain why we’re doing the things we’re doing to students so that they can make strategic choices when they’re in our courses and are trying to be successful in those courses. And then the other really important theme that came out when thinking about promoting conditions for everyone to succeed is, perhaps unsurprisingly, trying to make sure your course design and syllabus are accessible to as many groups as possible. That’s another way that the “Who’s in Class?” form can come in really handy because there are a lot of ways in which we might try to make something accessible to one group that inadvertently becomes less accessible to another. So knowing something about who is in your class, and what some of the accommodations they might need are, can really help you make strategic choices about how to be as accessible as possible. So those were really the big-picture things that came out about how to make your course more inclusive through the design of a syllabus.

Rebecca: Mallory talked a bit about syllabus design and setting a good tone up front, and the survey does that as well. So what are some things that we can do at key touchpoints throughout the course of the semester to keep this feeling of inclusion continuing and that sense of belonging continuing throughout the semester?

Khadijah: So that’s a great question. I would say that welcoming students begins even before the course starts, even before they lay eyes on the syllabus. So I think that you can set this positive tone, you want to think about it like a greeting card, to promote belonging from the beginning. And so we talked about the “Who’s in Class?” form, but even having a video that would welcome them to the course, kind of like a trailer for your class at the beginning. There are things like the physical environment, thinking about that if you’re in person, but if you’re online, think about what are the first images that someone sees when they log on to your learning management course or the course website. Thinking about what type of activities would emphasize diversity and equity and inclusion. And that would be at the beginning, such as the “Who’s in Class?” form, but throughout the semester. And so I think that those things are carried out. Building the relationships with the students are also important throughout the semester. But at the end, I think we never think about how the students, even at the end of a course, feel welcome. It’s never too late. So even on the last day of class, you can highlight as an inclusive instructor, and we saw this throughout our work, how much you’ve learned from the students themselves and thanking them for how much that they taught the instructor. And thinking about, by having this equitable participation that Mallory brought up, that acknowledging that at the end of a course, actually affirms them in their abilities. It encourages them to see themselves as members of that community of practice, and we know this is critical for various disciplines. And wrapping up with giving students a way to reflect and give feedback on how welcome they felt in that environment. And that is really critical, that feedback that they give, for helping make future classrooms more inviting.

John: And you also advocate not just doing that at the end, but also getting feedback from students regularly throughout the semester, I believe. Could you talk a little bit about how you might do that efficiently?

Khadijah: Exactly. So, I think when we think about content, we think about formative and summative assessment. It’s the same thing with the sense of belonging. So you can do a mid-semester check-in. That could be a formal survey, or it could be something as simple as, “What’s working?” I typically take a piece of paper and say, “What’s going great so far?” and “What would we like to work on as a community?” And so that gives equal onus in the shared space in the classroom. But it lets the students know that I’m hearing them and that they belong and what they’re saying is important.

Rebecca: That mid-semester check-in often times well with thinking about advisement and registration for next semester too. So I could imagine really reinforcing a sense of belonging before the continuity of the next semester, or thinking or planning for the future can actually be really useful. And it’s not something I had thought about before, but when you were talking about the end-of-the-semester sense of belonging, our advisement time is coming up right now and registration. So I’m thinking that right now is a really good time to just reinforce and underscore these ideas to make students feel like they really do belong in the spaces that they want to occupy.

John: One of the things we really appreciated in your book was the use of reflection questions. This is something that is really rare in books directed at professional development for faculty. And it probably shouldn’t be, because we all know the benefits of reflection. Could you talk a little bit about the importance of reflection in learning, both for students and for faculty?

Tracie: Yeah, I think that’s a great question and I’m very happy that you appreciated that. We were thinking very intentionally as we were thinking about designing the book in that phase. And you can kind of see there’s like a part one, a part two, a part three, and then these reflection questions embedded throughout, and then also in aggregate at the end of the book too. And so, in general, as you mentioned, reflection is so critical. We know in the science of learning that we need to take these points in time and moments to really think about our learning, to really make sense of it, and see that meaning that we’re making of it, and that we have or are growing. And so, in our book, we thought, it’s so important, this material, that we want you to think about it further. And, as an educational developer myself, I was thinking about all the people also reading the book, and I was like, “Oh, if we were in a setting, like a workshop or something like that, I could ask these questions. Like what would I ask for application or reflection?” And I’d want to have that. And thinking about the book, and talking with my co-authors about thinking about these reflection questions, it was kind of similar where it’s like “Let’s add these in, so that there are these opportunities to actually engage in that process.” With inclusive teaching in general, there’s so many things to think about, to think about how we do it, what we do. And we gave so much information that it was so important, I think, to process it and to allow time points for stopping to actually start to think about it further. The other thing that we thought about in terms of the reflection questions is that we know that, in our bigger study, we found that there are lots of barriers that instructors described to inclusive teaching. One of them was resources, another was discussions, and whatnot. And so, by embedding these reflection questions, it also has easier access if there is a discussion—or a book club, or reading, or opportunities—to actually take this book information and bring it back and talk about it in a community at their institution, whatever that might look like. And so that’s another reason we did include them too. And I think we later decided to include the aggregate too, but I think that was also helpful. And then also just being able to pick through those which you probably want to emphasize more and have that option to do so. Some might resonate more with some than others. So all of that to say that, that’s why we put it in there and I agree that I think it’s a really good thing in books to include that. Especially these types of books we’re really reflecting and we’re really thinking about intentional teaching, in this case inclusive and equitable teaching.

John: So you started writing this before the pandemic and then while you were writing this there was this global pandemic that popped up and it was a period in which there was also a great deal of social stress. How do you think this might influence the willingness of faculty to focus more on the importance of inclusive teaching?

Tracie: So for me, inclusive teaching has always been important as I’ve mentioned earlier. So the fact that all these things happened were just that they were made more public and people became more aware. And now people are trying to change these things a little bit more than the past. So I will say what it did do was really made me think what a timely book… [LAUGHTER] to actually be at this point in time. I think it was a great opportunity. And I think it’s really useful, and we hear that, that it’s been really helpful for many institutions during this time, especially with this increased focus on it, on thinking about these issues as well. I will say that we wrote most of the book, I think a big majority of it, before it happened and then there’s a whole process that happens in making a book so there’s some time. So we did later try to tie in more of the recent things that had occurred a little bit later. But the beauty of it is, it all kind of fit naturally in there anyway. It’s not like we had to majorly revise the book, we just had to address the issues that were facing our nations. So I think, overall, it’s just a timely book. And this has always been important, and we really do need to talk about it, and this increased that ability for us to do that.

Derek: Yeah and I’ll just add, along with increased appetite for tools to help around these ideas of inclusion and equity, there still weren’t so many of those tools out there. So it worked well that we felt that we could provide one of these tools, that we had been working on it, that it was really ready to go out there as this appetite increased. And, specifically related to the pandemic, so one of the the effects of the pandemic on higher education was it forced a lot of institutions and a lot of courses to move to either hybrid or online pedagogies. And interestingly, this was something that we had been considering all along in terms of some of the chapters we were writing and thinking about welcoming classrooms, but also pedagogical means and ways to work, both in and on the ground and in online settings. So as we saw this starting to happen, we did go through and make sure… Are we talking about things and making sure that it’s understood that many of these are applicable, whether you’re in-person or online? And if you are in an online setting, how can they be used in that way as well?

Mallory: Yeah, I would echo, I think Tracie’s exactly right: structural inequalities in academia and society are not new. And I think for a group of four people who are writing a book on inclusive teaching, they’re already thinking about a lot of these. So what was new was maybe the attention of universities, who maybe were not paying attention, were forced to start paying attention, which I think is a good thing. But one of the other things that I think made me reflect a lot on the value of this book, that came out of the pandemic was, in the shift to online learning—as an instructor who was frantically trying to move all of their classes online with a week’s notice over spring break—was how much I valued being able to learn from my colleagues, and troubleshoot things, and benefit from other people’s expertise. And that’s a lot of what we’re doing in this book by drawing on this survey and not just saying, “Well, here’s what the scholarship tells us inclusive teaching looks like.” But saying, “This is what inclusive teaching looks like by people who are in the classroom doing this work, whether they’re formally trained to do it or not.” I think the value of that became even clearer to me, as I was trying to do the same thing with my colleagues on a daily basis. Learn from other people’s expertise as we were trying to navigate this really challenging situation.

Khadijah: So for me, a lot of what my co-authors have said really resonates. I think that I always thought about inclusive teaching before we had such social challenges that have been more pronounced in the media. I think two things stuck out to me as we wrote this book. One of the parts of the book, we talk about what happens when your classroom is disrupted. And I think it’s interesting, we tend to think about internal things that disrupt, so the students or the instructor, but a part of it was what happens with things outside, so these social conditions disrupt our learning. And so, the fact that the book addressed that when so many things were going on, it kind of was a how-to and it gave practical tools, of models and activities that you can do to navigate that. And I think what’s really resonated is that these things that we talked about in the book transcend transient social things. So like Tracie mentioned, something can happen in the future and this book would still be relevant in the way that we think about inclusive teaching, and what would come further down the pipe. So I think that it helped me reflect on current situations, but also kind of forecasting how having these new tools, from people that we’ve learned around the country, how that would help with future application.

Rebecca: I agree, that’s one of the powerful pieces of the book, is that we know it’s going to keep being useful for folks moving forward. And I know that we’re really grateful that we were able to share that with our faculty in our reading groups this year.

John: It does seem from our discussions with faculty that people are much more open to inclusive teaching than they’ve ever been in the past because while the problems and issues have always been there, they were often hidden on campus because you didn’t see the inequity. But when we were teaching students in their own homes, we saw differences in their access to technology, to their living quarters, and other inequities. It was much harder for people to ignore that. And I think everyone came to appreciate the benefits of community and building a strong community as a result of working through the pandemic. I think everyone realized that having a productive community is an important part of our lives. And the importance of that in a classroom, I think, is much more visible to faculty than it had been for many faculty before.

Rebecca: So we always wrap up by asking, the big huge question, “What’s next?”

Derek: Well so one of the things that I’ll say, I’ll keep it short and simple. What’s next? Around the book, it’s spreading the word. It’s spreading the word of why inclusive teaching matters, why equitable teaching matters, and what tools are out there. Whether it’s our book or whether it’s some other tool, some other way to get yourself into that realm, and get some understanding and work with your colleagues and learn from the experts. However that happens, that’s great. And for me personally, it’s doing the exact same thing: constantly learning, knowing that I have room to grow, knowing that I can improve in my teaching personally and all of that, and looking externally and reflecting internally for ways to do that.

Mallory: I think, “What’s next?” is such a great question to end on. Because one of the things we focus on in the book is that inclusive teaching is an iterative process. You never reach the end of it, you never get the perfectly inclusive course. And so, “What’s next?” is always revisiting what you’re doing and trying to, both in your own courses, revise and work towards fixing the things you didn’t get right the last time and at the institutional level, trying to build more capacity for inclusive teaching and buy-in. And I think the big “What’s next?” question is: What happens as we move away from the immediacy of the pandemic? What happens when racial injustice is not the main topic of the news? Do we still have the support for inclusive teaching efforts, or does that fade into the background? So I think the “What’s next?” is making sure that the momentum that has been gained is not lost.

Tracie: Yeah, and I would agree with all of my co-authors so far. I think the institutionalization of inclusive teaching would be so wonderful as a next step. So whether it’s, like, not treating it as a fad, [LAUGHTER] but creating it as part of our cultures in our institutions. So I know, like at my institution, we’re working hard towards that in a variety of ways. For me also personally, I do a lot of work around this, and thinking about the research and whatnot. So one of my steps right now that I’m taking is really thinking about the tools that we can really think about and capture practices around inclusive teaching to have that feedback. So we have all these great strategies, but let’s talk about more tools to really get feedback on our actual teaching practices. So I am doing some research around that right now, and I do work with students, student partners, to help us really think about this thing called “inclusion” and this equity as well. And so that’s where I sit in this space. So I’m going to continue to think about tools like Who’s in Class? and then these new tools, and go from there as well.

Khadijah: So, I echo a lot of what Tracie, Derek, and Mallory said. I think for me, of personal interest, when we do a lot of the inclusive classroom teaching, it makes me think about my laboratory. It makes me think about my teaching laboratory and my research laboratories. And I think teaching and mentoring go hand-in-hand in this space. Particularly when we think about DEI and STEM. And so for me, I’m interested in: What does inclusive mentoring look like in these spaces? And what are some of those principles and practices that are translatable from what we think about in the classroom, but then also what may be distinct in the laboratory and mentoring?

John: Well we very much appreciate you joining us, it feels like we’ve been in a dialogue with you all through our semester so far through the reading group. And we very much enjoyed your book, and I hope many other people will join in reading through it and working with it. Thank you.

Tracie: Thank you.

Mallory: Yeah, thanks.

Derek: Thank you so much.

Rebecca: We look forward to seeing all your new work.

Khadijah: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Anna Croyle.

[MUSIC]

205. Tutoring

Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, Philip Oreopoulis  joins us to discuss his research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy.

Shownotes

Transcript

John: Equity gaps in educational outcomes play a major role in perpetuating economic inequality. In this episode, we discuss research examining how tutoring and computer-aided instruction can be used to reduce disparities in educational outcomes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Philip Oreopoulis. Philip is a Distinguished Professor of Economics and Public Policy at the University of Toronto, the Education co-chair of MIT’s Abdul Latif Jameel Poverty Action Lab, and an award-winning researcher who has conducted a wide variety of studies relating to education and educational policy. Welcome, Philip.

Philip: Thanks so much for having me.

Rebecca: Today’s teas are:…Philip, are you drinking tea?

Philip: My tea is coffee. I love coffee. I once looked for a reason not to drink coffee, I couldn’t find one. I love my black coffee.

Rebecca: A true researcher at heart. [LAUGHTER]

John: And I am drinking a bing cherry black tea, a custom Tea Republic tea made for Harry & David.

Rebecca: And I have Irish breakfast tea. I really need to get some new tea [LAUGHTER]. I’m going to a tea store this weekend, so I’m looking forward to getting some new options.

John: And we have lots of tea in the office, some of which may not be as fresh as it was a year and a half ago. But this one still is good. It was purchased right before the shutdown.

Philip: You guys are inspiring me. I think I’m gonna have some tea sometime today.

Rebecca: All right, good, good.

John: In a November 2020 Scientific American article, you describe a meta analysis that you worked on with some colleagues that found that tutoring results in significant improvements in student learning. Could you describe this meta analysis a bit and what you found?

Philip: To backtrack a little bit, how it got started: my colleagues at J-PAL, Vincent Quan, Andre Nickow, and I, had heard about the potential of tutoring to be an effective form for increasing test score learning performance. For example, there’s Benjamin Bloom’s seminal article in the 80s, where he had two very small studies done by his students that both found off the charts improvement from offering tutoring in randomized control trials. In fact, that’s why he called it the ‘“2 sigma problem” that he found estimated impact from these two small studies were raising learning performances by enough to potentially solve most of our problems that we would be having in education policy. There were a number of recent studies as well, a randomized control trial coming out from the University of Chicago’s Ed lab, also finding very promising results from an RCT looking at providing in-class tutoring to grade nine students. And so we wanted to explore whether there was some consistency in these results, so we decided to try to take a more systematic look, and we gathered up all the RCTs, randomized control trials, in the last 40 years for about 96 studies, and we took a look and we found that consensus was quite remarkable. About 80% of those studies found significant effects larger than .2 of a standard deviation, and the average effect size was .38 of a standard deviation, which is like the equivalent of almost an entire extra year of school, from receiving these programs. And not only were the impacts really quite meaningful, as about as large as you get from education interventions, but they were consistent across the board. I think that this is about as much consistency as you’re ever going to get in an education policy intervention. So we were quite excited about that. We found that the effects were pretty consistent no matter which type of program that you looked at. They were larger for things like in-school delivery, three days a week, one-to-one delivery, full time tutors, but even in cases where that wasn’t the case, usually there were still significant effects.

Rebecca: Can you talk about what age the students were, what grades they were in?

Philip: It was for K-12.

John: I think it’s probably safe to assume, though, that the same effect would hold in the college environment as well. Those are some pretty dramatic effects.

Philip: Of course, to some extent, maybe it’s not that surprising. Giving instruction one-to-one leads to higher learning gains, and the biggest challenge, of course, is cost. We can’t all have our own teacher when we go to school. And so the biggest challenge, which gets back to Bloom’s point calling this the 2 sigma problem, is I think we have a powerful intervention to help education, it’s just that it costs too much to implement it on a larger scale. So the fundamental problem is to figure out a way to scale this in a way that can complement the classroom instruction.

John: And so that’s one of the things I think you’re looking at now, how this can be scaled up in a more cost effective manner. Could you tell us a little bit about your current research in terms of computer-assisted learning?

Philip: Sure. So computer-assisted learning or computer-assisted instruction is a type of educational software designed to help students progress through topics at their own pace. It has a lot of similar features as what you might receive when you’re receiving tutoring. So a typical example might be Khan Academy, MATHia, there’s lots of other types of software designed to help with different topics, math and reading, but they all have these sort of common features that allow students to progress through topics at their own pace. You receive immediate feedback from trying to work through your own problems and a chance to understand where you went wrong. If you do make a mistake, there’s data that’s generated from going through it that someone like a teacher might be able to follow and respond to. And so computer-assisted learning can, in some ways, simulate the tutoring experience, but of course, at a much lower cost. The challenge is you don’t have a real person guiding you through it. So even though a platform like Khan Academy is easily accessible, your willingness or motivation to go through it on your own is probably not as great as if you had a real person guiding you through the same material. So there has been some experimental evidence on computer-assisted learning, not as much as theories on tutoring, but of the 15 or 20 randomized control trials that have been done in this area, they have also been showing quite promising results. In cases where computer-assisted learning is provided, especially during a school setting, those receiving it also seemed to be performing at significantly higher rates than those in the comparison group. So there does seem to be some promise at using computer-assisted learning to generate the gains that we see from tutoring. But the way to introduce it, the instructions that teachers need to learn how to use it effectively, are not yet maybe as developed as we’d like them to be. So getting to, I guess jump into what I’m working on, I think that there’s a lot of potential for leveraging existing resources to combine with computer-assisted learning in a way that might come close to the tutoring experience. And so what I’m thinking of is in the classroom, that the kind of facilitated practice that might go on, say, in a math subject might be much better through a tool like Khan Academy than paper and pencil that we often give students. And so the question I’m investigating is around reshuffling the classroom in a way where the teacher is trained how to use computer-assisted learning more effectively in the classroom to generate that type of experience. So in the context of the program that I’m looking at now, which tries to integrate Khan Academy more into math classes, the teacher is still instructing and presenting topics, but now emphasizing the students following an individualized roadmap that allows the students to progress at their own pace, rather than having to keep up even if they’re missing on topics and not understanding. So the program which we’re calling “Coaching with Khan Academy,” or CWK, has students receive a roadmap of incremental topics and videos to follow at the start of school that roughly proceed in the same order that the teacher is going through. Now, the teacher has the students to try to work on this roadmap for at least an hour, an hour and a half a week, and tries to facilitate that time during the class and encourage more done at home, and the students then have the ability to hopefully get into a routine of watching a video and taking the exercises, and if they don’t score high enough on the exercises they’re asked to try to understand why they made the mistake using the hints and tips and guidance that Khan provides or gets help from the teacher, and then repeat it so that they don’t move on to the next topic until they’ve mastered that. So the students are not proceeding all in the same pace, but it is just a much better way to learn math such that the students don’t go on to the next topic until they’ve established a strong enough foundation on the first one.

John: During the global pandemic, most high schools moved to emergency remote instruction for an extended period, and there’s quite a bit of evidence that that led to a decline in overall learning, but also some growing achievement gaps which are tied to household wealth and the wealth of the school districts in which the students reside. What types of policies could be implemented at the K-12 level so that students are more equally prepared for entry into college

Philip: On COVID, we’ve all been exposed to online learning now, and most research suggests that it’s not a great substitute for in-person but there are certain benefits from being able to speak with a real person over a computer in regards to tutoring. So the biggest one is convenience, both for the tutor and the tutee. It’s nice to be able to jump in on a call and spend just 30 minutes on that or an hour, and not have to drive to the person’s location or do this after school. The opportunity to facilitate more tutoring, I think, is increased by having this online access. So I think there’s a lot of interesting promises from that. This one particularly interesting study that was done during COVID last summer, where a group of Italian faculty organized a volunteer tutoring experiment where they got the Deans of their respective universities to invite university students to volunteer their time, three to six hours a week to reach out and connect with students who have been struggling in the high schools and lower grades. And on the flip side, they got the school districts of several locations in Italy to ask teachers to identify students that they thought could benefit from having this one-on-one instruction. And then the response was great in both ways, there were a lot of people willing to volunteer their time for this effort, and there was also a lot of perceived need for students that needed this. And so from this large set-up, they randomized who they were able to give this offer of assistance to. And it was done all online, sometimes over the phone, but more often through Zoom, or Skype, or whatever was most convenient for the match to take place. The tutors met with tutees, for three hours a week, over six weeks. The topics were either math, Italian, or English, and then at the end, the researchers collected the survey and found similar gains to what we were finding in the online overall. Not only that, but they also collected data on mental health and found improvements in feelings of connection, more positive outlook on life. And what’s also interesting as they seem to show improvements and positive outcomes for the tutors themselves, as well. So it stands the potential for a win-win, and this was all done online. So it’s like the only online study I know, but it seems to show the potential that it might be done there. One other example I should mention is Khan Academy has also initiated another organization that facilitates free volunteer online tutoring. It’s called ‘schoolhouse.world’ and it’s been interesting to watch that trying to get up and running. Their system allows anyone in the world to volunteer their time as a tutor, and then they try to connect anyone in the world wanting to receive that tutoring. And you get some sense of some of the challenges from doing that. How do you screen for quality? And also, how do you screen for safety? So they’ve had to go away from a one to one model to more of a group model. They’ve had to have systems in place to check the quality of the tutoring, what’s being discussed. They’ve had to switch to allowing only high school students to receive the tutoring and a few other challenges. And so there’s challenges but also a lot of potential in this that wasn’t available from always having to meet your tutor in school or after school or face-to-face. So the potential scalability is enormous, and that’s where the intriguing possibilities are with that tool.

Rebecca: So if we’re looking to reduce achievement gaps, we’ve talked a little bit about COVID and the mix of instruction that students might’ve had during COVID, the quality of instruction, access to technology, to even have interactions with teachers in some cases, and historically even, differences in ability when students arrive in higher ed. What are some of the things that the higher ed community might be thinking about in terms of this research? Should we be advocating for certain kinds of policies or programs in K-12? Should we be trying to institute some of these things in higher ed? What are your thoughts on that?

Philip: So just in terms of advocacy and thinking about facilitating more equality, there’s no question that tutoring has, in general, been an unequal program. There’s the whole private sector of tutoring where a lot of households for more affluent families seem to receive it than those from less affluent households. And so one thing we can do as policy-makers is to try to facilitate more tutoring to happen in schools, especially at schools for more disadvantaged backgrounds. We can also focus on providing tutoring to those who need it most. I think that there is a growing awareness of the potential for tutoring to make a real difference in helping address the learning loss that may have occurred with the pandemic and just helping address education inequalities in general. And so a lot of resources have started going towards trying to increase the amount of tutoring happening in schools. I think that the more we understand how to implement it successfully, the more guidance that we can provide the K-12 sector in trying to introduce that. I think that there is a lot of optimism now around its potential. I think tutoring is one of the most effective programs that we can offer to make a meaningful difference at scale, such that we can get more students arriving into post-secondary ready to handle it and succeed well there. So that’s on that end. I think that there’s no reason why we also can’t consider tutoring at the post-secondary level as well, and the potential benefits that might come from that. Even if we just look at first-year calculus, or other subjects in math, computer-assisted learning is well developed even at that level, the need for tutoring at that level is there as well. And so it really does go from that importance of establishing a foundation that one might benefit from tutoring at earlier ages. But even at the post-secondary level, regardless of what level the student is, we can all benefit from one-on-one instruction compared to being in a calculus class of 500, right? I think there has been less research that’s been done in that area, but the evidence certainly points to the direction that tutoring at the post-secondary level would be also effective and important to consider.

John: And you mentioned that Italian experiment where college students were providing tutoring, and you mentioned that that was a very positive experience for the college students as well. That might be an interesting model where college students could improve their own skills and develop a bit more automaticity and more practice in basic concepts, while helping bring students up to a higher level in secondary schools. That’s a program that I think offers a lot of potential.

Philip: So I would agree, absolutely, the expression is you don’t really understand something until you teach it. I think that there’s something to be said for that. I think that there’s also a lot of skills and experience that is gained from trying to help others, from trying to connect with perhaps younger individuals that have not had the same background as you. I think that the experience is also attractive to employers looking at who to hire. I think there’s huge gains from all the things that you might volunteer or use your time for in college, spending some time to volunteer to do something like tutoring could be a very rewarding thing as well. So I’m also excited about that model. I think that there are ways to try to facilitate that kind of model at scale and more research needs to be done to explore how to do that.

Rebecca: One of the things that I heard you mentioned early on in the conversation is the idea that, historically, folks who had access to tutoring are more affluent. So the students who most need the tutoring are the ones that aren’t always getting it, because they can’t afford it. So I love the idea of having it in schools or it’s a part of our programs. But also I think sometimes tutoring has a negative connotation to it. It’s like a deficit model. Especially I’ve seen this in higher ed, students don’t want to go to a tutor because it makes them feel like they’re dumb or something.

Philip: My first reaction to that is that tutoring can be beneficial at any level. For example, in the Khoaching with Khan project that I’m looking at, the potential is to help all students in the class regardless of their level, because every student can be given their own individual roadmap. And that not only includes those that are behind grade level that benefit from establishing a stronger foundation in that earlier material so that they can catch up, it also includes those at a higher level that don’t have to be held back or wait for the instructor to cover new material can use a platform like Khan Academy or a tutor to work on more challenging material that interest them. And so how to remove that stigma that exists in general, I agree the usual perception is when someone asks, “Do you need a tutor?” it’s because you’re struggling. It doesn’t need to be that way, but at the same time, I think the more we become aware of the benefits from the tutoring, the more we realize that it’s a great resource to take advantage of. Getting back at the college level, I don’t know about your own experiences, but it always amazes me how few students take advantage of all the free tutoring that’s being offered by the universities through, like, office hours. The opportunity for receiving one-on-one discussion is often there, and yet so few students seem to take advantage of it, perhaps because of that stigma or perhaps they’re too busy. Some of us, when we went through college, were pleasantly surprised by how much you can get with office hours of graduate students and extra tutoring and how much you can learn from that process.

John: As in a lot of classes, students are treated as if one-size-fits-all education and students come in, especially in subjects such as math where there is a very rigid structure, if you don’t have a solid foundation and concepts, learning new topics is not going to be very productive, because you don’t have that foundation to connect to. And I see that in my own classes, and it’s a bit of a challenge to try to do that. Because of issues of scale I often teach large classes, I try to rely on peer instruction as much as possible with small group activities. Could small group peer interactions in working through problems and problem sets achieve something similar to the one on one attention?

Philip: In the literature, it’s called peer-to-peer, we did not look at peer-to-peer in our meta analysis on tutoring, but there is some literature and there’s some effort to consider that. It’s a little bit of a different model, because you’re relying on slightly older students or similar students to help assist other students. I think more research needs to be done on how to make that happen effectively. On one hand, the potential is there to make this a scalable, effective program that doesn’t cost very much. On the other hand, monitoring quality and the potential to train to be a tutor and to do a good job with it may not be there as much as with the regular type of tutoring program.

John: In particular, I was thinking of activities in class where students work on problems in groups, and they try to argue out solutions. They work together and they can explain to each other things they don’t understand, but the key aspect of that is they get feedback on whether they’re correct or not, some constructive feedback on where they went astray. But I was just thinking that those types of small group interactions could provide some of the benefits without that stigma of needing to go to tutoring and perhaps at a higher scale than tutoring might work.

Philip: The advice that I often give my students is to study until you feel you can explain it to someone else. And so there’s a similar, perhaps, mechanism at play when we’re thinking about that. When you try to write down a concept or explain it, even to yourself, out loud or to someone else, you quickly realize what you understand and what you don’t. There does seem to be a lot of potential there.

Rebecca: Sounds like one of the keys to reducing stigma around all of this is making the coaching or this tutoring model just something that’s normalized. Maybe it’s normalized in class, it’s normalized through the school day, and then people might be more apt to take advantage of it because they have access to it. But also, it becomes a standard way of being, that’s what other people around them are also doing.

Philip: Absolutely! I think if we can reframe tutoring as just individualized instruction or personalized instruction, then we can all understand the potential benefits of receiving more personal help than in a classroom setting, and that goes for pretty much anyone.

Rebecca: It really also matches up well with a lot of universal design for learning principles of flexibility as well, and allowing students to go at their own pace and finding ways of teaching and learning that match well for students and where they’re at.

Philip: And of course, the issue is scale. Getting children to learn in a classroom of 25 to 30 students, when these students vary enormously in academic levels, is just really difficult. And trying to figure out a way to provide that individual attention is the challenge that all teachers face and have been facing for many, many years. And if we can find a way to scale adding on or providing more and more individualized attention, it has the potential, I think, to make a real difference in education. Of all the potential policies that we can be looking at, I do think that, at the school level, leaning towards more individualized instruction is where we should be looking at, for a solution.

Rebecca: It’s so interesting to me that we’re having this conversation early on in our semester, because after teaching online for a year, which I hadn’t done previously, I’ve really worked to make my classes more flexible and actually offer some of those kinds of models that you’re describing where students are going more at their own pace, and that they can get some individualized instruction when they need it and that they need to do this mastery learning so that they build on things over time. It looks to me like maybe I need to look more into tutoring and coaching models that have worked really well to see if I can’t implement some of that more during class time.

Philip: There may be different ways to do it. Some may be more effective than others, but I do think, getting back at what John was saying, it’s harder to provide that individual support or help to students arriving in college without that foundation. I have done some other work at the college level, trying to facilitate more personal attention to students arriving, trying to help them out and encourage them to get into better habits, and it has proved quite difficult to change behavior, and so I have found myself reacting to that by focusing more on earlier grades to see if there might be more promise on trying to foster better study habits, better learning habits, earlier on with the hope that students arrive in college more prepared.

John: I think that’s one of the things a lot of behavioral economic studies have found. Interventions that result in long-term changes of behavior are challenging in general.

Philip: Absolutely.

John: And I think you’ve done some research on that.

Philip: Absolutely. So if we have to change one-time actions, like helping students through applying for college, applying for financial aid, those types of interventions are much more promising at affecting one-time goals than to change habits or routines that involve much more continuous behavior. So helping someone study more effectively, spend more time studying, these are much harder problems to solve. And maybe low-cost nudges that we’ve been looking at in the literature may not be as effective. I think that does tie back into how my perspective has changed over time. It’s hard to have significant influence without personal connection. It’s a lot more expensive, but there’s only so far you can go with sending an email or a text message or a one-time meeting in trying to change someone’s learning trajectory or life trajectory. And the more you sort of look at education policies that have been successful, the more you notice that they often come with this personal connection that’s been important for making that meaningful change.

Rebecca: It seems like we should all be really advocating then for these much more early interventions. It’s much more cost effective if we get those habits in place really early [LAUGHTER].

Philip: I will say there’s surprisingly not enough research on the long-term effects of tutoring. I’ve seen one study that has found that the benefits of receiving that tutoring continued one year past the program ended; the effects faded, but not by that much, and that’s the only study I’m aware of that actually does a long-term study. So on the question of whether we can have these life-changing impacts from targeting earlier ages, certainly, there’s a literature for the very young… like, almost helping at the household, but at the school, I think that more work could be done.

John: And that could be a really productive research area. Before we started recording, we were talking a little bit about, with the pandemic, creating our own videos. Could you talk a little bit about how you try to implement what you’ve learned in your own classes at the college level?

Philip: Yeah, I think that using the situation last year to put my lectures online has freed up space in the actual lectures to be more interactive. So I think it was a benefit both ways. The videos of the lectures themselves became more streamlined, I got a chance to break them up into smaller parts, sort of like Khan Academy videos, where instead of one video that’s two hours long, that goes all over the place, and you’re staring at me and the Blackboard, I created five- to ten-minute videos of vignettes that I could focus on with slides and have a series of these videos that students could watch at their own pace. I could edit them and make sure that the video is as succinct as possible and gets across what I really want to say. So that was good on the video side, and then on the actual lecture side, we spent that time going through problem sets and answering questions and it was much more interactive, closer to the spirit of more personalized instruction. So there was more opportunity for questions, more opportunities for the students to get more involved, and I think it did lead to more satisfaction of that approach. Obviously, the big question is, ‘Do they really watch the videos when they’re asked to do it on their own?’ I think there are ways to try to incentivize that, but just like any class, the students really perk up when they’re working on a problem that was, say, a previous exam question.

John: I’ve used a very similar approach. I’ve used videos for like 20 some years in my classes, but one thing I started doing last year is I embedded questions in the middle of the videos, and that’s a pretty effective incentive structure. It does get them all watching the videos, and at least thinking about it and trying to make some connections while they do it, and that’s worked pretty well.

Philip: Not only that, but you can make them mandatory for class participation. So you stick those questions in and they have to watch the video to find the questions when they pop up, there’s software that can do that. And then you can make it as a way to encourage them to have to watch the video.

John: Do you think that more use of computer-aided instruction is going to be helpful in allowing more students to be successful?

Philip: I’m very optimistic on this potential of leveraging computers with teachers and parents working together on trying to facilitate high-dosage practice. We’ve been talking mostly in math, but it could also be language as well, and maybe other topics. But I think this really is a good way to learn, as long as the practice time is long enough, and the student’s not stuck. I think that it takes a while to get into the habit, getting used to the software, getting used to the routine, both for the teacher providing this and for the student doing it, and so that, for me, right now, is the biggest challenge. I am optimistic that if we can facilitate a way to help teachers and students get to that higher-dose practice using computers, then very good things will happen. I think that the evidence is highly suggestive that the high dosage is a worthwhile thing to get done. I’m hoping that we can generate evidence that that’s the case, but we are finding that there are challenges because there’s a learning curve, it is changing the way that the classroom is done and changing the way the student usually learns, but I’m optimistic that if we can get past that, the students and the teachers will come to like this approach, and that we can do more of it at scale.

John: And I think a lot of people began experimenting with some sort of a flipped approach where they created videos and then use the classroom for more interactive activities, ast least at the college level, I don’t think that’s happened quite as much at the secondary school level. But I think that has helped provide at least some professional development for faculty. But it is an adjustment that students are not adjusting to perhaps as easily as I would like, I know I always have trouble getting across to students that there is some benefit of working through problems in class and watching videos and learning some of the basic concepts outside of class. Students would rather be lectured to, there was that big study that was done at Harvard not too long ago, where students were asked about active learning classes versus lecture classes, and the research certainly showed that active learning in the classroom led to significant learning gains, but students perceived a higher learning gain from lecture classes, and that’s where I think that issue of students’ adjustment is a challenge, and until we get to see a large amount of this occurring, it’s going to be a while convincing students of this, because it’s really easy to sit there in a lecture and nod and smile and have it all make sense and it seems to fit together very logically, but then when you try to apply it, there’s a bit of a problem, and then the questions are somehow unfair. But when students are faced with problems and interactive work in class, they’re confronted by not knowing things as well as perhaps they thought they did, and it’s not as pleasant of an experience. And I think that’s the source of that metacognition, that students perceive that lectures are more effective, because it’s easy to sit there and listen in, and it all seems reasonable. But the problem is when they try to work through problems and realize they don’t quite have those connections fully there yet.

Philip: The lecture seems to make so much sense until you sit down when you get home and try to go over it again, but I do think there’s the potential for this middle ground that even in the experiment we’re looking at, we’re not entirely flipping the class, in fact, we want to work with the teacher to understand what their own preferences are, while still trying to hit this high dosage of practice, which may occur in class, but also could occur at home as well. And I think that there is something to be said by having a lecture of a new topic being done in class, in person, with the real person. It gets back to that importance of personal connection that the computer is not able to provide. And so maybe there is a sweet spot around providing real instruction, real empathy, but also enough time to be working through these problems at your own pace. My vision for the Khan project is that students say, in grade four, getting 90 minutes of math a day, maybe half an hour of that would be the teacher’s own instruction of a new topic, but then a lot of the other time would be students working on their own devices, while the teacher takes the time… instead of just sitting up at their desk… walks around and spends a lot of time looking over the student’s shoulder, using the data that they’re seeing to understand who’s struggling and where, and spends a lot of time working individually while the student is using the computer. So there’s still that interaction going on and taking advantage of the personalization. I think they too can go really well together.

Rebecca: That’s definitely something I’ve been experimenting with. I went all the way flipped before, and right now I think I’m right in the middle. There’s some flipped, there’s some demos that are live so that people can interact and ask questions, and then there’s lots of practice with individualized attention. And it does take a little time to get everyone on board, to get everyone trained to do things in a new way. So in a 15-week semester, it might take two full weeks to develop new habits and workflows for everyone, but really after we get over that two- week hurdle at the beginning of the semester, my classes tend to settle into a routine that seems really productive and that students have been pretty positive about.

Philip: A key feature of the coaching with Khan program, is that every teacher gets their own coach that we spell with a “kh,” and our coaches meet with the teacher prior to school to go over our suggested recipe to follow, but then they don’t just leave it at that, they keep working with the teachers to check in and try to troubleshoot or brainstorm or reassure and remind the teacher until things are going smoothly. But it can take longer than two weeks to figure out how things are going, and then on the student side, it can take a while for them to adapt and understand that there’s some independence on their own for wanting to do it. The hope is that the students start to gain confidence when they see their own progress, when they see that maybe they didn’t consider themselves a strong math student, but if you start them at the right spot on this roadmap, and then they proceed incrementally, and they can see that they are advancing, then they start to understand the potential benefits and internalize the desire to keep going on their own.

Rebecca: Yeah, that autonomy and that empowerment, I think, is really key to the whole puzzle. And I think something that probably tutoring historically helps students achieve is that they can do this. They might have a little extra guidance initially, but then they achieve it and can do it, and that’s really empowering.

John: That’s our hope

Rebecca: We always wrap up by asking: “What’s next?”

Philip: What’s next? I think I made some notes on that. [LAUGHTER] So I think the issue around tutoring and individualized learning is all about, now, scale. I don’t think we need another study to demonstrate that one-on-one instruction, or one-on-two is an effective additional tool for learning, that more should be done if it were possible. A lot of resources are now going into trying to provide individualized instruction. I think a lot of policymakers and governments are looking to tutoring as a way to address some of the learning loss that may have gone on during the pandemic, and I think, in that space, there’s some optimism by researchers and policymakers to try to understand what types of scale up are better than others in a way that we can make a meaningful difference at the aggregate level.

Rebecca: Well, thanks so much. I’m really excited to hear more as your research develops and more information becomes available!

Philip: It was a pleasure to get a chance to chat with you guys. It’s a topic I’ve been spending a lot of time on and losing a bit of sleep on trying to get things to work. The experiment that we have going on, this is going on in Texas, and one of the challenges of doing a field experiment is that so many things go wrong while you’re trying to deal with real people, real students, and provide evidence that this is a good idea. And it’s always a bit frustrating to face these challenges, like just account issues, students have trouble getting on to Khan Academy and the teachers getting frustrated, and it would be a shame to have those issues that can be worked out actually create this wedge from the program going smoothly and making the difference between having these great impacts or not. So it is stressful, but I think it’s worth it to try to keep at it, and I hope to be able to do so. With funding and policy support we’ll just keep trying. I think there’s a lot of interest in it, I think that it hasn’t been difficult to motivate these ideas and wanting to do more on it. So thanks a lot for giving me the chance to share these thoughts.

John: Your work is incredibly important. And so much income inequality is associated with differences in educational attainment, that understanding these achievement gaps and what we can do to narrow them can have a really dramatic impact on society.

Philip: Fingers crossed!

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle.

[MUSIC]

188. Student-Ready Courses

College faculty sometimes complain that many of the first-year students who enter their courses are not “college ready.” In this episode, Natalie Hurley joins us to examine strategies that can be used to ease this transition and help ensure that our courses are “student ready.” Natalie is a New York State Master Teacher and a 2018 NNSTOY STEM Fellow who teaches high school mathematics in the Indian River Central School District in Watertown, NY.

Shownotes

Transcript

John: College faculty sometimes complain that many of the first-year students who enter their courses are not “college ready.” In this episode, we examine strategies that can be used to ease this transition and help ensure that our courses are “student ready.”

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Natalie Hurley. Natalie is a New York State Master Teacher and a 2018 NNSTOY STEM Fellow who teaches high school mathematics in the Indian River Central School District in Watertown, NY. Natalie was also one of our students here at SUNY Oswego. Welcome, Natalie,

Natalie: T hank you so much for having me. It’s a pleasure to be here.

Rebecca: Today’s teas are… Natalie, are you drinking any tea?

Natalie: I am, I’m drinking a peach tea because it’s warming up outside and I want to feel that warmth.

Rebecca: That sounds right up John’s alley,

John: …and I am drinking a ginger peach green tea.

Rebecca: I knew there’d be peach. It’s almost like you guys coordinated. I have Irish breakfast today.

John: No Scottish?

Rebecca: I am almost out of the Irish breakfast. And then we will move on to a different container.

John: Move down the Empire [LAUGHTER] and head back to your English breakfast and afternoon.

Rebecca: Yeah. [LAUGHTER]

John: Okay, we’ve invited you here to talk about the transition between high school and college. Could you first tell us a little bit about the courses that you teach?

Natalie: I teach high school math. And I have a pretty nice spectrum. I teach algebra I, calculus, and precalculus. I’m in my fifth year of teaching precalculus and calculus, and I have taught algebra I, eight years.

John: Excellent. What were your majors in college?

Natalie: So, I was a math major from the get go. And then I had this fantastic professor who helped me find my love of economics. His name is Professor John Kane, and he’s here with us too. [LAUGHTER] And then, once I started to find my love of economics, I picked up a minor in it. Unfortunately, it was too late in the game to pick up a double major, but the minor was great too.

John: And we did talk a little bit about you going on to grad school in economics. And I was a little disappointed that you didn’t, but then I thought you could do a lot of good in the school system. So, one of the things we want to talk to you about is the differences between your experiences in college and the way you teach your classes now. Could you talk a little bit about some of the differences between what you observed as an undergraduate student and the way in which you teach now in secondary school?

Natalie: Yes. So there are definitely a lot of differences. Although I tried to definitely keep that idea of college and career readiness, being wholesome in my classroom. I remember in college it was a lot of lecture style, a combination between very, very large classes of hundreds. And my math classes were usually pretty small, less than 20, generally. In my classroom, in a non pandemic year, I have somewhere between 15 and 20 students usually. Of course, now we’re doing cohorts. So I have a class as small as two and another class as large as 13… is my largest class… on Thursdays and Fridays. So I just remember, in college, a lot of lecture style: the teacher talks, you write notes, you can ask questions, but it’s very much like you raise your hand, if a teacher calls on you, you can ask a question. In my classroom, it’s more a conversation about math, whereas I’m leading them to discover the math, hopefully on their own, with a little bit of questioning from me to lead them there. There’s a lot of… in a non-pandemic year, think-pair-shares, or working in groups, or talking it out with their partner, or somebody that they sit near… things like that, that’s a little bit harder to do these days, especially when I only have a class of two in there sitting six feet away from each other or more.

Rebecca: Everything’s a think-pair-share. [LAUGHTER]

Natalie: Right. So yeah, and as I grow, and I become more seasoned, it becomes more student centered and less about me than it had been before. So I want to put the students in the center, whereas when I was there, it was very much the teachers up there telling you how to do it. And then this is how you do the calculus. And then this is how you do calc 2, this is how you do calc 3, and they just show you and you find your study groups, and then you work with each other after class as opposed to in class. So that seems to be some of the differences. I remember in college is a lot of rote memorization: know these words, be able to regurgitate this proof, things like that, where just now, in my classroom, and really the focus of high school math, and really all math through the Common Core standards, is that deep understanding, getting to the core and understanding vertically, what number sense and number theory is throughout all the grade levels.

Rebecca: It sounds like you’ve incorporated a lot of evidence-based practices into your own practice, which we encourage all of our faculty, even at the college level, to do as well. We hear a lot of faculty complaining about students not being “college ready.” And maybe that’s because there’s still a lot of lecture and memorization, and we want to keep changing that. What are some ways that we can become more “student ready” and bring students in?

Natalie: I think it’s very important that colleges understand what it’s like in the high school, how teachers are teaching, what they’re teaching. I’m in my ninth year, and the next year, we’ll start to move into the next gen math standards. And then that’s going to be the third set of standards. And I think it’s really important for college professors at all levels to be able to understand what did these standards look like for these kids? And how do I need to change what I’m teaching to adapt to that? I recently had a very informal discussion with the master teacher program and SUNY professors at SUNY Delhi about their math and science curriculum. And there was just a lot of surprises on both ends about what’s being taught, what isn’t being taught any more… emphasis on the calculator… standards now are being written so that there’s a lot of calculator use and students are losing a lot of their number sense, because you don’t need to know how to add and subtract, multiply and divide fractions when you get into high school, because you have a calculator that’s going to do it all, which turns into a lot of button pushing. But that’s definitely going to have a trickle down effect into… or trickle up effect, I guess… in college, when these students get there and they’re so used to being fully dependent on calculator and technology use… which is great if you’re a college that also promotes that. But some colleges may also still say, “Hey, we want to limit technology.”

John: Since you were in college, I think much of our teaching has changed. I know in my own department, most of us were doing a lot of lecturing when you were a student. And there’s been a pretty steady transition away from that. And that’s been true in many departments, including our math department.

Natalie: That’s great to hear.

John: But we still see a lot of lecture. And one of the things we’re hoping is that perhaps the experience of the transition to remote instruction has encouraged more people to try some new approaches to teaching and learning.

Natalie: We have definitely seen that in the high school, a lot of teachers who were behind the ball, as far as utilizing technology in their classrooms, it’s been forced to become great at it… literally overnight. So I guess if there’s a silver lining to all of this, that is definitely key right there. I think something that’s extremely important in high school is making connections to real world, making everything very real life, how am I ever going to use this in real life and making sure that that is evident in the student learning. And I don’t teach in a college, I haven’t been in a college in 10-15 years. But I think that if you wanted to be more student ready is also connecting all of your curricula to career readiness. Students want to know why am I taking this class if I’m going to be a librarian, or if I’m going to be a social worker. So just being able to bring those connections to them, could also help colleges be more student ready.

Rebecca: Just that relevance alone is more motivating to get students excited about topics. So finding ways to connect to students, no matter the level, high school or college, is a really great way to bring them in and bring them along.

John: One of the challenges I know in economics we face, is with students saying, “I’m just not very good at math,” or “I have trouble with graphs.” And I suspect maybe you might have seen a little bit of that, too. How do you address that issue of a fixed mindset concerning student’s ability to engage in mathematics?

Natalie: In my calculus and precalculus classes, and you’ll be even surprised to hear it that I do hear that there. But it’s extremely prevalent in algebra I, that idea of “My mom wasn’t good at math, so I’m not going to be good at math.” “I’ve always had math support, AIS,” or “I’ve been able to get through because I’ve always stayed after with the teacher.” I definitely still hear that, but the idea is to kind of break that mold by saying, “Hey, listen, this is a new year. This is a new teacher, this is a new curriculum, maybe this is going to be the one for you.” I find a lot of students struggle through geometry. So as soon as I bring up geometry, it’s like, “Oh, no, I don’t want to.” And I say, “Listen, let’s connect it to algebra, then. Do you feel more comfortable making the connections to algebra?” …or I don’t even know. I don’t know how to tell them that you can be better. Maybe you had a bad teacher, maybe you had a teacher who didn’t make things relevant for you, or just some bad experiences, but every year’s a new year. I’ve seen all sorts of success stories of students who truly find their niche late in high school math. So in my class, I’ve definitely adopted the idea of a growth mindset. And I’ve been studying on best practices to help do this. And part of that has been through standards-based grading. I’m kind of a beginner, a novice, of standards-based grading… more of a cafeteria, I’m choosing which aspects of it that I’d like to implement versus which ones don’t work for me, yet, as I learn and grow, which is definitely not the high school or college that I went to, to implement things like re-quizzes and retests. And it was hard for me to even start with this idea. Nobody else in my department was doing anything like this. And to be honest, as I was starting my career, I started my career right as common core was being implemented. So there was a ton of work in developing all the new curricula. I started in an eighth grade and moved all the way up and I’m like, “Wait, you want me to make two tests? four tests? two quizzes? three quizzes? four? And you want me to give them to these kids and grade them and change their grades and all of the clerical work. But now that I’m nine years in, I’m ready for it. And I find that the students have such a better… I guess they feel better about themselves when they go in to take a quiz when they know that there’s going to be redemption. One thing that I do is, if their test grade beats their quiz grade, I let the test grade replace the quiz grade in the gradebook. If you can show growth, by the time you get to the test… and that idea actually comes from college. If your grade on the final is better than your grade in the class, some professors will do something like that. Or if your grade is high enough, you don’t even need to take the final or I don’t know if anybody’s still doing things like that. But that kind of way, it takes the stress of testing and quizzing, and it takes it off of the grade and puts it more onto the learning. And, however, I do understand that that practice might not be making my students “college ready” if their college professors are not following that, which probably they are not. However, it’s something that I feel I need to do in order to focus on the learning that needs to happen.

Rebecca: I think one of the things that we’ve been working really hard to do at the college level is to incorporate some of those practices as well, to encourage learning, and that it’s about test taking and the testing effect as a way of remembering and learning and practicing rather than some moment in time that somehow reflects all learning that has ever occurred, which is not really relevant or accurate.

John: And you would be happy to know that many people in our math department are doing mastery quizzing and mastery learning approaches where they allow students multiple attempts to demonstrate mastery.

Rebecca: One of the things that I think is somewhat problematic for the transition of students from high school to college is that we don’t have these kinds of conversations very often. They’re not in place unless we go out of our way to talk to local high school teachers about what’s going on in local schools and vice versa, where high school teachers are asking us what’s going on at the college level. Can you talk a little bit about how your Delhi experience evolved in ways that you could encourage others to have those same kinds of conversations?

Natalie: I believe it was somebody from SUNY Delhi reached out to the Executive Director of the New York State Master Teacher program, if you haven’t heard about it, it’s a STEM initiative to retain teachers in math and science. And it’s a four-year fellowship that you have to apply and interview and be selected to join. But, it ends up creating a very strong network of teachers statewide that are actually affiliated with SUNY, every region has a SUNY campus that they are affiliated with, Up here in the North Country, mine is Plattsburgh. And since our program has moved more to an online format, we were able to all participate with SUNY Delhi through this conversation. And I think that just leads to more ideas for the future or to somebody who has an in with some SUNY people. I think we all do professional development. I do it as a high school teacher, and you guys do it as professors. But then, where can we find the middle ground that can maybe mesh some professional development and share some best practices between our two groups?

Rebecca: Sounds like an excellent idea.

John: That is a practice that I think we should do more of. One of the issues that we have is that in general, high schools provide a lot of support for students. And often some of that support comes from parents who help encourage students to be successful in school, and then all of a sudden, people go away to college. And sometimes that doesn’t work quite as well, and we lose a lot of people. Do you have any suggestions on what perhaps colleges could do better to help retain more students?

Natalie: That’s a great question. And I think as we grow through the years, you’re noticing a lot more of, shall I say, helicopter parents or parents who are very, very involved in their children’s lives, and so much to the fact that they actually can become bulldozer parents and literally take roadblocks out of their children’s way so that they never experienced any kind of struggle. And that, of course, is not anything that me as a high school teacher, or you guys as college faculty can do, other than just encouraging parents to step back. Students should be very autonomous. I think that, if anything, we’re going to see out of this pandemic. Another I guess silver lining is the students are becoming more autonomous. They have to when their parents are working and they’re only going to school two days a week and they’re responsible for their own learning the other three days a week. They’re also learning skills, like do I care for online learning? So when I get to college, should I steer towards classes that are online or steer away from classes that are online. So these are definitely going to be skills. So you guys might want to look out for when you’re screening students or as advisors, “Hey, how did you do when you were fully remote? Were you able to get all of your work done? Did your mom have to come home and sit down with you and work with you? Or were you able to get up in the morning, eat your breakfast, and then get right started on your schoolwork?” I make a joke to my seniors about this time of year that if they are still being woken up by their parents, they are not ready for college. You will be home by Christmas if your mother is still waking you up to get you in the shower and get to school on time. Students are going to have a lot of free time when they get to college. Their lives are managed for 40 hours a week, an hour bus ride in and back, somebody tells them when it’s time to eat lunch, then they go right to sports practice and come back. They’re so busy when they’re in high school. And then when they get to college, and they have class that are for 15 hours a week, that’s a lot of extra time for these kids to have to manage. And then when they start to find out, “Oh, nobody’s gonna call my mom, if I don’t go to school?” That’s the kind of mentality that these kids are going to. And I ask my students often, “What do you think is going to be one of your biggest struggles?” or “What do you think is going to be a biggest struggle of your peers? …and it is time management. The students are used to doing a lot of their work in school, either in the class or given class time to start assignments, to do assignments, or they have a study hall where somebody says, “Okay, get your books out, get to work.” And now they’re just going to go to class, they’re going to learn for an hour, hour and a half, and then go back to their dorm and have to start studying on their own. And even that, just the idea of needing to study, a lot of students can very easily get through high school without studying, without ever learning how to study. They just are good students, they know how to sit and behave, they learn well, they do their assignments, and that’s enough. I was a victim of that. I was always above average just by doing exactly what is expected of me. And then when you get to the bigger pond and you become the smaller fish, that can cause a lot of struggle for students, that idea of time that’s not managed, that they need to learn how to manage on their own.

Rebecca: I think you’re highlighting a lot of great themes here. And we certainly experience as college faculty, but your students are right: time management is the number one struggle of our college students. [LAUGHTER] They already know that it’s a problem. We know that it’s a problem. High school teachers know it’s a problem. But we don’t actively necessarily all collaborate on finding solutions to that problem. We just expect overnight students are somehow magically going to have autonomy and know what to do with it. [LAUGHTER]

John: And that’s especially true for first-gen students who haven’t had family members talk to them about those challenges and those issues, and who don’t have that sort of support. And suddenly, they have all this free time. And they don’t have any tests until weeks away or months away. And they just had these big assignments due so they don’t have anything they have to do right away. But one of the things that we know makes that worse is online instruction. Because there’s a lot of research. I did some about 17 or 18 years ago that found that freshmen and sophomores just did dramatically worse with online courses. And we had a guest on our podcast several months ago now, who found the same thing in a much larger study… that juniors and seniors and older individuals, if they’ve been successful in making it to that stage of college, they tend to be relatively successful at managing their time effectively. So online instruction is much more challenging for freshmen. And one of the things that I know many faculty are concerned about is the fact that all of our students coming in next year will have spent at least a year in some sort of remote or online instruction. And the quality of that varies quite a bit across school districts. And I think that is going to be a challenge we’re going to need to address. Do you have any thoughts on how we can help students be better at this?

Natalie: You are definitely correct, there are going to be a lot of gaps to fill, based on how students have received instruction, in how much instruction, we were given guidance that perhaps we’d only get through 80% of our material. And as you know, at the end of the 2020 school year, some of the learning immediately stopped in mid March with the “do no harm” idea of you can’t do harm to any of their grades, whatever their grade is when they left is what their grade needs to be. So we saw students who had very high averages that said, “You know what, it’s good enough. So I know you can’t give me anything less than a 98. So I’m not doing anything for the rest of the year.” And that was a very real situation that’s not going to be reflected in a transcript that you receive as a college advisor or as a college applicant when these kids are applying to school. I’m hopeful that students could self assess where they’re at, that if a college is going to have a lot of remote opportunities, and they are good at online learning, absolutely, go there, that’s for you. And if not, then that might not be for you.

John: I think we can also refer back to an earlier podcast we did with Viji Sathy and Kelly Hogan, who talked about the importance of structure in all of our classes, that providing students with clear directions, with giving them more expectations, and a couple of other podcasts that we had with Betsy Barre, she discusses the importance of sharing information about the time that’s required for various tasks. So giving students more detailed instructions and giving them more guidance on how much time they should be expected to work on things, perhaps, may help.

Natalie: I completely agree, I remember the rule of thumb was a three-credit level class would have about nine hours, the total amount of work, including class time for that class. And I try to drill this into my students heads as a pre-calc and calc teacher that you’re going to be doing a lot of work outside of this class, which is not what they’re used to. They’re used to, “I’m going to go home, I’m going to do 15 minutes worth of homework, and it’s going to be graded based on effort. So I get a free 100 in the gradebook if I did it, and nobody’s gonna care if I did it correctly, I could do the whole thing wrong, I could copy from my friend, and there we go, I get the credit.” And that’s just not college readiness, or career readiness, for hopefully most careers. But I would love to see some kind of, even in high schools and then have it trickle right up to college, some kind of bridging from each course… ninth grade, these are our expectations, 10th grade, you get a little bit more strict restrictions, and then all of that being an idea of we’re bridging the gap from 12th grade to college. And as well as college professors and faculty looking back and saying, “Okay, this is where they’re at when they’re coming to us. And so we can pick it up from here.”

Rebecca: Yeah, that ramping is so important. And just acknowledging where students are at, and meeting them where they’re at, and not having some false expectation of where we wish they were, which is very different. And I think what John was talking about, like providing some time allotments and things, but just even providing students with a sample of what their outside of class time should just generally be looking like, maybe you are spending three hours outside of class studying or six hours outside of class studying. But what are those six hours look like? What does studying look like for this class? What are the kinds of exercises that would be really helpful. And faculty have a vision of what that should be, we just often don’t communicate it.

John: And often when faculty do, though, they communicate what worked for them, which is what had worked for their professors before them, which is not always what would work for a typical student.

Rebecca: What do you mean? Highlighting, John?

John: …and repeated rereading, and focusing on learning styles, but there’s a lot of things out there that faculty may encourage students to do that is not really always consistent with evidence. And going back to your point earlier, Natalie, about providing the relevance of things, one of the problems I think that faculty have is we got into these things, because we’re just really interested in the questions of the discipline. And the things that interest us now are based on having studied the discipline for a long time. And it’s a little harder, often, to connect with the types of things that would interest a student who’s just coming out of high school or is in a sophomore or junior position in college, because they don’t have that same network of concepts to make the topics that we find interesting as interesting to them. That’s something I think we definitely need to work on.

Rebecca: I know one thing that I always struggle with in the design classes that I teach is I’m constantly trying to get students to think about audiences other than themselves. But then when you start to put things in perspective, you realize they don’t really know much beyond themselves. They know maybe what an audience younger than them might experience or have expectations of, but they have a really hard time envisioning what a professional world might be like, because you just don’t have any experience in it. So we often come to the table with so many assumptions that we think that they have, and they just don’t have the life experience to match it.

John: We’ve talked a little bit about how instruction was affected during the pandemic, how did your classes transition?

Natalie: I saw this, the pandemic, as a way to be able to spread my wings and try and implement some different teaching strategies that I had thought about, but I wasn’t totally sure if I wanted to dive right in in one given year. And one of those ideas was the idea of a flipped classroom. I had never felt confident to be able to create a flipped classroom that I felt students would be responsible for. That being, watching the videos at home and then us just working on problems together in class. And to be honest, that’s pretty much what has happened. I have students who will have to watch videos on the two days, three days that they’re not here, and they have to, and I’ve figured out now how to assess that they’ve watched the videos, and to make sure that they are responsible for that learning. It’s not totally flipped. But I do have to provide instruction to these students three days of the week that they’re not around. And I did that in a couple of different ways for each one of my preps, because I didn’t want to get so solid in just one way was “the way,” and I wanted to try some things out. So for calculus, I teach calculus fully synchronous. My students who are at home Google meet live with the students that I have in class in front of me. We’re able to talk about problems as a bigger class instead of just the five who are in front of me. And that’s worked out quite well for us. Granted, class is at 10 o’clock, so it’s not too super early for these students. So, precalculus, I do kind of more HyFlex. They have the opportunity to come to the live session if they want to, or I do make videos for them to watch on their own. One thing that we had to keep in mind this year was that some of these students are responsible for younger siblings while parents are at work. So we needed to be a little bit more flexible in how we set up our classes. And when I had the conversation with my calculus students, they said, “Nope, we could totally do synchronous.” And if they needed me to make a video, I can make them a video. And then algebraI, I do completely asynchronous, the videos are already set up. There’s no live session. When they come in the next day, or whatever day follows watching the videos, I check and make sure that they watch their videos, give them a little bit of credit, just for doing what they were supposed to do. And, to be honest, I have become very, very graceful compared to how I ever was in the first seven and a half, eight years of teaching. My department very rarely ever accepts late work. Math is a sequential subject, you need to do the work today so that you know what’s going on tomorrow in class. This year, I’m just happy it’s getting done and getting done with integrity. So I’ve become so extremely graceful with students who are getting work done after due dates, right before the quarter ends. It’s not great, but they are also learning a very important lesson in “it’s a lot easier if I just do a little bit every day instead of trying to cram it all in at the end of the semester, or at the end of the quarter.” So, hopefully, that idea can trickle up to college as well. It’s not very easy to dig yourself out of a hole, especially in college.

John: You mentioned with your recordings that you were verifying that they actually watched the videos. How do you do that?

Natalie: With my algebra I, the videos obviously match their note packet, they have guided notes that matches very nicely. So the next day I just come in, check, they’re going to show me a front and back with all of the notes filled in. For precalculus, I feel like I shouldn’t have to check their notes, that they should be responsible enough for that level to actually watch the videos, I did get a sense that they weren’t watching the videos, mostly because I was bringing things up when they came to class, and I had a lot of blank stares looking at me. So I started using a program called Edpuzzle. They have to sign into it with their Google account. Lots of teachers in my school use this. And I was able to see who was actually watching. And it was funny, I had a student come to me and he goes, “I’m going to make a guess that 1/3 of the kids are actually watching the videos.” And as soon as I started counting up the number of students who actually watched the videos, it was about 1/3. So that gave a very stern talking to about what my expectations were in that they’re not going to have the best understanding and knowledge if they’re just practicing the homework assignments, that it needs to be the full package, coming to class in whatever shape or form that looks like, be it coming to the live session or watching the videos. But all of it needs to be done in order to be successful in precalculus.

John: Do you embed questions in the videos?

Natalie: I am just getting my feet wet with it. I just started using it. So no, I haven’t gotten that far. But I did hear that you can and you can do a little checkup and it will not let you get through unless you do the question. So is that something you guys have used?

John: I haven’t used EdPuzzle. But I have been using PlayPosit this year. And I embed questions in it and it is required as part of the grade in my econometrics class and in my introductory microeconomics class, and I’m probably incorporating it in more classes as I go forward. But students actually have responded really positively. They discovered that when they watch the videos, it’s really helpful, because when there was no grading involved or no questions, the level of use of the videos was dramatically lower. So this provides just a small incentive, it’s a trivial part of their grade, but it’s enough to induce them to watch them, though. I’ve been really pleased with it. One advantage of Edpuzzle though, is that It’s free, as far as I understand, whilel, I do have to pay for PlayPosit.

Natalie: Yep, it’s free. And we’re a Google school. So they are able to sign right in. And it was easy for me to use, I could see all the students, I could see how much of it they watched. So that was pretty cool.

Rebecca: That’s great. As I’m hearing you talk about how your teaching has transitioned, it largely matches how faculty at the college level have also had to transition. So in some ways, we may have been brought together unexpectedly. [LAUGHTER] So there’s a lot more college faculty also using a flipped classroom model, and using more class time to solve problems and work on the more complicated things. Because often the video lectures or that kind of part of the material is foundational. Some of it might be memorization things, it might be terms and things like this, and then you put it into action in the class, and you can have guidance and coaching. So what’s really interesting is that the pandemic may have brought these two experiences together, and that bridge might be more present than it has ever been before.

Natalie: Exactly, I used to think that there was no way for my students to learn, if I wasn’t the one who was telling them and showing them. If I wasn’t up at the front of the board, doing it with them, doing it for them, then they weren’t going to learn… there was no way they can learn this on their own, and boy, was I ever wrong. Even though I could do 10 problems with them up at the board, whereas they could do three working together in a small group or in a pair, it’s so much more valuable to listen to them have those conversations, and to hear them explain these things, in their own words, or what’s very powerful too, is when I hear my words come out of their mouth, as they’re explaining it to another student. So that’s totally cool to hear that. And I think that’s the direction of education. The prep work is more important than what the teacher role is in the classroom. So the role of the teacher plays planning and prepping for a lesson is so much more powerful than what the teacher is doing in the classroom at that time as far as leading instruction.

Rebecca: That design of experience is so powerful, just generally. I think one of the things that I know I have always struggled with using a flipped classroom model, but I’m definitely getting better, the more and more I do it, is really scaling back on how much can actually be done in class, when students are working through problems and they’re not having quite as much coaching. You’re really letting them struggle and fail and try again. That takes time. It’s not like something that can happen automatically. So we might be used to going through more examples, but less examples, but more depth can be really powerful. And your right, students explaining to other students is an amazing thing to see. But it also helps them articulate or realize where they don’t understand, which is something that you don’t necessarily recognize when someone’s explaining something to you. Because when someone explains it to you, of course it seems obvious, until you have to do it yourself.

Natalie: Absolutely.

John: One of the things I was thinking about is I have to create a video on log transformations in econometrics tonight, and that reminded me of all that.

Natalie: Do you want one of mine? I can grab it right off Edpuzzle. [LAUGHTER]

Rebecca: Can you imagine that, though, like sharing back and forth between high school and college and sharing resources? We should do more of this.

Natalie: Oh, absolutely. Why are we always reinventing the wheel for ourselves, when there’s so much already out there?

John: When you first create a flipped classroom model, it’s a lot of work, creating those videos, maybe embedding some questions in them. And doing that is a tremendous amount of work upfront. But it’s taking this stuff that’s relatively easy for students to learn and shifting it outside of the classroom. So that when students are in class, they’re able to focus on the things they have the most trouble with. Because, as Rebecca said, you can provide a really good lecture on how to do something, but if students haven’t wrestled with it, they’re not going to learn it as well. I know for many years, I was doing an awful lot of lecturing, and I’ve cut back to very, very little now other than when I’m recording lectures, and I’m seeing students struggle with materials, but then when they are assessed on it, there’s much more balance in their performance. Because before there were always some students who would pick up on everything you did… people like you, Natalie, and people like those who are faculty. We were able to learn the stuff by listening to people tell us how to do it and figure it out on our own. But that doesn’t work well for a lot of students. And when they’re there explaining it to each other, they learn it much more deeply than they ever would by trying to do the more difficult parts on their own. Because in the traditional model, we’d give them the basics and show them how to do things. But the only time when they really got to apply that was either when they were working alone, doing homework, trying to struggle to solve some problems. or on a high-stakes exam. And those are times when perhaps they need the most support and the flipped classroom model can really address that really nicely.

Natalie: Yep, I absolutely agree.

John: We always end with the question: “What’s next?” …which is a question I think everyone in education is wondering.

Natalie: I think what’s next is filling in the gaps. There is going to be a lot of gaps for a lot of years. And we keep saying I can’t wait till next year, I can’t wait till it’s back to normal. But pandemic aside, I don’t think it’s going to be normal for a long time, we’re going to have a lot of students who need a lot of support for a lot of years, and it’s going to affect me. And then it’s going to affect college faculty, as the students get up there, addressing where the gaps are, how can we fill them? How can we give these students the support that they need, even under just budgets that were being given that were cut in this past year or could be getting cut? We’re going to need a lot of support from each other, from our colleagues, from our administrators from the community… and I think a lot of understanding… these students are going to need understanding, we’re going to need understanding.

Rebecca: I really like the underscoring of this empathy towards one another, both between faculty and teachers, as well as between students and teachers. I think that’s a really nice note to end on. Thanks so much for sharing your insights, Natalie. I think many faculty will want to take the time and effort to reach out to some high school teachers and make some connections and start to figure out ways to bridge those gaps.

Natalie: Thank you so much for having me, this has been such a valuable experience. And I’m always grateful when anybody wants to listen to my opinion and share conversation with me.

John: Well, it’s great talking to you again, I very much enjoyed working with you when you were a student here, and it’s nice to see just how successful you’ve been. And I suppose I should mention that this came about because one of my current students had listened to the podcast, had shared it with people on Facebook, and then she said, “And you know, you really should invite one of my former teachers,” and so we invited you.

Natalie: That is so awesome. I feel very honored that that student thought that I had something great to share. And it’s been so great to be back in SUNY Oswego, among other Lakers, and of course, a professor who was a great mentor and led me down a path of success.

John: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

169. Statistical Simulations

Abstract concepts can be really difficult for students to grasp. In this episode, Matt Anderson joins us to discuss how simulations can be used to make statistical concepts more tangible. Matt is a lecturer in the psychological sciences department at Northern Arizona University. He was a recipient of the 2020 College of Social and Behavioral Sciences’ Teacher of the Year award at NAU.

Show Notes

Additional simulation resources:

Transcript

John: Abstract concepts can be really difficult for students to grasp. In this episode, we look at how simulations can be used to make statistical concepts more tangible.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Matt Anderson. Matt is a lecturer in the psychological sciences department at Northern Arizona University. He was a recipient of the 2020 College of Social and Behavioral Sciences’ Teacher of the Year award at NAU. Welcome, Matt.

Matt: Thanks very much, John. It’s a pleasure to be here. And I’m really excited to be talking about the use of simulations in introductory statistics classes.

Rebecca: Today’s teas are:

Matt: Today’s tea is lemon ginger.

Rebecca: Well, that sounds good.

Matt: Yeah, that’s a fave.

Rebecca: How ‘bout you, John?

John: I have Christmas tea…

Rebecca: Aha.

John: …with a cinnamon stick.

Rebecca: That’s only because I was drinking it last time.

John: You inspired me to buy some Christmas tea, which I’ve been drinking for the last couple of weeks.

Rebecca: I almost sent you some. And I have a Scottish afternoon tea today.

John: We’ve invited you here today to talk about how you’ve been using simulations in your courses. But first, could you tell us a little bit about what courses you normally teach?

Matt: Absolutely. My main mission is to teach the labs for the undergraduate statistics courses, the introductory statistics courses that are taught within the Department of Psychological Sciences. And so I teach about six to eight of those a semester. And I think it might be useful to just contextualize the labs. They’re one-credit labs, and they’re embedded in four-credit, introductory statistics classes. And all of our faculty use the same schedule, and we use the same textbook, and so there’s a lot of coherence among the sections that we have. So, the thing I love about the lab the most is that, because I’m getting all the psychological sciences majors, I have a chance to meet almost all of them. And I have a chance to do that early in their academic trajectories. And that provides an opportunity to get things on their radar for those who are going on to graduate education such as the graduate record exam and things like that. We have a little bit of a focus on SPSS in our lab, in addition to the normal course content that we would see aligned with the textbook. In addition to the course faculty teaching the lecture portions, and I’m teaching the labs, we’ve got some wonderful tutors, and those tutors come from our Academic Success Center. And we’ve also got a tutor who’s a second year graduate student in our Master of Arts in Psychological Sciences program who adds a lot to the course delivery. So it’s a really wonderful place for me to be teaching. I feel very well supported, and I love the mission, even though I understand that statistics may be, to be contemporary, not everybody’s cup of tea. [LAUGHTER]

John: So how many students are there in these classes? And what level are they? You mentioned they’re fairly early in their career, are they mostly sophomores?

Matt: Well, it’s a mix. Yeah, I think that the tendency is that we’ve got sophomores and juniors, we do have first-year students, and we also have seniors, but most students are sophomores, or juniors. And in each section, I’ve got maybe 30 students. So you can see when I’m teaching eight of those, that’s a lot of folks. And that’s complicated a little bit by the online delivery that we’re using right now as well. I also teach a thing called an undergraduate teaching apprentice class. And in this very small class, I’ve got students who are interested in learning more about the science of teaching and learning. And we focus on statistics, and they have applied assignments where they might help me with our learning management system. I think I’ve been inspired by all of the great simulations out there, and I’m going to add an assignment related to those as well. And then I also teach a fully online statistics lab for undergraduate students who are transfer students, who might not have had a lab experience when they took their lecture. And so this uses the same materials that we use in the face-to-face labs or the labs that we use for our basic introductory classes.

Rebecca: So, you participated in a redesign of your introductory statistics classes, can you talk a little bit about this redesign, maybe where it started, and now where it’s ended up?

Matt: The redesign that I was involved with had to do with the lab portion of the class. And it started around 2013, when I was hired to teach the labs for these courses, and also for our research methods in psychology classes. And up to that time, the labs were taught quite capably by graduate students, but there was variability in content and delivery and things like that. And so it was in the interest of the department to consolidate those into a single uniform experience. And that’s what I had the pleasure of putting together. And so what I started doing was building these what I called lab modules, and they would be used in each of the classes. And when I first started doing those, the version I used on Thursday was much different than the version I used on Tuesday. And there’s a lot of evolution that took place. And teaching eight of these labs a week, it was nice to have some development take place that was meaningful over the course of a single semester. And right now that lab manual is still being used. It’s fortunately not something I have to stand at the copier and print, but it comes in a bound book through our NAU bookstore. And it’s got modules that are aligned with each of the chapters in the textbook that we use, as well as very specific freestanding modules related to things like SPSS assignments and power analysis, and a little bit of Excel that’s built in there as well. And the fun thing about putting this thing together, I just loved the creative process. And I benefited enormously from the input from instructional designers at NAU. We’ve got some just phenomenal folks there who had some really important insights to provide that we put into the lab manual. So, it’s got QR codes, for example, in it. And so if a person gets to a particular part in an SPSS assignment and can’t remember how to do this, they can just use the QR code to see a very short little tutorial on how to do that. And I think being able to build those kinds of resources in something like this, make it interactive, I think is useful for the students and for me. It’s a really fun part of the creative process.

John: When you started working with constructing these labs, did you start using simulations right away? Or was that something you’ve gradually been adding since then?

Matt: Well, I started adding them soon after I started building them. But it wasn’t until maybe a year later that I started embedding simulations in as assignments. I was one of those students who really struggled, I probably shouldn’t say this out loud. My first stats class, it was very abstract. It was early in the morning, which complicated things for me. But, what I found was that I really benefited from seeing things to help marry these abstract concepts to real data. And about the time that I started teaching, there was a series of videos that were put out one was called “The dance of the P values.” It was by Dr. Geoff Cumming and it had a beautiful simulation attached to it. And so I was just starting to learn R at the time. And so I started seeing if I could replicate his findings using R and was able to, and that gave me a little bit of encouragement about building them. And at the same time, in 2014, our mathematics and statistics department helped host a International Conference on the Teaching of Statistics here in Flagstaff. It’s a huge international event. And it brought people together who were just marvelous at explaining and had these beautiful simulations. And they also talked about how to teach courses using R. And I just found that whole thing inspirational, in addition to having the pleasure of meeting some of my colleagues in the math department that I might not have met otherwise. And so that opened a whole new window into what simulations are out there, created by these really incredibly bright and capable and devoted teachers to the introduction of statistics and psychology,

John: I have to ask, what does “The Dancing P values” do as a simulation?

Matt: Well, they don’t actually dance. They do move. This is very similar to the way that I dance I suppose. But what it shows was that with small sample sizes, the P values just really were not consistent. And that was a message that was really central to what he was trying to put across. And the way that it was articulated and illustrated, I thought, was really compelling.

Rebecca: And who can argue with that title. That’s the hook. You have a good stimulation with a good hook, you got your attention.

Matt: It is a hook, right? Yeah, even if you didn’t have an interest in statistics, there might be dancing involved. Yeah.

John: And p-values is a concept that students often have trouble with. So, having that practical application, I would think would be helpful.

Rebecca: For those that aren’t familiar. Can you describe what R is?

Matt: Yes, R is a statistical software package that was built from the ground up to do stats and represents statistical graphics. It’s incredibly powerful. It’s free. And it’s also open source in the respect that people build these things called packages for them, which extend their capabilities quite a bit. And so if you can think of almost any esoteric statistical procedure that you would like to implement in your own lab, for example, there’s probably a package out there to do that. And the thing that I liked about it was that it was able to be paired with a thing called Rstudio, which I thought was a nice integrated development environment, which has some additions that allows you to take some of the things that you do on your local machine and put them on the web. So it was really a nice match between what I wanted to do in the lab and what I wanted to put out on the web for people to be able to see.

John: How do your simulations use R.

Matt: That’s a great question. They basically are simulations that I’ve built in R in this add on called Shiny. And so the students don’t see any R code at all. That said, in the labs themselves, I do think it’s useful for them to be able to interpret statistical output from different software packages. So I do give them some R output and ask them to make meaning out of it. But I don’t ask them to do any coding at all.

Rebecca: Can you talk about the difference in students experiencing simulations versus different kinds of exercises you might have had them complete prior to introducing the simulations into your course.

Matt: Maybe right now, I should just define what I think a simulation is. And so this is a very wide net. And I think it’s basically any visualization that allows you to “What if?” questions, to explore and demonstrate connections between abstract concepts and real data. Some of the simulations that I’ll talk about allow you to do statistical inferences as well… so, incredibly powerful. So I think these simulations and the use of the simulations exist across a continuum. I think there are some environments, such as the one in which I am operating, where we use simulations to try to reinforce critical points. So, Central Limit Theorem comes to mind. But there are also some courses where they build the entire semester around the use of simulations, they start them very, very early in the course, leveraging people’s natural inquisitiveness and their desire to see patterns and use that over the course of the semester to develop this deep understanding, not only of the details regarding statistics, but the big picture, how these things are all wired together.

John: Going back to Rebecca’s question, how have students responded to the use of the simulations compared to what you’ve done earlier in some of these lab assignments.

Matt: One of the simulations is one that’s done by the Rice virtual statistics lab. And it’s one that has to do with sampling distributions, which for me, when I was learning it and teaching it and for students still, a difficult concept. Imagine, if you will, just this normal population at the top of your screen, and then three boxes below, and the box immediate below, you have the mean of a sample that’s drawn. And then below that, it gets pushed into what emerges as a sampling distribution based on that sample size. And then the fourth box, the one at the bottom, would allow you to do a different sampling distribution. So you can do two at once, if you will. But this is visually really appealing, because it allows you to see the random sample being taken and where that winds up being put and how those samples aggregate to develop the sampling distribution. So that’s one that I built an entire assignment around, because you can predict some of the values of the sample distributions based on the math. And so it was nice to be able to put all those things together, I think, and the added beauty of this is that you can take that normal parent population, and you can make it one that’s non-normal. And you’ll see when you rerun the sampling distribution that you wind up with, in most cases, a very normal looking sampling distribution that allows you to run those inferential statistics. So it helps connect some of the dots that might not be connected otherwise. And so, while I don’t have any p-values myself to evidence how successful this has been, I have heard a lot of “a-has” when I’m talking to students about this, which to me is the Holy Grail. And they seem to get it with these simulations. As I researched simulations in preparation for this particular conversation, that was something that was echoed in all of the presentations was just the students really getting it and being able to leverage previous knowledge and being able to put all these things together so they can anticipate what happens in the future, when they do other simulations.

Rebecca: There’s something really powerful about being able to observe something and make that rule for yourself rather than just being told the rule that you have to follow. Otherwise, it seems really arbitrary.

Matt: Rebecca, that’s absolutely true. And it’s kind of fun to see these things played out with real world data that is much more compelling to students.

John: Inferential learning about inferential statistics.

Matt: [LAUGHTER] Absolutely, yeah.

John: But those are things, again, that students do have trouble with. They have trouble understanding that the estimators themselves have distributions. And this should make it a whole lot easier for them to see it. I’m getting a lot of ideas here, because I’m teaching an econometrics class this spring, and many of the things you’ve mentioned are things that my students have trouble with.

Rebecca: As someone who just learned some statistics this January, I’m thinking this could have been really helpful. [LAUGHTER]

Matt: Yeah, and so when I look at what I’m doing, I’m really happy to be using simulations. But as I look at the universe of simulations that are out there, I can see that there’s more that I can do, and that I’m really motivated to do after seeing some of these wonderful things. I shouldn’t get too far without talking about some of the wonderful things that are being done on a grand scale with simulations in introductory statistics courses. There are actually textbooks out there, which are built around these. And what they’ll do is they’ll start off early in a semester using simulations, and without giving names to things like sampling distributions and confidence intervals and P values. But they’ll take some real world data. And then they’ll say, what’s the model that you would use to best describe these data and then run some randomization samples to collect data. And then ask, “How likely is it that the original data were from that distribution?” And so that’s a powerful thing because a person doesn’t need to know any statistics coming into that class and being able to make meaning out of a lot of those things. There are multiple textbooks that use this simulation-based inference testing process to great effect and in the links that are going to be associated with this podcast, you’ll be able to go and find those, and just see the rich resources that they have supporting those texts, which actually can be used independently as well for reinforcing specific points that a person might have about their own statistics class or econometrics class. Another thing that I think is useful to point out is the fact that there’s a document that helps guide all of this. And the American Statistical Association has got the Guidelines for Assessment and Instruction in Statistics Education. This is called the GAISE guidelines. And they were revised in 2016. And they provide some very implementable recommendations for improving introductory statistics classes. And some of them are very consistent with what we’re talking about today, increasing the use of technology and the use of simulations, and decreasing that distance between abstract concepts and these students’ real worlds.

Rebecca: Can you talk a little bit about how to get started in implementing these sorts of things into your classes? So if you’ve never used simulations before, how do you start?

Matt: Well, I think, for me, the best thing to do would be to reach out to colleagues who might be doing that. So, for example, here at NAU, colleagues in the mathematics and statistics department are using these. So I would go to them. And I would say, “What have you found most useful, and how might I implement that in the class, given the context I have?” But, you can also do some wonderful internet searches. And I think I’ve curated a few really good starting places for you, in the resources attached to this podcast, I would recommend, for example, just seeing what’s out there by looking at these lists of applets that exists to teach this and to teach this and to teach this. And if you are thinking about something that’s on a grander scale, listen to the video by Nathan Tintall and Beth Chance, about how they implement this simulation-based inference testing in their classes and the rewards they have from doing that. I think getting a real broad sense of what’s available early can be really helpful in figuring out how you might want to do this. But I think that the nice thing is that you don’t have to do it on a grand scale, to start. You can use a single applet to reinforce a point that you might find your students struggling with. There are some other resources. There are journals on teaching statistics that are very, very useful. And I think this cross-pollenization between mathematics and the psych stats classes, is really useful. So I think it’s helpful to get this strong situational awareness of what others are doing to help inform how you might do what you’re doing better.

Rebecca: I think this idea of “one small step” is always a great approach to trying something new, and it seems very manageable. So I can try one simulation and see how it goes, and then feel confident to implement more and more. But that kind of iterative approach seems really helpful. Of course, not everyone has eight sections that they can iterate through all at once. [LAUGHTER]

Matt: Yeah, even if you’re doing it once. I mean, think about the environment that you have. You’re explaining these findings that are very visual in nature, and how all these things are wired together. And I think one of the most important things the faculty contribute to students’ education is not just the facts, but how these things are all linked, and being able to hear from a seasoned faculty member can help develop student’s ability to think these things through in a more expert way, rather than just memorizing simple facts. So I think that not only do we get some sense of accomplishment in putting those things out there for students to use, but the students do as well, because they want to get this too, and they’re much more enthusiastic about content when they think they’re really getting it, or they know they’re really getting it.

John: I think anyone who has been teaching for any length of time knows where some of the pinch points are, the things that students always have trouble grasping. And those would be a good place to start, not just in statistics, but more broadly, in any discipline, where there’s some concepts that students don’t always make the connection between theory and practice or practical application. So those would be the places I would think where people should get started… thinking back on where students are having trouble making connections, because it’s generally the same areas year after year after year. And that information could be used to help us improve our instruction by using tools that make it easier for students to see those connections.

Matt: Those are all good points. And you know, one of the things that was evident when I was looking into this more deeply was the frequency of which these simulations are being used in AP statistics and earlier. And so it’s much more likely now that we’re gonna see students coming into our classes who are somewhat familiar with this way of presenting information. And so they’re going to get it pretty quickly. And so a nice way to make them feel more at home might be to put these things in and, again, to leverage their learning, give them this feeling of self efficacy, that’s going to be really helpful to them as they get into more difficult concepts.

Rebecca: How have you adapted your instructional approach during COVID-19 and teaching remotely?

Matt: That is a great question. And one of the things that helped was that this online stats lab that we’ve put together over the years really made it so that these labs were kind of ready to go. So, in that respect, the materials had been developed as had many that people had put together at the end of the spring 2020 semester. Those were just in the bank and ready to be used in the fall and are even stronger now. There are lots of models being used throughout the country, for dealing with COVID-19 and instruction. So maybe I’ll just drag the one that we’re using so that listeners can get a better sense of how it all fits together. We’re using a modification of the HyFlex system, which is called NAUFlex. And we started using that really in the fall of 2020. As is the case, I think, for many, after spring break of 2020, people went into mostly completely online mode. And so the NAUFlex system starts off with teaching being done completely online. And that allows students to get on campus and be tested and all of the things that build that strong safe infrastructure. And then somewhat later, students are able to opt in if they choose to participate in in-person classes. Now, the online classes that are held are synchronous, so there’s an expectation that students will be there for those. And then we also have COVID adjusted room capacities. And so what that means for some classes, is that they have two groups or three groups of students who can come in, so that we can maintain that distancing. My experience has been that most students have opted to stay online, which means that they show up for the lectures or the labs in either Zoom, or what I use is BB Collaborate, which is built into our learning management system BB Learn. And so that’s how it works for us. And kind of the unsung heroes in this whole evolution have been the instructional designers who helped make this work, and also the folks from Information Technology Services who found hardware that works, that allows us to both interact with our students in the classroom and push it out there to students who may be in places that have varying degrees of connectivity. What I’ve done to modify my instruction, somewhat based on feedback I’ve gotten from students from the spring and fall semesters: what they liked, what they didn’t, how it worked for them, and try to really bring to the lab, this sense of organization and consistency and safety. One of the things, as educators, we’re used to doing is walking into a classroom and being able to gauge energy levels and look around the room and be able to tell who’s got that faraway look, and maybe we need to go back and regroup and cover some material. And some of those cues aren’t there anymore. And so what I’ve found is that I’m much more elaborate in my explanations so that I don’t leave anybody behind. And I tried to foster an environment which makes everybody feel comfortable asking questions when they want. And it’s very rewarding for me when they do. And I know that, in a class of 30, if a student asks a question, there are several others who probably have the same one. So the other thing that I think, and I know that this is something shared by your listeners too, is just that the notion of teaching with compassion, these students are really out of their academic element, if you will, of the in-person classes and going from one place to another. That sequencing is no longer there. And so it’s a much different world in which they need to learn. And some of them learn very well in an online environment and some don’t, but they’re forced into that anyway. And so I tried to have lots of compassion for what the students are going through and try to extend that in my syllabi for late assignments and things like that. And I think I’m a little more careful with humor, because I don’t know the backgrounds of all the students. I have not had that experience with them in the classroom. And so I’m really careful about how I express things so that everybody can get it, and it’ll be something that everybody can accept and understand. The nice thing about the labs is that we have all of these resources that we can use. And so it’s designed to be delivered in an online environment completely. And so students have interactive tutorials they can go to that help them master the content, complete the assignments. As an aside, one of the things that I found helpful in this communication, is that I wear a clear face mask when I’m teaching. Part of it is because I appreciate that myself. I’ve got some hearing loss. And so I’m a little bit reliant on reading lips. And in the classroom, students need cues that “this is important,” or “I’m trying to be funny here.” And I think that it helps the students understand the content and my commitment to their success when they can see my face better. That’s a little thing, but I thought I’d put it out there. I’ve gotten feedback from students that they found that helpful. And the other thing is that having taught this lab so many times, you mentioned pinch points before, I’ve got an idea of where those are going to occur. And so when we’re coming up on one of those, I can be more explanatory, give them a much better foundation for getting past those. So those are the things that I’ve changed.

John: One nice thing that may come out of this whole difficult teaching experience this past fall, is that I think all faculty have learned to be much more inclusive in their teaching approaches for all the reasons that you mentioned. And I’m hoping that that’s something that will continue as we move past the pandemic.

Matt: I agree with that completely. I think that there are really important initiatives to promote that in every classroom. But I do think that the situation which we find ourselves in now does encourage us or motivate us to do a better job with that. And so that’s one of the things that I would throw out as well is this whole idea of universal design for learning, something that I think is really important, and simulations play into that nicely, don’t they? …in that they provide this other way of representing information, content that students can get, particularly those that the students can work on themselves time after time after time until they feel like they really get it. And so I think that this universal design for learning thing is something that we should probably keep in the forefront as well. And some systems really do a nice job with making that easy for us. So for example, there’s an add on to BB learn that takes PDF files, for example, and creates those in alternative formats.

John: Ally.

Matt: And so you’re familiar with that. And for some students, it’s the only way to get that content. For others, it’s a convenient way to listen to content while they might be on the bus or in the car. And so I just think with these simulations, it just feeds nicely into what I think is a mandate to try to make things available in as many ways as possible, so they can really resonate with students.

Rebecca: Do you have any other tips related to simulations that you want to share with folks who might be teaching similar kinds of courses?

Matt: Well, that’s a great question. And while I’ve talked about simulations, one of the things that might be on the border of that, but I think is very useful for incorporating into classes, are some games. And in the face-to-face labs, I used to really enjoy doing like Stats Jeopardy and things like that. It’s a little bit more difficult to do in an online environment. But one of the things that I’ve done in the correlation module is to use a system put together by John Marden. He’s a Professor Emeritus at the Department of Statistics at the University of Illinois in Urbana Champaign. And he’s got this nice little system where he provides students with panels: four scatter plots and four correlation coefficients, and they need to match those. And so what I’ve done in previous semesters, and look forward to doing again, is having a competition across all the sections to see who has the longest sequence of correct panels, the winner of that gets a copy of a book by Tyler Vigan called Spurious Correlations, which if you haven’t seen it, his definitely worth a look. And there’s a website online as well, which is kind of fun. And one of the things that I’ve noticed with this particular gamified module is that students really work hard to get it. And at the end, they do, there are heroic efforts to win that book. And at the end, they really do know how to look at a scatterplot and get an idea of what the correlation coefficient might be.

John: For people who might want to go a little beyond using simulations in class, do you have any suggestions on where they might go to learn how to use, say, Shiny in R in order to create their own simulations? Is there a good reference out there?

Matt: I think there are some good references out there. They’re not, I think, specific to building simulations for teaching psychology. Although I have to say that one of the links that I’ll provide following the podcast will take you to an array of Shiny apps that were built specifically for teaching introductory statistics. And here’s the thing, they were built by undergraduate and graduate students for that express purpose. So this is a beautiful selection that were student built. But I think people who start working in R will look at some of the blogs that are out there and start being able to put these together themselves. But again, I think with all of these things, it would be starting off simple and going from there. Some of the ones that you’ll see out there are incredibly elaborate, and I know that they’re not in my skill set to build at the moment. So I would start with simple and go from there. But in the meantime, take a look at some of the other ones that are out there either to implement directly or try to emulate.

John: We always end by asking what’s next?

Matt: Well, for me, what’s next is a nice organized, gradual wind down of 2020. I think all of us are looking forward to 2021. I mentioned how grateful I am to have the opportunity to talk with you today. In preparation for this, I did lots of looking at things that are out there and I’m just really re-inspired to find simulations to put into my lab wherever I can. And also, as I’ve mentioned, planning on maybe building some assignments into my undergraduate teaching apprentice class about how they can use this. But I think I’m missing the contact with students in the online environment and in the lab, and I’m looking forward to being back in the classroom and using some of these things. But, I think, immediately what’s next, maybe another cup of lemon ginger tea.

Rebecca: Sounds like a good way to spend an afternoon.

John: This has been fascinating, and I’m looking forward to doing more of this during the spring myself. Thank you.

Matt: You’re very welcome. Thanks, John. Thanks, Rebecca. It’s a real pleasure to be here today.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

165. Educational Pipeline

A college degree, especially in one of the STEM fields, can provide students with higher incomes, more stable employment prospects, and more pleasant working conditions. Many  students who could benefit from a college degree face a variety of barriers that prevent them from successfully completing their degree. In this episode, Jill Lansing joins us to discuss what colleges and universities can do to help smooth the educational journey from Pre-K to college and to careers for all of our students. Jill is an Assistant Vice Chancellor and Director of Education Pipeline Initiatives at the State University of New York. Before moving to this position in 2009, she had been the Coordinator of P-16 Strategic Planning for the New York State Department of Education.

Transcript

John: A college degree, especially in one of the STEM fields, can provide students with higher incomes, more stable employment prospects, and more pleasant working conditions. Many students who could benefit from a college degree face a variety of barriers that prevent them from successfully completing their degree. In this episode, we discuss what colleges and universities can do to help smooth the educational journey from Pre-K to college and to careers for all of our students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Jill Lansing. She is an Assistant Vice Chancellor and Director of Education Pipeline Initiatives at the State University of New York. Before moving to this position in 2009, Jill had been the Coordinator of P-16 Strategic Planning for the New York State Department of Education. Welcome, Jill.

John: Welcome, Jill.

Jill: Thanks so much for having me today. It’s really truly an honor and a privilege to be here. And

John: Today’s teas are:

Jill: I’m drinking iced tea because of the podcast, and some water.

Rebecca: It’s a little cold outside for me to have iced tea today, but I have Big Red Sun again. And as you can see, I am drinking my way through one whole canister of the same tea. [LAUGHTER]

John: I am drinking Tea Forte’s Earl Grey today.

Rebecca: So Jill, we’ve invited you here today to talk a little bit about your work on educational pathways. Can you tell us a little bit about your role?

Jill: About 10 years ago, almost 11 years ago now, I started at the State University of New York in this role to really try to help to strengthen the alignment between K through 12 education and higher education and adult education, also encouraging adults to return to college. And it has been such an amazing opportunity, I have to say. It’s amazing, It’s been 10 years already, 11 years to think about this pathway. But, I think the idea was, many of the faculty that are listening today have been doing it forever. So they know what works and what doesn’t work. What we did when we started this work about 10 years ago was to really try to create more opportunities to bring resources to the table for faculty at our colleges. So the idea was really, and I’ve been lucky in this regard, was really to figure out what grant opportunities we could apply for, what pathways were available that we could actually say: “This works. This doesn’t” …to try to avoid places where people have had difficulties in the past. So, I spend a lot of my time actually learning from the faculty at SUNY about what does work, and then also try to bring resources to those things that are successful. Before my role at SUNY, I was working as the coordinator of P-16 strategic planning for the State Education Department. And I worked in many different capacities there, one of the places that I worked, that I loved the most, was working in the Office of the Professions and Higher Education. And in the professions, I was so new, I was a graduate student when I started and I got the opportunity to be in the role of the public management internship program at the time, which was to really try to create opportunities for new people to have the opportunity to learn about the government and learn about public administration. And so the idea was to create a public information campaign around career opportunities in the professions like architecture, dentistry, engineering, nursing. And so I was able to learn about so many opportunities there, there are nearly 45 licensed professionals in New York State, all opportunities for students to have career opportunities in. And also from there, I was able to work in the Office of Higher Education, and also the Office of at the time, it was Elementary, Secondary and Continuing Education. And so I really did get to see all of the amazing work that goes on in the State Education Department to support students, and also the pathways that are available for student success. So it really gave me an idea of what we could do at SUNY to really support students beginning at the very early age of pre-kindergarten all the way through elementary school, secondary school, and then into college. So lots of good experience that led to, hopefully, better outcomes for students,

John: What are some of the barriers or the challenges that students face as they move along their educational pathways in the state.

Jill: So I always like to start with opportunities. [LAUGHTER] But I think that barriers are real. In fact, they very much are real. So last year, I had the great opportunity to work with the National Center for Education Statistics as part of the Data Institute team to kind of look at the data and to find out some of those barriers. So, we did some regression analyses, and one of the things that we knew were true, but it really became evident in the numbers, is that economics and race matter. My dissertation was around college choice. And it’s amazing to me that, when the odds are stacked against you when you’re looking for college, that puts you in an unbelievable place where you’re really trying to figure out how to go about college choice, how to go about succeeding in college. So we really do try to look at that in the beginning. But barriers are, you know, your a risk for college. The Education Trust has been an enormous resource for us. They were looking at things like student suspensions in high school. It’s a demoralizer, it also really places students behind where they could be if they didn’t have to experience those kinds of challenges in their high school. So, I would say that the barriers are real, and we’re trying, through our programming, to really address those barriers. I also think another barrier is, and I know a lot of our faculty are interested in this, it has to do with the growth mindset and to what extent is students believe they have the ability to succeed and I think that pervades all economic lines and it’s really an important thing to keep in mind because, as faculty are thinking about how do we work with new students, and they’ve given me so much advice, as we look for money and programming, how do you really try to build a student up and help to empower them as they try to be successful in college? So, I think the barriers are real. And we really need to think about that. And it’s good to have this opportunity to talk about what we can do to really make a difference.

John: What does the data suggest is effective in helping students overcome some of these challenges?

Jill: That’s a great question. And there are so many data points that help to think about what the challenges are, and how to create change around the barriers. So one of the things we’ve been working on right now in collaboration with the Community College Resource Consortium, and also Jobs for the Future is the idea of Guided Pathways. For a long time, students would come into college, particularly in community colleges, and take courses that they were interested in, but didn’t necessarily count toward their major. So, they would take some courses, they weren’t sure what major they were interested in studying, and then they would take some courses, and then they may or may not add up to the total number of credits they need to graduate. So, we’ve been trying to really focus in on college completion and graduation. And that consortium has led us to become part of something called Strong Start to Finish, which is a national initiative to try to get students to complete gateway courses in their first year of college. So, really try to focus in on what matters, also try to avoid remediation through co-requisite coursework, and also opportunities where students don’t have to kind of get behind in their classes and can really kind of get up to speed on what they need to complete their gateway courses. So, that’s been very helpful. And also the Guided Pathways Initiative. If you talk to college presidents across the state, they are head over heels about Guided Pathways. So students enroll in college, they enroll in a meta major, instead of necessarily a particular major, and really try to take the courses in the very beginning of their curriculum that would lead to success in the health sciences professions, Hospitality and Culinary Institute, business professions. So, it’s not we’re asking high school students to make a decision right away about their major, but kind of “tell us what you’re interested in, and then we can connect you with the courses that would actually add up to success.” Because, I think, where the Community College Research Consortium has found many students fall away from college is when it’s not adding up to their degree program or to their area of interest. So, that’s been very powerful. Another area is our P-Tech and Smart Scholars programs that have been guided by faculty, and I’ll say the Faculty Council of Community Colleges, in the very beginning of any early High School initiatives, put in place standards for what quality looks like. And one of the things I was going to talk about later, but I’ll say it now, because it’s so critical, is that rigor matters. So, when faculty put rigorous standards in place, it inspires students to succeed. And I think that they feel a different benchmark about where they’re supposed to be in their collegiate life. So I think rigor matters, and I applaud all those faculty who do this day in and day out, because it’s hard. It is hard in times of COVID-19, it’s hard every day, but just to keep the rigorous standards in place, and then support students on that pathway really matters. So, those are some opportunities, many more. We were able to, a couple of years ago, win a grant from the National Science Foundation, and also from Battelle Institute to help our graduate students work with middle school students to be, in their words, deputized as scientists beginning fourth, fifth and sixth grade in high need school districts. And having the connection between the graduate students and the middle school students was powerful, and all this was under the direction of faculty in our colleges. So, we had our scientists, across SUNY, overseeing this effort. And their ability to really create opportunities for these middle school students, who otherwise would never have had the chance to become scientists in laboratories at our SUNY Colleges, was huge. So I think that there are so many examples that I could share with you, but the ones where I think faculty, graduate students, aspiring faculty, put their passion and their heart into it, as well as their intellect and rigor, those are the ones that really matter for our students.

Rebecca: I wanted to follow up on the picking a major piece a little bit, because I think that is a real struggle for students when they head into colleges with an expectation of like, “Well, what are you going to do when you grow up? What are you going to study?” And the names of majors are so abstract to students, and there’s many careers that students have no idea even exist, and if you’re already in academia, you might have a clearer vision of what those things can lead to, but our students just don’t, and there’s no reason for them to know that

Jill: Exactly. When I was an undergraduate student, physical therapy and occupational therapy were big majors in my school and I didn’t know what they were at all. So the idea of helping students to come into a major in health sciences really will allow them to understand what opportunities are available to them. Another resource I should talk about is a resource called the Empire State STEM Learning Network. And in my career, one of the things that I’ve learned is that often when grants are distributed to an organization, the project period ends when the grant period ends. And one of the lucky things that I’ve been a part of, thanks to the regional leadership across the state, is something called the Empire State STEM Learning Network. And the idea for the STEM Learning Network was to bring together educators and business leaders in regions across the state to really focus on the educational pathway that leads to jobs in healthcare, in manufacturing, in Information Sciences, and architecture, and so many other fields. And I will say that one of the things that made the Empire State STEM Learning Network successful was the commitment regionally to develop partnerships to help lead students into careers. Because, you’re right, students don’t really know what they want, necessarily, in the very beginning of their life. But once they have the experience to be exposed to these opportunities, then all of a sudden, that could open the door for them and really spark an interest in them that we hadn’t thought about before. And even if they don’t go into architecture, maybe they would be interested in something related in engineering, or in teaching architecture. So I think the spark matters. And you’re absolutely right, Rebecca, it’s a really good question, because how would you know what the opportunities are for you? But really trying to create a level of interest with the students and also a level of “I can do it, this is real for me,” that really does matter. And I think our faculty, I mean, I’m talking to our faculty, and I’m so honored, because our faculty are often the drivers of everything that we do, and you know what drives students, what motivates them, and how to really think about what their needs are. And I think young high school students, I am always amazed at what they can do when they see exemplars of success, and also when they understand rigor. But also, I would say, in the way of how to be successful, I think we need to meet students where they are. One of the things that we learned from our colleagues at the Community College Research Consortium was that remediation can be very detrimental to students… the word remediation, the idea behind “I need to still catch up.” But, instead, trying to help to develop the skills that the student is interested in. So astrophysics is a big dream. But really, if you start, piece by piece, level by level, it can be a possibility, or another career option in that same field or discipline could also be possible for students. So really trying to motivate the students to success, I would say, but you’re right, students figure out what they want right off the bat when they’re coming to college is really a challenge.

John: What types of approaches have been effective in overcoming gaps in prior education? Because one of the main issues is that there’s quite a bit of disparity in the amount of training that students receive in their high schools, and that’s tied to a whole host of issues related to funding and housing segregation and other issues. While remedial work can be discouraging, or just even the term “remediation” is discouraging, what strategies have you seen that have been effective in helping students bridge any gaps in their background, to help them get to speed more quickly without being faced with a whole sequence of courses needed to enter, say, the STEM fields?

Jill: Precisely. So, good question. I would say to look at the following programs for good benchmarks and good outcomes data, Educational Opportunity Program (EOP), the EOC, the Liberty Partnerships Program, also very powerful. STEP and CSTEP have been incredible about meeting students where they are and getting the students where they need to be P-Tech has been amazing. And I think the best P-Tech programs and the best early college programs, the best programs that we have across the board, are those that are led by faculty who are determined to help students succeed. For example, in our community colleges, which are open access, and many students come to as the first opportunity they have to enroll in higher education, where there are collaborative opportunities for the students to participate in STEM research opportunities, that is so powerful, it almost closes the gap in the way of letting students have the opportunity to become scientists. I was talking earlier about programs where we worked in our middle schools to help our graduate students come to the middle school students and talk about “Here are opportunities for you to become a deputized scientist.” So, then when that student actually comes to community college, and they’re saying, “Oh, I’m gonna research climate change with my faculty member,” or whatever the research opportunity may be, it becomes a way for the student to see themselves in that career. And then they understand the pathway to the career. So I can’t say enough about the hard work of faculty in this effort, because (and this is my own experience as well), is that faculty in my life have taught me how to think. And I often tell students in high school that college is a place where they can learn to think. And they think that they’re behind the eight ball in some ways, and that they feel like they have to tell the professor about how much they know. And, instead, if you listen to faculty, and I’ve worked with both of you, and you’ve been teaching me about, like, how to think about this, how to think about talking to people, how to think about getting a message out, and I think often new college students feel that they have to prove themselves versus they can learn something, and you’re actually willing to help them. [LAUGHTER] So, I wanted to share an example. A couple years ago, I was interested in the process of statistical process controls as a way of thinking about, “Well, if we are trying to achieve student success at scale, how can we look at numbers and data to help us do that?” So, I was thinking that statistical process control could be helpful in higher education, about seeing trends over time, seeing when there are differences, and why there are differences. Maybe, perhaps, that the spring semester wasn’t working out for students because of X, Y and Z. And the faculty, again, provide such leadership, maybe we could learn from one another. So anyway, I went to this course on statistical process control that was offered by SUNY Polytechnic Institute. And there with me, were three Community College faculty. And they were interested in statistical process control, because they wanted to be able to integrate that curriculum into their coursework. They said, “So many students are coming to us, and they’re interested in manufacturing and machining. And they keep talking about statistical process control.” So they were seeing how can we integrate those into our work, so that the learning opportunities are more aligned with their goals, professionally and personally. And so I thought it was really amazing that these faculties sought out these professional development opportunities to really try to make learning matter for the students and make it something that they were interested in that they could actually apply to their jobs. And I think we talk a lot about workforce, we also are talking about educating a citizenry, and especially with the political climate change, I think we really have an opportunity. We’ve learned a lot in 2020, about COVID-19, about our responsibilities to citizens in this country, African-American citizens in particular. We focus a lot about Black Lives Matter. And after the passing of George Floyd and so many other unfortunate events this year, we’ve learned a lot about why we really have to be focused in on those liberal arts opportunities, and also about governance, and religion. And we need to have active citizens in this country. And I think that is equally important. Often, we talk a lot about STEM and job opportunities, but we have to think about the big picture. And I think that our faculty are leaders in that work. And so, to the extent that we can learn from them, and, again, create opportunities that help to advance our agenda. is really powerful.

John: A couple of times you mentioned P-Tech, for those listeners who may not be familiar with that, could you describe that program and how it works.

Jill: So the P-Tech program has been a program that is being led by IBM, and they’ve been really a front runner in this work. And they’ve put a lot of thought into how to connect high school faculty, high school programs with our colleges, and then also with workforce opportunities in high need fields. And I think that it’s been an awesome opportunity for our students. They are able to enroll in the P-Tech program as high school students, free of charge to students and their families. And I would also add, at this point too, that one of the big assets of SUNY is affordability. And so there are often times where students can actually earn SUNY degrees free of charge. And that is really incredible to graduate debt free. That also reduces a lot of the burdens that we talked about in the very beginning, around economic burdens, challenges around race, all of these opportunities are now readily available to students. So that’s been very powerful through the P-Tech program. And I think the P-Tech program is something that we can model across the board. And we have tried to do that in collaboration with regional BOCES, regional K-12 leaders, to actually develop a scope and sequence of courses beginning in grade 12, that continues on to higher education and then into the workforce. So, what happens is K-12 leaders, community college faculty (also many of our technical colleges also have P-Tech programs), and our business leaders come around the table together. And they think about what would a student need in order to be successful in life and in business in the long term. And I think what it develops is a real true camaraderie that allows successful programming development to happen. What is fortunate about the P-Tech is that it is enabled by grants from the State Education Department. So it does provide opportunities for faculty at K-12, faculty at higher education, business leaders to come together. But that seems to be very powerful when college faculty can also work with K-12 faculty to find out what the needs are, where the gaps are, and how they can work together to try to mitigate those gaps. So, often what I heard when I interview faculty in our community colleges in our technical colleges is that often they are unaware of some of the challenges that K-12 faculty face and then K-12 faculty are so eager to learn more about all of the expertise that community college and our technical colleges and also even graduate programs have to offer. So, when they have the opportunity to speak to one another, then all of a sudden, they’re all on board on changing curriculum and helping students to succeed. So I think the magic in the P-Tech is about this collaborative experience. I often find that regional collaborations are really also very powerful in that our faculty in our regions know what kind of high school students are coming from. And then when they talk to faculty and they can identify, you know, maybe the student didn’t have a fourth year math course in high school, or maybe there are some skills that for some reason the student wasn’t able to develop, and especially this important in COVID-19, because we’re hearing students that are struggling. So I think the connection between high school and college and then regional workforce leaders are very important. And we talk a lot, too, in our work, about regional economic development, and helping to really strengthen New York as a whole through our regional workforce capital. So we’re looking at that. And again, I would say, to create and develop good citizens of the state and of the nation, I think it’s very important. So, I feel like the opportunity for faculty at the K-12 level to connect with our faculty at SUNY, and then also with our workforce and industry leaders (many of whom are SUNY alumni), is a very powerful connection.

John: What are some examples of good collaborations between colleges and P-12 programs? What has worked in terms of helping to bridge the gap in terms of creating an environment of good communication? You mentioned P-tech. Are there any other models that have worked well?

Jill: I can name so many examples of what has been successful. I’m thinking about also the gaps between community colleges and our baccalaureate degree granting programs and our graduate programs. I was thinking offhand about Binghamton University and its collaboration with Broome Community College, really trying to help students who perhaps in the beginning didn’t meet the admission requirements for Binghamton, but had a lot of promise and a lot of opportunity for skill development in STEM. And so they entered into a collaboration with SUNY Broome to help those students to develop the skills at SUNY Broome that would make them eligible for upper-level credits at Binghamton University. So there’s so many examples like that. Our early college high school programs, in addition to our P-Techs are also powerful for making the connection between high school and college. We’ve also done a lot of work with our teacher education programs. We’re really very fortunate to have the opportunity to work with the deans of all of our SUNY teacher education programs, to help them to develop more practical experiences for their teachers to work with the students and try to bridge the gap between high school and college. And I think that that has been so successful, our teacher education programs. And the faculty there have really led the way in terms of helping us to better understand K-12 experiences and create clinical opportunities for prospective teachers. So that has been huge because educator preparation is so essential to really helping to bridge the gap between K-12 and college and they’ve come back to us with so many ideas and opportunities, things that we’ve been able to win grants for and bring resources into the classroom about. So I’ve mentioned the Empire State STEM network before and they have just been amazing. So our colleges will frequently partner with our K-12 leaders to do things like in Buffalo, there’s something called the hands-in-hand program where the students were able to use 3D printing to develop hands for other students internationally, who actually need hands or limbs for some reason in their life. And they’ve been able to actually generate that through 3D printing. The other thing is computer science, cyber security. Our students right now in the P-Tech program are working with SUNY-Orange to develop skills in cybersecurity. So P-Tech is one example. But there are so many, and again, the graduate students that are working across the state with our K-12 students in STEM is also pretty awesome.

Rebecca: The ingredients of successful programs seem to include some collaboration between faculty at different levels to develop curriculum and mentorship, with either faculty at the college level with students, or college students with high school students. Are there other ingredients that we should focus on or highlight as being important to the success of these kinds of programs?

Jill: I think you said it so well, that it’s mostly customized opportunities for students and the places where we find success. And I was just reading the Journal of Higher Education this month, and it was talking about STEM student success and STEM student success at the community college level. And it was really focused around this idea of faculty in the very first year creating an experience for students that gets them engaged in their learning experience across the curriculum… and I think that idea of faculty engagement at the very beginning of learning experiences that are aligned to the student’s personal and professional goals. And again, it comes back to rigor. The students are so inspired by rigor, the students are so inspired by the benchmark is high, and these are the things that I can learn to be successful and to be like faculty. And so I just think faculty role modeling, faculty advising, academic success is so important. We also have the great benefit of SUNY of so many amazing student affairs professionals who also provide coaching and leadership and just a really well rounded approach to student development. Like I’ve mentioned before, the EOP programs are so successful because of this. So I think you’re right, Rebecca, the magic ingredient is collaboration. It’s also about rigor. It’s also about looking at the research and data and talking about what programs can actually generate data to improve success. I also think I’ve learned a lot from faculty about the idea of sort of an interdisciplinary approach to learning I often find that for me in education, if I go and look at that economics literature or the anthropology literature, or the political science literature, I can also learn a lot about how to help to bring these other resources to the table to really think about what success looks like. I recently had the opportunity to work with some medical professionals around the development of guidelines and the rigor to which they looked at guidelines and looked at evidence in the field to make judgments about what they should do, and the way of supporting medical advancements was so powerful that I thought we need to benchmark this in education. So I think looking at what the data tells us, and we’ve had our colleagues, like I mentioned before, at the Community College Research Consortium, at Jobs for the Future, at Achieving the Dream, and so many more places looked at the data for us. But I think that we need to be focused on the long term. And we need to think about what is the data telling us and how that can continue to improve our practice. And then I often think about the importance of qualitative data, what the faculty are telling us in the different disciplines. So that’s why it’s such a great opportunity in this position, because I’m working with faculty, often from economics, from English literature, from science, from political science, from public administration, so many different perspectives. But they all really come to bear when we talk about student success. So that’s been really tremendous for us here at SUNY,

John: Many of the programs you talked about have been at the community college level. And these don’t seem to be quite as common yet at comprehensive institutions. What can traditional liberal arts colleges do to reach out in the ways that community college have been doing for decades? What else should four-year colleges do to emulate the success that many community colleges have had in bridging some of these gaps?

Jill: I really think that you do it well. And I think this is where the SUNY advantage comes in. Because the quality and the dedication of faculty at the four-year colleges and the comprehensive colleges and the University Centers are amazing.

John: What can faculty at four year institutions or University Centers do to reach out to students in high schools to help them see the potential that’s available to them.

Jill: When they have an opportunity to listen to podcasts from faculty or actually engage in their research, it becomes really powerful to these students. And I think part of the challenge with faculty is really in finding a way to kind of create an inroad with these students. And so that’s why I was talking earlier about research opportunities, or really introducing students. I think you also have to help the students learn more about your expertise and try to again meet them where they are, trying to contextualize learning so that the expertise that you have at the comprehensive level and the university center level really can be matched to something that they might see an interest in long term. I’ve worked with an organization called the Army Education Opportunity Program. And this was through an opportunity that we had with Battelle. And the army is trying to recruit the future researchers, scientists, engineers. And what they encourage students to do was to work in groups with a faculty member and a science mentor to address challenges in their own local community. So, what kind of STEM challenges would they need to accomplish in their community? So, they look at things like recycling or safety and parks or child safety. But when the student can see how your expertise at the faculty level applies to their lives, then they are so engaged and faculty will tell me that they’re often outperforming them… that we have students in Long Island, for example, that are applying for patents on downloading data from their computer to a flash drive in instantaneous time, because they actually were listening to a faculty saying, “Ah, this takes so long to download that I actually need some help with this.” So they figured out the magnetic structure behind the thumb drive, and they actually created something that would help them to expedite the time in which they could download data. It was a big win-win, because the faculty was overjoyed. And also the student was able to learn the fundamentals from the faculty member to actually create innovation and to create change. So I think you have it, I was going to talk a little bit about the college choice process, which is so interesting to me, because students in high school have so many options, so many choices, there are 270 plus colleges and universities in New York State alone. And so when you think about the whole universe, and how students go about selecting a place where they can see their baccalaureate success dreams come true or their associate dreams come true. And think about graduate studies, what are they looking for? And based on the literature, after they go through the first predisposition phase, and then the search phase, they actually always talk about student culture in terms of why they make a decision about one college over another. And I also have seen literature that suggests that retention rates are impacted by the choice process around how much the student believes that they fit in. And so it’s hard for faculty at that point to connect with students. But I think that, to the extent that they could appreciate students’ development process and understand that their professional and personal goals and really try to connect them with their own research interests or their own academic interest, I think that can be very powerful in and of itself to really connect the student with the college culture, and with the rigor and academic excellence expectations that faculty have. I think it’s powerful. The other thing I think, is that Chancellor Malatras has been very committed to faculty diversity, because I also really believe that students need to see themselves in their faculty and see themselves in their mentoring experiences. So faculty diversity is also very critical. Our graduate students have been helpful with that as well, to try to make the connection between the faculty ranks and also with our colleges. The Education Trust has some amazing data around faculty diversity at the college level and also at the K-12 level. Because I really do think we need to do more in both arenas to really achieve change for our students, because our students are changing. And the other thing that drives success, and the main thing that drives success is student voices, because we have to hear from our students. And the more that I work with students and work within the framework of student success, I keep going back to the literature to find out what it is that motivates students of various backgrounds, how to really hear their voices, and how to integrate their voices into our activities to really strengthen the outcomes of students in our colleges. So student voices are fundamental to this work.

Rebecca: I think you’ve given a good view of how much has already been done, but also how much there is still to do. So. we usually wrap up by asking what’s next? [LAUGHTER]

Jill: So, so much exciting things are next, I think. In our office, we have an opportunity that was made available to us through the Bill and Melinda Gates Foundation around trying to strengthen communications, advocacy and success tools for students. So the idea would be: how do we really make students understand the value added of SUNY, the value added of the work that we’re doing to try to improve college completion rates, and student success Initiatives. So I would welcome faculty feedback on this. As I said before, faculty have always driven the work that we’ve been leading in our office. And so ideas around how we can better communicate these opportunities that are available to our students and to our parents, also to our guidance counselors, and then policymakers and government leaders about what is the SUNY advantage and about how they can have a part in this, I will say that if you look at SUNY, affordability matters, affordability is huge. The student debt crisis is alarming. And that’s where SUNY has the value and the added advantage. Also, the graduation rates of SUNY often surpass the national average, and are very amazing in terms of what the experience is that post grad and into careers that we’ve been able to achieve for students. And the alumni networks are so strong. And so someone asked me the other day that the SUNY Community College Trustees have been very powerful, great advocates, and they asked, “Can you provide an example of what it looks like to be student of Guided Pathways? So, what is the outcome? And how is it different than it was before? So I guess if faculty could share with us their ideas on first of all, how to continue to improve outcomes around student success, and also how to improve communications, because often faculty are doing this in our classrooms, but they’re saying how do we get the word out that this is so good, that we’re really making a difference, like this student came to us not knowing all the opportunities, and now is head of the class or now is really making huge changes in their field? What are those things that we can highlight, also transfer opportunities, students that might come to community college and then go to our baccalaureate degree institution, or I’m always interested in those that go on to graduate studies, our business and industry partners have been trying to create programs that meet students at the high school or community college level, and then drive them all the way to be our future engineers and scientists and researchers and physicians and faculty and all of the great career opportunities are available students. So what are those pathways look like from your perspective? And I think with this funding, and with this opportunity to really try to communicate these outcomes, we could really have some power that would drive us into the future. So I appreciate the question. And we certainly welcome your input and opportunities.

Rebecca: Well, thank you so much, Jill, for sharing your experiences with us.

Jill: Well, thank you for the opportunity. And I would just say again, our faculty at SUNY are top notch, number one, and really have driven student success as we’ve started this work where we really put an anchor into the ground to try to help to strengthen the alignment between K-12, higher education and workforce, and citizenry. And I would just keep encouraging your great ideas, and we hope to continue to work with you into the future. So thanks for the opportunity to talk to you today. And it’s really an open door. So please keep your ideas coming.

John: Thank you. And we will share your contact information if anyone has any ideas that they’d like to sharee. Thank you very much.

Jill: Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

125. Peer-Led Team Learning

Many studies have found that peer-led team learning is effective in helping students learn. In this episode, Dr. Christina Winterton joins us to discuss her study of the factors that result in more productive relationships between peer leaders and the students they work with. Christina has returned to SUNY Oswego as a full time visiting professor in the Department of Biological Sciences, and was previously the Associate Director of the Collegiate Science and Technology Entry Program at Lemoyne College.

Show Notes

  • Julia Snyder, Research Assistant Professor in the Department of Science Teaching in the College of Arts and Sciences at Syracuse University
  • Jason R. Wiles, Associate Professor of Biology and Science Teaching in the College of Arts and Sciences at Syracuse University
  • Peter Arcidiacono (2020) 122. Differential Grading Policies. Tea for Teaching Podcast, February 26th.
  • Student Assessment of their Learning Gains (SALG)
  • Michelle Miller, Director of the First Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University.
  • Winterton, C. I., Dunk, R. D., & Wiles, J. R. (2020). Peer-led team learning for introductory biology: relationships between peer-leader relatability, perceived role model status, and the potential influences of these variables on student learning gains. Disciplinary and Interdisciplinary Science Education Research, 2(1), 1-9.
    https://doi.org/10.1186/s43031-020-00020-9
  • Ryan Dunk, Graduate Student in Biology in the College of Arts and Sciences at Syracuse University.

Transcript

John: Many studies have found that peer-led team learning is effective in helping students learn. In this episode, we discuss a study of the factors that result in more productive relationships between peer leaders and the students they work with.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Fiona: My name is Fiona Coll. I teach in the Department of English and Creative Writing here at SUNY Oswego and this is my turn to sit in as a guest host.

John: Our guest today is Dr. Christina Winterton. Christina has returned to SUNY Oswego as a full-time visiting professor in the Department of Biological Sciences, and was previously the Associate Director of the Collegiate Science and Technology Entry Program at Lemoyne College. Welcome, Christina.

Christina: Hi, thank you for having me.

Fiona: Our teas today are:

Christina: I am drinking green tea infused with raspberry, pomegranate, and strawberry.

Fiona: Amazing. I am drinking Tazo’s Refresh Mint tea with a wink.

John: And I am drinking Republic of Tea’s Spring Cherry Green tea. We’ve invited you here today to discuss the dissertation research that you’ve done on peer-led team learning in an introductory biology class. Can you first describe what is meant by peer-led team learning?

Christina: Sure. Peer-led team learning is an active learning technique that typically gets implemented in courses where there’s going to be a high enrollment of students, so a lot of the introductory sciences. I specifically helped with the peer-led team learning at Syracuse University. So the intro to bio course there had between 600 and 700 students for the first semester, and then it went to about 500/400 for the second semester. So with all those students, we still had two lecture periods. So all those students were in one of two lectures. So to supplement the lecture, we found a bunch of research on PLTL, and they implemented peer-led team learning at Syracuse University. And what it is, is it takes students who have recently and successfully completed that introductory biology course, they become peer leaders. So, they need to get a B plus or higher in the course, but then they also have to enroll in a one-credit course with a learning specialist. And our learning specialist at SU is Dr. Julia Snyder, who actually was a huge factor in implementing PLTL at all with Dr. Jason Wiles, who is the professor of intro to bio. So, the premise of peer-led learning or PLTL is you take the student who just recently and successfully completed the course, you give them a course about how to be a leader. So, it covers pedagogy, Bloom’s Taxonomy, different teaching styles. And then after they meet with the learning specialists, they are on their own to lead a 55-minute section of the supplemental peer-led team learning course. So, they get modules that they work through with the students. And it’s different than a teaching assistant or a faculty because, first of all, there are no grades involved. The peer leader’s not grading the students, they’re just there to help, and a big difference, a big key, is that the peer leaders are not provided answer keys to these modules. So, they’re truly acting like a fellow learner, a fellow student. They’re relying on what they remember, what their notes are, and they’re just facilitating the conversation. And as a group, the current students are working with this peer leader to construct new original answers and in that they become friends, and they talk more about their feelings and what they really think of certain aspects and how to work through it, and the peer leader is able to relate and share stories because they just took the course.

Fiona: Could I ask some very practical questions about how this works?

Christina: Yeah.

Fiona: So how big are the groups, the peer groups?

Christina: So each group will have six to eight current students with a peer leader.

Fiona: And these are sections that are scheduled along with the massive lecture courses.

Christina: Yeah. Yep, they’re supplemental. So it’s a co-enrollment.

Fiona: And they’re required?

Christina: They are not required. The literature breaks it down with different incentives for different universities to have the peer leaders and the students both choose to be in peer-led team learning. So, for the leaders, typically it’s a learning credit or a leadership credit, and it’s a resume builder. It’s a way for some of the students who are going to maybe take an MCAT to refresh themselves on the material. And then for the students, a lot of the times it’s connected to maybe a portion of extra credit. So that’s the different carrots.

Fiona: Yeah, that’s fascinating. And I want to talk very, very much about the way you assessed this program, but could you give us maybe an example of a teaching technique or something that the peer leader might learn in the one-credit course you were mentioning they’re a part of?

Christina: Sure. I was lucky enough to be able to fill in for Dr. Snyder sometimes when she was out. One of my favorite lessons that we did was as the instructor for the day, I stood up in front of the classroom and all the leaders all had different pieces of paper. And I said, “Okay, listen to what I say and do it as I say it.” And then I said aloud instructions, “Fold the top of the paper to the right towards the center, make a triangle on the side, do it again with the other side,” and I verbally gave them instructions on how to fold a paper airplane. Well, obviously, that didn’t go so great. So I said, “Okay, keep that to the side.” I didn’t even tell them what they were making. They were like “What am I doing?” So they put it to the side. The next thing, I gave them a handout that it said “Step one, fold this” and it was just written instructions. And then for the final round, I put it on the doc cam and folded it with them. And then we looked at the three different outcomes, and we talked about different strategies used to teach or explain. “Did everybody do it the best the last time? Did maybe somebody do it better the second time? Is that right? Is that wrong?” And we talked about different ways to convey the same information to different groups of students when they’re actually leading a session.

Fiona: And do the peer leaders who are forming a cohort of their own essentially, as they’re going through this…

Christina: Yeah.

Fiona: Do they report back through the semester with questions or problems, does that fold into this?

Christina: Yes, I know that Dr. Snyder, as part of the course, she had them keep journals. And so they would reflect on the session. And each leader really only had one group of students. So they knew their names, they knew their habits, they were able to write really good reflective journals and keep notes on “They liked this module or they didn’t like this module. You know, they really liked the debate that we did this week.” Or next week they’d say, you know, “Those concept mappings?…the students struggled, but I think I personally saw them learning even though they didn’t.” So we were able to get really good insight on the models just from those journals.

John: There’s a lot of research that shows that peer instruction can be really effective. What makes it so effective?

Christina: What makes it so effective is actually what my research honed in on, because I was obviously a member of this Biology Education Research Group at SU and I saw Dr. Snyder and Dr. Wiles doing all these really cool things with PLTL. And I know it got results in the form of the research coming out was saying the students enrolled in PLTL had higher grades than their non-PLTL peers, because like you asked, it’s not mandatory. So, some students were in it, and some students weren’t. And you’re able to see the difference in final grades. And I just wanted to know “Why,” right? “What is it about the session? Is it the environment? Is it that there’s no faculty? That it wasn’t in a lecture hall? (It was in a small library room.)” So I just wanted to figure out what about the interaction made it so effective, and I thought like you’d ask that it would have to have something to do with the dynamic between a student-to-student rather than a student-to-faculty. So I was reading up on some literature trying to find out in particular for the STEM fields, what makes peers help each other more. And that led me to some research on role models and relatability and seeing someone you can relate to be successful or do something you want to do. So that led me to develop my question of “Is the PLTL setting enough to leave a student calling their peer leader a role model by the end of this? And what makes up a role model? And did it have an effect?”

Fiona: Don’t keep us in suspense.

John: Although maybe we want to save the answer until a little bit later.

Christina: That’s another question. Yeah.

John: So make sure you listen to the end of the podcast to find out what happened. [LAUGHTER] So you talk about, in your dissertation, the importance of education in the STEM fields. Why is it important to focus our attention on STEM programs?

Christina: So it’s important to focus on STEM programs in particular during the first year or so of college when the students are choosing their major and deciding if they want to pursue STEM after they get a sample of it because the highest rate of students leaving STEM comes between the freshman and sophomore year. So a cause could be what they perceive to be an unwelcoming, large intro course. They come from high school where it’s a smaller class, students know each other, the teachers know them, then you go to the lecture hall and nobody sees you and attendance doesn’t matter, you know. So we can make it feel welcoming and inclusive and that your voice is being heard. And if you have a question, don’t be afraid to ask it and other people struggle as well, then maybe we can keep them and say, “Okay, I still want to pursue this. I got through first year and now I’ll be able to choose an elective that hones in on my specific interests.” So, we particularly don’t want to scare them away with large intro courses, because the overall number of degrees conferred in STEM in the United States is lower than China and Russia and other countries comparable to us.

Fiona: And we also know that the struggle to complete let’s say, STEM is not evenly distributed across different sorts of student groups. That attrition that you described…

Christina: Absolutely.

Fiona: is exaggerated in certain underrepresented groups.

Christina: Absolutely, actually, again, we’ll speak to some of the amazing people I was able to work with while I was at SU in that Biology Education Research Group, one of their studies specifically showed that peer-led team learning closes the gap between underrepresented minorities and their peers in the course. So peer-led team learning can provide this avenue to level out the playing field.

Fiona: Did you have any experience with peer-led learning yourself in your own education?

Christina: I didn’t actually, no.

Fiona: Everything you’ve described makes so much sense. Motivating students, making learning spaces accessible to students.

Christina: Yeah.

Fiona: And I can’t help but think what would be different about my educational path if I had had peers involved in the process in some way?

Christina: Yeah. And I actually went to SU myself, and this opportunity wasn’t available. So it’s fairly new.

John: I started doing it after quite a few years ago, maybe 12/13 years ago, I went to a conference. And I wish I knew who presented this. But there was an experiment, and I believe it was a chemistry class, where they introduced your instructors and they measured the students’ performance for everyone, including the TAs. There was a larger group, and they had larger sections, but they had graduate TAs and they were using undergraduate TAs for the first time. And they measured the increase in performance of students from the beginning to the end, including the TAs, and what they found is among the undergraduate students in the class, the graduate student TAs, and the undergraduate TAs, the undergraduate TAs actually learned the most, and the students who work with the undergraduate TAs had larger learning gains than those who work with the graduate TAs, which is very consistent with the types of things that are being found here. There have been quite a few studies, and I wish I knew some of the citations off-hand, that have found similar effects. It’s interesting. I should also note, in a podcast that we released on February 26, we talked a little bit more about the dropout rate where we specifically looked at the gender difference in dropout rates, and that women are disproportionately likely to leave the STEM fields in response to the relatively lower grades that they receive in the STEM field compared to other disciplines, even though their grades were higher than males in the classes, at least in that study. So the lower performance that many students experience in the STEM fields does seem to be a pretty significant deterrent for, as you said, underrepresented groups. Also, for first-gen students, it seems to be pretty significant in a number of studies. So we’re losing a lot of people who could be really successful in these fields, and as you noted, we’re falling behind other countries. And as we close our borders to the people coming from other countries that could seriously hurt us in the STEM disciplines if we don’t bring more people through here or eliminate some of the barriers to foreign students entering.

Christina: Actually as part of my research, when I was delving into some of the psychology research that came up because I was looking at perception and role models, so we went past biology and education into a little bit of the psych realm, and there was one study that just laid it out really clearly like exactly what you’re saying. And it just got me so interested because their study specifically aimed to put students in a situation and give them exposure to things that would specifically counteract the two prevalent STEM stereotypes and identified the two prevalent STEM stereotypes as: 1. STEM is for white European males, and 2. STEM is only for those who are innately gifted at it. So, that really caught my attention because as an advisor, and as a professor, the students have come up to me a lot of times in STEM courses and said, “I got a bad grade on this. I don’t think I’m good enough at this. I studied for 50 hours that week.” And they always put an emphasis on how hard they had to work to get their grade as if it’s a bad thing. And it really just speaks to that that they truly think that people who are successful in the fields are just born with it and that even your professors never had to put in work. So now after reading that article, I always try to directly counteract those stereotypes for them as well and say, you know, “I have to put in a lot of work to maintain too, you know, it’s normal. And it’s not just for those people and don’t give up, everybody has to put in work, even though they don’t maybe show it or maybe you don’t see that.”

John: And that’s where the peers can be really useful because they just went through it. And serving as you said, as that role model, can be really effective in letting them know that this is just normal. This is how we learn.

Christina: Exactly.

Fiona: And I could see how hearing “This is normal, this is how we learn” might resonate very differently, if that comes from a peer than it does from a professor, even if you’re being…

Christina: Definitely

Fiona: entirely sincere and open.

Christina: Definitely, and especially from someone who has no power over their grades, or what’s going to be on the quiz, you know, so if they can just hear it from a peer.

John: And who not only had just been through it, but had been through it successfully and was able to do well in the course.

Christina: Yeah, and if they can share with them, how many hou rs a week they put in and how they prioritized it as well. That’s gonna all lead to part of the finding.

Fiona: You mentioned that you didn’t have any experience with peer-led team learning in your own education.

Christina: Right.

Fiona: Can you tell us a little bit about how you discovered this as a topic? And what made you decide that this is what you wanted to research?

Christina: Sure. So, I actually discovered this because I went to Syracuse University myself, and I was a biology undergraduate. But, then I decided to get a master’s degree in forensic science because the pieces of biology that I enjoyed most were DNA and genetics. So I pursued a Master’s of Forensic Science, and as a graduate student, I was assigned to be a teaching assistant in intro to bio. So I began doing labs, and as I mentioned before, Jason Wiles is the professor of Intro to Bio, so he was my supervisor then and he became my PhD advisor as him and I started agreeing on some teaching strategies and techniques. So, I saw the students as a teaching assistant in the lab. And the lab by itself is such a different feeling than the lecture. I saw 24 students at a time, not 600, so I was able to see that perspective when I was a teaching assistant, and I was giving them weekly quizzes and grades, and they could come to my office hours, and it was like a tutoring type thing. So, I saw that perspective of them. And then they implemented the peer-led team learning. In addition to the lecture in the lab, this became an option. And they started doing research on it, and Dr. Wiles was leading that research. And then when I chose to get my PhD, he still had active projects going on. And I was looking at those results of how come this increases their grades, and this does increase their grades. And then it just made me think: Why?

John: So that brings us to your study, how did you design a study to investigate these issues?

Christina: Alright, I went back and forth on the design for this a lot because a lot of the research was focusing on assessments, quiz grades, test grades throughout the course and final course averages. So, I wanted to include that because I knew that there had been good results as far as a positive benefit for PLTL and you could find this in the grades, in the difference between the grades. So, I knew I want to collect data on the final course grades, which is easy enough. But then I also wanted to hear from the students what they felt PLTL gave them in terms of their learning gains. So I used an instrument called the Student Assessment of Learning Gains. And I had the students fill that out. And I was able to use that to see their learning gains, and you calculate the total learning gains by just adding up all of their different categories. So, you ask them questions on a Likert scale, and then you get the summation, but I knew I wanted to do learning gains rather than just course grade because there’s a lot of different ways to get a good course grade: you could memorize you could just remember from the slides, you could just be a very good test taker and good at multiple choice. So I just wanted to see if they were truly learning and I really was drawn to the SALG because that tested the things that they were able to carry on to upper-level courses like their ability to connect concepts in biology to one another is one of the questions your ability to connect concepts in biology to other courses you’re taking. “How strong do you feel you could explain a biology concept to a family member? How strong do you feel that you could create your own position after reading a graph of results and defend your position of how you interpreted the graph?” So this was testing actual skills and true learning gains, not just their end course grades. So I want to look at both of those factors. So, I gathered up their final course grades, and I got the assessment of their learning gains as well.

Fiona: So, they provided assessments of their own learning gains.

Christina: Yeah.

Fiona: And the peer leader?

Christina: So, as I got thinking, like a year into the project, I was thinking, all right, my whole study is testing interactions between the students and student-to-student interactions and what makes a good peer leader. So first, I gave out at the end of the semester, there was a qualitative questionnaire one of the questions asked “Do you consider your peer leader to be a role model in any way? Please explain, yes or no.” So they gave me qualitative data there for why they saw the peer leader to be a role model. But then, as I got thinking that I wanted to do this interaction between them, I thought, why not ask the peer leaders, because this is all the student perception so far, Let’s ask the peer leaders, “Do you consider yourself to be a role model in any way? Yes or no? What makes you a role model to your students? Why do you think you’re not a role model?” And we’re able to look at those two qualitative sets of results and find the discrepancy so that if there is a difference between what the student’s looking for and what the peer leader thinks the student’s looking for, but they’re not providing, we can address that in the leadership course, and say, try to implement these strategies, try to display X, Y, and Z. So, we collected that qualitative data. And then in addition to that, I was also wondering, okay, maybe the student just likes that peer leader, like maybe they’re just a funny person. Maybe they’re just nice. So I thought it would be interesting to see how well the peer leader gets to know the student because as we discussed one of these benefits, this is a very small group of six to eight students, you can spend one-on-one time you see this person every week you talk outside of class. So, I also asked the peer leaders to fill out the Student Assessment of Learning Gains, but ask them to do it in regard to their students. So I said, “Alright, Jane, you had Joe as a student, fill out what you think Joe’s learning gains are.” And then I took Jane’s assessment and compared it to Joe’s assessment and found that the student and peer leader pairings who were closest together had the least amount of differences between their assessments had higher course grades and saw the peer leader to be a role model and relatable and higher learning gains.

Fiona: So very elegantly designed approach to this incredibly thorny, difficult, complex process of learning. And so you’ve essentially got two perspectives on a single student’s experience.

Christina: Yes.

Fiona: From their sense of perception, which is also a fascinating idea. There could be a difference between learning and a student’s perception of learning.

Christina: Right.

Fiona: But by having this very elegant combination…

Christina: And part of what led me to saying, you know, there have been errors in students’ self-reported data before, but perception is valuable, the student perception is valuable. And there are other studies that show if a student perceives themselves to be learning and doing well and happy in a course, they’re going to stay in the discipline despite whatever their final course grade is, and that only helps our retention.

John: But you also said that the people who had the better match that way also had higher course grades as well. So, that seems to validate at least the general nature of that instrument.

Christina: Right.

Fiona: Could we talk about these very interesting concepts, so the idea of being a role model, but also the idea of being relatable?

Christina: Yes. So, originally, the design of the project was on role models because there’s so much literature on role models to support it. So the role model question, the qualitative portion, was asked to the students and the peer leaders. But then, as I was analyzing the results of role model, relatability got mentioned so often that it almost seemed like it was a stepping stone to role model status, that to become a role model, you had to be relatable. So, I didn’t have those students anymore, so I did a follow up question to the peer leaders because we still had access to the peer leaders, their course was a little bit longer. So I was able to ask the peer leaders to write about relatability and how relatability came into context with their relationship with the students. So, for the relatability factor, we have the qualitative answers from the peer leaders. And for the role model factor, we have it from the students and the peers, but students were mentioning relatability in a lot of the role model answers.

Fiona: And what does relatability mean in this context?

Christina: So relatability, I was able to categorize into a few different codes and categories. So the students were calling the peer leader relatable. Sometimes it was very clear, they would say the phrase “In my shoes, you know, they were just in my shoes, they know exactly what I’m going through, they’re in my shoes.” Other times, they attributed it to a common career goal. So these are all bio majors. They’re all “I want to be a doctor and my peer leader is taking the MCAT. That’s what I’m going to do in the next year or two. So, I can relate to her because I see her doing this thing that I’m about to do.” What was interesting is that if there was a commonality there, it would always be a shared attainable goal. Like a “In the near future, I’m gonna do that.” It was neat to see that and it’s definitely an attainable goal of the students. Sometimes when you think role model, you think, “Oh, they need to be a famous celebrity and they’re rich,” so it was neat to see the students just saying “They’re a role model because they’re succeeding in what I see to be a hard path that I also want to do.”

Fiona: They’re a role model because “I want to be there next year.”

Christina: Yes. And I relate to them.

John: So we have all that evidence about the effectiveness of your instruction. But this is discovering why it’s working or what specifically allows it to work.

Christina: Right.

John: So, what does this suggest that in terms of developing your lead team leader program, what should that training focus on?

Christina: I would recommend that we share the results with the peer leaders and show them that in a lot of the student responses, when they were saying what they thought made the peer leader a role model, or the peer leader relatable, a chunk of them did say that they got a good grade in this course, and that’s what I want to do, and that’s my goal is just to pass. But, there was a lot more in depth answers than I think maybe I was even expecting, and it went into things like “They were just kind, they were nice. I could tell that they cared if I was learning, they stopped and adjusted the course to my pace,” and that’s a huge benefit that we don’t get in the lecture halls when you just have to keep going, and you’re not able to assess “Are all 600 students with me? Do they all understand?” And a lot of the responses just came up that there was a caring peer leader who truly cared if they were learning, and talked to them and asked them. So, I think that showing the peer leaders that would be a little bit of a difference, because I think a lot of the times it just gets focused on “Well, I did really well in this course.” So, here’s what I think, that there’s so many personality attributes that go into it.

Fiona: I keep thinking of Michelle Miller’s line, that teachers are professional motivators, fundamentally the role that teachers play in drawing out the learning potential in a student in some way. And this seems to point to that in some sense, that what matters the most to the students… yes, the content and skill development matter… but that, in fact, there was some other piece to the puzzle.

Christina: Yeah.

Fiona: That might be effective or emotional.

Christina: And motivate is a word that was frequently used, so much so that I used it as a code for the qualitative piece.

John: And I understand you just heard that a paper from this study is going to be published. We’ll include a link to that in the show notes, and we’ll list it as forthcoming until the paper becomes available.

Christina: Just got accepted. Yes, thank you.

John: Congratulations.

Christina: Thank you. The first paper shows the quantitative aspect, that students had a difference in either perceived learning gains or final course grade, if they saw their peer leader to be relatable and a role model. And then part two is coming up with the qualitative research that explains what those definitions were in the eyes of the students.

Fiona: I have a question that I think goes a little beyond the scope of what you were focused on. But it comes back to this idea of what being relatable means. And I’m thinking about that gap, that the attrition rate or the achievement gap or however you want to talk about that for certain sorts of underrepresented groups. Does your research suggest that the identity of the peer leaders might matter? We know representation matters. We know that this is part of envisioning yourself in a discipline or a field process that your research is getting at. And could you imagine the approach to setting up a peer-led team learning approach could be useful in like specific interventions?

Christina: Actually, that’s a really interesting question that you brought up, because a lot of the research points to that. And then I was expecting to see that, but it was the only time that that was really brought up about the identity of the peer leader was out of all the responses I had, I believe six times the student just said, “Because they’re my age” and interestingly, two times when the identity of the peer leader was brought up for the “No, they’re not a role model,” the answer was “Because they’re my age.” So, I thought that was a really neat finding, that some of the students are viewing age as this positive thing. And other students are using it to say, “No, they can’t be a role model to me. They’re my age,” and other students are saying, “Yes, they’re a role model, because they’re so successful, and they’re just my age.” So, I thought that was really interesting, and I’m not sure, there wasn’t any answers about ethnicity or even gender, and I’m just wondering if the students maybe felt a little like restricted because it was a written out response, and if I just looked at the data of the ethnicities of each group and did a comparison there, if maybe I would find something. But I didn’t go that far in this experiment, but it would be really neat to do so in the future.

John: That could be an interesting follow-up, just to see if the gender and race of the team leader affected the performance of people within the group if they’ve matched or if they diverged.

Christina: Yeah, I agree. I did look at if STEM major and non-STEM major, because in semester one of peer-led team learning, it could be used for any student. So, even if you’re not a bio major, you could be using it to satisfy your science elective. So, I was trying to see if there was a trend with the students who were STEM majors getting more out of it than the non-STEM majors, and we didn’t see a significant difference.

John: Interesting.

Fiona: You mentioned the fascinating finding that when the peer leaders’ assessment of the students’ learning and the students’ assessment of the students’ learning matched, that there was a definite correlation with the overall performance. Did you pull out any information about when there was a mismatch? Or was there anything interesting or useful about the opposite scenario?

Christina: Yes. The peer leader responses, in general, were always a little bit more informative and longer than the student responses. A lot of the time when there was a mismatch, the peer leader would leave a note at the end or something that would say, “This is what I think. But so and so didn’t come to class very much.” In those cases, there was a big discrepancy where interestingly, the student would say, “Oh, I learned a lot, high learning gains,” and the peer leader would put them at a low learning gain, and then there would be a big discrepancy, and I would try to see why and the peer leaders would say, “I never saw them, so I don’t know if they learned or not.” My research group at SU, one of my good friends, Ryan Dunk, who was actually on this paper that just got published, he helped me with some statistics for this. Another question that came up was, “Could it possibly just be that some peer leaders care more and know their students more?” But, we were able to find that there was no significant difference between the peer leaders or the section, so time of day didn’t matter and peer leader didn’t matter, so that was neat to find out too.

Fiona: Fascinating.

John: You mention that the participation in the peer-led team leader program was optional for the students.

Christina: Yes.

John: What proportion of the students in the classes participated in that?

Christina: I can answer that for the semester of the data set that I’m working with, it was after you eliminated students who either didn’t consent to research, or were under 18 and couldn’t be included into research. Out of around 650 of them, 241 participated in my study. So, that means that at least 241 were in peer-led team learning.

John: and it could have been more…

Christina: And it could have been a few more.

John: …if they chosenot to participate or were under eighteen.

Fiona: Are you currently, are you still crunching the numbers and figuring things out from all the data you currently have?

Christina: There’s a good three more publications that are all in progress in various stages. Review, sending it to a partner, and working with my advisor still.

Fiona: Yeah.

Christina: He’s still doing different projects too, though they’re still being studied.

Fiona: Do you think peer-led team learning can work across disciplines?

Christina: I do, and a lot of research was shown to back that. And it’s actually pretty interesting because it started in a chemistry class, and then other sciences picked it up, biology and physics. But, it’s expanded into mathematics, and computer science has had success. And other studies also supported the finding that it would work in smaller classes too. So, it could work at universities, it could work at colleges, it could work at private institutions. So, it just looks like it can work across the board.

Fiona: There’s still room to find out, room to experiment.

Christina: Yeah.

Fiona: We’d like to finish by asking you “What comes next?”

Christina: I think if I don’t answer that I’ll get those three publications out that my advisor is gonna… [LAUGHTER] I’m gonna hear about it. So all the publications.

Fiona: We very, very much look forward to the results from those various publications.

Christina: Thank you.

John: This was fascinating.

Christina: Thank you. I’m so happy to be here. It’s such a privilege.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]

123. New Trends in Science Instruction

Science instruction in K-12 education has long been provided as if science consisted of a body of facts to be memorized. The Next Generation Science Standards, however, rely on an inquiry-based approach in which students learn about science by engaging in scientific exploration. In this episode, Dr. Kristina Mitchell joins us to discuss this approach and its implications for college instruction.

After six years as a director of online education at Texas Tech University, Kristina now works for a science curriculum publishing company and teaches part time at San Jose State University.

Show Notes

Transcript

John: Science instruction in K-12 education has long been provided as if science consisted of a body of facts to be memorized. The Next Generation Science Standards, however, rely on an inquiry-based approach in which students learn about science by engaging in scientific exploration. In this episode, we discuss this approach and its implications for college instruction.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Kristina Mitchell. After six years as a director of online education at Texas Tech University, Kristina now works for a science curriculum publishing company and teaches part-time at San Jose State University. Welcome back, Kristina.

Kristina: Thank you. It’s good to be back.

Rebecca: Today our teas are…

Kristina: Diet Dr. Pepper.

John: The same as last time. [LAUGHTER] I have a green tea today.

Rebecca: And I have a nice pot of brewed English Breakfast tea.

John: We’ve invited you here today to talk about the Next Generation Science Standards. How did you get involved with the Next Generation Science Standards?

Kristina: During my work at Texas Tech University in online education, I was doing a lot of consulting for publishing companies on their online and curriculum offerings, which led to a full-time job offer from a curriculum company that works specifically in K-12 and some higher ed science curriculum, and I learned very quickly that one of the newest trends in science education at the K-12 level are these Next Generation Science Standards that were created by college-level science professors in order to change the way we teach science to K-12 students.

John: How did these standards come about?

Kristina: I have to admit that because I’m relatively new to the science curriculum world, I am not an expert on their creation. But I do know that there were several panels of scientists from various higher education institutions that got together to try and figure out from the perspectives of different disciplines, “What do we want our students who come to university to know about science once they get there?” When I was in middle school, I don’t know how much you remember your middle school science lessons, but I remember dissecting frogs and doing our Punnett squares about genetics, but what I remember most is that we would learn those principles of science like Newton’s laws throughout the week, and then Friday, we would do a lab to prove that they were true. And unfortunately, that’s not how any of us do science. Being a social scientist myself, that’s not how I did science at the college level, as a researcher. So, the panel of experts from various scientific disciplines sought to change the way students think about science to think about it more as a process of investigation, rather than something you sort of memorize and then confirm.

Rebecca: The Next Generation Science Standards have three dimensions. Can you talk about those a little bit?

Kristina: Sure. So, these sort of three dimensions, or the three pillars of the way students are learning in K-12 about science, the first is they’re learning disciplinary core ideas. So these disciplinary core ideas are the core features of disciplines like physics or chemistry and the things that they really have to know about that specific discipline in order to build on that knowledge. The second one is science and engineering practices, so these are the ways we study science. And the great thing about the ways that we study science is that it doesn’t change across disciplines. Whether you’re a physicist or a chemist or a sociologist, you’re going to use the same types of scientific practices to answer your research questions. And then the final dimension, the third dimension, are cross cutting concepts. So, these are things like observing patterns or observing cause and effect, concepts that also span across all disciplines, not necessarily limited only to science. So it’s concepts that students can use to understand the world around them, regardless of what questions they’re asking.

Rebecca: Can you give us an example of how these three dimensions play out in a particular grade level?

Kristina: Sure. So when we’re thinking about what students are doing in science classrooms these days that might be different from the way we learned science when we were in seventh grade, students are doing a lot more investigation centered science. So, you walked into your classroom, your college classroom, and somebody walks in and says to one of your students, “What are you guys doing?” And your student might say, “Oh, we’re reading this chapter on…” Rebecca, I think you’re an art person, right?

Rebecca: Yeah, design.

Kristina: “So we’re reading this chapter on Photoshop,” and you ask them, “Okay, well, why are you doing that?” And they’d say “Well, because Professor Mushtare told us to.” In trying to get them away from that instead, the way they’re trying to implement science in the classrooms now would be to get students, you know, if they came into your classroom to say, “Well what are you guys working on?” And they might say, “Well, we’re trying to make a flyer for this project.” And you’d say “Why?” And they’d say, “Well, because we’re trying to figure out how we can make people more excited in this project that we’re working on.” So, getting students to start from the point of “What are we trying to figure out and why do we have to do this activity in order to figure that out?” …rather than “What are we memorizing, what are we learning because the teacher told us to?”

John: So it’s more of a project-based learning or inquiry-based approach to the discipline.

Kristina: Absolutely. And we’ve seen that a lot emerging in higher ed. So, it’s really great to see it reflected in the K-12 space too.

John: But it’s important probably for faculty to be aware of this, because if students are starting to use this approach, as they move forward, they might expect to see more of that in higher ed, where they might see it as a step backwards if they have to move to something that involves more rote learning, for example.

Kristina: Exactly. We’re seeing a lot less “sage on the stage”-type activities, both in K-12 and in higher ed. So it’s really important, I think, that we all kind of know what both sides of education are doing so we can at least know where our students are coming from.

Rebecca: How do you see this new approach helping students be better prepared for college?

Kristina: Well I definitely think that the idea of empowering students to be in charge of their own knowledge is really important. Sometimes I find that a struggle that my students have is doing this sort of figuring things out on their own. If I don’t give explicit instructions with explicit rubrics, they sometimes feel very lost. And so empowering them to recognize themselves as people who can ask questions and figure out the answers to those questions, that goes far beyond just physics class. That goes into just being a responsible and productive member of society. So, I think hopefully we can see students taking a little bit more ownership in their learning because of these trends at K-12.

John: Does it also perhaps give students a little more motivation when they have a goal in mind, and they know why they’re doing things rather than just doing it because their instructors told them to?

Kristina: Yeah, I would definitely think so. I definitely remember my seventh grade year not being passionate about the science, other than “Wow, cool. It’s a dead frog.” So thinking about empowering students to ask these questions might get them more excited about what they’re learning about.

John: In one of our earlier podcasts, we talked to Josh Eyler, who emphasized the importance of curiosity and nurturing curiosity and this approach seems to be very nicely tied into that.

Kristina: Absolutely. And I think about my own political science classroom. One of the fun activities I’ve done and I swear the first time I went through it, it was one of those days where I thought, “Ah… I’m tired. I don’t have anything that I’m interested in lecturing about. We’re almost through this semester, what am I going to do today?” And so I decided, I’m just going to have the students write questions on index cards related to political science and hand them all in and I’ll answer all of the ones that I can. And that was one of the best days of class. I’ve recreated that many times in semesters. And so seeing that these Next Generation Science Standards where we’re trying to get students to think there are some things you learn about each discipline, but there are some practices and concepts that apply everywhere and you just learn how to ask the right questions and you use these concepts and practices to answer them. It’s something that I’m using in my political science class, even really, before I realized that I was specifically doing so.

Rebecca: Do you have any idea how students are responding to this curricula?

Kristina: So every individual curriculum and program is going to be a little bit different, but they are all going to tie into these main ideas. And one thing that we’ve seen in student performance and student preferences is that the students really like doing things in class, having activities and hands on specifically to science. Of course, that often means labs. But one thing that I often talk to teachers about and get a lot of really good feedback on is the idea that we kind of think of doing science looks like it’s in lab coats with safety goggles, but we all know as researchers ourselves that science looks like talking sometimes. And sometimes it looks like asking questions, and sometimes it looks like taking notes or revising models, and getting the teachers and students to realize that that’s all part of what doing science is. They get really excited at the idea of doing science themselves. Of course, they like to see things explode in class, but sometimes just getting them the opportunity to have discussions with each other. It’s amazing to me what the middle school students are capable of, and our elementary age students, what they’re capable of. I have kids myself and I didn’t know that they were learning such higher level concepts of science in class.

Rebecca: There’s big movements to move STEM to STEAM. Is the humanities represented in this new model?

Kristina: So, when we think about STEM, that’s kind of what everyone’s prioritizing these days: the science, the technology, engineering and mathematics… and the A to add it to STEAM would be the arts. So the idea that the arts are also important, but I think there are some key pieces missing here. When we focus exclusively on STEM and on creation and innovation, that’s really good. We need creation and innovation. But as I feel like sometimes we’re learning right now, without a good background knowledge of history, we might be creating and innovating things that either already exist, or have already been tried, or that don’t have a good basis in what we’ve experienced as a society. It’s like they said on Jurassic Park, you spend so much time wondering whether you can and you don’t stop to think about whether you should.

Rebecca: So you were just raising that question about ethics and things like this. So,how do you see this curriculum evolving so that the humanities are better represented or integrate more into our integrated learning in K through 12?

Kristina: Well, one thing that I think is really great about the NGSS is that it definitely starts to sort of de-silo the disciplines of science. So, when we think about physics and chemistry and earth science and life science, when I was in school, those were all completely separate years, both in middle school and in high school. And the NGSS standards definitely start to recognize that we can’t silo our disciplines like that because they have so much to speak to each other. Some questions you need many disciplines of science to answer. And so I’m hoping that slowly, more and more curriculum writers will apply that same standard to the humanities, to the social sciences, and to the arts, the idea that nothing can truly be siloed because social science speaks to even questions like “What scientific research do we do? What research questions do we ask?” That can’t be explained using natural sciences alone. Because yes, natural sciences might seem really unbiased, but we’re asking certain questions and we’re not asking others. So using social science and humanities to help answer those bigger questions. I’m hoping that we see less siloing across disciplines over time. And even at universities, we’re starting to see increases in things like cluster hires, where multiple people from different disciplines all come at a sort of similar question, kind of like what we’re doing here. We’re three people from very different disciplines. But we’re all interested in the question of, you know, what makes teaching good or bad.

Rebecca: One of the questions that’s risen a lot in higher ed, and I know as a popular conversation on our campus, is about fake news and debunking pseudoscience. Do you see that this curriculum helps with that,? …perpetuates that?

Kristina: I think that’s a really good question, and kind of a difficult one, because I think it’s really important to teach students how to interpret the world around them, and what science is and how it’s done, because that can help us determine the validity of the information that we get. But I also think, like we were just talking about, there is a move away from rote learning, from the idea that an expert in a field tells us what is true about that field and we learn it. The fact that we’re moving toward anyone can be an expert, anyone can do the research themselves, anyone can ask questions. Sometimes, I also worry that that pendulum might swing too far, because that’s sometimes where we get people who go online and find whatever Google can tell them about vaccines not being safe or about climate change not being real. And because they feel empowered to be their own expert, they’re treating themselves and their research as equivalent to people who have devoted their lives to getting advanced degrees and who know the answers to these questions. And so I worry that maybe we’ll see the pendulum swing a little too far in the other direction, but I’m not sure.

Rebecca: Based on knowing what you know about higher ed and what you know about this science curriculum, what do you think faculty should be thinking about in terms of what their students might know or might not know, or how they might know differently than they did before?

Kristina: I think that it’s important to teach students what expertise is to begin with, when we think about what it means to be an expert and know something. So, when we go back to those next generation science standards, and think about those three dimensions, the disciplinary core ideas, to me, those are the things that experts already know and that we can accept as the core ideas of each discipline. So, starting from that point and saying the question of vaccine safety has already been established, we know that the vaccines that are here are safe. That’s a disciplinary core idea. But now let’s use our science and engineering practices and our cross-cutting concepts to think about other types of preventative medicine that maybe don’t have the same consensus or new types of vaccines? How would we know that those are safe or how would we know that those are effective? So, I think it’s important not to forget about those disciplinary core ideas that are things that experts have already figured out questions that have already been settled, and then allowing people to start from that point and ask their own questions and do their own research using the scientific method.

John: Changing the topic just a little bit, we get a fair number of people who are interested in the STEM fields when they’re in middle school and even high school, but then they get to college, and all of a sudden, they have to start taking calculus classes, and Matrix Algebra, and Organic Chemistry. Is there any way that perhaps lessons learned from this approach could be used to make college classes more effective and not having so many people get lost along the way once they hit these courses that have been traditionally a bit of a barrier to progress in those areas?

Kristina: That’s a really great question, and we even see that in political science. I teach our research methods class in political science and it has just the tiniest bit of statistics in it. And students are like, “Why am I doing math in a political science class?” And then of course, in graduate school, we’re seeing increasing amounts of quantitative knowledge needed to finish graduate programs and then to compete on the job market. So, I definitely think that the rise of math as something important is a deterrent to some students who might otherwise be interested in entering STEM or social science fields, but I think that also comes down to the pendulum again. I think that it’s important to realize that quantitative methodology is a method of understanding scientific questions, and you don’t have to use quantitative methodology to answer every scientific question. Some of them can be answered using qualitative methods, and even discussing political theory or other types of theories in other sciences or in the humanities. I think it would be good if we could open up the doors a little bit to people who have really high level intelligence and interest and curiosity, but maybe don’t have an interest in math. I think both by making math more accessible and less required, we might be able to get some really interesting new perspectives in our college level courses. I don’t like the idea of using hard math classes as a way to weed students out. And I think that’s sometimes what they’re used for, because I took Matrix Algebra in grad school, and it was really hard. And I’ve never used it since that class… ever again. So it’s not as though it was a skill I needed to be a PhD in political science. It felt more like a class that was used to weed out students who weren’t as interested or weren’t as good at memorizing how to do Matrix Algebra.

John: The point I was trying to get to, I think, is that perhaps if more of these inquiry-based methods were used to motivate the material, it could help get past that. I actually had a similar experience with Matrix Algebra. And when I was learning, I was able to do the proofs and work through it, but I didn’t really understand it until I started using it in econometrics and other disciplines, when all of a sudden it just made so much sense and why all those theorems were there and useful. But we lost a lot of people along the way there. And perhaps that’s something we could build into all of our classes to help motivate students and not just hit them with a lot of theorems and not necessarily even just rote learning, but a lot of inquiry for which the motivation may not be obvious to people when they take it.

Kristina: Definitely. And I think that when I teach my Statistics and Research Methods class, it always is easier for the students, and they always do better, when I tie the math we’re doing to a real question. So even something as simple as “We’re trying to figure out whether the average American is conservative or liberal,” and this is our question that we’re trying to figure out. Now we need to do a difference of means test to figure it out. And so that would be, going back to what we were saying earlier, the idea of you walking into my class and asking my students what they’re doing. Well, they could answer “We’re memorizing how to do a difference of means test from chapter 12 because Dr. Mitchell told us to,” versus saying, “Well, we’re trying to figure out what the ideology is of Americans from different regions, so we have to use this mathematical test so we can answer that question.” So really about framing it in terms of “What are you doing and why are you doing it?” and making sure that we know that the students would answer in a way that gets them excited about the material.

Rebecca: I think context is everything. The real world problems that you’re talking about in the new science curriculum is not unlike the kinds of things that you’re talking about in your classes or even when you have to use math in design class. So, a lot of visual arts students think math is completely irrelevant to them, but when it’s put in context, when you need to do certain things, all of the sudden it makes more sense. I teach a creative code class and we do fairly complicated geometries there, actually. And if you can see the results, sometimes it makes a lot more sense to some of our students.

Kristina: Exactly. So the fact that they would know why they’re doing this and it’s to accomplish a specific goal, rather than just “We’re memorizing this because the professor told us we had to, it was going to be on the exam.”

Rebecca: Yeah. [LAUGHTER]

Kristina: Lord, help us for how many times we hear “Is this going to be on the exam?” That seems like the motivation for trying to learn something. And it would be great if we could move it away from that being the motivation, toward the motivation being “Well, we’re trying to learn this because it helps us figure out this question or this design problem.”

Rebecca: Or the motivation, like, “Hey, I really need to understand this mathematical principle, because I really want to know the answer to this question.”

Kristina: Exactly.

Rebecca: And that leads to where they actually start asking for the things that we’ve prescribed previously. [LAUGHTER]

Kristina: Yeah, and it becomes just a tool to figure things out. And if they want to figure the answer out, then they’re going to need all the tools.

John: Could you tell us what this would look like if we were to stop by a seventh grade class, and this was being put in practice?

Kristina: Absolutely. So if we think about what a seventh grader might be doing, so if we start from that point of the goal is to figure something out, and maybe they’re learning about Space science. They’re learning about Earth and Space and trying to figure out “Why do some objects in Space orbit other objects?” …like “What does that mean? How does that happen?” So, one really fun activity that I’ve seen middle schoolers doing is they have these embroidery hoops and these marbles. And so what they do is they put the marble in the embroidery hoop, and they sort of spin it around on the table. And so the marble is swirling in circles and then they lift the hoop and marbles start shooting everywhere. So it’s very funny in a seventh grade classroom, because you got kids running everywhere trying to chase their marbles. And so at first, they’re just having fun, tracking which way the marble goes and trying to figure out when they lift the embroidery hoop which direction will the marble head, and then eventually they start realizing, “Oh, well, the marble kind of wants to go straight, but the embroidery hoop is keeping it going in a circle.” So, if we think about the sun, with the planets orbiting it, if the sun suddenly disappeared, Earth would kind of want to go straight out in whatever direction it was headed. But because the sun’s gravity’s in the middle, it’s keeping Earth going in a circle around the sun. And so they start putting these pieces together and eventually they come up with this idea of the relationship between inertia, which keeps the earth going, and gravity, which is what keeps it in that same orbit. When they think about what is the core idea they’re learning about, well, they’re learning about orbit, but the way they’re figuring it out is by looking at this small model of what is actually happening at the planetary level. And that’s really reflective of the kinds of things that scientists do in their labs. We’re running models that try to simulate what’s happening in space, or what’s happening in the political world, or whatever discipline we’re doing. We’re trying to create these models to help us understand it. And so that’s what the students are doing, too.

John: That reminds me of a talk that Eric Mazur has given in several places, including at Oswego, where he talked about the Force Concept Inventory, and how, while students were able to plug numbers into formulas and solve problems very well, when they were asked about questions like that, specifically, if I recall, “Suppose that you’re swinging a rock on a sling, and you release it, what would the motion look like?” And basically it ends up flying in essentially a straight line… well, other than the effect of gravity. But when people were asked that, they would bring up things that they had seen in cartoons where they expected it to loop through the air, perhaps as it’s moving. That’s actually one of those things where people in really good institutions who had taken a college class had some really poor intuition. And this type of practice can help prepare them with better intuition on which to build later.

Kristina: There was also a video at, I think it was Harvard’s graduation, where they asked a bunch of STEM majors who had just graduated if they thought they could make a little circuit that would light up a light bulb. And so they handed them everything they would need to do, and everyone’s like, “Oh, yeah, yeah, I got this no problem.” And they couldn’t, they couldn’t figure out how to light the light bulb up, because as much as they had done the math in class, they just had never really done the practical aspect. And so getting students hands-on to do the things with those science and engineering practices in whatever discipline we’re in. It’s really important to get our students in the driver’s seat to start practicing what they’re going to be doing when they leave the classroom.

Rebecca: It also seems like having multiple pathways to the same information is a good way to play with our memory, and make sure that it sticks and it stays there. So, something that’s embodied, something that you memorize, something that’s a couple different kinds of examples.

Kristina: Exactly. Maybe it can move it from our short-term back to the long-term memory to help us keep that knowledge.

Rebecca: So we always wrap up by asking “What’s next?”

Kristina: So I think I’d really like to start exploring what this does look like in a classroom setting for myself. So rather than just focusing on what it looks like for students in K-12, I’d like to see if I can use a similar style of teaching in my political science classroom, and just see what kinds of evaluations I get and how the students like it. I think the idea of getting rid of these silos and recognizing that we all use the same practices and concepts could help us with keeping our college experience for our students interesting and consistent. And we start realizing that they’re going to learn in each of our disciplinary core ideas, but there are a lot of things that we all have in common. So, I’m really going to try this semester, maybe my student evaluations will suffer greatly, but we all know from my past talks that they’re biased anyway. But I just kind of want to see what it looks like if I let my students take a little bit more of a driver’s seat in asking the question, “What are we going to figure out? What do we want to know about the world?”

Rebecca: Do you have a piece of this semester that you’re most excited about that you already have planned in this domain?

Kristina: Well, I’m doing the research methods class this semester, which of course, involves some math. And I think that I would like to use their ideas about what kind of data we should be looking at and what kind of questions we should be answering. Because I think that’ll make it a lot more meaningful for them, rather than if I say, “We’re going to look at the American national election study,” letting them figure out “What question do you want to answer?” And then I’ll teach them whatever method it is that they need to know in order to answer that question. So, we’re going to try it. We’ll see how it goes.

John: In a past podcast, Doug Mckee talked about a similar situation in his econometrics class where he was using a time-for-telling approach. Where basically you face students with problems and let them wrestle with it a bit before giving them some assistance and helping them resolve things, and there’s a lot of evidence that those techniques can be really effective.

Rebecca: So, it sounds like well wishes on your new adventures. [LAUGHTER]

Kristina: Thank you.

Rebecca: Thank you so much for joining us.

Kristina: Of course, this is great. It’s always great. I think eventually I’ll have to come visit y’all.

John: That would be great. Well, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Brittany Jones and Savannah Norton.

[MUSIC]