333. High Structure STEM Classes

Multiple studies have found that increasing course structure reduces equity gaps and provides benefits to all students. In this episode, Justin Shaffer joins us to discuss several ways to increase structure in STEM classes.

Justin is the Associate Dean for Undergraduate Studies and a Teaching Professor in Chemical and Biological Engineering and in Quantitative Biosciences and Engineering at the Colorado School of Mines. He has taught a variety of both small and large STEM classes in multiple modalities using evidence-based approaches and has won multiple teaching awards as a result of this work. Justin is also an active researcher with 16 peer-reviewed publications and serves as the editor for four STEM education journals. He is the author of a forthcoming book on high-structure course design coming in late 2024 or early 2025 from Macmillan.

Show Notes

Transcript

John: Multiple studies have found that increasing course structure reduces equity gaps and provides benefits to all students. In this episode, we examine several ways to increase structure in STEM classes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca:Our guest today is Justin Shaffer. Justin is the Associate Dean for Undergraduate Studies and a Teaching Professor in Chemical and Biological Engineering and in Quantitative Biosciences and Engineering at the Colorado School of Mines. He has taught a variety of both small and large STEM classes in multiple modalities using evidence-based approaches and has won multiple teaching awards as a result of this work. Justin is also an active researcher with 16 peer-reviewed publications and serves as the editor for four STEM education journals. He is the author of a forthcoming book on high-structure course design coming in late 2024 or early 2025 from Macmillan. Welcome, Justin.

Justin: Thanks for having me. I’m really excited to talk to you today.

John: We met at the POD conference back in November, and we talked about doing this but we finally got around to actually scheduling it. So thank you for joining us.

Justin: Thank you, John. Yeah, I saw you sitting there at a table by yourself. I knew your face from LinkedIn and I built up the courage to introduce myself and I’m glad I did.

John: Today’s teas are:… Justin, are you drinking any tea?

Justin: I am drinking some tea. I got prepared. It’s from a small little coffee/tea shop in West Middlesex, Pennsylvania. I’m from the near that area in western Pennsylvania. We call it the Shenango Valley and it’s called O’Neill’s and they make lovely Western PA coffees and teas and I just have some nice black tea here. It is that time of day for me that I need some.

Rebecca: Today I have a jasmine green tea, John.

John: Very good, and today I have a Prince of Wales tea, and a peppermint tea because I didn’t have very much to drink today, and my throat was getting really dry in my econometrics class. So I wanted to be well hydrated. [LAUGHTER]

Rebecca: Lots of variety, too. So we’ve invited you here today, Justin, to talk about how you’ve implemented high-structure course design. And for listeners who might be new to the concept, can you talk a little bit about what you mean by high-structure course design?

Justin: Yeah, so just like I’ll tell you all like I tell my students, I didn’t come up with this term. I wish I did. But I’m building off the work of other giants in the field. So as far as I know, the term really came out in the mid-2000s, 5-, 6-, 7-ish range from the University of Washington in Seattle. There’s some pioneering folks, Scott Freeman and Mary Pat Wenderoth in biology and they use this term structure to think about, “Well, how do we scaffold student learning before class, during class, after class, and we’re continuously building as we go up through the three levels.” So pre-class involves some kind of content acquisition, that can be reading your textbook, reading journal articles, watching Khan Academy videos, other sources like that. And then some kind of formative assessment before class because we all know students need a little bit of a carrot there to get some of that work done sometimes, which is totally great, because you give them a lot of chances, sometimes even unlimited attempts… again, formative assessment before class, make sure everyone has the same solid foundation, then you come to class, high-structure course design, just like any good class nowadays should be really rooted in active learning. So we got a lot of activities going on: group work, problem solving, case studies, real-world connections, any kind of flavor you want to do there with your active learning as long as students are engaged. And the big thing here though, is that you’re aligned to what happens before a class. We’re building off what happens before a class, I’m not in class just repeating what happens ahead of time. Rather, I’m using the foundational material students acquire before class, we build that up with another level in class, and then we keep going. And then after class, there’s a bucket there, where students are going to be practicing those skill sets they’ve been developing through the before class and in-class buckets, applying their skills to new contexts, solving new problems, doing projects, essays, more authentic assignments, whatever you want it to be. And then there’s typically a frequent assessment piece with a high-structure course design too, whether that’s a multiple midterm model, weekly quizzes, every other weekly quizzes, things like that, in addition to some more authentic assessments, which is something I’ve tried to get more into this past year, by adding more realistic projects and course assignments in my high-structure courses.

John: Could you give us some examples of some of the authentic assessments that you use in your classes?

Justin: Yeah, thanks, John. So this is really new to me still, this past year. I know I’m behind the curve. I know this is a term has been around there for a while. I always like to think I’ve had some authenticity in my classes for a long time. I integrate news stories, I use case studies, I have this new thing I called pod cases where I take podcasts and combine them with case studies and we use those in the classroom. But I thought I could try to be a little bit more authentic, even still… getting that advice from my wife for my personal life. She always says that can be more authentic. So, maybe I can make my courses and my assessments more authentic too. One example I’m doing right now. I’m teaching introduction to biomedical engineering, and I’m having the students write the actual legit National Science Foundation Graduate Research Fellowship proposal. Those of you familiar with that, there’s a personal statement part and there’s a technical kind of research proposal part. So we’re just doing that. It’s a two-page proposal, and we’re working on this in a structured way throughout the entire semester. Last week, students had to turn in the research topic and research question. This week, they’re turning in their hypothesis, in a couple weeks they’re turning in the literature review and experimental plan. We’re gonna have two rounds of a formal peer review of the entire proposal. And then we’re going to conclude with a mock panel discussion, just like you might have in the real NSF or the real NIH where students are going to be reviewing each other’s proposals and basically pitching them to each other and seeing if they should get, quote, unquote, funded or not. I don’t have any funds for them. Maybe we’ll bring candy on the last couple of days. But that’s kind of the idea of having a real thing that students could actually use some day if they’re going to graduate school. But even if they’re not, being able to write technically and persuasively in two pages on a proposal is a really important skill that they should be working on, in my opinion.

Rebecca: Sounds like you’re going to need some chocolate coins for that day.

Justin: That’s a great idea. I will get chocolate coins. [LAUGHTER] I didn’t think of that. [LAUGHTER]

Rebecca: Can you talk a little bit about what motivated you to attempt and move in the direction of high-structured course design.

Justin: I kind of started out this way, I kind of started in the deep end, if you will. So my postdoctoral program was called spire, S-P-I-R-E, and it’s one of these bigger programs called IRACDA through the NIH. They’re postdocs where you still do research. So I was a molecular biologist, biochemist at the time. I have a weird background in chemical engineering, bioengineering, molecular biology, but at the time, I was studying muscle proteins in squid, super fun stuff. And then I also got training, though, on how to teach evidence-based college science and engineering courses. In my postdoc, through great people like Ed Neil, Brian Ray Bozic, Leslie Lauria, and then also Kelly Hogan was someone that took me under her wing at the time, she was able to, along with the rest of the SPIRE team, helped me be introduced to using evidence-based course design practices, which included with high structure. So this was around 2010-11, right when these papers are starting to come out from Scott Freeman and Mary Pat Wenderoth. I started with high structure because the evidence shows it works. And that’s as a trained scientist, as a trained engineer, we like to go with evidence ideally to guide our decisions. And it’s no different with course design and teaching as it would be with thermodynamics or reactor design or anything else. The literature on high structure is really strong, even then it’s been growing over the years, some of my own data and a lot of other papers. The main gist, though, is that by having the scaffolding before class, in class, after class, students are able to really monitor their learning, develop self-regulated learning skills, be more metacognitive. And the data in a lot of papers shows all students are doing better in the class. But it’s not just everyone, using high structure and moving up in structure will start to close or reduce achievement gaps or performance gaps between historically underperforming students and their majority counterparts. So for example, female students and male students in physics, there has traditionally been a gap there, add some more structure to the class, those gaps tend to get smaller. And I’ve been participating in that kind of research here over the last many years with undergraduate students and my research team. And we found some similar results across different STEM disciplines. It’s really the evidence guiding me in these decisions to do it. I also think it’s a lot more fun. Once I have in mind one of my very first faculty interviews, oh a long time ago, now, I had an interviewer say to me, “Well, hey, I’ve looked at all the papers on active learning, and they’re all wrong. [LAUGHTER] All the statistics are wrong. What do you have to say about that?” And I paused for a second I said, “Well, even if that’s true, it’s a heck of a lot more fun to teach actively with active learning than not, I think the high-structure design model is more of a fun way to teach. And again, you’re helping the students with those study skills and metacognitive skills that they can then take forward to their future classes.”

John: There was a recent paper that did argue that many of the studies on active learning were flawed in some way in terms of a very strict methodological approach. However, there are so many of them, and the results are so remarkably consistent. I don’t really buy that argument in terms of the questions about methodology.

Justin: You’re right, it was a big deal, I think it was either Beckie Supiano or Beth McMurtrie in the Chronicle had articles on this going around for sure. And that’s the thing though, with us, as DBER folks… I want to introduce a new term, DBER – discipline-based education researchers… which I consider myself to be in. So I’m in the discipline. I’m in chemical engineering, I’m in biology, I’m in Biomedical Engineering, and I do the education research from that lens, but also from that training background. I’ve never been formally trained in education research methods in psychology or social sciences or psychometrics. I am, totally honest, I fake it. I do what I can. I know my boundaries. I know when I need a collaborator in the true social science disciplines that helped me with my studies. And so I think that paper was getting some of that. Sometimes the folks like us that are doing this work, whether we call ourselves DBER people, or Scholarship of Teaching and Learning (SOTL) people, we might not have that true background, and therefore the rigor of the paper, the methodologies, aren’t as advanced as maybe they should be. But like you said, there’s so many studies showing benefits of this, whether it’s simple correlational studies… sometimes they get to the mixed methods, even quasi-experimental studies, but the evidence just keeps mounting and mounting and mounting. And that key paper there was the 2014 one from Scott Freeman that PNAS meta-analysis showing that hundreds of papers, hundreds of studies… again, they have their flaws, don’t get me wrong… but you start to see these themes everywhere and you think, well, there must be an effect, something’s going on.

John: And we will include a link to that paper in the show notes, if anyone would like to follow up. But the fact that there may be these flaws in some of the research methodology suggests that maybe we need to see more people doing work in this area, and that there’s a lot of areas where there’s potential growth.

Justin: Totally agree.

John: And you’ve done quite a bit of research already.

Justin: I’ve tried, as a teaching faculty member now for this is going on year 12-ish, but I’ve always made that a part of my career. Before I was at Mines, I was at University of California, Irvine, where we did have a scholarship expectation or to earn tenure as teaching faculty. The UC system is one of the few in the country where you have actually tenure-stream teaching faculty, which was really cool. So I earned that, but I only had tenure about a month because then I left and came to Colorado, which… no regrets, I love it here at Mines… but I don’t have tenure here. But there’s still some expectations for scholarship, but not as high as it was in the California system. But I continue to make that part of my career, because, again, as a scientist and engineer, I think going based on data, based on evidence to guide your decisions, moving forward is a really important thing to improve what you have, because that’s all it comes down to really whether you call it SOTL or DBER or whatever, we’re trying to improve our courses, we’re trying to improve our students’ experiences. And yeah, so I’ve tried to do what I can over the years. And now I’m at the point where I want to help other people too. So I work with faculty individually, I do workshops via my side business, and even at Mines this semester, starting next week, I have a three-part getting started in DBER series kicking off for about 25 faculty on campus. I’m going to help them get started, get their feet wet in this field, and then we’ll see where it goes. Because I think a lot of folks, like you said, John, want to do this work, they’re interested in it, they might not have the time, they might not know how to, but you kind of give them the start, hold their hand a little bit and then see where it goes, and we’ll get even more evidence to support these claims.

Rebecca: I think a lot of faculty have an inclination to have a reflective practice and continuous improvement. They’re wanting to do SOTL work, but just don’t know how to formalize that practice. And we can certainly lean into helping people move in that direction.

Justin: 100% Rebecca, even when I started, I didn’t know what I was doing. If it wasn’t for Kelly Hogan and a few other folks at Carolina and then moving on from there at UC Irvine folks like Brian Sato, Diane O’Dowd, Adrianne Williams, they really helped me understand the field more. And then you start going to conferences and going to meetings and learning more about the tricks of the trades.

Rebecca: Can you talk a little bit about ways that a high-structure synchronous course might be different from a high-structure asynchronous course?

Justin: So I think the high structure while what I described earlier, you might be thinking of okay, a Monday, Wednesday, Friday class meeting at two o’clock in a lecture hall works great. I think it works even better in online for some cases, because you think about an online situation, whether again, it’s synchronous or asynchronous, your students you’re not really seeing and there’s that human connection of having eyeball to eyeball being able to understand, “Okay, I’m here I’m motivated, I’m ready to learn.” But whether it’s the recording, that’s the worst case scenario, when it’s totally up to you, or if it’s going to be on that live Zoom remote, where you can make your screen black, and who knows what else you’re doing. I’m guilty of that in meetings for sure. But having the structure though, having that scaffolding, having the clear deliverables of what is due when and seeing that step up from pre-class, in-class, after class, whether it’s the in-class is out again, synchronous Zoom session, or the recorded video you do on your own time, there’s that scaffolding helps students stay more connected with material, they understand more of what’s going on, when it’s due, what is due, how I’m being assessed, because that’s the nice thing about with structure. And just to be clear, I’m from the STEM disciplines, but you could do this any field. The research on high structure, as far as I know, is only been published in the STEM fields. I bet it’s being done in art history, I bet it’s being done in the languages, I just don’t know about it. So if anyone listening does this kind of teaching or research, I’d love to hear from you. But you can use that scaffolding again to help students be more transparent with the course. And that’s a word that I think comes through with high structure and is really true in that online environment, especially as transparency. I always tell my students, I want to know what you know, not what you think I want you to know. And by being uber clear with the learning objectives, starting with backwards design, just a clear Canvas website for the online cases can be really helpful for that. One distinction, though, with the async online versus synchronous online, is that kind of active learning piece, because we’re not there together. So one tip I like to share with faculty is that okay, if you’re recording a video for the in class, mimic, because you’re not actually meeting in class. What I always say is “okay, well make the video, but if there’s a point where you’d want them to answer a clicker question… I’m a big clicker guy… or if you want them to do an activity, just say, Okay, everyone, pause the video here. Take a couple minutes, think about this problem, what’s the answer or work on this, whatever it might be. When you’re ready, unpause the video and keep going. That’s a way that you can add the active learning to an online asynchronous class as part of the bigger high-structure model. I find when I have to record an online video, if I have traveling for a conference because I have to miss a day of class. I’ll have a 50 minute in-person class, when I make the video recording it’s only about 20 to 25 minutes, because all the active learning I don’t do in the video recording, but I give that tip… I tell my students, hey, pause the video here, answer this quick clicker question, solve this problem, come back when you’re ready. And so that’s a way to build the active learning into the asynchronous part, which again, is part of the overall high-structure piece.

John: And also, if you want to take it a little bit beyond that, you could use one of the tools where you can embed the questions directly in the video, and have them auto graded. I use PlayPosit for that, and it’s possible in Panopto, and EdPuzzle is another alternative that many people use. PlayPosit works really well. And I’ve been using it for a couple of years.

Justin: I just learned about PlayPosit maybe three weeks ago, I was doing a workshop at CU Boulder on clickers. And I kept hearing PlayPosit, it does seem like a very powerful tool to automate that active learning in the video piece, which I really like. I just haven’t had a chance to play with it.

Rebecca: So you did just mention that you’re a clicker guy, I believe. So do you use a flipped classroom approach in all of your synchronous classes? And how are you using clickers in your classes?

Justin: Yeah, I think that’s the technical term these days, clicker guy. So, I’ll go with that. I’ve been using them for as long as I’ve been teaching in various platforms. You name the platform, I’ve used it. I’ve primarily been using iClicker, just because that was the platform we had at UC Irvine and the platform we have here at Mines as well. Yes, I love using them. I think it’s really important. And you mentioned the flipped classroom design there, too, Rebecca. So when you think of high structure, you might think, “Oh, it sounds like a flipped classroom.” And it kind of is because we’re having students do stuff ahead of time. In class, they’re very active, but there’s still that structure piece even within class. I guess maybe I haven’t seen a truly flipped class because I think flipped classes, I think in class is like purely work time, purely activities, purely worksheets, homework type assignments, whereas I do a series of clicker questions, discussions before or after numerical problem solving, if it’s more of my engineering classes that I teach in thermodynamics, or material and energy balances, more conceptual sometimes if it’s biology or anatomy and physiology, but I’m breaking up a lot of activities and any talking with clickers. And so the clickers… I love all the modern bells and whistles that they have as that’s the another technical term I use, moving beyond multiple choice. And when I do workshops for faculty with clickers, that’s what I triy to get them to do. Okay, yes, you know, you can do A B C D E, but can you do multiple answers? Can you do target questions or hotspot where you touch the screen? Can you do ranking or ordering? numerical answers? …and I’m trying to get this traction on my own campus too. So even this semester, my first five questions of the semester, which I didn’t do intentionally, but after I wrote them, I realized, oh, I should do this every class now… my first five questions in my introduction to biomedical engineering class, each one was a different question type. So we did multiple choice, we did multiple answers, we did short answer, text entry, we did numerical, and we did target or touch the screen. And when I was done, I asked my students, “Hey, who’s ever used those other question types before?” And maybe four hands went up, I said, “Cool. What class did you do them in?” and they said, “Oh, your class last semester.” I seem to be the one and only that I know of on my campus that’s trying out these other tools, although like I said, I’m trying to change that. So yeah, if you’re using any kind of device like this, any type of ed tech, for that matter, make sure you justify it, justify the cost, and you make use of those bells and whistles in class. Or if it’s out of class, courseware, whatever it might be, just make good use of the tool, and you’ll find your students that are going to be able to buy into it and use it more effectively, and hopefully learn more from it.

John: And one option that we have here is a campus license, which actually is much less costly per student. And that reduces that barrier of students having to pay for it, which has led to a pretty rapid expansion in the use of it on campus.

Rebecca: One of the things that I really like about your example of the five questions is that extra level of scaffolding, of introducing all of the question types right away, and giving everybody an opportunity to practice those question types that you’re going to be using all semester. And so I just wanted to make explicit that other layer of scaffolding that you have built in there.

Justin: Yes, teaching students how to use the tools is very important. Thanks, Rebecca, for mentioning that, because whether it’s the app now on your phone for these clicker devices, or if it was the old physical device, there’s a learning curve with them. And same thing, even if it’s computational tool, or like Desmos, or something for a calc class. Yeah, you got to be able to teach students how to use the technology properly first, before you expect them to use it well. I will say though, with the learning curve of the clickers, when I was at Irvine, the app wasn’t really there yet. So everyone had the physical iClicker remote, and this was the iClicker w, so you could do text entry, and you could do number entry, but you had these up and down arrows, which you had to click to go through the alphabet. So I do the type of clicker question where I like to give four statements on the screen, and I say, “Okay, well tell me if they’re true or false, and you have to click in all four.” So you would click in a string of T’s and F, so it’s like TTFT, or FFFT, whatever it might be. And so when I would do this in intro bio there with 440 students, it sounds like a swarm of locusts, all the clicking to get to the T’s and then someone says, oh, you should change it. Just make A be true and B be false? I said “Great idea, I’ll do that.” But now with the app, we lose the sound effects and iyou can just type in T’s and F’s and it’s a little bit easier to use.

John: One thing I haven’t tried with iClicker, and I do plan to do it this semester is allowing students to rate their confidence in their answers. Have you tried that yet?

Justin: I did just last week. So I just found out that was available this semester, and I gave it a shot. And it was kind of fun. Like the first day I turned it on, they’re like, Oh, look at this, this question was 80% confident, and now it’s down to 40% confident and it kinda reflected the percent answers that were correct. So like the one that was 80% confident it was a single multiple choice answer. And it was pretty much most people got it right. The one that was lower confidence, like 40%, was a multiple select answer. So they had five options, but two or three could have been correct. And so the confidence goes way down through a different question types even. So, I think it was really fun for a day or two, and then kind of the novelty wore off, unfortunately. [LAUGHTER] So I’m going to try to revisit it maybe more in targeted times and targeted questions rather than doing it for every single one. But it was fun to try it. I’d encourage you to do it, too.

John: One of the reasons I was considering it is that it might help students improve their metacognition a little bit by reflecting on how well they thought they knew something and then getting fairly immediate feedback on whether their expectations about how well they knew it actually matched the response, [LAUGHTER] or the feedback to the response.

Justin: Absolutely.

Rebecca: When you’re using clickers, are you also using strategies like think-pair-share as part of your implementation?

Justin: Yeah, absolutely. And there’s that sweet spot of clicker responses that lead to that. So the best practices, I’d say, and a lot of this work came from Carl Wieman Science Education Institute, which I believe is still housed through Vancouver, British Columbia, but he’s at Stanford now. But there’s this great flyer they made like this trifold flyer that kind of walks you through how to use clickers properly, or best practices suggested, and then what are the possible outcomes? Typically, you’d start off with that silence. Okay, click in on your own at first, I tell them I want to hear crickets, just click in by yourself, and then I’m able to see on the app the distribution and so at that point, there’s typically kind of three outcomes. So one is maybe 80-90% have the correct answer. At that point I’ll say “Okay, let’s move on.” But I make sure to take time to have students respond and say, “Okay, well, why is it C?” or to have a little discussion about why it’s not the other answers. Because even though you get 90% correct, there’s still 10% of students who got it wrong. So we want to make sure that everyone’s on the same board. The next option is usually that kind of random distribution. Let’s say if it’s a multiple choice question, you get ABCD each are in the 20 to 30% range. At that point, I call that Hmmm, what’s going on? Was it a really hard question? Did I goof the question? Is there a chemistry exam this evening, so there’s some reason why maybe it didn’t click? And then what I call the Goldilocks case is when you get the kind of 50-50. So maybe you’re split between two answers. And I love actually, when it’s exactly 50-50, I’ll show that on the screen. But sometimes I don’t, and I’ll say “Okay, at this point, now, you’ve all had a chance to try on your own, now talk to each other.” So that’s kind of the think-pair-share that’s built into the clicker there. So this is peer instruction at its best. This was pioneered by Eric Mazur at Harvard, people like Jenny Knight at CU Boulder have studied this. And she actually has a Science paper with Carl and a few others that showed when students are doing this… So you give them some quick information on their own. Don’t show them the answer, have them talk to each other. They put a bunch of microphones around the lecture halls at CU Boulder, and they listened into their conversations. And they actually found students are legitimately teaching each other and debriefing the topic, discussing it, tearing it apart. It’s not just the one student says, “Oh, it’s A ,everyone else, click A.” They’re actually debating it and talking about it, so that you get that true peer teaching and peer instruction going on. And then typically, you see everyone moving more towards the positive correct response that you’re looking for, at which point you can say, “Hey, look, you don’t need me as your instructor, you’re teaching each other. You’re doing great. Kudos, keep up the good work.” Yeah, there’s so many ways to riff on clickers and use them in different respects. Whether it’s the more traditional question types, or now the more advanced question types, and no matter how you do it, you’re getting students more engaged, they’re having more fun with it. And fun actually is a word they use with clickers. I have a paper a few years ago from the HABS educator where I looked at clickers vs. Kahoot!s, and students rated Kahoot! and clickers both equally fun, where Kahoot! being the more gamified version with trophies and a podium and music and sound and stuff like that.

John: And both Eric Mazur and Carl Wieman and some co-authors had done some experiments on the use of clickers with that methodology with answering it individually and then discussing it and voting again. And they found some really substantial learning gains in both of those studies. And we’ll include links to those in the show notes. In all areas, we’ve been seeing some significant equity gaps, but I think since COVID, we’ve been seeing much worse equity gaps in terms of students because learning losses were pretty pronounced everywhere. And that’s especially in math and in the STEM fields, but they were especially large in lower-income communities where schools were not as well funded and where students did not have the equipment and so forth, needed to thrive as effectively in remote instruction. One of the things that some institutions have been using increasingly is co-requisites in place of prerequisites to help students who want to pursue a major in a STEM field, but who would otherwise fall way behind or take many years just to get up to the basic level of math needed for the entry level courses in the discipline. In 2021, though, you had done a study that did a comparison between the effects of prerequisites and co-requisites. Could you talk a little bit about that study and what you found?

Justin: Yeah, absolutely. We saw the same thing probably everyone else did with students struggling, returning from online instruction in high school and having online instruction here. I taught remote synchronous for three full semesters, even though at Mines we stayed open for business pretty much the whole time with in-person classes, just lower capacity. I had prior online instructional experience in my career, so I volunteered to do those online sections for our department. But with the co-rec/prereq issue, I think that’s really important to consider for a lot of reasons. And this work really started when I was at Irvine with Brian Sato and Pavan Kadandale and a few others, looking more in the biological sciences and “do prereqs matter?” It was kind of our big research question. And then I carried that over to Mines and applied that to chemical engineering too. The short answer is: it depends. So first of all, and kind of why do we even have prereqs or co-reqs? A lot of it seems to be more of a management thing. So if we want to control class sizes, moving from first year to second year, and so on, and so forth. But there’s definitely an idea of well conceptual understanding and carry forward for if you have Bio I, well, those skills you develop could be used and count for something when you get to Bio II or same thing with the calc series or whatever it might be. However, we don’t know. It’s unfortunate that I think programs don’t sometimes look at those connections, that vertical alignment, if you will, between courses in terms of prereq to the following class, or with the co-req issues. At Mines, what I did with the paper you’re referencing, John, in chemical engineering, we have these two courses. One is called material and energy balances, MEB, I’ll call it for short. And that’s kind of like our intro bio version for chemical engineering. It’s kind of like a survey course that covers a lot of different topics, gives you some foundational skills that you’re going to need as a chemical engineer. We also have introductory thermodynamics, and an intro thermo, it’s related to MEB, covers some same conceptual understanding and problem-solving skill development, but it’s more on the nature of energy, and that transformation understanding. So typically in chemical engineering curricula, and I’m getting in the weeds here, so apologies for anyone who’s tuning out cuz you’re not a chemie, but MEB comes first, that’s typically either the fall of sophomore or spring of sophomore, you take that. First, it’s kind of like your gateway class, and then that’s a prereq to intro thermo and everything else you take. At Mines though, we do a little bit different. Mines, we do intro thermo first, fall of sophomore year, and then in the spring of sophomore year, you take MEB and coming from that background myself as a student at Penn State, I wish I would have had that way because intro thermo is the more… I use the word gentle… it’s a gentler introduction to chemical engineering than material energy balances is. But the idea here is that you can take thermo as a prereq in the fall or as a co-req in the spring with MEB. So the natural question came up… it’s a nicely well organic experiment to test is, well, if you have the thermo as the prereq do you do better than if you take thermo as a co-req with MEB. And this has implications for advising and its implications for progression through our major through repeating classes, all sorts of things. So it’s a really worthy study and my student, Jordan Lopez, at the time, he was an undergraduate student in mechanical engineering, and I love working with undergraduates on these SOTL and DBER projects. Jordan came along with me for this we did a quantitative assessment to figure out okay, well if students in MEB, did you have thermo as a prereq or co-req? How’d you do in MEB? And then we also asked the students qualitatively “Okay, well, do you think having thermo as a prereq or co-req matters? And if so, what do you think? And what we found was there was actually a big deal. Students who took thermo as the prereq did significantly better in their overall course performance in MEB than if they took thermo as a co-req, and that’s using linear regression models controlling for incoming GPA and things like that. Students also felt the same way. They felt that if you have thermo as a prereq, you would do better than having thermo as the co-req with MEB, and again, the quantitative data bore that out. The reasons they gave for that was because they get some opportunity to get more familiar with the discipline, with how we approach chemical engineering, how we do problem-solving skills, and having that foundation in the prior class allows them to do better in the follow up MEB course. In this case, it mattered, prereq vs. co-req. However, in our studies at Irvine, with Brian and Pavan and others, we found it didn’t really matter so much whether you took for example, microbiology lecture before after microbiology lab, you would think those two would be highly correlated one as a prereq to the other, it didn’t really matter in the result. Same thing with anatomy and physiology sequences, it didn’t seem to matter too much, in other papers in which we cite in these studies looking at organic chemistry sequences, economic sequences, it’s kind of hit or miss with whether prereqs matter or not. Long-winded segment here, but I think we as programs should be taking careful look at our prereq sequences and core sequences in general to see what, if anything, our students carrying over from one course to the other, and maybe we should loosen that up a little bit, because if they don’t matter, that can help students be a little bit more flexible with their degree planning, and time to graduation and things like that.

Rebecca: Speaking of prerequisites, faculty are often complaining that students don’t necessarily recall [LAUGHTER] information from prerequisites as they go into a sequence like we might expect. In 2018, you conducted a study that investigated the effect of physiology prerequisites on the future anatomy and physiology courses. Can you talk a little bit about that study and what you found there?

Justin: Yeah, so that was what I was alluding to there, Rebecca, with some of the work I did with Brian and Pavan. And in those cases, there wasn’t too strong of an effect, if anything, with the sequencing and the ordering that students took these classes. At Irvine, we did have human physiology as a prereq to human anatomy. I know that sounds a little funny to those of you in the A&P space. But the reason we did that was because of kind of a controlling the size of things. Our physiology course there was a lecture-based course, it would have 3-4 hundred students in it every quarter… talking about over 1000 students a year taking this class, versus the anatomy class, which had a lab, we can only handle about 144 per quarter, so availability of seats in the human anatomy was much, much lower than physiology, and also not as many students wanted to take anatomy versus physiology. So that’s kind of why that ordering existed. And so we tested that, though, looking at with what Brian and Pavan and I came up with, we call it a familiarity scale. How do you do on questions that are based on how familiar you are with the content based on the prereq? What we did was we had the prereq instructors look at the follow up course exam questions and write them as very familiar or not familiar based on basically whether it was covered or not. And then we looked at student performance on those questions based on having the prereq or not, and we didn’t see much of a difference based on the very familiar versus not familiar questions. The way we designed those studies, we didn’t come up with any strong conclusions, again, on the biological sciences side saying that, yeah, these prereq sequences matter. But on the chemical engineering side, which I just described, that seemed to totally make a difference, the ordering of the classes.

Rebecca: It seems like it’s really important to think about why we’re structuring the curriculum in the way that we are. And is it just traffic control? Or is there actually a legitimate reason why we’re hoping that they are scaffolded in the discipline?

Justin: Yeah, I think it’s reality, a little bit of both that happens, which makes sense, kind of have that traffic control, but also making sure that that alignment occurs through our new programs or existing programs, and just taking the time as a program or department every so often to say, hey, let’s look back at this and see, why are we doing it this way? Things need to change in this modern day.

John: One thing I wonder a little bit, though, when instructors are complaining about how little students remember from the prerequisite courses that they’ve taken earlier, is how much of that may be due to the incentives provided to students. The way many courses are structured, not in the sense of the structure you were talking about, encourages students to do a lot of cramming before the high-stakes exams. And we know that that results in very little long-term recall. Might some of these issues be resolved if we could encourage students to adopt better learning strategies, and if we could design some of our courses at the lower level, perhaps, to build that in, using the type of structure that you’ve been talking about? Because at least from what I’ve seen, that’s not so common? You picked up this stuff fairly early, but I don’t think that’s part of the training of most of the faculty in STEM fields or in most disciplines.

Justin: Yeah, I agree with that. And there was even in 2018, I believe, in Science, a paper, I think they call it “The Anatomy of STEM Education in North America,” it was something along those lines, where they looked at a big survey a big swath of courses and trying to see who’s using Active Learning who’s not, they use the COPUS protocol, the observation tool to characterize what’s happening in the classroom, and they broke it down in these seven different buckets. And even at that time, a very small percentage of the courses surveyed in STEM were taught using these student-centered strategies, not necessarily high structure, but the active learning, student-centered ideas. But I completely agree with you, John, if we can get more faculty on board with using these types of evidence-based practices, which yes, it takes some time, takes some energy, takes some foresight to develop these courses. But you do the same thing with your research, it takes your time and energy and careful thought to put in your research experiments. We should put the same energy into our classes too. And if we can do that at the early stage, which again, there’s such a huge issue with loss of learning. Not only that, but also lots of students from the early stage. There’s a paper after paper for these bigger courses, taking them as quote unquote gateway classes, we get a lot of attrition at these levels especially of students who might not be as well represented in STEM. We can add high structure. We know that works. We know that helps students do better and succeed. It might not work the first time, don’t get frustrated everyone. There’s a great paper from Anne Casper and Scott Freeman about having True Grit with being able to try and try again. So in her case, she tried adopting high structure at a smaller school in Michigan, it took her about three tries to get the right formula for it to work for her students to see success. And that paper documents that trial and error over a few semesters. But if we can use these types of methods in our classes, ideally, our students are going to be better prepared then for future classes. I do these things called Reading guides, which again, another shout out to Kelly, I got a lot of inspiration from her at Carolina for these, these are just Word documents that I make. And I have a bunch of these on my professional website. If anyone’s interested to download them for free, go for it. They’re just Word documents that help students read the textbook and again be more transparent with what I want them to know. So it has them define terms, fill out tables, make little drawings, they answer checkpoint questions in the text, summarizing things in their own words. And with the reading guides, we’re able to help students again acquire that content and then use it in the class as needed. I’ve had students in follow-up classes say to their instructor, “Well, where the reading guides at?” And they’ll come to me like, “Hey, what are these things? What are they talking about?” So the point being with that story, though, is that I tell students that “Hey, even if you go to a follow-up class, where it’s not highly structured, you’ve developed so many great metacognitive skills, you’ve developed so many great self-regulated learning skills through this class that you can then apply it and add your own structure to follow-up classes, if needed. And these are the kinds of tips that I love working with faculty about to try to help them transform their classes when I visit a campus and do a workshop or work with faculty one on one or on Zoom or whatever, trying to take these practices, which again, are rooted in the evidence, they’re rooted in the literature, but we have to tweak them to your own personal situation, your own student body, your own courses and demographics and everything. You all know your students way better than I do. What I try to do is combine what I know from the literature and the evidence. And that says a lot of this is in the book I’m writing on high structure, it’s going to guide you through the course-design process, a lot of practical outcomes and deliverables. But you got to tweak it and match it with again, your unique circumstances to achieve those positive outcomes.

Rebecca: I think one of the nice things about introducing students to some strategies for high-structured learning and retaining information is that if you have things like your reading guide, and they show it to another instructor, now they might request that of their instructor and the other instructor might decide, “Hey, this is a good idea. This is a strategy I want to adopt too.” And that’s some of the best ways to get some of these techniques to spread.

Justin: It really is. It’s that camaraderie building through seeing each other’s practices, seeing what’s going on. I’ve been part of a project here at Mines since I got here almost six years ago now where we’ve been working on kind of revising how we evaluate, and what we consider effective instruction to be. So we’re trying to take a more holistic review of how we take into account the best practices of teaching and student learning. And one of the big pieces there is peer observation. So we’re actually rolling that out, as of this week and last week with doing departmental trainings and helping faculty see how we can do peer observation to learn from each other. I learned so much from going into another person’s classroom or looking at their Canvas site, or their syllabi or their materials. And I really got this from Malcolm Campbell, who was at Davidson College in North Carolina, when I was a postdoc, this is going back 14 years or so. And Malcolm told me once on the phone, he said, “Find someone who you consider to be a master teacher, and go watch him teach.” But he said “don’t go just once, go a lot. Once a week, if you can for several weeks, and you’ll get to see them handle different situations, you get to see the ups and downs of the semester, you get to see what it’s like on the day where the midterm gets passed back. Or you get to see what it’s like when they’re getting to a really difficult topic or there’s some kind of disruption that class and how they handle it.” That’s the one great thing about peer observation is that if I go to your classes, sure, I’ll be able to give you some comments or feedback on what I saw, but I’m going to learn probably more from seeing you two teach than whatever I could give you and then I’m able to incorporate that into some of my lessons and my teaching style. If we can just change that culture, make it a little bit more okay sometimes to talk about teaching, share teaching strategies, see each other teach and have fun with it really and learn from each other, I think that can go a long ways towards, again, improving student outcomes and helping them do better in all aspects of college and life.

Rebecca: I found one of the most fun opportunities for observations is doing observations outside of your own discipline. It’s really informative to see how people in wildly different disciplines approach teaching and the way that they operate their classroom. I remember visiting a dance instruction classroom when I was at Marymount, at my prior institution, and it was really interesting, it was fascinating to me about how specific and transparent every little move was and how to correct things and the anatomy that was being discussed and just the level of depth that you might not realize how something transpires in a classroom unless you’ve actually had the opportunity to observe it.

Justin: I’m going to play that some for my faculty Rebecca. I thank you, because that’s what we’re trying to do at Mines is not intra-departmental, but rather inter-departmental observation. So getting across the disciplines for sure. Because yeah, you learn so many different things. And even if you’re just going from mechanical to electrical engineering, there’s still a lot of differences between those types of courses, you might see what’s going on. And I’ve learned that myself, I’ve watched some economics instructors teach at Mines, no idea. I feel like I maybe should know this stuff to be a better financially literate person myself, but I learned so many cool ways about how they’re teaching, but I also learned some cool concepts.

John: We do have an open classroom project here on our campus, we keep hoping more people will open their classes and more people will attend. But the people who have participated have found it universally to be valuable. So we’re going to keep doing it, and we just hope to see it become more a part of the culture.

Rebecca: Sometimes just indicating to a faculty member that you’re interested in their content, and you just want to sit in because you’re curious is a great way to learn new content, and maybe get an opportunity to observe someone teaching. [LAUGHTER]

Justin: And yeah, I definitely pitch it that way. Like I want to learn from you. And you get some new ideas. And I feel like in that case, faculty are more than willing to have you in. Well, yeah, Best of luck with that open classroom project. I’ve seen those types of programs in the country kind of sometimes hit sometimes miss. But best of luck with that.

John: One of the things you’ve done is just some research or some experiments involving two-stage exams. Could you talk a little bit about how you implement them, and what you found in general? And how did students react to the two-stage exams?

Justin: Yes, so I’ll try to be brief on these because I could spend probably 45 minutes on these myself. But yeah, two-stage collaborative exams, or more affectionately called group quizzes or group exams for short. There’s been a ton of literature on this Jim Cook, UC San Diego, he really helped me get started in this about five years or so now. But it’s all disciplines. You look it up, and the one paper I have, you mentioned in the intro, there’s a lot of references to different STEM disciplines that use them and some non-STEM too. The idea, though, is that we push active learning, we push group work, we push collaboration, but then we assess you by yourself with a pencil and paper and a calculator on a piece of paper, so why not change the dynamic there a little bit. And the two-stage idea is that first you have students take an individual quiz or individual exam, and I personally have gone to a weekly quiz model with my high-structure courses. I don’t do midterms anymore, I don’t do finals sometimes even. I know that’s taboo sometimes, but it depends on the class and I might not even do that. My colleague in Irvine Adrienne Williams says with high structure by going to the weekly quiz model, instead of having these giant peaks of anxiety with the two or three midterms, we just have a smooth level of anxiety the entire semester, and spread it out a little bit. But, for me, group quizzes then, what they look like is students take the individual quiz, turn it in, then they get in a group, this is a predetermined group, they know who their group is going to be. And then I give them a group quiz. That group quiz in the paper that you referenced was an identical one. So they took the identical two problems. And this was a MEB material energy balance class. So it was just two numerical problems, problem solving. They did that again as a group. However, my other courses now I do a different quiz. It’s either an isomorphic question, so slightly similar, but a little bit different, maybe in intro thermo or if it’s biology or anatomy and physiology, completely different set of questions. Then they do it as a group. The reason we do this, again, was trying to get this collaborative piece in the group setting. But also, when do you find students are most excited to talk about a quiz or an exam? It’s as soon as it’s over, they go walk out, the hallways abuzz, everyone’s all excited, good or bad excited, but they’re talking. So let’s capture that energy and do it in the classroom through that group assignment. Now, the group quiz is usually a fraction of the weight. In the literature, you’ll see, it’s usually let’s say you call the whole experience 100%, the individual quiz, maybe 80%, the group quiz is 20%. But that can vary from 75 up to 95, however you want to do it. And again, they’re covering the same kind of concepts just in that group dynamic. And so when I did this the first time, I got some consultations from my co-instructors, Rachel Morrish, Dave Marr at Mines, and I was a little bit nervous to see how it would go. But it was a hit. It was an absolute blast. The energy in the room is one thing I cannot describe it strongly enough here through audio, it’s just you got to see it. The students are energized, they’re talking, they’re debating. They’re working through the problems together. They’re trying to figure out, I will use the F word here. It is fun. It is a very fun experience to see. I’ve had students tell me that after the quiz is over, like that was a lot of fun. Talking about spreading things on campus. This is one thing that has spread a little bit at least within my department of chemical engineering, a few other classes have been using them now. And I collected some data on this, which I published, showing that… which makes sense, there was no shock… but the group quiz score is higher than the individual score because you’re tackling this together right away. And then also though, students strongly preferred it. The affect, the positive attitude response was through the roof. Both of those results, these are replicated from other disciplines, they see the same kind of things. I think it’s a really nice way to promote that community in the class, show that you’re working together, add that active part to the assessment. However, I’ll end with saying there can be some resistance to this, especially from students who might feel like “hey, I can do it all on my own, I’m fine.” And then they have to work together with maybe some students, they don’t know as well. You might get some resistance from students saying that I don’t want to work with someone else, I wanted this on my own. So be prepared for that. Also, accommodations with testing, if you have students who get extra time, that can be difficult to manage. I will say though, from doing this now about five years and four different classes at Mines, I’ve never had a single issue with students and extra time. What I typically do is I’ll have students get the extra time for the individual part, then they come join us for the group part and have the normal amount of time because in that group setting the students with extra time, they’ve always told me and I always ask them, I say, “are you okay with this?” And they’re like, “Yeah, it’s fine to have the normal time because I’m in a group.” So just some things to be aware of, if you’re going to try them out. And again, I could go on and on with these, so if anyone wants to learn more, hit me up. I’m happy to chat with ya.

Rebecca: Sounds like a good day to visit a classroom, huh?

Justin: It’s an energetic day for sure. Although you might not see much teaching going on if it’s just the assessment part. But it’s fun to drop in and see what’s happening.

John: There is a lot of teaching going on in their second stage, though.

Justin: Absolutely.

John: In fact, when I first did it, I was so excited about it, I made a short video recording and sent it to Rebecca, so she could see the dynamics of that interaction.

Rebecca: And hear it.

John: It was just really, really different than going over a quiz, where some of them would be really happy, but not that focused. And the students who did poorly were generally pretty disengaged when we went over a quiz ift they didn’t do so well. But when they were explaining it to each other, it was so much more positive.

Justin: Yeah. And you’re right, the teaching is happening. It’s coming from the students to each other. Not for me necessarily, but it’s coming from them. And it’s so powerful to see… absolutely to hear that live.

Rebecca: But you’ve set up the structure to facilitate it. So therefore, you’ve done the teaching.

Justin: [LAUGHTER] Thank you. Yeah.

Rebecca: So we always wrap up by asking: “what’s next?”

Justin: Yeah. So for me, what’s next? You mentioned at the top, John, I have this new position, I’m Associate Dean of Undergraduate Studies. And I mentioned in passing, I have a side business I called Recombinant Education. I do workshops for faculty. I’m writing a book. For me, I don’t know what happened, but last year, I had a birthday the ends in zero. Things have been a little wonky in my brain lately. And I listened to Taylor Swift now, sometimes. I never used to, but I think a lot of people do that. So it’s not just me turning that age. But the point being is that I’ve been doing this enough time now that I feel like I have things to share. And I’m trying to do more of that. So I’m trying to, and I’m still nervous to do things like this kind of stick my neck out and even talk to folks like yourselves, but I want to share best practices with teaching and learning, specifically with STEM education, because I still hear from students and I still see it from time to time, students are just not getting the best classroom experience they should be getting. College keeps getting more and more expensive. Students don’t deserve to have less than excellent experiences, in my opinion. I’m doing what I can on my own to share these best practices. Someone called me a one man teaching and learning center, it’s probably a fair way to say it. One of my students said, “Oh, you’re teaching teachers how to teach,” and that’s, I guess, another fair way to say it. But again, I got the energy for it now. I kinda have this midlife crisis going on too where I want to make some bigger impacts on the world outside of my own bubble, whether it’s at the administrative level at Mines now or across the country, or even the world. So I love working with other faculty on all these different areas. And that’s what I’m trying to make a big part of my day-to-day life now. But I’m still in the trenches. I’m still teaching classes every semester, because that really keeps me rooted, lets me try some new things out, still doing the DBER work and the SOTL work, working with students and just trying to have a blast and do what I can here in the time I have left.

John: Well, thank you for joining us. And we’re really looking forward to that book. And I hope you’ll come back and talk to us again when the book is close to coming out. Or sooner.

Justin: I would be honored to come back anytime John and Rebecca and if both of you are ever in the Denver golden area, please let me know. I’d be happy to show you around out here.

Rebecca: Well, thank you so much for joining us. It was fun to talk to you and to hear about the fun happening in your class.

John: More F words.

Rebecca: Yeah, [LAUGHTER] a lot of F words.

Justin: Yes. [LAUGHTER] Thank you both. I appreciate it. It was fun.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

GANESH: Editing assistance by Ganesh.

[MUSIC]

318. Reducing Equity Gaps

Gender and racial equity gaps exist in economics and other STEM fields. In this episode, Tisha Emerson joins us to discuss research on strategies to reduce these inequities. Tisha is the chair of the economics department and the James E. and Constance Paul Distinguished Professor at East Carolina University and is the incoming Chair of the American Economic Association’s Committee on Economic Education.

Show Notes

Transcript

John: Gender and racial equity gaps exist in economics and other STEM fields. In this episode, we discuss research on strategies to reduce these inequities.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Tisha Emerson. Tisha is the chair of the economics department and the James E. and Constance Paul Distinguished Professor at East Carolina University and is the incoming Chair of the American Economic Association’s Committee on Economic Education. Welcome, Tisha.

Tisha: Thanks for having me. I’m so glad to be here.

John: I’m very happy to be talking to you. And our teas today are:… Tisha, are you drinking any tea?

Tisha: Yes, my favorite tea, I found a couple years ago, it’s a Golden Moon brand of Tippy Earl Grey.

Rebecca: That sounds nice.

Tisha: It’s delicious.

Rebecca:I have Harvest Memories today.

John: I don’t remember that one.

Rebecca: Well, it is from my little favorite tea shop in Canandaigua, New York, and it’s autumn flavors.

John: Autumn flavor?

Rebecca: Autumn flavors.

John: Do they have leaves dumped in there?

Rebecca: I mean, it almost looks like that. [LAUGHTER]

John: Ok. [LAUGHTER] Well, that’s one way of getting rid of all the leaves that have been falling. And I have English breakfast tea today. So we’ve invited you here today to talk about some of the research you’ve done, and also your new role and your past role with the AEA’s Committee on Economic Education. One of the things that we’ve observed is that there’s some significant equity gaps in economics and in STEM disciplines in general in terms of race and gender. And in one of the studies that you’ve done with this, in a 2023 paper in the Southern Economic Journal, you and KimMarie McGoldrick examined retake behavior for students who are not successful in their initial attempt at completing an introductory microeconomics class. Could you give us just a general overview of this study?

Tisha: First, let me say that this started a long time ago. And we got access to this great data, the MIDFIELD dataset, that was actually originally funded by the NSF to study the gap in engineering education, but it’s student transcript data. So we said, well, you could use that for economics too. And we have. And so this particular study looked at almost 180,000 students who take Principles of Microeconomics for the first time. And what we ultimately wanted to do is to think about the likelihood of success, and actually more so, failure, because a lot of papers already look at success and they stop there, because you have too small of a sample, fortunately, of the failures to continue on to look at them. But when you start with 180,000, you have enough to continue. So we find, as many others do, that, of course, aptitude matters, but that, unfortunately, females tend to be much more likely to be unsuccessful in their Principles of Microeconomics class, as are underrepresented minorities, or URM. And then we follow those students and we say, okay, so if you were, in fact, unsuccessful, and we define that as a grade of a D or an F, so grades that wouldn’t really let you continue in your study of economics, or you withdraw. And we say, if you’re in that category, what do students tend to do?… or first of all, for those students that are unsuccessful, what helps predict that? So given that you are in the unsuccessful category, what is more likely to cause that? And we saw that students who were carrying more concurrent credit hours were more likely to withdraw. Which makes sense, because if you have constraints that lead you to want to be full time, if you start with 12 hours, and the course is not going well… any of them… and you were to drop, then you would not be full time anymore. And so it gives them sort of this flexibility, they have more concurrent credit hours. But we found that students who got D or Fs, given that they were unsuccessful, they tended to have taken more related courses. So things like accounting (financial accounting in particular), calculus, and macro principles. And so we didn’t actually see a lot of gender or racial differences there. But then those students who were unsuccessful (D, F, or W), on retake decisions, we found that women were much less likely to retake if they were unsuccessful. But we did find that women who were of higher aptitude, who were unsuccessful initially, they were more likely to retake. So that was a positive. And we did find that underrepresented minorities were more likely to retake the course, but then more likely to be unsuccessful again, on their second attempt. So there’s some good news, but also, a lot of not so good news.

John: So you were able to follow students over time, how long a time period was there in the data set that you were analyzing?

Tisha: We had over 20 year period of data, but that doesn’t mean we followed any particular student for that length of time. Once we saw that they were unsuccessful, we made sure that they had at least one more semester at the institution so they did in fact have a chance to choose to retake the course or not and then we gave them three years post their initial unsuccessful attempt. And if they didn’t retake it in that period, then we said they chose not to retake.

Rebecca: In a 2019 study in the Journal of Economic Education, you and KimMarie McGoldrick use the same dataset to examine the decision to switch into an out of the economic major. In this study, you find that very few students selected economics as a major when entering college and that 83% of economics majors actually selected the major after taking their first economics course. What did you find in terms of the gender and racial composition of those that selected the economic major?

Tisha: So what we found was that, for the few students who started out as economics majors, so going into their principles class they already had said they were majoring in economics, we found that females were as likely to persist or not. That’s good. And we found something similar for underrepresented minorities, that is the females and URM students who went into the principles class planning to major in economics, they were as likely to persist as not.

John: What happened to those students who were not majoring in economics.

Tisha: Well, unfortunately, females were not as likely to persist and decide to major in economics. There was a larger proportion of females who opted to remain in their original major. So, they come into the principles class, they have some major other than economics, they were much more likely to stay in their other major and not switch to economics. Men were considerably more likely to switch, on average, than females were. But on a more positive note, we did find that students from traditionally underrepresented racial and ethnic minorities and Asian students were more likely to switch to economics.

John: And one thing we should note, and you mentioned this in your study, is that because this datset was focused on engineering, the schools in this sample have relatively large engineering programs and some students switched to economics because economics was perceived as being easier than their original major. So some of this may not apply as broadly to liberal arts institutions that do not have engineering programs.

Tisha: That’s possibly true. And in fact, business majors were even more likely to switch to econ. So it was about 10% of those switching in were switching from engineering and 27% were switching from business. So at the liberal arts schools, to the extent that they don’t have business or engineering, there wouldn’t be that source of majors coming in.

Rebecca: Both of these studies are really interesting. But, how do we think about some strategies to create a more inclusive environment in economics and STEM classes? So what did these studies tell us? What should we be doing?

Tisha: Well, I don’t know that they tell us what we should be doing. They tell us that what we’re doing is not really working.

Rebecca: Fair.

Tisha: …which is unfortunate, because that’s what we’ve been doing for a really long time and we’ve been really unsuccessful at attracting women and underrepresented minorities. So there’s a new sort of strain in the literature that’s really trying to address some of this. And there’s a study, for example, by Porter and Serra, and that was published in 2020. They did a randomized control trial at Southern Methodist University, where they, just by happenstance, picked a couple of very charismatic female alums and they randomly selected which principles classes they would speak to. And this was just like a 15- or 20-minute exposure, where they talked about majoring in economics, and how that helped them in their career. And they found that this significantly increased the number of women from those courses who decided to major in economics and additionally, not necessarily just major, but take more economics courses. And I thought it was really fascinating that there was no effect on men in those treated classes, it was just the women. So that suggests possibly that there’s some room for role models. Other work that looks at role models from other directions, like some of my own work with KimMarie and John Siegfried, didn’t find any evidence that supports the idea of role models. So I think, still mixed, and the exact sort of interaction that students may or may not have with these people I think is going to be important.

John: You’ve done a number of studies on these types of issues, and one of them was looking at the effect of classroom experiments. Could you talk a little bit about what types of classroom experiments have you looked at and what’s the impact of those on students?

Tisha: Sure, I talk about that a lot, because I really love the classroom experiments. I think that they’re a great pedagogical technique. And so since your audience is more general, maybe I’ll explain just what we mean by classroom experiments. And that’s basically the idea that we’re going, in some cases, simulate markets or other decision environments for the students that will mirror the types of environments that we’re talking about in class. So for example, if we’re talking about a market, then we will have students participate in an experiment where some are buyers and some are sellers and they negotiate and trade. In my classes, we always do the experiment first, I don’t talk about any of the concepts. And then we collect the data as we’re going through the experiment on the decisions that they’re making: prices, quantities that are traded, then I’m able to talk to the students about “Well, look, this is what actually happened, this is your actual data that you generated, and now let’s talk about the theory.” And this is what the theory would predict the outcome to be. And they can see how well they match up. And they are able to discuss in class, this is why I chose to behave in this way. And they see how it fits in to the economic theory. And I did a study back in 2001, which was published in ‘04 in the Southern Economic Journal, where we were looking just at the effect of experiments and we looked to see if there was a differential gender effect. And in fact, there was. So you often see that women underperform men in these principles classes. But what the experiment did is it closed that gender performance gap. So it raised sort of everybody’s performance, but it raised the females disproportionately, so that they were closer to the same outcome as the men if they were in a classroom with experiments. So I do think that there is definitely room through pedagogy to improve women’s outcomes, and make them more attracted to economics, but also make economics more attractive to them. There’s a large literature that suggests that women are much more grade sensitive than are men, various people have gone about showing this in different ways. And if you can bring up their performance so that it’s on par with men. And it’s not really even that comparison, but you’re bringing up the females’ performance, then they’re getting better grades, and they’re going to be less likely to want to leave the study of economics.

John: And we’ll throw in a reference to an earlier podcast we did with Peter Arcidiacono, who had looked at that very impact in STEM fields in terms of the decisions of people to move from one major to another. And in particular, in that study, they found that women did better on many of the tests but still left because the grades were relatively lower than in other classes. So one issue I think might be worth addressing is the issue of grade differentials, that if we’re going to continue to assign grades that are lower than most other departments, there’s a good chance we’re going to lose majors in general, but disproportionately more female majors, based on what the research literature tells us.

Tisha: There’s this really cool study, McEwan, Rogers, and Weerapana, 2021. They’re at Wellesley, and so it’s all females, but they were trying to address grade inflation. And what that meant was, of course, there are some disciplines that have higher grades, they had to bring their average grades down, and economics is treated in the opposite direction, and that it actually drew more students into these harder grading disciplines, like economics and the other STEM fields. And so yeah, that’s certainly an issue. I don’t think we’re gonna across the board be trying to treat grade inflation. I don’t think this is published yet. But I was reading another study out of Wellesley, where they were allowing students… I think it was maybe for their gen ed classes… to take them all pass fail. And again, you saw students flowing into these harder grading disciplines.

John: I know a number of schools during a pandemic made all of their first-year classes pass fail, which is a good way of letting students get into a discipline without worrying so much about the grades so they can explore things without worrying about the impact on their GPAs. That might be another useful strategy.

Tisha: Yes, I agree, and then this is really anecdotal, but I’ve heard other people observe the same thing, that women, they’ll be in your principles class, and they’re doing well, but maybe they’re going to get a B plus. And they say, “Oh, either I’m not gonna take another economics class,” or I’ve even had students withdraw at that point, and I’m like, “but you’re getting a B plus, I don’t understand,” but they’re so sensitive to grades. And then there will be male students who are getting a C, barely, and they’ll say, “I’m going to major in economics.” and I say, “Okay, are you sure?” It’s hard to understand sometimes.

Rebecca: It’s interesting to think about withdrawal policies that have really kind of loosened up due to COVID. So our institution changed the policy more recently so that students can withdraw through the last day of classes without documentation. And so students who are doing really poorly might take that option, but as you’re discussing students with higher grades withdrawing, that sounds really concerning that policies like that could be kind of counterintuitive, or really have some other negative impacts. Classes all cost money, [LAUGHTER] and there’s financial impacts to withdrawing.

Tisha: It’s hard to really predict how this is all going to play out. And I have to say, I’ve never heard of such a generous withdrawal policy. At my previous institution, I think they could withdraw up to sort of two weeks out and not even have a W necessarily on their transcript. So this all seems like a lot. There’s a lot of moving parts. But on the positive side, so let me just say, first of all, part of me feels like it’s a little bit too generous. But part of me also thinks back to the work that KimMarie and I did. And if you don’t have that D or F, that you have to overcome, it’s a lot easier to persist, not just in your major, but towards graduation, whether you switch majors or stay. And part of me feels like that’s helpful to the student. So in some ways, I like a generous withdrawal policy, I definitely want students to know what the withdrawal policy is, what the deadlines are, so that they can make a good decision.

John: Yeah, the only concern I have with the policy is that it can lead to students making slow progress towards a degree and they could run out of funding before they complete their degrees. But on the other hand, it does provide a bit of a safety net for students who are adjusting. And as our student body has become more diverse, and we have more first-gen students, and we have more students coming from schools that did not prepare people particularly well for a college environment, it provides a safety net, which has allowed students to take some chances early on and not be harmed. And I’ve seen some students struggle in their first year or so, drop classes, and come back and be very successful, when before they might have been deterred from further study in the discipline.

Tisha: I agree, I can see that.

Rebecca: So, definitely one of the things that I’m always thinking about when students are talking about withdrawing is making sure that they know all of the ramifications. So what does it do to your GPA? What does it look like on a transcript? What are the financial implications? …so they can make a well informed decision. And depending on who they’re talking to, they are not necessarily always getting all of the information, and that can be problematic, too.

Tisha: Yes. So I think that points us to the issue of how important advising is, so that there are well informed advisors and that students have access to them, not just at the end when it’s time to register for next semester, but they have a relationship with their advisor, and feel that they can go in and ask those questions, because as John said, a lot of those students can’t go to their parents, they’re first gen, their parents don’t have advice for them. And even if your parents went to college, they don’t necessarily know all of the rules and regulations of the institution that you’re at. It’s really hard, I think, for students to have all the information that’s necessary.

John: I’ve had trouble keeping up to all the changing information [LAUGHTER] over the last few years. In terms of those requirements. You’ve done some work with cooperative learning. Could you talk a little bit about that?

Tisha: Sure. So that really came out of the fact that I had done a lot of efficacy work around classroom experiments. And KimMarie and I are really good friends. She is a cooperative learning expert. So I should say that she is the expert on CL. And she wanted to do an efficacy study, so we decided to team up and do that. So what we did is unlike a lot of some of the other work in efficacy, is that a lot of the work is comparing lecture. So you don’t do anything to this active learning technique. And with cooperative learning, the students are working on exercises. And when we talked about it, we said you know, it’s really not fair, and not even interesting to compare a student who is in a lecture-based class where they don’t get to see any of these problems to students in a cooperative learning class that are working on these problems in class in teams. So we thought a lot about the collaborative learning approach. And we’re trying to isolate what we thought could be the mechanism that might be driving different outcomes. And what we decided was we were going to compare the active team-based learning to individual learning, working on problems. So in our control, the students were exposed to the same problems, they had the same amount of time in class to work the problems as in the treatment, which was the collaborative learning. It’s just that in the treatment, they were working in teams in this think-pair-share share framework, and we didn’t find a significant difference. So at the end, we said that the cooperative component didn’t seem to be driving any difference. Although there are other people who’ve done work comparing collaborative learning to lecture based and they did find significant positive effects from cooperative learning.

John: One type of thing that I’ve done that seems to have been fairly successful is using clicker questions. Following the methodology of Eric Mazur, students are given a challenging question where typically half or so them will get it right the first time when they’re asked individually. But then they get to talk it over with the people around them, preferably someone with a different answer, they get to debate and argue it and I normally see between a 10 and 20 percentage point increase in the correct answer. And the really absurd answers tend to disappear down to about nothing. So that’s not a very formal study, but that second stage of that process where students are engaged in some peer instruction seems to have had a pretty significant impact. And I point that out to students, that when they talk to each other about it and explain it to each other, they do much better than when they’re working on their own. And I encourage them to try doing that outside of class, to work with other students, because there are a variety of studies… I haven’t seen too many in economics, but there are quite a few studies that find that that type of peer instruction, either in the classroom or outside, can be fairly effective.

Tisha: Yeah, so I’ve observed that too. I haven’t used clicker questions, but just going in and observing my colleagues when they teach, and they are using clicker questions. And I have to say, I’m stunned, because I don’t know how the right answer bubbles up as opposed to the person who had the wrong answer convincing the other of the wrong answer. I don’t know how that doesn’t happen. It’s like magic to me, almost. So part of me wants to think a bit more about what is the mechanism and how is that working? That’d be really cool to do a study on that.

Rebecca: One of the things that’s always curious to me about that particular dynamic is sometimes students discover, as they try to explain something to someone else, [LAUGHTER] that they really don’t know what they’re talking about. [LAUGHTER] They realize, huh, I can’t actually explain this concept, so maybe I’m the one that’s not correct. [LAUGHTER]

Tisha: Maybe that is what’s happening. Because everyone always says, “You have to really know your stuff to teach it,” and so if you are trying to explain it to your partner, you do have to know the material.

John: And when they’re raising objections and you don’t have a counter argument, that can help break down your misconceptions, because getting rid of those incorrect perceptions can be as important as trying to build new ones.

Tisha: Yeah, I could see that and maybe the people who are initially giving the wrong answe are just guessing. So maybe they’re not trying to explain and convince their partner of their answer, I guess. I don’t know. So maybe it’s not magic.

Rebecca: [LAUGHTER] Feels like magic.

John: It does. And it works.

Tisha: When I see it happen, it does seem like magic. In general, I think that active learning is helpful, because I think that women are going to be more interested when you show them the applications, and that when you have active learning, there’s some community building that happens. So you’re illustrating relevance, you’re building this community, giving them a sense of belonging in the classroom, which is a lot of what I think actually happens with the experiments. And then actually also with the experiments, they’re kind of discovering some of the theory for themselves, which is a growth mindset. And there is research that shows that if you can show relevance, belonging, and growth mindset, and develop these in your classroom, that students are going to do better, but that it also is more appealing to females and underrepresented minorities.

Rebecca: There’s lots of things for all of us to think about, even if we’re not necessarily in economics classrooms. Definitely good food for thought. Switching gears a little bit. Can you tell us a little bit about your work on the Committee on Economic Education?

Tisha: So I’m not officially on it at the moment, I’m the incoming chair. But I have served on it in the past. And I’ve been shadowing KimMarie McGoldrick, who is the current chair for the last year. So the committee is doing a lot of really great work. I think. When I first served on the committee, we basically just organized sessions for the American Economic Association meetings. And when I say “just” I don’t mean “just…” Compared to now, it’s just but we have, I think, seven sessions that we organize for the meetings every January. But while I was first on the committee, then chair Mike Watts, who was at Purdue, he said,”I have an idea. We should have a conference, so an all economic education conference,” and we started it while he was chair, and it’s CTREE, so the Conference on Teaching and Research in Economic Education. So the committee is in charge of that. It’s every end of May, beginning of June, depending how the calendar falls. So our next CTREE will be in Atlanta, not that far. It’s on the East Coast, maybe you can join us. And it will be May 29 through the 31st of 2024. We have many other things that we also are overseeing: an educate program, which is professional development for college faculty, and that includes two-year college faculty, in course design, different active learning pedagogies, and we talk about how to bring diversity and inclusion into the classroom. KimMarie actually started this this year, we have a newsletter called EconEdNews. And that features different pedagogies, it features the winner of the AEA Distinguished Economic Educator Award, which the committee also oversees, and some centers that we have for Econ Ed across the country… just a lot of information that’s in there, we have two issues a year. So we’ve had our first two, and our next one will come out in March of ‘24. So just a lot of great work, I think, that the committee is doing and I think really thanks to Mike who started us with our own conference. And then Sam Allgood succeeded Mike and added and then KimMarie is an overachiever. And she added a lot more in her six years as the chair.

John: And going back earlier, early surveys of economic instruction suggested that economists were using much more chalk and talk than many other disciplines were using. So these efforts have been fairly important in shifting people away from that. It’s a somewhat overdue change, but it’s nice to see that happening. And I would love to go to CTREE. But there’s two barriers that I’ve run into one is we run a series of workshops here, right around the same time, and the other is I teach at Duke in the summers. So I either am doing workshops here, or I am in North Carolina teaching econometrics now down there, so I haven’t been able to get there. I would love to go to a CTREE meeting.

Rebecca: I think I would have no clue what’s going on if I went. [LAUGHTER]

John: Oh I think you would.

Tisha: I think you would, I think you’d have a nice time. Everyone there is just so lovely and very engaged in teaching, and just really cares about the enterprise.

Rebecca: It’s nice to see education groups out of these professional organizations to really formulate communities of practice around teaching.

Tisha: Yes, I agree. I think economics, unfortunately, was late to the game. But we’re increasingly appreciating the importance and the importance not just of focusing on four-year colleges, but two-year colleges. So going back to the point of diversity, and the lack thereof in the discipline, the students in four-year colleges that I’ve worked with have predominantly still been very much what economists already looked like. And if we really want to improve the diversity in economics, I think we need to reach down to the high schools to the two-year colleges, and make economics more interesting and accessible.

Rebecca: We always wrap up by asking: “What’s next?”

Tisha: There’s lots of things. I guess, with regard to the committee, I’m hoping to just kind of not start anything new in my first term, and keep everything running as well as KimMarie has it now. But then for my own research, I am really interested now in looking at comparing pedagogies. So it’s been a lot of efficacy work looking at a particular pedagogy compared to sort of the standard chalk and talk. But now, how do they compare to each other? Is there a better pedagogy for students? And my first study, and I have the data, so I just have to start working on it will compare experiments to cooperative learning. And I told KimMarie, if experiments lose, then she won’t hear any more about this. I won’t write it up. I won’t talk about it. Because I don’t want to say that experiments lose, but no, I mean, seriously, that’s the next thing that’s first on the agenda. But also, I have a deep interest in diversity questions. KimMarie and I with Scott Simkins have a paper looking at HBCUs compared to PWIs and the study of economics there. And that was inspired by some work that showed that HBCUs contributed disproportionately to the STEM pipeline. And since economics is part of STEM, we thought, “Well, that should be true for economics as well.” So we did some work in that area. And we show that there are some positive contributions from HBCUs to the economic pipeline. But I think there’s a lot more to do there. So I would like to dig down in that. I’ve heard a lot of people at HBCUs talk about secret sauce, and I want to learn what that is. And not just at HBCUs but look at MSIs as well, more generally. And I do want to look at some questions at two-year colleges, so that’s for the next five to 10 years…

Rebecca: You’re going to be busy. [LAUGHTER]

John: For those who aren’t familiar with the terms PWI are predominantly white institutions and MSI are minority serving institutions.

Tisha: That’s correct.

John: Those terms are becoming more and more common, but just to make sure all of our listeners are familiar. Okay, well, thank you. It’s great talking to you, and it’s nice meeting you. And we will keep trying to get KimMarie McGoldrick on the podcast. I’ve been asking her for about four years. She’s been a little resistant, but we’ll see if we can get her here in the future. Well, thank you.

Tisha: Thank you so much. I hope you have a good afternoon.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

314. Handbook of Online Higher Ed

Since its start in the late 1990s, asynchronous online instruction has spread throughout the world and has been the subject of extensive experimentation and study. In this episode, Safary Wa-Mbaleka, Kelvin Thompson, and Leni Casimiro join us to discuss their new handbook that examines effective practices in online learning from a global perspective.

Safary is an Associate Professor of Leadership in Higher Education at Bethel University in St. Paul, Minnesota. He has authored and co-authored more than 40 scholarly journal articles and more than 20 books and book chapters. Kelvin is the Vice Provost for Online Strategy and Teaching Innovation at the University of Louisville. Kelvin developed the BlendKit Course open courseware as part of the Blended Learning Toolkit, and he co-hosts TOPcast: The Teaching Online Podcast. Leni is a Professor of Education, the Associate Dean of the AIIAS Graduate School and Chair of its Education Department and the Director of AIIAS Online, the virtual campus of the Adventist International Institute of Advanced Studies (AIIAS) in the Philippines. Kelvin, and Leni are frequent invited speakers on topics related to online instruction. They are the co-editors of The Sage Handbook of Online Higher Education.

Show Notes

Transcript

John: Since its start in the late 1990s, asynchronous online instruction has spread throughout the world and has been the subject of extensive experimentation and study. In this episode, we discuss a new handbook that examines effective practices in online learning from a global perspective.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Safary Wa-Mbaleka, Kelvin Thompson, and Leni Casimiro. Safary is an Associate Professor of Leadership in Higher Education at Bethel University in St. Paul, Minnesota. He has authored and co-authored more than 40 scholarly journal articles and more than 20 books and book chapters. Kelvin is the Vice Provost for Online Strategy and Teaching Innovation at the University of Louisville. Kelvin developed the BlendKit Course open courseware as part of the Blended Learning Toolkit, and he co-hosts TOPcast: The Teaching Online Podcast. Leni is a Professor of Education, the Associate Dean of the AIIAS Graduate School and Chair of its Education Department and the Director of AIIAS Online, the virtual campus of the Adventist International Institute of Advanced Studies (AIIAS) in the Philippines. Kelvin, and Leni are frequent invited speakers on topics related to online instruction. They are the co-editors of The Sage Handbook of Online Higher Education, which we’ll be talking about today. Welcome Safary and Leni and welcome back, Kelvin.

Safary: Thank you.

Leni: Thank you.

Kelvin: Good to be here.

Safary: A pleasure to be here.

Rebecca: Today’s teas are? Safary, are you drinking tea?

Safary: I’m having water this morning.

Rebecca: A key ingredient to tea it might add. [LAUGHTER]

Rebecca: How about you, Leni?

Leni: I used green tea, particularly this Japanese matcha. [LAUGHTER]

Rebecca: Nice. How about you, Kelvin?

Kelvin: I have deconstructed tea. That’s also called water.

Rebecca: [LAUGHTER] Popular globally.

John: And speaking of globally, Rebecca and I are both drinking Moon Bird tea, which is a gift from one of our listeners in France who sent this to us a few weeks ago. So again…

Safary: Wow.

John: …thank you, Myriam.

Rebecca: Yeah, it has a nice hint of pear and elderflower.

John: …which is also a green tea.

Rebecca: Yeah.

John: We’ve invited you here today to discuss the SAGE Handbook of Online Higher Education. Could you tell us a bit about the origin of this book project?

Safary: The origin of this project is actually something that has to do with me having worked with Kelvin several years ago at the University of Central Florida. And right after that, I decided to work in the Philippines and that’s where I met with Leni Casimiro and we worked together. And at both institutions, we were working with online education. And eventually I was transferred to work for two years in Kenya. During the COVID-19. I happened to be in Kenya, and I quickly saw the great need of people wanting to have online education. The resources went up in the place. The things were scattered all over the place. And immediately the idea came that we needed a project that captured the whole world because now this was a worldwide phenomenon, it was no longer something peculiar to Kenya or Philippines or U.S., the whole world was in need of a tool like this. And that’s how I reached out to Kelvin and to Leni.. Thankfully, they both agreed to be part of the project. And I think, from my perspective, that’s where it came from. I don’t know about them… how they think about this? [LAUGHTER]

Leni: Well, for me, it’s really a big project that we did, combining the different parts of the world. You see where Kelvin comes from, representing the West, I represent the opposite, the East. And although Safary comes from the East as well, but he can represent the African continent. And so this really makes the book a global project, really a blend of different perspectives. And so I can say that online learning is represented all over the world in this particular book. And this is indeed, a big surprise to all the readers and a big discovery for everyone.

John: Speaking of readers, what is the intended audience of this book?

Kelvin: Well, I mean, honestly, I would say anyone, anywhere, around the whole planet, who in any way touches online or digital education, should access this book. It’s great for libraries and institutions to acquire and be in their communities. It’s a big book. There’s stuff in there for everybody. So I think it’s a great resource.

Rebecca: Speaking of the size of the book, the handbook contains 50 chapters. Can you talk about how you selected those chapters?

Kelvin: I think the scope and the sequence and the layout of the chapters and the sections sources originally to Safary’s proposal with the publisher, but it was intended to be rather comprehensive with sections like fundamentals and student support and administration and instructional design, instructional delivery, regional specifics, particular regions around the world, and how online education might differ a little bit in, say, the African context versus the European context. But over time, as we were recruiting authors, and as the writing process started, you get a little bit of evolution, the sections might morph a little bit, the distinctives of a given chapter might adjust based on interest and specializations of the authors. So that’s a little bit of the insight into the evolution. But I credit Safary for the vision, which I would say, is probably about 80 plus percent of what he originally had envisioned in the layout. That’s my guess. Safary, would you agree with that?

Safary: Yeah, the thing is that, when you work on a huge book like this, especially a handbook for Sage, they want to have the complete plan when you submit your proposal. Before I can get my co-editors to agree with me, they need to have kind of ideas, okay, this is what I have in mind. So usually, when I work on a handbook like this, I come up with a rough draft. And Kelvin and Leni were very good in catching certain things that I wouldn’t have caught because of their expertise, their experience, and their regions that they represent. And so in the end, what we have here is a product of the Table of Contents was really the product of these three brains that are speaking today.

Leni: I really liked the way Safary has chosen the chapters of this book. Well, we can say that he really originated the choice of these chapters. As you can see, from the perspective of a reader, when you look at the content, you can look in the sequencing, and you will find that you are actually looking into the step-by-step development, or the step-by-step process of engaging in online education. I will say it’s almost like a manual, almost every step that you will go through in undertaking online education in your institution is covered in this book. That’s why it’s really a very important book for every school to have.

John: We had some challenges coming up with a brief intro for each of you, because each of you has done so much with online education in many different roles in many different places. But you also have an editorial board for this book, which is a little bit different than many other books that we’ve seen in terms of handbooks. What was the role of the editorial board in putting this handbook together?

Safary: Yes, we had an editorial board. When you have a project of this magnitude, it is really important to have experts from different parts, especially at the global perspective of experts, and of course, experts on the different topics that are represented in the handbook. As much as we have experience with online education, we cannot assume to know it all… areas where we definitely need help. And so we selected very well known, very well recognized experts from different parts of the world. As far as online education is concerned, all the names that are there are people who are very well respected in the field of online education within their respective countries. The role they played was, for them to be our experts in checking the accuracy and the quality and the completeness of the chapters that were submitted to us. So basically, each chapter went to two to three reviewers and the editorial board members were the primary reviewers to help us really catch everything… and the work they did, I know that some chapters had more feedback than others, but I can say that contribution they gave through their feedback was very substantive in improving this handbook. I don’t know, Leni, how you found that when you’re working with the editorial members who are assigned to you?

Leni: Yeah, actually the editors we chose, I can say they are truly excellent and helpful. During the early parts of the writing of the chapters we lead editors are having like a tug of war with the chapter authors. They tend to bargain their thoughts with us, but when the editorial board came into the picture, it gave a more balanced outlook into writing the chapters. And so we really appreciate their services. The other thing is that this editorial board members are experts in the area and so we can truly depend on them. Their feedback were truly much valued and contributed much to the excellence of the contents of this book.

Rebecca: So the handbook is divided into seven sections. Can you provide a brief overview of each of those sections to give us the lay of the land?

Leni: Oh Yeah, seven sections, it’s nice to give an overview for people to know what the book contains. First section, of course, is the fundamentals of online education. It contains the introduction to the topic of the book, online learning, and some variations in online delivery, like blended, MOOC and ERT, emergency remote teaching, we just really call it ERT, and that became popular during the pandemic. The second section, online education around the world. This section is the most colorful part of the book, at least for me. Because it tours us around the world and gives us a view of how online education grew in varied contexts like US, Canada, Europe, Asia, Latin America, Africa, Australia, and the Middle East. The third section, Online Instructional Design, this section now brings us to the T-cell of online learning, the design of online instruction with focus on how learning happens online. This is now the more serious part of the book. While we came from the most colorful, we now go to the serious part of the book. And then the fourth one, Online Instructional Delivery, this section focuses on the hammer and nail of online learning, the actual online teaching, and this is the most exciting part. Because this is now the delivery, the previous one was the most serious part, this one is the most exciting part. And then perhaps, Kelvin, can you say about the fifth section [LAUGHTER] Instructional Technology for Online Education?

Kelvin: Here’s what I would say about that, if you’ve got the most serious, you talked about the most exciting that you talked about, maybe the fifth instructional technology for online education is the most invisible, maybe that’s what it is. Nobody thinks about plumbing until it doesn’t work. [LAUGHTER]

Leni: Thank you. So that’s technology, I would say this section is essential, because you cannot teach without knowing how to use technology [LAUGHTER]. And the sixth section, Online Education Administration and Management, I would say this is the driver’s seat of online bandwagon [LAUGHTER]. Online education can never prosper without the support of the school administration. So, leading school reforms, like entering the field of online education requires certain strategies to be certain of success. Therefore, I would say this section will indeed equip the readers with those skills, perhaps Safary, can tell us what section seven is?

Safary: I would say the last section is the Customer Service, given that the students are the customers. So the customer service, how to make sure we deliver the best customer service to the online students. And so it discusses all those different aspects of how to really prepare, plan effective service to the students, because many times when people are migrating from face-to-face to online or integrating online education, they forget that online students actually need serious support. And this support definitely needs to be defined. And people who are dealing with the students need to be trained. And so the last chapter actually deals exactly with that.

Leni: For me, because I was looking at the table of contents, and I was smiling in my mind, wow, this is really neatly done. And so this works came to my mind, and I said, Oh, the seventh section, this section focuses on the heart of every online classroom, the students. And so because the students are the reason why we offer online learning, thus we ought to know how we should support them.

Rebecca: One of the things that I love about working on collaborative projects that are really big, and then you have these opportunities to reflect together, is how you summarize what you did. It’s probably really different than while you were right in the middle of it. And it’s fun watching the facial expressions and things as you guys are describing the different sections.

John: With 50 chapters, there’s a great deal of breadth and depth on these topics. In section one, though, you address two topics which are not always considered as part of traditional, at least, online education, which is the use of MOOCs and ERT, emergency remote teaching. But these have played fairly important roles. Could you talk a little bit about the role of MOOCs and Emergency Remote Teaching in the larger environment of online higher education?

Leni: As I see it, MOOC and ERT are connected to the overall theme of the book, because technically they are both delivered online. Online learning can be synchronous or asynchronous. And it’s mostly taken asynchronously while ERT is done synchronously, because it is generally a replication of the face-to-face classroom through the web. However, there are certain arguments in the field as to whether can we classify these two under online learning, because they are believed to not use the principles of effective online teaching. And they say, is their instructional design in ERT? There are more questions to raise to the point that some people believe they should not be called online learning. But for me, we have a common denominator, course delivery through the web. Maybe we can hear from my co-editors here, Kelvin and Safary, what they think about it?

Kelvin: I was thinking, John, when you asked that question, I think the combination of Emergency Remote Teaching and Massive Open Online Courses, it’s part of the popular conception of what online education is, it’s sort of like what a layperson might think, is, it’s just one big thing. So if you didn’t address Emergency Remote Teaching, Massive Open Online Courses, maybe even Blended or Hybrid learning, those mutations, it might not provide quite the same way in for the broadest possible audience. But then, once we’ve ushered you into the house, through the front door, I hope we do a good job of taking you on a more detailed guided tour through the nuances and everything that online education can be, without just being stuck at that surface level.

Safary: If I may add something to the ERT. Personally, the reason why I wanted to see this chapter there was that outside of the United States and maybe Canada, and a little bit of Latin America, when ERT came, Emergency Remote Teaching came, many people call it online education. And as we know, online education, the way we know it traditionally, is much more than translating your face-to-face class to a Zoom class or Google meet class. And let’s face it, that the word there is emergency. This was an emergency modality, which obviously emergency is never the best option, it means better than the chaos that you’re going through. And so many people who didn’t know online education, they came to believe that Emergency Remote Teaching means online education. And many people who were against online education to start with, it was like, “Okay, we have already said that this thing is really bad because it was an emergency.” So it was very important to distinguish what Emergency Remote Teaching is. And in the future, if somebody wants to use that for another calamity that happens, then they know what steps to take, but it does not replace what is known, what we define as quality online education.

Rebecca: One of the parts of your book, The second section is about online education around the world. And getting that tour around the world is not something we typically get the opportunity to have. So can you talk a little bit about what some of the global differences in how online higher education is structured and practiced across continents and regions?

Safary: This section came up as we were trying to make the book global. We really wanted to hear the voices of the people from around the world and not just the United States… the United States being the lead on online education, no question about that. We wanted to know where things are in different regions that were represented. We had to even go online to try to track people down from different countries. It was not easy finding people from certain regions where we didn’t have a network. So as a result, we’re able to bring on board chapters from different parts of the world. We had a chapter from the United States, we had a chapter from Europe, from Canada, from Asia, from Latin America, from Africa, from Australia, and from Middle East. So we were able to see what was happening in each one of them. And these chapters we had, they were kind of similar in a way where we wanted to know what is happening, what are the challenges, what are the achievements that people have in those regions, so that people from those regions who decided to do more work on online education, they have a place where they can learn of what is happening in the whole region from this book. They can have this as a reference to understand what was happening in their region. It is true that when you have one chapter, for example, I co-authored a chapter on Africa, because I was still in Africa at that time. It’s a chapter that’s covering 52 countries, you cannot really cover 52 countries, we just had to have illustration from some of African countries, because there’s no way we have data on all the 52 countries, but at least, there were some common themes that were coming up from a different African countries if I can speak from that specific region.

Leni: I can speak from the perspective of an Asian because I come from Asia. And I would say, we cannot deny that online education started in the West. But because we live in a connected world, it spread easily. Basically, I can see a lot of similarities around the world. The only differences I noticed, because your question says what are some of the global differences in how online education is structured and practiced? Now, I would say the only differences I noticed are the approaches to online learning, depending on the level of their maturity, in using this modality, and the resonance of the context they serve. Institutions that have been engaged in online learning for a long time definitely deal with issues that are different from those of newcomers, the needs of the context they serve also differ, so the strategies utilized also differ. One thing I would highlight, though, is that you can clearly see the creativity and continuity of people in different parts of the world in running online education. And we still can learn from each other. That’s why I said a while ago, the section on the global online education is really colorful.

Rebecca: One of the things that I think is really interesting about that section, is that it can also give us insight as instructors that teach a global audience about what the contexts are that students might be coming from. And that’s something that we often don’t have a little bit of insight into.

Safary: I think that is a very good point. Now that we have online education, people are teaching in many different countries. I remember just a couple of weeks ago, I was approached by one of my former students who wanted me to teach a class in the Caribbean. If things worked out for me, for that class, I would have just glanced at that chapter that covers a little bit of the Caribbean and see what I need to watch out for. So that is definitely a good point for the section on the different regions. In this handbook.

John: When online education first started, there wasn’t really that much known about what would work effectively. And as online education evolved, we saw the role of instructional design become an important part of the practice of online education. And section three deals with online instructional design. And that’s helped facilitate and inform online education, along with a lot of research that’s been done since the early stages. How have instructional design practices evolved since the early stages of online education in the latter part of last century?

Kelvin: That’s a good question. And I guess I’ve been in this field watching this first hand and touching it for about 25 years now. So I sometimes say not exactly the first floor of the building, but just one step above. And what I would say is that when I started in the late 90s, what we saw a lot was adaptation of traditional instructional systems design models and practices, that is constructs that were used quite often in corporate education. See if this takes you back to the past: CD ROM development, military learners. Those kinds of methods, practices, and models were adapted to this online context. And some of that’s constrained, like you’re making a system, like it’s a bounded system that was, quite often the context, like a CD ROM. And now you’re talking about the internet, a network open system. And I remember some of those early days, like, “Okay, what can we learn from these models? How can we adapt those?” Over time though, we learned that this is a unique context, which then began to have its own models and practices and processes and research and iterations and development. And I think of even things like much newer developments, like alongside of constructs like inclusive pedagogy, we see practices and thrusts, like inclusive design, as being a very specialized subset. So we’ve got a very robust research and professional practice literature that has grown up and these, arguably, two and a half decades of online education experience to draw upon. And I guess I’ll just say this, about that. Throughout my time in this field, what I’ve seen is that online tends to make the formerly invisible, visible; formerly implicit, explicit. And I think that evolution of instructional design and development field, it has learned from that. Online education has drawn us along in what does it mean to bring learners in from really anywhere and bring them together in a learning community, and how do we excel in that. That’s been a really rich progression over these last two and a half decades.

Safary: If I may add to that, the reason why we had this section was that many people who are new to online education, they think that online education is about uploading all the files that you have been using face to face, and then let the students read that, and that’s online education. It leads to a lot of frustration from the students because there was no instructional design for online learning. And so we needed to have a section that would guide people into that. And also for instructional designers in college and universities where they already have instructional designers. Some of them have not gotten a degree in instructional design. So they have limited knowledge. They just happen to know a little bit more than everybody else, but they don’t really have a solid foundation. And so that section helps to kind of guide people in the proper instructional design for online learning.

Rebecca: So sections four and five focus on online instructional delivery and instructional technology. These are topics that we love to talk about and have episodes of this podcast on. But given the time constraints, we probably can’t dig in fully here. But can you help us identify some of the most important changes that have occurred in how well designed online courses are taught?

Leni: That’s a nice question. Kelvin also said a while ago, he was mentioning about the early years of online instructional design, I would say, perhaps 1998 to 2000, those are the early years I’ve been involved, still in the planning stages of online delivery. Most of the online courses we developed were primarily text based, and are delivered asynchronously. That was after the military, Kelvin used, online learning, it was already in the university. Why text based? Because even our students, in the context we are serving also did not have the capability or the capacity to access videos or higher level technology tools. That’s why we designed the way they can access us. And so, yes, it was primarily text based and asynchronous. However, through the years, I would say two forces caused the major changes in the way we design online courses, first, technological developments, particularly in instructional technology. And second, changes in the needs or nature of our stakeholders, the students. Well, technological developments without a doubt have increased the repertoire of instructional media that we can use in designing truly engaging online courses. But as I’ve said earlier, technology is not the heart of online learning… it’s our students. And we saw how the nature of our online students change over time as well. While many of them were happy with plain text based asynchronous online courses during the early days, now they want more real-time meetings. And the flexibility they want is indeed tremendous, I tell you. We notice that there is a greater demand now for more flexible and personalized learning approaches. And these topics are dealt with in this handbook. I know Kelvin has written on this. And some other chapters also addressed this flexible learning, personalized learning approaches. These are now the needs and demands of the new generation of online students.

John: This is bringing me back to a time when I started back in 1997 teaching online when many of the students had 300 baud… [LAUGHTER] …or 1200 baud modems, and you couldn’t do much more than text. And I remember putting in some flash-based videos, and many students couldn’t access those because they didn’t have the download speed, especially students in more rural areas. So there was a lot of resistance to online education when it was first introduced, which is one of the reasons why I think instructional design practices became a part of early online education to help ensure the quality of that. And we do have, in most institutions a fairly elaborate process of instructional design assistance and instructional design review for online courses, which is something that’s never really happened in the same way for most face-to-face courses. Might it be time to start applying some of the techniques and practices of design that’s being used for online course delivery to in-person course delivery?

Safary: I remember about 15 years ago, I was training faculty on online teaching in the Caribbean. And I remember many of them, at the end of the training, saying, “I have improved my face-to-face teaching because of the training that I have been going through for online teaching.” So I definitely believe that if people get the proper training in online teaching, they can use that knowledge to improve face-to-face teaching. Because let’s face it, many people are teaching not because they have a degree in education, but because they have a degree in whatever field they come from, they have never learned how to teach. And so when they go through the training for online teaching, they discover a lot of principles that they should have even been using face to face. So I definitely agree with you on that one.

Kelvin: Yeah, it’s true. I say it all the time online makes the formerly implicit, explicit; formerly invisible, visible. And I think that’s why online has been a vehicle for applying thoughtful design and teaching practices and the improvement thereof. Once you sort of concretize the elements that make up an online education experience, then you can see well, how are they arrayed? Are they lined up properly? Does this cause lead to the desired effect, and you can work on improvement, no offense to anyone in this, but when we just are dealing in the ephemeral, we will walk into a space, four walls and a door, and we say words into the air, it’s much harder to see how those parts fit together or don’t. And it’s harder to be reflective. So, I think that’s the reason that online education has brought more emphasis to potential improvements, continuous improvements, and so I welcome it as a vehicle for a more thoughtful process in general. I love this elegant turn of phrase Caroline Boswell says she frames teaching as a student success intervention. Or as I put it, I’m one of those odd people who sees a connection between teaching and learning. And not everybody does.

Rebecca: You’re kind of queuing up our next question perfectly Kelvin. The final section of your book is really about student support. And our students are often distributed when we’re teaching online. So what are some of the biggest challenges in terms of supporting students that are in these online programs or online courses?

Kelvin: Yeah, I would welcome Leni’s and Safary’s viewpoint on this as well. But to me, I’ll keep it simple and say that the biggest challenge is the diversity of student profiles. The different backgrounds, the multifaceted demographics, and resource or not resource, or technological connection or not technological connection, that diversity makes it awfully hard to assure kind of an equitable experience for everyone. So that’s the gap that emerges, that student support is trying to offer… not to mention the diversity of approaches to design and development in the actual experience. But I’m curious what Leni and Safary would say to that.

Leni: I would go for the opposite, on the side of the teachers, I would say the greatest challenge in student support is personalizing your support. It’s related to your diversity. Almost every online student has her unique needs and contexts. So considering different personalities and backgrounds as well, you may be able to personalize your support. But in the name of efficiency, you’ll find yourself dehumanizing the process. What do I mean by this? Well, machines can never replace human touch. And human touch is what every online student needs.

Safary: If I may speak a little bit from experience I had in Kenya during the COVID-19, we migrated our classes to the online delivery. And I quickly realized that… and this was something that was going on in all of Africa, I know this because I was involved in different international association for online education all over the continent…. and so we were meeting and discussing some of these issues. The major challenges that were going on at that time, I don’t know about today, were dealing with infrastructure, because most universities that didn’t have online education platforms, or online education structural systems, so the technology was not in place. Many students there were not access because the internet was extremely slow, some were using loads of data to access the materials and they would run out. Some had issues with electricity. These are things we take for granted in the West. These are the not issues that we will discuss even in textbooks of online education, but they are real issues that cannot be ignored. And so that was a major challenge in supporting online students, because the infrastructure was not in place. And I think the issue is still the same. But more and more work is being done. I remember, for example, in Kenya, what the government did, they gave the free data access to all the faculty in the whole country, as long as it was used only for instructional purposes [LAUGHTER]. If you want to use it for something else, it wouldn’t work. I mean, that was quite creative, to try to help people to help education move forward, because everything was just stuck because of COVID-19.

John: Over the past year, we’ve seen a fairly explosive growth and use of generative AI large language models, including chat GPT, Claude, and a few others that have come out very recently. And that opens up a lot of interesting opportunities, but also some challenges for online education, particularly concerning the assessment of asynchronous learning. How do you see online education adapting in response to the widespread availability of tools like this, which will only become more powerful over the next few years?

Kelvin: It’s sort of the very definition and epitome of disruptive innovation or disruptive technology. And just to be clear about this, I don’t think it’s limited or focused on asynchronous online education, I think it’s everything. For me, it’s really an opportunity to address learning and assessment of learning much more meaningfully, and I’ll use one of Leni’s words, more personalized and relational. I think one of the things we’re seeing with the injection of these various forms of artificial intelligence into the learning setting is the value proposition of the human. And I think it was Cathy Davidson, years ago, from HASTAC said something like, “If we faculty can be replaced by a computer we should be.” That is, if all you’re offering is something that is easily rendered more efficient and scalable by a machine then, well, what are you doing it for? I think that the opportunity to really gauge learning, which is a very personal and a meaningful thing, we act like it’s something that’s kind of homogenized and industrialized, but learning… I don’t know what learning is, frankly, I can’t crack open a human and see what all is happening with the connections and making of meaning in all the background experiences. All I can do is get insight, but in dialogue, in the creation of artifacts I get a glimpse. If we’re product oriented, to the exclusion of the process, and to the exclusion of the human context, well, that can be certainly disrupted, maybe stolen by artificial intelligence in machines. But if we keep the emphasis on humans, on “Well, John, tell me about this…” that’s more meaningful. I learned a practice a long time ago from a faculty member that I studied under, where she adopted a practice of a learning summary. And in any course, again, that’s just one artifact, but it gives a glimpse into the articulation of what learning is really about. So I think we need to push the envelope in “What does authentic assessment mean? What does meaningful learning look like?” Now, that’s hard to do at scale. Are you going to have a personal oral defense with every student for every assignment, probably not. But if we see artifacts, and products, as breadcrumb trails leading to a destination of a more substantive dialogical process, well, then maybe that’s something. So I don’t think we know yet how this is going to play out. And I think your listeners are gonna find cold comfort from me in getting to an easy solution. But I think the future of responding to generative AI is to lean more into the human and the relational than less.

Rebecca: So we always end by asking what’s next?

Safary: Well, as far as this project is concerned, what is next really, we want to continue building a community of online higher education scholars, practitioners, so that this momentum that has been created by this book can continue, because this is one of the few maybe rare books that really have so much global contribution to online education. Many of the books that are written, they’re usually kind of regional to a specific region of the world. And so this is the first time we have a network of, I think, around 100 people who contributed to this, coming from many different countries. And I feel this has created synergy on the discussion of online education in a way that we should not let that go. So one of the things that we have been talking about is the possibility of holding a summit on online higher education in the next few months, once everybody has gotten a chance to hold a copy of this book, and to bring different experts together from different parts of the world, and try to address online education from different parts of the world, while addressing common issues such as assessment, which is one of the major controversial issues anywhere have been, everybody talks about the challenges of online assessments. So that’s things like this, and probably this artificial intelligence, which is a new thing, we may want to go deeper into that… we’re not able to dig too deep with that, although we addressed it in the book. But we didn’t go too deeply because it was still kind of new ChatGPT was just coming out when we were finishing the handbook. And so that is one of the things that we are looking into, there is another handbook in the making with SAGE that will focus specifically on instructional design in higher education. So that would be like an extension of this project. So we want to continue building on this work, because we consider it’s very important.

Leni: I’m really optimistic about the next steps on this because it’s like a seminal book that really got there’s a global perspective, as Safary says it’s not the same as the other online learning books. So we can also see a lot of developments coming up. And so I will say, this book is just step one, the next steps will really be coming up definitely, because the field is always growing. We have seen its growth, and it will still grow. And so there’s more to follow, I believe.

Rebecca: Well, thank you all for joining us. I know that our listeners will really enjoy the handbook and all that it has to offer.

John: Well, thank you, and it’s great talking to all of you and we’re looking forward to reading the book.

Safary: Thank you so much for the opportunity. Really appreciate that and wish everybody a wonderful reading experience.

Kelvin: Thanks for having us, Rebecca and John.

Leni: Thank you very much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

Ganesh: Editing assistance by Ganesh.

[MUSIC]

303. Higher Ed Then and Now

Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, Todd Zakrajsek joins us to reflect back on the changes that have occurred in higher ed during our careers.

Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson.

Show Notes

  • Zakrajsek, T. and Nilson, L. B. (2023). Teaching at its best: A research-based resource for college instructors. 5th edition. Jossey-Bass.
  • Zakrajsek, T. D. (2022). The new science of learning: how to learn in harmony with your brain. Routledge.
  • Harrington, C., Bowen, J. A., & Zakrajsek, T. D. (2017). Dynamic lecturing: Research-based strategies to enhance lecture effectiveness. Routledge.
  • EdPuzzle
  • PlayPosit
  • ChatGPT
  • Wayback Machine

Transcript

John: Teaching practices have gradually evolved as we’ve learned more about how humans learn. From one year to the next, these changes may appear small, but the cumulative effect is profound. In this episode, we reflect back on the changes that have occurred in higher ed during our careers.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Todd Zakrajsek, and I am with Todd here in Durham, North Carolina. Todd is an Associate Research Professor and Associate Director of a Faculty Development Fellowship at the University of North Carolina at Chapel Hill. He is also the director of four Lilly conferences on evidence-based teaching and learning. Todd is the author of many superb books, and has published four books in the past four years. His most recent book is a fifth edition of Teaching at it’s Best, a book he co-authored with Linda Nilson. Welcome back, Todd.

Todd: Well, thank you, John. Well, this is exciting. And Rebecca may be a long ways away, but I have never been arm’s length from a person who interviewed me for a podcast before.

Rebecca: Isn’t that cool?

John: And we’ve really done that before either at a conference or at Oswego,

Todd: I feel very special.

Rebecca: Well, we can celebrate with our teas. So, today’s teas are:… [LAUGHTER]

Todd: I’m drinking a peach mango that I got from some teas that John brought, which are fantastic.

Rebecca: John, how about you?

John: I am drinking a Tea Forte black currant tea, which I brought from Oswego, in a new mug that was given to me by Claire McNally, when she visited this area last week.

Todd: Love Claire, she’s fantastic.

John: And it has kangaroos on it.

Todd: Yeah.

Rebecca: And I can’t see it. Let me see it, John. Oh, that’s a cool mug.

Todd: It’s a good mug. I got a mug from her university. But I didn’t realize I should have brought it. So I feel bad about that. But it is a podcast. So I didn’t think about what it would look like.

John: That’s true, we generally don’t do a lot of visuals on here.

Rebecca: And I have a blue sapphire tea in my Tea Rex mug.

Todd: Well, that’s a nice mug,

John: We’ve invited you back to talk a little bit about how some of the changes you’ve observed in college teaching across your career have impacted how you teach today. When did your work in higher ed begin?

Todd: Actually, it started when I was a graduate student. So back in 1987. So there’s no reason to try to figure out how old I am. Now I’ve basically specifically dated myself here. I started teaching, I got to teach an introduction to statistics course. And I had so much fun that I taught again the following year. And by the time I left my graduate program, I had taught more courses in that program than any other graduate student had ever taught in the psychology department there. I really loved teaching right from the beginning, when from the beginning, very concerned about student learning, and just getting rolling.

Rebecca: What was it about the teaching, Todd, that really got you hooked?

Todd: Just watching the studentsis. it’s the same thing as it is today, when you have an individual who’s struggling with something, and suddenly they get it and you realize that they may eventually get it on their own, but you realize how much you’ve helped them to move that along very quickly. And facilitating the learning process, I just really love that. That doesn’t mean I was fantastic at it. But I really did love it.

Rebecca: Sometimes the things we love the most are things that we’re not great at to start with.

Todd: That’s true.

John: My experience was similar, actually, I started in 1980, with a course where I had a fellowship, so I didn’t have to teach. But there was a sudden shortage in the department. And they asked me to fill in. And I was planning to go on into research. But it was just so much fun teaching that I’ve never stopped.

Rebecca: I taught as a graduate student too, and taught the whole time I was there. But I started a little bit later in 2003.

Todd: Alright, so that was a couple of years later.

Rebecca: Just a couple.

Todd: Yeah, I had kind of a funny start, I will mention that when I first started that after the first semester of teaching, my students got almost all As and Bs. And the department chair called me in and he said, “I’m not going to have you teach any more courses.” And I said, “why not?” And he says, “Well, you give grades away like candy, we have to have better standards than that.” And I said, “Well, how are you basing that?” And he says, “Well, you know, we looked at the grade point averages.” And I said, “Well, how about if I bring in my final exam, and just walk through it, and then you can tell me how it could change to be more rigorous.” And so it was great. I showed it to him at the beginning. And like the bottom of the first page, the students had to calculate a statistical value, then I had them explain how they came about that number. But if they had used a different test how might it been inappropriately found and what the interpretation might have been, based on the fact that they had done it wrong with a different test. I thought it was important for them to understand how these things can change. The Chair said, “I can’t believe you have your students in the first class actually talk about various tests like that.” And I said, “Yeah, I did. Then we turned the page he says “You did nonparametric tests?” I said, “Well, yeah, we did parametric tests, but then I thought they should know the equivalent.” And he said, “We never do that.” And then he turned the last page and he said “You had them do a two-way ANOVA? You’re only supposed to go through one-way ANOVA.” I said, “Yeah, but we’d finished everything and we still had a week left. And I figured I might as well introduce the next concept to them. And so I showed them how to do a two-way ANOVA and they ended up with all As and Bs. So if you could help me in how to push their grades down and give them lower grades, I’m perfectly happy to do that.” And he then set me up with two courses the next semester, but it’s that reliance on the teaching evaluations is always funny.

Rebecca: Todd, it’s just funny, as we’ve gotten to know you through the podcast [LAUGHTER] it sounds so perfect that that was your first experience. [LAUGHTER]

Todd: Yeah, I’ve lived my entire career on the edge. [LAUGHTER]

John: And those sorts of arguments are still occurring in a lot of classes today about rigor and the need to keep grades lower.

Todd: Yeah.

John: They’re less severe than they were a few years ago.

Todd: Yes, but also looking at how well a person’s teaching based on student evaluations. I mean, we should be looking at authentic assessment. Some things have changed through the years, some things have not changed through the years.

Rebecca: Well, technology is one of those things that has changed.

Todd: Woosh, yeah.

Rebecca: Can you talk a little bit about what tech was like in the classroom when you first started and how it’s evolved a bit?

Todd: Yeah, I know you have some listeners who have been teaching for a very long time. So those of you have been teaching for like 30 to 40 years, just stop and think back about what it was like when we first started. For those of you who have been teaching like Rebecca since 2003, let’s just mention that technology back then was mostly pens and chalk and chalkboards. So back then, of course, there’s technology, there’s always technology, but we were using overhead projectors. This was long before the internet came along to really be used in the classes. LCD projectors were not out yet. Canvas, Blackboard, Sakai, all those learning management systems were not around. We didn’t have any of the ways to email individuals, you couldn’t email your students back then. And there was no ChatGPT to write your papers for you.

Rebecca: But there were calculators that could do all the work for you.

Todd: Yes, but this is the cool part. Back when I started teaching statistics, I’m glad you mentioned the calculators, huge debate back then was whether or not the students should calculate the statistical values by hand using the calculator, because computers had just come onto the scene and we could punch the data into a computer and have a computer run an ANOVA for you. Should you calculate it by hand? Should you run it to the computer? And there was a huge camp that said you should do it by hand or you will never understand a statistical value. And I said, “You know, we’ve got the technology there. Why don’t we have the students use the computer to do the mundane stuff, and we’ll have more time to talk about the theoretical and the important implications.” But even back then we were having the discussions about whether to use the technology at hand or not. Oh, and by the way, we are also hanging grades on doors. So we would figure out the grades, we’d tack it to the door, and then the students who want to know what their grades were for the class would swing by and look at the door.

John: And they were sorted alphabetically, to make it easier for people to find where they were in the grade list.

Todd: Yeah, it was great. We listed them according to their social security number, [LAUGHTER] which was a little different back then. And yeah, we actually did that back then. But as John pointed out, they were listed by number so nobody knew whose number went with whom, except, surprisingly, they were alphabetical on the door. So not only could you figure out Armstrong’s exam score, you’d get Armstrong’s social security number as well. Yeah, times have changed.

John: And it was also back in the day of dittos and mineos as well, which was the only way of disseminating information on paper.

Todd: This is so much fun. We’ll get to some real meat of this thing. But that walk down memory lane has some fun stuff too. The dittos…

Rebecca: I remember dittos, just for the record, okay.

Todd: Yes. So you probably remember, if you dittoed just before class, and you handed it out in class, the students would all pull the ditto up to their face, so they could smell the ditto fluid. And they got that smell. I was running dittos one time in the graduate student office, and I noticed when I looked down because it ran out of fluid, and I had to put some more fluid in, and I looked down and I noticed that the floor was kind of eaten away by this ditto fluid. And then… this is the best part… About a month later, I was digging for something in the closet and I found extra tiles and I thought they should put these tiles down to replace the ones that are all eaten and on the side of the box it said these tiles were long lasting and durable, reinforced with asbestos. So that ditto fluid was eating through asbestos tiles. That’s some strong stuff.

John: …to make it a little bit more friable so that it would disseminate in the air nicely.

Todd: Well, there had to be something to help the faculty members who were running all their own dittos to not mind doing it, and one way of doing this is to have them use ditto fluid, because I’ll tell you, you may not have liked it when you started, but by the end, it was all right. [LAUGHTER]

Rebecca: It’s funny that we’re taking this walk down memory lane, because on our campus, I was in our historic lecture classroom today in Sheldon Hall.

John: What are some of the other changes that have occurred and how have they influenced how we teach?

Todd: Yeah, so it’s interesting, I did the walk down memory lane and we were chatting about this stuff. It’s all fun, but thinking about how the changes have taken place. I think that’s really important. So there have been massive changes. I think that we tend to forget, it’s so easy to communicate with students now. Heck, people are texting now so that you can text back and forth with students. But think about how that has transcended or gone through time. There was a time when I would have to call and leave a message for a student on an answering machine, and then they would call back and we would try to find a time that we could talk on the phone. If we wanted to have a conversation. I could either leave a note for the student or I could call and leave a message that says least come see me after class. So even having a conversation with a student was difficult, then it became easier with email because you could start emailing back and forth. And now we have Zoom. And the equity in the way that this has changed, just think about the difference of this, if I’m leaving a message for a student, they may not even have an answering machine, if they’re living off campus with limited means back then. So even getting in touch with a student would be challenging. Now I can have a Zoom conversation with a student who doesn’t have to hire a babysitter, who doesn’t have to find reliable transportation, who doesn’t have to drive across town and burn gas, and to do all of those things that it would take to have a 15-minute conversation that in the past would have been really hard, and even four or five years ago would have been challenging. The grades, why in the world would a person have to leave… and I was teaching in very northern Michigan, there were days that the wind chill was 75 degrees below zero… and students would leave their dorm rooms and walk across campus to see a grade on the door. It’s actually physically dangerous. And now we have learning management systems, we could post things for students. Interlibrary loan used to take weeks to get a document that you can now go on and get. People can lament all of these technological changes at times, but we’re actually creating more and more equity within the higher education system as we make certain things easier. Not saying that we’re anywhere near an equitable system yet, but we’re moving in a really good direction. And a lot of those changes are helping us to get there.

Rebecca: I’m thinking about all the times when I get to go to the door or meet after class, it really assumes that students are a certain kind of student, they’re full time, they have time. And our students now are working [LAUGHTER], and where they’re juggling a lot of different schedules and things.

Todd: Yeah, and I mean, we want to be careful too. And I agree with you 100%. But they were juggling back then too. But some of the things we were doing, for instance, I taught a night class. Now I would probably suggest if I was going to teach a class from 7 to 10pm that I would teach it through zoom, because there’s a lot of reasons that it’s good to do. But I had students that I noticed in class, would very quickly at the end of class would start talking to other students and I couldn’t figure out what it was doing because a lot of buzzing and stuff. And what I found was that there were certain students who were uncomfortable, and we were in a very safe campus, but they were uncomfortable walking to their car at 10 o’clock at night. So I started saying to the students, “Hey, I’m gonna park a car… and when we showed up, there were quite a few cars there… but I’ll be under the second light, I drive a little red Chevette, not a Corvette, a Chevette, but I’ll have my car there. If you want to park near me, we can walk out together.” And there were students that were not paying attention to almost any of the class because they were fearful of how they were going to get to their car safely. When you think about Zoom and stuff, it’s even safety factors, I would never have a review session now like I used to at 8 to 9 pm the night before the exam because I’m exposing people to potentially dangerous situations. Now we’d have zoom sessions. But I could tell you 40 years ago, there was no even concept of what zoom would be and how it would work. Even Star Trek didn’t have stuff like that.

John: And there was also, besides the inequity associated with people who were working, many campuses had a lot of commuting students who could not easily get back to campus for office hours. Or if they were just taking classes on Monday, Wednesday, Friday and your office hours was on a Tuesday, they’d have to come in that extra day, arranging childcare, or their work to be able to fit that into the schedule.

Todd: Yeah, it really did start to change that system. So we got a little bit more equity, and like you were saying too, the commuting students, the part-time students, the students taking distance courses. When I first started teaching, I was writing… oh my word, remember the correspondence courses? …and you mail away and get a packet of material, you take a test at a local library and, and they talk about distance education being not as good as on campus, but at least better than nothing. And now we’re finally getting to a system where we can stop assuming that those folks who are coming in for part-time courses and stuff are just getting something better than nothing. They’re actually getting something similar to full college courses, which some of those online courses are actually as good or better than college courses that are on campus. But all that’s changing with the technology. It’s crazy.

John: And there’s a lot of research that supports that in terms of the relative learning gains with online and face-to-face, as well as hybrid courses, which seemed to outperform others in a few meta studies that have been done. But those were options that just weren’t available back then. And the early online courses were often designed to be replicas of face-to-face classes, and they probably didn’t work quite as well. But we’ve learned since that, which brings us to the issue of research. During the time that you’ve been teaching, there’s been a lot of research on teaching and learning. While some of it was taking place, it wasn’t very widely disseminated to faculty.

Todd: Yeah, that is true, too. It’s so much easier to get technology out. It’s easier to gather data, it’s easier to write it up. It’s easier to edit it so all of those types of things that are happening now that couldn’t happen before. And as a result, we’re learning a lot more about how people learn, you know, the book I did on the New Science of Learning, looking at a lot of the ways that students learn. And part of it’s just the ease of getting to information. But also part of it’s just being able to investigate how people process information. I used to teach Introductory Psychology back then, we would talk about the stages of sleep. And nobody really knew, for instance, what REM sleep was about, we knew that you had to have it or else it caused some problems. Deep sleep we knew was important, we now have indications that deep sleep for consolidation is necessary for semantic memory. If your sleep is interrupted, you can get eight hours of sleep. But if you don’t get deep sleep, the information doesn’t get consolidated. Procedural memory, how to give shots and kick balls and do anything procedurally looks like it’s more solidified during REM sleep. So again, the different types of sleep are associated with us learning long term, different types of information. We never knew that before all this technology was running around. In fact, back then I gotta say, I remember from my intro psych class being told that you were born with a certain number of neurons, and as you live through life, neurons would die. And if you killed them by drinking or doing something like drugs or something, they were gone forever, and you would never get more. And if you broke a connection, it was broken forever. That’s just simply not true. But it’s what we thought back then. So technology has really allowed us to look better at how people learn, different ways of helping them to learn and different ways they can even study. By the way, before we move on, we now have this physiological demonstration that staying up all night and cramming the night before the test. Even though it gets you slightly higher grades on the test, we now know that because the information is not consolidated that it won’t be there a week later or two weeks later. So we’ve always told students, you shouldn’t do it, but now we can actually show them why it doesn’t work.

John: And the LMS itself has offered a lot of ways of giving more rapid feedback to students with some automated grading with some things to give them more low-stakes testing opportunities. And those were things that we just couldn’t easily do back when you started teaching.

Todd: No, John, that’s a really good one. And we know that one of the most consistent findings right now in all of learning and memory stuff is that the more often you do something, the easier it becomes, long-term potentiation. Which means the more frequently you retrieve information from your long-term memory, the easier it is to retrieve. And just like you’d mentioned, we can now do LMS systems that are set up so that you could do practice quizzes, you could do dozens or hundreds of practice quizzes and keep pulling that information out over and over and over again. That was just not possible before this. And so the LMS helps with that, it helps by giving feedback, really good feedback so that students know what they’re doing well, and what they’re not doing well. And it helps faculty members to design feedback specifically for certain types of projects, and so that I can more easily give more feedback without spending a lot more time on it. So LMSs have done a tremendous amount of work. And that’s not even mentioning the fact that you can have all of the LMS systems loaded with the content. So students can log in and get their information without leaving their house. If there’s fiscal challenges with your class, you can put in articles, the students may not have to buy a book, they could read the articles. And so we’ve got students who were able to come to classes because they can afford to be there. By the way, I remember being on a committee when I was a graduate student, and we were looking at financial aid and different financial systems. And I remembered asking the Chief Financial Officer, I said, “What increase in tuition does there need to be before you start to see students drop off because they can’t afford to be here?” And this was about 40 years ago, but he said $100 for a year, if they have to pay $100 this year more than last year, some students won’t come back. If we look at the price of textbooks now, textbooks can cost $400. So, a book like that is definitely going to make a difference between some students being able to take the class or not. So LMS systems make this possible.

John: And they also make it easier to share OER resources that don’t have any cost for students, or some less expensive adaptive learning platforms, giving all students that first-day access. I remember, not so long ago, when I was still using textbooks in some classes, students would wait several weeks before they got that book. And that put them at a severe disadvantage. And the people who were being put at a disadvantage. were generally the students who came in with the weakest backgrounds because they came from lower resourced school districts.

Todd: Yeah, if they had the resources, they would have the better background foundational material, but they’d be able to buy the books. And you mentioned OERs. So open educational resources are really another thing that are really valuable because back then, before the technology, you couldn’t produce something that would be readily available like throughout the world. And so this project that’s going on now where they’re doing introductory level books in all the different disciplines, you can get an OER introductory psychology textbook that students can log in and read. None of that was possible before the technology. So even the creation of OERs has changed so much.

Rebecca: Well, speaking of digital materials, libraries have changed significantly too over time from having completely physical collections and interlibrary loans and things that take a lot of time to having a lot of digital resources, which changes access to research and materials that you can populate into your classes, but also can aid students in the work that they’re doing. Can you talk a little bit about the change in libraries and how that’s impacted how you’ve taught?

Todd: Yeah, you know, libraries have been fascinating to watch over the last 40 years, because it used to be the biggest challenge librarians had before them was which books to put on the shelves because there was a finite amount of shelf space. And there were lots and lots of books. And so that was the big thing. We used to take out journals that weren’t used very much to make room for other journals. Through time, little by little, they started digitizing all that stuff. And I can remember chatting with librarians, one conversation I had was back around 2001. I said, gonna be interesting, because there’s gonna come a day where there’ll be no books in the library, and the Dean of Libraries said “Well, there’s always going to be books.” I said, “Not always, potentially.” But even if we reduce them, I said, “What is your foresight? How is the library going to change?” And so he had a couple of ideas. But what it basically boiled down to our conversation is, I always felt like a library was like the brain of the campus, it had the books, and it had all of the information that you could go and get. As the books left, and things were diversified in a way that you could find this stuff, you could get all the information right from your dorm room, or from your apartment, when the internet came along, you could get anything you needed, then the library was still a physical space that was in the middle of campus. And what it should become is a learning commons, a place where people go to share and to learn from one another. And I think that’s what’s really changed is individuals still just pile into libraries and use the space, but they use it in different ways. They go there to meet other individuals to work, which they did before. But they took away that aspect of going there for the book part. And it meant all of those shelves got emptied, and they started pushing them out. And you can go into libraries right now that have very few shelves. But they have webcams, they have smartboards, they have spaces where folks can plug in their computers and share with one another. They’ve got screens set up so that you can project and have students sitting around a table, they’ve got Google Glass set up, all of these types of things that bring students together to use technology to learn from one another.

John: And they have cafes to help support that to make it easier for people to gather.

Todd: Yeah, you could swing by and get a cup of tea.

Rebecca: It’s funny, even when I was in high school, my sister and I would rely on going to the library to have access to a computer so that we could even type of paper, because we didn’t have one at home. And that kind of place of having the technology started a long time ago, but it’s amped up quite a bit over the last 20 years.

Todd: Yeah, and I agree completely. And the computers that are there. I mean, even right now, with the books dissipating, there a’re still large numbers of computers. And oftentimes, they’ll even be an area in a library that’s carved out with really high-end computers. But it gives students an opportunity to go. We make this assumption that everybody has a computer and they don’t. But libraries give them that opportunity.

John: Yeah, for those students working on smartphones or Chromebooks, that gives them access to all the tools that students with $2000 or $3000 or $4,000 computers.

Todd: Yes, because smartphones can work for lots of things. But they’re a little tough to write a paper on

John: When I started teaching, and probably when you did too, the predominant mode of instruction, which actually still is often the predominant mode of instruction in many departments, was lecture. That’s changed quite a bit since then. Could you talk a little bit about the shift from lecture-based courses to courses that involve much more active learning activity?

Todd: Yeah, or they just involve a lot more of everything. The concept of flipped classrooms, which was almost impossible 30, 40 years ago, because you really couldn’t get the information to the students. Yes, it was kind of possible, but whoo, if it was hard now, it was really hard back then. But the ability to get information out to students that they can read it before they come to class. But coming back to the lectures… So I’m going to take this moment and those of you who know me know that I’m going to do this, is that we still have no evidence that lectures are bad, but there’s something that we need to really keep in mind. I think this is vital. I do think it’s important for us to be able to talk about buzz groups and jigsaws and fish bowls and lectures and Socratic lectures, discussion lectures, all those different methodologies out there so that we know what we’re talking about when we chat with one another. But I do think it’s time that we stop talking about lectures being more effective than one thing or fishbowls being more effective than something else and look at the components of what is valuable in a learning experience.

John: And a good reference for that is a book on Dynamic Lecturing, which you happen to be a co-author of.

Todd: That is true and in fact that there’s the Dynamic Lecturing. And then there’s a chunk in that about The New Science of Learning. And then there’s a whole chapter in that about Teaching at its Best, because that’s a good point, John, thank you.

Rebecca: It’s almost like you’re trying to slip it in everywhere you are.

Todd: Because the research… people keep talking about one methodology being better than another. Here it is, folks, you can be a hideous lecturer, you can be a phenomenal lecturer. And if you’re a hideous lecturer, you’re not going to learn anything. If you’re a phenomenal lecturer, students will learn from you but they won’t learn all the time, it depends on some student factors. I’ve actually been exposed to group work in flipped classrooms that were awful. And so that concept is we start thinking about and this is why it’s going to come back to the technology, we think about the elements that need to be there, that are necessary for learning to take place. I’m just going to do this, because it’s not the topic I’ll make it very brief, is let’s just go with three things. If you don’t have your attention, as a teacher, if my learners aren’t attending to what I’m saying, if they’re on their phone or thinking about bacon, then they can’t process what I’m presenting. And if you’re having a think-pair-share, if they’re not attending to the person they’re sitting next to, you have to have attention. Number two, they have to have some value. If I’m hearing somebody or I’m reading something, and this has no value to me, it’s really hard to get it into your long-term memory and to learn it. And number three, I have to have a clue of what’s happening, I got to understand some aspects. Now if we think about attention, value, and understanding, now we can flip back to the technology. This is why gaming works. Gaming draws the attention, it increases the value, because you want to win the game, and it has understanding. We have all played games. You open up the old board games, and now it’s digital, where you don’t have a clue what the game is. It’s like, if you advance a player four pieces and the opponent advances five pieces, you have to go back three spaces, unless it’s a Tuesday. When those instructions are that complicated, you don’t understand. So we can use technology to help with attention, we can use technology to help with the value of what’s going on. And we can use that technology to help with understanding. Those are things that were very difficult before. And they allow us to do things like a mini lecture and then shift over to an active learning exercise, and then say, take all this information and create a Zoom session tomorrow that will go over it again. So the technology has really helped us to be able to do all of these things to get at the core of learning, a topic I barely care about. [LAUGHTER]

John: That’s an important one, because people often see this as this binary issue where you lecture or you use active learning. And there are some really effective ways to combine them. And in fact, in that book on dynamic lecturing, it was suggested that lecture can be more important in introductory courses, when students don’t have as much of a knowledge base.

Todd: You’re absolutely right. Discovery learning is a really great way to learn if you’ve got a lot of time. I can just put you into a room with some other people and say, “Here’s some data, and here’s some things we need to know. Go.” And if you don’t have any foundational knowledge at all, it takes forever to figure it out, you go online, you know what to look for, I could do a five-minute lecture, and at the end of five minutes, set it up and say, “Now go and work with your neighbors. In fact, here’s what we’re going to do, we’re going to have you each work in small groups in class, I’m going to open up a Padlet. At each table, I want you to go in and add your information or put it into the column that corresponds with your group number.” As an instructor, I can watch everything develop in front of me. While I’m in the room, I can look at my laptop and see it and walk over to a table and say, “looks like you’re struggling a little bit.” I’ve lectured, I put them into small groups, I’ve had them use technology, I’ve created a little bit of competition on who can come up with what and I’ve had a way for me to monitor it and give them feedback. That is so different than what teaching used to look like. So pulling it all together, that’s what we do.

Rebecca: The tools to be able to monitor have been really helpful in my own teaching and being able to get a better pulse on what’s going on and get a nice overview and then be more targeted in how to interact with small groups rather than just kind of wandering around more aimlessly like I think I did initially. [LAUGHTER]

Todd: Yeah, and this is all going to be great until we get our cognitive load headbands that I’m waiting to be developed. So anybody who’s listening, take this idea, run with it, you can make a bazillion dollars and then take me out to dinner or something. I want a headband and the headband has a light and it measures brainwave activity. And then as I’m teaching, if you start to be a little bit like it’s a little bit too much, you’re moving out of that zone of proximal development, the light turns from green to a yellow. And then when it hits red is like when you’re trying to put together an Ikea bookcase and someone comes by and says “What do you think of this?” and you say, “Errr, I’m working on an Ikea bookcase right now.” …that shutting down with that red light. I’m telling ya, that’s going to be the technology we’ll want next.

Rebecca: It would be so helpful. [LAUGHTER]

Todd: You can actually look and see somebody else’s zones of proximal development and their cognitive load. Whoof. Which by the way, there’s a little party game that they’ll do periodically at parties. It’s like if you’re a superhero, what would you want your superpower to be? And I was in a room one time and one person said they wanted to fly and somebody else said that they wanted to be invisible, which real quickly in my head, I’m thinking, what could you possibly gain that wasn’t illegal or creepy if you’re invisible. So aside from that, transporting and everything else, and they got to me, and I said, “I want to be able to see people’s zones of proximal development. If that were my superpower, I’d be the best teacher.”

John: I bet that went over really well at those parties. [LAUGHTER]

Todd: Yeah, my friends all said “You are amazingly smart and quite insightful.” They used different words, but that’s what I heard. [LAUGHTER]

Rebecca: They didn’t start with what is that? [LAUGHTER]

Todd: As soon as I start talking, most of my friends just shake their head and drink whatever beverage they have near them. [LAUGHTER] So yeah, it’s good times, good times. They’re all impressed. They don’t say it all the time, but I know they are.

Rebecca: I think one of the things that often happens with technology is that it allows us to get things quickly and move through things quickly. But sometimes, as you just noted, learning doesn’t happen quickly.

Todd: Yeah.

Rebecca: Can you talk a little bit about speed and the difference between maybe not having all the technology and all the things really quick versus maybe now where we have it at our fingertips, but do we always want it at that speed?

Todd: So there’s another study that I’m waiting to see. This is an easy study, folks, somebody can run this one quickly. We all know that students are listening to any recorded lectures or recorded material that they have to watch, 1.7 is about the best speed that we tend to see people listening. 2.0 is a little bit fast for some folks. 1.0 is like normal speed, that’s no good, too slow. So what I’m curious about is the space between words and between sentences that our brains, because they move so fast, we can listen faster than somebody can talk. And we have all this other stuff going on is I can be thinking and processing while you’re talking to me. But if I bumped that up to one seven, I think we close the gaps. And I hear it a lot faster. But what I don’t think is happening is the cognitive processing while I’m listening. The active listening component to it. So I think technology can create concerns in those directions. And students who do try to process material too fast… we’ll wait and see.

John: And that’s especially important in flipped classrooms where students do watch these videos outside. One of the things I’ve been doing with those, though, is embedding questions in the video so they can watch them as quickly as they want. But then they get these knowledge checks every few minutes. And then if they find they’re not able to answer it, they may go back and get their attention back and watch that portion again.

Todd: Yeah, I think that’s a really good way to go. EdPuzzle’s kind of a fun technology to use. I don’t know if that’s the one you used.

John: I’m using Playposit, which is a bit more expensive, it works beautifully. I love it, they did just double the price this year, though, it was bought by a new company.

Todd: This is the tricky spot now as the prices are going up. You know, inflation is a terrible thing to waste. Anytime somebody can raise prices now it’s like, “ooh, inflation”. So you know, prices double, inflation is 8% with runaway, now it’s back down around three. But when inflation was 8%, they doubled the price and say, “Hey, we’ve got to,” but yes, it’s some of them are expensive. There’s lots of things that are less expensive. Oftentimes we pay for functionality that help us but the freemiums kind of thing. So stuff that’s inexpensive. I just wanna let everybody out there know just about anything you want to do in class or can think about doing it, there’s a way to do it for either free, or probably under $100 a year, which I know $100 can be expensive for some people, it’s about eight bucks a month. And so things like Padlet that I think might be up around 140 now, so maybe $12 a month, can change how much time you spend doing things, and how much time for students. But yeah, I love the embedded questions to help slow things down.

Rebecca: I think that the cognitive load can happen really quickly if we’re piling lots of information in but not always providing the time to process and use that information in some way in the kind of activities that you were talking about. Or knowing when everybody’s red light is going off in the class.

Todd: Or when people try to do multiple things. I mean, now you’ve got the technology around. So if students are trying to listen to an assignment while they’re texting their friend and have a TV on, I mean, we’re living in an age where there is a lot going on, and people believe they can process lots of things. Evolution doesn’t happen quite that fast. And so I think we have to be careful with that one.

John: One other thing that’s happened is back when you and I both started teaching, the only way students generally communicated their learning was either on typed pages or on handwritten notes. Now we have many more types of media that students can use. And also we’ve seen a bit of an expansion of open pedagogy. How does that help students or how does that affect student learning?

Todd: Wow, that’s really changed a lot as well. Blue Books. Remember the blue books? I think they still sell blue books in the library. They may cost more than the I think it was eight cents when I started, but the concept of writing things down, you turn them into the faculty member, the faculty member would grade them and turn them back. One of the big things that I caught years and years ago was so much wasted cognitive energy in terms of what they produced. I’d read a paper from a student and think this is amazing, and no one will ever see it. It was written for me, I graded it. And now it’s done. I think the technology has changed so many things. One of the biggest things, I would encourage all the listeners, any faculty member out there is, whenever possible, create something that will take the students’ work, the things that they’re doing, and use it to make society better. It’s not that hard. There’s assignments that you can do on Wikipedia. Anybody who wants to complain about Wikipedia, if you don’t like it, I’m gonna go back to Tim Sawyer, who is a faculty member of mine, my very first time I ever did TA work. I was complaining about some students. And he said, “You can complain three times. And after you’ve complained three times, either stop talking,” he was a little bit ruder about that, “or do something about it… just shut up or do something.” And so I complained about Wikipedia for a while, that it wasn’t all that effective. And I thought, well, if I don’t like the page on cognitive load on Wikipedia, I could give an assignment of my cognitive Psych class to go on to Wikipedia and fix it. And so you can have Wikipedia assignments, there’s so many things you could do. Here’s one for you. If you’re doing one on communication, you could have your students go and take pictures or short videos somewhere on campus of something that’s meaningful to them, and then jot down why it’s meaningful, take that compilation of stuff and send it over to the office on campus that does publicity. What better way of drawing students to campus than to have all of these students that have said, I love sitting by the pond because… and in the past, we would have had students write a paper about someplace on campus that you think is effective, put it in the blue book, we would grade it, we turn it back to the students. And that is a waste of possibilities. And so I think we do have lots of ways that we can get the students involved in helping through technology,

John: One of our colleagues in SUNY, Kathleen Gradel, had an assignment for a first-year course, where the students went out, took pictures, geocoded it and added it to a map layer that was then shared with other first-year students about useful resources on campus and their favorite spots on campus, which is another great example of that type of authentic learning.

Todd: Yes, for the authentic learning, there are just so many possibilities because of the technology. If anyone doesn’t have ideas, ask deans, ask the provost, ask the president on your campus, like what kind of information would be helpful, either for the next round of accreditation or for just helping the campus and we can design those things. Another one I did was we took students to the museum. We’d go to the museum, almost any class could kind of find some way to tie museums in, and through the museum, not only would they write stuff that the folks at the museum who did curation would help use, but also just helping the students to see how issues from the museum, how artifacts and things can be used in their own life, to better understand.

Rebecca: When I first started teaching, community-based learning was popular, in fad at the time, and I think having the experience of being a student in a class like that, but then also a faculty member teaching classes like that has really informed the kinds of projects that I do. Maybe they’re not always community-based learning, but they’re often community oriented, whether it’s the campus or even the surrounding community that the campus is situated in to help students get connected. There’s so many nonprofits that need partners and love, there’s always a project that can be done. [LAUGHTER]

Todd: There is. And I used to be a director for a service learning component of the campus. And yeah, there’s just so much out there that we can do to help others.

Rebecca: And students always had such a strong connection. And they didn’t want to fail because other people were depending on them. And so there was a real investment in the work that they did on projects like that.

Todd: I will admit that I’ve never experienced it myself. I’ve never even heard of anybody that if the students are doing some kind of authentic learning, that their authentic learning is then used to help somebody else. I have never heard students say “What a waste of time” or “I hate that class,” or “those assignments are just busy work.” They’ve never used those terms.

John: One common sort of project is to create resources that could be shared with elementary or secondary school students in the disciplines. And again, they can see the intrinsic value of that.

Todd: Yeah. Students could write short manuals on how to learn and then pass that on to the first-year students. And so upper-division students could be helping the lower-division students because not everybody can get a copy of The New Science of Learning, third edition.

John: …available from… [LAUGHTER]

Todd: Available at… used to be Stylus. Since Stylus was sold to Routledge, now it’s available at Routledge. [LAUGHTER]

Rebecca: Given the historical background that we’ve walked through today, what if we think about the future? Where do you see technological changes or learning theory changes impacting the future of higher ed?

Todd: Yeah, we’re living at an interesting time. I like to point out to folks that when you go back to Socrates, Plato kind of time there was a thought that if you wrote something down, it would weaken the mind so we shouldn’t write things down. Luckily, some individuals wrote things down or we never would have known. We’ve gone through several iterations of those kinds of things. Samuel Johnson, I believe it was, who said “With the ready availability of books, teachers are no longer needed. If you want to learn something, you could go get a book on it.” Well, that was a couple of 100 years ago. And we still have faculty members, we have students writing things down, we’re reading, I don’t imagine how you could teach without writing things down and having books. The internet came along, as we were discussing earlier, while we were teaching, we watched the internet show up. And there were people who said, “Well, with the internet, there’s going to be no need for teachers anymore, because students can get whatever they want.” I can’t imagine teaching without the internet right now. So as we’ve gone through each of these iterations, there’s been this fear that maybe we’d be supplanted by some technology followed by “I don’t know how I’d work without that,” it’s a little trickier now, because with generative AI, we’re talking about not just something being available, but actually creating something. I don’t know what that’s going to look like. But there’s some real possibilities that the generative AI ChatGPT, could do things like help students who have writer’s block, get started. And that’s an individual that maybe could produce something really cool, but just can’t get started. I didn’t publish my first book until about seven or eight years ago, because I’m one of those individuals who has a terrible time from a blank screen. I just have a terrible time with that. And so now, I don’t use ChatGPT to actually write anything significant. But I will tell you that I will use it for the first paragraph. That’s all, just one paragraph. And then I completely rewrite that. And there’s no actually trace of it. But it’s something that gets me going.

John: So can we count on more than a book a year going forward? [LAUGHTER]

Todd: No, no, no, no, you can’t. So exhausting. But the concepts that will help students that can do that, I think that’s going to be helpful for them. So there’ll be a type of student who couldn’t have produced before, but now can. We are definitely going to run into some challenges, though, with students who are going to just use generative AI and use artificial intelligence to actually create and to hand something in instead of doing the work. So I do think we’re in a challenging time right now. And I wouldn’t make light of that. There’s actually something that I find fascinating from this. Right now, more than ever before, we can actually have artificial intelligence create something for us, especially in higher education, this hasn’t been done before. The tricky thing is that we were the ones to be able to make that possible, because we learned things. If we let a machine do that work for us, we’re not going to be put into the situation or our students coming along, will not be put into a situation where they’re intelligent enough to do the things that need to be done when they need to be done. And so I do think we’re facing a real dilemma right now. If my students, for instance, always do use some artificial intelligence to create a paper and hand it in, if I can’t catch it, they may end up with an A in that portion of the class. But there’s going to come a day when they’re going to have to write something or be able to read something and tell if it’s written well. And so I’m a little bit nervous, we’re entering a phase where by bypassing some cognitive processing that needs to be done, we may be limiting what we’re able to do in the future. Wrapping this up, though, I don’t want to be the person who says if you use a calculator, you’ll never understand this statistical test. So I don’t know where the balance is. But I do think we’re going to have to have decisions coming up that we’ve never had before.

John: Generative AI is drawing on that wealth of knowledge that has been produced. And for that to continue to grow in the future, we do need to have some new materials being created. So that is an interesting challenge, unless it goes beyond unless….

Todd: …unless it creates it. So that was one I thought about, by the way, sometimes you’re sitting around just thinking about stuff and it’s interesting. I was thinking how do I acquire new information. And the way I acquire new information is I go read articles, I read books, I read a ton of stuff. And then I say I think this is valuable, I don’t think that’s valuable. And then I put it together and say here’s what I’m thinking. And now I’m looking at this generative AI who goes out and scans the environment and pulls these things and then creates something new. It doesn’t have the cognitive processing that I have at this point, but…

John: it’s in the early stages.

Todd: We have some folks who are very concerned out there, especially in European countries that are starting to put some guardrails out, because at the point that it keeps grabbing stuff, and then generating and then it grabs the stuff it generated, then it’s going to be interesting. But as of right now, I just read another article, I think it was yesterday, that they’re going out and grabbing the most popular or most frequently written things and then putting it down as if that is right.

Rebecca: The way that you might prioritize as a human with an expertise in something, is going to be really different than a system that’s prioritizing based on popularity, [LAUGHTER] or like how current something is like when it was last published. That’s a really different value system that really changes priorities.

Todd: Yeah, and I think it changes how we teach. I think the way we teach is going to fundamentally shift because we’re going to have to work with students with all these things being available and explain to them and talk to them about the learning process and the value of the learning process. And keep in mind, this isn’t just about ChatGPT writing papers, everybody’s freaked out about that right now. We shouldn’t lose sight of the fact that you could get fresh, cleanly written papers that have not been plagiarized at all, we’d be able to do that for 20 years. There are paper mills, I can either write away or contact somebody and say, “Please write me a 10 page paper on Descarte, and they would write it, I could turn that in. What actually has happened recently is that everybody can do it, even those who can’t afford to have a paper written at $10 a page or whatever it’s costing. And so equity comes back again. [LAUGHTER] Now we’re an equal opportunity cheater. So we have to be careful with that. But I think the way we teach is going to change because all that information is going to be available, kind of like the internet on hyperspeed. And then what do you do with that? It’s going to be really intriguing. I think it’s an exciting time.

Rebecca: So Todd, this episode’s gonna come out right at the beginning of this semester. So you’re saying we need to be thinking about how to change our teaching. ChatGPT’s here, what are you doing for the Fall differently?

Todd: Well, I think the biggest thing is what we were just talking about, looking more at the learning process, which has been a big thing for me for the longest time, is explaining and talking through the learning process, I can hand you all this information, but if I hand it to you, you don’t learn. In fact, one of my favorite examples came from a friend of mine, and it was the gym, if you want to get in better shape, I could pay somebody to go do sit ups for me. And then I could somehow log in the book at the gym that 100 Sit ups were done, use the passive voice there, and somebody else did them for me, I’m not going to get in better shape unless I do the situps. So I have to do the work, I have to run, lift weights, do the situps in order for me to be able to gain. We need to just turn that into a cognitive process for our students to really gain cognitively they have to do the work. And so I think more than ever, it’s how do we convince students of that? And for the faculty members who say, “Well, that’d be great, but my students just want the grade.” If that’s the case, we have a bigger problem than whether or not some technology can write a paper for them.

John: So how do we convince students that it is important for them to acquire the skills that we hope they get out of college?

Todd: I think this is probably going to come down to the community building, it’s been there forever. If you really want your students to do the work, the best thing you can do in my view, and that’s why I’m gonna say, Rebecca, I don’t think a lot for the way I teach, has changed. You build a community, you build relationships, you talk to the students about importance of things, if you’re sincere about that, and they get that then yes, there’s going to be some students that are going to mess with the system, they have always been there. But you’re also going to get a lot of students who will say, “Yeah, that’s a good point.” And then they’ll do the work. I don’t teach as many undergraduates as I used to, I’m teaching more faculty than ever because of being the faculty developer. But there were years that I would have to tell my students don’t put more time than this in on your paper, you have other classes, you need to do the work in the other classes. Because, and I’m telling you, I am very proud of this, my students would spend a ton of time on this stuff for my class, because they didn’t want to let me down. And I would say you’ve already got an A, I’m proud of what you’re doing, please go work on your other classes. That kind of scenario happens when you build community. And I’m not saying it’s easy, I would never say it’s easy. And it’s not going to happen for everybody. But it is the foundation of good teaching.

Rebecca: So we always wrap up by asking, what’s next?

Todd: There’s just so much going on right now. I think that what’s next for me is I am still in that headspace of coming kind of back from the pandemic, anybody who says, “Yeah, but the pandemic’s all over,” wait for November, we won’t know, we’re going to see. But I still think that’s next is kind of thinking about how we teach and learn in this environment. So moving in that space, it’s probably not surprising. I’m working on the next book here. One of the things I want to do now is the last couple of books that I’ve done had been pretty heavy books. And now I want to write something that’s a little bit lighter. So it’s going to be more of a quick guide with more narrative and having some fun, I love telling stories. I love having fun with people. So I’m going to try to create a book that’s kind of like a science of learning and teaching at its best but really accessible and more of a story-based kind of way of looking at things.

Rebecca: Who is your audience for that book?

Todd: Anybody who will read it? [LAUGHTER] Anytime I write anything, I have to have the audience firmly in mind and think about who am I talking to. And I really believe there is a pretty big overlap with students and faculty who don’t know specific things. And I’m not saying this in a mean way toward any of my faculty colleagues at all. But there’s a lot of people who aren’t taught about things like long-term potentiation and deep sleep in terms of semantic memory, and looking at depth of processing and those types of things. So the same type of thing we can say to a student, we know you shouldn’t cram, but here’s why you shouldn’t cram… faculty learn a lot from that as well. And so my audience for this book is going to be faculty and students, students, because I think it’ll be more fun to read about how to learn in a narrative form like that. And faculty because it’s more fun to learn when you read in that kind of a format for some people. we’ll see.

John: And if faculty design their courses to take advantage of what we know about learning, it can facilitate more learning.

Todd: Wouldn’t that be cool? We could just keep rolling, rolling. What a great amount of work. I mean, a huge amount of work that faculty do. They’re hard working folks that are just cranking away all the time. Number one, making their life a little bit easier by helping to understand things would be great. And just having a little bit more fun would be fun, would be nice way to go to0.

Rebecca: Hey, anytime you can save time, so that we can have more play in our lives is better.

Todd: Yeah, just to do whatever you want to do.

John: Yeah, ending on a note of fun is probably a great way to end this.

Rebecca: Well. It’s always great talking to you, Todd. Thanks for chatting with us and going on the Wayback Machine.

Todd: Oh, you know, I love the Wayback Machine.

Rebecca: I love it too.

Todd: For those of you who don’t know about that, you should check out the Wayback Machine

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

302. Flipped Team-Based Learning

Flipped classrooms allow for class time to be used to put content into action. In this episode, Tina Abbate joins us to discuss the team-based approach that she uses in her classes to help develop the real-world skills important in her field.

Tina is a Clinical Assistant Professor at Stony Brook University’s School of Nursing. She holds a collection of credentials including a PhD, MPA, an MS, and is a registered nurse (RN). She teaches in-person and online undergraduate nursing classes at Stony Brook and conducts research on active learning strategies and the retention of information. She works as a nursing supervisor at two local hospitals.  She is the recipient of the 2023 SUNY FACT2 Award for Excellence in Instruction and was a recipient of the Stony Brook University Award for Excellence in Teaching an In-Person Course.

Show Notes

Transcript

John: Flipped classrooms allow for class time to be used to put content into action. In this episode, we look at one instructor’s team-based approach that emphasizes real-world skills important to the field.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist….

John: ….and Rebecca Mushtare, a graphic designer….

Rebecca: ….and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Tina Abbate. Tina is a Clinical Assistant Professor at Stony Brook University’s School of Nursing. She holds a collection of credentials including a PhD, MPA, an MS, and is a registered nurse (RN). She teaches in-person and online undergraduate nursing classes at Stony Brook and conducts research on active learning strategies and the retention of information. She works as a nursing supervisor at two local hospitals. She is the recipient of the 2023 SUNY FACT2 Award for Excellence in Instruction and was a recipient of the Stony Brook University Award for Excellence in Teaching an In-Person Course. Welcome, Tina.

Tina: Thank you. Thank you so much for having me here today.

John: We’re very happy to see you again. We saw you at the SUNY Conference on Instruction and Technology (or CIT) about a month or so ago. And our teas today are:…. Tina, are you drinking tea?

Tina: I am. I am drinking a chai tea. Very good.

Rebecca: That sounds nice and warming.

Tina: Yup.

Rebecca: It’s a little chilly here, although it’s summer and it was hot yesterday. It is not hot today. [LAUGHTER]

Tina: Yes, for sure the weather has been very odd.

Rebecca: So I have my tea for teaching mug today. And in it, I think actually a mix of a couple of different black teas because I switched when I had a half a cup left. [LAUGHTER] I’m not sure what we call this today, but it’s a mix of black teas.

Tina: That sounds delicious.

John: Well, it sounds like a great tea to have while discussing blended learning.

Rebecca: A high quality blend. [LAUGHTER]

John: And we’re having a real cold spell here in Durham, North Carolina. The temperature has dropped down to 87 today, and I am drinking a tea forte black currant tea.

Rebecca: That’s a nice summer tea.

John: It is.

Rebecca: So we’ve invited you here today to discuss your use of active learning tools. But before we jump into that, we were curious about your wide range of degrees, credentials, and certifications. We didn’t even list them all. Can you share a little bit about your pathway into your current position at Stony Brook?

Tina: Sure. Well, when I went back to grad school, I certainly didn’t intend to get three graduate degrees. I had gotten into Binghamton and gotten into their BS to PhD program because I wanted to do research and my ultimate goal was to do executive leadership position at a hospital because I really enjoyed the leadership role of nursing. So just to backtrack, I graduated Binghamton University in the year 2000 and started right in the NICU (neonatal ICU) at Stony Brook and I worked as a NICU nurse for six years. And in that time, I knew that I wanted to go back to school. And like I said, I got into the BS to Ph. D. program at Binghamton. They awarded me a fellowship program. So I moved from Long Island. My daughter was one at the time. And I started my education there at Binghamton, continued it for the graduate program. And about a year into my doctoral studies they had asked if I wanted to teach clinical and I’ve taught in other capacities. I used to teach violin and piano when I was younger and I never really thought of teaching as a career goal for me. However, I was a poor graduate student, and I said, “Sure, I’ll do it.” And I had about six students in the NICU. I was teaching clinical, and, I don’t know, something came over me. I found my professional soulmate, something clicked so hard for me in that clinical that I wanted more. So I continued asking for teaching assignments. And it’s hard to articulate the feeling that you have, but I felt like I found my niche. And so I did clinical instructoring for about six years and then I moved into the classroom setting. So at that time, I still worked as a nursing supervisor, so I enjoyed the leadership role. And Binghamton started a dual master’s degree program, where you get your master’s in nursing with a concentration in whatever you wanted, I chose education. And the other part of the dual degree was a Master of Public Administration. So I was the first cohort to move through that program. So I graduated first with my Masters of Science in Nursing and my functional role was educator. Then two years later, I completed the Master of Public Administration, and then eventually the PhD. And it all just aligned so perfectly in my current career, because obviously I’m an academic at heart through and through. So those degrees have assisted me in that role. I still work in administration. I teach research, I teach leadership and management. So all of the degrees I’ve utilized and I still utilize actively every day. So this pathway was kind of carved out for me, I think, and I just feel very fortunate that I’m able to apply all of the degrees that I’ve gone for.

John: At the SUNY Conference on Instruction and Technology, you gave a presentation on how you structure your courses. And you mentioned that you were using a flipped team-based learning class structure. Can you tell us a little bit about how your classes are structured, and what a typical class day would look like in one of your classes?

Tina: Sure. So any class that I’m involved in or coordinate, the structure that I utilize is a flipped team-based learning approach. And this essentially requires students to prepare prior to coming to class. It has some benefits there, there’s flexibility, students can learn at their own pace, it really amps up the student responsibility for learning, as we know, and then it also gives us the opportunity for higher level learning because they’re interacting with the concepts outside prior to class. And the team-based part of it I like is because that increases that collaboration amongst students. We know that nursing healthcare is a team sport, so I like to engage the students in teamwork so that they can collaborate and work on their team dynamics, and their own personal team skills. So how my classes operate is, prior to each class, students complete a set of videos, and they’re interactive videos, they’re accessible videos for all types of learners, and it carries weight in their grade. So basically, in these pre-class videos, students get a little voiceover content from me about a concept, and then they get tested on it using a variety of types of questions: matching, true-false, multiple choice, hotspots, you name it. As they move through the videos, they are taking notes on a note-taking guide. So all the concepts are there for them to just follow along, take notes. So they’re seeing, hearing, they’re doing something as they move through the videos. And that note-taking guide eventually acts as a study guide for them, because they have to take a quiz every single class. So they complete these videos before class. And then I start each class with a micro-lecture review using Kahoot!, which is just a game-based learning platform. And in this micro-lecture review, I’m really drilling down to the concepts and helping these students reconcile any last residual confusion that they may have about these concepts. And then after the Kahoot!, they take a quiz. Now, since they’ve interacted with the concepts so many times prior to taking this quiz, I push the level of the questions in these quizzes. There are 15 questions and I try to push the level as high as I can. And the students are able to rise to the occasion because they are not hearing the information for the first time when they walk into class. They have a vague sense of the concepts, we nail it down, and then they take the quiz. After the quiz, the rest of class is comprised of team-based activities. And that’s how every class looks like for me.

Rebecca: Can you talk a little bit more about the embedded questions that you have in the videos and how students have responded to that aspect of a flipped classroom?

Tina: Absolutely, I use a program called Articulate 360. Articulate 360 has many different types of functions in it. But I focus more on the storyline aspect of this product, where I’m able to set up these video clips. So if you already have voice overs, you can basically chop up that voice over into different bits, put it into a story, the type of file that they reference there. And then in between each clip, you can embed any type of quiz question that you could possibly imagine. And you can set up different parameters. So for example, I like to elevate the stakes a little bit, so the students, for these pre-class videos, the grade that counts is their first pass. So it’s not like they can retake the video for a higher grade. It’s whatever they get at the end of that first pass of the video is the grade that counts. And they have two opportunities to answer each quiz question correctly. And I also embed a lot of feedback, so if they get the answer wrong, they’ll see a pop up with some review, and then if they still got it wrong, or they got it right, then there’s an explanation that pops up for the right answer. So I do survey my students in the middle of the semester using a Google form. And then at the end using the university platform, and the feedback about the videos has been very positive, they really do appreciate even though it means extra work, I’m still not giving them 20 chapters to read. I’m giving them something that passes along a bit more quickly and has a better chance of sticking in their memories. And they also appreciate the note-taking guide because it also becomes a study guide, not just for the quiz, but for the final exam at the end.

Rebecca: Like I’ve counted four or five layers of countability on that same content. [LAUGHTER]

Tina: Exactly.

Rebecca: We’ve got the note taking guide. We’ve got the embedded questions, and we’ve got the Kahoot!, and then we’ve got the quiz, and then the exam at the end.

Tina: Yeah, so it’s all about building on these concepts, having the knowledge and then being able to apply it in the classroom

John: In your presentation, you mentioned that you were de-identifying the names of students taking the Kahoot!, but maintaining a leaderboard in the classroom. Could you tell us a little bit about how that works.

Tina: So Kahoot is based on answering the questions correctly or incorrectly. And part of the score is how quickly you answer the question. So ideally, you want to answer quickly and answer the questions correctly. So at the end of the Kahoot!, they get a score. And just again, to raise the stakes, students have to hit a certain benchmark of points to receive full credit. And I try to push that benchmark a little bit, not to make it impossible, but just to make it a little bit challenging for them to give them something to work towards. So for example, in one semester, they have to reach 70,000 points to get the full credit, and then it’s prorated from there. So every time I have a class, I load the data into this program that was built by our Center for Excellence in Learning and Teaching on campus. One of the computer scientist was able to put this leaderboard, showed me how to upload the files, which are basically just CSV files. And what it does is this leaderboard shows their rank in the class, their total score, and the score for that week, so that they can monitor their progress. And everybody else is de-identified and random words, but they can see their name, and they could see their rank in the class.

John: And one of the advantages, I think, of using Kahoot! is it does provide some practice in developing automaticity. So that students can practice retrieving information quickly, which I would think would be especially important in health-care situations.

Tina: Absolutely. And I’ll have some students that come to me and they just absolutely despise Kahoot! because of the stress. And if you’ve ever taken a Kahoot!, and I have, it is stressful, you have to really think on your feet very quickly, especially since your score is based on how fast you answer the question. So what I tell them from the beginning is if you really are struggling with Kahoot!, and you don’t like Kahoot!, Kahoot is really for you, it’s meant for you, because I want you to think of a situation in a hospital setting. If a patient is deteriorating, we call something called a rapid response. And a team of people flow to the room to address whatever issue it is, perhaps the patient’s having difficulty breathing, whatever, chest pain, this now has become a very emergent situation. And in that situation, you have to be, as the primary nurse or a nurse assisting someone else, you have to have laser focus, and someone may ask you to just go get a piece of gauze. And if you’re new in the role you may be so flustered, just by getting that piece of gauze. So, this is really like a precursor to that. So I tell the students to use Kahoot! as a mechanism to help with your laser focus in situations where the outcome is dependent on what you’re doing.

Rebecca: Another thing that seems really relevant to a healthcare setting is the team-based learning aspect of your course. Can you talk a little bit about how you arrange the team-based activities and also how you set your students up for success on teams.

Tina: So with team-based learning, as we know, it’s simply a collaborative learning strategy and how the team activities look depends on the course. So I can talk to you about my research course. That happens every fall semester, and I have 160 students, this is the graduating class. These are the seniors, they’re in the last two semesters of the program. And what we do in that course is the team-based activity portion of class is working on a project. So I’ll tell you a little bit about the project which is experiential in nature. Stony Brook University is attached to Stony Brook University Hospital. So every year, I pick a unit, I meet with the manager, and they give us a clinical problem to solve. So for example, this fall the students and I will be working with the surgical ICU and the clinical topic is nurse wellbeing. So, as we know, we’re in this post-pandemic world and wellbeing has really moved to the forefront. Things like burnout, compassion fatigue is very prevalent in the healthcare environment and just globally as humans. I think we’re just a little tired of living in this fight or flight for so long. And now we’re trying to come back from this. So this fall semester, the students will be working in teams to find a solution for the surgical ICU for nurse wellbeing. So what we do is we search for articles together, and that’s how they get to their solution. We use a framework we use Melnyk’s seven steps for evidence-based practice. So in undergrad nursing, even though it’s called the nursing research course, the students are expected to utilize the research that has been done on a topic to make changes to their practice. Our expectation is not for them to actually conduct research. That’s a PhD level thing, but according to our essentials in baccalaureate nursing, that our accrediting body tells us what curriculum to teach to the students, the expectation is that they know how to read the research, how to critique it, how to appraise it, how to synthesize it, and how to use the research to develop solutions. So from there, they work in teams of eight throughout the semester, they develop their solution, they put it into a video project, the six-minute video project, and I choose the top two projects. Those top two projects then move on to the implementation phase. So then the unit will implement and evaluate the solution. And in addition to that, we put in for posters at conferences. For example, last year, we had two posters at ENRS. I was assigned the course of research, I was like, “Oh, boy, how am I going to make this interesting?” …because we know that research content can be a bit dry. So I ran the course for a couple years, and I knew that I had to do something with it. And that’s where I started moving towards this more experiential learning opportunity for the students. And so far, it’s been going really well.

Rebecca: So I heard you say something about teams of eight, and I almost maybe had a heart attack, [LAUGHTER] just thinking about how big that team is, and how to manage that. Can you talk a little bit about some of the structures you have in place to help a group that size, which is relatively large, be successful?

Tina: Sure. So teams of eight… that means I have 20 teams in total. And we’re all reviewing the same articles. So then I know the answers to all the questions. And basically, Google Drive is my answer. Every team has their own folder, within that folder are subfolders, I have them buddy up and be assigned to a certain number of articles. As a team, they have like individual and buddy responsibilities, which is clearly articulated in a contract that they review and fill out at the beginning of the semester. So they have individual responsibilities, buddy responsibilities, and they have team responsibilities. And every single class looks the same. So by the second class, they’re already into the mode. I don’t throw them any curveballs, every class structure is exactly the same, so they know what to expect. And they have appraisal forms to fill out. They have tables to fill out as a team to keep all of their literature organized. And the structure that I have in place seems to be working because there’s very little confusion now that I’ve kind of worked out all of the kinks. And I also always keep instructions projected just to make sure that they are apprised of the flow of class.

John: You mentioned Melnyk’s, seven steps of evidence-based practice. Could you give us a brief overview of that framework?

Tina: Absolutely. So there’s many evidence-based practice models out there. Stony Brook goes with Melnyk, and there are seven steps and actually I begin with step zero, step zero is igniting that spirit of inquiry. And that’s one of my main end goals of the course is for them to stay curious about how they can improve practice as a nurse for their patients. So that’s step zero. And then basically, what we do is we take the clinical problem, and then we frame it in the form of a question, a PICO question. And that helps us to find our articles. So once we find our articles, we go through the articles, we decide what we’re going to keep, what we won’t want to keep, then we start to critically appraise these articles, review them, read them, understand them, the students put that information into a literature review table, which is just the main elements of each article. After we’ve appraised all of the articles, the next step is to synthesize all of the articles. So what is the bigger picture? For that synthesis class students do complete synthesis tables. And when they create these synthesis tables, now the beginnings of their proposed solution begin to emerge. So then students put their solution together based on the synthesis table. And then the next step in this process is to implement the solution and then evaluate the solution. And of course, dissemination is always the last step.

John: You also mentioned that you use collaborative testing on exams. I’ve done this with a two-stage exam process where people take the exam individually first and then submit that but then take it again as a group. It also appears to have been and that’s been tremendously successful. It’s also appeared to have been really beneficial in terms of student learning, and it’s just so much more fun to watch the students work in groups on exams, than it is to go over the exam the next day with the whole class. That collaborative exam format has been so much much better than I ever expected it to be. Could you tell us a little bit about how you do collaborative testing on your exams?

Tina: I absolutely adore collaborative testing. If you have to assess students using exams, this is really maximizing the use of exams. So in my courses, students take collaborative exams in teams of three. And as we know, the research says that collaborative testing may decrease test-taking anxiety, the students have to take a large licensing exam at the end of the program. So it may help some of these students with that, like you said, immediate feedback on test performance, it really scales back the number of questions, I don’t even do exam reviews anymore, because the immediate feedback that they get, they’ve reconciled any confusion on the exam, that an exam review is no longer required. It increases student engagement and collaboration. I love how they, like you said, they debate, they discuss, that peer instruction. There are some people out there who can read a book and retain 100%. But generally speaking, you’ll have a better chance of retaining more information if you’re teaching someone else versus reading a book. Of course, that just varies learner to learner. So that could be really something to hone in on when it comes to collaborative testing. So, yes, the traditional way is to take the test individually, and then they take it again, in a team. And in our program, the clinical courses like medical surgical nursing, pediatrics, all of those, I would always recommend to do individual than collaborative because you really want to assess that individual on their performance and understanding of the concepts. And so I teach research, and I teach leadership and management, these are non-clinical courses, I skip the individual part, and take them right to a collaborative exam. So for example, for my research course, the students don’t know who they are paired up with, or in a team with, until about an hour before the exam. They get two articles, a quantitative article and a qualitative one. And then they have a set of questions to answer. Essentially, we’ve been preparing for this type of exam throughout the semester. So they end up doing really well. In my transitions to professional practice, where I’m teaching leadership and management, that is a traditional final exam, multiple choice, select all that apply, type of questions. And again, I actually do it on Zoom, they go into breakout rooms, they share their screen, and they take the exam, there’s a scribe who enters the answers. And also when it comes to accommodations, kind of as a side note, I’ve been able to set up strategies for individuals that do have accommodation so that they can maximize their experience as well.

John: When I first tried this, I was so excited about how the students were reacting with the collaborative exam that I took a short video clip while they were doing it and sent it to Rebecca. She was working with me in the teaching center at the time. It was just a remarkably positive experience.

Tina: Do you notice a difference like I would say an estimate of 10 points between the individual and the collaborative mean.

John: Generally, yeah. And the group one is virtually always higher than each individual score, except in one case in my class, where one student had a higher score than his group, and that’s because during the group discussions the student gave in to peer pressure within the group. I encouraged him to be more assertive when he’s confident about his answer. But that only happened with one student on one exam.

Tina: that’s pretty rare. I just love just watching them engage like that. So I’ll pop into like the breakout sessions, and they’re collaborating and negotiating and it’s just fantastic.

Rebecca: You mentioned earlier about your research class having a project coming up about well being. And I think that’s a topic that we’ve been talking a lot about in higher ed in a lot of situations. Can you talk a little bit more about that project and some of the research that’s going into it and some of the outcomes of it?

Tina: Absolutely. I mean, wellbeing is such a hot topic right now in probably every type of job you could think of. And it’s interesting wellbeing is kind of always been in the background. And I think the pandemic really shoved it into the forefront where it really should have been. That really needs to be, in my opinion, the top priority of any workplace because if your employees are well, it has a positive trickle down effect. So it has gotten to the point now where our accrediting body who tells us the essentials that we need to teach to our students, they have added a wellness component, and we’re adopting these new essentials in the next year. These are new essentials for us to follow. So it made it into the essentials, which is very telling. And now faculty are charged with teaching students, monitoring students, about their wellness and wellbeing. So this was pretty timely, because of the pandemic, the clinical topic that we’ve been doing for this EBP project has been things like compassion fatigue, burnout. And now this year, we’re doing wellbeing. Last fall, we worked with the cardiothoracic ICU. And the EBP project topic was compassion fatigue. And we wove in a lot of wellbeing into the solution, which is actually kicking off on July 1. So this year, instead of doing compassion fatigue, which has a bit of a negative connotation, let’s flip it to the positive. And like I said, we’re working with the surgical ICU, and we want to customize a wellness solution for that unit. So in the meantime, by proxy, I can teach the students about their own wellbeing and their own wellness. So I have a lot of content in there, so that they’re learning about this clinical topic to help develop a solution, but they’re also learning about it for themselves. And I do a few things with them, and definitely evolving this as we move along. And I’m lucky enough that I have the graduating class in the fall and the spring. So I move it through from the fall to the spring semester. So in addition and educating them on the different ways to promote your own wellness, we start each class with a mindfulness activity. I have a sound bowl that actually a student gifted to me, we do meditation, mindful breathing, every class is something different. This year, I’m inviting students to lead some of these sessions. So I want it to grow so that other students can participate and lead us and it’s literally three to five minutes at the beginning of every class, all lights down, devices off, phones flipped down, and we just take the time to be as present as possible. And I also help them keep an eye on their level of burnout. And I give them the professional quality of life survey at the start of every class. And halfway through, I’ll do a comparison of statistics between the different cohorts. Because I have the traditional cohort and I have the accelerated one, we look to see how our scores are doing over time, just to have that educational component to it. And then also the Insight Timer app, that’s an app that you don’t have, I would highly recommend that you download it. It has so many mindfulness type of activities that you can do. There’s a journal, you can track your progress. They have classes, and even the paid version, which is I think, maybe $60 for the year, they offer so many different bells and whistles, it’s really just a phenomenal app to use if you’re looking to promote your own wellness. So the other thing I wanted, I attended that CIT conferences, is I would love to use ChatGPT to develop a wellness assignment. So I’m still thinking about the inner machinations of how that would work. But hey, you know, if AI is here, might as well see if we can use it to promote wellbeing.

John: And it’s nice to have that focus of using ChatGPT positively because this is something that’s going to be part of students’ lives going forward, maybe not this specific tool, but AI tools are not going to disappear and using them for good would be a nice alternative for the concerns that many faculty have about the use of these tools. During your presentation at the CIT conference, you also mentioned using a variety of edtech tools. What are some of the tools that you use in your classes?

Tina: Sure. So I’ve trialed some apps here and there, I’ve used Plotagon. I used Go Animate for Schools, which is now VYOND, just for them to create case scenarios in their leadership and management class. And based on feedback, the one that they really liked is now a bit pricey. So I tried a free version of an app, and it really didn’t go well based on feedback. And that’s how it works in education. You try something out and you survey the students and if the experience over time is really not positive, you need to move on to something else. But things that really have stuck is I told you about Kahoot! and Articulate 360. How I communicate with the students. I use GroupMe. I prefer to communicate with them using that application over Brightspace or traditional email. They join via QR code and I have them all in one group chat and I can post quickly. They could send me direct messages, they could post questions in our group chat. And it just seems to really streamline communication because we’re all competing for their cerebral real estate, they have a lot going on, a lot of deadlines, so I find that this GroupMe app is really helpful. And I also try not to spam them with too many messages, thoug. It really seems to work. And then again, Google Drive, I can’t even begin Google Drive for everything, whether I want to survey them or whatever it is, Google Drive has it for us.

Rebecca: So speaking of Google, [LAUGHTER] you mentioned earlier using a Google form for a mid-semester evaluation. Can you talk a little bit more about that, and how you’ve used that to make adjustments in your class for the latter half of the semester?

Tina: Sure. So a Google Form is a pretty nice way to just give a quick survey to your students, I do that in the middle of the semester. And I have to tell you, that’s where I get my best data, because they are in the throes of it. And my response rate is typically over 90%, as compared to at the end, where they’re kind of just fizzling out, tired, maybe a bit over it, generally speaking. So I don’t get the response rate in the final that I do in the mid semester, when I analyze it, very short, a couple Likert questions: What do you like? What don’t you like? …and if there’s enough of a theme in the qualitative questions, or in the Likert scales, I’m able to make changes prior to them departing from me, instead of waiting for the next cohort to come in. For example, some things that came up was: “It can be a bit loud in the classroom.” So I’ve done something to control the volume in there, because it’s a very active classroom, or we feel like we’re sitting around too long during the TBL activities. So now I have a mechanism for them to let me know when they’re done with their activities, so that they’re not sitting around waiting. So those types of things. If they say, “let’s skip the final exam,” then that’s not anything that I can honor. But I’ve gotten some really good raw feedback that’s helped me evolve my classes. I’m just always so grateful for the student experience, because they inform me where this needs to go. Another way that I use a Google Form is with team-based learning. Michelson says that you should have the team members evaluate each other on their team performance. And typically, this is done at the end. But I like to do it in the middle of the semester, where they’re evaluating each other so that they have an opportunity to remediate, and then by the end, hopefully, their team’s performance scores have gone up. The challenge, though, with a Google Form is it’s very hard for me to share the feedback back to the students, it requires a lot of copying and pasting. And there’s a lot of room there for error, human error. So currently, I do bring in the students that are rated poorly just to give them some one-on-one guidance on how to improve their team performance. But in the meantime, to work around that I did trial a product called Kritik that offers that ability where the students will get their feedback back. But I reached out to our Center for Excellence in Learning and Teaching. And right now what we’re doing, we have a sandbox, and we’re working on trying to do a Kritik-like type of peer evaluation in Brightspace, using PeerMark. And we’re getting very close to ironing out some of the finer details. So I’m going to finally have an evaluation where every student can see their feedback from their team members based on their performance, so they know what they’re doing well and where they need to improve.

John: You teach both face to face and also online. Do you use many of the same techniques in your online classes that you use in your face-to-face classes? How do you modify your class for online delivery?

Tina: So I do everything the same, except that it’s in an asynchronous format. So students really have to be self disciplined in an asynchronous online type of environment. The online classes that I teach are post-licensure undergrads, so they have their two-year Registered Nurse license, and they’re looking to get their four-year degree. So some of the assignments, we tailor a little bit differently just because they have nursing experience, whereas my pre-licensure students do not. So maybe the assignments vary a little bit, but the structure is the same, using Articulate. I don’t use Kahoot! with them, only because I don’t have them in front of me, but they do have the quiz. And they have the TBL activities and things of that nature. So it’s the same, but it’s just in an asynchronous format.

Rebecca: I know that we mentioned in the intro that you do some research on some of your teaching practices. Can you tell us a little bit about some of that work?

Tina: Sure. So a colleague and myself got IRB approval, and we’re just starting to do some research on this evidence-based practice project that the students do in my class. And we’re just starting off with a cross-sectional study. We have a valid tool that’s been out in the literature that measures their perceived knowledge, skills and attitudes regarding evidence-based practice. So, I’m not building logistic regression models or anything yet, but starting off with a cross-sectional study to understand pre and post, the beginning of class and at the end of their research class, if there’s any impact or change in their knowledge, skills, and attitudes regarding evidence-based practice. So that’s where I’m starting. And I’d like to move on from there eventually.

John: And speaking of moving on, our last question is: what’s next?

Tina: So, I just would like to continue publishing and presenting. And continuing my research. Like I mentioned earlier, I’d like to introduce an AI tool for wellbeing, and Stony Brook just purchased several VR headsets. And because my courses include a lot of content about compassion, wellness, well being, I would love to develop a simulation about empathy. I think that would be a fantastic use of VR, apart from like, typical clinical scenarios. And that’s really my plan for now.

John: Well, thank you for joining us. And when you do have some results from your research, we’d love to have you come back and talk about it.

Tina: Thank you. Definitely. I really appreciate you inviting me. This is a wonderful opportunity for me. Thank you.

Rebecca: Yeah, thank you for letting us use your class as a little case study for folks to think about ways that they could change, improve, and reconsider their own classes. Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

299. My Professor Cares

Students from marginalized groups often question whether or not they should be in our classes and disciplines. In this episode, Michal Kurlaender joins us to discuss an easy to implement intervention that faculty can use to improve retention and student success. Michal is a Chancellor’s Leadership Professor in the School of Education at UC Davis and is a co-Director of the California Education Lab. She is a co-author with Scott Carrell of a National Bureau of Economic Research working paper entitled “My Professor Cares: Experimental Evidence on the Role of Faculty Engagement.” (This article is forthcoming in the American Economic Association journal, Economic Policy.)

Show Notes

Transcript

John: Students from marginalized groups often question whether or not they should be in our classes and disciplines. In this episode, we discuss an easy to implement intervention that faculty can use to improve retention and student success.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Michal Kurlaender. Michal is a Chancellor’s Leadership Professor in the School of Education at UC Davis and is a co-Director of the California Education Lab. She is a co-author with Scott Carrell of a National Bureau of Economic Research working paper entitled “My Professor Cares: Experimental Evidence on the Role of Faculty Engagement.” This article is forthcoming in the American Economic Association Journal, Economic Policy. Welcome, Michal.

Michal: Thank you for having me.

John: Our teas today are:… Michal, are you drinking any tea?

Michal: I am not drinking tea, but I did have some not too long ago today.

Rebecca: Do you have a favorite?

Michal: I’ve come back to Earl Grey. I used to be an Earl Grey person. I left it for a while, and it’s just made a comeback for me.

Rebecca: Nice. It’s a classic. I have Christmas tea today, despite the fact that it’s June.

John: And I’m drinking a ginger peach black tea today.

Rebecca: I have my Christmas tea because we had our presidential announcement today. And it was like celebration tea.

John: White smoke has come out of the towers [LAUGHTER] and we have a new college president here.

Michal: Congratulations.

Rebecca: So we invited you here today to discuss the study of the impact of specific faculty behaviors on historically underrepresented minority student success. How did you decide on this specific intervention?

Michal: My colleague and collaborator Scott Carrell and I do a lot of work to try to understand College Access and Success. And in particular, we’re interested in understanding inequalities in graduation rates at more open access institutions, like the California State University system, which has, across its system, some more selective campuses and some more open access institutions. But in particular, what we’ve noticed for years is that the graduation rates of students of color, particularly male students of color, black and LatinX men, were really much lower than other groups. And this was a puzzle to us, largely because the eligibility to get into four-year colleges, including the CSUs is quite substantial. These are primarily B-plus students who have finished a comprehensive set of courses required to be eligible for the CSU. And so to see their graduation rates lag so much behind other students was really troubling to us. And so that’s why we decided to focus particularly on the CSU system. And we focused on one campus in particular, that’s a less selective CSU campus.

John: What was the intervention that you used?

Michal: We didn’t go in knowing what intervention to use, we actually started with a focus group with particularly men of color at this campus and asked them what their challenges were. In particular, what we learned was that their challenges were not necessarily social or more broadly campus level, they were primarily in the classroom, and that is they felt disconnected from their instructors and from what to do to be successful. These were all students who reported feeling quite successful in their high school, feeling quite connected to their high school instructors who encouraged them to go on to college. Many of them were in competitive fields like STEM and engineering. And then they felt like they really struggled in college, and in particular, how to seek help and how to understand what instructors wanted from them. And so we came in quite agnostic, I would say, about what could work, what is helpful here? Is it more writing centers, more coaching, more nudging? We didn’t know. And what we came out with is feeling quite sure that we needed to tackle the classroom. And in particular, I think what we wanted to think about was this untapped source of potential support or hindrance in that is faculty, and think historically, we know that many times we just think of the classroom as kind of untouchable, and we put other support centers, writing centers, and tutoring centers, and other supports for students. And we kind of leave the classroom alone and leave faculty, including ourselves… we’re both faculty… to do what they will. And instead, here we wanted to really think about, could we intervene with faculty to provide more support for students?

Rebecca: Yeah, it’s funny how we often overlook that particular option given that’s a key touchpoint with our students, right?

Michal: Exactly. So we came out deciding that we’re going to do an intervention that was classroom based, and that was going to try to work with faculty to give students more information about what it takes to be successful in the classroom, how to seek help. And then we decided to pilot that to see if the proof of concept worked. And we piloted it at a large, more selective institution on a small-scale pilot, and found some promising results and then launched it at full scale. This article describes that whole kind of research process as well, which we think is also an important contribution to the literature… to not just immediately do something, but to actually think about the way in which it might function and just really to understand from students what they tell us they need.

John: Your initial pilot used a light-touch intervention, could you describe that intervention?

Michal: So initially, in the pilot, what we did was a slightly underpowered pilot in the sense that we took students who didn’t complete their first homework assignment in a classroom where you have to complete a set number of homework assignments. And you could miss one, but historically, we knew that students who missed the first one often struggled in the class. And so that’s the point of randomization that we took for the pilot. That is we took those who didn’t submit their first homework assignment and to half of them we sent an email saying, “Hey, we noticed you did and submit your homework assignment. Just to remind you to do well on this course, here are some things that might be helpful.” We also provided some information on what’s coming up and reminding of the office hours and how to seek help. And then we did two others. Importantly, the two other emails provided information that showed the students that we knew how they were doing in the class so far. So after the midterm, and before the final, and we found again, it was underpowered, but we found positive effects among the treatment group. And then that was conditional on some other ex post characteristics that we added to the pilot, but then we decided to launch it at the CSU campus that we worked with at full scale across a random sample of introductory courses.

John: For those listeners who are not familiar with statistics, when you mentioned that this is an underpowered test, could you just explain that in terms that…

Rebecca: …Rebecca can understand? [LAUGHTER]

Michal: Absolutely. So they were underpowered statistically, to detect a statistically significant difference between the treated students, those who got the emails, and those who didn’t. And so for that, we need a large enough sample size of treated versus control students, particularly if we’re going to add other kinds of observations about them, like their gender, or their race, or their prior academic achievement. And so when we say something’s underpowered, we might see the positive effect that is better achievement in terms of the final or in satisfactory progress in the course. But that difference may not be large enough in statistical terms to consider it statistically significant, even if the mean differences are actually in the direction that you expect. So to get that, to be able to actually detect significant effects, you need a large enough sample size. And that’s when we launched into the population that we were specifically focused on, which was the population at this less selective campus.

Rebecca: At that less selective campus when you scaled things up, did you keep the intervention the same? Or did it change? What did that scale up look like?

Michal: Great question. So the first thing we did was we really focused on introductory courses. This was also piloted in an introductory course. But we wanted to focus on particularly large classes, especially because the information was going to come from the instructor and we were doing a randomized control trial, that is some students are going to get this treatment and others are not. So the class had to be large enough for it to not be weird that some students were getting it and others were not. If you’re in a classroom of 30, that might be strange if you’re talking to someone next to you, and they get this email from an instructor, but basically what we did is we recruited faculty, we randomly chose 30 large undergraduate courses. And then we recruited those faculty and said, “Will you be part of an experiment with us? …and here’s what you need to do.” And the important thing here is that we’re not trying to dramatically change faculty style in the classroom, we all have our own style, the own way we write a syllabus and what we expect from students. What we wanted, we had several key principles. The key principle first was that faculty need to directly communicate with the students showing them that they know who they are. So it very much said in an email, Dear Rebecca, or Dear John. They needed to provide information that was specific to their class, it couldn’t be quite generic. We provided them some templates, but the goal was for them to personalize them and say, “Here’s this upcoming unit, here’s what to look out for, here’s how I would study for it, Here are my office hours, and so it provided information. So the way we changed it is… it was a semester-long courses. and so we requested that faculty sent a minimum of three emails to students, one after the first assignment or exam, sometimes earlier if they didn’t have an exam, and two later, we wanted it within the first three weeks of the semester, one after the midterm, and one before the final. And the important thing is, in the second two emails, those were further personalized to sort of say, again, “Dear Rebecca, I see that you’ve gotten a 72 on your midterm, it’s not too late to improve your grade in this class. Here are the things that you could do. And so it was personalized also just showing that we know how you fared in this class. And so again, the goal was to let faculty, in their own words and in their own course formats, personalize these emails, with the principal being information to students, personalization to students, and help seeking behavior advice.

John: And this process is a personalization, was this done in a mail-merge type format? Because I would think to scale this intervention would be a lot easier if you did do it either using the tools within the LMS or using some type of a mail merge.

Michal: Great question. So again, this was a grant-funded study and where we could provide some support to faculty, some faculty didn’t need additional RA support from us and either knew how to do a mail merge, it really worked with their course management system, like Canvas or Blackboard and found it very efficient to work on the own. Others you may or may not be surprised, did request our help from our graduate students. And we did provide support including one actually helping a faculty member directly write individual emails for students to support. You’ll probably ask me how the faculty feel about this. And I will say we actually asked them how long it took. It didn’t take more than a minute an email and so we do kind of try to guesstimate the investment on the faculty’s time to do this, and it very much varies on their comfort level with the course management system.

Rebecca: In the scaled up version of this study, did you continue only interacting with students who had struggled and missed their first assignment or is that a shift from the pilot to the big study?

Michal: Yeah, thank you for catching that. No, it is a shift. We did open it up. We believed actually, theoretically, our priors were that anyone could benefit from this. So if you were a B or on the cusp, we have lots of researchers suggest students, especially in introductory courses, some students, particularly first gen students might take a B or a C as a signal that they shouldn’t be in a particular major. We really did want to encourage across the achievement distribution for everyone. As to John’s earlier question, as you scale this up, or as people have talked to me since this experiment said, what if I want to do this, but I teach 400 people, you could one year, one term, try it with your lower achievers, another term, try it with those at that C range, or others. And so we did in this initial intervention want to do it across the board.

John: How large was the sample on the scaled up version?

Michal: It was 20 faculty, some were in multiple classes, we had 22 classrooms overall, and roughly 3000 students.

John: Excellent. How large was the estimated effect in the scaled up version?

Michal: First, it’s important to note, as we’re talking about findings, that our findings are really concentrated on first- and second-year students or new students, and who are from underrepresented minority backgrounds, so URM students, and we find that their treatments are about five percentage points more likely to earn an A or a B in the course by comparison to control students. So again, just important to note, we find overall positive effects for the whole of treated students. But they’re only statistically significant for the URM students that we target, that our intervention aimed to focus on.

Rebecca: Was the impact limited to just the classes the students were in, or was there an effect beyond that individual class?

Michal: Yeah, so that’s a great question, and we do find what are called spillover effects. And that is that those same students, those URM students, had a positive effect of being in the treatment group, even in courses that were not part of the experiment. That positive effect was much smaller in magnitude, it was like three quarters the size of the effect of the actual treated course. But still statistically significant at the ,10 level.

Rebecca: it seems so easy, just three emails.

Michal: Yeah, it does. It takes an investment. But yeah, I think it does beg the question, I think, for me, and maybe this is something that you want to talk about a little bit later. But we do so many things to introduce first-gen students or to get students in the classroom and again to provide support externally, but we do tend to sort of assume that they’ll just survive or just be okay in the classroom without training faculty on how people might experience their classroom differently. And so, again, we do test for other subgroups. I focus on first-gen, because it’s a concept that is helpful for people to think about students who don’t have at home, at least, people to tell them what to expect in the college environment and how you might go to someone’s faculty office hours, and they’re not there, or they’re there, but they’re sort of like, “Yes, did you have a question about the material?” Whereas to know, you could go just to review the material or you don’t have to have a question, you can just show up, things like that. Being able to feel comfortable asking questions in class and others who don’t or just step in at the end of class and sort of say, I found this part of the lecture really complicated, will you be reviewing it again the next day? So things of that nature. And so I don’t think these emails did those, but they sort of remind us that there are things that faculty can do to remind students that they see them, that they see that they’re in a classroom, and that they know that they may be enjoying parts or struggling in parts, and that there are some actions that they can take to be more successful in their classroom.

John: Did the effect size vary with class size? Was there a more substantive effect in larger classes than smaller classes? Because I would think it might be easier for students to feel more lost in a larger section, especially as a first-gen student?

Michal: It’s a great question. We aren’t able to test that, our sizes were all pretty similar, and we didn’t have enough. We actually chose the largest classes that exist in this campus, which don’t get much bigger than 150. I think it is worth testing. Absolutely. I can tell you from the pilot, that, in particular, that was a class of 400 students, and both for the pilot and for the full scale up the types of emails we got, that faculty got back. So that’s one thing we could talk about is how did students respond more generally, and many students emailed instructors back, and in particular, in the pilot, but also in the scale up, we got those emails back from the faculty who were in our experiment, and they very much appreciated the email and said, “No faculty members ever emailed me before,” or “especially in a class of this size,” or “I so appreciate the email. I’m working really hard.” So the first and sort of overwhelming finding from these emails is just gratitude from students that a faculty member emailed them, particularly we noted that in the pilot when the class was quite so large.

Rebecca: Yeah, I can imagine that just something that feels personalized, whether or not it’s super personalized or not, just feels personalized, really helps students feel seen.

Michal: Exactly that. That’s right. I should also say in addition to just grades and you asked me about graduation outcomes, we also included a survey, both in our pilot and in the scale up, to try to get at some of the mechanisms and in particular, we asked questions like, “Did you feel this instructor supports your learning? Did you feel you could reach out to your instructor?” And we do find consistent with our intervention that students in the treatment group reported more positive outcomes on these dimensions.

Rebecca: That’s so fascinating, because it’s so easy. Like it just isn’t that complicated.

John: In your study, you also examined how this effect persists over time, which certainly relates to the graduation thing. What did you find in terms of the persistence of this effect over time,

Michal: We did look at long-run treatment effects. That is, we waited to see what happened several years later, we presented on this paper in the shorter term outcomes. And we tracked these students and worked with this institution closely, we really wanted to know did it affect the outcome we care most about, which is graduation? In other words, we care about student success in a particular classroom, and maybe they’re slightly better grade, or not dropping out of the course. But we really care about their longer term outcome of finishing. Again, for this specific group of interest. We note that the treatment results in that 7.3 percentage point increase in persistence, one semester later, and then a higher four percentage points difference in graduation. So we do find positive effects on the likelihood of graduation.

Rebecca: We’ve talked a bit about the impact on students and how students have responded, how did faculty respond to participating in this intervention?

Michal: First, I’ll just say, again, we had to recruit faculty to do this. And so we do track the faculty who said no, and we did as much reconnaissance work, if you will, to understand that we did need to self select faculty, keeping in mind that if we did self select faculty who just had a proclivity to help students or being this intervention, if anything, we perhaps under reported some of our findings, but we do as much effort to understand how representative our faculty are, which we did determine they are, and in the paper, there’s some details about that. And they represent faculty from a real diversity of disciplines, from music to engineering, across the board, humanities and social sciences, we had the whole range, we had the range of faculty. So you know, “I do this a little bit sometimes. And the two of you are, I would have said, I’d do the same with a student who doesn’t show up or doesn’t complete to sort of track them down. But I’ve never done it systematically. And I’ve always wondered if it even matters.” We had everything from that kind of faculty member to a faculty member that’s like, “Well, I’ll do this, but I don’t think it matters. I mean, at the end of the day, the students who want to put in the work put in the work.” And so we had a whole range in their attitudes. We did offer a stipend to do this, because we did believe it takes time. And we wanted to sort of show that faculty who do have a lot of demands, especially at teaching institutions, that this was going to take some time. And so we haven’t done it again without an incentive, or with an institutional incentive, that’s part of like performance evaluations or something. So that’s yet another thing that in terms of where to take this in terms of where institutions might take this, for our perspective, it was externally funded, they were only accountable to us in their efforts to do this. And so we talked to them multiple times in the term, again, some were in both waves of the study, because we did it over two terms. And then we surveyed them at the end and really got some details from them. Some of this is in the paper around how they felt. And I would say most expressed similar to what you expressed, Rebecca, which is like “Wow, this some emails and I made this difference, especially in underrepresented minority students lives and in their classroom, and it felt really good about the impact.” …keeping in mind, we talked to them also, before we knew the results. And so at that point, they just were sort of documenting how much time it took to do the emails, and what kind of emails they got in response from students and most felt, I would say, humbled by the thank yous that something so small, like an email, got so much gratitude back from students. We did have some faculty that sort of said, this takes a lot of time, and I’m not sure it’s much of a help. In our last survey with faculty, we actually provided them the full scale results and said, “Here’s, by the way, what we learned from the study,” and then asked them to respond or to reflect on that. And many said, “Wow,” like, again, similar to your reaction, “a few emails could make such a big difference, I will be sure to continue.” We ask them directly if they will and we report this in the paper. Most do say that now that they know the positive impacts of this study, they’re likely to continue with these emails.

Rebecca: I imagine the workload for a faculty member isn’t necessarily in drafting those initial emails, but maybe the responses to the emails [LAUGHTER] you might receive back.

Michal: It’s a good question. I don’t know how many continued the conversation once an email was sent, the standard emails, part of the experiment and the student wrote back and said, “Thank you so much.” I’m not sure they continued. We did a lot of qualitative coding, which we don’t report in the paper, we report some but then we did a lot more internally. And there definitely were a lot of hardships described among students who did reply, the extent to which faculty replied with those hardships offering extensions or any other kind of augmentation to their requirements, we don’t know.

John: In an introductory microeconomics course I use the Lumen Learning Waymaker package which actually does automate emails to students based on their performance on weekly quizzes and so forth. And even though the students in the class realized it was automated, they’d still write back to me and I’d respond to it. And they’d often apologize, saying, “I had a bad week and I know I need to work harder.” But it did often start a dialogue that might not have occurred otherwise. And I’ve often wondered how large that effect was, but because it’s automated for the whole class, it’s hard to measure the differential effect of that. So it’s nice to see this result, that that type of approach works.

Michal: Yeah, I think that what you’re describing is exactly right, this sort of feedback. Our whole intervention is built around theories, not just from kind of behavioral economics or nudging or information source, but on the education literature on feedback, and the important role of feedback, and the timing of it. And it’s most useful if it’s not just performative like feedback, like your grade on an assessment like a “C,” but that actually gives you more information about how you’re doing or what to do to improve. And so this kind of thing you’re describing John is exactly right. I think we know that more touch points with students through assessments, as opposed to all hanging on a midterm and final also support students to get more feedback about how they’re doing.

Rebecca:I think sometimes students know that they’re struggling, you get a grade back, you know, if you’re doing well or not, but I think a lot of students need more coaching around what to do to improve or to better understand how the grade is calculated, to just take the time and attention. It’s there. It’s in the syllabus. But sometimes they don’t realize what they should prioritize. And including some of that kind of messaging makes a lot of sense. I know that when I’ve done that with my students, they’ve been really appreciative because they didn’t realize that they were putting all their energy into something that didn’t really matter as much.

Michal: Yeah, that’s right. And I think coaching is a great choice of words around what to do with it. I also think many students come to our universities with really uneven or unequal preparation for those courses. And so I think a lot about students who came from a high school where they took an AP course in economics or chemistry that might as well have been a college level course. Many of these questions are great on a grade curve. And so that C might be an excellent grade for them, given the type of preparation that they had, but they don’t know that necessarily, and they might, to them, signal that maybe this isn’t the right major for them. And so I also think coaching around what to do with the grade when you’re kind of passionate about a subject and not to give up on yourself too quickly. Many are juggling jobs, we know for some, it’s their only work is to get through this term, and others are doing this while working and taking care of family members or whatever. And so that grade that they got often conveys information that we as faculty don’t necessarily know anything about how they’re interpreting,

John: I would think just a signal, as in the title of your paper, “my professor cares” might create a sense of connection and belonging that might otherwise be missing for someone who is a first-gen student who might not feel that they belong in the institution.

Michal: Yes, I agree. And I think there are more and more studies coming out, particularly in social psychology, but elsewhere about the importance of belonging. We know from the K-12 literature that it’s having a teacher who cares about you matters, actually. And again, nothing dramatic happens once you get to college. But we assume it’s completely different, where in fact, I think having an adult or particularly your instructor care, you feel a sense that that instructor cares about your learning, or how you’re doing in their class, irrespective of the grade per se, just that you’re making progress in the class or feeling comfortable in the class, I think is really important. And I think it’s hard to test. And most of the belonging literature has been on survey type research, “I feel like I belong here.” And it’s not as much in the classroom, although there’s increasingly more studies about belonging in the college classroom beyond just a university at large.

John: A while back, we interviewed Peter Arcidiacono, on a paper that looked at the impact of differential grading between STEM and non-STEM courses. One of the things you just said reminded me a little bit of that, because one of the things that was noted in that paper is that many of the people, particularly female and underrepresented minority students who switch their majors out of the STEM fields had some of the higher grades in the class, but it was below their expectations. And I’m wondering if this type of intervention might have an effect of letting them know that that performance in that discipline may not be all that bad. Since we’re probably not going to eliminate grading differentials between STEM and non-STEM disciplines, perhaps some type of personal communication might help preserve some of those connections so we don’t lose as many people in the STEM fields where the returns to education are the highest.

Michal: Absolutely, I think well said. That’s exactly right. And I think that is among the reasons we wanted to get across the grade distribution, not just those who are really, really struggling. And also because we do think that students might give too much meaning from a signal of a grade early on in their academic pursuits where they can get through a certain amount of courses and then maybe where the fun stuff of their major where they really see that utility of a particular course for a future career path matters. And so I think that’s right and I do think Peter’s work and other people’s work looking at the impacts of particular kind of grades and grading distributions or signals of grades, I think, are really important. I think that’s an area that’s blossoming in economics and in other fields to sort of better understand heterogeneity or differences between subgroups around how particular information like an assessment, grade or a test score.

John: One of the things we’ve been seeing in a lot of studies is that many changes in education, such as using more active learning techniques, providing more course structure, benefits all students but disproportionately benefits those students who are historically underrepresented. And it seems like this study just provides more evidence of that, that good teaching practices benefits everyone, but especially benefits the people who are most at risk in higher ed. And those are often the people who can get the greatest benefit by persisting in higher ed.

Michal: Yes, I think that’s exactly right, if we’re really going to address disparities in college outcomes, and I think one really important source to go to is the importance of information gaps. And that would be particularly for first-generation students, for students of color, but also for students who come from unequal K-12 backgrounds. Colleges and universities often know and often are recipients of systematically particular high schools in their states, especially public flagships, community colleges, others, and so they are aware, they offer relationships with those K-12 high schools that are feeders to their institutions. And that is an important source of information that they can provide to high school students as they enter college for a kind of a warmer handoff, if you will, but that also faculty teaching introductory courses can provide. And so I think, again, if our goal is to address inequalities that we see in college outcomes, then I think information, particularly for those for whom their information gaps, is particularly key. Students want to be seen and not necessarily yes, there’s the anonymity of a large lecture hall, that maybe don’t want to be called on. But that doesn’t mean you don’t want to know that your faculty member sees you and knows whether you’re doing well or struggling, or how you feel about the class or how to succeed on the next exam or in the next course in that sequence.

Rebecca: So if there’s other faculty who think, “Hmm, three emails, that seems easy,” What recommendations would you make to those faculty?

Michal: Yeah, I mean, I think what the first recommendation I would make is to try it, to do it. I think thinking about how you communicate in your syllabus that about your forms of communication are important. So if you’re going to do an email, I think one thing that we would have loved to test and if we were to continue further is the format for the information. And so I think letting also students know that you want to hear from them over email or through other means, I think is useful. So first, deciding on the kind of medium like how you’re going to communicate this, I think email makes sense. When faculty start texting students, maybe we’d move to a text them information. But that’s not the case for most of us, so it’s through the course management system or email. I would say focus on again, what we know from the literature on feedback is that for it to be as specific as possible about what students can do with this information, and so that is looking at your syllabus closely, knowing… we typically do as faculty… where students trip up in the material, what’s complex, what’s up ahead. And so giving that kind of feedback as well about how to prepare for the next assignment or exam, what has tripped people up in the past, what you know, might work for them is really helpful. Again, other research has suggested the importance of going to office hours might matter. But that means you need it to show up, you have to think about how you structure your office hours. Incidentally, we did try to track that… quite hard to do whether it actually promoted more office hours in the pilot, we believe it did promote more office hour usage. More broadly, it’s something we’d like to test, the actual help-seeking behavior of students. So I would say faculty should do it if you’re teaching a 400-person class and you can’t imagine doing this for 200 students or even 100 students, maybe start with students who you see as struggling based on that first assignment, as we did in the pilot and see what you can learn from that… maybe do as John suggests, which is kind of get savvy on a mail merge and think about ways to do this so it’s more efficiently done, so that you can reach as many students as possible.

John: We just switched recently to Desire to Learn’s Brightspace platform, and it does have intelligent agents and it does have replace strings so you can automate an email conditioned on the grade on either your overall course grade or on a specific item. And if you do it on overall course grade, (which I just set up, by the way, for my summer class last week), students get an email saying, “I see that you’re struggling, there are some things you might want to try.” It would be nice if I could put in their grade without having to go to mail merge, but I don’t think that would be possible. And I’d like to scale this up. In any case, I’d encourage them to contact me during my office hours or to make an appointment to talk to me. The first iteration of that went out last week. None of the students responded, but it’s a small summer class. So I’m curious to see how this might work. And your paper helped encourage me to try this. I had other reminders out there, but this was one that I thought might be useful, using a specific grade trigger.

Michal: That’s great.

John: We always end with the question. What’s next?

Michal: Well, I think our lab and Scott Carrell and I continue to do this work, and in particular, we’re also spending a lot of our time these days doing work at community colleges in California, which serves one in four community college students nationwide. So also persistence outcomes are quite weak at community colleges, historically, and we’ve seen real declines in enrollment at community colleges since the pandemic. And so we are definitely doing some work at community colleges. We continue to track and follow graduation rates, particularly inequality in graduation rates at CSU. And we’d love to launch another intervention. So stay tuned on that. I can tell you, we are quite committed to understanding the college classroom beyond college settings more generally, and so hope that the college classroom continues to be a source of important information for the field about how to better support student success,

John: You’re doing some really wonderful research. And it’s really nice to see some of the attention that this got because your article has been mentioned in The Chronicle. It’s been mentioned in Inside Higher Ed, and I’ve seen people tweeting about it ever since it came out. It’s good to see this research becoming popularized.

Michal: Well, we appreciate it, especially since Scott and I did not succeed on that front, that is becoming those people who do good social media. Other people are better at that than I am. And I’m always a little troubled when I talk to more junior faculty around “Do you need to do all that?” …and 10 years ago or so I would say “No, just do good work and it doesn’t matter.” And now I confess the sort of buzz that some people are better able to develop around their findings in their papers seems to matter and so it’s really nice when it happens to you because we didn’t do it ourselves. [LAUGHTER] So I appreciate my friends Sue Dynarski and others who’ve done a really nice job promoting this paper.

Rebecca: Well, thank you so much for joining us and for sharing your work with our audience.

Michal: My pleasure. Thanks for having me.

John: And we will include a link to your study in the show notes and we encourage our audience to read it.

Michal: Wonderful.

John: Thank you.

Michal: Thank you so much. I really appreciate it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

293. Study Like a Champ

The study strategies that most students use may be helpful in passing high-stakes assessments, but do not generally support long-term recall of fundamental concepts. In this episode, Regan Gurung and John Dunlosky join us to discuss a new resource they have created that is designed to help students develop more efficient study strategies and improve their metacognitive and self-regulatory skills.

Show Notes

  • Hacker, D. J., Dunlosky, J., & Graesser, A. C. (Eds.). (1998). Metacognition in educational theory and practice. Routledge.
  • Gurung, Regan and J. Dunlosky (2023). Study Like a Champ: The Psychology-Based Guide to “Grade A” Study Habits. American Psychological Association.
  • Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological science in the public interest, 9(3), 105-119.

Transcript

John: The study strategies that most students use may be helpful in passing high-stakes assessments, but do not generally support long-term recall of fundamental concepts. In this episode, we discuss a new resource designed to help students develop more efficient study strategies and improve their metacognitive and self-regulatory skills.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Regan Gurung and John Dunlosky. Regan is a social psychologist and is an associate vice provost and Executive Director of the Center for Teaching and Learning at Oregon State University, He is the author of over 120 peer-reviewed articles and has co-authored or co-edited 15 books. John is a professor in the Department of Psychological Sciences and the Director of the Science of Learning and Education Center at Kent State University. He co-authored the first textbook on metacognition and has edited several volumes on education. Regan and John are the co-authors of Study Like a Champ, which was published earlier this year by the American Psychological Association. Welcome, John and welcome back, Regan!

Regan: Thank you.

John D.: It’s great to be here.

Rebecca: Today’s teas are:… Regan, are you drinking tea?

Regan: Just water because it is a scorcher here in Oregon. So water, and lots of it.

Rebecca: Hydration is important. How about you, John?

John D.: No tea for me for about five years, I used to really be addicted to oolong. And now I’ve gone full espresso. [LAUGHTER] Water for me today too. It’s a little bit late for espresso on the East Coast.

John: And I have an Irish Breakfast tea today.

Rebecca: Oh, John, you’re really pumping up the caffeine today. Also on the East Coast,

John: My day started at 5:30 this morning…

Rebecca: So did mine.

John: …and it’s going to be going really late. We’re in the midst of grading here, we’re recording this a little bit earlier than it will be released. So we’ve invited you here today to discuss Study like a Champ. What prompted you to create this book?

Regan: So both of us have been teaching for a long time, both of us do research on teaching and learning. And I think both of us really like taking stuff from the lab and testing it in the classroom. And, I think, as passionate teachers, we noticed that we knew the stuff, we knew what students should be doing, we told our students what they should be doing. We even tried to design our classes in that way. But clearly, we needed to do more of it. And the impetus behind this book was let’s put these expert tips in the hands of students everywhere. We didn’t want it to be just those students who had teachers like us who talked about studying, we want it to be in student hands. And that’s where it came from, specifically driven by the fact that even though there’s some really neat stuff out there on learning, very little of that, if any, is written to the student. And that’s something right from the get go when John and I first talked, we agreed that this would be in a voice that would speak directly to students.

John D.: Absolutely. In fact, I think that was the main impetus because of all the wonderful books and volumes out there on learning sciences. And just too many of them, I think, appropriately so written for teachers. And often they tend to be a little bit fact listing because of that. In other words, I’m going to tell teachers everything there is to know about learning, which can be overwhelming. And what Regan and I wanted to do is kind of find the most effective and best little snapshots of the learning sciences to share with students, the things that we think will move the needle the quickest, so to speak. So by no means do we tell students everything there is to know about the learning sciences, but we hope we tell them the best stories to get you back onto the learning track, so to speak.

Rebecca: So you begin your book by discussing a variety of widely believed learning myths. Can you talk about a couple of these myths?

John D.: Sure, one of the biggest and I really don’t think this holds true for college students, but K through 12, definitely, it’s just the myth of learning styles. So, we all have learning preferences. I certainly have preferences. I wish my instructors would sing to me all the time and do my learning through music. It’s just that’s not going to help me learn calculus, and so forth, where really hard work needs to be done. So it’s natural to accept a myth like that, because it makes everyone feel good. It really can also undermine learning because it makes us want to do things that are more aligned with our preferences, but sometimes misaligned with most effective practices. So that’s kind of one learning myth, which I hope, in 10 years, if we were to have this discussion again, it would not come up because it was completely dispelled. But there are a variety of others too, that I think can undermine student success too.

Regan: And I think something else, that we really tried to put ourselves in students’ shoes, and we built on what our students said to us. And another one I wanted to highlight in response to that question is so often I think our students think that learning is up to us, if they have a good teacher they learn. If they have a bad teacher, they won’t learn. And undoubtedly, teacher effectiveness is important, but it’s also up to the student. The student has to do something, the student has to take accountability. And early on in chapter one, we show that at least half of the variance half of the accountability for learning is in the student hands. Teachers are also responsible, but really, if a student isn’t doing their bit, then learning is not going to happen. And now once you are ready to do your bit, well the rest of the book tells you what to do. But you’ve got to realize that don’t just sit back and go “Come on teach. Do your magic.” We all like doing our magic. But we can do all the magic in the world if you are not joining us in this adventure, and I’m thinking adventure a lot, because Zelda Stairs of the Kingdom just got released recently, and my household seems to be quite into it. So yeah, join us on the adventure.

John D.: Absolutely. Let me just jump in with one other learning myth that really gets me, and it literally gets me, because I fall prey to the same thing. That’s if you’re struggling, you’re not learning because it’s frustrating. And I often try to pick up new skills, I’m teaching myself to play the guitar now. And I struggle, and I get frustrated, I want to throw my guitar down because things aren’t happening. And I have to remind myself that sometimes it’s the struggling when you’re doing the most learning. And if we could allow students to realize that if you’re not struggling, you actually may not be learning, and that you should embrace the challenges, the hurdles that are in front of you. And it turns out, some of those hurdles are embracing the most effective learning techniques, which produce struggles in learning, but yet also produce the best long-term outcomes. Whereas other ways to prepare for classes and so forth, the way I’d like to teach myself to play guitar, makes it seem very easy. But then there’s really no progress being made. And if there’s any method I could really want to undermine, that would be a major one, that sometimes struggling is a good thing, not all the time, and it’s up to the students and the instructors to figure out is this a good struggle or bad struggle and how to correct that.

John: Those times when we try to give students desirable difficulties, they often find those to be somewhat undesirable. And I often get comments in my course evaluations that “he’s making us learn it ourselves.” And my response is always “Well, I can’t really learn it for you, I can give you the tools, I can give you resources, I can help you learn this material, but ultimately, you have to do the learning.” Going back just a little bit to your comment, John, about learning styles, I had a discussion in an online class, which is mostly upper-level students. And about three quarters of the students in my class were very strongly convinced of that myth of learning styles. So it hasn’t gone away. We hope it ma, but it is very pervasive, certainly in K through 12, where they’re regularly testing it, but some of the students have mentioned that they’ve been tested here at the college in some of their classes. And it’s again, really frustrating to see that, but it’s pretty pervasive.

Rebecca: Yeah, I was struck too, John, when you were talking about both learning styles and the struggle, because as I was working with mostly graduate students this semester, I had those exact conversations with those students. But it may have been the first time that they’ve really come up with some barriers that they faced in learning, and really had to reassess how they were working towards their goals.

John D.: Absolutely, and nothing but utmost compassion for students and sympathy, because in many ways, the experiences they had before college allowed them to use ineffective strategies and still get by. So things didn’t seem like a struggle, the difficulty now, the first time, then you hit a struggle, when you really have to change strategies or techniques, you may not realize what to do. It might seem frustrating and overwhelming, when in fact, now’s the time when the real learning begins.

Rebecca: Yeah, and I think sometimes when the subject matters new, as well as the strategies are new, students can immediately feel a little cognitive overload. [LAUGHTER]

JOHN D. Absolutely.

John: And part of that I think may be related to how they’re often taught throughout their whole career. In general, much of the advice they get in elementary and secondary schools, and sometimes in college, encourages them to adopt strategies that may not be that effective. Often, people still use high-stakes exams. And people don’t do a lot of interleaved practice or spaced practice, which are things you talk about really nicely in your book. And the incentive structure for students is perhaps to favor things that align well with short-term recall without that longer term learning. And your book can help, if we can get students doing this, but it seems to me like we also have to do a bit more work and getting faculty to adopt some techniques that may encourage students to use techniques that are more effective. Might this be something we could build into our classes a little bit?

Regan: Actually the riff over a couple of different things you said, John, I think there are two things going on here. One is, a lot of the issues that we run into with our students are actually pretty easy to pinpoint. And that is high school. And when I say high school, I’m not saying “Oh, high school teachers,” I’m saying the whole high school environment. One of my favorite examples is the syllabus. Many of us college and university faculty complain that students don’t read the syllabus, and I had a great conversation with my 17 year old. I was talking about the syllabus and he said, “Yeah, the syllabus is useless.” I said, “Excuse me?” and he said, “Here’s what we do with the syllabus. We spend a whole day, a whole day, reading through every word on the syllabus, but the syllabus is not really designed to help us learn.” And I looked at his syllabus. And sure enough, the syllabi he had didn’t really map on to the best practices that many of us listening employ. And I could completely see why he had that whole. “Yeah, I’m not going to read the syllabus when I get to college. I said, “And of course, you will now,” but it was a great conversation where I really got to see the mindset. And I think that’s just one example, where a lot of this happened, where it’s like, let’s see, what have they just been used to? Then add on top of that, how learning changed during remote learning. That’s another whole mix to the whole issue. And then, add one more, the misinterpretation of, I think, learning styles. Before we started this call, we were joking about the fact that our book is now in an audio version. Now, because it’s an audio version, does it mean the auditory learners will be better than those….? No, it just means that sometimes when you want to listen to something that works well, other times reading something works better. So it’s not this whole “Oh, look, there’s this one modality and I’m a person of this one modality.” And I think that preference that John D. so nicely used… it’s preference, right? …and we have preferences. But let’s also not forget the take home message of that Pashler et. al 2008 study, which is the more styles that you use…. And here’s where it goes back to desirable difficulties, mixing it up, is actually better for learning, no matter what your preference may be.

Rebecca: Oh, but mixing it up is so uncomfortable. [LAUGHTER]

John D.: It makes things a lot more difficult, right? One advice I would give too, that my teaching has changed dramatically, as I’ve gotten more into learning sciences. And at least for someone who teaches, I hate to say this, a non-essential course course that I love and courses that I love for undergrads, I realized I was teaching them too much. So I put them into almost a defensive mode, where the only way to prepare, because they were overwhelmed, is to cram as much as they could and hope to pass the exam because it was a tsunami of information. So over the years, I’ve followed the model, less is more, it’s up to me as the instructor to figure out what is really essential, and how to give them just as little as possible to make their experience as large as possible. So that they then can use the techniques to learn the content well enough so that they can keep that content with them a lot longer period of time. So it’s really kind of changing styles. Now, don’t get me wrong, if I’m teaching chemistry to pre meds, and you need to know chemistry, then you need a different approach, because there’s a lot to learn, it’s difficult, and aspects of our book really focus on how to meet those challenges too. And I think students, just like instructors, need to pick and choose where they want their battles, you don’t have to ace everything as a student, but decide what you really want to do well at, and then use the most effective practices to nail it as you work forward toward a long career and lifelong learning.

John: One of the things you include early in your book is a list of what research finds to be effective and what research finds to be ineffective, side by side, which for faculty use could be really helpful except for some degree of similarity between those two lists. Could you just talk a little bit about that.

John D.: I like to think about it, as much as I’ve not talked about it like this, as more or less effective versus effective and ineffective. And I’m not gonna like reference to that table necessarily, but let me talk about one that I badmouth a lot, which is highlighting as a learning strategy. And like Sharpie has not contacted me with a lawsuit or something because it sounds like I don’t want people to have highlighters. And what we mean by less effective as a learning strategy here is, is that using a highlighter is just the beginning of a learning adventure and not the end of it. So there are great uses of a highlighter of rereading material, of these things that really don’t lead to a great deal of learning, because they kind of are the stepping off point or the catapult for learning, so to speak. So is using a highlighter ineffective for learning? Absolutely, it doesn’t help you learn the content that well. But what I recommend to students: to highlight everything they want to learn as kind of a different approach to that strategy, so that they then apply the most effective strategies in learning it. Yes. So as much as we kind of play off ineffective to effective, it’s more like every strategy in its right place. And some strategies that can be generally ineffective for very specific uses might be relatively good, and what I think Regan and I try to do, because it’s way too much to think about if you’re not a learning scientist, like one exactly show us the specialty strategy, but we try to focus on are just strategies that students could learn just more generally to do well in any particular content. So kind of low hanging fruit, so to speak, where you don’t have to be a learning scientist to make decisions about which specialty strategy to use in every place. But here’s some kind of fail-proof strategies that can help you anywhere.

Regan: I want to actually say there’s that flipside too and John, this is something you asked about earlier when you talked about teachers changing what they do, and I think John D, and I joke about the companion volume to this is Teach Like a Champion. But I think the nice thing about that table of high utility and low utility strategies is that it’s not just the student who needs to be aware of the fact that some are high and low. But I think we take it further to say why are some of those high and low, and then here’s the key companion piece, which is, we hope that instructors reading this can also take those tips, and be ready to share different options with the students. Because I know both John and I have compared notes on this, when a student comes to our office after an exam who’s not thrilled with their score, who has not scored well, we both do some pretty intensive. “Alright, so tell me exactly what you do.” Just that conversation is so important. We both ask to see their notes. And of course, every once in a while a student says I don’t take notes. Well, let’s start right there. And that’s why we have a whole chapter on note taking, and why take notes and how to take notes. But I think that’s exactly it, is so many of us. and I’ll say us, not just our students, so many of us are so used to doing things just one way that we’re not ready. And we don’t look enough about is this really working or not. And I think that’s what we really push with a lot of very real examples ripped from the headlines of our lives, as it were, where we talk about, look, this is what somebody’s tried, and here’s what they did wrong. And here’s how they can make it better.

Rebecca: I wanted to talk a little bit about that chapter on notetaking. Faculty often make the observation that students don’t take effective notes, but then don’t necessarily respond by teaching students how to take notes or giving them some strategies to improve that skill. Can you talk a little bit about some of the things that finally might show students or offer students to take more effective notes?

John D.: Absolutely. So the one thing that I would recommend highly is that if we all agree that taking notes in its own right is not where the learning occurs. And in fact, research comparing people who take notes versus who don’t take notes typically showed no differences between the two. Now, the key is, when you look at these studies carefully, you’ll realize the folks who took the notes, and who didn’t take notes do equally poorly. So it’s not like they do equally well. So the notes are a artifact that in many cases students can use to really learn the content well. So the question is, how do you get all the correct and appropriate information into those notes? So one thing that teachers can do, that research shows really helps well, it’s just flag important content, literally tell students “Okay, time to take notes, this is critical, I need you to know this.” One of my favorite statements, ”The next thing I say, will be on the exam.” [LAUGHTER] And see how many people start taking notes. Now, of course, I would love to see notes that are embellished with examples, with questions, and so forth. But at least getting the rudimentary content down there as an artifact, so students have it to utilize as they’re preparing is so important, or having teams of students take notes and share those things as well is also good. So I think for teachers to be compassionate, but also to help your students sometimes identify what’s most important, by speaking louder by saying, “Hey, this is really critical.” Sometimes I have to even flag it to not only just critical, but it’s going to be an exam, you’d think those two things meant the same thing, but sometimes students need a little help. I think it’s important, and there’s nothing wrong with telegraphing what is most important in your class. In fact, let me restate that. There’s everything not right about not telegraphing. Now, try to parse that. But what I’m saying is, we should be telling the students what they need to learn, how we’re going to test them on it, what we’re going to test on, and allow them to meet those successes. And I think good note taking, and helping students take notes by telegraphing what should be in there, can be very useful, in my own experience, and at least some of the research too.

Regan: Notice what else is in there. If we do want students to take good notes, we don’t want to be talking extremely fast. We want to make sure we have pauses, we want to make sure our slides, to the extent we use them, are not so packed with information that either they want to copy it all down or they get it all as a handout so they don’t have to, but there are all these nuances. And I want to go right back to that finding about the students who take notes and the students who don’t take notes. The bigger issue is what are you doing with those notes? I think far often students just think about it as “I’m going to record what the instructor said.” Well, you’ve got to go back and revise them. You’ve got to go back and check and see if your understanding is right. You need to review them. The issue is, I think, when people take notes, they’re not doing enough with those notes. And there are ways where we can talk about how to take better notes and they’re not. When I teach 100, or intro level classes, every once in a while, I will stop, and I do this during the first week, I will stop. And I’ll say, “Alright, here’s a reality check. If you took good notes, you should have the following. How many of you have it in your notes?” And I do that a couple of different times that first week or the first two weeks to get into that habit of you should be taking notes, you should be doing something about it. I also want to say something that didn’t make it into that chapter on note taking, but it’s gone up another shot, because I’ve seen this so often, in the last year and a half or two years, students taking photographs of the slides. Reality check, people, that’s not taking notes, taking a photograph of a slide is not taking notes. But I think in some students’ minds they’ve taken notes, just more effectively. And there are studies now rolling out that show that the photograph slides versus note taking versus controlled no notes. that the photographs and no notetaking is doing the worst on quizzes. But I think that’s something else we’ve got to take up” how you take notes is important and taking a photograph is not it.

John D.: Let me spin back to something that Regan said that I think is great, too, as far as helping students develop better notes, I begin, and I know Regan does this as well, every class with a no-stakes quiz. And I use that for a variety of different reasons, some of which we talk about in our book, but one of them is first try to answer the quiz multiple choice question without the alternatives from memory. And if you got it from memory, that’s important, because that’s going to help you learn. And then if you don’t get it from memory, now see if you can answer using your notes from the last class. And if they can’t answer the question using their notes, we have issues. So they’re not taking complete notes, they don’t understand their notes, or the issue could be with me. Occasionally, I look at the notes and realize, “Oh, my goodness, I must have said something wrong last class, because everybody’s notes are incorrect.” [LAUGHTER] And if everyone’s notes are incorrect, the source of the error is probably me. So it’s a way to help students understand that the notes are the vehicle to understand the content that we really want them to learn. And there’s ways to reinforce that through no-stakes quizzes, and so forth to help them understand that what we’re doing in the classroom is providing all the scaffolding to help them succeed.

John: One of the things you emphasize early in the book, and throughout the book is the importance of students developing their metacognition as well as their self-regulation skills. Could you talk a little bit about why that’s important, because I think that’s something that’s not emphasized enough throughout their educational experience,

John D.: I’m gonna just focus on one aspect of our particular viewpoint on self regulation, and then I’ll let Regan go with the more fun stuff. I want students to really succeed in school and in college. But quite frankly, as a college educator, I want students to succeed in life after they leave here. And one of the most important skills that anyone can learn is time management, that’s about self regulation. And believe it or not, I have actually students tell me, it’s like, “Well, I know, managing my time and developing plans, that’s just not my cup of tea.” (speaking of teaching for tea). But it absolutely should be, because effective people develop goals, plan on how to get them, and manage their time to get there. So if students can take something from this book that I think is highly valuable about self regulation is to look at our examples and our encouragement and inspiration to manage your time, make goals and plans, and develop a time management strategy that works well for you. Because that’s ultimately going to lay the foundations for a successful use of other strategies we discuss in the book, but also for a highly successful life, whether it be success in your job, your hobbies, raising a family, or whatever, successful people manage their time and self regulate. That’s something that we really do push in the book, and some of it’s not necessarily metacognition, per se, but it’s about really taking ownership of your life and deciding how to get things done effectively. So time management, for me is so important aspect of this book.

Regan: I think whenever you think about self regulation, I really try to stress two elements of it on one hand, especially think about metacognition is the: “What do you know? Do you know this stuff?” And there’s that whole “Am I ready for this exam? Do I really understand this concept? Do I know how to do this?” That’s the classic metacognition, but even more broadly, when we talk about plan and monitor and assess your knowledge, that’s, I think, really important as well. And of course, the tips such as retrieval practice is great for the assessment too and to assess yourself. But I think I love the way John talks about time management because when we talk about self regulation, a big issue that I hear from students a lot is where those two things interact, not being able to regulate how they use their time, especially when it comes to social media. And students will say to me, “Yep, I start scrolling Tik Tok, and before I know it’s an hour later. Well, that’s a basic self-regulation issue. That’s a time management issue. That’s a planning issue. And I think in our planning chapter, we recognize this and we tell students “Look, it’s okay to enjoy social media. But guess what? Plan your social media timr. So it’s not like you’re saying, I’m not going to use social media. Be real. If that’s what you enjoy, sure, look at some social media, but plan it out, allocate time for your social media, just like you allocate time for your work so that everything fits into your schedule.”

Rebecca: Allocating time, it feels like have a lot of conversations with students about that on a regular basis.

John D.: Oh yeah.

John: Are there other topics you’d like to emphasize about your book that we haven’t touched on yet?

Regan: Well, I think one thing that if you’ll notice, the last chapter in our book is all about things that, for the longest time, nobody thought related to studying: sleeping, eating, physical activity. But I think that is just so important. That is so important. And gosh, I know, I don’t think I ever shared this with John. But sometimes I almost played with putting that chapter first. We did put it at the end, because I think it’s really important. But I think most students don’t realize how those life things interact. And I think for those of us who read up on higher education, there’s just so much written about burnout among students, among faculty, and all of that relates to those things we talk about in the last chapter. It’s prioritizing sleep and prioritizing good eating and prioritizing physical activity, prioritizing just getting out and getting some air. And some of my favorite comments that I get back from students… it happens a lot when I teach health psych… but even otherwise, where they will say, “you know, yeah, I caught myself scrolling, I went outside for a walk, and it was just five minutes, but I came back and I could tackle things better.” And I think, people, it’s 360 degrees of living is what we’ve got to practice.

John D.: And I love that Regan. Because obviously, we saved the best for last with that chapter. But, come on, to do all these things. It’s a time-management issue. If I’m going to work, all of this in, how can I do so so I have enough time for play and I have enough time for work, which are two important aspects of life. So, totally agree.

Rebecca: Well, they always wrap up by asking: what’s next? [LAUGHTER]

John D.: Wow, that’s a big question. And what’s next, I would say for me, the strategies that I think are most effective, that work the best, unfortunately, have also been investigated the least, because they’re difficult to investigate, they involve multiple sessions for students to engage the same material, and so forth. So as someone who was kind of born and raised in a laboratory, either my own lab or laboratory of the classroom, what I’d love to see is just much more evidence-based research focused on further understanding, not only what works best, but how students can engage in the best practice in the most efficient way. So it’s balancing both an understanding of wanting students to obtain their learning objectives, but also understanding that they want to do so effectively and efficiently. Because there’s so many aspects to life beyond just school. So I’m always going to say more research, although I could totally understand some would say more application too, but I’m gonna go what’s next is some more research and all these really effective techniques and how to use them more effectively.

Regan: And it’s that last part, John, that is what’s exciting me right now, I’ve been learning a lot more about the whole field of implementation science, which is just because something is effective, and something is efficacious, what are the factors that influence its implementation? …because whether it’s a student reading our book and trying to implement our recommendations, or a faculty member, instructor, teacher listening to us and trying to implement what we’re saying, there are still nuances, there are still contextual factors. And I think we’re just getting better at intentionally and systematically teasing apart what some of those issues with implementation are. And for me, that’s been pretty exciting. I was fortunate to read a lot of a fellow Oregon State University, new PhD, Dr. Rachel Schweitzer, who’s done a lot of this work on implementation science, and some of her ideas. I’d love to be able to test them in my class, to dovetail with continuing research on study techniques and how to make it happen more,

Rebecca: Some really important stuff that needs to happen, for sure.

John: And thank you for this book and all the other research and work that you’ve done in support of improving student learning.

John D.: Thanks for having us on.

Regan: Thanks so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

290. Transparency in Learning and Teaching

While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, Mary-Ann Winkelmes joins us to discuss what happens when instructors make their expectations transparent to their students.  Mary-Ann has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd.

Transcript

John:While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, we explore what happens when instructors make their expectations transparent to their students.

[MUSIC]

John:Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John:…and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Mary-Ann Winkelmes. She has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd. TILT is an acronym for Transparency in Learning and Teaching in Higher Ed. We are very much fans of the TILT approach and have referred to it often in workshops on our campus (and on previous podcast episodes). Welcome, Mary-Ann.

Mary-Ann: Thank you. I’m really delighted to be here with you. And I appreciate the opportunity to speak with you on Tea for Teaching.

John:We’re very happy to have you here. You’ve long been on the list of people we’ve wanted to invite. So we’re very pleased that you’re here today. Today’s teas are:… Mary-Ann, are you drinking tea?

Mary-Ann: I am indeed. And I’m drinking a Sencha green tea today. That’s my new favorite kind of green tea, Sencha.

Rebecca: Nice. I have English breakfast today.

John:And I am drinking a mixed berry Twinings black tea…

Rebecca: Hmmm.

John:…which I haven’t had in a long time. I wanted to mix it up a little bit today.

Rebecca: …mixing it up with mixed berries. So, Mary-Ann, can you tell us a little bit about how the TILT project came about?

Mary-Ann: Sure. This was years back, I want to say in the early 2000s, late 1990s, where I was working at the BOK Center for Teaching and Learning at Harvard University. And I was leading a seminar group discussions about teaching and learning. And we began to think about the question: “What happens when you tell students why you’re teaching how you’re teaching, just what happens when you tell the students more about your choices as an instructor, how you’re choosing to shape the learning experiences for the students?” And that’s not often something that we think about first when we’re thinking about what’s the content of the course. But we began to think about that a lot. And we had a kind of metaphor about the Wizard of Oz, and pulling back the curtain to show what was happening behind the scenes to build the experience. And then somehow through that conversation, the word transparency emerged. And that became the word that we used pretty regularly from that time on. When I moved to the University of Chicago, that was the word we were using, and it kind of stuck. So that’s kind of where it started. And it started alongside of my career as an educational developer. And it’s kind of been, for me, in the background or on the side, as something that I’ve been kind of tracking along with as a project. It’s still there, it keeps going. And just about a year ago, I began to work on TILT as my full-time job, which I’m really happy to be doing now because it gives me an opportunity, not just to do a guest talk here or there, or a keynote address, which is usually a one time-interaction. But now I have the flexibility to connect with institutions around a longer-term project. So if there’s a faculty learning community that emerges from a first talk that I would give, I get to follow up with them later and see what’s happening and check in with them. Sometimes I get to see the assignments before and after, which I really like. And I invite those now, because we’d like to publish some of those on the TILThighered.com website. And there are some schools that I’ve been working with in the state of Washington for several years now running with their TILT projects. And that emerged from a project we did with the entire state system of Community and Technical Colleges in Washington State. So I have opportunities now like that, where I can work with larger scale TILT projects that take more time, because this is my full-time job now. And I’m really happy about how that’s working, because I feel like it’s getting larger beneficial impact for students in a way that’s more efficient than when my full-time job was at an individual institution.

John:Could you give us an overview of the TILT framework?

Mary-Ann: Absolutely. So the TILT framework is meant to be a very simple tool that is a framework for an ongoing kind of communication among teachers and students. And in all of our studies, we asked teachers to use this framework in their own way at their own discretion, because we know that it’s not really possible to expect that people would do the exact same things with it. So our research is based on the premise that people are using this framework in their own way, at their own discretion, in a way that feels consistent with their teaching style. So there are three parts to this framework: purpose, task, and criteria. And what we ask in all of our studies is for teachers to engage students in conversation about three aspects of a particular assignment or a project or even an in-class activity. Before the students do a piece of work that we want them to complete, we’re asking for teachers and students to have a conversation about three aspects of the work before the students start working on it. And those three aspects are the purpose, the task and the criteria. Now the purpose kind of consists of two pieces. The first part is talking about the skills that students will practice while they’re working on the assignment. And then how are those skills useful, not just now in this course, or maybe in college and other courses, but how are these lifelong learning skills that will be useful for the student in their careers after college or in their lives ongoing? And then the second part of the purpose is about the content knowledge. What new information or what disciplinary information will the students be researching, or gaining, or applying when they’re working on the assignment? And how will that be also similarly useful to them, not just now, or in college, but beyond in their lives? The task, that’s the second part of the TILT framework, and the task is sort of about what are the teacher’s expectations about how students will approach the work? And for the students, it’s kind of like mapping out their game plan, like, what’s the first thing they will do? Will they Google something? Will they go to office hours? Will they go seek out a research librarian? Will they go into the lab and start mixing something like, what’s the first thing they’ll do? And then a sequence of what they plan to do after that until they submit the work. In an ideal world, the teachers and the students would have similar expectations about how that would go. In some cases, though, teachers have a pretty legitimate pedagogical reason for hiding that, that they don’t want students to know how to do the task. And I found this to be the case, particularly in fields where creativity is really important: performing arts, studio arts, even engineering or some STEM courses, where teachers really want students to cast about for a while and kind of use their imagination and see if they can come up with something unique, if not into the discipline, at least unique for the student to try to figure out some new process. And there’s value in that. When teachers want to do that, we did have some pushback from teachers in our original TILT research studies, where they said, “What happens if we don’t want to tell students how to do the work, like part of the task is for them to figure out how to do the work?” So in that case, we asked for those teachers to just say something like, “Part of the purpose of this assignment, in addition to the skills and the knowledge we’ve talked about, part of the purpose is for you to struggle and feel confused, while you invent your own approach to the question.” And we think this is what helps to preserve the student’s sense of confidence and their sense of belonging. Because instead of having that moment of panic of “Oh, no, I don’t actually know how to do this, I don’t even know where to start, I don’t know where the resources are, I don’t know what I’m doing. Maybe I shouldn’t be doing this, maybe this isn’t for me, maybe I shouldn’t be in this major, or in this course.” Instead of going to blaming it on themself or to questioning whether they’re up to the task at all, students can say instead, “I am totally lost right now. And that is exactly where I’m supposed to be. I know I’m on track. I’m doing great. This is the confusion part that comes before the clarity. And I know that because we talked about that and the professor said, this is part of what we expect to happen. This is intentional, this confusion, you’re supposed to feel lost right now.” So that’s sort of what we can say about the task. And the benefit of students knowing upfront what the task is, or knowing how they plan to approach the assignment or the project, the benefit there is that students get to spend 100% of the time they’ve allocated to work on this project, doing their best quality work, and they don’t lose time trying different approaches to see if this or that is going to work or looking for resources that aren’t what the teacher intended for them to be using. Instead of losing time, on the “how,” students get to spend their time so that what teachers receive then is most of the time what we’re looking for, which is “What is the student’s highest capacity right now?” Let’s see an example of the best work that the student can do right now, so we know where they’re at and we can bring them further so that they can advance in their learning. But if we accidentally end up in a situation where a teacher didn’t intend for the students to be confused, they expected the students to take a particular approach that they may have even mentioned at some point in class. So that’s why they think the students know that that’s the expected approach. I don’t want to say the correct approach but at least what they expect students to do. So if we think that students know how to do what we expect them to do, and the students don’t know what we’re expecting them to do, then there’s this chunk of lost time, where what we’re measuring then in the end is what happens after the students spend a chunk of their time lost trying to figure out how to approach the work, and then whatever time is left after that doing their best quality work in the amount of limited time that’s left. So part of the “task” piece of the framework is about what do we want to measure? Right? Do we want to be assessing the best quality work that students can do? Or do we want to be assessing what happens when you give a really varied, diverse group of students a particular assignment to do and you don’t give them 100% clarity about how to do it, and then kind of what you’re measuring is which students have, through no fault of their own, not encountered that information in their lives before coming to this course. And then you also get to identify who are the students that maybe because they had some other kinds of privileges that not all the students had, who are the students that can figure it out faster, because they come equipped with those privileges. So you can begin to see that this is an equity issue. So if talking about the purpose of the assignment kind of speaks to the student’s motivation, and to the value that they will gain from doing the work, and maybe to their ability to assess if they’re getting that value while they’re doing the work, the task speaks to even more of an equity situation where we’re trying to get all of the students to the same starting line of understanding of how to do it, and of having all the resources they would need to do the work to complete the work. And we want to make sure that students are all at that same starting line before they start the assignment. So that’s kind of the equity piece of this. And then finally, the third part of the framework is about criteria. We want students to be able to understand while they’re doing the work, how well are they doing. We want them to be able to make corrections, if they end up with a finished version that doesn’t look like what successful work would look like in this kind of a scenario. But if the students have never seen what successful work looks like, and they probably haven’t, because why would you assign them to do something that they’ve already seen many examples of; they wouldn’t be learning anything new. So kind of by definition, students aren’t going to know what successful work looks like when it meets this or that criterion in the discipline. So what we encourage teachers and students to do there when they’re considering the criteria is to offer students more than just a checklist or a rubric, because the words on a rubric or checklist might mean something different to the student who hasn’t done this kind of work than they do to the teacher who’s really immersed in this kind of work. An example I sometimes offer is, let’s say, I asked students to write up an analysis of a 15th century wooden painted sculpture of the Madonna and child from when I was teaching Italian Renaissance art history courses. In an art history course, the word analyze, like the tasks, the actions that you take when you are analyzing something, that’s a very, very different activity than analyze in the context of an economics course, or in the context of a chemistry course. But if the student hasn’t done this kind of analysis before, you can’t know for sure that they know what you’re asking them to do. So we kind of have to talk that through and students are going to need to see some examples of real world work in the discipline so that they can, with you, in a class meeting, talk about how do we evaluate analysis in this example from the real world, or in that example from the real world. And you won’t find any one example that matches every criterion of the assignment you’re asking students to do, usually, so you need several examples. The benefit of several examples is also that you can begin to talk about the relative success with which different examples are meeting a particular criterion as well. So once we’re in a conversation with students, and we hear back from them, that they’re telling us, what we had hoped they would understand about the skills they’ll practice and the knowledge they’ll gain, that purpose, about how they’re going to approach the work, the task, and about how they’ll know that they’re doing good quality work, the criteria, once we hear students telling us that, that’s the moment that things have become transparent. It is that activity of communication, that conversation with students about purposes, tasks, and criteria, that’s where the transparency comes from. And when we are done with that conversation, we know that students are at the same starting line of readiness In terms of their understanding of what they’re going to do, and also, in terms of their confidence that everyone has the resources that they need, in order to complete that work

Rebecca: What faculty believe is important for students to learn doesn’t always align with the goals of students. Can you talk a little bit about some strategies for bringing these into better alignment?

Mary-Ann: Sure, I think that this kind of speaks to the purpose part of the transparency framework. And often teachers are expecting students to learn something that is very valuable, we wouldn’t spend our time teaching things that don’t have a lot of benefit for students or that they would only use today and it wouldn’t be useful to them later in life. We like to teach things that have value. And so, when we are communicating with students about that value, we’re talking about the skills that students will be practicing. They won’t perfect them on this assignment, but they will begin to strengthen a particular kind of skill set. And they will gain some sort of disciplinary knowledge that can be useful to them later. And we know that sometimes disciplinary knowledge changes over the years as people discover new things and publish new things in any field. Sometimes that knowledge changes. But having some knowledge now does give you important value if you’re going to continue in that discipline or if you want to understand basic principles of a discipline that you might find useful elsewhere. So if students and teachers have a transparent conversation or communication, it could be a written communication, it could be something that they record and put on a website, it could be an asynchronous kind of conversation in an online course. But whatever form that communication takes, I think students and teachers when they’re on the same page about what the knowledge is, what the skills are, that are the focus of this assignment, students will feel more motivated to do the work, because they’ll see that it has benefit for them. And it doesn’t feel like a rote exercise, or just churning out another problem set or another art history analysis paper. There’s some value here that the students know upfront what that value is. And when the teachers hear the students reflecting back to them in this communication, that this is the value that they will be gaining, then we know that students have a kind of motivation to benefit from this assignment.

John:One other issue is that students have come up with some way of learning while they’ve been in elementary and secondary school. But those methods that they picked up are not generally the ones that are most effective. How can we encourage students to adopt learning strategies that they may be resistant to because for example, students, when there have been surveys of what types of learning strategies they found most productive, students often say they prefer to be lectured at, because they learn more from the professor that way. And also, many students don’t like active learning strategies. While they learn more, they don’t perceive it that way. Partly because of those desirable difficulties you referred to before, that when they’re struggling with something, it’s a little bit less pleasant than sitting there nodding and smiling and having everything seem to make sense. How can we encourage students to accept those desirable difficulties associated with learning so that they can learn more effectively,

Mary-Ann: I want to say that this is something that the TILT framework can definitely help us with. And this is not an uncommon phenomenon at all, I even find in my TILT workshops that I do with instructors, that instructors don’t love collaborative learning either. And in fact, many of these TILT workshops that I do will begin with some kind of a research review about “How do we know TILT works? What are the studies and what do they tell us and show us the data?” So we get off on this kind of role, where we’re almost in a traditional lecture format, where like someone’s delivering some information, and people are listening, and then they have questions about it. Or maybe they have challenges to say, “Wait, this doesn’t make sense, let’s talk about this.” And then I kind of switch the method that we’re using. And I’ll ask people to break off into small groups and begin to analyze a particular assignment and talk about where do they see the purposes, the tasks, and the criteria? Before I do that, I acknowledge the fact that we are shifting gears, and that we were doing fine with this sort of Q&A format. You know, look at the research and then think about it and talk about it. Ask questions. Why would I switch that up now? Like we were on a roll, we were doing great. Everybody was sort of on board. Why would I change that now? And so I use the TILT framework to talk about why we’re shifting gears now. What is my purpose in having you use this different method? So if it’s a peer learning method, as it is in the workshops, or as it might be with students in a class, we want to tell students: “Why are we now manipulating your learning experiences this way? Why would I do that to you when I know that sometimes students resist this, when I know that it can be uncomfortable, because I don’t personally always like to do it when I’m in a learning experience?” So if we can tell students, here’s why this is going to benefit you, because you don’t just hear it, but you have to struggle to apply it, you have to fit it not to the situation that I was talking about, where it all sort of makes sense when it rolls over you and you’re hearing it. But you now have to take the principle of what we were talking about, and apply it to this new unfamiliar scenario. And the benefit of that is that you will discover you will hit a barrier at some point in that process, where you will discover the exact piece of information that’s missing for you. You will discover exactly where you hit a barrier to your understanding. And you will have an opportunity right now, right here with me, the teacher in this class, to address that confusing point. And the benefit of doing that now, as opposed to later when you’re doing a graded assignment, is pretty obvious, you get the benefit of having the difficult learning experience in a safe environment that doesn’t lose you any sort of points on your grade. It doesn’t have any negative impact on you the way that it might if you waited until the end of the term to do some massive project and you hadn’t really done a lot of the homework or done a lot of the practices and so you didn’t really know what you didn’t understand until it was kind of too late to do anything about. So I think in short, what I’m trying to say is when we’re asking students to do something uncomfortable, that has a really solid pedagogical reason, that has evidence behind that, it is an evidence-based practice, we want students to know that upfront, because that then will increase their motivation to do it, because they see how they’re going to benefit if they do this thing.

Rebecca: One of the things that students often struggle with is when they start new courses with new faculty, and new ways of doing things and determining what the instructor will expect out of them and out of that learning experience. Can you talk a little bit about how the TILT framework could allow students to shift their focus to learning if it was adopted in the design of the course rather than just an individual single assignments?

Mary-Ann: Yes. And in fact, this is a way that lots of faculty are using the TILT framework, is to think about how do I TILT not just a single assignment, but a whole course. So usually, when people are introduced to the TILT framework, the original ask for all our research studies is would you please do this two times in an academic term, just twice? Because we wanted to see how little change could you make and have a beneficial impact on students’ learning, because small change is much more likely to happen than massive change. But once you’ve made that small change as an instructor, and you see that when you do this with two assignments, there’s some real benefit for students. And on the TILThighered.com website, there are publications by faculty who talk about not just how the quality of students’ work increases, but how the teachers experience in grading, or in responding to students, or in how many students will ask for an extension at the last minute, like these difficulties that teachers often face are diminished, while the benefits for students and the quality of students work increases. So once you begin to see this in the small scale of assignments, teachers then, maybe in the subsequent term, will think about what else could I TILT? Could I TILT in-class activities? Could I TILT a unit of this course? Could I TILT the whole course? And then the effects or the applications can grow. So we can apply this to a single assignment, we could TILT a whole course, we could TILT a curriculum in a department, we could TILT a program, we could TILT an institution’s learning outcomes and thread them through not just all the courses, but through all the co-curriculars too so that students might discover in their work-study job that they’re practicing one of the critical thinking outcomes, that’s a goal for the whole university that connects with what they were doing in their accounting class. And then we can even think about this in terms of a national framework of learning outcomes as well. So there are many scales at which you can apply that to a framework. And one of the things that I’m really enjoying about doing TILT full time, is that I can work with groups of schools, groups of institutions, so not just the Washington State group that I mentioned to you, but several weeks ago I was in the state of Kentucky where working with teams of teachers from institutions across the state, for the whole state system, to think about aspects of how do you map out a path for students to succeed in fulfilling their curriculum? And then how do you pursue that path? How do you complete that path? And in that case, we were using the TILT framework as a strategic planning framework to think about once we know what the plan is, like, once we’ve mapped out our plan for how students can effectively complete their degrees, how do we then communicate the value of that degree, not just to the students who are doing the degree, not just to the students’ families who may be contributing to the costs of doing that degree, not just the costs of the student’s tuition, but the cost of the student not being an earner in that family. And we want to communicate this to all the stakeholders, so the students, their parents, faculty, and staff at the institution, to state legislators who may be voting on packages of funding to higher education in their state, to individual grantors who might be funding particular scholarships. And we want to be able to communicate the value of this degree to every stakeholder in a state system that way. And the TILT framework is very helpful for thinking across multiple audiences, because that’s a pretty difficult task to communicate clearly to all of those different kinds of audiences. But it’s pretty essential for the success of higher education in this country. And so we spent a couple of days using the TILT framework as a strategic planning framework to think about how do you communicate the value of a degree? There are lots of ways that you can apply the TILT framework. Another example is I was working with a school in Texas over the summer, and they were TILTing their entire college success course. Many institutions have that kind of course in the first year, and some of them had TILTed individual assignments. And they decided they wanted to put the team of all the teachers together, and then subdivide that so that a smaller team of teachers was working on each week of the course. And then all the assignments and the lectures or discussions that would go into that week. And then we use the TILT framework as a larger framework to connect that whole course. So that from week to week, the purposes, tasks, and criteria were pretty clear. And students understood the path for all of their learning across that course.

John:Have you tried taking on the Florida Legislature? [LAUGHTER]

Mary-Ann: I have not.

John:That’s a real challenge, I suspect.

Mary-Ann: Yeah, I have worked with schools in Wisconsin. Last week, I was working with a school in Tennessee, right after a couple of their legislators were expelled temporarily. This kind of a framework, I think, can be effective in a lot of different higher education systems and contexts. That’s one of the beauties of it. Because this is something that teachers can do, starting right now, to complement any kind of larger, institutionally driven or federally funded program that might focus on student success. A lot of the time, those programs don’t necessarily feel like they’re directly connected to what faculty members are doing in the day to day in their classes. But using this TILT framework is something that you can do that will advance students’ success that will then make you feel more like you’re connected to these larger ongoing efforts that might be focusing on something that you don’t do directly, like targeted scholarship funding, for example. But that’s part of the beauty of the TILT framework is that it can work in many, many different contexts, and across different scale sizes of projects, as well.

John:And it works nicely for faculty because you end up getting work of the quality and the type that you expect, rather than getting student work that you find disappointing. And similarly, students end up doing work that they’re much more happy with, because they were not guessing at what the instructors want. So it just seems really, really logical. But it’s not always so widely practiced. Your efforts are really helpful for all of this.

Mary-Ann: I think one of the reasons why people might be hesitant to use the TILT framework, you don’t necessarily want to try doing something different that could suck up time that could take time away from delivering important content in the course, and what teachers have discovered and written about and published in the National Teaching and Learning Forum and other places you can see on the TILThighered.com website, what teachers have discovered is that if you take some class time to talk about the purposes, tasks, and criteria for a project before students do it, by the time that practice is completed, everyone has saved time; that time gets recouped, and students have learned a larger quantity of what we had hoped they we’d learn because when we deliver content in a course, we don’t know that students are absorbing it the way that we’d hoped or that they could apply it the way that we’d hoped. So I think by the end of the course, if you’ve used the TILT framework a couple of times, you’re in a situation where you’ve worked in a way that is more time efficient, somewhat, and you arrive at a place that, as you say, is more satisfying for students and teachers, because more of the time has been spent with the students doing the highest quality work possible.

Rebecca: I think one of the things that can be challenging for faculty initially is that if you’ve never communicated in this way, it’s hard to do it the first time, because anything you do the first time is difficult. But once you have a little practice doing it, it’s easy to adopt and expand across a course or across a set of courses.

Mary-Ann: That’s so true. And I think that the way that we’ve structured the TILT framework, it looks so simple, it’s a three-part framework. Applying it then gets you into some complexities that are important to clarify. I think you’re absolutely right, the first time we try anything that’s unfamiliar, just like for students, it’s more difficult. And then we kind of get the hang of it. And then it comes much smoother, and much easier. The TILT framework for starters, is pretty simple. It’s got three parts, right? And I think you could probably share a link to the one-page version of the framework that we give to students, that sort of spells out the framework: purpose, tasks, criteria, the knowledge and the skills. And then at the bottom, there are some of the evidence behind why we know this works and some footnotes, so that students can see on one page, this is a real thing. It works, it helps you. It is, in some cases, equitable, and it is probably worth giving it a try. And if you can see all that on one page as a student, then you might be more willing, especially in a context where a teacher is describing to you why this will be good for you, why this is a benefit for all of us. And then for teachers who have not encountered the TILT framework, when students can bring in this one pager that has some studies listed at the bottom and footnotes, they can see that when the student is asking me, why should I bother? This is actually a legitimate question. This is not a troublemaker student, this is a student who actually knows that they will benefit from knowing a little bit more in advance about this assignment that they’re planning to do. So we try to make it as easy as possible to implement. And then we also try to say only a little bit of this will make a statistically significant difference for students’ learning, so that you only have to try it a couple of times in a whole term. And you’ll probably see the kind of differences that we saw in terms of increases to students’ confidence and their sense of belonging, and their metacognitive awareness of the skills that they were practicing and developing. So if you’re doing anything new or different for the first time, yes, there’s some difficulty to that, but this one is a very, very desirable difficulty. [LAUGHTER]

John:We’ll share a link to that one-sheet document as well as to your website in general. And you do have a lot of research cited on your website. And there’s also some ongoing projects. Could you talk a little bit about those?

Mary-Ann: Yes, we are sharing all the resources that we possibly can on the TILT higher ed website, because we want for everyone to have access to this. Some of the places that benefit most are places that might have the least amount of money that is allocated for faculty development or educational development. So we want to make sure that this is accessible to anyone who would want to try it. And then the studies that we’ve done in the past, there are a few studies that have indicated to us a number of the benefits of TILT. One of the first studies we did was the national study we ran with the Association of American Colleges and Universities. It was funded by TG Philanthropy and my colleagues working on that project were Tia Brown McNair and Ashley Finley. And what we did there was we worked with a group of seven minority-serving institutions from across the country that represented every possible type of minority-serving institution, as well as a range of educational contexts like urban and rural, two-year, four-year, research university, really small in scale, large, residential and non residential campuses because we wanted for teachers to look at our results and see, “Oh, well, this worked for those faculty at that institution, and there are students like my students in that mix, so maybe this would work for my students. And in that study, we started with 35 professors at seven schools and we surveyed about 1200 students and we saw that, for the students who received the more transparent instruction, their competence and their sense of belonging and their metacognitive awareness of the skills that they were developing, those increased, those were higher for the students who got more transparent instruction than for those who got less transparent instruction. And then we also saw in that study some differences that showed us that while all the students were benefiting to a statistically significant level, underserved students were benefitting slightly more. So first-generation students in their family to attend college and ethnically underrepresented students and low-income students have slightly larger benefits than the benefits for the whole group. And then in our second study, we focused on how long does this effect last. So we worked with a group of University of Nevada – Las Vegas students. At the time we were working with that study, University of Nevada – Las Vegas had the most diverse undergraduate student population in the nation, according to US News and World Report. And we know from other studies, like Walton and Cohen’s, 2011, Science Magazine article, for example, we know that when students’ confidence increases, when their sense of belonging increases, they tend to persist longer in a course. So in courses that have higher levels of confidence and belonging, fewer of the students would drop the course, for example, more of them more likely to complete the course. And we wanted to see how long does that last. Is it just that course? And some studies indicate that this could last for a year. And what we did was we kept looking at the retention rates of these students to see how many of them were still registered a semester later, a year later, two years later. And we saw that by the time students were in their third year of university as undergraduates, those students who had received transparent instruction in one of their large gateway intro courses in their first year, those students were a little bit more likely to be still registered as students in their third year. And we’re now tracking that out to six-year graduation rates. So we saw that not only does transparency have a beneficial effect, it’s statistically significant, but that effect lasts for a good long time. And then in the state of Washington, we’re now writing up that study I mentioned with the Community and Technical College System. And I think that TILT is particularly helpful in that environment, because the population of community colleges and technical colleges is a little bit more diverse. And we have more students who belong to that underserved category of students, first-generation, low income, ethnically underrepresented. And what we’re finding from that study is we’re understanding a little more about how does transparency work, and I want to thank all of the researchers who are contributing to all of these studies too, because I’m not an educational statistician, so Daniel Richard, and Carolyn Weisz and Kathryn Oleson are contributing to this study and doing a lot of the analysis, along with help from some graduate students who have been working on this project over the years. What they’re discovering is that transparent instruction has a direct impact on students’ awareness of the skills that they’re learning, and it has a direct impact, similarly beneficial, on students’ sense of belonging. And then separately, sense of belonging has a direct impact on students’ metacognitive awareness and skills that they’re developing. So TILT has this direct effect. And then there’s this other effect between belonging and skill development as well. So we’re finding out more about precisely how TILT works for the benefit of students in these studies. And I think in terms of next studies, I want to be asking questions that really matter to populations of faculty and students around the country. So we open up the TILT research team to anybody who’s curious about this, and a number of faculty have asked about, can we say something more about how this works in an online setting, in an online synchronous setting in an online asynchronous setting, and we’ve got a few publications up on the website about that, but others are looking at that a bit more. And then we have another person who’s looking into just the impact on low-income students to see if we can find out more there about the details of how this works. And I’m really curious to see if we can work with large state systems, what can we find about the most time efficient, most beneficial ways to apply transparency and learning and teaching in community college settings. And I’ve also noticed that as I begin to do more work internationally, because I now have more flexible time to be able to do that, the colleges of applied sciences, like in the European Union, for example, they have a kind of three-year degree that is similarly focused on students’ learning something from their degree like they do here in a community or technical college that will lead them on a path into sustainable long-term employment and a career. So I think that this is going to be a really beneficial place to focus TILT efforts and to do some more research about how can we long term have an impact on not just students’ education, but how that is a pathway into a career. And I’m hopeful that we can find out more about that, like the longer long-term effect of TILT. But I’m also really open to inviting anyone who wants to do more research with the mountains of data that we’re sitting on, to discover something that is of interest to them about how students are learning, and how we can help students succeed more.

Rebecca: I really love all the resources and examples and research materials, worksheets, that are on the website. They’re really handy for folks who are starting out. We always wrap up by asking: “What’s next?”

Mary-Ann: What’s next for me, and then what might be next for teachers and students too. So we’ve talked a lot in detail about how TILT works, and how we know it works, and what more we want to discover about how it works. But I want people to remember that this is really a small effort, it’s a very easy lift that has a really large benefit from the size of that lift. And so I would really encourage teachers and students, if they’re going to do anything at all, even if they have no time to adjust any assignment prompts or to adjust anything about the way that they’re teaching or learning in a classroom. If you use any one single thing, I would say use that framework that we built for the students that has the footnotes at the bottom, and it’s called the “unwritten rules” and that framework, and I think you could probably provide a link to it, that’s what I would hope people would do next, just take that framework with you to anywhere that you’re communicating with your students. And the students will tell you how to make the work more transparent for them. Ask students what they see as the purpose, the task, and the criteria. And you’ll discover very quickly, very efficiently, how you can make that work more transparent so that all students are starting to do the work with the same understanding about what’s expected and with the same set of resources that they need in order to do it. So that’s what I hope is next for teachers and students.

Rebecca: And I hear all the faculty cheering about efficiency, and quick. [LAUGHTER]

Mary-Ann: That’s good. Yeah. So that would be the most time efficient thing to do, I think is to have students teach us more about how to be more transparent. And then in terms of researchers, I’m hoping that researchers will think about what can we learn more about? Can we learn more about what motivates students? Or what forms students’ sense of belonging? Is there anything in our survey data that would shed light on any kind of work you’re doing around that? Is there anything in our survey data that would shed light on more of the research on neuroscience and how that’s impacting learning? Or is there anything in the research that we have in our survey data that might help clarify what would be most beneficial for the very most at-risk students? So if we look at federal government statistics, National Center for Education Statistics about retention rates and graduation rates of different populations of students? Can we double down and look at those students with the very lowest graduation rates? And can we find something about TILT that would be the most beneficial for that population of students? To me, that’s a really important and interesting question. And then I really do want to be finding more locations where TILT could be useful, small scale for teachers and students, large scale for state systems or national systems to be thinking about how to apply this all for the good of students success, and for the satisfaction and time efficiency for teachers work as well.

John:If you’re finding these results of long-term persistent effects from just a single intro course, imagine what would happen if all intro courses use the TILT approach. I imagine the effect would be magnified if it was adopted at a broader level and it is being adopted at many institutions at a broader level.

Mary-Ann: I absolutely agree with you that applying TILT across the largest introductory gateway required courses at any institution would be probably the most efficient way to improve retention and graduation rates. Because if you go for the largest group of students as they enter, and you reduce the number of those students who might be thinking or doubting or wondering if they should continue, and if you increase the number of students who feel confident, who are aware of the value of what they’re learning, in terms of skills and knowledge, and if you increase the number of students who persist from the first year on, then that’s where you’re going to have the best success in increasing retention and graduation rates. I agree with you. I think that’s a really strategically wise place to invest TILT effort.

Rebecca: Well thank you so much. We’re looking forward to sharing this with our listeners.

Mary-Ann: And thank you so much for the opportunity to talk with you this afternoon, I really appreciate it

John:Thank you for all the work you’re doing.

[MUSIC]

John:If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

289. The Cognition-Motivation Connection

Emotions can have both positive and negative impacts on learning. In this episode, Michelle Miller joins us to explore the relationships that exist between emotions and learning.

Michelle is a Professor of Psychological Sciences and President’s Distinguished Teaching Fellow at Northern Arizona University.  She is the author of Minds Online: Teaching Effectively with Technology and Remembering and Forgetting in the Age of Technology: Teaching, Learning, and the Science of Memory in a Wired World. Michelle is also a co-editor, with James Lang, of the superb West Virginia University Press series on teaching and learning.

Show Notes

  • Miller, Michelle (2023). “Revisiting the cognition-motivation connection: What the latest research says about engaging students in the work of learning.”  March 3.
  • Miller, Michelle (2022). “Ungrading Light: 4 Simple Ways to Ease the Spotlight off Points.” The Chronicle of Higher Education. August 2.
  • Remind
  • Transparency in Teaching and Learning (TILT)
  • Sathy, V., & Hogan, K. A. (2022). Inclusive teaching: Strategies for promoting equity in the college classroom. West Virginia University Press.
  • Ariely, D., & Wertenbroch, K. (2002). Procrastination, Deadlines, and Performance: Self-Control by Precommitment. Psychological Science, 13(3), 219–224.
  • Abel, M., & Bäuml, K. H. T. (2020). Would you like to learn more? Retrieval practice plus feedback can increase motivation to keep on studying. Cognition, 201, 104316.
  • McDaniel, M. A., & Einstein, G. O. (2020). Training learning strategies to promote self-regulation and transfer: The knowledge, belief, commitment, and planning framework. Perspectives on Psychological Science, 15(6), 1363-1381.
  • Miller, Michelle (2019). Attention Matters. Tea for Teaching Podcast. Episode 86. June 19.
  • Michelle Miller’s R3 Newsletter
  • Sarah Rose Cavanagh’s Once More, With Feeling substack
  • Cavanagh, S. R. (2016). The Spark of Learning: Energizing the college classroom with the science of emotion. West Virginia University Press.
  • Cavanagh, S. R. (2023). Mind over Monsters: Supporting Youth Mental Health with Compassionate Challenge. Beacon Press.

Transcript

John: Emotions can have both positive and negative impacts on learning. In this episode, we explore the relationships that exist between emotions and learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Michelle Miller. Michelle is a Professor of Psychological Sciences and President’s Distinguished Teaching Fellow at Northern Arizona University. She is the author of Minds Online: Teaching Effectively with Technology and Remembering and Forgetting in the Age of Technology: Teaching, Learning, and the Science of Memory in a Wired World. Michelle is also a co-editor, with James Lang, of the superb West Virginia University Press series on teaching and learning. Welcome back, Michelle.

Michelle: Oh, thanks for having me. It’s great to be here today.

Rebecca: Today’s teas are:… Michelle, do you have some tea?

Michelle: Well, not exactly. I’ve started hydrating with fruity water today. So, I’ve got my water jug and I’m working on it.

John: And I have just a little bit of a peppermint-spearmint tea blend here. And the reason is just a little bit as this is our third podcast of the day today. So I didn’t have a chance in between them to go back to my office and get some new tea or some new hot water. So I do have a little bit to get us started here.

Rebecca: A tiny bit left from my pot of blue sapphire tea.

John: …which is much more colorful.

Rebecca: It is, but not in my cup, only in the pot.

John: We’ve invited you here to discuss your March 3 blog post, addressing the relationship between cognition and emotion. In general, how is cognition influenced by emotion,

Michelle: I’ve been interested in this connection for a while and watching the evolution from within my field of cognitive psychology and kind of moving away from the approach that I came up with when I was just starting out as a graduate student, which is I recall was this kind of oil and water conception of cognition and emotion that here on the one side, we’ve got thought processes, we’ve got memory, and so on. And on the other side [LAUGHTER], we’ve got the emotions and so on. And we’re just going to really work from in our subfield to try to get our arms around just these cognitive processes, and don’t worry about the rest of it. And now, I think that most cognitive psychology theorists in the field would say that, yeah, our cognitive processes are definitely shaped by and infused by what’s going on kind of over in the emotional processing systems of our mind and in our brain. And if I had to describe, just from my own perspective, what I see is a change over time and an evolution in our field, we’ve kind of gone from really talking about parts of the mind in this very compartmentalized modular way, where different parts do different things pretty much independently. And now you see more discussion of how these different parts have interplay with each other, how they give what I would think of as a sort of a soft input to other subsystems, or even set some constraints on what those other systems are doing without totally determining them. So I think we are moving into this more nuanced view of how those two things work together. So that, yeah, our emotions affect what we believe, they also serve as a way to almost elevate or suppress different aspects of what we’re processing so we might remember things in a particular way, or think about them in a particular way. And it’s neat to me too, as somebody in the field, because I look and I see clinical psychologists, the people in the area of psychology who work on how do we help people in therapy and help people with different disorders and challenges. They’ve known this for quite some time, but they’ve looked at it sort of in reverse. So if you’ve ever heard the school of thought known as cognitive therapy, or cognitive behavioral therapy, one of the core tenets of that approach is that the emotional side of how we function, our emotions, are affected by our cognition. So what we feel, even our mood states and so on, that’s fundamentally driven by things like what we believe. And so they’ve come up with these really exciting and powerful techniques for addressing beliefs that people have and thereby affecting their emotions. So we can take a cue from that and have this more nuanced view of the interplay. So back to cognitive psychology. I also come at this really philosophically as what we would call a functionalist, [LAUGHTER] that’s sort of a lens through which I see how we address questions in psychology. So when we say, “Well, why does the mind work in this particular way? Why does it have this component or why does it do this in this way?” I would look at and say, “Well, how does that help us survive? What’s the function for helping us really survive and thrive in our world?” And when we look at things like emotions, our emotions are there for very functional reasons. I believe our emotions exist in order to kind of move us towards things that help us in our survival and move us away from things that are going to be a threat to our survival. And also they serve in this way to kind of alert us to what’s relevant. So it’s almost like a relevance mechanism. So if something provokes an emotional response in us, that may be an old shortcut that our mind has to say, “Yes, this is something that’s important. This is something that maybe you want to remember and that you want to pay attention to.” So I see emotions as kind of a feel for relevance, and that’s something I’m sure we’ll get into in our conversation about teaching and learning. later. And all of this is a practical issue too. I tell the story sometimes about Minds Online, and writing that book, where I got to about midway through the book and literally had this crisis, I remembered it happening like in the middle of the night, there’s something huge that’s missing right around this point in the book. And that book, for those who have taken a look at it, it takes a very cognitive view of how we select and use technologies. But I came to this realization, we can’t really talk about how to maximize the effectiveness of those approaches, unless we also talk about why students are going to do them in the first place, and how we can get them motivated to do them. So, in that book, I ended up covering some very basic elementary foundational concepts in motivation and motivation theory with that idea of what are just the essentials that every teacher needs to know and how might that also get involved in how do we choose certain technologies? How do we set things up in a particular way, for example, in an online course to keep students moving and that keep them putting in that productive effort. And so that’s been around in the back of my mind for quite some time. But now I’m reading all these new articles and this wave of interesting new research that is finding yet more connections between those two sides of the mind. And so how to get students to engage in strategies that work from a cognitive perspective and how to direct that feel for relevance. And early on in my career, as I mentioned in that blog post, I look back and it seems so harsh now, like, well, how do we get students to be accountable. And now I’ve kind of shifted that along with many others towards really looking more at the support side of this and bringing in things like empathy for our students, I don’t think I’ve ever been one of those super punitive “look to your left, look to your right” kinds of teachers, nor have I ever really advised that to their faculty. But I’m realizing that in this really critical case that I’m looking at this relationship in new ways, and I’m excited to share that.

Rebecca: So there’s been a lot of discussion about student motivation and engagement…, a crisis in it. [LAUGHTER] A lot of faculty have reported students being less engaged or less motivated. How can we, as faculty, address some of the challenges that people are experiencing at this moment?

Michelle: And it is such a pressing question, and that’s another thing that’s just really been registering as I’ve had my antenna up about what are people talking about right now? What are they bringing into conferences, and so on? And first off, as a little bit of a skeptic, I have to say, “Well, I think that we still need some more information to nail down exactly what the extent and the nature of the engagement crisis is.” And I think all three of us are attuned to what I guess you can call the fallacy of “students these days.” [LAUGHTER] So as so many people have observed, it’s so tempting to have that filter on of like, “Well, back when I was a student, I was always intrinsically engaged in my classes, I didn’t miss assignments, and so on and there’s a downward trend.” So there’s something about that. I put on my skeptic filter when I see that. But that said, we do have these experiences. And I don’t think regardless, even if we look back and say, “Well, maybe this wasn’t really part of a bigger trend as we thought.” Even if that were to happen, are we gonna look back and say, “Well, we shouldn’t have worried so much about engaging our students, we can almost always stand to engage them more.” So with that big caveat, I think that we should also be really reflecting on and separating out, as much as it’s really possible to do so, disengagement from other related things like prioritizing. I don’t have the capacity to cover all that I need to as a student, and perhaps also as a family member, a parent, a worker, and so on. So here’s how I’m going to go at it, or even just straight up overwhelm, and I think we can look at that from our own perspective, too, and say, “Well, right.” I think we’ve also seen quite a few faculty professional development directors and others who work with other faculty to say, “Oh my gosh, I put up a half a dozen workshops, and I’m having trouble filling them. So we too, as a lot of our demands have converged over the last couple of years, and as we’ve coped with those stresses, we too. It’s not that we’re disengaged from what we’re doing, but we’re having to make some different choices out of necessity. We have the economic costs of college and that whole dynamic that’s going on as well. I’m no expert in that. But I think we all know that students today are working more jobs, succeeding at every single course and getting through as quickly as possible is an economic necessity and so on. So the stakes are very, very high for students, and students are dealing with that. And so that’s one also very important thing to think about when we’re looking at this. So with that, though, have students been more disengaged? I mean, my experience immediately coming back to in-person teaching, I found myself that students were really excited. At the risk of sounding very strange here, it was like a box of excited puppies: Oh my goodness, we’re all here in the classroom together. And I felt the same way in some ways too. But really directing that in some, again, productive ways is what we have to do as the leaders of our classes. Now to practical tips for what can we do. If there’s a disengagement, students are elsewhere, they’re not doing the work, or they’re all excited but they don’t know how to manage that. But here’s a couple things that I think are very practical. So I’m a big advocate these days for flexibility and approachability in what we do. So I wrote a piece last year titled, “Ungrading Light…” I think that was the catchphrase in the title… which talks about “Okay, without sort of throwing out grades and say, ‘Well, students have the wrong motivation when it’s all about the grades.’” If we’re still going to have grades, what are some positive ways to keep students really focused on the learning and engaged with the learning and not just like, checkpoint, checkpoint, how do I get through this? And I do think that even some basic changes to policy can help here. So things like I really have gotten very flexible on deadlines. The caution here that this is going to look very different for people with different course sizes, section sizes, different disciplines, what the learning objectives really are in your course. So I don’t want to imply that everybody just can do this in the same way. And as I also mentioned in the piece, things like very flexible deadline policies can present a professional risk for people who do not have the security of, for example, tenure, and people who are historically minoritized, and are going to elicit different kinds of reactions from students to play out on things like end-of-semester evaluations. So for example, faculty of color. So with those big cautions in mind, now, here’s been my experience is I communicate with students… I say, “I want to be approachable,” I want to really show them and not just tell them that if you come to me and say, “I was pulling double shifts all weekend, and I need to do this paper draft, and I know that, but I need another two days,” that I’m not going to come down on them in a harder, personal way. And if they do that, just not all the time, I will say, “Yeah,” and then my catchphrase right now is, “Take the time, you need to do your best work.” And that turns out actually to be really good for my motivation, too, because I would really rather read what they put together [LAUGHTER] with a little bit of more time, that’s all about I want to do something I can be proud of in this course, and actually walk away with great knowledge. It’s more geared towards that and less geared towards “Oh my gosh, didn’t come in until 11:58 when it’s due at midnight, and I sort of just checked that box again. So that is something as well and other ways to be approachable, that can also elicit more engagement. How do we know students are engaged? Well, when they reach out to us. And so, here again, different individuals have to decide their appropriate comfort level and parameters. But I have a syllabus statement that says here are all the different ways to get in touch with me. If you’ve got a long question, and we need to talk, I’ve got a scheduling program, you click a button, and boom, now you have access to my calendar, and you can get on my calendar the same way my colleagues can. And that’s good. If you have a really quick question like, “Oh my gosh, there’s one thing I need to do in order to finish this assignment and actually be successful, then you can text me or send me a message in a program such as Remind, which can kind of buffer so we’re not trading phone numbers. But, that immediacy, it has not really resulted in this giant pile up of lots of inappropriate communications, which is what I was always warned about when I was coming up as a teacher. But instead, students get the question answered and then they can kind of stay engaged with the flow of what they’re doing. So just basic ideas, but ones that I think can help move us back towards a more engaged setting where students are excited to be there and so am I. If I could add one other thing here, too, we can also take a page out of the transparency philosophy. So if you’re familiar with Transparency in Teaching and Learning, the TILT framework, it’s so powerful. And it’s all about giving more explicit directions to students, as well. What you may read as disengagement or not caring, might be “I don’t know where to start and now, I really am feeling either alienated, overwhelmed, or something in between.” And I think that’s another we can all relate to is, we’re a lot more likely to take the first step down the path and keep going if that first step is lit up, or maybe if the whole path is lit up. So taking that little bit of extra time to say “And here’s where to start and if you get stuck, here’s what to do.” That can also help.

John: I’ve been observing the same sort of issue that many people have reported of students not completing work. I’ve seen students being much more excited to be back in the classroom, and they tend to be fairly engaged in classroom activity. But what I’ve been seeing and what a lot of faculty have been saying is that students aren’t doing the work outside of class at the same rates that they used to do. And one of the concerns in terms of making your classes more flexible, in some cases, you can do that really well and I do allow that with many of the assignments. But in classes where the material builds from week to week, if students start getting behind early in the semester, they’re going to be struggling a lot more later. So I have different policies depending on whether they’re producing something, some type of educational project… a podcast or something similar… as they do in some of my classes, then they can have more time. And I give them as much time as they need to do that with multiple iterations. But with other things like reading the materials online, where there’s some embedded questions, and so forth, there, I do insist that they get it done by a certain time, because then when they come into class and they’re asked clicker questions, some of which they’re graded on, they’re not going to be successful in that if they haven’t done the basic reading. And that’s where I’m seeing a lack of engagement, outside of class. I’ve had many fewer students complete the readings before class, or even weeks after they were due, they’re still not completing some of those readings. And that’s the concern that I’m having. And I have to say that I’ve also observed some of this with faculty too, that attendance at professional development workshops have been a lot lower this year than in the last couple of years. And some of it may be because of burnout after the pandemic. And I should note that on my campus, we’re also transitioning to a new learning management system. And a lot of people have been struggling with that, which takes up a lot of their time, reducing the amount of time they have to learn other new things while they’re struggling to learn the new system. But this issue of engagement does seem to be impacting the amount of learning that I’m seeing, at least in my large intro classes, I’m not seeing it so much in my upper-level classes. But I’m wondering if some of this may be because we have students who’ve spent a year or two with remote learning in schools that often had very few resources to do that, well and students may have just gotten out of practice with doing a lot of work, because in many school systems, students were just passed on to the next grade level without necessarily learning very much in many classes.

Michelle: Yeah, I’ve seen this dynamic, actually even at my upper levels as well became rather glaring the first time that we went back to an in person symposium, it was the kind of capstone experience in this class was to bring some research to the symposium. It’s a wonderful experience, but it dawned on me partway through the semester to step back and just say, “Okay, how many of you have never done a presentation like this before?” And yeah, previously, most of the class would have had some experience either in an in-person research lab that they were in, or in a methods class or something like that. It was one of those head slapping moments, at least for me, feeling, “Of course, of course they don’t know.” And I try to come at it like, “Well, this is the time to do what I probably should have always done for what was previously a small group of students. But it’s still an important group of students who are sitting in the back going, ‘Oh, my gosh, I feel lost. I don’t want to even raise my hand. I don’t know what she’s talking about with the poster or participation or even things like what to wear.’” And so I did, I went back and dusted off and created a few stopgap materials. I found some things out on the web that actually demonstrated a poster presentation that was in progress and what to do and not to do. So it can be an opportunity to do more of that transparency and kind of scaffolding and bringing everybody up. But yeah, it can be shocking to stand back and say, “Okay, who has not actually done this thing that I kind of always assumed would be the case by this level,” regardless of what that is.

Rebecca: I’ve experienced this, even with graduate students, this lack of knowledge of certain kinds of academic experiences, in part because they were learning online or not doing things in person, and now they’re in in-person classes and having any in-person experiences. So I had the same experience, Michelle, but with graduate students, and needing to really build in some transparency there that maybe didn’t need to be there before because in their undergraduate experience, they were very likely to have had a similar experience.

John: And as you said, Michelle, giving students more structure and more support is something that we probably should be doing anyway. We just finished a reading group on Viji Sathy and Kelly Hogan’s book on inclusive teaching. And that’s the message at the heart of that book: that giving students support will help all students at least some and will especially help those students who come from backgrounds that provide less preparation for success in college. So to the extent to which we as faculty all learn that lesson, that giving students more support is useful, it’ll be a better environment.

Michelle: I agree and what seems implicit in that., how you’ve put that too is, instead of like, “Oh my gosh, another thing I have to cram into the semester” …for our motivation and our engagement is to say, this is part of one of the most noble pursuits that we can have as educators, to give it that meaningful frame. So yes, a hearty I agree with that book, in particular, and their framework. And for me, that helps me kind of say, like, “Okay, yeah, this is not just an extra add on, this is what we’re here for.” And if I’m trimming back a few extra articles, or chapters, and I have done that, to some extent, in favor of being able to go more deep and into content, and be more supportive in these positive ways, I think that’s a win.

Rebecca: I think one of the things that I’ve noticed or experienced recently with students is high engagement in class, high engagement in the subject matter, but we’ve had really interesting conversations about procrastination or not doing things outside of class, largely due to a lack of confidence, or striving for perfection that doesn’t exist. And there’s a lot of that emotion around that. And so a lot of my students have talked about that, or shared that with me, which I’m grateful that they’ve shared that with me. But that’s what’s preventing them from getting started.

Michelle: Absolutely. And I think folks who follow the research on procrastination out there, it’s not as much in my specialty area, but I do think it’s fascinating. And it’s another one of these touch points between what we believe, what we feel, what we’re motivated to do, and then in turn, what we remember and what we learn. So I think that for people who are interested in this whole topic of procrastination, why does it happen? What are some really good ways to talk about it, and address it, there’s new stuff coming out. And it’s a good thing to talk about with students as well, I think years back, it was almost a taboo subject. But now from what I hear you saying, you just bring it up with students, and we can all talk about it not as like, “Oh, that’s some terrible thing that other people, bad students, are doing. This is all of us, right? [LAUGHTER] We live in a world of abundance, but also abundant distractions, and so many things competing for our time. So I like this idea of opening up that line of communication, saying, “What do we all do to tackle this when it occurs?”

John: Dan Ariely had a paper a number of years ago, where he did an experiment in class, I think it was an economics class, actually, where students wrote papers. And he and some co authors had two sections of the class, where in one section, students had three papers with fixed due dates spread evenly through the semester. And in the other section, students were able to pick their own due dates. And there was a penalty in either case of one percentage point a day for each day the work was submitted late. And what was found in that study is that the students did best who either had fixed due dates, they had higher quality papers, and higher quality work, and so forth, they wrote more, and the quality of the work was much better when they had fixed due dates, or when they chose evenly spaced due dates. From an economist’s perspective, the rational thing to do would be to put all three due dates on the last day of class, because then you could still plan to do it evenly throughout the semester, but you would have no cost of doing that. So if something came up, you could postpone it. But what happened is the people who put all their due dates at the end of the semester ended up procrastinating, turning in work later, the papers were shorter, they were lower quality, and in general, they didn’t do quite as well. So that’s one study, I often will cite to students when we talk about due dates and deadlines, and so forth. But it’s an interesting study. And I haven’t seen anything else in economics journals, at least, related to that, but I’m sure there’s more that I haven’t seen in the literature.

Michelle: Fascinating stuff from across the disciplines.

John: One of the things you talk about in your blog post is that the strategies that students use for learning are not the strategies that evidence tells us are most effective. Students tend to use strategies that provide some short-term benefit, and seem to be easier, rather than the strategies that require them to struggle a little bit more with the content. One of the things you talk about are some ways that we could encourage students to adopt strategies that may not feel quite as good in the short run, but result in more learning. How can we motivate students to use evidence-based learning strategies?

Rebecca: …motivate them to struggle? [LAUGHTER]

Michelle: Right, but that is really what we’re talking about here. And I do want to go into this… big qualification here… I don’t think that students are just out there wanting to get the best grade and for the least effort necessarily, that’s just not a narrative about students that I buy into. So, I don’t think students are trying to do low-effort strategies. But, just like the rest of us, we don’t have a very good or accurate view always, a very empirical view, of what actually pays off in terms of learning. It’s pretty rare that we sit down and actually kind of do the math and say, “Well, I did this, I systematically changed the approach in this way, and here’s the outcome.” So we don’t come at it that way. So no wonder that over time, we end up with kind of a distorted view of what actually does work. So that’s a big piece of it. And it is true at the same time that these strategies we’re talking about… well, let’s take one, for example, of blocked study. Now this is a term that I also want to unpack a little bit too, it’s not super intuitive. So this has to do with the principle of interleaving, which I always say it doesn’t always apply in all studying, but to cases where you’re learning how to apply different problem solving strategies and you have to choose from several when you’re having to categorize and learn to discriminate among categories. So that is a subset of what students are sometimes learning. And the thing is, we have this great powerful line of research that shows that actually mixing it up in an unpredictable way, the different problem types or category types, means it’s going to be a lot more memorable when you actually work through those practice problems or practice sets. And if that’s the case, the unpredictability of like, well, something’s gonna pop up categorizing different painting styles, I have no idea [LAUGHTER] if it’s going to be a Renoir or a Monet, what could be next? It’s that unpredictability. So people sometimes confuse it with just like mixing up topics or having variety, but it really refers to the systematic principle. Now, when students are offered the opportunity to structure their own study, what do you know, they tend to go with blocked study, and again, it’s not because there’s some dispositional factor, they don’t want to do their homework or something like that. Really when you look at it intuitively, it’s like an illusion, block study feels so effective. I’m going to work through all of this one painter or all of this one way of solving a statistics problem. And then I’ll go into the next and our textbooks are organized that way, too. So students have seen that, and so that’s what they fall back on. And there’s some recent work that I’ve talked about in that blog post and in a few other places, that has really studied in a very granular fashion… it’s presented students with different alternatives, like here’s a blocked study schedule. Here’s an interleaved one. We don’t use the technical terms, we show them both options, and say, “Okay, let’s pretend you have a math test coming up. Which one of these do you want to do? And why?” And yeah, students, they gravitate towards the blocked one. And they say, I perceive that this is going to be easier, first of all, and that’s fine. We want to use the most efficient strategy. So they say, this is gonna look easy and also, it feels more effective. Because I feel like, “Oh, yeah, I’ve got it.” But as we all know, sometimes that’s a false sense of security. So that’s the example as it lays out in that one case, and I think that that is a larger kind of big dynamic, that we do have to be aware of what looks easy, what feels easy, what looks effective, what feels effective. Sometimes, your brain is kind of playing tricks on you. And that becomes very serious when it is the case that things like interleaved study are more effortful, but they’re going to pay off more for the time invested. Retrieval practice, which I and a lot of other folks in the space talk about so much, that’s another that it’s gonna require a different level of effort and engagement to close my book and say, “Alright, instead of rereading this chapter, what did I actually get out of it, or maybe I can seek out a quiz.” And to me, I also think it’s not just the effort involved, and research, by the way, it’s also showing students also to look at this and go, “that looks difficult.” It also kind of emotionally, I mean, I was feeling okay about this chapter, and now, I can’t really kid myself any more. So to the extent that students might be kind of saying, “Yeah, yeah, I’ve got it, I’ve got it.” …like, we might all do this, this will kind of bust that unjustified optimism, and that doesn’t always feel great in the short term. So if that’s the case, I mean, we can set up these wonderful learning activities, and if students aren’t going to do them, then they’re not doing them or us any good or any benefit. So, that’s the case as well. And so if they have a sense of the value, and a lot of these strategies that maybe we can touch on, do have to do with exposing and revealing and convincing about the value, and finding ways to draw students into exactly those techniques. So just because these are difficult, I do want to make sure everybody doesn’t get this terrible impression of like, “Ah, studying is going to be this miserable slog, no pain, no gain,” …it’s more subtle than that. And they really do work, you really are gonna get so much more out of the time that you put in and for students who really are stretched really thin as we’ve touched on. That’s an important powerful message.

Rebecca: You mentioned a number of specific examples in your post, do oyu want to dive into some of those and share something like the snowball effect or self-determination theory, or some of your other really awesome examples?

Michelle: Oh, thank you, I appreciate that. And, after all, the big philosophy and approach matters, but let’s get down to the actual techniques. So I’ve referenced something called the “snowball effect,” and this is just my informal term, but really, the more you know, the more that you want to know. And the more that you know, because of the way memory works, the more you know about a particular domain or subject area, the easier it is to acquire new facts. And like I always throw out the example of folks we know who are just really committed to some hobby or area of interest, the sports fanatics and so on, they can run into a fact one time and boom, they’ve got it, they’ve maybe got it for life, and they don’t have to study [LAUGHTER] or do anything like that. And so there is that snowball, or rich get richer effect, just because of some factors about how memory is set up and how it works. From a very practical angle, like I ran into this really intriguing study. And it’s not one where we’ve got piles of research yet, but this really got me thinking. So they did a study where they had students through retrieval practice, learn some basic facts about an area that they picked. And they were able to systematically track that when students did learn this foundational information more solidly, then when they had the option, “Oh, would you like to know more about this subject?” …students were more likely to say yes. And that’s totally voluntary. And that’s the sort of thing that makes our hearts go pitter pat, as teachers we want students leaving and going, “Oh, my gosh, now I really want to read that next reading that Dr. Kane assigned, for example.” And like when you were talking about, students are coming into class, and we’re trying to get them into the next level of conceptual stuff and exciting things they can do, if they don’t have the facts, it’s gonna be really, really tough, so it also really points up the importance of doing that. And it also, I think, addresses one of these big myths about memory. And this is one that I’ve talked about in some of my recent workshops, and so on, and I mentioned in my last book, this big myth that if we do focus on having students concentrate on remembering foundational information, we’re going to turn them off of learning: “Oh, it’s going to be this sort of these nightmare of drills.” And, “sure, they’ll know it for the test, but they’ll walk out and they’ll never want to be engaged with the subject again. So it’s a big loss, [LAUGHTER] right?” And this is really calling that into question, saying that sometimes knowing some of these cool initial facts can start to set you down that path and then maybe someday, you will be that expert who can hear a fact one time in this area and we’ve got it. Why? Because we already know so much about it. So again, that whole Interplay there. There’s also the role of choice and autonomy. And this is one that I think a lot of really intuitive, committed, teachers really hit on early, even if they never really have some of the more formal terminology for it. So when there’s choice, not only are students more invested in what they’re doing, there’s possibly a role of curiosity here. So I talk about, in this blog post, this sham lottery study, it was one of these, where if you look at it on the face of it, you’re like, “What are they doing?” But as a psychologist, I’m like, “Ah, that’s really clever.” So basically, they had research participants going through this little pretend lottery of like, “Okay, you’re selecting out of this bucket of red balls, and so on. And what do you think it’s going to come up?” And the one twist that they put in there, is sometimes people chose which of these two little buckets, there are these little random drawings, that they were going to focus on? And then it’s like, okay, well, we can either just move on with the study, or you can see how it came out? Well, they want to know how it came out when they chose… even this incredibly arbitrary [LAUGHTER] low-stakes situation. So I think that’s also another kind of natural, emotional process, motivational process, that we can tie into… setting up curiosities or questions, but also having students say, “Well, which of these two projects do you want to do?” These days I offer options whenever I can. Would you like to write a term paper? The sort of formal paper? Or would you like to put together a slideshow that you can narrate and share? Big learning objectives are probably similar, but students can pick and I always present in practical terms, I say, “Well, if you are going to graduate school next year, and you need a writing sample for your portfolio, this is a great opportunity to do that orr if you’d like to stretch your skills with oral presentations, maybe because of the last few years, you haven’t gotten to do that as much, then you can choose this,” …but simply by having them make a choice, the research would predict that they are going to be more invested, and they’re going to be involved in these more effective things. So that’s one and oh, I’m really excited to see what’s coming out in this whole sub area of “Okay, we’ve done all the research we know that things like retrieval practice and interleaved study, all this engaged stuff. We know it works. We put it on a tip sheet, we gave it to students, nothing happened.” Uh oh.[LAUGHTER] Now what? So not just the like, “Okay, what should students be doing?” But “how do we get them to do those very things?” And, boy, if there was ever a time when we realize, yeah, my ability to just sort of exhort you and make you do things because I say so is limited, this is the time. Because I don’t get to go home with students [LAUGHTER] and say like, “Alright, that whole thing about quiz yourself. And so now you really have to do it.” So there’s this relatively new framework that’s come out too from some cognitive psychologists that I really admire, Mark McDaniels and his team. The knowledge, belief, commitment, and planning approach, KBCP. So this pulls in from some other research on intentional behavior change that’s also been perking along just all on its own for years and years. We know so much about how people set a course and decide to change their behaviors. And study skills are, after all, kind of an entrenched pattern of behavior for many students by the time they get to us. How do we go in and change it? And yeah, it’s absolutely not through my least favorite technique, which is put together a list of random tips and hope for the best. So they say, alright, knowledge is the first step. So just telling students like, “Hey, there’s all this research that shows that if you close the book and quiz yourself, you’re going to get more out of this. If you do the reading quiz that I set up for you, and do it as many times as you possibly can, that’s going to help you retain the foundational information.” I’ve told them, I’ve shared it with them, or something like interleaved study, if you’ve got different problem types, mix them up. But it doesn’t stop there. That’s only the first step. So the next step is belief. And that means changing beliefs, which means persuasion. So we kind of dust off a whole bunch of other things out of the psychologist’s toolkit. How do you persuade people? Well, you show, don’t tell. So this team proposes doing things like “Well, let’s run a head-to-head comparison, like a Pepsi Challenge, in class. Sure, your brain tells you that you learned a ton just from reviewing, but did you? Let’s try it.” And this takes some time. I mean, this is not easy. But this is one of the things they propose: commitment. So now that I’ve persuaded you that this is the way to study, now, what’s your next step? So getting your students to say, “Yeah, I actually authentically believe this. And I see how it’s going to help me and I’m going to try it.” And then of course planning. So instead of just like, “I will do this,” right? Those of us who are veterans of New Year’s resolutions of yor [LAUGHTER] know that that is not the way to go. So yeah, saying “Okay, but here’s what I’m going to actually do. So I’ve got a test coming up, I’m going to maybe set up a study schedule, instead of just cramming it all in the last minute, which is [LAUGHTER] a really good empirically grounded strategy. I’m going to find these practice quizzes, or maybe I’ll get together with a study group and do that. So here’s my plan.” And then if possible, circling back and say, “Well, did it work.?” And hey, if we’re right, then students will actually try it, they’ll say, “Wow, in less time, I knocked the top out of this test that I was really worried about.” And that is going to feed that virtuous cycle of going right back to those effective strategies. So KPCB, I love it too, because I’ve been doing something similar in the Attention Matters Project that I think I’ve talked about on some previous episodes as well, which is all about having students themselves come in and see how their attention is limited, learn about the effects of things like distraction on their learning. But we don’t stop there. We give them a few rudimentary tools as well, we say, “Okay, what is going to be your plan if things are dragging in class and your mom is texting you? That’s tough. How are you going to get through [LAUGHTER] that without then checking out of your class? What are you going to do if your neighbors are watching who knows what on their laptop or they’re texting and it’s bothering you? What is going to be your plan?” So getting students to really think ahead to those things, commit to doing them in a way that works for them, and puts that newfound knowledge into practice. So those are some of the things that I’m really experimenting with and excited about right now.

Rebecca: So in that approach, it seems really necessary to help set up a structure for students and then circle back and have a reflective piece so that maybe they will do that on their own next time.

Michelle: And there’s some exciting suggestions from research here, too. I mean, I know it’s easy sometimes as faculty, especially at the end of a long year, like this one, to say, “Ah, well, did it actually stick with them?” But there’s a couple of different projects out there that have kind of converging on this idea that once students really do see something like retrieval practice, active studying, and so on, and once they really experienced that, as part of the structure of one course, they absolutely will run with it. So they will go into the next class, whether in your discipline or not, and say, “Well, from now on, I’m actually going to have a study plan that’s set up in this particular way and I’m going to do this.” So I personally find that very, very encouraging that “Yeah, it takes some work to do this stuff, but the payoff, even if you personally don’t see it right in front of your eyes, the payoff is likely there.”

John: And so the more faculty you start doing these things too, the more likely it is that students will adopt new approaches. So spreading this more widely is helpful.

Michelle: Yes, yes, a hearty I agree to that statement. And I can test on my own campus, I’ve seen more faculty bringing in more structure, things like online reading quizzes, I have noticed that, so I guess that’s a counterpoint to the “Wow, my lived experience is telling me that there’s these issues in engagement. Maybe so, but my lived experience is also telling me that students are coming to me more ready to be proactive about their study, they need a little less persuasion to do things like reading quizzes, because they at least they’ve seen them before. So yeah, I think it absolutely can work that way you’re describing.

Rebecca: So we always wrap up by asking, what’s next?

Michelle: Wow, well, these days, I’m working a lot on my substack newsletter, it’s called the R3 Newsletter. And I think this is the mechanism by which we connected on some of these new topics. So I love that it’s already starting these great dialogues. And if you haven’t seen substack at all, it’s a bit of a blogging platform. And my substack is free, some subtasks are paid, but mine is definitely free. And so, for example, if you’re interested in this topic, I would definitely tell your listeners to check out Sarah Rose Cavanagh’s substack, as well, it’s called “Once More, with Feeling” and I also want to affirm that she is and her work are really at the forefront of this whole topic of motivation and emotion and particular in learning so a great other substack to follow it and buy her book, Spark of Learning is also just an absolute modern classic in this. So I decided to get in the fray since I saw these wonderful thinkers around me also doing this. And this has been a really good platform, and a way to structure for myself something that I felt needed a refresh, which was my reading of the literature that’s coming out. So what I do is twice a month, approximately, I’ve been putting out discussions of research that I’m reading. My little heuristic is anything that was published in approximately the last year, really privileging the new stuff. And I’ve historically just seen that then when I really get into the nitty gritty of the research and what it says and what it doesn’t and parsing that for folks, especially those who may be outside of social sciences, that’s where I get the most affirmation from folks and people saying, “Yeah, this was really helpful.” So I decided to run with that. And so that’s a big project ight now. I’ve been really happy with the reception and just working on that. I’ve been writing and thinking more about this topic of motivation and cognition. So as we mentioned at the top of our conversation, it’s one that goes back to kind of my initial ponderings, thrashing around as a beginning graduate student of like, “How does this all work?” Coming back to that, and really finding new ways that I can share that with my fellow faculty. So getting the word out there. I have a few new projects that I’m working on that tie back to that attention issue. So that’s another perpetual area of interest for me. So I have a few new writing and research projects that are going on with that and kind of in the development phase. And this summer, I am going to be catching up on a stack of books, just an epic number of these great books and works that are coming out. Seems like every week, there’s a new thing that goes on that list. So I look forward to a few weeks or more to really concentrate on that.

John: And we should note this is the second podcast that has come out of things we’ve seen posted on your substack blog.

Michelle: Oh, wonderful.

John: One other thing. Sarah Rose Cavanaugh has a new book coming out that we were fortunate enough to get a draft copy of and it should be out this summer: Mind over Monsters, if you’d like to see more about this topic.

Michelle: Oh, absolutely. Alright. It’s on the stack now.

Rebecca: That pile keeps growing.

Michelle: Yes, it does.

Rebecca: Better add on another week. [LAUGHTER]

John: We could all use an extra week or summer.

Rebecca: Right? Yeah.

John: Well, thank you. It’s always great talking to you, Michelle.

Michelle: Oh, my pleasure, you as well.

Rebecca: Thanks, Michelle.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

288. Mobile-Mindful Teaching and Learning

Faculty generally design courses on their computers, but many students interact with courses through mobile devices. In this episode, Christina Moore joins us to discuss the benefits of being mobile mindful in course design.

Christina is the Associate Director of the Center for Excellence in Teaching and Learning at Oakland University in Rochester, Michigan. She is the author of Mobile-Mindful Teaching and Learning: Harnessing the Technology that Students Use Most, which was recently released by Stylus Publishing.

Show Notes

Transcript

John: Faculty generally design courses on their computers, but many students interact with courses through mobile devices. In this episode, we discuss the benefits of being mobile mindful in course design.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guest today is Christina Moore. Christina is the Associate Director of the Center for Excellence in Teaching and Learning at Oakland University in Rochester, Michigan. She is the author of Mobile-Mindful Teaching and Learning: Harnessing the Technology that Students Use Most, which was recently released by Stylus Publishing. Welcome, Christina.

Christina: Thank you so much. So glad to be here.

Rebecca: Today’s teas are:… Christina, are you drinking tea?

Christina: I am.

Rebecca: Woohoo,what kind? [LAUGHTER]

Christina: I have to, of course. I am having honey vanilla chamomile tea. Just something refreshing and light.

Rebecca: That sounds perfect for a Friday afternoon. [LAUGHTER]

John: And I have one of my usuals, ginger peach green tea, today.

Rebecca: I just got a new shipment of my blue sapphire tea pack. So I’m back to drinking that. It’s a good spring tea.

John: And it’s all sparkly, isn’t it?

Rebecca: It’s not sparkly, it’s blue sapphire.

Christina: I never heard of sparkly tea, but I’m intrigued.

Rebecca: There is. We need to get our hands on some.

John: It was the same episode where you describe that tea. We’ve invited you here to discuss Mobile-Mindful Teaching and Learning. You note that you started writing this book on your phone? Can you tell us a little bit about the origin of this book project?

Christina: Yes. So probably the very first step is my interest in Universal Design for Learning. I find it to be a really useful framework for thinking as expansively as possible about how students can learn and how we, as instructors, can be involved. So my very first interest into mobile-mindful teaching in earnest was reading Tom Tobin and Kirsten Behling’s Reach Everyone, Teach Everyone: UDL in Higher Ed, and they have a chapter on “Meet the Mobile Learners” that was really this important call to learning with our phones and how, as educators, we’re really missing out if we’re not willing to consider the role that that could play, and I thought the argument was really convincing. So I started to do just a little bit of exploration into the topic. And that was probably in January 2020. And then a couple months later, COVID hit in earnest. It really upended our lives. I’m a mother of two young kids, and they were two and four at the time. So it was sometime in late March or early April, that we had been so cooped up, and we had used the family minivan so seldomly that we decided just to play in the car, that that would be the activity; not moving in the van anywhere, just playing in there. The kids would crawl around, listen to the radio… it was just one of those really comical moments just totally different than life in general. And it was really during those days that I was using my phone for work a lot more than I ever had before, because I just sometimes needed to keep things moving while we didn’t have childcare. I would read articles, I would take notes on how they might apply to something else I needed to write or work on that day, or it would spark an idea. And it was really at that point that I realized, it’s not just 18-, 19-, 20-year olds who want to learn on their phones, it’s really something that all of us, at least the vast majority of us, take advantage of. And during certain periods of our life, we need to lean on them more heavily. So I actually started to realize this while I’m sitting in the van, and I started writing down some notes about this experience. I was connecting back to some of the things I started to read and work on earlier. And then I sent the piece to EDUCAUSE and they were really interested in publishing it. It was a really short piece, but I was really surprised with how many people resonated with that, because mobile learning is still something that tends not to excite most instructors, it just feels like this distraction device, something people don’t want to think about, we’re already frustrated about it. But a lot of people recognize themselves, I think, in the story that I told and in some of the practical places to start. So in many ways that was sort of the seed to what would become this book, because honestly, while my interest has been in educational technology for a while, I would not have guessed, I would have written this book, but really it was the need to address something that I think we’ve been ignoring, or just haven’t been able to find a really accessible entry point to as far as a really good learning opportunity and even a good teaching opportunity for us. I was really inspired in this book to say “Okay, let’s come up with a starting point for at least considering what role mobile learning can play. And let’s start developing our curiosity and see where we go from there.

Rebecca: Can you talk a little bit about who the audience of the book is?

Christina: I really tried to make this book to the person who is excited about teaching, they really care about their students’ learning experience, even if they’re not so excited about the idea of students learning on a mobile device. And I would describe the audience that way, because I think that’s the audience that a lot of the enthusiasm around mobile learning has been missing. And we haven’t had that critical mass of instructors who are finding a good entry point in. So I would say it’s like the learner first, tech second, type of instructor who uses technology, normally, because they see it has a real benefit for students. And they may not always be comfortable with it, but they’re willing to try things as long it isn’t too overwhelming. So with that audience in mind, I really tried to take a beginner approach to tech. I explain how QR codes work, how you can create one, might even explain what the share icon is, because that’s really important for fluid learning and connecting our learning experiences. But I also allow space to dive into more course activities and possibilities that you can do with students, even if you don’t feel completely mobile tech savvy, because I’m somewhat in that boat as well. Of course, I learned a lot through this book, but I hope I did some of the learning and pre work so that faculty readers and other educators can just feel free to try things out, even if they’re not totally sure how they feel about these things.

Rebecca: Can you define what you mean by mobile learning, just to make sure that we’re all on the same page early on the conversation?

Christina: Yeah. So mobile learning, first and foremost, makes us think of learning on a smartphone, which mobile learning can sometimes mean like tablets, and even non-smartphones. But we’re normally talking about smartphones, phones that have the capability to connect to the internet, especially because increasingly, most people, most adults, have mobile phones. But I do think a little bit larger about mobile learning as well, just acknowledging the fact that we learn in motion. I think a lot of us sit in front of a computer for a lot of the work that we do. But what I try to guide us to think about is the fact that we learn and work in multiple places, with our laptop, with our phones, while we’re on a walk, while we’re on a drive. We do not stop learning or stop thinking the second that we are no longer in front of a screen officially working on things. So I also try to tease out that much larger idea of we are learners in motion and that we are learning and responding to our environments.

John: And smartphone ownership by students is close to being ubiquitous, nearly all students have a smartphone. And they normally have them with them all the time. And they use them regularly for learning. Yet there is some faculty resistance to students using smartphones in the classroom. How do you address that when faculty say “I don’t allow smartphone use in my classes?”

Christina: I think I take a balanced approach to this. So first, I acknowledge that and understand. I think even our students don’t really like how distracted they constantly feel by technology. Our institution actually just facilitated a student engagement panel with students talking about creating a collaboration around student engagement and learning with faculty. And even that was expressed by the students themselves. And that was corroborated by the research that I looked into. So, I think, and I have this mantra a lot throughout the book, which is somewhat the mindful aspect of it as “Well it’s okay, let’s acknowledge and notice that we have that skeptical feeling, but sort of suspend it and just be a bit curious.” So my first piece of advice is to talk with students directly about this issue, because our classes look different. They’re small, they’re large, they’re gen ed classes or major classes. So I think it helps to first talk with the students, maybe on the first day, if you’re discussing the course syllabus, “What should tech use look like in the class?” It can either be in class or in an anonymous form where you’re saying: “Some research has found that cell phones can be distracting not only to the person using it, but to the person next to them. How do you feel about this and your learning environment?” Then that can help at least bring them into the discussion, so that with whatever you decide, they feel like they have had some say and input, or at least some understanding of why you do what you do. That being said, I would definitely not support just a total tech ban. And that’s because, and the book does get into some of this research, there’s pretty strong indication that students are using these phones for e-texts, sometimes they are caregivers who really feel anxious if they don’t feel like they’re going to know right away if there’s an issue and they’re the primary contact. So there’s lots of evidence that by banning this technology altogether, we can do real harm to the students who really need it the most, which overwhelmingly are women, or people of color, and people with disabilities. So I wouldn’t encourage a total tech ban, a conversation with students, and really, similar to what James Lang has talked about in his book Distracted, that we don’t have to take an either/or stance, it’s not really reasonable to expect totally undivided attention. I mean, think of any faculty meeting and how many phones are even out during those. So just thinking realistically, but also maybe guiding your students, like, which point in class do you think it’s really important to put away the phones because you’re just talking to one another, and then maybe prompting students to do that. Whereas other active learning situations, you might not worry so much about the technology being used, because the activity itself is so engaging that you don’t really have to worry about that. So overall, I actually encourage us to think more about the mobile learning possibilities outside of a classroom, because I actually think that that’s where its virtues come out a lot more than it’s vices.

Rebecca: I’m really struck by the idea of this fluid learning and learning on the go. [LAUGHTER] And having learning in your pocket and work in your pocket. Your story was reminding me that just the other day, I was enjoying the nice weather, but had a lot of work to do, so I got out my mobile device and I talked through the presentation I needed to give so I could get an outline done while I was on a walk without having to be at a computer. And there’s lots of ways that we can use our devices. We talked a little bit about QR codes, and that might be one obvious way to use a mobile device in a classroom, but what are some of these other ways to use a phone that are in these other spaces that we don’t always expect?

Christina: So I think your example of being on a walk is one that I talk through, because part of the book, a fairly large section of the book is called “Start with Self.” And this is really guiding us through the basics of what it means to be a mobile learner, what are some basic skills that will help you actually become more familiar with what it is to be a mobile learner firsthand, because we’re really not used to thinking of ourselves in that way. And I like the walking to sort of get a break, but you’re still actually being very productive. And maybe you’re being more productive because you’re breaking up your thinking, your body is moving, your muscles are moving, so your brain is likely going to be better and more responsive. But I think, for most of us, we have to sort of walk through the steps of “Okay, but how do I make that happen? How do I use voice to text in order to be able to speak into the phone and have the text written out? Where do I do that? What app do I use?” And along with fluid learning, which is the idea that we design learning activities so that if we do our learning in one place, we can then access it in a useful way, in a different context, a different device, in a different situation. So it gets into that decision of am I taking my notes in a app that I can easily access when I do decide to sit back down at the computer? So I think going through those simple steps of “Okay, what buttons do I have to press? How do I find out how to do that? What tools and processes are going to work best for me?” I think is something that we have to start with, because many of us aren’t used to putting all of those pieces together. But just to use this example, again, the idea of being able to walk and learn in a productive way, is a really good example of something that’s good for our bodies that in a way actually takes us away from screens a little bit because we’re not so focused, even on the screen in front of us, we’re just using it basically as a recording device. And that’s why I do like us to think a little bit broader about mobile learning because yes, it is learning that is made possible because of phones but it is not always just us with our thumbs staring at this teeny tiny screen, but it’s also how it allows us to take pictures of things that we find so that we are actually connecting whatever we’re talking about in class to something that we are seeing in a completely different context. And if we think about the application that we make possible, how much more powerful is that learning that we’re able to take it from our environment, and then find some way to share it with our classmates, such as a shared messaging platform or a shared folder where people can put their pictures, I’ve heard really amazing examples, especially in like biology and STEM fields, where instructors are using, whether it’s something like social media or just a shared photo folder, where both the instructors are sharing photos and asking their students to identify them, sometimes on a daily basis, where also students are actively collecting samples via photos. And then they are working as a class together to label the genus or species of whatever leaf, plant, or whatever that they have identified. So I think that answer was a little bit mobile in and of itself, it might have kept going, but I think it provides some examples.

John: In the title of your book, you use mobile mindful, rather than mobile learning. Why is that distinction important?

Christina: Even stepping back to when I was thinking of mobile learning directly, I was really wanting to use my phone more productively so less mindlessly, because I was noticing I was just going to my phone to pass the time. And I was doing things I wasn’t even interested in or was consciously thinking of. So I began thinking, “Okay, how can I redirect this habit into something that is more intentional?” So that’s one reason for the use of the word mobile mindful is this idea of intention, and using our phones for the things we actually want to do rather than just for this pure distraction. But I also use the word mindful with it as a hopefully less intimidating and less techie sounding approach to it. So when I think of mobile mindful, I think of something that’s mobile aware, or mobile-ish, or it’s an adding a piece to our already existing rich ecology for learning that we create in our classroom. And just adding this as like an extra tool, an extra really powerful way to connect all of these pieces together and help our students learn more often and think about the content more often. So it’s a mindful approach to mobile learning, but also like, let’s start with mobile aware before we like go diving into mobile learning. So I contrast it with mobile first learning, which some people are doing amazing work on, which is putting in the constraint and challenge of let’s try to create a whole learning environment that can take place on a phone. That is not the approach that I take in this book. I think it’s productive for most college and university instructors to first start with, “Well, how can this be one piece and one delightful added element to all of the good work that we’re already doing?”

Rebecca: I’m curious in your role as an instructor and in a teaching center and your interactions with students, what are some of the most interesting ways you’ve seen that students have just adapted to using their phones to help with learning?

Christina: Well, I’ll answer the question, but I’ll also add something to it that I think is important for us to realize, that once we go through the learning process of being a mobile learner ourselves, there may need to be a little bit of prep work that we also do with students. Well, a lot of our students have only known the world that is mobile phone capable, it doesn’t always mean that they are ready to be mobile learners. They have definitely internalized messages that phones are bad for them, phones shouldn’t be in the classroom, even though they bring them in anyway. So there’s very much this vice type of attitude towards it. And therefore they haven’t had always a lot of opportunities to use their phones as these powerful learning devices. So I would add sort of the caveat of “Yes, students are doing amazing things and can do amazing things, but they may need to be guided into it a little bit just as much as we do.” So with that being said, I would say that some of the exciting things that students are doing and going back to QR codes, I really liked the example of audio essays that were taken to specific places. And again, QR codes have become so much easier to both use and create. You can basically create them from any browser. The QR codes in my book are purposely created for free by right clicking on any website and there’s a drop down option that says create a QR code. You can create fancier logo specific ones, but I decided to just use the default one as sort of a demonstration of the fact that they are really easy to create, and that’s how I went about it. So I think even just adding an element to maybe research presentations or things for a specific audience where you say, “Okay, how could you use a QR code to direct people to a different learning element?” And so it might be directing them to a piece of audio where you’re explaining something that is in a very specific environment. So again, thinking about learning being mobile is you are creating learning experiences that take place in very specific locations, or could take them to a form that they fill out or a petition or something like that. So I think it creates a lot of convenience and thinking about your audience and makes it a little bit more creative. And then the one other really interesting use that I cover, and I talk about the ethics of mobile phone use and inviting students to use their mobile phones. That’s actually a really good opportunity to get students to think more critically about the data being collected on them. So some faculty have done really interesting work in places like statistics, or other data analysis type of classes, and students have been invited to download the data that is collected about them on social media or on Pokemon Go, especially those that are mobile dependent, like especially Pokemon Go. It not only teaches the students a really useful content skill and applies it, but it helps them be a little bit more critical about what is actually going on behind the scenes when they don’t actively take a role in limiting data sharing about what they’re doing and where they are. So I think that type of application of getting students to think and actually dig into their own data is a really good example of what I think faculty could start to find as really exciting about mobile-mindful teaching, as they start to see that there is a lot they can have students do that really isn’t as possible in other ways.

John: You mentioned QR codes. When they first came out, they were really useful tools, but you had to dig up a specific application to scan them. Once smartphone manufacturers allowed the cameras to directly read and respond to QR codes, it became such a game changer in terms of their use. I don’t think I’ve given any presentations, either in class or at a conference in the last three or four years, except during the time when all instruction was remote, and then I was more likely to drop URLs in the chat. And it just opened up so many great possibilities for sharing resources with students, with colleagues, and so forth. I’m still amazed at the number of faculty who don’t know how to use QR codes. And I was really glad to see you had a discussion of that in the book in terms of how instructors could use those within their classes. You mentioned a little bit about the use of QR codes, but how might instructors use that in their class?

Christina: Yeah, so and just as a funny note, back to the audience, my mother is reading my book, and I asked if she’s tried any of the QR codes yet. And she said, “What’s a QR code?” [LAUGHTER] And I said, “Oh, maybe you haven’t gotten that far in the book yet.” But then I was explaining it to her. And she’s like, “Oh, it’s what’s been on all of the restaurant menus.” I was like, “Yep, those are the ones.” So I also think of like, you don’t know what a QR code is and you’ve seen it, but you haven’t connected the dots. I’m hoping that this book will connect a lot of the dots. So I will give a couple of really useful examples of an instructor intentionally using QR codes. So, I think in the spirit of Universal Design for Learning, it can be really nice to add QR codes to print handouts, because I think sometimes students do like to have print handouts because it helps them resist some of the distraction that comes with phones. They like having something tactile, but by putting a QR code on them, if they would rather consult something on a phone and take digital notes, they immediately have that option. So I think that’s something that’s fairly easy that can be done if you use print handouts, but want to be conscious of people potentially using mobile phones, or directing people intentionally to other websites by using their phones. My other favorite, which I think is also useful in other contexts, is when you want to get quick feedback from your audience, such as students, displaying a really big QR code on a projector. And then even in a really large class, they can pretty easily scan it and then they can give you some really useful feedback that you have in a digital form that can be automatically analyzed or you can quickly go through it. So I think of some of the really classic active learning strategies that we may be familiar with such as exit tickets, what you want to know from students at the end of class. It might be the muddiest point, where you want to know what students are still confused on, or one-minute papers, where you want a really quick reflection about what they’ve learned in the class. So by displaying that QR code, students can take the form there, and then you quickly have all the data. Inversely, you can also do this at the beginning of a class, if you want to ask students, either three review questions, or you want to ask them three questions that are just going to prime the pump of whatever you’re about to discuss to sort of see what they know before you’ve even covered it. While I didn’t do it with the QR code, this was one of my favorite mobile learning activities that I tried the last time that I teach. Because it was an asynchronous online course, I wanted to get a sense of students feeling like I am responding to what they’re learning and thinking. So I would start the week with a really short form and say, “This will take you five minutes, I just want to know what has your experience been doing primary research? Do you know the difference between primary and secondary research?” And then I could address that feedback directly into my instructional video. So it would create that sense of presence, even though it was an asynchronous course. And by telling them it would take them five minutes, they did it right away. So like, that’s kind of the magic of micro learning, which is, I think, one of the superpowers of mobile-mindful learning is if you can break things down into smaller chunks, students will do it. And that’s kind of the interesting course design pedagogical challenges, to figure out how to get things into smaller pieces.

Rebecca: I think one of the things that we often assume is that the students who are using their phones for learning or to complete work are more traditional aged college students. But from my experience, [LAUGHTER] it’s often adult learners who are using their phones the most, because they’re often double timing as your example in the van [LAUGHTER], or at soccer practice or during swim lessons and trying to complete a module or reading the captions on a video, or [LAUGHTER] any number of other things, getting a start on a paper or trying to edit or providing feedback to peers or something. So they’re taking it with them, and often maybe in an environment where there’s other things going on, but trying to make progress on something in the little snippets of time, those small chunks of time that might be available.

Christina: And that’s what I think is useful about thinking of this fluid learning environment is, of course, we don’t always want to be learning in that context. We want our students to think deeply, we want them to have time to really mull over ideas and work in larger chunks of time. But what I’ve come to realize is that there really is quality learning that can happen in those snippets of time, mostly by frequency. Because I think a lot of times with the way activities and courses are set up is that the students are cramming right before class, the last possible minute, to do everything they were supposed to do over the last week. And we know from experience that this does not produce quality work. And it doesn’t bring into our class, a really curious thinker who’s really been mulling over these ideas. So I think if we reframe this sort of micro learning as “How would your students think differently, if they thought about your course content four times a day, even if it was in really small snippets?” They would probably be a lot more prepared and have more interesting things to say once they do sit down with that hour to work on things. So even if we sort of dread and don’t love the idea of our students doing things while waiting in line for five minutes, or being in a waiting room for 10 minutes, I think reframing it as like this is a piece that will contribute to a longer work period. I think that’s a lot more inspiring.

Rebecca: You know, from my own experience, I get more reading done, because sometimes I have my device read to me and I can do that in the car or in other places… maybe not good for research kind of reading but good if I’m trying to get background knowledge on something or keep up with something that’s current.

Christina: Rebecca, you’re narrating and describing exactly the types of things that I want readers to notice about themselves, about how they learn, because I think we normally don’t notice these things. We just sort of do them because we’re trying to just do what we can. And what we want to get students to notice about their own learning emotion as well.

Rebecca: I mean, I’m a designer who designs for mobile devices, so I’m already sold. [LAUGHTER] But I think it’s important that we recognize how often people are using their devices already, and all the ways that we could use them that we don’t always know that we can. My students are often really surprised when I show them some of the features that are available to them on their phone that make their lives easier.

Christina: Yes. Exactly. And that’s why we have to take students along with us and pointing things out like, “Oh, do you know how to do this? Do you know that the learning management system has an app? Do you know that it will give you push notifications when I message you, so you don’t have to worry about catching up on my emails as much?” I think those little nudges like, “Oh, did you know or how do you keep notes on a phone?” I think those types of nudges and getting them in the right direction will help in your class and throughout their whole lives.

John: And you mentioned that you got started on this through a UDL approach. And smartphones enable a lot of assignments that can be done in multiple modes. Could you talk to us a little bit about how instructors might use that to give students multiple ways of demonstrating their learning?

Christina: Yeah, this reminds me of a course activity that I propose in the book. And it’s called “untethering the research presentation,” because I predominantly teach writing and rhetoric to first-year students, they’re required courses. So I think students are really used to doing slide presentations for their classes. And I think they do that because we’re comfortable with them and so it just becomes this routine thing that doesn’t have a lot of love and spirit behind them. So I think this idea of untethering the research presentation is like, let’s think of this in a little bit of a different way. If we’re not using slides, what else can you do? Is it a really engaging discussion without technology? Is it a video? Is it coming up with a social media campaign. And what I like about that idea is not only is it a more creative and authentic way to put whatever they’ve been researching into action with a real audience, but it gets them to think in a different way about how that information lives. So I think mobile learning can be a really important part of this, especially if students are thinking about who their audience is. They may determine that their audience is going to best be reached on a mobile device. So if they’re doing a video, and they determine that their audience is most likely going to look at this on a phone, how are they going to design that video accordingly? If it’s on social media, then doing something in portrait might be the best because it scrolls through better that way. So I think prompting our students to also be, depending on their field, be prepared to be mobile practitioners and how they can reach a mobile audience. Another example I use is if you are a math educator, we hear about the new math and reaching out to parents about how to guide them through that. How many of your parents are likely going to maybe be smartphone dependent, meaning that the only reliable internet they have at home is on a phone? So how are they going to use that sitting next to their child helping them with math? So I think by posing those types of ways of presenting information for a specific audience, is a good example of both inviting students to express their learning in a way that they are comfortable with and excited about and speaks to their strengths, but also getting them to think about the audience for the work that they’re doing too, and how to demonstrate that learning to an audience in a way that is relatable and accessible to them.

Rebecca: So one of the things that got me really curious about how students are using mobile devices is actually, how they might even engage in the learning management system. So we talked a little bit about having an app, but also sometimes there’s a web version that’s made responsively, and also exists on the mobile device. And what I’ve discovered often is that those are sometimes different, or the way you even get to information material is different. So that’s always something that I start talking to my students about is like, “Okay, if you’re using the app, you can do this one thing, if you use the website, you can use this other thing.” I’m mentioning this, in part, because the way that students are engaging with their materials sometimes is really different if they’re in the app of a learning management system versus the website version of it. We might have micro lessons or small activities that we’re doing on our devices, like videos and things and those experiences might be really different. I’m curious about the ways that we can help faculty become more aware of the different ways our students are using their learning management systems, even on our mobile devices.

Christina: Yeah, so that’s great. And I’m glad we’re bringing it up. I highly encourage us to regularly take mobile test drives through our materials. I think it’s a really good place to start. So pull up your syllabus on a phone, what does it look like? How easy is it to navigate how when you open up links that are there? How many clicks does it take or how many taps if we’re thinking mobile mindful? How many taps does it take for them to get to the content that they want to get to? So I think actually going through the tactile experience of going through your course materials on a phone is really insightful, because I actually hope that you’ll find things work a lot better than you expect, because I think mobile accessibility has gotten a lot better. And I think sometimes we might still be thinking in like a 2004, even 2012, realm of like, where everything just looked terrible on the phone. I think we might be surprised when we actually go through that things scale, and are more responsive than we expect. So I think that that’s a great place to start. If you use the learning management system at your institution, and you’ve never looked into whether they have an app, you can do that. Download it. You might discover things, like I mentioned sort of offhand earlier, that there are push notifications whenever you use the announcements forum, or whatever it’s called in your learning management system. So students get that right away, rather than hoping that they get into their email. So you may discover that there are certain surveys that work okay, but if you maybe used a slightly different tool, or you broke up the question in a different way, it might be even more responsive. And that might make you think, “Okay, if I actually just break this one 30-question quiz into three 10-question quizzes and open up that access so that they can take it as many times as they want, and then I tell them in class, I want you to do these, you can do them on your phone, it’ll probably only take you 10 minutes a day, I mean, then you start to think of how much they’re practicing and reviewing the material so that they don’t even have to think about it anymore, they can get right into the more complex thinking. I think even that test drive mentality of like, “Okay, let’s see how it looks,” then I can sort of guide students on what I think works well on a mobile phone and what I think doesn’t work well on a mobile phone. And then even, and this is what I was doing throughout the book, is taking screenshots that I wanted to save and show in class, okay, this is how it looks. It really helps reinforce that for students, and then going through your course texts, trying to identify what works well, on a mobile device, tell students to do that. You might also feel free to say: “This one text, it really doesn’t respond on mobile well, that’s something I would say to do on your computer, to do offline.” I think talking students through those options really gives them a lot more agency, because I think a lot of our impulse is to say, “Don’t use your phone for the course, it’s not designed that way.” But they are, for different reasons. Sometimes, they’re just going to do it that way. So if instead we can say, “Well, the discussion forums work well if you do a video post, but otherwise, if you need to cite things it might not.” So by giving them an action and plan, rather than just saying, “Don’t do it,” I think that’s gonna get us a lot farther. And I know that in doing this test drive and thinking about how we can leverage those five and ten minutes, it actually got me really excited to think about quiz design, how I get feedback from students, and even how I design my instructional videos. In the UDL mindset, I started to record my videos the same, but I would just pass them on to YouTube as well, instead of just in the learning management system, and then I would have a link that says access on YouTube. And then I would make that into a playlist. It’s not really any extra work, it’s just organizing them into one list. And then it gives students the opportunity to just keep playing through, which we probably know, as mobile consumers ourselves, is that it’s easy to get us to buy in, if it’s only a three-minute video. But then we’re like, “Okay, let’s just do one more three-minute video. And then we’ve been watching videos for a half hour very easily. So if we can use that capability for good, [LAUGHTER] I think that can be something exciting for us.

Rebecca: Christina, did you just suggest designing a learning rabbit hole? [LAUGHTER]

Christina: I sure did.[LAUGHTER]

John: We gave a workshop recently where we encouraged people if they were using videos in their class to do the embed, rather than sending students to YouTube, because that rabbit hole could often take them in directions away from the course. But if you’re directing them to a playlist with a whole series of videos, then having that rabbit hole could be very useful.

Rebecca: That’s downright sneaky.

Christina: Yeah, but let’s use the sneakiness for good. But, doing the test drive, we can also recognize where things are distracting if we tried to take them to a mobile device. And we might just be transparent about that, and for that reason, suggest that they don’t go in that direction. So it’s why it’s this mindful approach. It’s just being aware of what works well and what doesn’t, and giving our students some direction accordingly.

Rebecca: Sometimes that test drive can reveal even little details like should this open in the same window or a different window?

Christina: Mm hmm.

Rebecca: Because some tools are fine on a desktop, but as soon as you try to do it on the mobile device inside of the learning management system, it’s a nightmare.

Christina: I think that’s actually a really good example of how I think going through this thought process will reduce friction, and overall just improve the teaching design in general, because we found that with online teaching, too, is that when people began teaching online overall…I mean, as long as they did it, right, of course, and took a good approach… it actually often increased the quality of their in-person or on-ground learning as well, because it was just a different way of thinking about it. And it helped you see where there were barriers that you could take away. So I think that’s a good example of, it just helps you pay attention to the learning experience in a different way that could give you really good insight overall.

John: We always end with the question, what’s next?

Christina: Well, first is, a little bit of a break. [LAUGHTER] I definitely want to talk about the book, and I will be, but I’m taking just a little bit of a pause. But during this pause, I’m actually putting together content for a blog to kind of be the “what’s next” of the book, because the book is an invitation and it’s a framework for us to get started with mobile learning. But from there, I know that there are people doing brilliant things with mobile learning, or they’re going to have lots of light bulbs that go off because of this book. So I want to continue the conversation. I didn’t want it to end with the book. So I plan on contributing content myself, but also inviting people to share their mobile learning strategies, victories, challenges, stories. So I may provide my email address so that people can feel free to contact me if they would like to contribute something. I’m gathering up goodies so that I can start to share them out into the world. And then I also want to work with faculty to research how the application of these strategies are going, because I’d like to see the evidence and put them out in more formalized ways so that we can really build and make this a practice that is more common, more accepted and really is convincing that it is what students need and provide guidance on how to do it well.

Rebecca: Well, thank you so much for the book and really thinking about introductory audience.

Christina: Thanks a lot. This was great.

John: I really enjoyed reading the book and I’m really happy we can share this with our listeners.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]