168. Synchronous Online Learning

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

Show Notes

Transcript

John: As we approach the end of a really challenging year, we’d like to thank all of our guests who provided so much help and support to us and all of our listeners, and we’d like to thank you, our listeners, for hanging in there with us. We’ve all learned a lot in 2020 and we’re looking forward to a chance to apply what we’ve learned in circumstances in which there are fewer external threats.

…and now we return to our regularly scheduled podcast.

The pandemic forced many faculty to experiment in different modalities in 2020. In this episode, we reflect on our own teaching experiences with synchronous online courses this year.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Over the past few months, we’ve talked a lot about the pandemic and ways to adjust our teaching. And we’ve talked a lot about online learning, but we haven’t really focused on synchronous learning. John and I both taught synchronously this semester. So we decided that in this episode, we would focus a little bit more on synchronous learning and what we’ve learned about it in our own experiments in our classes.

John: Our teas today are:

Rebecca: I have Scottish breakfast once again.

John: …and I have a blend of spearmint and peppermint tea.

Rebecca: That sounds much healthier than my choice.

John: It’s not my first tea of the day.

Rebecca: This is not mine, either.

John: This is my first herbal tea of the day.

Rebecca: This is my second pot of the day. [LAUGHTER]

John: So, Rebecca, what classes were you teaching this fall?

Rebecca: I was teaching two design classes that are smaller. So I had one web design course that was stacked. So, it had beginning, intermediate and advanced students in it, 25 students, and we met synchronously, but also had asynchronous classes, it’s considered a studio course. So for a three credit course, we have six hours of class time, and three hours of outside work, which is a different balance than maybe other folks. And then the other class I was teaching was a special topics design course, which was smaller, it was about 10 students. And that class was also synchronous, but it was a project-based class, and we worked on two community design projects: one for a project called Vote Oswego, which was a get-out-the-vote initiative on campus, and the second is a project called “Recollection,” which is a storytelling project with adult care facilities.

John: And we do have an earlier podcast on an earlier iteration of Vote Oswego. So we’ll include a link to that in the show notes.

Rebecca: So John, what kind of classes are you teaching? We obviously don’t teach the same thing.

John: I was teaching two classes this fall. One was a large synchronous session with 288 students. And the other was a fully asynchronous section with 60 students this semester,

Rebecca: At what level were the students in both of your classes?

John: These were both introductory economics classes. So most students in the class were freshmen, and it was their first economics course.

Rebecca: So, your classes are much larger than mine, you’re teaching much more younger students or newer students, and my classes are smaller, project-based, and usually junior or senior students.

John: Yes. And there’s certainly some differences in the disciplines as well.

Rebecca: No, that they’re the same. [LAUGHTER]

John: Why did you choose a synchronous mode of delivery rather than an asynchronous mode, or a face-to-face option this fall?

Rebecca: So I chose not to do face-to-face delivery for my own health reasons, I chose to not be on campus for my own safety, because I have a chronic illness. So I chose specifically to have strong synchronous components, because a lot of our students are used to working in a studio together and having a community around each other and kind of feed off of each other’s work and work collaboratively. And I wanted, because of the classes I was teaching, to continue to have collaboration as a key part of my class. And I was really concerned that if I didn’t have a strong synchronous component, my students wouldn’t be able to effectively collaborate with each other, because there would be too much scheduling issues and what have you. So it’s a little bit of a carryover from the way that I would run my classes in person in that I give a lot of class time to project-based learning and team-based work and do a lot of lectures and things like that asynchronously in like a flipped classroom style. How about you, John?

John: Basically, I tried to preserve something as close as possible to what was originally scheduled or what was originally planned. And my large class is typically about 400 to 420 students, and I just couldn’t imagine taking that class and doing it in a completely asynchronous manner, because when I teach a class asynchronously, I give students lots of individual feedback, and it would be really challenging providing individual feedback to several hundred students. I just didn’t really have the time to do it in that sort of mode. So I thought it was better to work in a mode where I could give students feedback in a group setting using some interactive tools, where they’re all getting feedback at once. It was the only way I could see handling a group that large. If I was trying to do it as an online class, it would be effectively more in the form of a MOOC with very little interaction, either among the students or between me and the students.

Rebecca: One of the things that we both talked about before we started recording was how we both used a flipped classroom model to help with our synchronous session. So can you talk a little bit about how you did that and what students were doing outside of class.

John: This is actually, in many ways, similar to what I had done in a face-to-face class. Before each class, students would have some readings to work through. And I use the Lumen Learning Waymaker package, which is basically taking materials from a textbook, combining it with interactive multimedia, where they got to shift demand and supply curves and other curves around and see how they responded when they change parameters. And they read a bit in that online text and then they would work through some problems on it where they were allowed multiple attempts at those problems. I also created some videos with embedded questions that were at a somewhat higher level than the textbook readings, which was a little bit more challenging. And they were given unlimited attempts to work through those videos with the questions. And they also, outside of class, participated in discussion boards, where I asked them to relate what they were learning to things in the world around them, in their own lives and their own experiences.

Rebecca: …nice little inclusive teaching practice right there, right? …connecting students with their experience and making it relevant to them.

John: Right, because we know that students learn things most effectively when it has some salience, when they see the relevance to their life.

Rebecca: And the waymaker package, if I remember correctly, had some quizzing and stuff associated with that, and unlimited attempts, a version of retrieval practice there.

John: It’s a mix of things with unlimited attempts and limited attempts. So, the microeconomics Waymaker package is designed, and all of their Waymaker packages, for that matter, are designed, is that they start with a list of broad learning objectives. And they break it down at each module level to sub objectives. And they break those down into sub modules. So, in what would have been the equivalent of a chapter of a textbook, there’s usually two to four sub modules on particular aspects of that. And students work through that. And embedded in it, they have some review questions, some practice questions. And those they can take an unlimited number of times at any point in the course. Once they complete the module, they have a module quiz where they are limited to only two attempts at it. But they’re getting feedback on what they did well. And what they didn’t do well. It’s automatically color coded to indicate whether they mastered the material in one of the blocks of content in there. And then, if they take the module quiz, it will give them feedback on what areas they did well, and what areas they need to work at more. And they’re being directed back to the areas that they need to review. And there, they do have unlimited practice opportunities. And the other thing I did is I created my own videos that focused primarily on the topics that students generally find the most challenging. And in economics, that’s generally with either applications involving math or involving graphs. It was one module a week, and I would take the topics that I know, from past experience, they were likely to have the most problems with, I’d create my own videos with that. And I was using PlayPosit, which allows you to embed questions in there. Most of those videos I created were between five and 12 minutes in length. They would watch the videos and answer questions as they were going. And if they got one of the questions wrong, they could go back and replay that portion of the video and then try it again. And they were given unlimited opportunities for that.

Rebecca: I think you mentioned students really loved those opportunities.

John: At the end of the class, I gave them a Jamboard, which I know is something you’ve used more regularly, asking them what worked well. And there was very much universal agreement on the PlayPosit, as well as on the Waymaker aspects of the course. They really liked the fact that there were practice activities embedded right in their textbook, and that they could go back and try things over and over again until they mastered it. And it was giving them feedback on whether they had, in fact, attained mastery at every step. And it was a nice visual indication of what they’ve learned and what they still needed to work on more.

Rebecca: Excellent.

John: What did you do in your asynchronous components of your class?

Rebecca: Well, the balance of my classes, as I mentioned before, is a little wonky in that we’re supposed to spend more time in class and less time out. So asynchronously, I did a lot of independent stuff that students were not necessarily doing collaboratively. So this is where I had lecture videos that are recorded that were about the topics that they were going to be working on or introduce the component of the project that they were going to be doing. And then they also were completing things like LinkedIn Learning tutorials. And we also have access to D-Q University, which is a set of tutorials for accessibility, and teaches accessibility. So I took advantage of that package as well. And largely they were completing those kinds of tutorials, both of those have exercise files and that kind of thing that they can follow along with. They get little certificates. When they’re done completing there’s little quiz questions and stuff. So they were doing a lot of that kind of work asynchronously. They were also using Slack to communicate with their teams for independent things that they were working on that they needed to communicate out to teams when they were working on projects together. And I also use Slack as a place to have discussion. So like you, I had discussion questions that tried to make what we were talking about relevant. We were exploring design, specifically like web design and how they interacted as a consumer versus how they would interact as a maker and did a lot of observational studies. We also did some discussion boards that were really about design activities and things that got students off the computer. So they were just documenting what they did off-screen, offline. So things like listening to a podcast so that they didn’t have to be staring at a screen and what their takeaways were. They attended virtual conferences, which I guess was still on-screen, and did some sketching, like paper prototyping and some other methods that we like to encourage our students to do, just to kind of help balance the screen time a little bit for students. So that’s largely what they were doing asynchronously.

John: Were you having them submit some copies of that work in some way, or were they just reflecting on the work that they had done?

Rebecca: The little non screen activities were documented in a discussion, essentially, that we were holding on Slack, and then tutorials and things, they were just submitting their completed certificates. And so I broke down those LinkedIn Learning courses and things over multiple weeks. So they didn’t really submit those certificates until they were completed. But they were doing a little bit by little bit, but if they didn’t do the tutorials, they wouldn’t be able to do the projects or the actual work that we were doing of the class. So it was pretty important that they were doing those components outside of class.

John: Once your students were in class, what did you have them do in a typical class session?

Rebecca: The two classes I was teaching I handled a bit differently because of just the sheer volume of students in the bigger class, which was 25 students that were working on projects. They were working on collaborative projects, in teams of three, for the most part. And so what we would often do is show-and-tell’s or critiques in small groups. So let’s say there was two or three teams together that we would do a little critique with in a breakout room, while other teams were meeting and collaborating. We would also do things like come together to answer questions about things that they were working on, troubleshoot or whatever, and then go work on projects in breakouts for a bit. And then we’ll come back at a scheduled time. I also did one-on-one meetings with students during class time. So I’d set up things like a quiet work breakout room or the chatty breakout room. And students would pick the place that they wanted to go while they were working on projects. And then I would meet with them individually for critique, and often a lot of code troubleshooting is a lot of what I spend synchronous time doing. And students sometimes met with my TA to do the same thing, and with our small groups. I also did a lot of design challenges. And students really liked those and would like to do more to hold them accountable for the kind of material that they were learning outside of class or being introduced to outside of class in a low-stakes environment to test it out with some peers and troubleshoot. So I would pose a little design problem. And then they’d work in a small group to work on that problem in a very tight amount of time. They might spend 30 minutes… my classes are three hours long… or an hour, and then we’d come back and show them off or talk about different things. And I tried to make those design challenges fun and entertaining. So one of the first things we did, which worked really well to start gelling their teams that they were with the whole semester was designing an emoji for Slack that they used for their team. And they loved that assignment. It was partly about working at a small size, and so it was tied to some of the curriculum that we were doing, but it was fun. So they did that in a small team and then had to implement it. Later on in the semester, we did things like a 404 error page for their projects, which were just kind of entertaining. We tried to make them amusing, so that if you landed on a page, it was a good user experience. So things that maybe wouldn’t typically work on in one of my classes that were a little bit more fun, but really were emphasizing the technical and conceptual things that we were working on. The other thing that we use synchronous time for is I took advantage of our virtual platform, and I brought in alumni multiple times, and local designers multiple times and did little Q and A’s with them. Not every week, but every few weeks, or every couple weeks, I would bring in a designer for a 30-minute session. They’d introduce their work. And then students did a Q&A with them, which students really loved. And it broke up our time a bit and really gave them something special that maybe we didn’t always do in a face-to-face class that made the synchronous environment kind of special.

John: Excellent. That is a nice opportunity provided by Zoom that actually could work in the classroom too. But I think many of us just hadn’t really considered it so much. It doesn’t really matter where you are when you’re teaching in this sort of synchronous environment. So it’s very easy to bring in guest speakers and it’s something we’ve probably should have been doing more of in the past, but I think many of us will be doing more in the future.

Rebecca: So John, how did you use your synchronous time?

John: I had told students before each class session, what specific topics we’d be working on. And then most of the class time was spent asking him a series of problems of progressively higher levels of challenge. I basically adopted Eric Mazur’s clicker strategy of trying to find challenging questions where roughly half the class will get it wrong the first time and then letting them meet (in this case, I had the meet in breakout rooms), discussing it and coming back and voting again on it. And generally, you’d see a fairly significant increase in the performance after they’ve had that chance to engage in peer discussion. And that’s where a lot of the learning seems to happen when clickers are being used. I used iClicker. The only difference is students could not use a physical radio frequency clicker because they have a range of a couple 100 meters and students were spread out all over the world, I had one student in Egypt, I had students in South America and students spread throughout the country this time. So they needed to use either their laptop or a mobile device in order to do that. We discussed it as a whole class after they come back from the breakout rooms. And then I’d asked them to explain their choices. I generally have them use chat, and then I’d go through and correct any misperceptions they’d have. And I try to guide them to the correct answer by asking them questions, and letting them see for themselves why some of the answers were right, and some of them were wrong. And generally, that’s how we spent many of our classes. Initially, I was also using Kahoot! from time to time. They enjoyed Kahoot!, but I noticed a bit of a drop off when we were doing the Kahoot! sessions, because those were not graded. And with the clicker questions, they were being graded, and that tended to receive a somewhat higher level of interest. It was very low stakes, they got a certain number of points for an incorrect answer on either attempt, and they got a bit more points when they answered the question correctly. And initially, I was giving him three points for an incorrect answer, and five for a correct one. And they asked it perhaps that could be bumped up, because some of the questions were so challenging. And I did raise it. So they ended up getting four points for any answer, and five points for a correct answer. So it is extremely low stakes. So I tried to do a lot of retrieval practice in the class, where it started from essentially no stakes with the embedded questions in the reading, then it ramped up to in class applications of this, where they still get 80%, even if they got it wrong, but they had another chance to get it correct. And then they took that module quiz, and even there, they had two attempts at it. So if they made mistakes, they had lots of resources they could go back to and work on it. So I tried to set it up and provide them with many pathways to attain mastery of the content, and to encourage a growth mindset and to encourage them to recognize that people make mistakes when they’re learning and that there’s a lot of benefit from having those mistakes as part of your learning process. There’s a lot of research that shows that we learn things much more deepl if we get them wrong, when we first try it, we’re much more likely to remember it later on, then if we happen to get it correct, initially. In that case, we’re much more likely to forget it a bit later. And that was a bit of a challenge for students. But I think they eventually appreciated the fact that everything was fairly low stakes.

Rebecca: I think I’m seeing some themes in the things that, although we’re teaching very different classes in very different contexts, there’s some real big themes about how we’re using our synchronous time, and even how we’re using our asynchronous time. And so there’s an emphasis on peer interaction and establishing those peer networks, really enforcing or reinforcing things and dealing with muddy points. And then also just providing the encouragement and support like that low-stakes environment or trying to foster a growth mindset. So in my classes, I did the same thing. I was doing peer group work and trying to really get them to collaborate and troubleshoot together and they love that that… that was really valuable. I spent time doing live demos and troubleshooting, when there was a really troublesome technical component or something that they were trying to do that a lot of them were having trouble with, that they could ask me live questions. So that same muddy point kind of thing that you were getting to in what you were discussing. And then, finally, the growth mindset that you started bringing up, I’d also tried to do and, although I didn’t have a lot of low-stakes testing, or something like that, I set my projects up so they were done in sprints. So a long full-semester project was broken into multiple two-week sprints, where they would work on something, get feedback, and then could revisit whatever they did, and then add a new component to it. And so I did that throughout the whole semester. So there was a bit of retrieval practice, a bit of spaced practice in there, and certainly some fostering a growth mindset and the idea that you make mistakes and that’s how you learn. And I spent a lot of time… I don’t know if you experienced this too, John… but I experienced a lot of time in synchronous and saying like, “You can do this. It’ll be okay. And this is how the learning experience works.”

John: And I did have to do a lot of that, especially in the first few weeks of the semester, because they were not used to a flipped class environment. And they were not used to this notion of making mistakes and learning from mistakes as part of your learning process. Because most of them have come up through their elementary and secondary school system thinking that they need to memorize some things and reproduce it on exams. And they do well if they get high scores, and they don’t make mistakes. And that’s just not how we learn in general. And it was important, I think, to help remind them of that. Another aspect of the flipped class environment that we’re both using is that we let students learn some of the basic skills, the easy things that they can learn pretty easily on their own, from other resources. And we’re trying to focus our class time using essentially a just-in-time teaching approach where you focus on the things that students always have trouble. In a traditional classroom environment, what normally happens is students will learn the easy stuff in class where faculty will lecture them on basic definitions and basic concepts. And then it all makes a lot of sense until students try to apply it. And they try to apply it typically in assignments outside of class, or in high-stakes exams. And it’s much more productive if the students use the time outside of class to master those basic concepts. And then we hold them accountable for having done that somehow in class. And then we give them assistance on the things that they find challenging when they need it. Not after they’ve had that experience of a more high-stakes assessment in some way.

Rebecca: Yeah, I think what I found or that students really shared with me that was something that they really appreciated was that there was a lot of structured time to work on those difficult problems in class. This is true of my face-to-face classes too, but even maybe more so in this online environment where students were having a really hard time managing their time. I would allow time to work on a project during class… it was scheduled, but then there was a check in point later on in the day. You wouldn’t want to spend three hours staring at a screen on Zoom, like this makes no sense. So I certainly did not do that. And I don’t want anyone to think that I did that. But, I would do things like “Okay, we’re going to check in at 9:30. And then we’re going to do a little activity together. And then you’re going to have some work time to work on X. And then we’re going to come back at 11. And you’re going to show me what you did. And then we’re going to have a little discussion or do another little activity, and then we’re gonna come back again at 12.” And we would have a schedule where there was time to kind of come back. What I found is, over time, students often wouldn’t actually get off of Zoom. They would just turn their cameras off and their microphones off. And I would do the same if it was like a work time. And then when we all came back on, a lot of students would turn the media back on. That said, I, of course did not require that depending on where students were, I certainly had students that were in environments where they couldn’t turn their cameras on, or had really poor internet connections, we adjusted as necessary there, and we had a way to communicate in a much more low-tech fashion using Slack during class time. So if something happened with someone’s internet connection, or whatever, they could still stay connected with us and what we’re doing.

John: How did you assess student learning in your class?

Rebecca: My classes are all project based. So the majority of grades are built on projects, not entirely, we had discussion boards, and I had some collaboration things that they were doing, and they were evaluated on those things as well. But projects were the significant piece of the puzzle. And the way that I graded them was really just providing feedback about the kinds of things I was going to ultimately grade very regularly throughout the semester. So every couple of weeks, they were getting feedback on their code for my web class, for example, feedback on their design, feedback on their writing, not a specific grade, necessarily, but feedback on all of those elements that were going to go into the final project. And then the ability to revise all of those again and again and again and continue to get feedback on those.

John: Did you have your students engaged in any reflective tasks?

Rebecca: Yeah, that’s a really great question, John. I had reflection built in two ways. So at the end of each sprint, or kind of module in my class, they were working on two projects at the same time throughout the semester, so they’d work kind of two weeks on one project, two weeks on the next project and cycle back… that was so that I had time to give them feedback regularly. So that was part of my structure. But at the end of one of those modules, I had a reflection activity that I implemented using a Google form. So a few different prompts to think about what they got out of that sprint, goals for their next sprint, that kind of thing. And then I also had some big group reflections at different moments during the semester, I had one at the beginning, and a couple in the middle and one at the end. And I use Jamboard for that, which is a Google suite tool that has sticky notes, and is the same kind of way that you might brainstorm. So I use it as a way to collect reflections in sticky note form, essentially, virtually. And I would have a reflection question for folks to respond to or a couple of different boards with different kinds of questions. In the beginning, we did something called “hopes and fears,” which is something I’ve talked about before… setting up the class like, what are they hopeful that they’re going to get out of a collaborative project? What are they scared about? We find out that like, all the teams have the same hopes and fears. During the middle of the semester, what are some of the big takeaways that you’ve had? What are some things that you want to work on? What are some things that you’d like to see changed about the class and various themes bubble up on that. And then at the end of the semester, I asked questions like, “What was your biggest takeaway? What was the thing you were surprised that you learned? What is one recommendation of something you would change in this semester?” and “What is something that you want to continue learning?” and I got really useful feedback on what to change about the class but also, some really great themes bubbled up across the class, which really results in like kind of three or four things for each of those questions, which was a nice way to wrap up the end of the class and summarize for students after they completed that task. And one thing that I like about the Jamboard is that it actually ends up being anonymous. You can see people while they’re working on it, but it doesn’t keep a name with a sticky, ultimately.

John: So you can see who’s active in the board, but you don’t see who is writing which note.

Rebecca: Yeah, exactly. So that worked really well for me. How about you, John, were you able to build in reflection? I know you have such a big class. So it can be tricky.

John: I wasn’t able to do as much of that with my large section. But I did have them do that, to some extent in their discussions. For one discussion forum in both classes, I had them use a tool called Packback, which uses artificial intelligence to give students some feedback as they’re writing their prompts. And each week, students had to post a question related to that week’s material and they had to respond to at least a couple other people. But one of the nice things about Packback is it will check the cognitive level of the posts, it will give them some feedback in terms of grammar, it will also do a little bit of checking to see if the material has previously been posted. And it gives students some feedback, encouraging them to say more than “I agree.” And it also encourages them to document sources and to provide resources or references for the arguments they provide. And they get a score on that. So it takes a lot of the evaluation of that away. And so I monitored all of that. But it was something that seemed to function pretty well, just by the interaction between the users and that system. I haven’t really mentioned much about what I did in my online class. My online class uses many of the tools, but obviously, I couldn’t do synchronous, because that class is fully asynchronous. I couldn’t do the same type of instruction. But I had students do two other things in that class that provided opportunities for reflection, one of which was I had them work in a metacognitive cafe, low-stakes discussion forum, where they reflected on what they were learning and the learning process. And that gave them another way of making connections to their learning and reflecting on how well they were learning materials and what barriers they were facing, and also sharing effective learning strategies with each other. They were given some readings each week, generally on research-based learning practices as a primer for many of those discussions… others, they were just reflecting on what they’ve learned and how it might be useful in their life to tie it back to themselves. But the other thing I had students do is work on two podcast projects in that class. And in those they were taking what they were learning and making reflections about how that connected to the world around them. Many of them ended up being related to COVID and pandemics, but they were making some really good connections, and they were getting a chance to see how the material they were learning had some relevance in their own lives. And a lot of that came out in some of the things they were discussing in their podcasts. And they also did use Jamboard once at the very end of the term. But I also use Google forms a few times to have them reflect on the process of what was working, what wasn’t working in the class and what was working and what was not working in their own learning processes and what I can do and what they could do to help them learn more effectively.

Rebecca: I don’t know about you, but I was really surprised at how well synchronous learning actually went for me. I had some technical difficulties early on with my internet connection. It took me a while, but I got around to fixing that problem by hardwiring my internet and resolving some of those things. But I felt just as connected to my students as I would normally. I had a lot of interactions. And in some ways, I was able to facilitate those interactions a little more equitably online, because it wasn’t just the person who came to nudge me and stand in line and be the next person. Instead, I could really coordinate using waiting rooms and breakout rooms and really give everybody a chance to have one-on-one interactions with me, which I really appreciated. And I really did get to know all of my students quite well, which I was a little bit surprised about. And then, in an area where it’s really technicalaAnd we’re doing a lot of coding and things on screen, being able to share screens and take control of another person’s computer to fix things or show them how to do something was incredibly valuable. We use some of those kinds of tools in person. But it actually was, I think, in some cases more effective using this particular tool. So I was kind of surprised at how well some things worked. And I think that even when things go back to face to face, there’s definitely some components here that I would keep.

John: I’d agree. And I think students were amazed at how well some of those tools work. When in breakout rooms, they would be using the whiteboard features, they would be sharing screens, they’d be making the case, they’d be drawing on the screens, and that was something that would be much harder to do in a face-to-face environment. Initially, at the beginning of the class, I had some issues with chat being kind of flooded with irrelevant material, and I had to clamp down on that a little bit. But within a couple of weeks, they started actually using it very productively, and it provided a voice for all students, even those quiet students who would have otherwise sat in the back of this large lecture hall. They were able to type something in chat, after thinking about what they wanted to say before doing it, without being concerned about interrupting the discussion that was going on. And I think that was really helpful. And when I taught large classes with three to 400 students, there’s almost always 3 to 10 Students who have trouble not having side conversations when there’s other activities going on. And that mute option is kind of a nice feature and the ability to set their microphones so they’re all muted unless they choose to unmute… to have the default being muted until people click the unmute option… made it really easy. And I was amazed at how quickly they adjusted to muting and unmuting. By the end of the term there was maybe only once or twice a class where a family member or someone else would walk into the room and start talking. And then they’d remember, they had to mute their mics, and it was very rare. In a class that large, I was impressed by it… and working with students one on one, during office hours, it was so much easier to have students just share the screen and show you exactly what their problems were then to correspond with them with email, or even have them boot up their computer or you try to find what they were talking about when they came to your office. It was just much more efficient.

Rebecca: Yeah, I could actually see it. You can Zoom in, you can see what they’re talking about. I also found, and I was really floored, in this last week of classes, students were doing their final presentations,at how well they did develop facility with these tools. They’ve developed a lot of fluency in the kinds of tools that are actually very relevant to my particular discipline. It’s relevant to many disciplines. But designers use these tools all the time when they’re working with clients. And so it was amazing to me that we got through 15 presentations so efficiently. We didn’t wait for anybody to share their screen. They just knew what they needed to prepare, had it ready, they started developing slide decks really effectively, and could just do the things that they needed to do really efficiently. One of the last things I said to my class was like “I’m so proud of you just being able to do that. We didn’t have to wait for anybody today. That was amazing.” And so maybe a little bit of a blessing in disguise, you hate saying like, “Oh, the plague is such a great thing.” But they really did develop some useful skills and tools and they became more effective communicators. That was something that a lot of students reflected on and things that they didn’t expect to learn is how much better they became collaborators and just communicators generally… not just in person, like through Zoom or in text… like through chat in Slack.

John: Video conferencing is likely to be a part of their lives in the foreseeable future, especially now that everyone has adapted tp this mode, it’s very useful for them to learn how to use that efficiently. The one thing I do miss though, is seeing their faces in person and recognizing them. One concern that I have is, I’m hoping to be back on campus in the fall, there may be students that I work with who interact with me regularly, whose voice I would recognize or whose name I would recognize on the screen, but whose face I just wouldn’t recognize because a very large proportion of students just didn’t feel comfortable having their cameras on regularly, and I understand that. We’ve got a lot of students living in crowded living quarters or working with very poor network connections. But I do miss actually physically seeing them. And I had my last class session earlier today. And I encouraged them to stop by in the fall and just say hello.

Rebecca: Yeah, I mean, I agree that the physicality is certainly something that’s missing. But it was amazing to me how connected I still felt to all of the students at the end of the semester. And I think that they felt connected to each other too and they verbalized that, and also wrote that in their Jamboard reflection. So although there’s much to be improved, given this was the first time out and an experiment in many ways. I’m really thankful that I read Flower Darby’s book about Small Teaching Online because that actually informed a lot of my practices, even though it was synchronous, and a lot of her material was about asynchronous learning. It really did help me remind myself of things that I already knew that I needed to do, but to kind of make a checklist of things that I definitely needed to do as I was rethinking my classes for the fall. So thanks for chatting with me, John.

John: We always end with the question: “What’s next?”

Rebecca: I am sitting down to reflect on what worked and what didn’t work, to try to troubleshoot some things for the spring. And I’m teaching a class that’s brand new to our curriculum for the first time in the spring. And so we’re developing it for online synchronous, although ultimately, it’ll probably be a face-to-face class. We’ve had to re-conceptualize some of the things that we were going to do because of the technology limitations that students may have. If they’re online, we’re expecting that we might have a lot of students who are relying on their phones versus software and having access to high-end software packages or computers that can run them. So we’ve had to rethink things. But I’m pretty excited about being able to experiment with my students with all kinds of technology in the spring, but it’s definitely a puzzle that I’m currently starting to work on. How about you, John?

John: Well, I still have a lot of grading to do. But once that is done, one of the things I’m going to be doing is converting a textbook I had written in econometrics to a Pressbooks site, which will be a lot of conversion because it’s originally in LaTex, a typesetting language used for mathematical typing and I’m planning to create a lot of videos, I’m hoping to get many of them done over the break so that I’m not spending 15 or 20 hours a week creating videos as I was all fall. And I’m hoping to get a little bit further ahead of the semester this time, so I’m not doing as much preparation at the last moment. And we’re both going to be working on putting together a series of workshops in January for our faculty to help people prepare for whatever comes at them this spring

Rebecca: We’re just going to be really busy.[LAUGHTER]

John: I’ve never spent as many hours working on my classes as I have this semester.

Rebecca: I agree. There was a lot of startup costs converting to this modality, but I’m hoping a lot of that stuff I’ll be able to keep and reuse moving forward. Thanks, John. Always nice talking to you, John. We chat all the time. But it’s nice to sometimes hear about some of the thought process and things behind some of the decisions that you’ve made in your classes. So it was really nice to actually hear about how you did some of that stuff this semester. So thanks.

John: And I also appreciate hearing more about what you’ve been doing in your classes. We spend most of our time on podcasts talking to our guests and only mentioning little snippets of what we’ve been doing ourselves.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

144. Pedagogies of Care: Evidence Based Practices

This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, Dr. Michelle Miller joins us to discuss how the use of evidence-based teaching practices can be an effective way of demonstrating that you care about your students and their success.

Michelle is a Professor of Psychological Sciences and a President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology research interests include memory, attention, and student success in the early college career. Michelle is the author of Mind’s Online: Teaching Effectively with Technology, and has written about evidence-based pedagogy in scholarly as well as general interest publications. She’s currently working on her newest book, Remembering and Forgetting in the Age of Technology: What the Science of Memory Tells Us about Teaching, Learning, and Thriving in a Wired World, scheduled as part of the West Virginia University Press series on teaching and learning, edited by Jim Lang. The tentative release date is 2021. She is also a contributor to the Pedagogies of Care project created by authors in this series.

Show Notes

Transcript

John: This week we continue a series of interviews with participants in the Pedagogies of Care project. In this episode, we discuss how the use of evidence-based teaching practices can be an effective way of demonstrating that you care about your students and their success.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Dr. Michelle Miller. Michelle is a Professor of Psychological Sciences and a President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology research interests include memory, attention, and student success in the early college career. Michelle is the author of Mind’s Online: Teaching Effectively with Technology, and has written about evidence-based pedagogy in scholarly as well as general interest publications. She’s currently working on her newest book, Remembering and Forgetting in the Age of Technology: What the Science of Memory Tells Us about Teaching, Learning, and Thriving in a Wired World, scheduled as part of the West Virginia University Press series on teaching and learning, edited by Jim Lang. The tentative release date is 2021. She is also a contributor to the Pedagogies of Care project created by authors in this series. Welcome back, Michelle.

Michelle: Hi. It’s great to be here.

Rebecca: Great to have you back. Today’s teas are:

Michelle: I am drinking fresh mint and hot water, which I think is my favorite summer tea of all when the mint is thriving all around here at the house.

Rebecca: Sounds nice and refreshing. How about you, John?

John: I’m drinking Tea Forte black currant tea.

Rebecca: And I’m drinking Scottish Afternoon. I haven’t quite run out of that yet.

John: We’ve invited you here to talk about your contribution to the Pedagogies of Care project and your forthcoming book. Could you start by talking about your contribution to the Pedagogies of Care project?

Michelle: Right towards the end of the spring semester for many of us, as you know, we in the teaching and learning community and professional development and scholarship of teaching and learning space, were in just vibrant discussion with one another, just talking each other through the experiences that we were having as part of the pivot to emergency remote instruction, which I think for most of us in higher education, that was a big part of what we did in March all the way through May of 2020. So we’d been talking about these and there’s this very vibrant group of authors that have come together under the West Virginia University Press’s project, as you mentioned, edited by Jim Lang. And so we had this group, which was already exchanging very rich sets of advice and ideas about where we were going and really talking about how to help. And so under the leadership of Tori Mondelli, who conceived of this whole project, and also Tom Tobin, who has also been a real leader as part of this group, we talked about how can we put together some resources that grow out of the work that we’re doing, that capitalize on some of the rich conversation and collaboration that’s already happening, and whatever format that takes, put that out there into the world, so that people can use that and there’s all different ways that it could be utilized. We’re not prescribing that but we really had envisioned something that was open, that was helpful, and that was really contextualized within this moment of real upheaval and crisis and new directions that many of us are involved in.

John: We’ve gotten some really good feedback. I shared that with the faculty at our campus just a few days ago and I got about a dozen responses within a couple of hours saying “These resources are really useful. Thanks for sharing.” We’ll include a link to that in the show notes. So, we went through this traumatic switch that was a bit of a struggle for everyone, students and faculty, what can we do now to better prepare for the fall?

Michelle: At the time that we’re recording this, we are, for me, about midway through the summer. So, it really is starting to get real, for many of us, what we are going to do in the fall. And we’re seeing more and more institutions who are firming up and starting to commit to real plans for what the format of instruction is going to be like, what enrollments are going to be like, and all those kind of locally specific pieces of information that are so important for determining what we’re going to be able to do. So, what can we do differently to better prepare for the fall semester? First of all, let’s honor that what the vast majority of faculty that I’ve talked to, what we accomplished in such a short space of time in spring, providing instructional continuity. This was amazing. I mean, we really enabled students who, in some cases, they were set to graduate, they were earning their degree in maybe a month or two, and we made it possible for them to get to that finish line through a tremendous amount of ingenuity and hard work on everybody’s part. So, let’s not sell ourselves short. That said, we are headed into a very different environment. And so what I’ve really suggested in some other things that I’ve written about and definitely in my Pedagogies of Care project is a focus on what does quality really look like? And for me, being a cognitive psychologist, social scientist, totally acknowledging that that’s my perspective… forr me that comes down to aligning with the best of what learning science has to offer. And the neat thing is that we are in an era right now when number one, we really have converged on a set of principles that are fairly non controversial, and if not always easy to implement, it’s fairly clear what we can be doing. And we have technologies, in some cases, that map onto them very well. They don’t do the work for us. But they can really help implement things and make things concrete that we’ve known in theory for a long time were very, very important. So, that’s one of the things that I think that we can focus on. So, there is that. I’ve also really emphasized the reevaluation that we won’t be able to simply do what we’ve always done. I think those of us who work in this space are always quite adamant that teaching, say online or teaching a hybrid course, is not a matter of just sort of capturing a lecture. If that were the case, this would be very, very straightforward. We should just lecture all summer, record it and post it, but that’s not what it’s really about. So, what I think that we can focus on as we do reevaluate, in our teaching, what are we trying to accomplish? We can step back and say, “Well, what do students want to get out of this?” And that I think can help us winnow down from all the things that we could potentially do. It will help us let go of some things that we will not be able to do. And help us find, if not an easy path forward, a more clear one that will allow us to serve our students and also take good care of ourselves during this time.

Rebecca: I think anything that helps us figure out what our priority can be, in terms of content or goals that we have for students, but then also methodologies that we’ll use and why, I think is key because I think we all need to scale back and be reasonable with ourselves because there is so much to accomplish if we want to do it perfectly. But we just don’t have that kind of time. You just said it was halfway through the summer and I almost had a panic attack.

Michelle: Right. Not that I’m counting but it is actually just about the midway through the summer. And you, know, when I started reflecting even more on this Pedagogies of Care concept, which is the kind of overarching ideal that we eventually rallied under as a group, it’s occurring to me that that applies to faculty as well. I mean, self care is a kind of a term that’s very cliched, and it gets kicked around, but I think that we also really do at this time need to be recognizing that, again, what we did, what we accomplished as faculty in the spring was tremendous, that it did require people working weeks and weeks and weeks, sometimes months without a break. And although summers are not really traditionally a break,or vacation for faculty in any conventional sense of the word, they are a time to recharge and for many of us were also taking care of research obligations and other things that went completely by the wayside for a while out of necessity. So we really do have to balance that too. What’s the degree of faculty burnout at this point? What’s the degree of faculty receptivity to brand new things. So, the things that we are looking at also need to be kind to ourselves. We need really good communication and collaboration more than ever before, I think, in university communities. I think that’s really also the thing that’s going to make this fall successful, is being able to recognize what faculty have been through and work with that. So yeah, I think that we should recognize this effort. And with that, I also think that evidence-based teaching, incorporating learning science and those principles… that ideally shouldn’t be yet another thing on the to-do list. I think that if that’s the way it’s coming across, then we’re going about it the wrong way. I mean, to me, frameworks are always a way to simplify. Again, we have this infinite landscape of things that we could do in any given class, all these different decisions to make and choices. We do have a framework for whether it’s learning principles or another framework… that should help and simplify. So I think it kind of fits in that big landscape of possibilities as well. That’s how I see it. It should help; it shouldn’t add to what’s becoming a pretty serious burden for faculty.

John: One of the things I’ve really liked in your discussion, as an economist, is you sounded at times, like an economist, when you were describing that, in terms of this is the most efficient way of helping students reach their goals… that if we use evidence-based methods of teaching, we can let students learn skills more efficiently without wasting as much time and getting closer to that point, making it a form of caring, I think, as you referred to it. That one way of demonstrating your care for students is by using techniques that are more efficient, that provide the largest return on students’ time… there’s the economics part coming in. So I really appreciated that. And I thought it was a really good argument that we tried to emphasize ourselves in our workshops.

Michelle: Oh, thank you. And you said it better than I possibly could have as a non-economist, but that’s exactly the core of that idea, that it is kind to students and perhaps it’s kind to faculty as well. We can pre-select some of these avenues and techniques that, if you’ve got an hour to study (and for many of our students, that hour of study might be fractured and jammed in among all kinds of caregiving tasks) that you’re going to get more from that. If, as a faculty member, you’ve got four hours that you can devote today to preparing for the fall… and as well, that’s going to be divided up among other tasks among your caregiving responsibilities… how can we cut to the chase for faculty so that they can make those choices? So I’m glad that that comes across.

Rebecca: I think it’s important when we are planning for the fall that we are getting down to those essential elements. Can you talk us through some of the steps that faculty might take to focus in on those essential items and the evidence-based practices so they can have a good framework moving forward, not just for the fall when they might be teaching remotely, and that’s what they’re not familiar with, but all the time?

Michelle: Coming down to essentials, and here too, I think, that that has really resonated with many faculty and also with instructional designers and others tasked with making all of this work. That’s what’s really resonated, like what are some of the essentials, and I’ll never claim to be able to I Identify the complete and exhaustive list of exactly what to do. But here’s what comes to my mind. I think that perhaps returning even to those learning objectives, which we may have put in a syllabus long ago, and they can be sometimes kind of abstract, but coming back to those and saying, alright, what does it really look like when students have achieved these? Are there any that need to be perhaps modified, or dropped altogether? So if we are going to have a semester of really focusing on essentials, this might be a good time to do that. Naturally, we will want to think about the content. And oftentimes we talk about in pedagogy and developing pedagogy, we talk about re-focusing away from just coverage of content, that’s something that a lot of us get behind. And it’s okay to be thinking about well what content is going to be in the course. But then really pivoting to look at what’s the engagement with that content? How are the students going to engage with the content and how are they going to engage with you? So that’s a piece of it, asking yourself that question. And I think then, starting to bring in those really concrete logistics. Now, again, typically those of us who talk about pedagogy a lot, we kind of discourage people from talking about very specific tools or technologies, until they’re really, really clear on some of those high-flown ideals of what they and their students want to get out of the course. But I think in this case, we probably want to hold off on th.t, we are going to have to say, “Well, are you going to be expected to teach online but synchronously? And if you want an example of that, the Zoom meetings, which we’re all pretty familiar with, at this point, where we’re in at the same time, but maybe you’re in a different place? So is that going to be a part of what you do with students? Because that is pretty new to many of us. And if so, there’s certain considerations you’re gonna have to have in mind say, ‘Well, how is that going to work?’” Especially, if you’re expected to also be teaching say, a face-to-face course at the same exact time, which I think is going to present challenges. And I think for many of us, it’s going to depend on your local institutional context, but I think you can’t go wrong right now with setting up a robust online component to your course. I think that with the level of uncertainty we have, or even with individual students… if they’re going to need to say quarantine or take care of an ill relative or something like that… having some asynchronous, so different time activities and materials online, is going to be essential. So I think taking those concerns and saying, “Alright, what is this physically going to look like?” I wouldn’t typically push that as much but I think that that’s important now. And I think in the preparation for this, too, another kind of bare essential point that I talk about in my resource for our project is media creation. So in some cases, people are going to want to create, say, a set of videos, or let’s say they’re demonstrating a process. Let’s say they’re teaching studio art. They might want to have some pretty involved videos or other kinds of demonstrations, or perhaps there’s not good written material out there that might replace a series of face-to-face lectures. Maybe they’re going to be wanting to write a fair amount of content or maybe record, even, podcast-style materials. That stuff eats up a lot of time. So I think really being real about what you absolutely need to do in that department and getting started now, that’s sort of the wisdom of experience that I would share with folks as well.

Rebecca: I think that’s really good advice, Michelle. As I’m thinking towards the fall, I made a list of “this is absolutely essential… if I don’t have this content made, we’re screwed if we’re online,” versus like, “this stuff does exist out there that I could use…if maybe isn’t my favorite.” And then there’s well established stuff that’s fine or whatever. Because it does take a lot of time to write, produce and plan some of that stuff… even if you’re using methods that aren’t burdensome, where you’re not worried about production quality and those kinds of details. It still takes time. You need quiet space. There’s a lot of constraints, especially if you’re like me and you have kids at home. [LAUGHTER] You got to find the quiet time to record the thing. [LAUGHTER] So I appreciate the balance there… really thinking logistically a little bit. Because if you have a finite amount of time, then you have to prioritize what can get done ahead of time.

Michelle: Right. And you know, it may not be the way to go. And I though I’d share with you an experience that I had, well, right in the thick of the great pivot, the transition to remote instruction. I was talking to a faculty member who does happen to teach studio art. They teach drawing and painting in a small-class atmosphere, a very intimate atmosphere that’s very hands on… and not somebody who works at my institution. I happen to know them. And she called me up partway through the great pivot week and was distraught. She was really on the verge of tears. And she was saying, “Well, this goes live next week, I need to somehow carry my course forward, my studio art course. And I just learned that my colleague, the guy down the hall, what he’s doing is he’s got these videos that come down from the ceiling, and then we have these close ups on drawing and these techniques and he’s doing all this. I can’t do this. I’m a single parent. I’m at home. I’m overwhelmed. I don’t know what to do.” And I said “Alright, it doesn’t have to look like that. Your colleague may be doing that. It doesn’t have to look like that.” And I said alright, what is working in your course? That’s another thing you can use to kind of cut to those essentials. So what is the strongest thing? What do your students need right now?” She said “Well, they’re absolutely overwhelmed and I think they need a lot of support.” And “Well, is there any kind of social peer-to-peer support?” And she said “Oh, well, we have since the beginning of the semester, I put them into these pods of three. And so they’ve been developing these social structures where they consult with each other every week. And so they have ways of communicating with each other in these pre-existing social groups. Do you think that could be useful? And I said “Yes, go with that.” So what your course is accomplishing really well right now is setting an atmosphere where students are talking to each other and I said, “Well, maybe you can kind of divide and conquer. You can hand off this project to where students are critiquing each other’s work in these groups. So, definitely kind of double down on that arrangement that you’ve already put into place. Your colleague down the hall, maybe multimedia is his thing and this is easy for him. But he may be struggling to say how do we get students to socially support each other form connections and feel connected to the class, even though it’s now in a remote format.” To me, that’s something to really capitalize on. So I took away a lot from that and I’ll be reflecting a lot on that as well. Your “solution” to the challenges we face is going to look different and it really should go with whatever is strongest for you. I think as academics, we kind of say, “Well, if it’s easy, that must be the wrong way to go about things.” But sometimes the path of least resistance maps well and aligns well onto what your strengths happened to be and what your students needs are.

John: Going back to that point, though, about creating media. If you create materials for an online format, you can always use that to support face-to-face if by some miracle things return to some sense of normalcy, it’s probably not going to, but that material will still be there and will be useful. So, a focus on that, I think, is really helpful. And that’s what we’ve been strongly advocating for our faculty as well.

Rebecca: Just as long as you don’t have specific deadlines… don’t put deadlines, dates or anything like that in them.

Michelle: Right? See, that’s just a practice that is so important to create reusable media. And it’s a seemingly small thing, but until you really get into this and get practice, you don’t realize how important that is… that yeah, if you are going to sink the time into that, make it reusable. And that’s an important point for reusability.

John: And going back there, I’d like to once again, we’ve done this many times, recommend Karen Costa’s book on 99 Tips for Creating Simple and Sustainable Videos. It’s a really nice resource. And it does focus on keeping it simple. Don’t do the fancy transitions. Don’t do something where a half an hour video is going to take you 30 hours of production time. Keep it so that it’s easy for you so that you can keep doing it without imposing a burden that’s going to make you stop doing this.

Michelle: Absolutely. And I’m so glad for that recommendation. I went out and got the book myself. I think I’m on Tip Number 80 as of this morning, so I’m almost there and I’m finding these wonderful… everything from very specific guidelines to much more conceptual things about why you want video in a course to begin with. So yeah, I’m with you on that. It’s definitely worth a read and definitely this summer. But maybe also, to kind of put this into a different focus as well with the focus on creating media and doing so purposefully in a way that is sustainable, let’s not lose sight of the active learning component. So that’s something that I’ve really kind of watched with some concern and definitely some interest as this conversation evolves. So active learning at this point, I mean, people sometimes perceive it as a buzzword, but it is such a robust concept. And I think it’s easy, at a point where we are kind of saying, “Well, how can we make all this work in some different formats” to lose sight of that. And so we may be creating wonderful videos, instructional videos, or all kinds of things and just merrily perking along with that, but we do need to remember how are students interacting with it, which is why a beautiful film of somebody demonstrating a drawing technique might, in some context, not even be as valuable as somebody who’s having students talk to each other because of that engagement. So I think that too, this is going to be so critical as we see more schools pushing for things like recordings of lectures, or even synchronously bringing students in during a live session you’re having with other students, I think that we do need to remind people who are in charge of these things, that education is just never something you watch, it is something that you do. So it is really tempting to say, let’s record everything we can, that’ll be equivalent, but active learning is not a luxury that we can just put on hold for a while. It really isn’t. And so I’m hoping that we don’t see that happen. I think there’s a very similar story that’s going on with Universal Design for Learning. Another concept I know you’ve engaged with so much on this podcast and is so important. And I think you’re too, it’s easy to say, “Well, you know, given all this going on, maybe we won’t have multiple ways of engaging with these great media that we’re creating, or maybe we’re going to kind of shut down this avenue over here for a little while.” And I really hope that doesn’t happen. So that’s another aspect of this balance between the quality and ambitiousness of what we’re doing and the feasibility and protecting ourselves as we face another very challenging semester.

Rebecca: I think that’s a really good reminder about focusing on the learning as the essential element as opposed to the teaching. It’s really about setting up the framework and the possibilities for students to learn, and designing those activities and making sure that we’re spending the time on that, rather than all the time on just delivering something.

John: But having those videos can free up time so that if you do meet synchronously, you can engage in more active learning activities rather than just lecturing to students online, which is probably one of the worst ways of structuring synchronous meetings. And if you really want to do a little bit more work, you could use something like PlayPosit where you embed questions in the middle of a video that could be somewhat open ended and that you could even grade. If you happen to have an institutional license you can embed it directly in your LMS. So the videos themselves can be made, with a bit of work, a little more interactive, and they can serve as a replacement for lecture that allows for more active learning, I think.

Michelle: Absolutely, and I too. I’ve seen some wonderful examples in practice of that technology, and there’s a couple of different ways to do this. So there’s multiple tools that allow you to put a retrieval practice or comprehension questions somewhere in the midst of this online lecture, presentation or video and what better way to help ensure that students are attentive to them, to give yourself some opportunities on the other end to say what’s the actual level of comprehension that’s going on out there. And for students to really solidify and practice the material. That’s all bedrock learning science stuff, right? Retrieval, active practice, and so on. And it just takes a little bit of ingenuity to take that one extra step to say, alright, what’s the level of interactivity here. And that’s something that I hear too, from faculty, it’s quite reasonable. They have taught purely face-to-face and don’t have that level of first-hand experience with something like online teaching. It’s just like, “Well, how do I know what’s going on out there?” And, again, there’s not a technology that’s going to just magically replace the experience of looking at the sea of faces that we experience in a face-to-face class. But think about it. That’s one way to do it. Having something like an online gamified quiz, like Kahoot!, which is currently my favorite quizzing app that’s out there. I ran this just the other day quite successfully in a remote synchronous environment. So, there are two that could help give you that information right away about what concepts are they struggling with. And having other ways of reaching out to students, if not talking to them individually in something like a meeting, a phone call, or even a text chat, having some other ways to kind of figure out on the ground what’s the mood level of the course? How are we feeling about things and are there individual students who are struggling for one reason or another who we can reach out to?

John: One way in which I saw interactive videos being used was several years ago, I took a MOOC on behavioral economics that Dan Ariely had put together and he’d often discuss experiments, but he set up the experiment and describe what the experiment did. But then the video pauses, and you’re asked to predict what the outcome would be. And that type of prediction is a really useful evidence-based technique that you can even do with videos if you can embed the questions in the middle of them. And I thought that was really useful. And it’s something I’m going to be trying to do a bit this fall. But in terms of evidence-based learning, could you talk a little bit about some of the main principles that people should be using to design their fall classes? What should people be focusing on?

Michelle: So, when I talk about bringing down just a vast literature of learning science and I’m going to necessarily boil this down to what I think are my favorites and the most applicable… So, of course, retrieval practice,I think if there’s one success story that our field has had, I mean it goes back even over 20 years that we got the data, determined how this principle works and started flowing it out to practitioners in the field, it’s this one. So that is, of course, the principle that when we actively pull something out of memory, it increases our ability to remember it in the future. And of course, we’d naturally think of tests, exams, and assessments as the avenue for this, but there’s lots of other ways that it can take place. So I always love to direct people to the website retrievalpractice.org. I’m not affiliated with it, but I think they have a wonderful compendium of ideas for how to bring this into classrooms at all different levels, all different disciplines, and so on. So if you don’t have retrieval practice, quizzing, students actively talking about what they remember, great time to bring that in. So you can’t go wrong with retrieval practice. Then, of course, the principle of what’s the timing of your study. So, spaced study, and pretty much by any measure, when we spread out student engagement with material… again, whether it’s through quizzing or solving problems, you name it, you’re going to get more out of that… efficiency… when it is spread over time. And I think that this is one of the real unsung benefits of online and technology assisted learning, even among people who are saying, “Oh, I’m just using the basic learning management shell to organize some materials and students turn their stuff in online. I mean, let’s not sell that short for how powerful that is, for being able to stagger deadlines, change the timing of when we are getting students to be working on different aspects of the course and so on. So while we don’t necessarily always want to bombard students with deadline after deadline, we do have to be mindful and help them kind of organize multiple deadlines. This is something that we could definitely build in as a design principle. So just to be very blunt about it, we always discourage people from the two midterms and a final course design. That’s something that a lot of us have experienced. It could work of course, like that can be fine. But from a memory and learning standpoint, that’s really not ideal. We want students engaging quite frequently. And then the practice… so the practice of this skill. So that advice, bring that up again, about it’s not all about content coverage. It’s about practicing the application of the content knowledge that they’re getting. We can almost always stand to build in more of these, I don’t think I’ve ever seen a course where I’ve said, “You know, you really need to present more content to the students. Don’t have them solving problems so often…” I have never seen that in practice, I will just go on the record and say that. So, if we want students to be doing X,Y, and Z. And again, go back to the front page of your syllabus and remind yourself what you’re hoping they’d be able to do at the end of the course. We want them to do that, what are the opportunities for them to actually try, and try in small bites? In my contribution to the Pedagogies of Care project, I give a very brief example of this in my own courses. So one of the things you have to do… bread and butter skills as a psychologist… is you have to be able to look at a psychology research study and kind of break down the structure of it. So no matter what’s being studied, there’s probably… we call them independent variables and dependent variables. So, things that are being manipulated, things that are being measured, and students have to develop that as a thinking skill and it’s really not easy. So I will oftentimes have them in, say a research methods course, very frequently, as part of whatever we happen to be doing, I’ll say, “Okay, here’s a really short description of a study. Maybe it’s an abstract or just a description, you pull out from me, before we talk about anything else about this study, you tell me, what are the independent variables? What are the dependent variables?” So it’s something that traditionally we’d always put on an exam. But, we didn’t always have students repeatedly practicing. So knowing that students absolutely had to master this before they got out of my research methods course. That’s what I did. So practice, and that kind of segues back into that active learning principle, which…yeah, you cannot go wrong with students getting involved. Once again, I don’t think I’ve ever seen a course where I said “You, the professor, need to get out there front and center, don’t emphasize the students so much.” So, they need to be doing the thinking, the practicing, and quite frankly, the work. That’s where the benefits come from. So with those: the retrieval practice, spaced study, practice of higher-order thinking skills, and a real active learning orientation, I think that that’s something you can take to the bank as a faculty member. You could build on that, but if you start with those, you’re probably going down the right path.

John: And I remember reading this really good book that talked about how using computer mediated instruction or using the tools within the LMS allows you to provide students with lots of feedback and lots of retrieval practice without necessarily increasing the burden on you, as the instructor. I think that book was called Minds Online: Teaching Effectively with Technology. [LAUGHTER]

Rebecca: Yeah. I feel like I might know that author, I’m not sure.

Michelle: Yes. [LAUGHTER] And thank you very much. That’s what I was trying to go for. So, thank you. It is wonderful that people are finding many of those points really relevant right now. So, yes, thank you so much for pointing that out. I think it’s great. [LAUGHTER]

Rebecca: I think one thing that I’ve been thinking about in terms of having more remote time then maybe in-person time is that I often provide a lot of structured activities around retrieval practice and spaced practice in my face-to-face class and if students are working more independently when they’re working remotely, I’m not there to [LAUGHTER] facilitate it synchronously, that structure needs to really be in place, maybe even more so than when you’re in face-to-face class, that they have that structure and that they know they should be doing those things on a regular basis. Of course, we should be reminding them to do these things on their own as well. But, I think focusing a little bit more on having that structure or those reminders in our courses, when they might be remote is actually really, really imperative,

Michelle: Right? And those are learning skills and abilities and principles that are going to serve our students well, no matter what they study or what they may do after they leave a course. And it’s kind of neat. There’s some indication from the research literature that particularly for students who come in who are not from advantaged backgrounds, that when they’re exposed to courses, which as you say, they remind them, “Okay, do this kind of practice. Here’s what you should be doing. Here’s why you should be doing that” …that benefit really does extend not just into that course, but into future ones because students can pick these things up on their own. So, if we do really want to be thinking about how can we set our students up for success no matter what the future holds, I think that’s a pretty high ideal that we can work towards. So yet another reason to incorporate these powerful practices and perhaps, yeah, to talk about how students can adopt them, no matter what.

John: For those faculty who are struggling to prepare their courses, what are some heuristics they could be using in terms of focusing their time where it would give the most benefit.

Michelle: This is something that has definitely been on my mind, both for my own preparation and to share with others. So heuristics, shortcuts, and helpful hints and approaches. So, I talked earlier about looking at what you consider to be your strongest points as an instructor and kind of the highlights of the course… the things that you know, are memorable, that advance learning that you feel really strong and competent with, with the caveat that, yeah, we do want to make sure that those do align with student learning. I think that that’s a great place to start. Say: “Okay, what’s the great parts of my course? Forget about what anybody else is doing. What do I really want to use?” And putting those front and center. If you have a short activity that’s working great, maybe that’s something that could be done every week, or somehow extend it. But the flip side of that is this, and this is another that I didn’t invent this… This is something you’ll see repeated time and again, in teaching advice, which is the pinchpoint heuristic, flipping it around and saying, “Oh my gosh, if there is one thing that students are struggling with conceptually, or it’s something that I know they should be doing, and they don’t do it to the level that they need to,” that you focus your efforts, kind of train your sights on that piece of it. Especially in the discipline. I teach, psychology. I mean, there’s so many fun things we could talk about with psychology, and it’s easy to kind of spend a whole lot of time and effort shooting the videos or setting up the learning activities online and making a quiz that’s about something that’s just cool to learn about. But that can’t squeeze out “Oh my gosh, everybody gets unconditioned stimulus and conditioned stimulus wrong and they do it every single year, and I know it’s going to happen.” So I need to be pulling out those things. You know what, if I’m going to spend the hours on a video or an extra module or creating an interactive quiz with multimedia, spend the time on the places where students are struggling. People who work with UDL, also talk about, “Well, here’s where you want to be especially conscientious to ensure that you do have the multiple means of representation and expression is around these areas that are really, really tough for students.” So what’s working great, where’s the point where you just say, if I could wave a wand and make one thing happen, that’s what I would do. So really looking at those two tracks. So that’s one heuristic. I think, as well, I’ll share with you something that I’m working on for my own courses… big caveat, that this is my courses. I will probably not be teaching a very large set of classes just because of the vagaries of course assignments. So I know I have that a little bit easier. But, here’s what I’m going to do as a framework. I’m kind of thinking of splitting it up so each week, students have a set of kind of general categories that they have to meet, they have to do some type of work or meet some kind of expectations in that area. So, I might, for example, have a column that corresponds to engaging with classmates about the topics for the week, and then a set of options for that week. So maybe you came to a face-to-face class, maybe you participated in an online discussion. And maybe there’s even a third option that I haven’t thought about yet. So just to really simplify things, I say, “Okay, check off in that area, what’s another column or category that you have to participate in, you have to do some type of demonstrating mastery of material” or I’m not quite sure what I’ll call it yet, but that could correspond to taking a quiz or maybe playing a Kahoot! in class or playing a Kahoot! remotely online. And I’ll probably also have a column that constitutes working towards whatever the term project is, and I’ll give them a set of choices again of what that term project can be like. But I am a very big believer in if you’re going to have a big project that there’s lots and lots of formative steps to that. So I tend to take that to extremes. And every week or so, students are doing something to show that they are moving towards and making progress in that area. So it is still a little bit general around the edges. But, to me, that really helped me feel like I had a handle on how am I going to manage choices? How am I going to manage multiple formats, and manage uncertainty with that focus on the purpose? Why do we have this do this week? Well, because it falls into these different categories, all of which are important for your learning in this class. So, those are a couple of the shortcuts that I would share.

John: One of the other things you talked about in your contribution to the Pedagogies of Care is the importance of getting help when you need it or where you need it. Could you talk a little bit about that?

Michelle: So this whole idea of getting help, I mean, it’s very simple on the face of it. I’m a faculty member, I want to do this thing in my class. I don’t know how… I call somebody… magic happens. And in reality, in higher education, what I’ve seen over and over at different institutions is that that is not a direct path at all between support, assistance, and collaboration, and the faculty member and the time and place when they need that. And so I think that this is going to be an issue that, if it’s not on people’s minds now, in leadership and pedagogy circles, if it’s not on their minds, now, it will be in six months to a year, I think that this is going to be one of the differences between institutions that make it through this fall in good shape and those that really struggle is what are those processes? So for faculty members, I’m really encouraging them to say, “Alright, where are the points, in this process, where you could get some kind of assistance that either you invest some time and you get the capacity to do something very efficiently in the future, say, like a workshop on how to do sustainable videos, or how do you actually find somebody to share the load? …actually delegate some of the work? For faculty, they should be reflecting on that, but at that point, that’s where things are going to get complicated depending on what the systems are in place at their institutions. So first of all, I think that institutions don’t always, and faculty ourselves, we don’t always make that distinction. When I say I want help, do I mean, I want you to point me to a great website or a book I can read? Do you want me to spend half a week coming to a workshop series? Or are you going to get in there and say, “Okay, you have the content, I can build these quizzes, you have a script for what you want to do for a complex video, I can shoot that for you, caption it, and put it online.” So what kind of help are we talking about here? And then figuring out how do you approach your institution to do that? So I’ve just really been continually surprised as I do visit different institutions. I mean, almost universally there are these amazing instructional designers and other people who just devote their professional lives to teaching and learning. They’re up on all the new technology. They know what was the great new video editing software that just came out last week? You know, they’re the ones who have that. And oftentimes there’s a disconnect there. People don’t know how, they feel inhibited, or maybe they’ve been actively inhibited. Some institutions, they say, “Well, there’s a process, and we’re going to put a lot of strings on how we’re going to divvy up these resources.” Others actually discourage instructional design and similar staff from even talking to faculty. And there’s a little kind of social piece to it as well, I think, just because we haven’t yet fully incorporated this into what we do… that it’s almost like, well, who makes the first move? If I’m an instructional designer and I know, here’s these courses over here that I could be helpful with, you know, just email people out of the blue… and likewise, faculty, they say, well, should I call the support line for this more complex project that I need help with or not? So I think that institutions will hopefully be sorting that out, but presuming that there isn’t a giant revolution in how we have collaboration between instructional designers and faculty, being aware of that and at least having something very clear in your mind for what you’re asking for, the worst that can happen usually is that somebody says no, but to have any chance you at least have to know what specifically do you want.

Rebecca: I think knowing that’s really helpful too. Because if you start talking to faculty, for example, in other disciplines, they might have a similar goal or they need similar structure in place, you could actually work with those faculty to put the structure in place and share the structure, swap out the content or whatever too. Sometimes we don’t think about those kinds of collaboration.

Michelle: Right, and what you’re describing, that’s something that is kind of non-traditional and new. We come into this with a very strong tradition of “my class is my class” and a kind of an ethos as academics that you do things the hard way, and you do them by yourself. But maybe this can be an impetus for us to really be getting creative with swapping, even things like a syllabus. You say, “Well, you know, maybe the way that I’ve gone about this, you can actually springboard this even if it is, as you said, in a different discipline.” Maybe we’ll even see faculty putting together some more unconventional team teaching arrangements. Traditionally, we know a team teaching is we’ve we’re experts in the same subject. And we’re going to create this class that sort of articulates, or we’re going to pass it back and forth. But maybe I should be collaborating with somebody from another area of psychology. Do they have to be in my sub discipline to just come in and say, help me with discussion forums, if I’m not very good at that, and then I can come into their class and help them with synchronous video, if they need help with that. Maybe if we have to, we will do it that way. So if that comes out of all of this, I think that would be a great benefit. And I want to say I have been really hesitant and cautious about engaging in this narrative of the silver linings and “Oh, isn’t this a wonderful experience? We’ll learn all these new methodologies of teaching will come out of this and we’ll all love online teaching and be fluent with it.” I don’t think that that’s an appropriate message for faculty right now. I think we do need to recognize that this has been somewhere between disruptive and catastrophic for most of us career wise, and not imply that we should all just constantly be thrilled to be learning new things. There are so many new things that we could be learning right now. But fall is coming. And we only have so much time. So I do want to put that out there, and that’s something that I think is an important thread that needs to be, and I hope it will be, talked about more as the dialogue unfolds. But even without saying, “Hey, this is a great time to do new things,” we can recognize that there will be innovation that happens, and it’s already happened. We’ve seen it happen.

John: And while this may not be a silver lining, I know in our teaching center, we’ve seen a lot of faculty who I didn’t even know existed on our campus, because as Jessamyn Neuhaus has talked about, people have broken down some of those barriers where they think they have to do everything themselves, and they’re more willing to request help when they desperately need help in ways that they weren’t willing to do before.

Michelle: Absolutely. I think that Jessamyn Neuhaus has been such a clear and fresh voice on some of these development issues. She’s absolutely right. She talks about it in her own style, which is totally unique to her, but it really gets it across, that we’re Professor SmartyPants, and we are not used to collaborating, working together, or just saying, “I don’t know.” So I guess we can also say, even if we don’t formally work in a teaching and learning center, if there’s something that you know, that your colleague does not, and you can help with, get out there, volunteer it, and let’s all really do this in perhaps a new spirit, where it’s not all just about, “Well, here’s what I know and you don’t know it, and I’m gonna feel uncomfortable coming in,” let’s have a real reset in terms of really open sharing. It’s not about playing the game of who knows more, or who figured out the latest thing. It’s really about serving the students and doing so in a way that we can sustain what promises to be a pretty challenging semester.

Rebecca: These have all been really great tips and things to think about as we move towards the fall, as the fall moves towards us… maybe that’s a better way of thinking about it. [LAUGHTER]

Michelle: I think that’s a frighteningly accurate turn of phrase there. [LAUGHTER]

Rebecca: I want to make sure that we get to talk a little bit about your new book, though, can you share a little sneak preview?

Michelle: Oh, sure. And this book, of course, well predates the era that we’re in. But it’s been something that I’ve wanted to write about for a very long time. And then when I was able to make the connection to James Lang and to his series, I think it was really meant to be. So, it is about memory and technology. So, much has been written in the popular press, and a little bit in the scholarly press as well, about cognitive processes and how those change or not in the presence of technology and with a frame for teachers, of course, so those of us who want to make up even just very specific policies, like should I allow note taking in class on laptops or not, to people who are really interested in this broader sense of teaching and learning in our contemporary era. So what I’ll be talking about in the book are issues such as well, first of all, what do we need to know about how memory works in the first place as a teacher or a person who is really into learning. So what do we now know about how memory works and how it can be improved? I also talk about why anybody should even care about memory, because that’s one of the angles of technology as well… this question of “Well, do you really need to know anything in the age of Google?” And there are people on both ends of that spectrum… probably no surprise that I come in somewhere in the middle of saying, on the one hand, it’s really important to be able to find information when you need it. And yes, we absolutely should be de-emphasizing memorization for its own sake. However, we also know from current research that memory in a subject area helps us think in that area. So there’ll be something for everybody in that section of the book as well. And then we will talk about what is the effect of having something like a smartphone, always at our fingertips? Does that create any kind of global change in memory? Does it change our memory for specific things that we might be doing or thinking about what we’re using that technology? And how, again, can we turn this to our advantage as lifelong learners ourselves and also for our students. Now, of course, you can’t talk about any of this without talking about attention itself. And so while it’s not a book about attention and distraction, per se, we’ll talk about “Alright, well, what’s the flip side of that?” And so how, basically, can we take all the advantages that technology has to offer for building memory and de-emphasize all the things that it does to offset and degrade our memories, and come out of this with the best of both worlds? I will get into a little bit at the end of the book as well into some of these bigger questions of how is memory itself changed when we live in a technological era when so much of our lives are recorded? And what does that say about things like generational differences, or what memory might look like decades from now? So I’m absolutely loving exploring all those themes, and I think they’ll be interesting for anybody who’s in the arena of teaching and learning but also with a lot of practical tips about again, how we can reap all the benefits that technology can offer for memory and for learning.

Rebecca: You’ll have a lot of disappointed listeners to know that that doesn’t come out until 2021. Right?

Michelle: Good things take time. And yes, we will see. It is a work in progress. And although we definitely have all the themes and all the ideas nailed down, it’s something I’m working on as we speak. So that’s part of why I’m so excited about the project. But yes, I got to finish it first.

Rebecca: We’re definitely excited for it to come. We always wrap up by asking what’s next?

Michelle: I am, as many of your listeners probably are, when this comes out, absolutely in the thick of redesigning my own courses for fall. Without getting into too many of the specifics, my institution has kind of laid out a set of parameters that they want us to meet. And so I’ll be re-envisioning my courses and to practice what I preach. I’m going to try to flow that out as much as possible to my colleagues, both locally in my own department, my own college, at my institution, and also nationally. So I’m kind of looking at some different ways that I can continue to engage people in this and share out what I’m learning as we go along. And I’m also pretty excited to be preparing some even more in-depth materials for some institutions who are looking for help in exactly this type of thing, how to get faculty interested in this whole topic of flexible teaching, some specific techniques that are useful for what I’ll call flexible teaching, key resources, things to do and not to do, and so on. So I’m excited to be coming back at it on all cylinders in the fall, and looking forward to engaging students in all the different formats that we now have and seeing where it takes us. So that’s what’s next for me.

John: Well, thank you. This has been wonderful talking to you again. We’ve always enjoyed these conversations, and our listeners have very much appreciated them.

Michelle: Oh, thank you.

Rebecca: It’s always really helpful to know too, that you’re not alone. We’re all going through the same kinds of contemplations, and so thanks for sharing some of your own stories about developing and planning for the fall too.

Michelle: Thank you as well.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

108. Neuromyths

Faculty design their classes based on their perceptions of how students learn. These perceptions, though, are not always consistent with the science of learning. In this episode, Dr. Kristen Betts and Dr. Michelle Miller join us to discuss the prevalence of neuromyths and awareness of evidence-based practices in higher ed.

Kristen is a clinical professor in the online Ed.D. program in Educational Leadership and Management in the School of Education at Drexel University. Michelle is the Director of the First-Year Learning Initiative, Professor of Psychological Sciences and the President’s Distinguished Teaching Fellow at Northern Arizona University. She’s also the author of Minds Online: Teaching Effectively with Technology and a frequent guest on this podcast.

Show Notes

  • Miller, M. D. (2014). Minds online. Harvard University Press.
  • Online Learning Consortium
  • Betts, K., Miller, M., Tokuhama-Espinosa, T., Shewokis, P., Anderson, A., Borja, C., Galoyan, T., Delaney, B., Eigenauer, J., & Dekker, S. (2019). International report: Neuromyths and evidence-based practices in higher education. Online Learning Consortium: Newburyport, MA.
  • Mariale Hardiman
  • Tracey Noel Tokuhama-Espinosa
  • Dekker, S., Lee, N. C., Howard-Jones, P., & Jolles, J. (2012). Neuromyths in education: Prevalence and predictors of misconceptions among teachers. Frontiers in psychology, 3, 429.
  • Alida Anderson
  • Macdonald, K., Germine, L., Anderson, A., Christodoulou, J., & McGrath, L. M. (2017). Dispelling the myth: Training in education or neuroscience decreases but does not eliminate beliefs in neuromyths. Frontiers in psychology, 8, 1314.
  • Universal Design for Learning,” CAST website
  • Mchelle Miller, “65. Retrieval Practice” – Tea for Teaching podcast, January 23, 2019.
  • Vygotsky, L. (1987). Zone of proximal development. Mind in society: The development of higher psychological processes, 5291, 157.
  • Michelle Miller, “86. Attention Matters” – Tea for Teaching podcast, June 19, 2019.

Transcript

John: Faculty design their classes based on their perceptions of how students learn. These perceptions, though, are not always consistent with the science of learning. In this episode, we examine the prevalence of neuromyths and awareness of evidence-based practices in higher ed.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today are Dr. Kristen Betts and Dr. Michelle Miller. Kristen is a clinical professor in the online EDD program in Ed.D. Educational Leadership and Management in the School of Education at Drexel University. Michelle is the Director of the First-Year Learning Initiative, Professor of Psychological Sciences and the President’s Distinguished Teaching Fellow at Northern Arizona University. She’s also the author of Minds Online: Teaching Effectively with Technology and a frequent guest on this podcast. Welcome, Kristen and welcome back, Michelle.

Kristen: Thank you so much for having us.

Michelle: Hi, it’s great to be here again.

John: Were really pleased to talk to you. Our teas today are…

Kristen: I’m drinking Apricot Oolong, a green Tea. Nice for the afternoon.

Michelle: And, I have a wonderful hibiscus tea.

Rebecca: And, I have… big surprise… English Afternoon tea.

John: And, I have ginger peach black tea.

We invited you here to talk about the study that you both worked on together on neuromyths and evidence-based practices in higher education. Could you tell us what prompted this study?

Kristen: Sure. As a lifelong learner, I decided I would enroll in a wonderful program being offered at Johns Hopkins University several years ago in mind, brain, and teaching led by Dr. Mariale Hardiman. In one of the courses, I read several articles that looked at the high prevalence of neuromyths in K through 12 education. And, one of the things that caught me by surprise was: One, I was a K through 12 teacher early in my career. I was, at the time, a professor in the School of Education, and in looking at some of the neuromyths, they actually looked like things that I had studied as part of professional development. And, I had not assumed they would be neuromyths. And, so it really intrigued me in terms of: Why is there this high prevalence and why are we not more aware of some of the evidence-based practices that are out there? Not just in the United States, but clearly these were studies that were taking place internationally. So, I decided to start looking at this through the lens of higher education, because that’s where I work and it’s my area of expertise, and I reached out to Dr. Michelle Miller. I was at the Online Learning Consortium conference. Her focus is on cognitive psychology. So, I approached her after the session and told her about this interest in looking at neuromyths within the field of education… really, across disciplines, in trying to see was it similar to what the findings were in K through 12 education, and what was really being done to integrate evidence-based practices into pedagogy or even andragogy. So, we decided to connect and start looking at this. I had a wonderful PHD student who I was working with at the time as well, who is from Armenia, very interested in this topic, and we quickly grew our small group to include a total of ten researchers from the total of seven different institutions nationally and internationally across three countries. And, everybody brought different expertise, everyone from two-year colleges, four-year colleges, public, private. And, we also were very fortunate because we were able to find, really some of the seminal researchers in the area of mind-brain education science, such as Tracey Tokuhama-Espinosa. And, we reached out to the researchers who actually conducted the studies looking at neuromyths like Sanne Dekker, and we reached out to a Alida Anderson who worked with McDonald et. al. in their 2017 publication. So, it quickly grew from a point of interest in trying to identify what was happening in higher education, to really a much broader international study.

Michelle: Oh, and just echoing what Kristen has said here, we first did meet through the Online Learning Consortium, first at a conference and then they set up calls where we got to talk to each other and realize that even though we came from somewhat different academic backgrounds and published in some different areas, we really had this common ground of interest in how do we bring more evidence-based teaching to faculty in higher education and really throughout the world. And, to me, as a cognitive psychologist, it’s just an inherently fascinating question of, even though we live in our own minds, why do we not sometimes understand some basic principles of how the mind and how the brain works? So, that’s just an intellectually interesting question to me. But then it does take on this tremendous practical importance when we start to look at teaching practices throughout the world and bringing that really quality evidence-based design of teaching and learning experiences for our students.

Rebecca: Can you talk a little bit about how, once all of these researchers are now together, how did you put the study together and how was it conducted?

Kristen: I have to say it was not easy. Thank goodness, we reached out to some of the original authors. The survey instruments that looked at neuromyths and general knowledge about the brain. And, what was so interesting is almost all the studies were truly K through 12 focus, so the questions were very different. Even looking at lexicon, “girl and boy,” where we would want to look at male/female. So, we had a look at absolutely every question and make sure that we were able to revise that question within the framework for the lens of higher education. So, it was not an easy process, just in terms of time, because we had to go through so many iterations. And, I think that really helps with the integrity of the research. We had two pilot studies, even down to looking at the Likert scales that we used. One of the things that really stood out was the primary study that we looked at, which was a 2012 study by Sanne Dekker and several other researchers. They had a Likert scale that looked at correct, incorrect and I don’t know. There was a study by McDonald and colleagues in 2017 and they changed it to true and false. So, we decided early on, we would go with true and false. And, when we did that pilot, we ended up with half the participants stopping midway and simply putting, “I’m not sure if it’s true or false…” and they just didn’t complete the survey. And, I think, just looking at how we phrase the questions, it really affected the participation of our respondents. So, we went back, we modified some of the questions based on that, and we change the Likert scale. And, I think being able to have the ability to say whether it was correct, incorrect, or you didn’t know took away from saying it was true or false, because you can base it on knowledge or what you perhaps had been exposed to. And, we ended up having a wonderful pilot making some additional changes. And the feedback that we got, even after sending out the survey, we had a flood of emails saying “Can you please send us a copy of the study, we’re really interested?” So, we really looked at everything. And, I would say one thing that stood out most; and again I go back to the time we spent over two years on this study from point of inception to where we actually send out the survey, collected this study and then published it, was when we looked at the neuromyths, what we quickly realized was we needed to examine evidence-based practices as well. And, we looked at all of this from a metacognitive perspective. The prior studies that were done, looked at what they called “endorsing neuromyths,” and we weren’t so much looking at endorsing, we wanted to look at awareness, because all of us were involved in teaching… professional development. And, so it was a matter of trying to identify what the gaps were, what were instructors, instructional designers, and professional development administrators aware of and, if there is that gap, how could we develop a study where people would say “Wow, I also thought that was correct, but it’s incorrect… but, I would love to find out what the response is and how I can change my knowledge or understanding.” And, so we looked at absolutely everything and wanted to create a study that people would pick up and say, “This is where I am now. Gosh, after going through this in reading the report, this is where I am and my circle of knowledge needs to continue to expand, as things continue to expand through mind-brain education science.”

Michelle: As a collaborative effort, I haven’t been involved really in a study of this scale and scope. And, it’s simply the level of collaboration. You just heard about one of the iterations of the survey instrument that we put together and just how that piece of the study came about. But all the way through the analyses, the writing, it was such an opportunity, even apart from what we were able to share with the rest of the world, just from my own niche piece of the study as well. The opportunity, as a cognitive psychologist, to start infusing what I feel is more attention that needs to be paid to cognitive psychology and learning sciences. The opportunity to infuse that into this field in this area of thinking was also really exciting as well.

Kristen: So, in terms of how it was conducted, we sent the survey out for the Online Learning Consortium. When we originally started, we were just going to look at instructors, we were looking at neuromyth prevalence in instructors because all of the other studies that had been done were primarily K through 12 teachers and pre-service teachers. (although the McDonald study looked at a wider range). Once we started to bring together our team, then we started thinking, “Gosh, well, it’s not simply the instructors. It’s going to be the instructional designers, it’ll be anybody conducting some type of professional development as well because no course is truly an island.” There are so many people today involved in course design, course development and so the Online Learning Consortium was such an amazing partner for us and they touch on absolutely every part of that population. So, we reached out to them early on and said “We’d love to collaborate with you. You’ve got an extensive membership and listserv. Would we be able to develop this survey instrument, send it out through your membership, and ask them through snowball sampling to share it with others who may actually be involved in higher education, in one of these roles.” And, they could not have been a better partner. They’re just incredible to work with. So, that’s how it was conducted.

John: And, we were actually part of that snowball. I sent it out to a list of about 1200 faculty, staff, and professional development people on my campus alone. How large was your ultimate sample?

Kristen: We ended up with approximately 1300 respondents. And, then we actually looked at the full study, we ended up with 929, who met the criteria for inclusion. So, one of the things we wanted to make sure when we looked at the criteria for inclusion that they worked in higher education. You’d be surprised. So many people complete surveys, but they don’t necessarily meet the criteria. Even when you explicitly state you have to be within higher education: teaching or one of these areas. So, we had a total of 929 who met the criteria, and of those they also had a complete 95% of the questions for the neuromyths, and also for the evidence-based practices because we didn’t want to have any gaps. I would say it was an incredible response rate, especially for those completing the survey. They filled out I would say the majority of everything within the survey itself. The respondents were just incredible as well, because you talked about the cross section of participants, but we ended up with really an incredible number of instructors and that was broken down into full-time, part-time, instructional designers, the professional development administrators and it allowed us to run a lot of different tests that we’ll talk about when we look at the findings.

Rebecca: I think one of the things that’s really interesting about how you discuss the setup of the study is thinking about how many different individuals play a role in perpetuating myths, or even perpetuating good evidence-based practices too. That administrators is where funding comes from, so you have to have everybody in the institution on board with what you actually want to essentially Institute.

Kristen: Well, what’s interesting, and you bring up such a great point. One of the top neuromyths out there is learning styles. And, so when you’re looking at learning styles, this is something that almost seems to permeate. It doesn’t matter when you started teaching, whether it’s K through 12, or higher education at some point if you’ve been involved in education, you’ve come across learning styles. Now there are learning preferences and there’s lots of wonderful research on that. But this concept of teaching to learning styles, I think, unfortunately… we talk about this in section seven of our report kind of got mixed in with multiple intelligences. And, that is not at all what multiple intelligence was about, but it was almost the timing of it and so, having been a K through 12 teacher, I remember going through a professional development where we learned about learning styles and how it was something to look at in terms of teaching to learning preferences. And, even to this day when I do presentations, and I know Michelle has run into this as well, especially when we co-teach some of the OLC workshops, somebody will inevitably raise their hand or type in the chat area “Are you kidding? Learning styles is a neuromyth? We just had somebody on our campus six months ago, who taught us how to do an assessment to teach to learning styles.” So, it’s still out there, even though there’s so much in the literature saying it’s a neuromyth. It’s still prevalent within education across all areas.

John: So, you mentioned the issue of learning styles. And, that’s something we see a lot on our campus as well. We’ve even had a couple of podcast guests who we edited out there mention of learning styles and then had a chat with them later about it. I won’t mention any names because they had some really good things to say, but it is a really prevalent myth and it’s difficult to deal with. So, you mentioned learning styles. What are the most prevalent myths that you found in terms of neuromyths?

Kristen: When you look at the report, the first part of our survey had 23 statements. We had eight statements that were neuromyths. If you look at the K through 12 studies, they had many more neuromyths, but we had eight. And, I will tell you, the top five neuromyths in higher education, very closely parallel what you find in K through 12. Now our prevalence is not as high, but it still shows that instructors, instructional designers, and administrators are susceptible to them and that goes back to awareness. So, the top one: listening to classical music increases reasoning ability and that’s really that Mozart Effect. Another one: individuals learn better when they receive information in their preferred learning styles. Some of us are left brained and some of us are right brain due to hemispheric dominance and this helps explain differences in how we learn. So, that’s really that concept of “Oh, I’m right brained. I’m left brained.” And, this again, is something that goes across higher ed and K through 12. Two other really big ones: We only use 10% of our brain. And, if you look at section seven of the report, you will find all of the responses, literally evidence-based practices or research-supported responses to make sure that people aren’t simply saying, “Oh, it’s incorrect. Well, we want people to know why it’s incorrect. So, they can reflect on that and change their understanding, really the rationale and the research behind it. And, then lastly, it is best for children to learn their native language before a second language is learned. This, again, is a big neuromyth. And I think one of the things I’m hoping that will come out of this study, because we talked about this really when we go into evidence-based practices, is this concept of neuro-plasticity, the fact that the brain changes every time you learn something new. When you’re engaged in an experience, the brain is changing. And, sometimes the brain is changing at a cellular level before you might even see that change in behavior, and so we’re able to see now through technology through f-MRI through fNIR so much more than we were able to see before. So, really keeping abreast of what’s happening in the research should be informing our practice because we have more information available than ever before. But, somehow we need to get that into our professional development training, seminars, and workshops or into the classes that we’re teaching in our schools of education or into our onboarding. But yeah, these are the top five neuromyths in terms of susceptibility, and they cut across higher ed and K through 12.

John: In your paper, you also provide some crosstabs on the prevalence by the type of role of individuals, whether they’re instructors, instructional designers, or administrators. Could you tell us a bit about how the different groups due in terms of the prevalence of these neuromyths?

Kristen: Well, the one thing I will say is, everybody is susceptible to neuromyths, so it wasn’t as if there was one group, and I know that’s always in the back of someone’s mind, “Gosh, who’s the most susceptible?” Well, we didn’t find any significant differences, and one of the things that we wanted to do as well was to really be break the participants down and look at other factors. So, when we look at full-time versus part-time faculty, is one group more susceptible to neuromyths. And we found no significant difference in terms of gender, in terms of age, in terms of working at a two-year institution, a four-year institution. And I really think that talks to the amazing reality of the opportunity to integrate professional development in looking at the learning sciences and mind-brain education science in the opportunity to decrease that gap. So, it wasn’t one group over another. But it’s everybody who has this opportunity to increase this awareness across all of these areas.

John: Didn’t you also find that some of these myths were less common among instructional designers relative to faculty,

Kristen: We found with evidence-based practices, when we looked at significant difference with evidence-based practices, instructional designers actually had in terms of percent correct, higher awareness of evidence-based practices. It wasn’t a large difference, but there was a significant difference and Michelle can certainly talk to this point as well. But, this is really the importance of having an incredible team when you’re looking at course design, course development, and part of that may have to do with, when you look at instructional design, there is so much new literature and research that’s getting infused in to that area, and so that may have something to do with it. But, I think there’s lots of additional studies that we could do to follow up.

Michelle: Kind of circling back to the point of the design and delivery of instruction in a contemporary university or college is fundamentally more collaborative than it was in prior eras. And, so I think we definitely need to have everybody involved start to really break out of that old school mold of class is identified with the teacher who teaches it and that’s what a course is. And no, courses reflect, today, everything from the philosophy and the support that comes down from the top to the people that the students may never meet, but who put their stamp on instruction such as instructional designers. And, this is something that I get pretty fired up about in my just practical work as a program director and just being involved in these things in the university, that there are still faculty who you say, “Hey, do we have any instructional designers who are working with us on this project to redesign? Is anybody assigned to help us as we develop this new online degree program or something?” …and you sometimes still get blank look.? Or you get “Oh, aren’t those the people who you call when the learning management system breaks down and that’s their specialty?” I mean, this report, I think, just really hammers home that idea that instructional designers are a key part of this collaborative team that goes into really good quality higher education instruction today. And it isn’t just about the technology. I think that they’re getting exposure to and staying abreast of what’s going on in research that relates to teaching and learning. And, what a great opportunity for faculty to not just rely on them for technology, but to learn from them and to learn with them as we build better courses together.

Rebecca: Can you talk a little bit about the awareness that you found in general about evidence-based practices? So, we focused a lot on the neuromyths, but what shook out when you started looking at the evidence-based practices?

Kristen: Well, one thing that stood out was awareness was much higher. And, that’s really exciting. I think that’s a huge testament to the professional development that we are offering. But, there were still gaps in areas where there certainly could be a lot of improvement. So, a couple of examples that I’ll give because we literally spent months looking evidence-based practices, and we wanted to make sure that we could support them. So, for example, when we look at percent correct, where most individuals across all three groups were not as aware, like “differentiated instruction is individualized instruction.” So, we know that this is incorrect. But most of the respondents did not put that that was an incorrect statement. So, they either stated it was correct, or they didn’t know. So, again, this is an area that we certainly want to explore. Because differentiated instruction is something that really, I think, adds to the classroom. And, there are other ones. For example, we’ll look at Universal Design for Learning. So, one of the statements we had in there actually comes directly from the CAST website, and it says “Universal Design for Learning is a framework to improve and optimize teaching and learning for all people based on scientific insights into how humans learn.” Well, the instructional designers, they were the most aware. So, 87% of them got that correct. Of the professional development administrators 74% got that answer correct. For the instructors, 58% got that correct. So, you can see the difference in the responses and when we share this nationally or internationally…. when we talk about the study, you’ll have a lot of individuals who’ll say “No, universal design for learning, that’s about accessibility.” Well, it certainly is about accessibility. But, most importantly, it’s about learning and how humans learn. It is probably the most dynamic and the most powerful aspect that we can add into pedagogy or into andragogy. But just by looking at the data here, it may not be something that everybody’s aware of, and that’s again a great opportunity to integrate that into professional development. So, there are a number of things. I mean, it’s exciting because when you look at it, there are 28 statements. And, as I mentioned, overall, the awareness was much higher across all three groups, compared to neuromyths or general knowledge about the brain.

Michelle: Just to jump in here, again, from my kind of cognitive psychology perspective, those evidence-based practices that we’re talking about also include, specifically, some items that are related to memory, a topic that’s really close to my heart. So, I think those are just fascinating as well. So, for example, we asked a variation on a classic question that many cognitive psychologists have looked at: “whether human memory works a lot like a digital recording device or a video camera.” So, is your memory basically taking in information that’s in front of you? And, here again, we’ve got 69% of our instructors saying, “Oh, yeah, that’s right. That’s how it works.” And, that is not how it works. 79% of our instructional designers identify this as an incorrect statement and 74% of our administrators, and we have a few other related things such as we asked people whether testing detracts from learning. And, as Tea for Teaching listeners know, that goes to retrieval practice. Testing doesn’t detract from learning, testing builds up learning. So, these are some as well that I think it’s very interesting to tap into what people know and really think about while these maybe seem like inside baseball, or very metaphorical or philosophical questions, if I’m an instructor, and I believe these things, that students are basically just running video cameras in their heads… well, that is going to lead to some different practices. I might be very puzzled as to why I got up and gave this lecture and the students eyes were pointed at me and yet it didn’t end up in memory. So, those are some of the items that I was particularly interested to see when we got all the numbers in.

Kristen: You know, I would say one thing: when anybody reads the report, what we want them to do is look at how it’s presented in terms of the tables, because everything is looking at the percent of correct or accurate responses. So, as Michelle said, when we look at “human memory works like a digital recording device,” 69% of the instructors got that correct. 79% of the instructional designers got that correct. And, 74% of the administrators got that correct. So, that means we still have a fairly large percentage, basically 20 to 30% that either got the answer incorrect, or they didn’t know. And, even looking at these responses, do they actually know why they knew it? Or did they guess or did they make that assumption like, “Oh, that’s got to be right.” And so, really, the intentionality of this study was awareness, really bringing out statements from the literature to help anybody who’s involved in teaching, course design, professional development to look at these questions, and really think “Do I know this?” And, “If I know it, how do I know this? Is it based on some type of research or literature? Could I defend that? If I don’t know with certainty, where do I find that answer? And how can I learn that? And, how can I integrate those practices?”

John: On the day when your report came out, we shared that on our campus to everyone on our mailing list. One of the nice things about the report is that it has all the questions and also provides references for the answers explaining why the specific answer is true or false. And, it’s a really great resource and we’ll share a link to that in the show notes. It is long. When I shared it two people sent back email saying “Maybe we should use this as a reading group for next semester.” And it’s not a bad idea, actually. But, much of that is appendices and so forth. And, it’s a really informative document. I believe in your survey, you were asking people about their participation in professional development, and you looked at the relationship between participation in professional development and the prevalence of these myths. Is that correct?

Kristen: We did. So, one of the things that we wanted to look at was trying to find out if educators were involved in professional development, whether it be neuroscience, psychology or in mind-brain education science, did that actually increase their awareness of neuromyths, general information about the brain, and evidence-based practices? And it did. We found that that it was definitely a predictor and it was found to be a significant predictor and so, for us, again, it looked at what a wonderful opportunity to be able to say that training does have a positive impact. And, that was really the crux of the study… and it’s interesting, you talked about the length of this study, because originally we had thought about doing two different or three different studies. So, we do one on neuromyths, one on evidence-based practices, one on professional development. Then when we brought the data in, the question was: “Do we separate them out into three different long articles or three different reports?” And, we collectively, across all disciplines said, “No, we need to bring them together.” Because first and foremost, it’s about awareness. You can’t really talk about evidence-based practices, until you’re aware of what the neuromyths might be. What are some of the fallacies that you might actually believe? What are things about the brain that you may or may not know? And, once you’re there, and you have that understanding, you can then move into the evidence-based practices, because it’s all really connected. So, when Michelle talks about memory, you can’t really talk about memory without having some understanding of the mind or the brain. And, so we decided collectively, we would bring it together as hopefully a seminal piece that would really present anyone with a continuum as to: “Where am I? What am I possibly doing in my classroom?” …being able to really do that self assessment and then find the answers, as you said, in that section seven, and realize that they’re not an outlier. I mean, chances are anybody that goes through this is going to fall within that span in terms of their understanding and knowledge.

Michelle: And, what I hope is coming out here is that this study is unusual, not just in its scale, its scope and that we focused on higher education, but that it is so explicitly geared to not just identifying gaps in knowledge or awareness, but addressing those. It’s not like we came along six months later and said, “Oh, by the way, here’s a really nice resource we put together.” It is one stop, it’s right there. And, what an exercise that was, as well. Kristen, I think you’ll remember back just saying, “Okay, in a paragraph… this item, all of us look at this and go ‘oh my gosh, that’s wrong’ or ‘that’s right.’ Why is that? and what are the very best empirical sources that we will trace back to, to demonstrate that?” So, we are trying to provide that and also to really be a model to say: next time that you get that handout or that workshop that says, “Oh, here’s some great stuff about the brain.” What are they backing that up with? Can you trace it back to the solid research sources that makes some of these really powerful principles for learning, and make other things just misconceptions.

Kristen: One of the things that I would say was probably the most exciting and the most challenging. We had 10 researchers, we had 10 researchers from different fields: people from nursing, biomedical engineering, psychology; we had people who work in the area of neuroscience, education (as I mentioned), and we needed to come out with a collective voice, writing a report that would be understood across disciplines. And, so when we wrote section seven, all of us had to be reviewers and we vetted it multiple times. Not just within our group, but outside, to make sure when you read about neural pathways, it actually made sense. Because to write something where somebody would not understand or not be able to connect would be a challenge. And, we wanted people to walk away. I know one of the things that we were looking at: Why neuromyths? Well, a lot of the research out there looks at the fact that when you teach, your teaching and your pedagogy is based on your knowledge, and in your understanding of how people learn, and so we wanted to really look at this area in terms of awareness, because it may impact pedagogy. Our study did not do that. And, I want to make sure it’s really clear. Our study was not designed to say, “Oh gosh, the awareness of neuromyths wasn’t very high in this area, therefore, you must be integrating neuromyths into your teaching. That was not the intentionality of our study and that’s not something that we’ve ever said. There are certainly recommendations we put in the study to look at. If there is a high prevalence of neuromyths,how does that affect pedagogy? But ours was simply looking at awareness and could professional development address gaps? So, we could do this across all different groups that would be involved in course design and delivery.

John: That’s one of the things I really like about it, that you do address all these things well, you provide the evidence, and it’s going to be a great go to reference for those of us when faced with neuromyths, with issues about evidence-based practices. We can just go and grab some of the citations and share them back out or refer them to the whole document as I’ve done several times already. These things are really common even in professional development. I was at a session not too long ago, where there were two neuromyths presented during the session. One was the learning styles thing. But the nice thing is, unlike other times when I’ve seen that done, there were two of us who went up and waited until everyone else talked to the presenter. And, we were both ready to do it after other people had gone so we didn’t embarrass her, but it’s starting to get out there. And, I know on our campus, we’ve got a growing number of people who are aware of this partly because of the reading groups we’ve had, where we’ve had a growing number of participants… and that all started actually with Michelle’s book about five years ago now when we first did the group. You came out, you visited, people wanted to do more, so we started a reading group. We’ve done four additional reading groups since then. We’ve had many of the same participants, but it’s spreading out wider. I’m hoping we’re making a difference through these reading groups.

Michelle: And, that’s so gratifying as an author and as a researcher, and I remember well working with your group in Oswego and the great ideas I took away as well. So, I’m a big believer in virtuous cycle. So, maybe we’ve started one.

Kristen: I think what really came out of this study is the passion that everybody has for student success. Everybody from those that are offering the professional development, the instructional designers that want to make sure that the students are successful, even though they might not be teaching the course. And, then the instructors themselves… and so to be able to work with that many individuals who are not only subject matter experts across their disciplines, but so passionate about making a difference. But I think being able to integrate all of this new research relating to neuroscience, psychology and education, it’s going to transform not only how we teach, but it’s going to transform pedagogy, andragogy, and this whole concept of learning.

Rebecca: I really appreciate the bringing it together and that you decided to keep it all together and not to make three separate reports. I think it’s actually really important to understand how these are all connected and related. And, I think that’s one of the most unique things about the report. I think the community is probably very grateful that we have this resource available now.

Kristen: Oh, thank you.

John: One of the things I’ve often been concerned about is how some of these neuromyths, particularly the left brain – right brain thing, and the learning styles belief, often serves as a message to students that they can only learn in certain ways or they only have certain types of skills, and they’re not able to make progress in other ways. And, it can serve as a barrier and can lead, perhaps, to the development of a fixed mindset in students which may serve as a barrier.

Rebecca: …or not even allow those students to feel like they can enter particular disciplines.

John: If people become more aware of this, perhaps it could lead to more opportunities for our students or fewer barriers placed in the way of students.

Rebecca: …or maybe even just more inclusive pedagogy in general.

Kristen: You bring up such a great point. So, if you believe in learning styles, and you believe that you are truly a visual learner, Michelle and I’ve talked about this a lot, it almost becomes a self-fulfilling prophecy. But you probably are an incredible visual learner because you’ve been told you learn better in this learning style, so you’re going to seek materials in that learning styles. So, the challenge with that, especially when you’re looking at younger students or anybody during their education, you’re precluding really other ways to enhance your learning. So, when you look at Universal Design for Learning, it’s so important because you’re looking at multiple means of engagement, representation, action, and expression. And, when you’re looking at learning styles, if a student believes they’re a visual learner and suddenly asked to go in and take a Spanish oral exam, it could trigger, all of a sudden, stress. Well, what do we know about stress? And, Michelle can talk more about that. But, when you’re stressed, it affects working memory. And, so just that thought of, “Oh my gosh, it’s an oral exam. I’m a visual learner. How can I perform well on that?” And it’s really creating, as you talked about, a barrier or it may decrease, possibly, performance. I know that Dr. Tracey Tokuhama-Espinoza is very passionate about this. And, you’ll see in her presentations, she’ll come out and say “Neuromyths do harm.” And so, I think it’s certainly something that needs to be explored. And, Michelle, from a psychological point, I’d be curious to find out what you have to say as well.

Michelle: When you say “self-fulfilling prophecy” and things like that, it also kind of reminds me of a placebo effect, in a way… and learning styles, and continuing that as an example, yeah, I might go: “Oh, visual learning. It is absolutely me,” like “Now I feel like I can tailor all this to myself. I’ll just find teachers, opportunities, and disciplines that are right there in visual learning.” And, I might have some subjective impression that that’s helping me, or from the teacher’s perspective, I might feel like “Well, I brought in some different materials and engaged different modalities and, what do you know, because of learning styles, we’re doing better.” Well, there’s lots of different reasons why that might be happening. An individual may walk away, and maybe they weren’t individually harmed. I just feel like… just like in modern medicine, there’s sort of a promise that we can do better than mere placebos. I think that ought to be the promise of modern pedagogy as well, that we can do better than simply trying to build up expectations or giving people a false sense that they have something based on science that’s going to help them individually do better. And, I hear so many kind of missed opportunities that really kind of get me activated as well. I think about, for example, the energy that goes into faculty professional development. These things come from good impulses. I really believe that. I believe that people who really pursue something like learning styles or things like that, they want to do better and they want to be more inclusive, but that effort is directed down the wrong path simply because of this gap in knowledge and gap in information in getting the right information to the right people at the right time. And, I can’t stand the thought of faculty, especially as limited as faculty time is and as spread as thin as faculty are, to think that they might try to pick up on some better information about teaching and learning and go down the wrong path. I never want that to happen again. And, maybe our report will be a step in the right direction.

Kristen: I’ll say one thing that we’re trying to do with the report, is really to align the report with best practices and evidence-based practices. So, when you look at the concept of neuromyths the wonderful study that was written by McDonald (and this was in 2017) and her colleagues, the title is “Dispelling the Myth: training and education in neuroscience decreases but does not eliminate beliefs in neuromyths” and so professional development is not a silver bullet. Simply offering one workshop that’s going to address neuromyths is not going to necessarily get rid of neuromyths. So, we have to do what? We have to look at spacing. We have to look at interleaving. So, with professional development, how do you take information related to evidence-based practices and integrate spaced practice into our own professional development? How do we integrate interleaving? How do we integrate low-stakes assessment? So, maybe when faculty or instructional designers come in, you do a quick self assessment and find out what that baseline knowledge is, and then at the end to say, “Okay, at the end of professional development, we need to get to 95% or higher.” But, they’re able to actually test their own knowledge. So, we need to kind of turn professional development upside down and make it active learning and really engage everybody in what we’re looking at within pedagogy and andragogy.

Rebecca: Yeah, I always find it really ironic that a lot of training and things on evidence-based practices is not using evidence-based practices… or using really traditional formats: lecture or getting lectured at and not really engaging with the material. And, it’s no different when we’re working with our students. And, if they’re practicing in a way that’s not going to be effective for them, and they’re not successful. They could spend tons of time on something and just not really make progress. The same thing can happen with our faculty and staff who are designing curricula and what have you as well. They can be really invested.

Michelle: Absolutely.

John: We do have an excellent podcast on retrieval practice. In fact, it’s one of our most popular episodes. We’ll share a link to that in our show notes. We don’t yet have any podcasts on interleaved and spaced practice, but I’m sure we’ll be asking Michelle to come back and talk about these things at some point in the future, if she’s willing. So far, we’ve been focusing on the types of neuromyths that are common. What can we do to reduce the prevalence of these neuromyths?

Kristen: Professional development is certainly key. But, I would look at things such as onboarding, making sure that when people are getting hired on, that they’re really introduced to evidence-based practices from the very beginning. And, even individuals that would say, “Gosh, I’ve been in instructional design for 20 years, I’m familiar” …there may still be those gaps. And, it’s almost like adaptive learning. Everybody that comes in very much like the Vygotsky’s work of zone of proximal development, they may have all been teaching for 20 years, but it doesn’t mean that we don’t have neurodiversity in terms of experience, knowledge about different practices. So, it’s important that it’s from the very onset of when people get hired and making sure it’s understood that we’re committed to best practices, evidence-based practices and what we do builds upon the literature and the research. Not only do we introduce it here, but we move it forward and integrate it into our pedagogy and what we’re doing in our classrooms.

Rebecca: So, we always wrap up by asking: What’s next?

Michelle: Conference season is upon us. We’re recording this fall of 2019. I’m gearing up to go to the Online Learning Consortium’s Accelerate conference in November. And so, I will just personally say come find me if you’re there and you want to talk more about this. I will be presenting on a related but different topic having to do with our ongoing Attention Matters project, which is also the subject of another Tea for Teaching episode. So, I’m really working on getting ready for that, and also the upcoming POD network conference. So, for those educational developers who will be attending that, I’ll be speaking there and hopefully having lots and lots of sidebar conversations with plenty of other people who are interested and fired up about these very topics. So, I/m working on those. I’m working on what I will now call a forthcoming book. It’s under contract with West Virginia University Press, tentatively titled Remembering and Forgetting in the Age of Technology. So, maybe someday in the not too far off future, we’ll be talking about that project as well.

John: We should note that this podcast will be released during the OLC conference. In particular, it’s coming out on Wednesday of the conference.

Kristen: Oh, that’s exciting.

John: And, I should also note that we’ll be presenting there as well. I’m hoping we’ll get some people to listen to this podcast because we’re presenting the next day. So, we might get some new listeners. [LAUGHTER]

Kristen: Oh, that’s exciting. In terms of projects that I’m engaged in and working on. We’ve just launched a new lab in our School of Education at Drexel University. So, we’re bringing everything together and trying to align projects coming up for 2020. But it’s a lab called ELABS, Education, Learning, and Brain Sciences Research Collaborative. So, we’ll be looking at different studies related to the learning sciences and mind-brain education science. I am wrapping up an article with several researchers at Drexel University, some of our PhD students, that looks at immersive virtual reality and practice as well as transfer of learning. We also have a report that I’m working on. It’s an update to research that I conducted earlier on online human touch. So, I’m wrapping up that study and putting together an article there. And, then also looking at two publications for books looking at neuro plasticity and optimal learning. One would be for students to really understand neurodiversity, neuroplasticity, how you can optimize the stress response, and then looking at neuroplasticity and optimal learning from the instructor or instructional design perspective. How do you integrate this into your practice? So, those are the initiatives that I’m working on.

Rebecca: Sounds like lots of things for all of us to look forward to.

John: Thank you very much for joining us. This was a fascinating conversation. And, we’ve been looking forward to this report since I first heard a bit about it when you initially did the survey, and when I saw a preliminary presentation at all see last year.

Kristen: Well, thank you so much for having us. It’s such a pleasure to discuss this topic with you. And, I’m looking forward to listening to many of your upcoming podcasts that clearly is connected to this report.

Michelle: Thank you so much. It makes all the hard work worthwhile and we love the opportunity to get the work out to exactly the people with the power to spread it to faculty and instructional designers and leaders in universities today.

Rebecca: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

67. Iterative OER Development

Imagine course materials that are always up to date and evolve continually to become better at supporting student learning. In this, Dr. Steven Greenlaw joins us to discuss how some publishers of open educational resources are trying to set up sustainable practices to achieve these goals. Steve is a Professor of Economics at the University of Mary Washington and the author of the OpenStax Economics textbooks. He has also developed the materials for Lumen Learning’s Waymaker Introductory Economics texts.

Show Notes

Additional Resources

Transcript

John: Imagine course materials that are always up to date and evolve continually to become better at supporting student learning. In this episode, we discuss how some publishers of open educational resources are trying to set up sustainable practices to achieve these goals.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Today our guest is Steven Greenlaw, a Professor of Economics at the University of Mary Washington and the author of the OpenStax OER Economics textbooks. He has also developed the materials for Lumen Learning’s Waymaker Introductory Economics texts. Welcome, Steve.

Steve: Thank you.

Rebecca: Today teas are:

Steve: I’m drinking coffee. Thank you.

John: …and I have Enchanted Forest Fruits black tea from Epcot which I picked up while I was out there for the OLC conference where I last saw you, Steve.

Rebecca: You’ll never guess what I’m drinking.

John: English afternoon?

Rebecca: Yeah, it’s my favorite.

Steve: Well, honestly, I switched to tea in the afternoon.

Rebecca: See…

Steve: But in the mornings, I tend to drink coffee.

Rebecca: Yeah, you and many other people.

John: What prompted your interest in using and developing OER materials?

Steve: I have to say the developing came first. For a long time, I’ve experimented with textbooks going back into the 1980s, which at least John can remember. And I came to the conclusion that that it didn’t really seem to matter what principles book you used. Students needed a book, particularly for the analytical parts of the course: the models and things like that. But whichever book I used, they seem to learn just as well. And more recently, I’ve come to the conclusion that intro textbooks are commodities, that where companies are going to make their money is in the aftermarket products. But we’re not there yet. At least, the majority of the textbook industry is not there yet. So I had that and I didn’t really pay much attention in the 2000s about what textbook I was using, because I didn’t really think it mattered. But I did notice how high textbook prices were going and it was around that point that I became aware of and interested in OER. Again, this is dating myself, but when I was in college during the mid-1970s, I remember a teacher in my intermediate macro class—John, that’s for you—saying he would never assign texts for a course that collectively cost more than $10. [LAUGHTER] And so that’s sort of my base year. So, I sort of had this in the back of my head, I basically tried to choose around the least expensive textbook that I thought would work. And then out of the blue, OpenStax contacted me and said they had funding to create a principles of micro-macro text, and would I be interested in helping them out. I actually jumped at the opportunity, it sounded like a lot of fun. At that point I had already published one textbook commercially for an upper level course and I knew something about the commercial publishing process. I knew that I didn’t really want to go through that again, but I did want to get my ideas out there. One of the things about commercial publishing is they ask you, “What are all the innovative things you want to do?” and then once they have you on contract, they say, “Oh, but you have to do it like everybody else’s.” So that was the start. A year after the OpenStax book got published, I got contacted by Lumen Learning who said essentially the same thing. They said, “We’re building this digital platform, and we wondered if you would like to be the principals subject matter expert.” That’s the term of art that I become a SME.

John: So could you tell us a little bit about Lumen Learning’s project and the Waymaker version of this?

Steve: Sure.

John: What does it add?

Steve: It adds a lot. So, just to be clear, I wrote the OpenStax principles book. And we can talk about that process later if you want to, especially about peer review and things like that. And then I wrote the Lumen Learning Waymaker version, which was essentially an improved version. When we did the OpenStax principles book, we did it in an incredibly short period of time, I think it was nine months. So, when I did Waymaker for the first time, it allowed me to flesh out some of the things that weren’t ideal in the OpenStax book. And then OpenStax came back to me maybe three years ago and said, “We have funding for a new edition. Would you like to do that?” so I wrote a second formal edition for the OpenStax principles book. And then right after that, I did the same thing for Lumen. So, in my mind, I’ve gone through four versions of this now. And it’s not done and that’s part of the beauty of OER… at least the OER business. So to get back to your question, the OpenStax principles book is a textbook, it’s available in print and a variety of online options. My particular favorite is the phone app. So if I’m in class and a student asked me a question about something, I could literally look it up on my phone. Waymaker is a very different animal. It’s digital courseware so it’s a more immersive, interactive experience for students. And it’s not available in print. For example, how would you show a video or do a simulation in a print textbook? You can’t. The most you could do was provide a URL or something and have the student go out to that. In Waymaker, it’s all in one. So Waymaker, aside from text, it includes video, it includes animations, it includes simulations. Just to give you a specific example, instead of students looking at a graph of supply and demand, they actually get to climb in and take it for a test drive. Students really liked that. Many students seem to get it in a way that’s just looking at a two-dimensional graph, or reading text it is much harder for them.

John: I saw you present on this at the OLC conference…

Steve: Yep.

John: …And you demonstrate this. What software did you use to create those interactive graphs?

Steve: Those little interactives are H5C… maybe… it’s called? [It is H5P]

John: Okay.

Steve: It’s a European company, and it’s open source, and it’s really easy to do. I can say that even though I didn’t create the interactives. That’s the joy of working with a company… they actually have people to do the stuff that you don’t know how to do… unlike my earlier career, when I was the programmer, I was the graphic designer and all of those other things. Talking a little more about Waymaker, it’s more than a source of course content. It’s designed to teach students to study more deeply and more effectively. I don’t know about your students, but my students don’t seem to have learned how to study well. They’re very good at the game of school, but they’re not so good at learning. And I don’t mean that as a criticism. It’s just sort of a fact. They think study means read, highlight, read again, highlight again. When we know a lot from cognitive science now, that learning comes from working with the material. As I like to say, “the best way for students to learn economics is to do economics.” So Waymaker emphasizes mastery learning and personalized tutoring. The tutoring comes both from the software and also from the instructor. It’s designed to give students actionable feedback so that they can make their own decisions about how to allocate their study time. This is a really different way of learning, so I’m going to say it again a little bit differently. Assessment is integral to the learning process, it’s not just or even primarily about the grades. Rather, the assessment is designed to make students interact more deeply with the content and interact in a more intelligent meta-cognitive way. I can go into more detail about what it looks like from the students perspective, if you want.

John: Sure. Could you talk a little bit more about that? It’s a great approach. I tried to do that myself, but it’s always an add-on. Having it integrated is a nice feature, and one of the reasons why I’m planning to adopt your package in the fall.

Steve: This is really different for students, but also for the instructor. I’ve been working on this product for three years. When it finally came out in beta, I thought I knew what was going on, and I was really surprised at how little I knew about how it actually worked. Waymaker is organized into modules, which are analogous to chapters in the text. Students begin each module with the “show what you know,” which is basically a formative assessment. The purpose of that is to identify what content they already know. So, it gives them feedback on how they can efficiently use their study time. So, if there’s stuff that they absolutely already know, they don’t need to read about it again, they can just go into the stuff that they don’t know.

John: And even if they don’t, it activates prior knowledge. And it helps them make connections so that they can learn more effectively…

Steve: …Yes.

John: So there’s a lot of benefits, even for the areas they don’t know.

Steve: Yes. And I’m actually adding a little exercise for my first day of class next week, where I put my students in small groups. Some of whom who’ve had the first semester, and some of them who have not. And I’m going to give them a basically a problem to work with, knowing that some of them won’t really know what to do with it. But I want the groups to start working together. But anyway, I digress. So, as students progress through the content, there are a series of learning activities. The original one is called a “self-check.” It’s basically a short formative quiz. The purpose of the quizzes is not summative assessment. But as I said before, it’s to help students think more about their learning. Think about the idea of a Socratic tutor. The tutor doesn’t ask questions to assess the students’ knowledge, but rather to help them work through the content, help them really understand it. So what happens in Waymaker is: the student reads a page a text, or watches a video, or plays a simulation. And then they’re posed a very short quiz, like one or two questions. If they pass the quiz, the “gate” opens and they move to the next section. If they don’t pass the quiz—and on a one-question quiz, either you get it or you don’t—Waymaker suggests that they review the content before attempting the quiz again. They can take those quizzes as many times as they want to. So they can really build some expertise. There are other sorts of learning activities, but I want to focus on the quizzes today. At the end of the modules, students take a module quiz—essentially a chapter test—which is summative. Again, if they fail to achieve mastery—and the default mastery level is 80%, so it’s pretty high level. As an instructor, you can change that to whatever you want. But I like 80%. So if they don’t achieve 80%, they’re encouraged to study again and they’re given information about what areas to study. And then they can take the module quiz one more time. They’re only allowed to take the module quizzes twice. Now, here’s where it starts to get really interesting from the teacher’s point-of-view. The instructor receives reports from the module quizzes whenever a student fails. So for me, the first really good thing about Waymaker was that I don’t have to go to some website and look at some spreadsheet and see which students are struggling. Rather, anytime a student fails, I get pinged from the software. So it says, “so and so…” Well it’s a little boilerplate language… but basically it says they worked through the module, and they scored a 46 on the module quiz. You might want to reach out to them at that point. So the software is flexible. So you can get these things in real time, you can get them once a day, you can get them once a week, if you want to. I get them once a day. That seems reasonably quick for me. If the students taken the quiz at three in the morning, I’m not up anyway, so it hardly matters. It’s not like I’m going to give them that fast feedback. But what happens is I get that information, and then I get to decide, “What am I going to do about it?” If someone gets a 76 on their first attempt, I generally figure, “Okay, they’re gonna figure this out.” And so I don’t worry about it. If someone gets a 46, then I immediately want to reach out to them and say, “Hey, I see that you’re struggling with this. You know you can take it again. Go back and review the material. And if you’re not sure that you understand it, let me know and I will work with you on this. Because the goal here is mastery. It’s not anything else. Anyway, Waymaker helps me, the instructor, make better, more efficient use of my time. In any given week, Waymaker allows me to know two important things. It allows me to reach out only to those students that need my help. And it lets me know what topics the class is struggling with, so that I can tailor my in-class time to the material where the students need help and not spend it on material where they already know this stuff. Basically, it gives me a better feel for the effectiveness of my teaching and student learning. And that’s really, really important I think as a teacher. I’m embarrassed to think of my early years and teaching, when if I got all the way through the 50 minutes, I counted that as a successful day.

[LAUGHTER]

John: I think many of us started like that.

Rebecca: It ties really nicely to your blog post series that you’ve just recently published. The first one being the critical importance of instructional design…

Steve:Yup…

Rebecca: …where you talk a lot about the instructor’s role is designing the experiences, rather than delivering content. Can you talk a little bit more about how Waymaker helps you do that as an instructor?

Steve: There’s a “just-in-time teaching” element to this. I have a course outline, I know what I’m supposed to be doing on a week-to-week basis. But what happens on any given day depends on the stuff that came before it. I’m absolutely not wedded to the calendar. If the students haven’t figured out what we did on Monday, I’m going to start by spending a little more time on that. But also because of the feedback that I’m getting from Waymaker, there are times when I spend 90% of the class on 10% of the material. Because that’s what I know students are having trouble with. I know that if it’s something analytical, probably what I’m going to want to do is instead of talking to them about it—I mean, certainly I’m going to talk to them about it—I put together some group activities. I do a lot of group activities, small groups, generally two to three people. And then I essentially turn the classroom into a lab experience for that day. They seem to enjoy it more, they seem to get more out of it than me just lecturing over the content. After all the content is in the book. I don’t need to just repeat that stuff. So I guess that’s my short answer to your question.

Rebecca: Can you give an example of the kinds of activities that you’re doing with your students?

Steve: Oh, sure. Supply and demand is the first real model that the students work with. And so one of my learning goals is that they ought to be able to take a scenario… something happens… use supply and demand to analyze the effect on the market for x, gasoline or something like that. Typically what happens is, hopefully they will have read the material in Waymaker. Typically, I spend a day talking about “here’s how you would do it” and then generally what’s going to happen is, I spend a day where I have a couple of problems, like three is all that we’re going to have time to do. And I say, “Get in groups of two or three.” Basically, I count the number of students that showed up that day, because my classes are pretty small. And if it’s divisible by three, I put them in groups of three, if it’s divisible by two, I put them in groups of two. And then I say, “Okay, here’s a problem,” I show them the problem. And I say, “Take 10-minutes to work through this, draw the graphs.” And then they know that I’m going to call some of the groups up to present the results to everyone else. So there’s a little bit of competition. It’s not very stressful. It’s a little stressful for people that don’t like to speak in class, but you’re not there by yourself. You’re there with your group, so it works better that way. So I do a couple of those problems until I’m convinced that most people know what they’re doing. So that would be an example.

John: You also mentioned—when I saw you present at the Online Learning Consortium—how you use some of that feedback to improve the text in your current edition. Could you talk a little bit about that process of revision and creation of the text?

Steve: Sure. While I can’t take all the credit. From the beginning of Waymaker, at least from when I began to get involved… once I realized how integral the assessment process was to Waymaker, I pressed Lumen to make sure that the assessment questions were good. One of the things that I’ve noticed over the years is, test banks seem to be the lowest priority of textbook publishers. Because after all, they’re selling the text but they’re giving away the test bank. So what I want, I guess what we all want, is that the questions in the test bank that Waymaker uses, are discriminating correctly. And that’s harder than you might imagine. To their credit, Lumens put a tremendous amount of effort into this. And more generally, into the design the courseware. This has resulted in a process of continuous improvement. Now, continuous improvement is not a term that excites most faculty. I think that’s a fair statement John? [LAUGHTER]

John: Yes.

Steve: But what it really means is that, Lumen has an ongoing process for improving OER, making it more effective every single semester. And they’ve done this, and we’re now in your five and a half. So how does it work? I have a short answer and a long answer. The short answer is, after every semester Lumen downloads the data from every student who’s given them permission at every school using Waymaker across the country. And then they analyze the data. The analysis identifies where the students are having problems. At that point, we go in and either revise the content to make it clearer, or add some learning activities. Or else we revise the assessments to better capture student learning. We do this a little bit in a panicky way over the winter break, because we only have a month. But we do it intensely every summer. Here’s the longer answer. Over time, we’ve gotten better at doing this more efficiently. Lumen has developed something called “RISE Analysis.” RISE is an acronym. I don’t remember what the letters mean. [LAUGHTER] But basically it asked the question, “Which course materials would benefit the most from improvement?” Or to put it differently, “Which changes would have the greatest impact on learning?” So what we’ve done—and this is all programmed now. So Lumen has dozens of Waymaker courses, not just an economics. Though, I like to think that some of the most interesting stuff is started in the econ Waymaker platform. I’m not just making that up, it’s actually true. [LAUGHTER] So, instead of just doing the aggregate sweep on the data, we particularly look at student learning outcomes. And everything in Waymaker is driven by the student learning outcomes. This is out of order, but let me just throw this in for a minute. The way that Waymaker started is they brought together—I want to say 50 principles instructors from everything from community colleges up to R1s. And we spent four days together. And we asked the question, “What do you have to have in your principles courses?’ And so from that we created a list of primary learning outcomes. And then we drilled down and we now have secondary and tertiary outcomes. So the assessment questions in the test banks are coded down to the third level. So everything is really granular, if you want to think about it in those terms. What we look at is not just which student learning outcomes are students struggling with. But rather, which student learning outcomes where students are doing relatively poorly, are they putting a lot of time and effort into. Because that’s where we’re going to get the biggest bang for the buck in terms of fixing things. So what we do is we look at three things. We look at, “Are the questions badly worded?” We’re mostly done with that at this point. “Are the questions testing what they’re supposed to be testing?” There are some psychometric tests that allow you to do that. And then finally, what we do is—after we’ve exhausted all those—we look at the content and we create new content, or different types of learning activities, and we integrate those into the course. So, the interactives that you saw at OLC John, they were the big new innovation from last summer. So we do this, and then we teach the courses again, and then we start the cycle all over again. So, the process just goes on. It’s not continuous, as in every day, but it’s continuous, as in regular. I’ve used the courseware since the first year, and the courseware has gotten noticeably better. Fewer students are failing to achieve mastery on the module quizzes. And fewer of them are crashing and burning. More of them are in the 60 to 70% range when they fail. But what’s really cool is Lumen has shown no sign that they’re ready to quit, that they’re done with this. As long as they’re willing to do this, I think I’m willing to do this.

Rebecca: I like the iterative process.

Steve: Yeah.

Rebecca: That’s something that, as a designer, I’m very comfortable with… that I do all the time, especially designing online. But one of the things that’s really interesting about this model is that, as the author of the textbook, you don’t just have this finished thing. It’s an ongoing…

Steve: …It is.

Rebecca: … thing. So that’s a really different model of authorship.

Steve: Yes, it is. I think it’s fair to say that we make small changes all the time. And then every summer, we make larger changes. And that’s pretty interesting. Because as a user—as you pointed out—I can see that this is helping.

Rebecca: Yeah, that’s really exciting.

Steve: Right now, the hardest part is getting students to trust the process. Because it’s a very different model of learning. And so one of the things that I’m going to do this semester is, build in opportunities for me to remind them that this is a different process, and that they need to trust the process. One of the things that I did last year, which seemed to help with that was I started using exam wrappers after the midterm exams. And ask them to think about how they were studying, and what they would do differently, and what I could do to help them. It’s real easy to see in 30 seconds, I can tell if they’re taking it seriously or not. And if they’re taking it seriously, I learn a whole lot from what they say. So, anyway, just another little wrinkle.

Rebecca: So we’ve talked a lot about the students and the different learning process for students. You talked a little bit about the different processes being the expert, or the writer of the book. And you also mentioned earlier about the peer-review process for an OER being a bit different. Can you talk a little bit about that?

Steve: Sure, and that’s really important. First of all, people have a wrong idea about how OER is produced. The OER that I have experience with is working with OER publishers. It’s not the loan faculty member working in their spare time in their basement, or something like that. Both the OpenStax and the Lumen experience for me, have been very much a team effort. There have been a lot of people involved. So this is really important because one of the concerns about OER textbooks is their presumed lack of quality. There was an article in The Chronicle about that, today in fact. I have to tell you that the peer-review process that I went through with OpenStax was extensive. The way we did this is, OpenStax purchased a manuscript from Tim Taylor—a prominent economist—as the basis for the first edition. They sent copies of the manuscript out to about two-dozen reviewers all over the country, asking them to identify strengths and weaknesses. Based on those review comments, I rewrote each chapter. Each chapter was then sent out to half a dozen new reviewers. And again, the reviewers were from a range of schools, from community colleges through research universities. I took that feedback and I revised each chapter again before it went through the editorial review and production process. I have to say, this was much more detailed and extensive then when I worked with a commercial publisher. The review process for Lumen was similar, there was a lot of peer review involved. And as I said before, I’ve now written two formal editions of both texts. We’ve gotten lots of feedback from users. I’m pretty happy with that.

Rebecca: Do you find that the difference between OER and a commercial publisher is that you keep getting this feedback from users? And that you’re able to revise based on the use of other faculty, rather than working in a silo?

Steve: If I’d written the principles book for a commercial publisher, I would be better able to answer that. I got no formal feedback on my commercial book. I got a lot of comments from people at conferences and things like that. But we have gotten tons of feedback on the OER books, and that is interesting. You can’t satisfy everybody. Somebody says, “This chapter is too long.” Somebody says the same chapter is too short. But, in general, the feedback has been really, really helpful. And we’ve tried to incorporate it as soon as possible. And with these digital text, it’s really easy to do. I can literally go in and edit if I have five-minutes on the fly. And then it’s out there.

John: While with regular publishers, there’s usually a three-year cycle on intro textbooks.

Steve: Yes. And that’s the other thing that—now I’m not a typical user, but I know that if I want to make a change, it’s going to be done by the next semester. The same thing is generally true of other people who give us feedback. Though, they don’t necessarily know that. W e take that feedback very seriously. And there is no three-year review process. So that’s wonderful.

Rebecca: I love the user-centered design process, like that’s clearly what’s being used.

Steve: Yep, we try.

John: And that iterative process is what we should all be doing with our courses, all the time…

Steve: …Yes.

John: …But the fact that you’re doing it makes it easier for instructors who perhaps, don’t have to do as much of that.

Steve: Yeah. But again, let me just say one thing; Waymaker is not my course, Waymaker is my text. So there’s whole levels to my course that go beyond Waymaker. That’s just one element of it. Not that I’m disagreeing with what you said.

John: I’ve seen you present at conferences on teaching principles, for decades now. And I know you’re constantly changing how you’re teaching your courses and trying new things there. And you’ve been doing a lot of great work for quite a while.

Steve: Thank you.

John: Going back a little bit though, to the question of mastery quizzing. When students take the quizzes at the end of a block, you said there’s one or two questions. When they do it a second time, do they get the same question or different questions?

Steve: No. We are adding questions fairly regularly, and so the test banks are getting larger. From the beginning, I think we started with 2000 questions. But again, that’s across the whole book. The questions are randomly chosen, so the odds are that students would get different questions at the self-check level, at the section level. There’s a different test bank for the self checks than there is for the module quizzes. But there are similar questions. In fact, we wrote two at a time basically when we did that.

John: This is a question more generally about Waymaker. Does it do any type of interleaved practice, where later in the course, does it call back earlier sections? Or is it just based on the current module?

Steve: No, it’s just based on the current module. But my more nuanced response to that is, economics is sort of cumulative. But I have thought about that, we just haven’t thought of a way to build it in yet.

John: In my classes, I’ve been adding that the last couple years where I just randomly pull in questions and the module quizzes from earlier modules. Maybe 10 to 15%, building up to about 20% at the end, just to help do a little bit more spaced practice as well.

Steve: I think I know how you could do that pretty easily. Because instructors have access to the test bank that their students are using, so that you can edit your own questions. But what that also means is that you could move questions from earlier into the course to later in the course. So I think there’s a way to do that.

John: Excellent.

Steve: So John, we learned all this in our graduate training, right?

John: [LAUGHTER] You know, it’s getting a little bit better. Some people are learning these things. We have someone in my department who actually came out of Kentucky where he had a lot of training and teaching and learning. But it’s still pretty uncommon.

Steve: Yep.

John: You mentioned two ways in which, OER materials are developed. Some by primary developers, such as the OpenStax and Lumen. And others, with people working in their basements…

Steve: …Yes.

John: …or working in a dark room somewhere. Which is how I often do a lot of my work. Is that process sustainable? And what role do for-profit publishers such as Lumen play in providing these services, or in continuing the development of OER materials?

Steve: There are a couple questions here. One is, is the development process for published OER materials, or OER materials created by publishers. Is that sustainable? And then the second one is, is the individual scholar model sustainable? And those are very different questions. The individual scholar model, I don’t know if sustainable is the right word. I have a colleague who did this, she did it all on her own. I’m so impressed. She didn’t have any support from the school other than a small summer grant. And she did it without any sort of extrinsic motivators. I think that over time, at least at schools like yours and mine, faculty are going to get credit in tenure and promotion, for creating OER, especially open textbooks. I think that’s really important. I think that people will eventually be able to get sabbatical leaves to create these materials. And I think that’s really important to keep that side of the OER creation process going. As far as revision, I don’t know enough about that to really answer that. But I’m curious. I may have to go talk to my colleague Katie now. As far as the publishers go, and I don’t mean the traditional publishers, every publisher has a plan for how they’re going to do this. Some work better than others. I know something about OpenStax and I know a lot about Lumen, about what their sustainability plan is. OpenStax have develop partnerships with a variety of ancillary publishers like Sapling Learning or Knewton. These people provide aftermarket functionality for the OpenStax books, and in return, they get kickbacks from these ancillary publishers. And by kickbacks, I don’t mean anything pejorative about that. I just mean that they contribute financially. I don’t know any more about how sustainable that model is. I know that that’s what OpenStax has been using. Lumen from the beginning, has been a commercial publisher. It took me two years to figure out how a commercial publisher could make money giving their content away. Maybe others haven’t thought about that, but I sure did. So, the short answer is, Lumen gives the content away, but charges a very modest amount, $25, for the intelligent backend. All the feedback that goes both to the students, and the instructors. Today, you personally, either of you, could go and get a copy of the Lumen Principles and Micro book, or the Principles of Macro book, and it’s yours forever, you can do with it what you want. But if you want to take the full Waymaker course, they charge $25. The idea is, that amount of money is both affordable to students, but also enough to maintain revisions and corrections, and keep the servers running and all of those things. So that’s the answer to that question. And I will say that every semester, I try to be completely transparent, and say, “If you don’t want to pay the $25, you can get all the content for free. But here’s what you lose.” In five years, I’ve never had a student who didn’t pay the $25, because they thought it was like beer money for the weekend, or something. Compared to spending 300 bucks on a traditional text that was nothing to them.

John: What are some of the barriers that you see to faculty adopting OER? You mentioned that people may have this perception of lower quality…

Steve: …Yes .

John: …but there’s quite a bit of evidence that the quality is not weaker in any way. And I think you had done some studies on that a while back, didn’t you?

STEVE. Yes. The number one problem I think is misinformation. The majority of faculty today don’t know what’s available in their discipline. Many of my colleagues have told me, “Yeah, OER sounds like a great idea, but there’s nothing available in my field.” Now, that’s flat out wrong. For your listeners, there is OER available for nearly every Gen-Ed course taught today. So that’s number one, is lack of knowledge of what’s available. Number two is, as you mentioned before, the belief that OER is inferior, that there’s no peer review. And that’s just not true. There’s a couple things here. One is that OER publishers don’t have a sales force, and so it’s going to take longer to get the word out. There’s been a lot of progress over the last few years. But at my school, we’re only in the second-year of our formal OER initiative. So we’ll see how it goes. The other thing that I think gets in the way of adoption of OER is path dependence, and the unwillingness of many faculty to change their textbooks because of the fixed costs involved. “I’m going to have to go through my lecture notes and make sure that I’m using all the same terms as the textbook does,” and that sort of thing. I don’t know the answer to that question. I know that some schools have used financial incentives, fairly modest financial incentives, to get faculty to try to make the switch. As far as my own assessment goes, every summer, I do statistical analysis of the effectiveness of the texts that I’m using. I looked at both the OpenStax Principles book, and also most recently, the Waymaker package. What I’ve looked at is, textbook alone, textbook with ancillary website, digital courseware, and because I used to teach a writing intensive version of the principles courses, I also looked at writing intensive. And what I found is pretty predictable, at least from somebody who has done this for a while. What I found is that there is no significant difference between student learning using OER, with commercial textbooks. I found that using either courseware or an ancillary website improves student learning outcomes, regardless of what the text is that you’re using. And I’ve also found that writing intensive courses seem to work better than non-writing intensive courses, because the students are getting into it in more detail. Over the last two years, I’ve been doing a randomized control trial, where I can really drill down and see what’s going on. And what I found is that using the full Waymaker package seems to have a statistically significant positive impact on student learning. So I’m going to rerun the analysis using last semester data, which I haven’t had a chance to get yet, but I’m anxious to see how that goes too. I believe this stuff works. And so I think sooner or later, more and more publishers—the commercial publishers too—are going to move towards digital courseware type products.

John: I think most of them have started to at least.

Steve: Yes, but it’s like turning the Titanic. Their base is so large that it’s going to take a while before even all of those people get on-board with this.

John: One thing I was wondering is whether you see more collaboration or competition in OER textbooks?

Steve: Initially, there was more collaboration in the early years. And the reason why is because anybody who was doing OER, was increasing the interest in users for everybody’s OER. Now, I think we’re going to see more competition between the users. Especially as more publishers are going to adaptive and personalized learning type courseware. I think that’s a way that publishers are going to be able to say, “Well, yeah, we’re doing that. But we’re doing better in our own particular way.” So I think there’s going to be a fair amount of product differentiation. And it will be harder for faculty, it’s going to take more work to dig in and see exactly what’s going on. I would love to see more published assessment of efficacy on the part of the commercial publishers. They’re only now starting to do that, and the studies that they publish are heavily controlled by them. So it’s not clear that they’re telling us about all of their things, just the ones that work. But at least it’s a start.

John: One of the things I see in most of those studies is comment to the effect that, “Students who use our adaptive learning platform have letter grades on average, one letter grade higher or point eight points higher.”

Steve: Yes, that’s right.

John: And there’s no evidence that they’ve done any control for the students who chose to use it versus those who didn’t.

Steve: That’s right.

John: But it would be nice if we could see more research on that.

Steve: And I think we will. At least I’m hopeful.

John: Earlier you told us a little bit about how your course is structured with some “just-in-time teaching,” and some activities there where you have students work on problems. Could you tell us a little bit more about how you structure your course so that it’s not duplicating the textbook?

Steve: The first thing that I would say is that, my intro course looks like almost anyone else’s Principles of Micro or Macro course. If you look at the course outline, it has all the normal topics in it. A very slight difference is, instead of assigning students chapters to read and problem sets to do, students have modules with content and learning activities to complete. There is some difference between my face-to-face sections and my online sections, because I teach both. My face-to-face sections are pretty much the way I described them to you earlier. My general approach is to do Socratic lecturing with a lot of in-class activities, like the supply and demand problems that I mentioned. I also like to have formal in-class discussions on interesting questions that don’t have a right answer. In the macro class, I spend a day talking about what is money. And I spend the day talking about what is government. And those are things that aren’t done in the same way and the same degree with a textbook, whether it’s Waymaker or something else. My online course is roughly similar. But what I do is I add group and individual activities to the online course to mimic what I do in class. I also have a weekly Google Hangout, a synchronous Google Hangout, where I can give students guidance about what I think they should be doing. And I can give little mini lectures on things that I know students have trouble with. But it also gives them a chance to ask me individual questions in a real time basis, one on one. Not a lot of students come to those Hangouts. I usually have between five and ten, and my classes are about 35. But more than 90% of the students watch the recordings. Google Hangouts are automatically recorded and archived in YouTube. So the students seem to like that a lot.

John: You mentioned that a number of people at Mary Washington have switched over, what proportion, would you say, of the faculty at Mary Washington has moved to using OER?

Steve: Single digits, a handful, probably less than ten at this point. But this semester, I have two new people. So I’m excited about that. And we haven’t yet given them any money or anything to do this. I’ve just been talking to people. I was invited to the College of Business’s summer retreat, and I gave a little talk about OER. And I got two people who expressed an interest in following up. One of whom has already done it. So I think we’re getting there. We just have to be patient.

Rebecca: So we normally wrap up by asking, well, what’s next?

Steve: What’s next for me is I’m continuing to iterate to improve Waymaker. I’m going to continue doing my own statistical analysis. So I get access to the aggregate analysis that Lumen does, but I also have my own analysis. So I can tailor that to my particular students. I also want to do something this semester that I’ve wanted to do for a long time, but have never done it. And that is to write a new non-traditional chapter for the micro book, which is relatively easy to do. It’s just really a question of me sitting down and doing it. So I know it’s doable, but I do want to actually make my version of Waymaker different from the standard version. In part, because it’ll better match the way I teach. But also because I want to see that it’s relatively easy to do so that I can talk about that to faculty.

John: Very good.

Steve: I’m going to the CTREE conference this summer to talk about Waymaker. And this is the first time we’ve actually reached out to a disciplinary conference. So I think that’ll be fun.

John: You know, I always want to go to the CTREE conference, but I teach at Duke in the summer and it runs right into that. So I haven’t been able to go. And we should note that the CTREE conference is a Conference on Teaching and Research and Economic Education.

Steve: I love to talk about this stuff, because I believe it.

Rebecca: Yeah, it was really interesting.

John: Thank you.

Steve: Oh, you’re very welcome. Thanks for the opportunity.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kelly Knight, Kim Fischer, and Jacob Alverson.

[MUSIC]

65. Retrieval Practice

Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, Dr. Michelle Miller joins us to discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Her academic background is in cognitive psychology and her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications.

Show Notes

  • Miller, M. (2014). Minds Online: Teaching Effectively With Technology. Cambridge, MA: Harvard University Press.
  • Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319(5865), 966-968.
  • Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 1199327.
  • Kahoot
  • Bray, Niki (2018). 43 to 0: How One University Instructor Eliminated Failure Using Gamified Learning. Blog post
  • Retrievalpractice.org
  • Lang, J. M. (2016). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • Pennebaker, J. W., Gosling, S. D., & Ferrell, J. D. (2013). Daily online testing in large classes: Boosting college performance while reducing achievement gaps. PloS one, 8(11), e79774.
  • Pauk, W. (1984). The new SQ4R.
  • Thomas, E. L., & Robinson, H. A. (1972). Improving reading in every class. (a discussion of PQ4R)

Transcript

John: Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, we discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today we’re welcoming back Dr. Michelle Miller. Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications. Welcome back. Michelle.

Michelle: Thank you so much. It’s so great to be here today.

Rebecca: We’re so happy to have you again. Today’s teas are:

Michelle: Well, I’m drinking Coco Loco, which is a blend from a local tea shop here in Flagstaff, Arizona. Steep Leaf Tea. And Coco Loco is a lot like what it sounds like. It’s chocolate and banana. So tea snobs may scoff at my choice, but it’s wonderful.

John: IIt sounds good.

Rebecca: And I think I saw a nice silver teapot that was poured into a green and blue tea mug.

Michelle: Yup.

Rebecca: A nice tall one.

Michelle: That’s what I need.

John: And I’m drinking ginger, peach green tea.

Rebecca: I went with a Christmas tea today. So Michelle, we invited you here today to talk a little bit about retrieval practice. Can you first start with defining that for us and letting us know what it is?

Michelle: Right. So retrieval practice is essentially the act of pulling something out of memory. So that is, in memory research, what we would term retrieval. So something is stored in memory and we want to pull it out so we can actively use that information, have it in our conscious minds and so forth. And so we go through this usually very fast process called retrieval. So retrieval practice is specifically the act of doing it, and we contextualize that with learning. So, when I’m trying to learn something or I’m the process of learning something, to say, “Oh, what was that fact I remembered, or what can I say about this?” When we do that, it produces something that we can call the testing effect. So this is kind of the clearest example… not the only… but the clearest example of retrieval practice in action during learning is when we sit down to take a quiz, take a test or something like that. So all the excitement that’s happened around retrieval practice in higher education, and really in the rest of education today, is around this finding, which has been replicated many, many times: that tests are, as one person put it, not neutral events in learning. When we take a test on something, that has a very powerful effect on our ability to remember it in the future. So, really simplified down to its core, tests help us remember in the future; when we take a test it strengthens our memory. So that’s what retrieval practice is and it can, as we’ll maybe talk about today, take many, many forms in learning settings. And I did want to clarify too, this is something that I definitely don’t want to take any kind of credit for discovering this. This has been around and has been known about for a long, long time. Some of the big names who are associated with this: Jeffrey Karpicke, Robert Bjork. Roddy Roediger, there are quite a few really heavy hitting cognitive scientists and cognitive psychologists who have established this. But there are many, many of us who are out there trying to disseminate this to other teachers around the world so that we can all tap into the power of this. And I have done a little bit of work in this area with my colleague here at Northern Arizona University, Laurie Dixon, who’s another psychologist… and we teamed up some time ago to look at a very practical implementation of retrieval practice in an Introduction to Psychology course that we conducted some years ago and this is a course, you can imagine, where just trying to get students to perform even a little bit better is a big project. So we examined how even something kind of basic… it was very high tech at the time… but just basic web quizzes that came packaged with the textbook. We said, “Well, if we actually assigned students to do these as part of the course, and if they went through and treated these as opportunities to learn, not just assessments in and of themselves, would that have any systematic impact on course performance?” And we found that in fact, there was a significant improvement associated with that. So that’s kind of the landscape of what retrieval practice is, and why we’ve been so interested in discussing this in the psychology of teaching and learning.

John: In fact, I saw you present on that about 11 or 12 years ago in Orlando at one of the NCAT conferences and it convinced me to completely revise how I was giving my classes and it’s made a big difference and resulted in some significant improvements in student learning. Given that we know so much about retrieval practice. Why are faculty resistant to doing this?

Michelle: Wow, that’s a great question, and that is one that I have been really facing a lot these days in my own practice, talking to other faculty members, and disseminating this through some different activities I do in this area. It is easy for those of us who work in this area, cognitive psychologists in particular, but a lot of us who like you heard about this a long time ago. We’ve seen the power of it. We forget that to other faculty, this can be a very off-putting concept. And so it’s really great for us to think about why that is. And I always think that’s really good for me to kind of go back to that and say “Yeah, not everybody is sold on this idea. And there are good reasons for that.” So like with a lot of things that we talk about in teaching and learning, I think that these really break down into two neat categories: there’s the philosophical issues that people have with it, and then there’s more practical and logistical issues with it. So kind of tackling those one at a time. Philosophically, when people say, “Yeah, I understand about the research but this just goes against something that I believe as a teacher or how I want my classes to be. Here are some ways that that can play out. First off is this idea, that I’ve heard in one form or another quite a few times, and that is this notion of superficial learning. So “Okay, sure, there’s a study that showed that maybe people retain something a little bit better. But surely that’s not this deep learning, whatever that is.” That’s a concept that we all want. So do tests and exams… just testing… create just a superficial form of learning? And while, of course, I understand that, and I absolutely applaud faculty for really thinking deeply about that issue, and caring about it. I Here’s the thing… we got to define that. We social scientists, that’s what we do… we have to kind of break things down and say, “Okay, what does deep learning mean?” and I don’t know that anybody has kind of definitively done that. But when I look at that, I say, “Well, this is not just one research study that showed a little improvement in a lab test. There’s quite a few studies that do use realistic types of materials. It’s not all just contrived laboratory studies. Furthermore, there’s also studies that show that when students engage in more quizzing and testing on material that they actually are able to transfer that learning better. And that is a very, very big deal in teaching and learning as a lot of us know is not just getting students to be able to solve a problem in one context or work a concept in one context, but can they do it in the next circumstance? And that very difficult process is aided by quizzing… and to me, what could be deeper learning than learning that transfers? So that’s part of it. Some of it is perceptions around multiple choice quizzes and tests. There’s an assumption too that if we’re talking about quizzing we must be talking about multiple choice questions… and first off, sometimes in larger classes, those are the final assessments… in that Introduction to Psychology course that we studied years back, that’s what the assessments were… so, having students practice in that format I don’t think that we should necessarily dismiss that. And as we can talk about in a little bit, there’s lots of ways to induce retrieval practice that actually don’t involve multiple choice questions. So, there’s a bit of that as well. And something that I’ve talked about with some faculty recently, too, is this baggage around K through 12. And maybe that’s something that’s resonant with you all.

John: Yeah, that’s given the testing effect a somewhat bad name, because high-stakes testing is being used in a lot of what’s going on with K to 12. But that I don’t think is what retrieval practice as you’re suggesting is all about.

Michelle: Right, and I have to be cautious here. I really like how you laid that issue out in K through 12, that there is a reputation problem… and that has happened because of the high-stakes standardized testing policy in the United States. And I got to be careful because I don’t want to represent myself as an expert in K through 12 or in K through 12 education policy. But I don’t think you have to be an expert in that to know that there’s been a lot of same pretty well justified public pushback against over-testing in K through 12. And yeah, I think that we absolutely do have to be aware of that. Students come to us in higher education… that’s a system that many of them have been through and our faculty are very aware and very cognizant of that too. So, nobody’s a blank slate here, not our students, not our fellow faculty. We have assumptions and ideas and experiences about testing that happen. I think those can be addressed. But, yeah, that is definitely another very big barrier. We got to differentiate between high-stakes standardized testing for the reasons it’s done in K through 12, and low-stakes testing and quizzing for learning as proponents of retrieval practice would have it.

Rebecca: Some of the pushback I’ve heard from faculty fall into two categories related to this as well. One is that they assume that retrieval practice is best implemented in 100, 200 level introductory classes instead of upper level 300, 400, graduate-level classes. And then the other area is that paper and pencil tests don’t make sense and all disciplines. And so they assume that a test has to be in a paper/pencil format, which could be online testing, or it could be multiple choice or it could be essay questions. But I think that, from being someone in the arts, like there’s other ways to test beyond that, but we don’t think of those as tests.

Michelle: Right. That’s another great lens through which to look at this issue; that we do need to broaden the definition to draw more attention to this and to make it a more appealing concept. But yes, how can we make it broadly appeal across lots of disparate disciplines? Not only does it not have to be a multiple choice type of exam, maybe it’s not a pencil and paper exam at all. And we as faculty have to think about what makes sense there. You make a good point about the levels concept. I think, these days most of us have heard of, “Well, there’s one particular Bloom’s taxonomy…” which is a wonderful framework for getting us thinking about being systematic about what we’re asking students to do with the information that we’re bringing to a course and trying to do things like align the teaching we do with the assessments that we have. That’s wonderful. However, I think it does ingrain in us that idea that “Well, just knowing things is sort of at the bottom… You sort of get that out of the way, and then we go on to the good stuff.” And from a cognitive perspective, these relationships are much more fluid and much more interdependent, so that yes, absolutely, the higher thinking that is what we want. That is what we should want. Or if we’re in highly applied disciplines (if we’re in the arts, for example), we need students to be able to do things with that information. But they have to have that. So I think it challenges us to think of new ways with that concept as well.

John: One of the barriers I think some people have is they don’t like to grade tests and so forth. But one of the things you mentioned in your book is that the testing effects been known for a long time, but it was really difficult to implement in terms of low-stakes testing, particularly when you’re teaching at a larger scale. But, as you’ve done yourself, and as you suggest in your book, computer technology makes it easy to automate some of this… certainly more easily for multiple choice and free response and similar things. But it makes it a whole lot easier for both students to have multiple attempts at learning something using some type of mastery quizzing and it makes it a whole lot easier for faculty who don’t have to spend all their time grading.

Michelle: Right and that’s absolutely where the practical stuff comes in. So, we’ve worked through some the philosophical objections, that: “No, this does not turn your classroom into some terrible assembly-line concept of learning. It’s not going to create a bad relationship with your students. It’s not going to simply carry on a legacy of bad policy as people perceive it. It’s not going to do those things.” And then, we do get to: “Alright, but what is this going to do to my life as a faculty member?” …and this is important stuff. Those who know me know I’m a big fan of James Lang’s work in his books. One of his more recent books is called Small Teaching and there’s a lot of different takes on it in that book, but it really hits home with respect to “We do have to think about not everybody’s in a position to, nor should we even try sometimes, to just take everything down to the foundation and rebuild.” That we do really need to think about “Do I want to do this, if it’s going to create 800 more questions for me to grade?” Is this a sort of a situation where, “Well, I don’t want multiple choice, so I’m going to have to give these open-ended questions. I’m gonna have to give feedback and I have 200 students, what will happen? If I am going to go with multiple choice questions… well, how am I going to do this? to have to write all of this?” And yes, it’s one of the most powerful outcomes of the educational technology revolution that makes it workable, and scalable in a sense, even with large classes to do these types of things and to bring them in. So, that is definitely a message that I hope faculty think about if they’re on the borderline of wanting to bring in more retrieval practice into their classes.

Rebecca: I’m in a discipline where the multiple-choice questions are using things digitally doesn’t always work for testing and practicing some of these basic things, and there’s not a good way to automatically grade it. But one of the strategies that I’ve used is actually some self grading, which has actually worked pretty well. I just check and I have them write notes about anything that they got wrong. So, it demonstrates that they’ve tried to understand when we go over it, and I give credit based on how thorough those notes are, rather than whether or not they got the question right or wrong. And that’s made a difference in my classes. And had I not come up with that solution, I think I would have abandoned it because it would have been too much work. But it it actually is working pretty well.

John: When we had a reading group on Small Teaching last year, one of the things that was widely adopted by faculty was a very simple form of retrieval practice where they had students at the start of each class reflect back on what they had done in the previous class. And most of them have continued to do that in subsequent classes as well.

Rebecca: One of the other barriers that faculty might raise is the idea that students don’t take low-stakes things seriously, or that they don’t put the same kind of time into it that they might for something that’s high stakes. Can you talk a little bit about how we might help students find value in retrieval practice and subsequently also with the faculty then?

Michelle: Right, so that, ”Well, we can put it out there, but will they do it?” I’ve kind of crossed this philosophical and practical barrier for myself of giving some credit for pretty much anything that I am hoping that students will do. I put the work in to set it up and I do believe as a teacher that there’s reason to believe that will help their performance, that I need to work it into the syllabus somewhere. I don’t think it detracts from learning, necessarily, to say, “Yeah, there’s some points associated with this.” And especially with our students who, for many of us, are going in a million directions at once. They’re juggling jobs, multiple classes, sometimes their own families. So having an incentive in the form of points—having some kind of a payoff—I think, helps them make that decision that this is at least worth the time to do. I think the other thing that we probably can all do more of, and that I’ve done more over the years, is framing and honestly marketing this to students… communicating with them about why. And when students disengage from an activity like this, when they say, “Ah, why do I have to sit down and do this thing? This is just another test. Oh, no.” …really conveying the excitement and the goodwill that we have in setting those things up can go a very, very long way. Of course, a student, if they just look at it inside and they have no context for why this was put into place, they’re going to have them say, “Well, maybe I won’t do that.” But when we can market to students, we can say “There is a lot of research that shows that this is a very, very good use of your time. And hey, you’ve probably taken a lot of tests in your life that were really about measuring or sorting you and figuring out what you know and what you could do. This is a very different kind of test.” So that can go along way and get the C students nodding: “Alright, I get it. I get why she put this assessment right here.” I think a lot of us have hit on the practical strategy too, that the little Easter eggs or goodies that we plant in the form of questions that get re-used on the higher stakes assessment. So most of us will have tests for measurement at some point in our courses. And yes, students really do pick up on it when you use one, two or more of those items that were in, say, the gamified quiz that you ran in class or the reading quiz that they did beforehand. They can see those and say, “Oh, wow, I got feedback on that. I got an opportunity to practice…” and if it draws in a few more students who see it as almost a legitimate form of cheating, honestly…. like a fun and sanctioned form of getting an advance sneak peek at the exam, then great! Then that maybe is an opportunity for them to come in and see that. So there is that. Actually taking it seriously ourselves, not just in the form of saying, “Well, here’s the points I’m going to give you for this,” but spending class time on it. That’s a big bridge for a lot of us to cross, right? Because we as teachers tend to be very focused on “Oh my gosh, class is for covering material. We use all these sort of distance metaphors to talk about what we want to do with our class time. But if I say, “You know what, guys, I believe in this, and I believe in it enough to where we’re going to spend the entire class period before the final exam….” I did that twice last week myself, when running exams. Or we’re going to spend five or 10 minutes of the beginning of every class period doing this, as one project recently published about doing. If you show yourself doing that, and offer them that, I think that also goes a long way towards it. And I guess to just say, well, taking it seriously… here again, what does that look like? What does that mean to different people? And we can kind of a little tongue-in-cheek say, “Well, why do we have to take it so seriously?” Sometimes games and learning can happen when it is presented in a more fun context. So not everything has to be deadly serious or spending hours and hours and hours of stressful time on. There are occasions when a light-hearted approach can be perfectly good and can still get us involved in that really critical activity of retrieval

Rebecca: I can share an example of doing that in my classes. We’ve done design challenges and sometimes we challenge other classes that are happening at the same time. That reinforces some of the basic principles that we think that students should be doing and reminds them of it… and they might work in a team…and then we have a competition. And it’s fun and what have you. And students like those, it breaks up the day, it makes it more fun. And then I’ve also done things where I give class time to do little design challenges in class that might be individual and then they can level up to working with a partner to finish solving a problem or something… and students value that. They recognize that next time they’re trying to do a project on their own,that it’s easier because they’ve had that practice or that opportunity for the retrieval practice. And my students have actually ended up asking for more of those opportunities.

Michelle: Great.

John: Could you go back just a little bit and tell us what you did in those couple of classes last week before your final. We’re recording this, by the way, in early December during finals periods in both of our campuses, but we’ll be releasing it a few weeks later… to put that in context.

Michelle: Oh, okay. Well, I’d love to. …and I’ve been talking about retrieval practice as you pointed out for years, and I still discover new ways to infuse this into courses. And the context for this is my cognitive psychology undergraduate course. It’s a 200 level. So it’s a lower-division course. And it’s about 60 or 70 students. And as you can imagine, it is a bit of a tough sell. For many of the students it’s their first encounter with this side of psychology. It’s not as intuitive as some other areas of psychology. So there’s a lot to learn and a lot of motivation to be done. One of the ways we bring this and in this course is using a technology called Kahoot, that’s spelled K-A-H-O-O-T,, and it’s really very intuitive and functions very smoothly… relatively free of bugs. That’s good stuff. It’s a program for doing gamified quizzes of various kinds. What I did, and at different points in the semester, and then really amped up in the last week of the semester is running these gamified quizzes. And this is something, by the way, that I hit on and got the idea to try based on a colleague named Niki Bray, who’s from Tennessee, and has actually done some really systematic work in reformulating some of her courses around in class quizzing in just really ingenious way. So I saw some of her presentation and I said I’ve got to try this for myself. I went with multiple choice questions. Kahoot does have some parameters… questions do have to be short…very, very short. And to some people that may be off-putting, but you can put together quite a few of these. And so we would put this up on the projector and students have the option of dialing with either their laptop or their smartphone and weighing in on each question. The neat thing about it is it has an algorithm for giving points based on your speed as well as your accuracy and it’s got a little leaderboard so you can actually have a little in-class competition. Now some people who use this do require all students to do it and they actually issue points for performance. Now, I presented it very much as a practice activity… and made it very, very clear because of my philosophy, I’m not going to assume that all students have devices or have smartphones or laptops or that they want to do that. But I said, “Look, remember we’re talking about retrieval practice, guys. So the real meat and potatoes of this is not buzzing it on your phone. That’s fun. But the real benefit of this activity is what you’re doing sitting there in your seat. And you could be doing this with a piece of paper if you want.” And that is what a few students opt to do. They try to answer the questions, they know what they need to go back and review and so on. And it’s nice because it spits out at the end, a whole report that tells me right away… Okay, which questions do we need to revisit? Which ones did students have the hardest time with and so on. That’s one of the things that I just brought in. And yeah, it was a big deal. I sacrifice a chapter of “coverage” so that we would have more time at the end of the semester. But to me, I would rather have students going into the final knowing that they’ve had this retrieval practice and they have a better chance of performing really well. Earning a good grade on this material I care about than honestly cramming in a little bit more mileage in terms of the quantity.

Rebecca: Sounds like fun.

Michelle: You know what, it does really bring a fun factor. There’s been a lot of different variations on in-class polling, and I will admit to this, I actually purchased and am the proud owner of a physical buzz-in quiz device, complete with a whole spaghetti nest of wires and an incredibly abrasive, buzzer sound, and everything. So previous to this, my educational technology did include… I think I could have up to eight intrepid volunteers who would play a quiz game and then I would have to appoint a points keeper and all this… and props are always a lot of fun. So when I say I believe in bringing in retrieval practice, I really do walk that walk. But I will say doing it by a smartphone does allow for more participation, and I don’t have to worry as much about the minutiae of scorekeeping and stuff like that.

John: I played with Kahoot a little bit at my classes at Duke and students have loved it.

Rebecca: We’ve talked a little bit about framing things so that students take the practice seriously. What do we do about the students who just push back, it’s like “This is too much work. This is a lot of extra time…” or that sort of argument.

Michelle: I think that that’s another piece of that barrier to more faculty adopting this, not just the work involved, but realistically, student opinions. Student evaluations matter a lot to faculty life. And of course, we all want to have that wonderfully rewarding semester, not the semester where we feel like we’re at odds with our students. So I do think a little piece of this is we do anticipate sometimes worse and more pushback than what actually happens in the end. So I think we have a fair amount of sort of a dread factor when we go into something new like this. But that said, when students have an issue with more quizzing or more testing, here’s how those come out. I think, first of all, we do have to sometimes separate out the technology aspects of it from the quizzing or testing, per se. So, as John mentioned, this is one of the amazing things that educational technology does. But the flip side of that is, if you’ve ever used technology for education, in any shape or form, you know that it breaks down. And when it breaks down at a big class, that’s a headache for everybody, it’s a misery. So if you’re trying it, and things are not working out, you just got to figure out “Okay, how much of this is the assignment, the activity per se, and how much of it is that wonky quizzing thing that I got from the publisher and it fell apart, or students hanging up when they’re trying to dial in with the poll and fine tuning those. I mean, when I first used kahoot, I decided it would be a lovely idea to put four chapters worth of material into a 40-question quiz. And when you get used to as you know, that they took a long time, I mean, 12 questions can keep you going for a very long time, especially if you’re discussing… and some students got kicked out part way through and then they weren’t on the leaderboard and I could have set the whole thing aside. But really, that was more I needed to get the technology working. So that’s a big piece of it. I think that sometimes it is a perception issue with the timing. Now, those of us who are just all over this as a teaching technique, we like to do reading quizzes before we talk about that material in class. So chapter three is up on the syllabus and your chapter three reading quizzes due on Sunday night before we do that…, and that can provoke a fair amount of confusion and honestly griping with students. They say, “Why did I get tested on this when we didn’t do it… we didn’t cover it in class yet?” And what do you know, that’s another framing and communications issue. Once we know that that is why we’re doing that you get so much less on that side of things. But here too we do also ourselves have to follow along with what we say. If we say you’re doing this reading quiz so that we’re establishing a foundation. we don’t have to teach everything in the class itself. We could spend the time applying. Well, guess what, then you do have to do that. So if I assign you to do the reading quiz on chapter three, and then on that Monday morning I go, “Okay, we’re going to go over chapter three starting at the top. And I’m going to show you all the PowerPoints for all these things that you already read.” Well, yes, then student morale and student support will fall apart at that point.

Rebecca: Those are very good reminders.

John: Yeah, I’ve pretty much adopted this approach all through my class beginning when I first saw you present it. So I have students do the reading in advance, I have them take reading quizzes with repeated attempts allowed on those. And then in class, I have them working on clicker questions, and I and the TAs go around and help them when they get stuck on problems. But there is an adjustment and students, especially when I first started doing it, would generally say, “But you’re not teaching me” …and you do have to sell students on this a bit. One source of resistance is that when students take quizzes, they often get negative feedback. When they read something and they read it over again, it looks more and more familiar. They’ve got that whole fluency illusion thing going and they become comfortable with it, and they feel that they’re learning it… or similarly, if they hear someone give a really clear presentation on a topic, it feels comfortable. They feel like they’re learning it until they get some type of summative assessment where they get negative feedback, and then they feel the test was somehow tricky. But it’s a bit harder to convince them that actually working through retrieval practice, watching videos at their own time and pace, reading material as needed, and then spending class time working through those problems is as effective. I’m getting better at it. But it’s been a long time trying to convince students and I did take a bit of a hit in my evaluations, especially the first few times.

Rebecca: Do you mean learning’s not easy?

John: Students would like learning to be easy.

Michelle: You know, it’s funny, I think almost in a way… see what you think about this idea. But it’s almost a mirror image of our illusions as teachers, right? That I gave a wonderful clear lecture and I assigned wonderful readings and I saw students highlighting them. Therefore, students must have assimilated this knowledge and it must be in there. So I think it challenges our students but it also challenges us as well. It can be quite an eye-opening experience to running something like a Kahoot. And that brilliant point that I gave this great example for…. what do you know…. 7% of the students actually nailed it. And so we can all use a reality check… teachers and the students. And you mentioned re-reading, that’s another one where I’ve had some pretty intensive conversations with other faculty, I’ll kind of say, “Oh, well, and there’s this great research that shows that students tend to re-read when they’re studying and we know that from a memory standpoint, that is really, really ineffective.” And faculty will say, “Whoa, whoa, whoa.I want my students to be re-reading.” Of course, now when they say re-reading, they may be picturing deeply interrogating a text… annotating it… looking at it from a different perspective. And absolutely, that’s a wonderful part of scholarship. But that isn’t what we’re talking about. We’re talking about students re-reading as a study form and mistaking highlighting for deep interrogation. So, just like with the rest of knowledge bases versus higher-order skills. This is another words “both and” it cannot be “either or.” Yes, we want students re-reading in the right ways, but students or teachers cannot mistake that for learning sometimes.

Rebecca: So, let’s say you’ve just convinced all of our listeners that we need to be doing this, how do we bake it into our course designs?

Michelle: Well, I think that, really getting creative with this, and as I talked to faculty when I visit other schools or talk to faculty of my own institution, I just see all of these new ideas all the time about how to do this. Once you do get that critical epiphany of alright, a test doesn’t have to look like a test on the surface… it does not have to be the ritual of “Okay, you’ve got a number 2 pencil that’s breaking and I’m standing over you while you have a panic attack for an hour.” That’s not what it is—that it’s about the retrieval. Once you get that, all kinds of creativity opens up. So I’ve also started sometimes on the very first day with the syllabus quiz. So, especially if you have a smaller class, we all struggle with that “I spent all week writing the syllabus and it’s incredibly important, and then I’m getting questions about stuff that’s on it all the way through the semester.” So, really on the first day, what I do is I take a very light-hearted approach and I divide the students up into teams, just physical teams. Everybody’s got a copy of the syllabus, sometimes I fake them out and I say, “Okay, we’re going to go over the syllabus point by point…” and I say, “No, you can read the syllabus. We’re actually going to do this other thing which research shows will actually help you remember it.” So the task here is that each team formulates a set of questions… you can make just a few. like three… so a little bit of teamwork. So formulate three good questions off the syllabus. And you’d be amazed, they come up with questions that even I can’t answer sometimes without having to cheat. So they talk about it. And then of course, you go around the room in some arrangement and each team gets to ask the other team their questions, and if you really want you could keep score and everybody loves bragging rights for being the team that stumped the other teams and won the most points. So, it can literally start right then. With that idea that I don’t read stuff to you in this class, this is about you. And I’m also not piling a huge amount of work on myself either apart from being the moderator and the MC and having written the thing in the first place. I don’t have to write questions. They’re writing the questions and they’re answering them too. So those are some of the ways that we can do that. I think especially in our larger classes we do want to think about things like peer grading or peer review of open-ended question responses. We do want to take advantage of things like publishers’ test banks to set those up as reading quizzes. And you’d mentioned earlier about “Well, what about this not being suitable for upper-division classes?” I had an upper-division class that just wrap this semester where we had reading quizzes as well. It may have been an upper-division course but it also serve that purpose of “Hey, you take a basic quiz over the chapter on Sunday and that really sets the stage for us to have a more substantive discussion.” All of those things. are ways that we can do this. I think open-ended reflection as well… so tests that look nothing like a test but are still retrieval practice. You’d mentioned about reflecting on what you learned last class period. So this sometimes goes by the name “brain dump.” And I did a bit of this last semester as well. So, this by the way, I do want to credit the great website retrievalpractice.org. So retrieval practice is actually that high profile it has its own website. It’s an absolute treasure trove of ideas. So, with the brain dump the way that I did it, or similar to what it sounds like you did, every now and again we start off class with you writing down everything you remember from last time on a piece of paper. I had students then turn to their neighbor and compare notes and see what they came up with. And I didn’t grade those. This was not a heap of grading for me. But in the end what they did turn into me for accountability and a few points was a very short reflection, just on an index card. So, “What surprised you?” and I review those and they’re very eye-opening but again I’m not there to police or micromanage what they put on their cards. So, that’s retrieval practice too. There’s lots of different flavors that we can bring in.

John: I do something similar with asking students to reflect on the reading in each module we work through. But because I teach a fairly large class, I didn’t want to have to deal with all the index cards. So I just have them fill up a simple Google form. And then I can skim through it and assign grades much more easily than shuffling paper. But it’s the same basic idea and it’s worked quite well.

Michelle: Absolutely.

Rebecca: Do you have any other examples of the kinds of little tests that faculty can run that don’t look like tests?

Michelle: Right? So tests that don’t look like tests. Well, here’s another that was probably a little bit more practical for a small class, and this is how I ran this. It, like the brain-dump exercise the way I ran it, was also very cyclical and very student generated. So we started out each class at the beginning of each week with students generating a set of quiz questions based out of the assigned reading. So we didn’t actually, in this particular example, have pre-quizzes or something like that. But students came in knowing that they can bring their book if they wanted. But they would need to sit down and write for me three questions, whether short-answer or multiple choice. Then I can flip through those and I would really quickly put those together for a quiz that went out the subsequent day. So, we’re alternating between generating questions and then returning to those questions. And then I would have pass out. I told them treat this like a realistic test. Actually try to retrieve everything, but, you know, when times up, you’re going to get to go back to it. And then they would grade it themselves. So I didn’t actually do the grading either. And that is really great for spurring discussion. And “Oh, my gosh, I thought I knew the difference between reliability and validity, but now that I tried to answer it, I realized that I didn’t.” And then you can throw it back out to the student who now has bragging rights for having had their question selected and say, “Hey, what did you mean? What was the right answer? And why did you put that down there?” And at the end of the day, they kept that quiz too. So it was really very much in their own hands to do. So there’s that. Other creative ideas that I’ve run across over the last couple of semesters… There’s a great project out there, run by Bruce Kirchhoff at University of North Carolina at Greensboro. So I got the wonderful pleasure of talking to that group last year. And he and some colleagues working in the area of botany actually put together a freestanding, custom built mobile app that students could take with them that presented different kinds of quizzes over the sorts of things that botany students really need to know like the back of their hands like how to identify different plants and how to discriminate among different examples and they found some empirical evidence that this actually raised performance up quite a bit. I’ve heard too of another Professor put together some surveys in Qualtrics, the surveying software took advantage of its ability to actually text message people and send them the questions. So this was an opt in sort of activity and it’s one that they didn’t just have it run 24-7 because it was a little intrusive, but what it did is it sent students questions that they could answer at different intervals throughout the day, which also takes advantage of another principle from applied memory research which is spacing. So students are getting these unpredictable questions, they have the option to answer them, and they could be happening even when they’re not in class. So those are some other ways. Some of them look like tests, some of them don’t. But those are creative ways to get students engaged in that practice.

Rebecca: As a faculty member, we often advise students and mentor students who might be struggling in other classes, ones that we don’t have control over. Are there ways that we can help those students use this methodology to do well in those classes where it might not be embedded?

Michelle: Right. And that’s so great that you bring that in as well because ultimately that is what we need as teachers and that maybe circles back to yet one other piece of objection that I’ve seen… actually this time in a published article from a few years back that said, well are we doing too much for students? Are we scaffolding them too much so they’re going to grow really dependent on these kinds of aids like reading quizzes, and reading questions. But if we also have in our mind very intentionally, that what we want students to have at the end of the day is also something they can walk away with, I think that we do have to be very mindful of like, “Okay, let’s not create the impression that just because retrieval practice is so important that you have to sit there and wait for me to put together this specific kind of reading quiz for you.” So I think here, the really powerful message is once again that one students take this to heart… once they’ve not just been told this, that “Oh well, you should quiz yourself as a study strategy,” but they’ve seen that I believe in it to the point where I’m going to put time, energy, and work into it as part of my class. And maybe they’ve even seen the results… they’ve now had their own little before and after experience of what happens when I do this… that they can be more likely to take this forward. So having more faculty across the curriculum endorse this is a powerful idea that “Hey, this is how your mind works. This is how your brain works. Your brain doesn’t just soak up stuff that’s in front of it. You soak up stuff that you have to answer questions about.” That I think is going to be a powerful message. And there’s actually another article out there that I just absolutely love it was done by a psychologist named Pennebaker and some colleagues at University of Texas at Austin some time ago. They replaced high-stakes assessments in their Introduction to Psychology course with these mobile in-class quizzes that were done every day. And one of the things that they report in this article is not just that students did better in that class, but that certain subgroups of students actually showed improvement, specifically closing of achievement gaps, really, in classes that they took after the psychology course. And these were classes that we’re not even in psychology. So you think about that for a minute. How does that happen? Now, nobody knows exactly for sure, because this was kind of an unexpected finding in the study was my impression, but a real possibility is that when students have sat in this class and every single day they have seen the power of taking a quiz… of spacing out their learning… and attacking it in a very active learning approach… Once they’ve seen that happen, they’re more likely to go home and say, “You know what? I can do this in my biology course too. I don’t have to sit there with the teacher’s quiz. But, if I attack it in the same way I might get the same results.” So I think with those things, we can have students walking away with that enduring practice that we want them to have. And it is funny too, because it brings up one of the, I’ll just say it was a really heartwarming teacher moment that I had this last semester when I did bring in these Kahoot quizzes. In the run up to the first exam, I had done my fancy little in-class quiz and was kind of patting myself on the back of what a great leader I was. So I came in to, I think it was the class right before this exam, and I come plowing into the classroom and it’s dead quiet. And there is a student at the front of the classroom. He has commandeered the podium and the computer and what has he done? He’s accessed the Kahoots, which I gave them the links. You know, they’re out there for all the students to see. And he is running them and the rest of the students in the class are taking them just as seriously as when I was administering them. So they’re running through the questions again, giving it another shot. I didn’t tell them to do this at all. It was one of these moments where I just backed out and closed the door after me. And I came back in when they were finished. And so I think those are the kind of moments that we can set ourselves up for when we really do bring this into our classes and we get behind it and a really authentic way.

Rebecca: I think one thing Michelle, in the way that I asked the question, I had also asked it from the perspective of a student hadn’t had the experience of doing a retrieval practice in a class. So, if you’re working with a student, maybe outside of class or as an advisor, are there things that we could do to help those students adopt those practices, even if they haven’t seen it modeled for them?

Michelle: So how do you help students when they haven’t had some of these experiences on their own? And I think this is a part of a bigger package of goals that I think a lot of us should have in supporting our students to really put students in the driver’s seat. To say, “Yeah, you don’t have to wait for a certain style of teaching or certain subject material in order to succeed with it.” I see that isn’t really part of a larger package of growth mindset honestly. So what can students do to make themselves the masters of this? Now, I think that there are some old standard, very traditional, approaches that are worthwhile. And when you look at those approaches, sometimes retrieval practice is at the core. So it may not be a matter of trying to get a very unfamiliar set of terminology or anything for students before, but really getting them to look at some of these approaches in a new way. So things like you’ve ever heard the term SSQ4R or PQ4R, there’s a couple of those that have acronyms and the things they have in common are that they tell students when you sit down to study or read a text or prepare for a test, here’s what you do. You don’t just start reading from the top with no goal in mind other than “Oh, I want to get a good grade later.” What you do is first of all, you set yourself up with questions. Which is, after all, a lot like retrieval practice. Start with a question, say, “What do I want to answer? Can I answer that now?” And if not, why not? To read your text or go through your material very intentionally around this questions, and then there’s always this piece of recite or review. That’s what one of those Rs stand for in some of those traditional systems. And that means closing the book. That means closing the book and saying, “Okay, I’ve sat here with this material for this amount of time. What can I actually say about it at this point?” So, sometimes directing students back into some of those and saying, you need to adopt these strategies, which are completely teacher-independent, they’re fairly discipline-independent as well. That can be good. If you’re doing them for the right reasons and with the right approach in mind, that’s very good. Encouraging students to take advantage of the publishers’ companion sites… Now this is a little bit more of an uphill climb. That is where we run into “Well, if there’s no points for it, Why should I do it?” But encouraging students to say “Look, you already paid for this textbook. To be crass about it, you paid all this money, did you know there’s a website over here and if you interact with those materials in this particular way that’s really, really likely to pay off for you?” So, besides just ensuring that our students have what I consider to be these basic foundational pieces of knowledge about the mind and brain: that we remember through testing, we remember through retrieval and an active engagement. All students should have that, but those are some specific things that we can counsel students to use across all their studies that really should pay off.

John: Do you have any other suggestions for those faculty that are thinking about expanding their use of retrieval practice?

Michelle: You know, just to really encourage and support faculty who are starting this journey, as it sounds like that you all have, to really re-examine how we can bring in this incredibly powerful principle, and to really reassure each other that “Yeah, this is not about really just piling so much more work on yourself.” We can sometimes even just re-examine the assessments and assignments that are already in the course and so that’s kind of one last piece of practice that I think that a lot of us can really stand to bring in. And that was a big thing for me as well to say, “Well, if I’m going to administer these tests… we administer tests, we administer midterms anyway… what else can we do to increase their value as learning experiences… as learning events. So here’s another where I’ve brought in various forms of test discussion activities. Instead of standing in front of the class with that deadly the day-after-the-test class period where I say, “Let’s go over it” to realize, you know what? That probably will not work that well. But one of the amazing things that we have learned from the research on retrieval practice is that it’s not just in the moment in the taking of the test, where this advantage to memory happens. It actually creates a sort of a receptive window for learning and for review when we’ve just taken a test on something. So if you’ve got a midterm in your class already, well, hey, why not carve out the time after that test is up to say give it back to the students… like I photocopy the exams before I’ve graded them. They have no feedback on them whatsoever. And I hand them back to students as a group discussion exercise. I say “Alright, here’s a blank copy of the exam, your group can fill out this blank copy together, just knowing what you know, revisiting all those questions, having those good discussions about what you understood and what you didn’t. And I offer a little bit of extra credit for really good performance on that. And there’s other ways to work that out so it actually takes off as a small group exercise in class. But regardless of the specifics of how you make something like that work, the spirit of it is the same. That when you sat down to do this, to try to drag all this information out of memory, that is not an end unto itself. It should be part of a bigger picture of learning in the class. And it’s sort of an untapped vein of potential we have as teachers and that our students have as well and that we can access it regardless of what the discipline is or how the class is setup.

John: …and students see it’s very relevant. I started doing something very similar in my econometrics class last year following a suggestion from Doug McKee who had been doing that in his class. And it worked remarkably well… and turned what was normally a pretty unproductive class period where we’d go over the test… and the people who did well with just be really happy and pretty much ignore any discussion, and the people who did badly were just sitting there unhappy and not really being very receptive. But when they sit there, and they’re explaining it to each other, it seemed like a really ripe time for them to learn the material much more deeply.

Rebecca: I remember the first time that you implemented that. You sent me a text message with a photograph of his students taking a test and it looked very active. [LAUGHTER}

John: …and they were having fun.

Rebecca: Yeah.

John: One of the nice things that came out as I was wandering around listening to their conversations, and I was hearing people say, “Oh, yeah, now I see where that came from.” Or someone would say, “Well, when did we learn this?” And then someone else would say, “Well, remember when we worked on these problems?” …and it just helped them make connections… and the power of peer instruction is so remarkable. I’m going to do it in as many of my classes I can.

Michelle: That’s perfect. That’s what we want as teachers, and that’s what our students benefit from.

Rebecca: So as you know, we always wrap up our podcast by asking what’s next?

Michelle: Oh, wow. So I am looking forward to kind of rebooting a course that I have not taught in some time. My senior capstone course in technology, mind, and brain. And that is a fun one. But it’s one that is going to take a lot of revision, since it is technology and things change so rapidly, as we all know. So that is going to be a big part of next semester. I also have a crop of research projects in various angles on teaching, learning and educational technology that I’m really excited to be moving forward in the next calendar year. One of those… really foremost among them is a project on virtual reality for learning. We have an incredibly creative and dynamic team looking at virtual reality here at Northern Arizona University. They put together an amazing series of interactive exercises that are part of the organic chemistry course here and teach some of the challenging concepts in that extraordinarily challenging gateway course. And so we now have a whole set of data from students who went through and did this at varying points during the semester. We got their feedback, we’ve got all different kinds of psychometric measures that we gathered from them at the time as well. So, I cannot wait to be tackling that and looking at all kinds of angles on how this part of technology is impacting student learning in this course.

Rebecca: Sounds like some exciting adventures.

John: That sounds like a wonderful research project. …looking forward to seeing what you find.

Rebecca: It was a real pleasure to talk to you again, Michelle. Thanks for spending some time with us.

John: It’s wonderful talking to you. Thank you.

Michelle: Oh, likewise, always a pleasure to talk to your listeners.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.