65. Retrieval Practice

Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, Dr. Michelle Miller joins us to discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Her academic background is in cognitive psychology and her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications.

Show Notes

  • Miller, M. (2014). Minds Online: Teaching Effectively With Technology. Cambridge, MA: Harvard University Press.
  • Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319(5865), 966-968.
  • Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 1199327.
  • Kahoot
  • Bray, Niki (2018). 43 to 0: How One University Instructor Eliminated Failure Using Gamified Learning. Blog post
  • Retrievalpractice.org
  • Lang, J. M. (2016). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • Pennebaker, J. W., Gosling, S. D., & Ferrell, J. D. (2013). Daily online testing in large classes: Boosting college performance while reducing achievement gaps. PloS one, 8(11), e79774.
  • Pauk, W. (1984). The new SQ4R.
  • Thomas, E. L., & Robinson, H. A. (1972). Improving reading in every class. (a discussion of PQ4R)

Transcript

John: Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, we discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today we’re welcoming back Dr. Michelle Miller. Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications. Welcome back. Michelle.

Michelle: Thank you so much. It’s so great to be here today.

Rebecca: We’re so happy to have you again. Today’s teas are:

Michelle: Well, I’m drinking Coco Loco, which is a blend from a local tea shop here in Flagstaff, Arizona. Steep Leaf Tea. And Coco Loco is a lot like what it sounds like. It’s chocolate and banana. So tea snobs may scoff at my choice, but it’s wonderful.

John: IIt sounds good.

Rebecca: And I think I saw a nice silver teapot that was poured into a green and blue tea mug.

Michelle: Yup.

Rebecca: A nice tall one.

Michelle: That’s what I need.

John: And I’m drinking ginger, peach green tea.

Rebecca: I went with a Christmas tea today. So Michelle, we invited you here today to talk a little bit about retrieval practice. Can you first start with defining that for us and letting us know what it is?

Michelle: Right. So retrieval practice is essentially the act of pulling something out of memory. So that is, in memory research, what we would term retrieval. So something is stored in memory and we want to pull it out so we can actively use that information, have it in our conscious minds and so forth. And so we go through this usually very fast process called retrieval. So retrieval practice is specifically the act of doing it, and we contextualize that with learning. So, when I’m trying to learn something or I’m the process of learning something, to say, “Oh, what was that fact I remembered, or what can I say about this?” When we do that, it produces something that we can call the testing effect. So this is kind of the clearest example… not the only… but the clearest example of retrieval practice in action during learning is when we sit down to take a quiz, take a test or something like that. So all the excitement that’s happened around retrieval practice in higher education, and really in the rest of education today, is around this finding, which has been replicated many, many times: that tests are, as one person put it, not neutral events in learning. When we take a test on something, that has a very powerful effect on our ability to remember it in the future. So, really simplified down to its core, tests help us remember in the future; when we take a test it strengthens our memory. So that’s what retrieval practice is and it can, as we’ll maybe talk about today, take many, many forms in learning settings. And I did want to clarify too, this is something that I definitely don’t want to take any kind of credit for discovering this. This has been around and has been known about for a long, long time. Some of the big names who are associated with this: Jeffrey Karpicke, Robert Bjork. Roddy Roediger, there are quite a few really heavy hitting cognitive scientists and cognitive psychologists who have established this. But there are many, many of us who are out there trying to disseminate this to other teachers around the world so that we can all tap into the power of this. And I have done a little bit of work in this area with my colleague here at Northern Arizona University, Laurie Dixon, who’s another psychologist… and we teamed up some time ago to look at a very practical implementation of retrieval practice in an Introduction to Psychology course that we conducted some years ago and this is a course, you can imagine, where just trying to get students to perform even a little bit better is a big project. So we examined how even something kind of basic… it was very high tech at the time… but just basic web quizzes that came packaged with the textbook. We said, “Well, if we actually assigned students to do these as part of the course, and if they went through and treated these as opportunities to learn, not just assessments in and of themselves, would that have any systematic impact on course performance?” And we found that in fact, there was a significant improvement associated with that. So that’s kind of the landscape of what retrieval practice is, and why we’ve been so interested in discussing this in the psychology of teaching and learning.

John: In fact, I saw you present on that about 11 or 12 years ago in Orlando at one of the NCAT conferences and it convinced me to completely revise how I was giving my classes and it’s made a big difference and resulted in some significant improvements in student learning. Given that we know so much about retrieval practice. Why are faculty resistant to doing this?

Michelle: Wow, that’s a great question, and that is one that I have been really facing a lot these days in my own practice, talking to other faculty members, and disseminating this through some different activities I do in this area. It is easy for those of us who work in this area, cognitive psychologists in particular, but a lot of us who like you heard about this a long time ago. We’ve seen the power of it. We forget that to other faculty, this can be a very off-putting concept. And so it’s really great for us to think about why that is. And I always think that’s really good for me to kind of go back to that and say “Yeah, not everybody is sold on this idea. And there are good reasons for that.” So like with a lot of things that we talk about in teaching and learning, I think that these really break down into two neat categories: there’s the philosophical issues that people have with it, and then there’s more practical and logistical issues with it. So kind of tackling those one at a time. Philosophically, when people say, “Yeah, I understand about the research but this just goes against something that I believe as a teacher or how I want my classes to be. Here are some ways that that can play out. First off is this idea, that I’ve heard in one form or another quite a few times, and that is this notion of superficial learning. So “Okay, sure, there’s a study that showed that maybe people retain something a little bit better. But surely that’s not this deep learning, whatever that is.” That’s a concept that we all want. So do tests and exams… just testing… create just a superficial form of learning? And while, of course, I understand that, and I absolutely applaud faculty for really thinking deeply about that issue, and caring about it. I Here’s the thing… we got to define that. We social scientists, that’s what we do… we have to kind of break things down and say, “Okay, what does deep learning mean?” and I don’t know that anybody has kind of definitively done that. But when I look at that, I say, “Well, this is not just one research study that showed a little improvement in a lab test. There’s quite a few studies that do use realistic types of materials. It’s not all just contrived laboratory studies. Furthermore, there’s also studies that show that when students engage in more quizzing and testing on material that they actually are able to transfer that learning better. And that is a very, very big deal in teaching and learning as a lot of us know is not just getting students to be able to solve a problem in one context or work a concept in one context, but can they do it in the next circumstance? And that very difficult process is aided by quizzing… and to me, what could be deeper learning than learning that transfers? So that’s part of it. Some of it is perceptions around multiple choice quizzes and tests. There’s an assumption too that if we’re talking about quizzing we must be talking about multiple choice questions… and first off, sometimes in larger classes, those are the final assessments… in that Introduction to Psychology course that we studied years back, that’s what the assessments were… so, having students practice in that format I don’t think that we should necessarily dismiss that. And as we can talk about in a little bit, there’s lots of ways to induce retrieval practice that actually don’t involve multiple choice questions. So, there’s a bit of that as well. And something that I’ve talked about with some faculty recently, too, is this baggage around K through 12. And maybe that’s something that’s resonant with you all.

John: Yeah, that’s given the testing effect a somewhat bad name, because high-stakes testing is being used in a lot of what’s going on with K to 12. But that I don’t think is what retrieval practice as you’re suggesting is all about.

Michelle: Right, and I have to be cautious here. I really like how you laid that issue out in K through 12, that there is a reputation problem… and that has happened because of the high-stakes standardized testing policy in the United States. And I got to be careful because I don’t want to represent myself as an expert in K through 12 or in K through 12 education policy. But I don’t think you have to be an expert in that to know that there’s been a lot of same pretty well justified public pushback against over-testing in K through 12. And yeah, I think that we absolutely do have to be aware of that. Students come to us in higher education… that’s a system that many of them have been through and our faculty are very aware and very cognizant of that too. So, nobody’s a blank slate here, not our students, not our fellow faculty. We have assumptions and ideas and experiences about testing that happen. I think those can be addressed. But, yeah, that is definitely another very big barrier. We got to differentiate between high-stakes standardized testing for the reasons it’s done in K through 12, and low-stakes testing and quizzing for learning as proponents of retrieval practice would have it.

Rebecca: Some of the pushback I’ve heard from faculty fall into two categories related to this as well. One is that they assume that retrieval practice is best implemented in 100, 200 level introductory classes instead of upper level 300, 400, graduate-level classes. And then the other area is that paper and pencil tests don’t make sense and all disciplines. And so they assume that a test has to be in a paper/pencil format, which could be online testing, or it could be multiple choice or it could be essay questions. But I think that, from being someone in the arts, like there’s other ways to test beyond that, but we don’t think of those as tests.

Michelle: Right. That’s another great lens through which to look at this issue; that we do need to broaden the definition to draw more attention to this and to make it a more appealing concept. But yes, how can we make it broadly appeal across lots of disparate disciplines? Not only does it not have to be a multiple choice type of exam, maybe it’s not a pencil and paper exam at all. And we as faculty have to think about what makes sense there. You make a good point about the levels concept. I think, these days most of us have heard of, “Well, there’s one particular Bloom’s taxonomy…” which is a wonderful framework for getting us thinking about being systematic about what we’re asking students to do with the information that we’re bringing to a course and trying to do things like align the teaching we do with the assessments that we have. That’s wonderful. However, I think it does ingrain in us that idea that “Well, just knowing things is sort of at the bottom… You sort of get that out of the way, and then we go on to the good stuff.” And from a cognitive perspective, these relationships are much more fluid and much more interdependent, so that yes, absolutely, the higher thinking that is what we want. That is what we should want. Or if we’re in highly applied disciplines (if we’re in the arts, for example), we need students to be able to do things with that information. But they have to have that. So I think it challenges us to think of new ways with that concept as well.

John: One of the barriers I think some people have is they don’t like to grade tests and so forth. But one of the things you mentioned in your book is that the testing effects been known for a long time, but it was really difficult to implement in terms of low-stakes testing, particularly when you’re teaching at a larger scale. But, as you’ve done yourself, and as you suggest in your book, computer technology makes it easy to automate some of this… certainly more easily for multiple choice and free response and similar things. But it makes it a whole lot easier for both students to have multiple attempts at learning something using some type of mastery quizzing and it makes it a whole lot easier for faculty who don’t have to spend all their time grading.

Michelle: Right and that’s absolutely where the practical stuff comes in. So, we’ve worked through some the philosophical objections, that: “No, this does not turn your classroom into some terrible assembly-line concept of learning. It’s not going to create a bad relationship with your students. It’s not going to simply carry on a legacy of bad policy as people perceive it. It’s not going to do those things.” And then, we do get to: “Alright, but what is this going to do to my life as a faculty member?” …and this is important stuff. Those who know me know I’m a big fan of James Lang’s work in his books. One of his more recent books is called Small Teaching and there’s a lot of different takes on it in that book, but it really hits home with respect to “We do have to think about not everybody’s in a position to, nor should we even try sometimes, to just take everything down to the foundation and rebuild.” That we do really need to think about “Do I want to do this, if it’s going to create 800 more questions for me to grade?” Is this a sort of a situation where, “Well, I don’t want multiple choice, so I’m going to have to give these open-ended questions. I’m gonna have to give feedback and I have 200 students, what will happen? If I am going to go with multiple choice questions… well, how am I going to do this? to have to write all of this?” And yes, it’s one of the most powerful outcomes of the educational technology revolution that makes it workable, and scalable in a sense, even with large classes to do these types of things and to bring them in. So, that is definitely a message that I hope faculty think about if they’re on the borderline of wanting to bring in more retrieval practice into their classes.

Rebecca: I’m in a discipline where the multiple-choice questions are using things digitally doesn’t always work for testing and practicing some of these basic things, and there’s not a good way to automatically grade it. But one of the strategies that I’ve used is actually some self grading, which has actually worked pretty well. I just check and I have them write notes about anything that they got wrong. So, it demonstrates that they’ve tried to understand when we go over it, and I give credit based on how thorough those notes are, rather than whether or not they got the question right or wrong. And that’s made a difference in my classes. And had I not come up with that solution, I think I would have abandoned it because it would have been too much work. But it it actually is working pretty well.

John: When we had a reading group on Small Teaching last year, one of the things that was widely adopted by faculty was a very simple form of retrieval practice where they had students at the start of each class reflect back on what they had done in the previous class. And most of them have continued to do that in subsequent classes as well.

Rebecca: One of the other barriers that faculty might raise is the idea that students don’t take low-stakes things seriously, or that they don’t put the same kind of time into it that they might for something that’s high stakes. Can you talk a little bit about how we might help students find value in retrieval practice and subsequently also with the faculty then?

Michelle: Right, so that, ”Well, we can put it out there, but will they do it?” I’ve kind of crossed this philosophical and practical barrier for myself of giving some credit for pretty much anything that I am hoping that students will do. I put the work in to set it up and I do believe as a teacher that there’s reason to believe that will help their performance, that I need to work it into the syllabus somewhere. I don’t think it detracts from learning, necessarily, to say, “Yeah, there’s some points associated with this.” And especially with our students who, for many of us, are going in a million directions at once. They’re juggling jobs, multiple classes, sometimes their own families. So having an incentive in the form of points—having some kind of a payoff—I think, helps them make that decision that this is at least worth the time to do. I think the other thing that we probably can all do more of, and that I’ve done more over the years, is framing and honestly marketing this to students… communicating with them about why. And when students disengage from an activity like this, when they say, “Ah, why do I have to sit down and do this thing? This is just another test. Oh, no.” …really conveying the excitement and the goodwill that we have in setting those things up can go a very, very long way. Of course, a student, if they just look at it inside and they have no context for why this was put into place, they’re going to have them say, “Well, maybe I won’t do that.” But when we can market to students, we can say “There is a lot of research that shows that this is a very, very good use of your time. And hey, you’ve probably taken a lot of tests in your life that were really about measuring or sorting you and figuring out what you know and what you could do. This is a very different kind of test.” So that can go along way and get the C students nodding: “Alright, I get it. I get why she put this assessment right here.” I think a lot of us have hit on the practical strategy too, that the little Easter eggs or goodies that we plant in the form of questions that get re-used on the higher stakes assessment. So most of us will have tests for measurement at some point in our courses. And yes, students really do pick up on it when you use one, two or more of those items that were in, say, the gamified quiz that you ran in class or the reading quiz that they did beforehand. They can see those and say, “Oh, wow, I got feedback on that. I got an opportunity to practice…” and if it draws in a few more students who see it as almost a legitimate form of cheating, honestly…. like a fun and sanctioned form of getting an advance sneak peek at the exam, then great! Then that maybe is an opportunity for them to come in and see that. So there is that. Actually taking it seriously ourselves, not just in the form of saying, “Well, here’s the points I’m going to give you for this,” but spending class time on it. That’s a big bridge for a lot of us to cross, right? Because we as teachers tend to be very focused on “Oh my gosh, class is for covering material. We use all these sort of distance metaphors to talk about what we want to do with our class time. But if I say, “You know what, guys, I believe in this, and I believe in it enough to where we’re going to spend the entire class period before the final exam….” I did that twice last week myself, when running exams. Or we’re going to spend five or 10 minutes of the beginning of every class period doing this, as one project recently published about doing. If you show yourself doing that, and offer them that, I think that also goes a long way towards it. And I guess to just say, well, taking it seriously… here again, what does that look like? What does that mean to different people? And we can kind of a little tongue-in-cheek say, “Well, why do we have to take it so seriously?” Sometimes games and learning can happen when it is presented in a more fun context. So not everything has to be deadly serious or spending hours and hours and hours of stressful time on. There are occasions when a light-hearted approach can be perfectly good and can still get us involved in that really critical activity of retrieval

Rebecca: I can share an example of doing that in my classes. We’ve done design challenges and sometimes we challenge other classes that are happening at the same time. That reinforces some of the basic principles that we think that students should be doing and reminds them of it… and they might work in a team…and then we have a competition. And it’s fun and what have you. And students like those, it breaks up the day, it makes it more fun. And then I’ve also done things where I give class time to do little design challenges in class that might be individual and then they can level up to working with a partner to finish solving a problem or something… and students value that. They recognize that next time they’re trying to do a project on their own,that it’s easier because they’ve had that practice or that opportunity for the retrieval practice. And my students have actually ended up asking for more of those opportunities.

Michelle: Great.

John: Could you go back just a little bit and tell us what you did in those couple of classes last week before your final. We’re recording this, by the way, in early December during finals periods in both of our campuses, but we’ll be releasing it a few weeks later… to put that in context.

Michelle: Oh, okay. Well, I’d love to. …and I’ve been talking about retrieval practice as you pointed out for years, and I still discover new ways to infuse this into courses. And the context for this is my cognitive psychology undergraduate course. It’s a 200 level. So it’s a lower-division course. And it’s about 60 or 70 students. And as you can imagine, it is a bit of a tough sell. For many of the students it’s their first encounter with this side of psychology. It’s not as intuitive as some other areas of psychology. So there’s a lot to learn and a lot of motivation to be done. One of the ways we bring this and in this course is using a technology called Kahoot, that’s spelled K-A-H-O-O-T,, and it’s really very intuitive and functions very smoothly… relatively free of bugs. That’s good stuff. It’s a program for doing gamified quizzes of various kinds. What I did, and at different points in the semester, and then really amped up in the last week of the semester is running these gamified quizzes. And this is something, by the way, that I hit on and got the idea to try based on a colleague named Niki Bray, who’s from Tennessee, and has actually done some really systematic work in reformulating some of her courses around in class quizzing in just really ingenious way. So I saw some of her presentation and I said I’ve got to try this for myself. I went with multiple choice questions. Kahoot does have some parameters… questions do have to be short…very, very short. And to some people that may be off-putting, but you can put together quite a few of these. And so we would put this up on the projector and students have the option of dialing with either their laptop or their smartphone and weighing in on each question. The neat thing about it is it has an algorithm for giving points based on your speed as well as your accuracy and it’s got a little leaderboard so you can actually have a little in-class competition. Now some people who use this do require all students to do it and they actually issue points for performance. Now, I presented it very much as a practice activity… and made it very, very clear because of my philosophy, I’m not going to assume that all students have devices or have smartphones or laptops or that they want to do that. But I said, “Look, remember we’re talking about retrieval practice, guys. So the real meat and potatoes of this is not buzzing it on your phone. That’s fun. But the real benefit of this activity is what you’re doing sitting there in your seat. And you could be doing this with a piece of paper if you want.” And that is what a few students opt to do. They try to answer the questions, they know what they need to go back and review and so on. And it’s nice because it spits out at the end, a whole report that tells me right away… Okay, which questions do we need to revisit? Which ones did students have the hardest time with and so on. That’s one of the things that I just brought in. And yeah, it was a big deal. I sacrifice a chapter of “coverage” so that we would have more time at the end of the semester. But to me, I would rather have students going into the final knowing that they’ve had this retrieval practice and they have a better chance of performing really well. Earning a good grade on this material I care about than honestly cramming in a little bit more mileage in terms of the quantity.

Rebecca: Sounds like fun.

Michelle: You know what, it does really bring a fun factor. There’s been a lot of different variations on in-class polling, and I will admit to this, I actually purchased and am the proud owner of a physical buzz-in quiz device, complete with a whole spaghetti nest of wires and an incredibly abrasive, buzzer sound, and everything. So previous to this, my educational technology did include… I think I could have up to eight intrepid volunteers who would play a quiz game and then I would have to appoint a points keeper and all this… and props are always a lot of fun. So when I say I believe in bringing in retrieval practice, I really do walk that walk. But I will say doing it by a smartphone does allow for more participation, and I don’t have to worry as much about the minutiae of scorekeeping and stuff like that.

John: I played with Kahoot a little bit at my classes at Duke and students have loved it.

Rebecca: We’ve talked a little bit about framing things so that students take the practice seriously. What do we do about the students who just push back, it’s like “This is too much work. This is a lot of extra time…” or that sort of argument.

Michelle: I think that that’s another piece of that barrier to more faculty adopting this, not just the work involved, but realistically, student opinions. Student evaluations matter a lot to faculty life. And of course, we all want to have that wonderfully rewarding semester, not the semester where we feel like we’re at odds with our students. So I do think a little piece of this is we do anticipate sometimes worse and more pushback than what actually happens in the end. So I think we have a fair amount of sort of a dread factor when we go into something new like this. But that said, when students have an issue with more quizzing or more testing, here’s how those come out. I think, first of all, we do have to sometimes separate out the technology aspects of it from the quizzing or testing, per se. So, as John mentioned, this is one of the amazing things that educational technology does. But the flip side of that is, if you’ve ever used technology for education, in any shape or form, you know that it breaks down. And when it breaks down at a big class, that’s a headache for everybody, it’s a misery. So if you’re trying it, and things are not working out, you just got to figure out “Okay, how much of this is the assignment, the activity per se, and how much of it is that wonky quizzing thing that I got from the publisher and it fell apart, or students hanging up when they’re trying to dial in with the poll and fine tuning those. I mean, when I first used kahoot, I decided it would be a lovely idea to put four chapters worth of material into a 40-question quiz. And when you get used to as you know, that they took a long time, I mean, 12 questions can keep you going for a very long time, especially if you’re discussing… and some students got kicked out part way through and then they weren’t on the leaderboard and I could have set the whole thing aside. But really, that was more I needed to get the technology working. So that’s a big piece of it. I think that sometimes it is a perception issue with the timing. Now, those of us who are just all over this as a teaching technique, we like to do reading quizzes before we talk about that material in class. So chapter three is up on the syllabus and your chapter three reading quizzes due on Sunday night before we do that…, and that can provoke a fair amount of confusion and honestly griping with students. They say, “Why did I get tested on this when we didn’t do it… we didn’t cover it in class yet?” And what do you know, that’s another framing and communications issue. Once we know that that is why we’re doing that you get so much less on that side of things. But here too we do also ourselves have to follow along with what we say. If we say you’re doing this reading quiz so that we’re establishing a foundation. we don’t have to teach everything in the class itself. We could spend the time applying. Well, guess what, then you do have to do that. So if I assign you to do the reading quiz on chapter three, and then on that Monday morning I go, “Okay, we’re going to go over chapter three starting at the top. And I’m going to show you all the PowerPoints for all these things that you already read.” Well, yes, then student morale and student support will fall apart at that point.

Rebecca: Those are very good reminders.

John: Yeah, I’ve pretty much adopted this approach all through my class beginning when I first saw you present it. So I have students do the reading in advance, I have them take reading quizzes with repeated attempts allowed on those. And then in class, I have them working on clicker questions, and I and the TAs go around and help them when they get stuck on problems. But there is an adjustment and students, especially when I first started doing it, would generally say, “But you’re not teaching me” …and you do have to sell students on this a bit. One source of resistance is that when students take quizzes, they often get negative feedback. When they read something and they read it over again, it looks more and more familiar. They’ve got that whole fluency illusion thing going and they become comfortable with it, and they feel that they’re learning it… or similarly, if they hear someone give a really clear presentation on a topic, it feels comfortable. They feel like they’re learning it until they get some type of summative assessment where they get negative feedback, and then they feel the test was somehow tricky. But it’s a bit harder to convince them that actually working through retrieval practice, watching videos at their own time and pace, reading material as needed, and then spending class time working through those problems is as effective. I’m getting better at it. But it’s been a long time trying to convince students and I did take a bit of a hit in my evaluations, especially the first few times.

Rebecca: Do you mean learning’s not easy?

John: Students would like learning to be easy.

Michelle: You know, it’s funny, I think almost in a way… see what you think about this idea. But it’s almost a mirror image of our illusions as teachers, right? That I gave a wonderful clear lecture and I assigned wonderful readings and I saw students highlighting them. Therefore, students must have assimilated this knowledge and it must be in there. So I think it challenges our students but it also challenges us as well. It can be quite an eye-opening experience to running something like a Kahoot. And that brilliant point that I gave this great example for…. what do you know…. 7% of the students actually nailed it. And so we can all use a reality check… teachers and the students. And you mentioned re-reading, that’s another one where I’ve had some pretty intensive conversations with other faculty, I’ll kind of say, “Oh, well, and there’s this great research that shows that students tend to re-read when they’re studying and we know that from a memory standpoint, that is really, really ineffective.” And faculty will say, “Whoa, whoa, whoa.I want my students to be re-reading.” Of course, now when they say re-reading, they may be picturing deeply interrogating a text… annotating it… looking at it from a different perspective. And absolutely, that’s a wonderful part of scholarship. But that isn’t what we’re talking about. We’re talking about students re-reading as a study form and mistaking highlighting for deep interrogation. So, just like with the rest of knowledge bases versus higher-order skills. This is another words “both and” it cannot be “either or.” Yes, we want students re-reading in the right ways, but students or teachers cannot mistake that for learning sometimes.

Rebecca: So, let’s say you’ve just convinced all of our listeners that we need to be doing this, how do we bake it into our course designs?

Michelle: Well, I think that, really getting creative with this, and as I talked to faculty when I visit other schools or talk to faculty of my own institution, I just see all of these new ideas all the time about how to do this. Once you do get that critical epiphany of alright, a test doesn’t have to look like a test on the surface… it does not have to be the ritual of “Okay, you’ve got a number 2 pencil that’s breaking and I’m standing over you while you have a panic attack for an hour.” That’s not what it is—that it’s about the retrieval. Once you get that, all kinds of creativity opens up. So I’ve also started sometimes on the very first day with the syllabus quiz. So, especially if you have a smaller class, we all struggle with that “I spent all week writing the syllabus and it’s incredibly important, and then I’m getting questions about stuff that’s on it all the way through the semester.” So, really on the first day, what I do is I take a very light-hearted approach and I divide the students up into teams, just physical teams. Everybody’s got a copy of the syllabus, sometimes I fake them out and I say, “Okay, we’re going to go over the syllabus point by point…” and I say, “No, you can read the syllabus. We’re actually going to do this other thing which research shows will actually help you remember it.” So the task here is that each team formulates a set of questions… you can make just a few. like three… so a little bit of teamwork. So formulate three good questions off the syllabus. And you’d be amazed, they come up with questions that even I can’t answer sometimes without having to cheat. So they talk about it. And then of course, you go around the room in some arrangement and each team gets to ask the other team their questions, and if you really want you could keep score and everybody loves bragging rights for being the team that stumped the other teams and won the most points. So, it can literally start right then. With that idea that I don’t read stuff to you in this class, this is about you. And I’m also not piling a huge amount of work on myself either apart from being the moderator and the MC and having written the thing in the first place. I don’t have to write questions. They’re writing the questions and they’re answering them too. So those are some of the ways that we can do that. I think especially in our larger classes we do want to think about things like peer grading or peer review of open-ended question responses. We do want to take advantage of things like publishers’ test banks to set those up as reading quizzes. And you’d mentioned earlier about “Well, what about this not being suitable for upper-division classes?” I had an upper-division class that just wrap this semester where we had reading quizzes as well. It may have been an upper-division course but it also serve that purpose of “Hey, you take a basic quiz over the chapter on Sunday and that really sets the stage for us to have a more substantive discussion.” All of those things. are ways that we can do this. I think open-ended reflection as well… so tests that look nothing like a test but are still retrieval practice. You’d mentioned about reflecting on what you learned last class period. So this sometimes goes by the name “brain dump.” And I did a bit of this last semester as well. So, this by the way, I do want to credit the great website retrievalpractice.org. So retrieval practice is actually that high profile it has its own website. It’s an absolute treasure trove of ideas. So, with the brain dump the way that I did it, or similar to what it sounds like you did, every now and again we start off class with you writing down everything you remember from last time on a piece of paper. I had students then turn to their neighbor and compare notes and see what they came up with. And I didn’t grade those. This was not a heap of grading for me. But in the end what they did turn into me for accountability and a few points was a very short reflection, just on an index card. So, “What surprised you?” and I review those and they’re very eye-opening but again I’m not there to police or micromanage what they put on their cards. So, that’s retrieval practice too. There’s lots of different flavors that we can bring in.

John: I do something similar with asking students to reflect on the reading in each module we work through. But because I teach a fairly large class, I didn’t want to have to deal with all the index cards. So I just have them fill up a simple Google form. And then I can skim through it and assign grades much more easily than shuffling paper. But it’s the same basic idea and it’s worked quite well.

Michelle: Absolutely.

Rebecca: Do you have any other examples of the kinds of little tests that faculty can run that don’t look like tests?

Michelle: Right? So tests that don’t look like tests. Well, here’s another that was probably a little bit more practical for a small class, and this is how I ran this. It, like the brain-dump exercise the way I ran it, was also very cyclical and very student generated. So we started out each class at the beginning of each week with students generating a set of quiz questions based out of the assigned reading. So we didn’t actually, in this particular example, have pre-quizzes or something like that. But students came in knowing that they can bring their book if they wanted. But they would need to sit down and write for me three questions, whether short-answer or multiple choice. Then I can flip through those and I would really quickly put those together for a quiz that went out the subsequent day. So, we’re alternating between generating questions and then returning to those questions. And then I would have pass out. I told them treat this like a realistic test. Actually try to retrieve everything, but, you know, when times up, you’re going to get to go back to it. And then they would grade it themselves. So I didn’t actually do the grading either. And that is really great for spurring discussion. And “Oh, my gosh, I thought I knew the difference between reliability and validity, but now that I tried to answer it, I realized that I didn’t.” And then you can throw it back out to the student who now has bragging rights for having had their question selected and say, “Hey, what did you mean? What was the right answer? And why did you put that down there?” And at the end of the day, they kept that quiz too. So it was really very much in their own hands to do. So there’s that. Other creative ideas that I’ve run across over the last couple of semesters… There’s a great project out there, run by Bruce Kirchhoff at University of North Carolina at Greensboro. So I got the wonderful pleasure of talking to that group last year. And he and some colleagues working in the area of botany actually put together a freestanding, custom built mobile app that students could take with them that presented different kinds of quizzes over the sorts of things that botany students really need to know like the back of their hands like how to identify different plants and how to discriminate among different examples and they found some empirical evidence that this actually raised performance up quite a bit. I’ve heard too of another Professor put together some surveys in Qualtrics, the surveying software took advantage of its ability to actually text message people and send them the questions. So this was an opt in sort of activity and it’s one that they didn’t just have it run 24-7 because it was a little intrusive, but what it did is it sent students questions that they could answer at different intervals throughout the day, which also takes advantage of another principle from applied memory research which is spacing. So students are getting these unpredictable questions, they have the option to answer them, and they could be happening even when they’re not in class. So those are some other ways. Some of them look like tests, some of them don’t. But those are creative ways to get students engaged in that practice.

Rebecca: As a faculty member, we often advise students and mentor students who might be struggling in other classes, ones that we don’t have control over. Are there ways that we can help those students use this methodology to do well in those classes where it might not be embedded?

Michelle: Right. And that’s so great that you bring that in as well because ultimately that is what we need as teachers and that maybe circles back to yet one other piece of objection that I’ve seen… actually this time in a published article from a few years back that said, well are we doing too much for students? Are we scaffolding them too much so they’re going to grow really dependent on these kinds of aids like reading quizzes, and reading questions. But if we also have in our mind very intentionally, that what we want students to have at the end of the day is also something they can walk away with, I think that we do have to be very mindful of like, “Okay, let’s not create the impression that just because retrieval practice is so important that you have to sit there and wait for me to put together this specific kind of reading quiz for you.” So I think here, the really powerful message is once again that one students take this to heart… once they’ve not just been told this, that “Oh well, you should quiz yourself as a study strategy,” but they’ve seen that I believe in it to the point where I’m going to put time, energy, and work into it as part of my class. And maybe they’ve even seen the results… they’ve now had their own little before and after experience of what happens when I do this… that they can be more likely to take this forward. So having more faculty across the curriculum endorse this is a powerful idea that “Hey, this is how your mind works. This is how your brain works. Your brain doesn’t just soak up stuff that’s in front of it. You soak up stuff that you have to answer questions about.” That I think is going to be a powerful message. And there’s actually another article out there that I just absolutely love it was done by a psychologist named Pennebaker and some colleagues at University of Texas at Austin some time ago. They replaced high-stakes assessments in their Introduction to Psychology course with these mobile in-class quizzes that were done every day. And one of the things that they report in this article is not just that students did better in that class, but that certain subgroups of students actually showed improvement, specifically closing of achievement gaps, really, in classes that they took after the psychology course. And these were classes that we’re not even in psychology. So you think about that for a minute. How does that happen? Now, nobody knows exactly for sure, because this was kind of an unexpected finding in the study was my impression, but a real possibility is that when students have sat in this class and every single day they have seen the power of taking a quiz… of spacing out their learning… and attacking it in a very active learning approach… Once they’ve seen that happen, they’re more likely to go home and say, “You know what? I can do this in my biology course too. I don’t have to sit there with the teacher’s quiz. But, if I attack it in the same way I might get the same results.” So I think with those things, we can have students walking away with that enduring practice that we want them to have. And it is funny too, because it brings up one of the, I’ll just say it was a really heartwarming teacher moment that I had this last semester when I did bring in these Kahoot quizzes. In the run up to the first exam, I had done my fancy little in-class quiz and was kind of patting myself on the back of what a great leader I was. So I came in to, I think it was the class right before this exam, and I come plowing into the classroom and it’s dead quiet. And there is a student at the front of the classroom. He has commandeered the podium and the computer and what has he done? He’s accessed the Kahoots, which I gave them the links. You know, they’re out there for all the students to see. And he is running them and the rest of the students in the class are taking them just as seriously as when I was administering them. So they’re running through the questions again, giving it another shot. I didn’t tell them to do this at all. It was one of these moments where I just backed out and closed the door after me. And I came back in when they were finished. And so I think those are the kind of moments that we can set ourselves up for when we really do bring this into our classes and we get behind it and a really authentic way.

Rebecca: I think one thing Michelle, in the way that I asked the question, I had also asked it from the perspective of a student hadn’t had the experience of doing a retrieval practice in a class. So, if you’re working with a student, maybe outside of class or as an advisor, are there things that we could do to help those students adopt those practices, even if they haven’t seen it modeled for them?

Michelle: So how do you help students when they haven’t had some of these experiences on their own? And I think this is a part of a bigger package of goals that I think a lot of us should have in supporting our students to really put students in the driver’s seat. To say, “Yeah, you don’t have to wait for a certain style of teaching or certain subject material in order to succeed with it.” I see that isn’t really part of a larger package of growth mindset honestly. So what can students do to make themselves the masters of this? Now, I think that there are some old standard, very traditional, approaches that are worthwhile. And when you look at those approaches, sometimes retrieval practice is at the core. So it may not be a matter of trying to get a very unfamiliar set of terminology or anything for students before, but really getting them to look at some of these approaches in a new way. So things like you’ve ever heard the term SSQ4R or PQ4R, there’s a couple of those that have acronyms and the things they have in common are that they tell students when you sit down to study or read a text or prepare for a test, here’s what you do. You don’t just start reading from the top with no goal in mind other than “Oh, I want to get a good grade later.” What you do is first of all, you set yourself up with questions. Which is, after all, a lot like retrieval practice. Start with a question, say, “What do I want to answer? Can I answer that now?” And if not, why not? To read your text or go through your material very intentionally around this questions, and then there’s always this piece of recite or review. That’s what one of those Rs stand for in some of those traditional systems. And that means closing the book. That means closing the book and saying, “Okay, I’ve sat here with this material for this amount of time. What can I actually say about it at this point?” So, sometimes directing students back into some of those and saying, you need to adopt these strategies, which are completely teacher-independent, they’re fairly discipline-independent as well. That can be good. If you’re doing them for the right reasons and with the right approach in mind, that’s very good. Encouraging students to take advantage of the publishers’ companion sites… Now this is a little bit more of an uphill climb. That is where we run into “Well, if there’s no points for it, Why should I do it?” But encouraging students to say “Look, you already paid for this textbook. To be crass about it, you paid all this money, did you know there’s a website over here and if you interact with those materials in this particular way that’s really, really likely to pay off for you?” So, besides just ensuring that our students have what I consider to be these basic foundational pieces of knowledge about the mind and brain: that we remember through testing, we remember through retrieval and an active engagement. All students should have that, but those are some specific things that we can counsel students to use across all their studies that really should pay off.

John: Do you have any other suggestions for those faculty that are thinking about expanding their use of retrieval practice?

Michelle: You know, just to really encourage and support faculty who are starting this journey, as it sounds like that you all have, to really re-examine how we can bring in this incredibly powerful principle, and to really reassure each other that “Yeah, this is not about really just piling so much more work on yourself.” We can sometimes even just re-examine the assessments and assignments that are already in the course and so that’s kind of one last piece of practice that I think that a lot of us can really stand to bring in. And that was a big thing for me as well to say, “Well, if I’m going to administer these tests… we administer tests, we administer midterms anyway… what else can we do to increase their value as learning experiences… as learning events. So here’s another where I’ve brought in various forms of test discussion activities. Instead of standing in front of the class with that deadly the day-after-the-test class period where I say, “Let’s go over it” to realize, you know what? That probably will not work that well. But one of the amazing things that we have learned from the research on retrieval practice is that it’s not just in the moment in the taking of the test, where this advantage to memory happens. It actually creates a sort of a receptive window for learning and for review when we’ve just taken a test on something. So if you’ve got a midterm in your class already, well, hey, why not carve out the time after that test is up to say give it back to the students… like I photocopy the exams before I’ve graded them. They have no feedback on them whatsoever. And I hand them back to students as a group discussion exercise. I say “Alright, here’s a blank copy of the exam, your group can fill out this blank copy together, just knowing what you know, revisiting all those questions, having those good discussions about what you understood and what you didn’t. And I offer a little bit of extra credit for really good performance on that. And there’s other ways to work that out so it actually takes off as a small group exercise in class. But regardless of the specifics of how you make something like that work, the spirit of it is the same. That when you sat down to do this, to try to drag all this information out of memory, that is not an end unto itself. It should be part of a bigger picture of learning in the class. And it’s sort of an untapped vein of potential we have as teachers and that our students have as well and that we can access it regardless of what the discipline is or how the class is setup.

John: …and students see it’s very relevant. I started doing something very similar in my econometrics class last year following a suggestion from Doug McKee who had been doing that in his class. And it worked remarkably well… and turned what was normally a pretty unproductive class period where we’d go over the test… and the people who did well with just be really happy and pretty much ignore any discussion, and the people who did badly were just sitting there unhappy and not really being very receptive. But when they sit there, and they’re explaining it to each other, it seemed like a really ripe time for them to learn the material much more deeply.

Rebecca: I remember the first time that you implemented that. You sent me a text message with a photograph of his students taking a test and it looked very active. [LAUGHTER}

John: …and they were having fun.

Rebecca: Yeah.

John: One of the nice things that came out as I was wandering around listening to their conversations, and I was hearing people say, “Oh, yeah, now I see where that came from.” Or someone would say, “Well, when did we learn this?” And then someone else would say, “Well, remember when we worked on these problems?” …and it just helped them make connections… and the power of peer instruction is so remarkable. I’m going to do it in as many of my classes I can.

Michelle: That’s perfect. That’s what we want as teachers, and that’s what our students benefit from.

Rebecca: So as you know, we always wrap up our podcast by asking what’s next?

Michelle: Oh, wow. So I am looking forward to kind of rebooting a course that I have not taught in some time. My senior capstone course in technology, mind, and brain. And that is a fun one. But it’s one that is going to take a lot of revision, since it is technology and things change so rapidly, as we all know. So that is going to be a big part of next semester. I also have a crop of research projects in various angles on teaching, learning and educational technology that I’m really excited to be moving forward in the next calendar year. One of those… really foremost among them is a project on virtual reality for learning. We have an incredibly creative and dynamic team looking at virtual reality here at Northern Arizona University. They put together an amazing series of interactive exercises that are part of the organic chemistry course here and teach some of the challenging concepts in that extraordinarily challenging gateway course. And so we now have a whole set of data from students who went through and did this at varying points during the semester. We got their feedback, we’ve got all different kinds of psychometric measures that we gathered from them at the time as well. So, I cannot wait to be tackling that and looking at all kinds of angles on how this part of technology is impacting student learning in this course.

Rebecca: Sounds like some exciting adventures.

John: That sounds like a wonderful research project. …looking forward to seeing what you find.

Rebecca: It was a real pleasure to talk to you again, Michelle. Thanks for spending some time with us.

John: It’s wonderful talking to you. Thank you.

Michelle: Oh, likewise, always a pleasure to talk to your listeners.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

47. First-year classes

The first semester of the first year is pivotal in helping students see themselves as scholars. In this episode, Dr. Scott Furlong, a political scientist and Provost at SUNY Oswego, joins us in this episode to discuss how first-year classes may be used to captivate student attention and ignite a passion for learning.

Show Notes

Transcript

Rebecca: The first semester of the first year is pivotal in helping students see themselves as scholars. In this episode we explore one strategy for captivating student attention and igniting a passion for learning.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

John: Our guest today is Dr. Scott Furlong, a political scientist and our Provost at SUNY Oswego. Welcome Scott!

Scott: Thanks John, I’m glad to be here.

Rebecca: Today’s teas are…

Scott: My tea is coffee because I stupidly forgot that they serve tea here. [LAUGHTER]

Rebecca: We’ll accept coffee drinkers too.

John: And I’m drinking a blend of peppermint, spearmint, and tarragon tea.

Rebecca: I reverted back to my old good time…

John: English afternoon?

Rebecca: Yeah [LAUGHTER]

John: SUNY-Oswego is introducing a series of new first-year courses this fall, and before we talk about what we’re doing at Oswego, could you tell us a little bit about what your own experiences with first-year courses at Green Bay?

Scott: Sure, back probably almost 12 years ago at Wisconsin, Green Bay, I was director of our first year programs on our campus. We had recognized that we have some pretty good first-year programs, but we were missing what I would have considered the most critical part, which was the academic aspect of it. I had been to a number of first-year conferences, had done a lot of work, reading, in first year and we were behind in that area. We did not really have any type of academic course for our first-year students. So, a number of our faculty (myself and about five others) decided we were going to do this on our campus. And, literally a colleague and myself… we’re sitting on an airplane coming back from a first-year conference and literally on an airplane napkin sketched out what we wanted to do in development of a first-year seminar for our students. And when we started at Green Bay, we needed to deal with some of the traditional questions around resources: How are we going to afford to do this? So, we made a conscious decision that we were going to take some of our existing general education courses that our basically introductory to the major and bring those larger sections of classes down to a smaller seminar size class, but we wanted to make sure that we were also going to infuse into these courses some amount of co-curricular activities and programs a student would have to go to that were diversity based, leadership based, health and wellness based, and academic lectures. And then we also incorporated interdisciplinary exercise where we would bring the students from the six classes together in a big room and and break them all up and have them solve what we thought was going to be a very interesting interdisciplinary problem using their disciplinary perspectives that they were learning throughout their normal semester. So, that was the birth of our first-year seminar courses. Those courses grew in terms of the number, we offered six the first year, twelve the next, fifteen the third year, and eventually got up to 20. And we got to a point where we were assessing the heck out of these things and it was clear that they were making a difference in terms of student engagement. We got up to the point where we were adding it to our general education program. The courses at that point were a lot different than how we originally initiated them, they were not Intro American government, they were not Intro to Psychology, they were what we started calling passion courses at UW-Green Bay and we stole that term that came out of Millersville University down in Pennsylvania. And they were courses that were interdisciplinary in nature and in topic, but they basically were around topics and areas that faculty cared a lot about and some of them were very much within their research or teaching interest; others were really far afield, where they would bring their discipline and other interdisciplinary perspectives into that course and in those courses we found were much more amenable to a first-year seminar than trying to ensure that we got all 26 chapters of an intro psych book in addition to everything else we wanted to do. When you can actually build the course around some of these activities, we found it to be a much more successful process.

Rebecca: Did you maintain those classes as part of the general education requirement or did it shift to being something else?

Scott: Two years into the first-year pilot, I had become Dean by then. The provost at the time had asked me to lead a general education reform effort. We knew pretty early that we wanted… because we had already collected a lot of very positive data that adding a first-year seminar would be something that would be a strong aspect of our general education. We really followed some of the AAC&U perspectives around general education: that your gen ed should be mission-based, should be based on what you’re most proud about at your institution. And again at Green Bay we were really strong around interdisciplinarity and almost all of these new freshman seminars were interdisciplinary based. So it ended up being sort of its own three credit requirement not meeting any type of disciplinary or domain type requirement, but just the idea that you had to take a first-year seminar.

John: Did that interdisciplinary requirement stay as part of the program?

Scott: It was when I left. [LAUGHTER]

John: Ok. …and you said there was a there were multiple classes that work together on a general problem.

Scott: Yeah, that didn’t last as long. There’s a great story there that I’ll tell. The faculty got very excited. One of the things that I most enjoyed about the process at Green Bay was the informal faculty development that sprouted up around the first-year seminar development. So, we would meet about every other week in our coffee house and pitch ideas and develop ideas and sort of frame what we thought the common learning outcomes ought to be. And one of the things we did is we came up with this common learning assignment and the idea we had (and at the time we thought it was a great idea) was that a new planet was discovered. And we had to send people to this new planet and teach them about the planet Earth. And how would you do that? And how would you set up a institution of higher education in a way that would teach these this new alien race about planet Earth? And we got cute and the name of the planet was trahe (that’s earth with a little bit of turning around of the letters) and we thought it was the coolest thing since sliced bread. The students hated it. [LAUGHTER] They couldn’t get it. They weren’t sure what they were doing. Although I will tell you the presentations they gave were dynamite given that they were first-year students that didn’t really know what they were getting into. They really give some really dynamite presentations, but we found out a little later in the semester that they had actually created a Facebook site called “I hate trahe.com”.

John: So, it was a unifying experience for them.

Scott: It was a unifying experience… and so we tried that one more year, realized it wasn’t working, shifted the interdisciplinary assignment a little bit, where it was a little bit more problem focused and probably more lecture oriented. We looked at issues and had different faculty from different disciplines try to talk about a problem or an issue from their perspective and then eventually we moved away from that sort of common group assignment. It became a little bit unwieldy as we got to 12 classes, 15 classes, to try to get those that many classes together or even as subsets of classes.

Rebecca: You mentioned that you did a lot of assessment related to the first-year passion courses, can you talk a little bit about what your findings were? You mentioned student engagement, but can you dive a little bit more into that?

Scott: One of the things that I’ll say right off the front is we went into this project knowing from our NSSE scores that our student engagement was pretty bad compared to the rest of the UW comprehensive campuses. So we knew we had a problem that we needed to address. We entered into this first-year seminar not so much around issues of “We need to address retention…” which is often a reason that’s put forward for bringing forth the first-year seminar, but rather we wanted to improve engagement. With the idea, and again research bears this out, if you increase your engagement you’re going to have a positive impact on retention. So, I had become friends and known some of the folks that work at NSSE… and specifically, Jillian Kinzie, who’s one of the lead researchers in the NSSE movement in Indiana. And I wrote to her and I said “Listen, I know we’re not on cycle for NSSE” (we were on, I think, a three-year cycle much like Oswego I think is now) and I said “…but we’re starting this new pilot program, we’d like to pull some of the NSSE questions and not only ask our pilot, but also ask some of our students who are not in the pilot. And what we found was engagement scores that were significantly greater across the board for the first-year seminars. And I had a colleague that used to talk about this when we would go to conferences of red bars reaching to the sky because we had a nice little bar chart that we would show on our PowerPoint which very dramatically showed the increase in engagement across a number of the NSSE criteria that they were looking at. We also found, and it didn’t hold, but in the first year we saw an 8 percent increase in retention as well for those students. Now, I know there was some selection bias there in terms of the students who were going into those courses, but we never saw anything less than 3% increase in all the years of the pilot. …and so we knew we had found something that was going to work, at least at Green Bay.

John: …and you taught one of these first-year courses. Could you tell us a little bit about that?

Scott: Sure, well I taught my first one, it was an intro to American government. And that was the the first year or two that we were doing this… and that was fun and it was great, and it’s always nice to teach a class with 19 students rather than 120 (which is what I was teaching). So you got to delve into some issues in a lot more detail, a lot more discussion based. But, when I became Dean, one of the things that I wanted to do (at least occasionally) is try to stay in the classroom a little bit. And it’s sometimes hard as an administrator to carve out the time because you never know when your boss is going to ask for you. So, I worked with a colleague and we team-taught a course around issues of Disney and we got cute and I came up with the name of “Inter-Disney-planarity” as the title of the course to sort of highlight the interdisciplinarity aspects of the class. She was an experimental psychologist and we used our various perspectives to really examine issues of Disney both in terms of the parks, the films, the culture. For example, I did a couple of different sections around how, at least Disney World, the one in Orlando, really is set up as its own government. Almost like a Vatican City in Florida, because they have their own police force. They have everything. Their own regulatory bodies, things like that. My colleague did a lot of work around architectural and planning background and planning theory… looking at people like Frank Lloyd Wright and others (names I’m not remembering) in terms of how they did some of the urban planning and suburban planning in the United States. And how Disney really pulled a lot of those issues in the building of the parks, and why they were doing it, and why it works the way they did. And then together the two of us taught class a part of the class on racial and gender issues around Disney, particularly some of their early films, still to this day… but it was really biased in the early years. So, it was a lot of fun. It was always a great way to engage academically in a fun topic. I will tell you, the students who signed up, they all thought they were watch Disney movies. We showed clips, but we rarely would show full films and so I think they were disappointed in that. But I think they had a lot of fun in the class.

John: And they were learning things.

Scott: They were learning about the disciplines, we did have some common learning outcomes: we had a writing requirement, we had an oral communication requirement and we had a critical thinking requirement. So, all of these sort of skill based activities that we all value as part of a strong liberal arts education is what we were introducing to them, and it was a way for them to engage in college-level work around topic areas that students find interesting. So, you mentioned before we started about zombies, we had courses on zombies, and what would happen in a zombie apocalypse and we had students who would put together basically action plans and where would you go on our campus in order to survive a zombie apocalypse. And why would you do that? and so on and so forth, and it became competitive within the faculty in terms of the titles of the courses and whose course would fill first as part of the registration process.

John: This is a nice follow up to last week’s podcast with Wendy Watson, where we talked about writing a constitution after a zombie apocalypse.

Rebecca: As an instructor how did you find the experience of teaching this passion course to be different from other courses that you taught?

Scott: Well, you go into your other courses, if you will, your normal course load, at least after a few years, you go in relatively easily to these courses… at least I found… and in the case of one of my classes. I’ve taught an intro to public policy class for 20 years, and actually wrote a book on it. So, you kind of walk in there and you don’t need to think too much about what you’re doing. I mean, that sounds terrible, but you get into a rhythm of your teaching and you keep current but some of the theories remain the same. The highlights of the course remain the same. This course, there was all a whole new set of readings because I was working with a colleague, it was not just making sure I was up-to-date on what I was worried about and taking lead on. But, at least having some type of knowledge on what she was talking about because a lot of the class was discussion based. Which again was probably a bit different compared to some of my lower level-classes in the past which, because they’re large you have to do a little less discussion in those situations. The other thing I would say is different when you’re teaching a first-year seminar compared to classes that have first-year students in it is that it’s a rare situation for most faculty to teach all freshmen or all first-year students… and there is a dynamic change, teaching a class like an economics class or a American government, where you’re going to have a lot of freshmen but you still have upperclassmen… and there is a dynamic that changes in that classroom in terms of modeling behavior and things like that. They’re not too far away from being high school students. You’ve got to get them focused. You really need to engage them as: “You are college students now. There’s an expectation we’re not going to go through every page of the textbook. We really expect you to do a large part of this work on your own so that you can bring your own perspectives and ideas to the classroom.” And again, that was something that was different for me and a lot of our faculty other than our English comp faculty that did this.

John: Because they’re used to small classes.

Scott: They’re used to small freshman classes. Most of us, we’re not used, so that was a difference.

Rebecca: …having that experience right now. We have a freshman colloquium in my department that I’ve never taught before until this semester. And it’s like: “Yeah, alright. We have to do these things that I don’t generally do in my other classes.”

Scott: And you’ve got to be really intentional with the students, which is a good thing anyway… but you can’t just assume that they know how to do college work.

John: That’s one of the benefits, I would think, of these courses… that it provides that bridge where you can focus on that without losing the upper-level students and that intentional focus on their needs could be really helpful in getting them acclimated.

Scott: Getting them acclimated, being intentional about the type of work that you expect, the type of writing you expect. That you can’t just copy and paste a wikipedia thing and call it a paper, and the acclimation to the rest of the campus was a big deal for us as well and is for the Oswego courses. There are a lot of resources here, there are a lot of events that happen here, and yes, we’re going to make you go to some of those, but the hope there is not that we’re making them do it, it’s once they get there they understand “Hey, I actually enjoyed this and I’m going to another one, just without being required to do it.”

Rebecca: Part of it is just figuring out the logistics if you’ve been doing it or where to find the information. Some students, if they don’t have that guided experience, might never discover it. There are so many other things going on.

Scott: Yeah, and we got to a point, at least for the first few years, where we actually were creating sort of cheat sheets of events, so that they had a calendar in front of them, so they didn’t have to worry about finding those of those types of things.
REBECA: This year, we’re piloting a first-year program at Oswego. Can you talk a little bit about that program?

Scott: Sure, we’re piloting nine first-year signature classes. That was the title that they wanted to put on our group here at Oswego. The program was developed by a committee of faculty and staff that developed a number of common learning outcomes that are very similar to what we did at Green Bay: strong communication, critical thinking issues… and then we recruited nine faculty from across the campus to engage in these ideas of, I won’t say common pedagogy, but some common learning outcomes. …and structure classes, and we did call them passion courses, at least internally. What is it that you want to teach? Is there something out there that maybe doesn’t fit traditionally into your curriculum but is of interest to you? Be creative about it. It’s okay to have fun. Be fun about what you want to do… and then really think through how you’ll get at these common goals, but also the goals of the class itself. So, we got a good group of nine courses… diverse courses… and they did just a great job in the development and even went above and beyond in terms of how they pitched and advertised their courses to the incoming students. They all did one- to two-minute videos… that our students actually did, which is great… and it really comes across as very professional. You can see the passion in their faces and I’ve already been told that a number of the faculty that have developed these first-year courses, it’s affecting how they think about their other courses as well.

John: That came up at several of the meetings (because I’ve been attending those too) and many of the people are saying that once I’ve learned how to do these things or I’ve tried doing these things, and some of it was credited to workshops that Rebecca did in the spring, but it’s changing how they’re teaching all their classes.

Rebecca: The conversations around the first- year class has been really interesting. Hearing those faculty talk through what they’re doing and work together has been really interesting, and so what you described at Green Bay as being that informal learning community certainly evolved here as well.

Scott: Well, that’s my sense too. Again, I specifically tried to stay away from it a little bit because I didn’t want my perspectives to fully guide what was happening and I wanted this to be a bottom-up faculty-led thing. But everything I’m hearing, is that the faculty are getting a lot out of those discussions and to really engage in teaching in a different way and around some different types of topics. And I think also to really think through the entire learning environment that we are providing here at Oswego, not just what’s going on within the classroom. I think all the faculty (I know at least one or two) require their students to go to the info fair over in the arena last week and actually I got passed on an email from a student who really credited “I never would have gone to this unless I was required to and by going I actually signed up for four different organizations.” …and this is exactly why we do these types of things.

John: …and that type of connection makes a big difference in retention and student success and engagement.

Scott: Yep.

Rebecca: How did the students end up in these courses?

Scott: They self selected as part of the information that goes out as part of the registration process. Late spring, early summer, these were offered up as an opportunity for them to sign up as part of their process of submitting their list of desired courses or preferred courses for the Fall. If they wanted to be in one of these courses or any of these courses, they put it on and then our our first-year advisers then made their made their schedules much like they do now… but they just included that particular course. I think there was a little bit of a concern initially since these courses count, but they’re electives. They’re electives within our 120 graduation requirements, so I think there was some concern upfront: “Why would students take these courses?” They don’t count for Gen Ed, they don’t count for the major but they filled pretty quickly, which I think speaks to both the marketing but also the topic areas that students find interesting. …and I think there are mechanisms for us to move forward to think that some of these courses could fit general education in a traditional way.

John: I’m not sure if this has changed, but in the early discussions of this, the goal was to have students request these courses with the hope that there’d be more people requesting courses than there would be slots, and then the students who applied for them but didn’t get them could serve as a control group so that you could get a benchmark without that self-selection issue. Has that been maintained?

Scott: I don’t know if that’s been maintained or not. I wasn’t part of those discussions. There are other ways of getting at some of the control groups if we need to do that, whether it’s simplistically students who did not take those courses or even pulling or surveying students that might be in like the English comp classes or the introductory math classes and using them as a pseudo control group. I’m gonna let IR worry about about how to get at some of those assessment issues. …and I will say that some of the issues around assessment… some of the issues around the success of this program… won’t show up immediately, and they won’t necessarily show up in data. We had a situation at Green Bay our first year where a student did not come back her second year and the faculty member actually got a letter from the student that said: “I want you to know that I really noticed how much time you spent with me, I noticed that you were paying attention to me and trying to get me involved and I’m not gonna be back in the spring semester, but I had a great experience here. This is just not the place for me.” That’s going to show up as a non-retained student and not a good statistic, but in many ways that’s a success story, and that’s something you can’t do in a lot of normal classes because you don’t have the ability to really engage with students in that type of a close way.

Rebecca: Do any have sense with the launch of the program this year whether or not the students in those classes are in the same major as the faculty member teaching them or do you think it’s more mixed?

Scott: I don’t know. I think it’s more mixed, but it’s a great question, and I’m going to guess they’re mixed, but I haven’t actually seen the the enrollments. And the reason why I’m going to say they’re mixed is that an incoming student would really have to pay attention to the bio of the faculty member, the description of the course, to be able to figure out “Is this course really within some major that I’m interested in? The courses themselves do not scream communication or business or any of that, so you’d a student would have to do a lot of sort of… not digging, it’s all there… but they’d really have to pay a lot attention to that. I’m sure there were some that did, but I’m not sure if that would be the majority or not.

John: Could you give us a few examples of some of these courses?

Scott: Sure. We’ve got nine as I mentioned and they are from all across campus. So Kat Blake is doing a course out of anthropology entitled: “The Talking Dead: Understanding Life from the Human Skeletal Remains.” …and I actually did print out the description a little bit here and what she had written was: They help forensic anthropologists investigate murders, bioarcheologists reconstruct life in the past, paleopathologists examine past disease and trauma… These are the bones of the human skeletons and they have stories to tell and students will learn about the scientific techniques for evaluating skeletal remains… so on and so forth. Who doesn’t want to play with bones, right? That’s great. And then another course that that is being offered is by Alison Rank out of the political science program and the title of her course is “The Witches are Hunting: Contemporary Feminist Activism in America” and she’s looking at the #metoo movement and feminist theory, and how these things have developed. And the interesting thing that she’s doing with her course is, she is occasionally (I think once a week) linking up with Mary McCune’s course out of history, and Mary’s teaching a course entitled “How New is the #Metoo? The history of Gender Activism in the United States.” So, those students will have the added benefit (at least from my perspective it’s an added benefit) of having some of these discussions in an interdisciplinary way. These are all highly engaging type topics. We have a course on how comic book characters are portrayed, and why is it that we turn to comic book characters when we’re looking at issues of justice? Why aren’t we doing these things ourselves? We have a course out of theater that’s looking at how black characters are portrayed within the arts and how that has evolved culturally. Another one out of theatre that’s actually looking at the interconnection between theatre and sports. Again, these are all topics that frankly students coming into a college/university setting would never think that they would be able to study. Frankly, a lot of things that we offer in the first year, students would never think about [LAUGHTER] studying coming out of high school. But, I really believe strongly that wrapping these accessible topics around college-level work is a really effective way to get students to think like college-level students and to do that get them prepared for the type of work that we want them to do as they’re moving through their years on campus.

John: When I heard some of the topics, I wanted to sit in on all of those classes. They all seem fascinating.

Scott: I think I’m going to, I’m gonna try to make some time to just sit in on these and try to get a sense of how they’re going.

John: They sound like a lot of fun.

Rebecca: They sound like a lot of fun. Yeah, definitely, and the videos are pretty fun too.

John: Yeah.

Scott: The videos are great. There’s a balance of the the funness. I’ve had people… frankly, I had a former Provost, when we’re really implementing our first-year seminars at Green Bay, talk about these courses as fluff courses. And I really had to push back on her, because I think, in many cases, these courses are more rigorous than some of the courses they would be taking otherwise (or in addition). They’re doing much more writing than they probably would be otherwise. I know, compared to an old large lecture class, where you’re taking a bunch of multiple-choice tests (because that’s the only way you can keep your sanity sometimes as a faculty member), that these are much more rigorous. The expectations are higher… and you’ve got to be present in order to do well in these types of classes, and I think we’ve all experienced situations with larger lecture halls, where it’s not unusual for a third of the class not to be there because they can get what they need out of a book or by copying notes.

Rebecca: As soon as you start tackling a topic that’s not traditionally a textbook, then you don’t have a textbook to rely on and you’ve got a start thinking about things differently.

Scott: …and it’s that’s a great exercise in and of itself to be, to move into sort of OER/direct digital access type things. There are all sorts of things out there that are not textbooks but are still primary source type materials or even current events type topics that you can really pull into these classes… and even the theoretical aspects of the discipline. How does psych address some of these issues? How does art address some of these issues? How does economics address some of these issues. Even around things like the #metoo movement or how comic books are portraying justice issues.

John: …and it shows students perhaps that these are really useful methodologies for approaching and analyzing things in the world that they may not generally see those connections, I think.

Scott: That’s right. I agree with that. You start looking at some of the popular culture issues through a different lens. I hope that the class that we taught on Disney really opened the eyes of students in terms of how Native Americans are portrayed or had been portrayed in Disney films, or black Americans or how gender issues are dealt. I mean it’s fine to just sit there and enjoy a movie, but at some point you want to start thinking through the larger social context that the film is being produced in and shown in as well.

Rebecca: I think it’s when you start hearing the students say things like well I can’t go to an experience like that without thinking X, Y, & Z now… or I can’t help but seeing… whatever it is… and I think that’s it’s a good sign of success.

Scott: It is. That’s what we’re about generally on our campus, is beginning to open up their mind and open up different ways of observing and interacting with the world.

Rebecca: Which, I think, leads into a good question about how are you gonna assess this particular program?

Scott: This program was started probably with a little bit more intentionality around retention, so we’ll look at retention rates. We’ve again been in contact with the NSSE folks to see if we can pull in some of their questions, even though we’re not in a NSSE year, and we’ll look at that as well. We’ll do some self assessments or surveys of the students and their experiences and what they thought of those experiences. And, frankly, I want to get the faculty response. I want to see how they reacted to the course. How did they think it went? How did they perceive the students responding to these classes? These classes do not necessarily automatically, just because they have interesting topics, lead to high faculty evaluations. Oftentimes new course development does not lead to high evaluations. You got to do these things a few times before you sort of get in your rhythm and really know what you’re doing. So, I’m hopeful that they’ll start looking at student outcomes and and are they maintaining connections with the students beyond the course? Which is something we saw on our campus that even though they weren’t their formal advisors, they would continue and seek out those faculty member for other courses. They would seek them out as they were walking across campus; or, if their office door was open, they would just stop in in a much more relaxed way than you might expect any other student to do that.

Rebecca: …sounds more like a mentorship kind of role, in some ways.

Scott: Yeah… that mentorship is probably a little strong, but it could develop into something like that. It’s the connection… it’s really focusing on what I feel is the most important connection that students can make, and that’s with the faculty member… that’s what’s going to keep them here… that’s what’s going to lead to their success. Yes, of course, it could lead into the mentorship as well but that’s where they’re spending their time… it’s with the faculty across campus. So, to the extent that we can facilitate that relationship… sometimes it’s good to bring them down to equal levels. We need to remain some level of distance and we have to ensure that the faculty is respected, but we’re also people and sometimes students don’t see that [LAUGHTER]… that we’re people. But, if we can get them in a small environment we can encourage them to talk. I used to require them to come to office hours initially just to make sure they at least stopped in a couple times. Those are all things that we can do to to help make that connection to SUNY Oswego.

Rebecca: …a strong connection to the episode that we had with Jennifer Knapp, talking about interpersonal relationships between faculty and students and that some of those outside of class relationships that are built (often through the classroom) are really important and really powerful. So, I think what you’re describing is exactly some of the research that she was describing in that episode.

Scott: I’m passionate about this area generally, and in this project in particular. I think there’s room to grow this, I actually think from a resource perspective, SUNY Oswego is in a better place than Green Bay was in terms of sort of scaling this; not that we go from 6 to 40 in a year. But, I think as we move forward if we find the type of success that I think we will find, we’ll need to have some good conversations around: How do we scale? How do we engage more faculty, more departments in this? How do we sort of expand these informal faculty dialogues around these important issues? …and we’re always going to be focusing on retention here. It’s an important element in student success. All of these are our building blocks to what I think is already s strong SUNY-Oswego education, but this is the beginning of the experiential learning that we’re trying to promote within our students.

John: Those informal discussions among the faculty are really incredibly important. In many of the meetings, when people were asked about what they were doing in their courses, many of them said: “Well, I stole this idea from Allison…” or “I stole this idea from Maggie…” or “I stole this idea from one of the other participants.” …and it was nice to see that sort of informal discussion.
So, we always end with this question of what are you going to do next?

Scott: Ooh… Well, clearly we are going to assess and look at this very strongly, and I’ve already mentioned that we’ve had some discussions around: “Can these courses be structured around general ed learning outcomes as well,” so that students don’t feel as if it’s a… I hate to use the word a “wasted” course, but sometimes that’s the way they’re looked at because they don’t count in GenEd… they don’t count in the major. It’s hard to explain sometimes to students that it doesn’t matter, you need 120 credits. That’s a harder discussion for a new first-year student than it is for a sophomore or junior. We’ll look at expansion. There are some things behind the scenes in terms of that expansion that I need to get a handle on in terms of numbers, and what would it take, and who’s doing what and how do we do that. I think, generally, the other thing we’ll probably start thinking and doing about is how can this seminar be the anchor to perhaps a more engaging elaborate first-year program for our students? How can we improve our advising process for our first-year students? How do we make that transition from that first year to that second year for students? How do we really get the faculty to engage with the idea that the entire campus is a learning community? There are resources out there that not everybody knows about, but people can tie into. Those types of discussions, I think, will be some of the things we’ll think about as we move forward.

Rebecca: Well, thanks for sharing.

Scott: Oh, this is great.

Rebecca: Some great stories.

John: Thanks for joining us.

Scott: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.