46. Creative risk-taking

When you teach the same classes every year, it’s easy to fall into routines. Classes, though, can be much more fun for you and your students if you are willing to take some risk by experimenting with new teaching approaches. In this episode, Dr. Wendy Watson, a a senior lecturer of political science and pre-law advisor at the University of North Texas. joins us to discuss how she has engaged her students by introducing some very creative and fun assignments in her classes.

Show Notes

  • Ishiyama, J., & Watson, W. L. (2014). Using Computer-Based Writing Software to Facilitate Writing Assignments in Large Political Science Classes. Journal of Political Science Education, 10(1), 93-101.
  • Watson, W. L., Hamner, J., Oldmixon, E. A., & King, K. (2015). 14. After the apocalypse: a simulation for Introduction to Politics classes. Handbook on Teaching and Learning in Political Science and International Relations, 157.
  • Wendy Watson (2016) Best and Worst Teaching Moments (Center for Learning Enhancement, Assessment, and Redesign, UNT video) – This contains a description of the zombie apocalypse project.
  • Center for Learning Enhancement, Assessment, and Redesign at UNT
  • Olson, Katie (2017). “Local Author Gets Cozy with Mystery Genre.” The Dentonite. October 3, 2017
  • Wendy Lyn Watson – author website

Transcript

John: When you teach the same classes every year, it’s very easy to fall into routines. Classes, though, can be much more fun for you and your students if you are willing to take some risk by experimenting with new teaching approaches. In this episode, we examine how one professor has engaged her students by introducing some very creative and fun assignments in her classes.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.
Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]
Rebecca: Today our guest is Dr. Wendy Watson, a senior lecturer in political science and pre-law advisor at the University of North Texas. Welcome, Wendy.
Wendy: Hi, thank you for having me.

John: We’re glad to have you here.
Our teas today are:
Wendy: I am drinking Paris. It’s a blend from Harney and Sons.

John: We have that next door.
Rebecca: Yeah, a tasty one. I have Irish breakfast tea today.

John: …and I have ginger peach green tea.
We invited you here to talk a little bit about some of the interesting things you’re doing with your classes. Could you tell us first a little bit about the classes that you normally teach.
Wendy: Sure. In the state of Texas there is a requirement that every student take two Introduction to American Politics courses in our department. We refer to that as the full employment plan. So, I teach both of those courses and then, other than that, I teach all of our law related courses. I’m not the only one, but I teach all of the law related courses: our legal systems course, civil rights and civil liberties, the rights of criminal defendants, constitutional law, an LSAT prep class, gay rights in the Constitution, and a seminar on the death penalty, in varying cycles.

John: You do quite a few innovative things in your classes, and one of those is having your students rewrite the Constitution after a zombie apocalypse. Could you tell us a little bit about that activity?
Wendy: Yeah, the idea is that the zombie apocalypse has occurred. This is actually for one of the flavors of Introduction to American politics, and this particular course deals with institutions: the founding of the Constitution, federalism, the legislature, the executive, the judiciary, and civil rights and civil liberties. The idea is that the zombie apocalypse has occurred. Huge portions of the population of the US have been destroyed and the remaining members of the country are required to rebuild the United States and part of that is rewriting the Constitution. Essentially, what they’re doing is building a government from the state of nature, but they don’t know that. They think they’re building a constitution after the zombie apocalypse, and that’s way more fun. It’s a guided exercise; they get worksheets every week making them think about “What is bicameralism? What are the benefits of bicameralism? What are the drawbacks of bicameralism? etc. They don’t just get to go off and write crazy things. They actually have to think about stuff and then they work in groups to create these Constitutions. One of the things that I really love about the course is that they actually do have to grapple with these issues. They sometimes get pretty heated.

John: How large are the groups?
Wendy: Usually these introductory courses are about a hundred and twenty five students and I put them in groups of about five to seven.
Rebecca: And are these things that happened outside of class, in class, online?
Wendy: No. I’ve taught the class as an online class in which case it obviously happens online, but when I’m teaching the class as a face-to-face course I actually do give them class time. Having them do it outside of the class nothing ever happened, giving them the time in class keeps them from hating me and also ensures that they actually do provide some sort of useful product at the end.
Rebecca: What assignments or exercises or things that you would normally do in class does this exercise replace?
Wendy: You know it doesn’t actually replace any exercises because if I weren’t using this activity, all of their homework would be outside of class and they’re still doing all of that. So what it’s really replacing is me lecturing and I’ve got no problem with that and I don’t think they have a problem with that. It’s more exciting or more interesting for them to be doing something, talking to each other than it is to be sitting in a seat listening to me, I think, I’m pretty sure. And I think it’s actually more educational for them to be engaging in the material as opposed to passively sitting and listening to me. Yeah. So although all they’re missing out on is me talking.
Rebecca: How did you how did you decide to go in this direction and develop this particular activity?
Wendy: I was trying to think of a way to create a simulation that would last throughout that semester, so something that kind of continued over the course of a term. And I wasn’t really sure what that would be, and I think we were watching The Walking Dead. But honestly how that all came together I couldn’t tell you, but yeah I’m really happy with that. It’s been adopted by several of my colleagues and by a professor at University of Whitewater. She used it in a summer program for high school students, and yeah, I’m really happy with that how that one turned out.
Rebecca: How did the students respond?
Wendy: You know,of course there are always students who are not going to respond at all. But I’ve never had a student who actively said that they hated it which is, I guess, good. I’ve had a lot of positive feedback and I had you know one of my best student interactions ever over this particular assignment. Again, I’m going to apologize to all of my biology friends out there. One of the features of the assignment is that the zombies fall into two categories. The type one zombies who are traditional brain-eating zombies and then the type two zombies who have developed a lesser mutated form of the virus. And so they have features of zombies, they have the shuffling gait and the slurred speech, but they have higher order cognitive thinking and they don’t eat brains, and they’re just generally safe. But if two type two zombies have a child together there is a probability or a possibility that their child will be a type 1 zombie. Again, this makes no sense at all, since it’s a virus and that just doesn’t make any sense. But, it raises this question of what do you do with type 2 zombies? Do you sterilize them? Do you kill them? What do you do with them? And they were grappling with this issue one day. And this poor student comes in, and he was, I swear to God, he was almost in tears. Because his group had decided collectively to exterminate the type 2 zombies and he said,” what do you do when you encounter people who are terrible?” And so he ended up having this long talk about how do you deal with the notion that there are Hitler’s in the world. I was like “Well, you have to remember that there are Gandhi’s in the world.” It was a long and lovely conversation about the essence of mankind and the balance of good and evil. And I kept emphasizing to him that this wasn’t real and that his friends were not evil, but anyway it was it was a great conversation and I was so touched that he took it so seriously. It’s just a testament to me of the fact that students really are interested in the material if you give them an opportunity to be interested in the material.
Rebecca: It sounds to me too like it allows them to really grapple with the really difficult conversations that are around rights and lack of rights and who gets those rights. That might be really uncomfortable if you talk about it in a in a real situation, but in this safe simulation you can have some of those challenging conversations that you might not be able to have as effectively.
Wendy: Yeah, I think that’s right. If you’re talking about things like race or sexual orientation, you’re always confronted with the fact that there are people in the room whose actual rights are implicated, and that does tend to make people sent to themselves perhaps, and that’s not necessarily what you want in real active discourse. So, when you’re talking about something that is seemingly unreal, it is unreal… they’re zombies… it’s not real. I do think that it gives people the opportunity to think through issues in a way that is safer, but also more honest.

John: The type 2 zombies add to the degree of difficulty or the level of challenge there.
Wendy: Yes, exactly.

John: You’ve also created a 500-person learning community, could you tell us a little bit about that?
Wendy: Yeah, that was nuts! My university decided to try to create a variety of different models of learning communities, sort of all at once, that alone was nuts. But I was going to be involved in a combined course learning community, so without any residential component. And I found this wonderful man in the psychology department who probably had no idea what he was getting into, and we created this community that was 500 students. His Introduction to Psychology course and my Introduction of Political Behavior course, that’s the other half of our introductory American politics duo. And our courses were back-to-back, so there were times when he could have two hours, and times when I could have two hours. And we focused on political psychology, specifically as it related to campaigns. And over the course of the semester, they each had to read three or four articles and write one page papers about them, little summaries, and then they came together and they shared their information, and they had to come up with the campaign strategy for either one or two presumed political presidential candidates. At the time we thought that was going to be Clinton and Rubio… that obviously didn’t happen. But they created these poster presentations and then we picked from among those poster presentations the 10 best, and we took those to UNT on the square which is a little gallery space in downtown Denton. And we invited faculty and university administration and we invited the Denton Record Chronicle which is our local newspaper. And the students really got into it, the ones who won showed up with their little red bow ties if they were representing Rubio and they had candy at their stations. And it was really awesome. It was great.
Rebecca: What do you think one of the biggest learning gains was for students who were in this learning community scenario where you were diving into something in depth from two different points of view?
Wendy: I think one of the things that they gained was an understanding that these two disciplines actually interacted with one another, that psychology and political science weren’t sort of siloed ideas, that they actually were related to one another. And I think one of the other things that they learned is that what they learned in class actually had implications for the real world. That things that we were learning in psychology and political science had implications for how politicians were actually running their campaigns. And that they could take the skills that they were learning at UNT and potentially apply them to a job, which is always a big thing. [LAUGHTER] Getting a job is good.
Rebecca: What level are the students in these classes?
Wendy: In those particular learning communities, most of the students were freshmen, first-year students, because they had to be advised into them, somebody had to sort of point them towards this pair of courses, so they tended to be freshmen. Otherwise these courses actually tend to draw students all the way up to their senior year, because they put them off until they have to graduate. But for these particular communities, they pretty much have an advisor say, “Hey, here’s a good idea. Take both of these courses.” They tended to be freshmen.
Rebecca: Did you find that the learning community method works particularly well with first-year students?
Wendy: I think for a lot of types of innovation it doesn’t necessarily, but I think for this, it did, because I think their desire to please was strong. And I think that they didn’t any preconceived notions of what college classes were supposed to be like, so they were maybe more receptive to the idea of doing something different. For all they knew this is what it was supposed to be like. [LAUGHTER]

John: …and getting that introduction to an interdisciplinary view of the world is probably good to do before they get too deeply into the silos of their major.
Wendy: Yeah, I agree.
Rebecca: So you’re full of brainy ideas and another one that you pulled off was an online Electoral College simulation game, can you tell us about that too?
Wendy: Yeah. So that was a lot of that was a lot of fun. I actually have to give most of the credit for the online component to our office, here it’s called CLEAR the Center for Learning Enhancement Assessment and Redesign. The assessment component has largely gone out of clear, but that’s still what we call them. They do all of our online support, learning management system, redesigned helping us create online courses, all of that sort of work. And I had a sort of a low-tech version of this course. Originally they were working in groups, I always make them work in groups, I don’t know why. But they had groups and the idea was I used the map from 270 to win, which has sort of the baseline Electoral College predictions, and which states are going red, which states are going blue, etc. And then students had campaign money and they could essentially bet their money on individual states. And if you were the Republican Party and you bet fifty dollars here, but then the Democrats get 51, then the Democrats won the state, so whoever bet more money in a state won the state. And so you could see the strategy of betting in different states of spending more campaign money more campaign resources in each state, and as you won a state, the states that were blue moved around or the states that were red moved around and you could see the total – who was winning the electoral college. And it was played in three rounds. But this was a huge pain to implement in the classroom with having to update this Excel spreadsheet every round and get people’s votes every round. It was a nightmare. So CLEAR created an online version for us that allowed students to play against each other online and it was really slick, it was beautiful, I loved it.
Rebecca: So, I’m noticing the “loved” as opposed to “I love it”.
Wendy: You notice that didn’t you.[LAUGHTER]…… Yeah, so I think another point to make here is that if you’re going to launch into one of these grand plans, you really do want to have some long-term commitment from your University. I love my university but long-term commitment is not their forte and for the learning community, for example, Adriel and I (my co-conspirator and I), we put a lot of effort into that course and we ended up offering it twice. It went really well both times but to the extent we needed money it came from a Title III grant that ended. So, we didn’t have the money anymore and then we also depended very much on help from the registrar, from advising, and from admissions to help us coordinate all of the the details. Because it was no small matter, right? It was actually very difficult. It wasn’t just us. There are all sorts of offices that had to help us out with this. And the university basically was like, “Oh, we’re done.” That was difficult and so we just lost the necessary institutional support for maintaining that program. And with the electoral college I went for like a year and a half without teaching that course, so it didn’t get used because nobody else was using it. And so CLEAR stopped supporting it on their website. It just went away and it’s just gone. So, it’s just one of those things. You kind of need to get it in writing, because there’s a tremendous amount of start-up costs associated with these programs and unless you know that that’s going to carry forth and this investment is going to pay out over an extended period of time, tt could be a little bit demoralizing.

John: In one of your other experiments in class, you did something with a mystery room. Could you tell us how that worked?
Wendy: Oh yeah, that was this last year. That was so much fun. Yeah, so the game was actually called Free Lucky. Lucky is UNT’s unofficial mascot. He’s an albino squirrel; he’s actually not lucky at all. We’ve had a series of Lucky’s on campus and the only two that I’m aware of… one got carried off by a red tail hawk and the other one got hit by a car, so they’re not lucky. [LAUGHTER] But we call him Lucky and you can get little lucky dolls. And so I got little Lucky dolls and I shoved them in little cloth pencil cases and I put combination locks on the pencil cases, so he had to get him out by undoing the lock. And I’m put my groups of students… groups again… in various study rooms in the library and they each had a little encased enshrouded Lucky in their room. And then they started the game with a question on their learning management system on Blackboard. This was for an LSAT prep course and the beautiful thing about the LSAT is that you have these questions with very specific answers. No question… here’s the answer… that’s it. The first question, if they got it right, it led them to a webpage with another question; if they got it wrong it led them to a webpage that had nothing and then it sent them back to the original page, and so forth and so on. It sent them to various pages around the web, some of them with clues, some of them with other questions, eventually it would’ve taken them off of the web and sometimes it pointed them to different clues around the room. There were various and sundry things on the table, some of them which mattered… there was a playing card… it actually was a clue, but then there were things like spools of thread that meant nothing. There was envelopes taped under the table that had a whole series of questions. And the questions there, if you answered them all, there were four of them and those gave you letters and then there was a tongue depressor on the table that helped you translate the letters into numbers and that was the code to the combination lock and that allowed you to free Lucky. And the first team that got Lucky to me… I was sitting in the lobby of the library, first team that got Lucky to me won… and they won packets of colored highlighters, which doesn’t sound exciting but they were all pre-law students and that’s like gold in the legal community… is colored highlighters. So it was exciting, they were really thrilled.

John: It sounds like fun.
Rebecca: It sounds like a lot of fun.
Wendy: It was.
Rebecca: What made you decide to do a mystery room?
Wendy: Well, you know, we have one here in Denton, and I think it looks really cool and I want to go, but I can never get people to go with me, and so I decided well I’m just gonna create my own. I wanted to do something, again, that was interesting. As much as the LSAT prep stuff was really interesting and important for my students, it’s not super engaging. We could stand up there and write logic game trees on the board, for hours on end, but that’s not exciting. That’s not even lecture exciting, that’s just really really boring. So I wanted to at least break up the class a little bit by having something that was more engaging, more active, something that was interesting.

John: And it brings in gamification too, where there’s some incentives and competition.
Wendy: Yeah. Oh yeah, the competition was big. I had one group that came down with Lucky after about a minute and a half. I was like, “You did not answer all those questions.” The guy who handed me Lucky, he’s like, “You gave this puzzle to a marine .” [LAUGHTER] I was like, “So, did you just bust the lock?” He’s like “No, I didn’t have to bust the lock. I could get him out without busting the lock.” I was like “You have to open the lock, you can’t cheat.” [LAUGHTER]. So they went back, they did it. But anyway, yeah, it was definitely a game to them. They were serious about it.
Rebecca: That’s hysterical and unexpected, right? [LAUGHTER]
Wendy: Completely.

John: A common theme of all this is that you seem to experiment with your classes and take some risks in trying new things. Could you tell us a little bit about what prompts you to do that?
Wendy: A couple of things. One, is that honestly it keeps me interested in the courses. I can get bored with the material as much as they can. In fact, they sit through it for a single semester, I sit through it for semester, after semester, after semester. And you can only talk about the appointments clause for one or two times before you’re like “Oh my god, I’m gonna dig my eyes out. This is really dull.” And that’s something I actually enjoy, right? I think the appointments clause is interesting. You still want to shake it up a little bit. And the other reason is that I really do believe that students learn better if they are engaged. As much as I love to hear myself speak, I don’t necessarily think that they love to hear me speak. I think that they get more out of my class if they are doing something. If they are seeing some connection between what we’re doing in the real world. If they can see themselves actively engaged. If they have a sense that they have power in the class. Some sense of control over their own education. I think all of those things are really valuable to them. So it’s a little more effort for me, but I think the payoffs are worth it.
Rebecca: So all of these examples that you’ve shared with us today are really different from one another: they use different technologies, different setups. What is your advice to someone who wants to take some risks and try something new, but it’s something that they’ve never done before?
Wendy: Start small. Don’t start with a 500 person learning community, which is what I did. That was dumb. It worked out, but it was dumb. Yeah, start small. Collaborate with somebody so you have somebody to lean on and share ideas. That’s maybe why the learning community worked, is that I had something called the Core Academy, so we were focused on these sorts of things together. And then I had my my co-teacher, Adriel, to work with. I think having a support system and starting small is the way to go. You don’t have to do a semester long simulation, you can devote one class to something. Use a method that lots of people are using, like team-based learning. You don’t have to do that all semester you could do it for one class. There’s nothing wrong with starting small and then getting bigger.
Rebecca: Did you start small?
Wendy: I did not [Laughter].

John: Somehow I suspected that would be the answer.
Wendy: Yeah, that’s not my style. But again, I think that if you’re worried about getting started, if you are less stupid than I am, then don’t hesitate to start small. There’s nothing wrong with that.
Rebecca: Have you had any student resistance to some of the alternative or non-traditional methods that you’ve been using in your classes?
Wendy: I get a little resistance sometimes. For the most part, they actually seem to enjoy it. Every now and then I’ll get a student who seems to think that I’m not doing my job. I mean I’ve had students who flat out on evaluation have said “I expected to come to class and hear you talk and you didn’t.” Like “Really? That was what you expected?” I mean, yeah, I assumed that is the expectation, but like, “You’re disappointed that didn’t happen?” I can’t imagine that. And of course there’s always, as I mentioned, a lot of these things involve group work, and a lot of students have resistance to group work. Even when the group work ultimately works out okay, they still are annoyed that I put them in groups. Just the anxiety associated with group work carries over to the end of the semester. Of course, some groups don’t work out. You’ve always got somebody in some group that either doesn’t pull their weight, or is responsible for a part of the project and fails to turn it in, or somebody in the group who is bossy. You always have some group that’s got a problem and I usually try to mediate that situation, but sometimes they don’t come to me until it’s too late. There are always points of contention. But they’re relatively few, and honestly I’ve always got a few complaints when I lecture too. I’d like to say I never have complaints there, but I do.
Rebecca: I read this really great article about you being a mystery novelist.
Wendy: I am.
Rebecca: Can you talk a little bit about that?
Wendy: Sure, yeah. I am a mystery writer. I started writing a long time ago, right around 2001 actually. A bad year. But I had my first novel published in 2009. I write a type of mystery called a Cozy, which is exactly what it sounds like, it’s cozy. They are light, often funny mysteries. Amateur sleuths, so no cops or private investigators. They can be in the book, but they’re not the primary character. Usually female sleuths, small town, no sex or violence on the page. I mean obviously somebody dies but it happens off the screen. They’re really quite delightful. I said PG-13, I actually included the word “bitch” in my first book, and it wasn’t even calling somebody bitch. It was like son of a bitch. I hope I’m not destroying your podcast by using that word [Laughter]. I actually got nasty emails about using that word. Really? Oh my goodness. I don’t use that word there anymore. Yeah so I started writing I’m working on number seven right now and, that’s that.

John: How do you manage that along with all your innovation in class? It seems like that’s a lot of demands on your time. How do you allocate your time?
Wendy: Not well. Yeah, I was talking about this with a colleague this morning, we were talking about this LSAT prep course (she’s teaching it this semester) about the fact that prelaw students really should be studying a lot for the LSAT. It’s a huge portion of their application. Yet, for some reason, they don’t and instead they focus so much on their GPA, which is important, but honestly, not as important as their LSAT score. They shouldn’t let their GPA slide either, let’s be clear. But in the grand scheme they should be focusing on their LSAT score. We were discussing the fact that the LSAT is way far away but their GPA is right in front of them, and so that just feels like the thing they need to tackle right now. And for me my deadlines are way far away and my courses right in front of me. So I tend to focus on my coursework and I’m not so great about meeting my deadlines, and I apologize deeply to my editor, but that’s just the way it is. I do though have a calendar, a very detailed calendar, that I keep, that has specific time set aside for every single thing that I do. Not always true to that calendar, but I do have a calendar, and it includes time set aside for writing.
Rebecca: Do you find that your writing life and your teaching life influence one another?
Wendy: Yes. Certainly my academic life has influenced my writing life. One of my books was set on college campus and I got to kill off a couple people that I didn’t like so much, which was awesome [Laughter]. Certainly, I think that my tendency toward narrative, toward storytelling, influences my use of hypotheticals in my classes. To the extent that I’m sort of telling stories. Like the zombie apocalypse, I didn’t just write a paragraph: there has been a zombie apocalypse. It’s this, probably too long story, about this has happened, and it’s all dramatic, and that’s definitely a carryover from my writing life.
Rebecca: I imagine that those details though and that spike in the climax to a story, are all the things that get students really engaged and interested and and buy into the simulation and take it seriously. As opposed to something that’s a little more surface level and that it’s a little harder to imagine.
Wendy: Yeah, and I think sometimes one of the things my cozies tend to include is humor, at least I hope it’s humor. I tend to inject that into my hypotheticals a lot and I think that that helps. One of the simulations that I do in my legal systems class is a negotiation divorce case. Each side in the negotiation has information about their client. Some of its common knowledge, that both sides have, and the wife’s attorney has knowledge that only the wife has provided and the husband’s attorney has information that only the husband has provided, and they know that that information is going to come up during the negotiation in a series of PowerPoint slides. They don’t know when that’s going to happen, but the idea is that all the sudden the wife is going to blurt something out during the negotiation. They also don’t know that there’s information that the husband and wife have not told their own attorney and that’s going to come out in the course of the negotiation. So I had great fun crafting the simulation; like the things that the husband and wife have done, and the pieces of information that come out are delicious, and the students have so much fun finding out about these details. And yeah, I think that that makes the whole simulation so much more engaging, instead of just calculating the appropriate alimony. I think it’s a lot more fun.
Rebecca: Can you share a couple of tips from your creative writing self that might help other people come up with hypotheticals or examples that they could use in their classes?
Wendy: Yeah, I think one thing that you want to do is provide detail. If you’re going to create a hypothetical, create a character to go with the hypothetical, and then provide some detail about the character and the setting and those sorts of things. It really enriches the hypothetical. It doesn’t all have to be completely relevant. In fact, sometimes it’s better if it’s not all relevant because then it forces the student to look past the things that aren’t relevant to find the things that are. I think that’s probably the key is to include at least one person in your narrative and then provide some detail. Provide a setting, provide some description of your character, provide some element of detail about what’s happening, so that it’s not sterile or clinical. Because that’s, like you said, that’s really going to draw the student in, in a way that’s sort of, A happened, B happened, C happened, or not.
Rebecca: That’s great advice [LAUGHTER].

John: We always end with the question, what are you going to do next?
Wendy: So this year I’m actually not teaching, which it is really weird for me. Last year this time, I took a position as the director of the university’s core curriculum. So, this year I’m going to be continuing with my pre-law advising but otherwise I’m focused on the university’s core curriculum. I will be engaged in assessment, which is everybody’s favorite thing, but I’m also gonna be developing a lot of programs related to our cores. So some programs related to writing across the curriculum, some programs related to bringing back, I hope, some of our learning community endeavors, and possibly exploring some other options that would allow us to really enrich our university core curriculum for our students. When I talk to students now they talk about them as the basics or the things that they have to check off, and I want them to think of those classes as something more than that. So that’s what’s next for me.
Rebecca: Sounds like the right person might be in that job to help inspire students. [LAUGHTER] I think sometimes that’s a hard sell these days, helping students recognize the value of a liberal education, and get them excited about it and help them find connections.
Wendy: Yeah, I agree. I think I have a tough road ahead of me but I’m going to do my best.
Rebecca: I look forward to hearing more about it.
Wendy: Yeah, thank you. I’d love to come back sometime.

John: We’d love to have you back.
Rebecca: Thank you so much for spending time with us this afternoon and sharing all your great initiatives in your classes, I hope it’ll inspire a lot of our listeners.
Wendy: Thank you, I really enjoyed it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer.

36. Peer instruction

Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know (and don’t know). Did you picture a lecture class, students taking a test, or students writing? In this episode, John discusses three ways in which he has been using peer-instruction in his classes: classroom polling, calibrated peer review writing assignments, and two-stage exams.

Show Notes

Transcript

Rebecca: Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know. Did you picture a lecture class, students taking a test, or students writing? If not, stay tuned, this episode explores ways to use peer-instruction to transform the learning experience.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

Rebecca: Today’s guest is my co-host John Kane. John is the director of the Center for Excellence in Teaching—that’s not even right…

[LAUGHTER]

John: …Center for Excellence in Learning and Teaching at SUNY Oswego.

Rebecca: Yeah, woops! Welcome to your own show, John!

John: Thanks, Rebecca.

Rebecca: Today our teas are…

John: Prince of Wales.

Rebecca: Oh, that’s a good one.

Rebecca: I have Golden Tipped English Breakfast today.

John: Excellent.

Rebecca: One of the areas you’ve been teaching experimenting in, and that I’m fascinated in, is peer instruction. Can you tell us a little bit about what peer instruction is and why you’re drawn to using this methodology in your courses?

John: Peer instruction involves using peers to assist with instruction, where students explain….

Rebecca: Thanks John.

[LAUGHTER]

John: …where students explain things to each other. One of the issues that we have is that, once we become experts in the field, it’s very hard for us to express things in terms that are easily understood by students. There’s a “curse of knowledge;” once you become adept at something, it’s really hard to explain things at a level that’s appropriate to the level of understanding that students may have. There was a classic study done in which a researcher gave people a list of songs, very well-known popular songs, and asked them to tap out the beats from that song.

Rebecca: Oh, I would fail…

John: …and then before actually seeing if people would recognize it (who had the same list), she asked them to make a prediction of what proportion of people would understand it based on their tapping… and they overestimated that by a factor of I believe, somewhere around 20 times. Basically, it was purely random if people happen to guess it. But the issue is, once you hear something in your own mind, it’s clear to you, but it may not always be clear to the people who don’t have the same rich net of connections. When students are explaining things to each other, they benefit from taking a position, arguing that position, trying to filling gaps and they’re also explaining in terms that are appropriate for people at their level of cognitive development for people who have a similar background in terms of what they know and their prior knowledge.

Rebecca: Sounds like a really good way to expand and refine mental models and also just develop better metacognition. Because, as soon as you go to explain it, you realize what you don’t understand.

John: …and if you don’t understand it yourself, your peers will often help you understand. they’ll say: “Well, you haven’t considered this…” and that sort of interaction is one that doesn’t work as well when it’s instructor to a large group of students. But, it does work very well one-on-one.

Rebecca: You’re known on our campus for teaching really large lecture sections. Implementing peer instruction in a large setting can seem pretty daunting, especially to someone who teaches smaller classes like I do. What strategies do you use?

John: The most commonly used one is to use clicker quizzes… and I use a methodology that Eric Mazur developed slightly over 20 years ago, where you ask the students a challenging question… you try to find questions that about half of them will get wrong… and over time you can develop that, you can come up with a pool of questions that fit somewhere in that range… and you let students first vote on the response themselves after they’ve had a little bit of time to process it, and then you look at the results. If you see that 90% or more of them got it correct or even 80% or more, you can just go over it and move on to the next topic, because most students understand it. But, if you see that somewhere around half of them get it right and somewhere around half of them get it wrong (plus or minus 20% or so), then the next stage is to let them explain it to each other, and that’s where the peer instruction comes in. When you have students argue it and take a stand and a position on it, we get a very significant gain and improvement when we then let them vote on it a second time… and the usual practice is not to reveal the poll results or the answer until after they’ve had that opportunity to engage in that discussion.

Rebecca: Just make sure, to make sure I understand correctly: you do the poll, you see the results as students don’t see the results…

John: Right.

Rebecca: …based on their answers or their responses when you decide whether or not they do the peer instruction piece. How long do they usually talk to each other about the topic?

John: It depends on the problem and normally I will have some undergraduate TAs and I’ll wander around the class and see what they’re talking about, listen in, answer some questions from them and the TAs will be doing the same thing…. and it’s usually pretty clear when they’re coming to a consensus. You can see them reaching for their clickers or their phones and getting ready to vote, so generally it may only be a minute or two, it could be longer… it depends on the complexity of the problems. Some of the problems require a bit of effort and require some calculations, but normally they’ve already done that… so, the second stage, where there’s a discussion, you can hear the volume build-up and then as they’re approaching solutions and consensus, it tends to drop back down again. It’s fairly easy to get a pretty good read on where they are and when they’re ready to vote again.

Rebecca: I imagine that you would really need to keep your ear to the ground, otherwise chaos could ensue. Because now, if they’re finished talking about the problem and there’s still time, then they could easily derail if you’re not quick to get back to the clicker question.

John: Right, and normally the time is generally held fairly tight. I suspect sometimes it’s only 30 seconds to a minute, other times it may go up to a couple minutes, but if I see them getting distracted and doing other things, the polling starts immediately.

Rebecca: Obviously technology is your friend in this particular situation. Can you talk a little bit about the technology you’re using to manage this many students all at once?

John: Here, we’ve adopted iClicker as a campus standard, so we use that in pretty much all of the classes where we’re doing polling and there’s both a physical radio frequency clicker that students may buy or they can buy an app and pay by the semester or over four years for the use of the app.

Rebecca: How do you make sure that the cost doesn’t get prohibitive to students?

John: That’s an issue, and it’s been a major source of concern…

Rebecca: They’re not very expensive, right?

John: Well, they can be expensive. A new clicker costs somewhere around $40. A used one can often be purchased for $15 to $20, sometimes less… and the apps I think, are somewhere around $12 to $15 for a semester and I think about $35 for four years.

Rebecca: …and you can use the clickers in all of the classes, right? So if multiple faculty member(s) are using all the same system, then the investment is a good one for students.

John: …and that’s why we have a campus adoption because in places where you don’t have that, students might have to buy two or three or four different clicker systems in different classes. So, once they buy the clicker for one as long as they hold on to it, they can use it in classes for the rest of their career. Almost everyone in the economics department, for example, now uses clickers, so if they’re economics majors or business majors, it’s very likely they’ll use them in multiple courses. The cost is much more tolerable when it’s spread out over multiple classes.

Rebecca: The other area where you do some peer instruction in these large classes is in writing. Which seems kind of crazy. You have all these students in this big classroom and somehow you manage to do writing assignments.

John: Yeah, my large class generally is somewhere between 350 and 420 students. At one time, for actually about a decade or so, I was giving weekly online discussion forums. But grading that or evaluating that and providing feedback was taking an awful lot of time…probably 30 to 40 hours a week. So, I pretty much…

Rebecca: A full-time job in and of itself…

John: I stopped that a few years ago and, a few years back, I replaced that with calibrated peer review assignments. The calibrated peer review system is something that Eric Mazur talked about while he was here… a visit in 2014… and when he mentioned it, a lot of people got excited. The way the system works is that you create an assignment, you store it on a central server at UCLA, and then it’s something that other people can adapt and use and modify—it’s released under a license, which is similar to a creative common license within the system… and you create the assignment… you create an evaluation rubric for the assignment… and you have to be really careful in designing that to make sure it’s one that students will be able to apply, because other ones that do that… and then you create three sample assignments yourself: a low-quality one, a medium quality one, and a high quality assignment… and you have students submit their own assignments first (according to the rubric and guidelines you provide to them)… then they go in and they evaluate the three that you’ve done. They’re given in random order, and they’re assessed in terms of how closely their evaluations match yours. That’s the calibration part. Students receive a calibration score based on how similar their evaluations are to the ones that you assigned to the sample responses. Then after they complete that stage, they evaluate each other, using the same rubric, and a weighted average of those scores is assigned as a component of the grade. They’re graded in a number of dimensions. One is based on the weighted average of the peers, where students who had a high calibration score will have evaluations that rate more highly in evaluating other students. They’re also rated in terms of how closely their evaluations match the others during that stage. So, if their evaluation is an outlier… much higher or much lower than other students… they lose some points on that… and then after they evaluate the other three students, they rate their own work… and one of the goals of that is so that they have improved metacognition. That, by the time they go back and look at their work again, they’ve rated three works by the instructor and three assignments done by their peers and then they’re asked to evaluate their own using the same criteria. What’s really interesting about the calibrated peer review process is their grade on this is tied not to whether they give themselves a high or low score on this, but it’s how close their self evaluation comes to the weighted average of their peer evaluations. So, they have an incentive to try to look at their work more objectively, and not try to game the system… because if they score their work too high or too low, they could end up with no points on the self-evaluation stage. So, the closer they get to the weighted average of their peer evaluations, the higher the score will be on that component.

Rebecca: I think that’s an area that we often see students struggling, is being able to effectively evaluate their work or other work. So, really training them to use a rubric and understand and think about what’s important or what’s not important about particular kinds of assignments or particular kinds of work could be really valuable to students in a way that we don’t really have other systems to do that.

John: The nice thing about this is it scales really easily. There’s a lot of upfront work in creating the assignments, creating the rubric, and a really good practice is to test them thoroughly before you give them out the first time. What I normally had done is asked some of my peers to look at that, some of my colleagues to look at it, and sometimes I’d have some upper-level students were…. and this does give students a little bit more reflective practice, where they get to look at their own work a bit more critically, perhaps, and reflect on it and see how they’re doing compared to how other students are doing in the course… and I think that’s helpful.

Rebecca: I think that the rubric would probably be a challenge to make but I think what would be more challenging is putting up those different assignments that are scored at different levels at the very beginning as your calibration tool. What strategies have you developed to make those in a way that it doesn’t take forever?

John: Well, I only do this three times in a semester, and once you’ve done it once, if you design it in a way so that it won’t go stale… and I generally have students, for example, find some articles in the news in the last six months that relate to a topic that we’ve talked about, or I ask students to find some examples in their own life to illustrate behavioral economics concepts in one of the assignments, for example…. where it’s not something that they could easily copy and paste from other people’s work. Because, there is always a concern with academic dishonesty and so forth. You don’t want these things showing up on Chegg or any of those other systems, where it would be easy to copy and paste good responses. So, I’ve tried to design assignments where once they’re done, they can be used for multiple years in one form or another. I modify them each year based on how they work. But perhaps a more serious problem is what happens when students really don’t like the evaluations. One of the things I’ve done when I’ve used this is to have three of these assignments, but I drop the lowest score… because, sometimes people will get some scores back that they didn’t expect or they may have neglected to look at the rubric I sent them and they may have omitted a major part of the assignment and ended up losing quite a bit of points all the way through that. But, as long as one of the scores is dropped, they have the opportunity to learn from their mistakes and do a little bit better. But, there are procedures built-in that make it easier to catch any outliers when you have someone who is just rating everyone extremely highly or rating everyone really poorly—inappropriately highly or poorly. There are tools in it which will give you a list of all the cases where there’s a high variance across reviewers or where someone happened to be evaluated by people who had very low calibration scores… so, if you end up with two out of the three peer reviews with low scores, that’s something that’s flagged by the system. I check all the cases where it’s flagged and I tell the students if they’re unhappy with their score or if they have any questions about it, to contact me, explain why they’re dissatisfied with their score, and then I’ll go in and look at it. In nearly all cases, it’s been an issue with the students submission and not with the peer reviews. Because, while some people tend to overrate things and some tend to underestimate some of it, compared to where I would evaluate the wok… on average, it’s been very close, typically, to what I would have scored or what I would have assigned as a score. But I do make, in rare cases, some adjustments when I see that something went wrong in the process.

Rebecca: Do you prevent students from seeing the score then, until you’ve reviewed all of the scores to make sure that you’re okay with what has happened before they have access or…?

John: In this system, that really can’t be done easily…

Rebecca: ok.

John:… because what happens is they get the results as soon as the last stage is completed. I’ll send a note out saying, “Now that the stage is completed, you can review your scores, you can read all the comments that your peers have provided, and you can see what your grade is at each component…” and we have gone over that in class so they know what they’ll be seeing.

Rebecca: What kind of workload do you end up with, dealing with problems?

John: In general, when I’ve used this in the class of 360 to 420 students, there’s usually 3 to 5 students who find their grade unreasonable, and sometimes, I found the grades perfectly fine. Occasionally one or two of those, I’ll make some minor adjustments to—if something went wrong where one of their peer reviewers didn’t show up, for example, one or two of them didn’t complete that stage of the assignment, and someone was overly harsh or perhaps overly harsh in their grading, but it’s rare.

Rebecca: Can that system be used for things other than writing? Like other kinds of documents?

John: It could be used for any type of document because basically students will either write something up or they’ll submit something and it could be an image, it could be used for peer review, or calibrated peer review, on pretty much anything as long as it can be disseminated in digital format. It could be used for websites, for example.

Rebecca: Well, that’s what I was getting at when I was asking.
You also teach some upper-level seminar courses with 30 or so students. This semester, you tried a two-stage exam after talking with Doug McKee when he was on campus about it. What is a two-stage exam and how did it work?

John: Backing up a bit, I was considering it even before Doug came here because I heard the episode of the Teach Better podcast where they discussed a two-stage exam and then when we were talking here and he was in one of our earlier podcasts and we discussed this very issue, I became more interested after we talked with Doug. A two-stage exam is one where in the first stage of the process, students take the exam by themselves and then in the second stage, they do some group work– either on a subset of the questions or on some very closely related questions. It’s being used quite a bit in the sciences and there’s a growing amount of research indicating that it has been successful. Some studies have found weak results, others are finding stronger results, but it’s still fairly early in the exploration of this. The Carl Wieman Science Education Initiative has quite a few resources associated with two-stage exams. This leverages peer instruction in the second stage.

The usual process, or the most common practice, is to take the exam period and have students work on this for the first two-thirds or so of the exam time slot and then they work in a group in the last third. I did it a little bit differently than this. In my case, I gave the exam on a Wednesday and I graded the exam but didn’t get them back to the students and then I selected a subset of the questions and I had them work on them in groups on that Friday… and that worked pretty well too, they had a little chance to review in between, they didn’t get to keep the exams, but there were only seven questions on it. They could go back and review things. I didn’t tell them which questions would be on the second stage in large part because I didn’t know. I told them that two of the questions would definitely be on it, but it would depend on how they did on the other part. So, I was able to look at the exam, find the parts where they had the most trouble, and assigned those as ones for the second stage… and in general, it was a remarkable experience. It was really nice to be giving an exam and to see students working in groups of three or four, actively discussing the issues, arguing over them, trying to explain things to each other and it was a really fun experience. It was very energizing to see that much effort being devoted to try to understand concepts that students had some difficulty with.

Rebecca: I remember seeing an image of your class being really actively engaged, really talking about the core class material that you shared during your test and I think the caption was: “This is during a test!”
[LAUGHTER]

John: Yes, I took a picture of it from my phone and I think I sent that to you during the exam because it was just so exciting to see that… and it was also a reminder for myself just how well this was working. I wandered around the room and listened in on the discussions and they were all very focused and coming up with much better explanations of these things then they would have likely been able to see if it was a whole class discussion… because they were very focused, they were arguing over what was the best approach to deal with some of these problems. I could see people making connections and suddenly understanding how things they had done before fit in and pulling together a lot of concepts that they might not have done as effectively if it had not been for those small group discussions.

Rebecca: Were you tempted to join in on those conversations because they were so lively?

John: I was, but I mostly just listened in and let them work it out themselves… and in general, they did quite a bit better… and what I should have mentioned before is that the overall grade for the exam is a weighted average of the first part and the second with most of the weight being on the individual part. One of the things that really appealed to me is that typically, when we give an exam and then grade it and return it, the students who did well generally just put it away and are happy with the results and they may glance at some of the things they got wrong (if they got many things wrong), but they’re not going to spend a lot of time actively processing it. The students who did poorly tend to get discouraged, some of them may give up a bit, but rarely are they likely to go back and try to put in the effort to correct their mistakes and to see where they went wrong. It was really nice to see that processing taking place by both groups. The students who did really well the first time deepen their understanding by explaining it to others and I suspect that should increase their long-term recall of this. The act of explaining it to others in some studies seems to be really helpful in encouraging transfer, where you can take concepts and apply them to other circumstances and when you’re in a course like econometrics, you have to be able to apply the same concepts in a wide variety of topics and areas. I think it was a very useful experience.

Rebecca: I think it’s a great method to allow some time and space for a reflective practice, because students tend not to do that on their own unless they’re asked to do it and if you do it as a homework assignment, I suspect that students don’t really spend that much time doing it, but this time they spent the whole class period doing the reflection. So, that seems really valuable.

John: Because I know a lot of people will do that. They’ll have an exam, they’ll give it back to students, and they’ll tell them they can make up part of the grade if they turn it in with corrections… and many students would do that, but I don’t think that would be as effective as having the group discussion on this. Some of them were able to make very clear what they didn’t understand and then they were able to get explanations from others and sometimes the explanations were right, sometimes they were wrong, but they had to process it much more actively and that’s always helpful, I think.

Rebecca: The grade weights is what seems most compelling to me in this situation because I’ve offered quizzes in my classes, more low-stakes assignments where I let students work on it for a while and I don’t tell them that they’re gonna get to do some peer instruction as part of it, but then they’re struggling with what they’re doing and then I say, “Oh, well, you have five minutes to work with your peers to revise anything you want to do before you turn it in.” And those generally result in some pretty active conversations as well, but there still are those few students who just copy down the answer and don’t engage in the conversation… but I think if there was that wait between before and after, that would really change that dynamic. So, I think that that’s a really compelling opportunity.

John: I thought it was useful and another reason why I didn’t do it all at one stage in one day is because I’m teaching on a Monday-Wednesday-Friday schedule and we only have 55 minutes and I have quite a few students in the class who are not native English speakers and they always take more time or they need more time to process and write information in a second language. So, I didn’t want to constrain the time and make both parts of it much shorter.

Rebecca: If you encourage people to practice and retrieve that information in extra time outside of class, there’s nothing wrong with that either.

John: Exactly.

Rebecca: I’d rather the students learn the material rather than just panic about a test. What do you recommend to our listeners to read to learn more about this evidence-based practice?

John: In terms of peer instruction, Derek Bruff has a really good book on using clickers. Eric Mazur’s original book on this, which is now slightly over 20 years old, is still very good… where he describes a process of developing this peer instruction technique. Eric Mazur also gave a talk here a few years ago and we have a recording of his presentation on this. There’s a really great example in there where he used peer instruction and what was most compelling about it, and Rebecca’s heard this before, but…

Rebecca: I was there!

John: …and Rebecca was there, was he used this example where he gave a really short presentation on what happens to the hole in a plate of metal if you heat it up… and people were asked to vote on that and then they had a chance to discuss it.

Rebecca: …and he never told us the answer!

John: …and then he noted how energized people were and he said, “You were so actively discussing these things…” When he tried to go on after making a point about how they suddenly were interested in something they normally wouldn’t have been interested in… he started to go on to the next topic. People were really upset, because they wanted the answer and he finally gave the answer, but he did that deliberately to show that this sort of thing… where the students don’t know the answer but they committed to a position and they want to know if they’re right… builds a sort of interest in learning that might not intrinsically be there otherwise.

…and that’s exactly what I saw, by the way, in my exam. They were so actively discussing things that normally they’d be bored out of their minds with. So, that environment can be very supportive of learning.

Rebecca: Yeah, it really gets people curious. I remember being in that room… dying to know what version was right? People had such compelling arguments.

[LAUGHTER]

John: Exactly, and that’s why it’s really good to pick questions, with any of these things, where it’s not going to be clearly obvious, where they have to process it, and they have to make connections, and you could build a case, correctly or wrongly, for different answers, and people want to know what the answers are.

Rebecca: I mean it was key that he finally gave the answer, right? So there was some corrective feedback there, so that people didn’t continue to mislearn the information.

John: And that was nearly four years ago, and we remember that very vividly. If that was just a point in a class that was given… say, four years ago, we probably wouldn’t be talking about that now.

Rebecca: I can’t believe it was that long ago.

John: I think it was.

Rebecca: It was a while ago.

John: Yeah.

Rebecca: …and so I’m dying to know, what are you gonna do next?

John: One of the next things I’m going to do is a follow-up to something we talked about in an earlier episode, when we talked to Judie Littlejohn about the metacognitive cafe. One of the things I’ve been observing is that the use of this process by having students work to improve their metacognition about how they learn and what they’re learning… Students, at least, perceive there is being some significant learning gains from that. That’s convinced me that I’d like to do something similar in a large class, but an online discussion forum for 400 students again doesn’t scale quite as well. So, I’m going to be doing some weekly activities and I’m working with Liz Dunne Schmitt who teaches our large macro class in the spring semester, and a couple of other people: Kris Munger, and Michelle Miller, who also who’s the author of Minds Online: Teaching Effectively with Technology (and was a guest here a while back). We’re going to try to put together an experiment where we use some evidence-based methods as weekly assignments, say for ten weeks in a semester…. that’s our current plan at least)… and students will be exposed to this… and they’ll engage in some sort of reflection or some practice with one of these activities… and then in terms of evidence-based methods of learning, such as retrieval practice, spaced practice, and interleaved practice, and similar things… and then we’re going to see how that exposure along with some reasonably easily assessed activity, which could be just some short responses in a forum or it could be perhaps some online quizzes, evaluating whether that impacts their actual behavior in the class, and their actual performance in the class. One-half of the group will be exposed to those types of interventions, and the other half will be exposed to some form of standard study skills module, because most of the students in this class of freshmen and basically what we’re looking at is, if we present students with evidence on what really increases our ability to learn, whether that will result in significant change in either their behavior, or in their performance. So, we’re going to try, at least the plan, is to try to see whether that affects the number of times they take quizzes that can be taken repeatedly, whether it affects the number of times they log in and view other materials, and whether it changes a perception of how we learn. so right now we’re at the…

Rebecca: And performance too, right?

John: …and their performance.

Rebecca: And is the plan to start collecting that data in the fall?

John: The plan is to put all this together the spring, I’m hoping and then to submit a proposal to the IRB, and then to conduct the study and the fall and the spring, at least for a first stage and then we’re hoping to be able to follow these students up, to see if this has a significant effect later in terms of their grades or their persistence.

Rebecca: Sounds pretty exciting. I’m looking forward to hearing how that goes.

John: It is. I’m looking forward to it being all together and actually being implemented. I think it’s an interesting study.

Rebecca: We’ll have to have you back, John.

[LAUGHTER]

John: I think we can manage that.

Rebecca: Well, thanks so much for sharing all this information about peer instruction. I know it’s something that I’m always kind of asking you about and like to hear about, and I’m sure others will too.

John: Well, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.

33. The Marmots of Finance

In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, Alex Butler, a Professor of Finance at Rice University, joins us to discuss how rich imagery can be used to help students make connections and deepen their understanding.

Alex received the Rice University Presidential Mentoring Award and the George R. Brown Award for Superior Teaching in 2018. He is also the recipient of the Jones School’s Award for Scholarship Excellence in 2011 and 2012.

Show Notes

  • Duke Talent Identification Program (TIP)
  • Medina, J. (2011). Brain rules: 12 principles for surviving and thriving at work, home, and school. ReadHowYouWant.com.
  • Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185-205). Cambridge, MA: MIT Press.
  • Alex’s web site at Rice

Transcript

Rebecca: In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, we’ll discuss how rich imagery can be used to help students make connections and deepen their understanding.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Alex Butler, a professor of finance at Rice University. Alex received the Jones School’s Award for Scholarship Excellence in 2011 and 2012. Alex teaches financial markets and corporate finance in the undergraduate MBA, MBA, and PhD programs. Sometime in the latter part of the last century, Alex and I spent three summers teaching introductory economics to highly gifted middle school and high school students at the Talent Identification Program at Duke University.
Welcome, Alex.

Alex: Oh, thank you. Thank you for having me, John.

Rebecca: Today’s teas are:

Alex: I am not drinking tea…. just water at this point.

Rebecca: Another one… epidemic.

John: My tea is ginger peach white tea.

Rebecca: I’ve Prince of Wales today.

John: Could you tell us a little bit about the courses that you normally teach?

Alex: Sure. I have, over the years, taught almost every course that we have… either here or at other schools. Right now what I teach (and I’m glad to finally have settled into a group of classes that are my classes year in and year out) and those classes are the undergraduate business finance class and PhD courses – one in corporate finance, and one that’s a topics class on causal inference. I really enjoy teaching the undergrad business finance class in particular because I was an undergraduate student here at Rice many, many years ago, and so it’s fun to be back and be on the other side of the podium teaching the students. One of the things that I really like about teaching the undergraduates here is that they are able to appreciate my dorky sense of humor.

Rebecca: …which is indeed a very important thing.

Alex: Agreed. For years, I tried to suppress my dorky sense of humor in the classroom… ad tried to teach the course sort of straight up and dry, and it made everybody miserable… the students…. me… and so, eventually over time, as I grew more and more confident teaching, I started incorporating more and more jokes here and there… and then I allowed them to become more elaborate, and then before you know it, I’ve developed a full-blown dad sense of humor and full-on dork mode.

John: That’s actually why we invited you here. We read a little bit about that in a teaching award you just received. What do students expect the course to be about when they take an introductory finance course?

Alex: At Rice, the students have a good grapevine of information about what courses are about and so, at this point, the students come in with pretty solid expectations that line up with what the class actually is… and whereas some students are hoping that I’m going to teach them how to become millionaires in the stock market, what the course mostly is about is that’s very hard to become a millionaire investing in the stock market and how to make decisions in a corporate setting that will maximize firm value.

Rebecca: So you mentioned that some students come in with this misperception of becoming a millionaire. What are some of the strategies that you use to dispel that myth?

Alex: I should say this class is one that is a fairly standard course that’s taught in lots of different business schools all over the country, all over the world. One way that I do it differently is in the order of material that I cover… and so the very first week of class, I talk about market efficiency. That’s the notion that it’s very difficult to earn abnormal returns in the stock market. In other words, you can’t beat the stock market very easily, unless it just happens to be by luck… and so I come in the first week talking about the reasons why it’s very difficult to beat the stock market… and the reason why is because there are literally tens of thousands of people who have more money and more resources and who are faster and are doing this as a full-time job who are also trying to find the stocks that are mispriced… and so unless you are the investor who is faster, and smarter, and has more money to throw at the trading strategy that you think of, it’s very very difficult to beat the other 10,000 people… and so I spend the first week of class introducing this concept and then providing copious amounts of evidence… research that highlights how difficult it is for lay people to earn abnormal returns in the stock market. That sets the setting for the rest of the course, which is this notion of how competition affects prices and how that feeds through to other applications and the corporate domain as opposed to the financial markets domain.

Rebecca: What made you switch the order?

Alex: What I realized after teaching this class for a long time is that most of corporate finance, most of business finance, is about discounting cash flows back to the present at some appropriate discount rate, and I found I was having a hard time getting the students to understand the notion of what interest rate, what discount rate, should be used to make these cash flows that are spread through time to get a present value equivalent. I figured if I started with some aspect of that, where that rate comes from, where those prices come from. but that would make the rest of the course easier for them to understand as we go through. So, that’s why. Most people wait until after they’ve introduced things like: “oh, portfolio theory” and “capital asset pricing model” and other asset pricing concepts before they talk about market efficiency… and so I just sort of turned that around backwards… and I open the course with that. So, point number one is pedagogical, and point number two is that the lectures that I do on market efficiency are really fun, and so I really like starting the course off with something that’s really fun. So, we can talk about stories of insider trading, and we can talk about stories of surprise announcements and how that affects stock returns, and we can talk about “oh, so you think you’re going to beat the market, well let me explain to you how hard it is and the reasons why …” also were very fun. So, we can spend the first week talking about fun stuff. It’s a giant bait and switch.

[LAUGHTER]

I lure them in with fun stuff and then beat them to death for the next fourteen weeks with discounted cash flow analysis.

Rebecca: It also sounds like it’s a good way to motivate students. Not only is it fun, but it gets students motivated and interested and they buy into the class, which I wouldn’t discount that. I think that’s an important task.

Alex: Oh, absolutely.

John: …discounting in a different sense, but…

[LAUGHTER]

Rebecca: Yeah, sorry… I’m a designer

John: …but it also starts a class by dispelling that myth that they’re going to learn tools that will allow them to become really wealthy in financial investments and so forth. So, you’re setting it up by getting rid of that myth and they’re ready to start actually learning without having that at the background.

Alex: Correct.

Rebecca: I really like the idea of just meeting students where they’re at. If this is where some students are coming with, and maybe they’re super motivated in some ways but not in others, that you just tackle both of those in one week.

Alex: That’s right.

Rebecca: Cool.
You recently won a 2018 George R. Brown Award for superior teaching, and the Georgia R. Brown Awards are based on an interesting selection process. Can you tell us a little bit about that?

Alex: Yeah, so what the university does is they solicit feedback from recent alumni, people who graduated, I think, it’s two and five years ago. Now, they have graduated and they’re looking back what teachers would they want to see when these awards… and so, it’s really a neat honor, because once the students are two, three, four. five years out, they’re not responding to the short-term incentives that some professors used to gain teaching evaluations and things like that, they’re looking back and they’re actually remembering what the course was, what the professor was like, and whether it was meaningful to them. It’s really been just a phenomenal honor, and one that I honestly thought that I would never win. So, it’s been really special to have gotten that award.

John: It’s a really interesting idea to have an award given in that way, because it focuses on that long-term learning, that you’ve taught them something that’s going to benefit them later rather than, as you said, something that people do short term. I know some faculty, when we’re doing course evaluations, will give out cookies or other things just to boost their short-term course evaluations. But, that’s not going to pay off very much two to five years after graduation.

Alex: That’s correct. I went through the cookies and brownies phase myself years ago and decided that that just wasn’t who I wanted to be, so…. Now, we’re on to just just the pedagogy at this point.

Rebecca: I think there’s a couple of interesting things that I see about those awards, too, and one is that is more meaningful probably to win it because you can feel the impact but two it meshes really well with what we know about evidence-based practices in that students generally don’t like them while they’re happening, but they have longer-term effects… and that students tend to learn that material and transfer that material later on.

Alex: That’s absolutely correct, and I’m a big subscriber to that basic view that students often don’t like things that make them uncomfortable, and learning new and difficult things is uncomfortable. I think one could make students happier in the short run by giving them lots of little assignments where they feel like they’re making progress every day but they’re not actually being challenged, they’re not actually being pushed, and so instead holding them to very high standards for demonstrating their mastery of material while that is very uncomfortable for some of them, it is that that makes them better students and better scholars of that topic.

John: It reminds me of Bjork and Bjork, in their writings on “desirable difficulties.” that the most learning occurs when students are faced with feasible challenges… that if things are too easy, they get bored, and they may be happy with the course if they don’t have to struggle much… but they learn the most when they’re struggling but they see it’s possible.

We’ve heard that this award, though, based on some of the write-ups we’ve seen at your institution, may somehow be tied to marmots, wolves, and The Princess Bride. What do they all have to do with finance?

Alex: Several years ago, I was reading some books I thought would be…. some I thought it’d be helpful for my teaching, some that I just thought would be helpful for me, and one of the books was a book called Brain Rules by John Medina, and it’s basically a book that tries to take cognitive science, brain science, down to a level that lay people can understand and gives several rules of thumb of how the brain works and why the brain works the way it does… and as I’m reading this, reading it mostly for my own consumption so that I can be a better researcher and more thoughtful person, smarter, that sort of thing, I realized “Gosh, a lot of the rules here applied very directly to teaching, at least in the lecture format that I use in most of my courses…” and so one of the things that really stood out to me is how people learn better, remember better, I guess I should say, when they see images images that relate to whatever the topic at hand is. So, text maybe a PowerPoint slide with text, and you remember X percent but if you see an image, you remember much more of that material later on… and so this gave me just a license to, all of a sudden, start having fun on a completely new dimension. Reading this book and sort of embracing the notion that I could maybe help students remember the material better simply by infusing my lecture slides with some relevant images, was just eye opening for me… because now I could take my completely dry, boring slides with words and numbers and equations and now I can have fun with them…. and have this entire new dimension, a degree of freedom, to play around with what the slides are gonna look like and how the students are going to experience them. So, that’s sort of the extensive margin. The first part is: “Hey, I need images…” so the intrinsic margin is “what kind of images do I need?” “what will work best?” and Brain Rules comes to the rescue there again… and it says people respond to images that are faces, that are things that are scary, things that they can eat, things that might want to eat them, and things that they might want to mate with. Well, that last one’s kind of out for most of my lecture slides… [LAUGHTER] I can’t really incorporate that very directly. But it got me thinking “okay, what’s scary?” So, I started looking around for images of things that are scary to people… and I found this great image of this really, just terrifying, snarling wolf. I’m like “Ah, I’m gonna use that to get people’s attention…” But, as I started thinking about it, I wanted to lead into the wolf a little bit… and so the main prey of wolves in North America are marmots… yellow-bellied marmots… and so I found this great image of a little cute yellow-bellied marmot sitting on a rock somewhere in the mountain somewhere… and now when I come in to teach what could be the absolute driest lecture of the entire course which is time value of money and understanding how to discount cash flows (it’s the tool that everything else builds on, so it’s incredibly important to get it right, but it’s also potentially incredibly technical and boring)… and so I start that lecture not with an equation not with numbers but with a giant image of a marmot filling the entire computer’s projection screen…. and I just leave it up there… and the class is all looking at it… and I look back at them… and I pick someone at random. I cold call… and I say: “So, Charlotte, what do you think?”

“What do you mean what do I think?”

“What is it?”

… and we go through a series of guesses, and the guesses range from just ludicrous things: “it’s a gopher.”

“No, no, clearly it’s not a gopher. Gophers are a lowland creature. This is obviously in the mountains.”

“It’s an otter.”

“No, no, no. Otters prefer marine habitats, and this is clearly not there.”

… and go through this for a while until usually somebody recognizes it as a marmot. I say “Yes, very good. Alright..”

…and so then on to the next slide and the next slide is this picture of George Soros with no caption, no explanation, but again filling the entire screen, here’s this giant picture of George Soros… and so I go back to the first person: “Charlotte what do you think about this one? You didn’t get the marmot, how about this one?” …and so invariably somebody will eventually guess it’s a hedge fund manager.

I say: “Yes, very good. That’s George Soros, a famous hedge fund manager.” then the next slide is the wolf, the snarling wolf, and so at this point the captions on the slides read “This is a yellow-bellied marmot,” “This is a hedge fund manager” … and then it’s obvious what the wolf is. People get that right away. So, Charlotte gets to redeem herself at that point.

[LAUGHTER]

“This is a wolf, one of the main predators of the yellow-bellied marmot,” and then the next slide is another picture of a marmot but this one looking somewhat quizzical and the caption here is: “Why do wolves eat marmots, but not hedge fund managers?” …and the answer that I propose is because hedge fund managers understand the time value of money, but marmots do not. Now, that obviously doesn’t actually follow, but the correlation is there… that part is true… and so I just sort of lean into that and the ridiculousness behind that statement and from there on the marmot becomes our time value of money mascot… and so every time throughout the course I introduced a new application of time value of money then I bring in another image of a marmot to sort of tie all that together. iIt’s hard to see that the tie that binds everything together is time value of money and so the marmot, the mascot, is the visual cue, that “oh, this isn’t special, this isn’t different, this is just another application of the same technique we’ve been doing over and over again.

John: That notion of using visual cues goes back to the Greeks who use it to remember long stories before there was much printed word, and one of the arguments is that it’s because visual imagery developed much earlier than the use of language and the things you described in terms of things that might eat you may be tied back to our evolutionary adaptation, and so we’re tying into things that evolve fairly early in the evolution of human beings.

Rebecca: …and it also is just that emotional response.

John: Right, when you trigger an emotional response and fear is a major one…

Rebecca: Yeah. mm-hmm

John: Do the wolf’s chase the Princess Bride? I don’t remember that…

Alex: Well no…. So, the Princess Bride lecture is where I talk about decision rules using discounted cash flow techniques…. and so this sort of standard playbook for any finance professor anywhere as you talk about net present value and internal rate of return and payback period and maybe a few other rules as well, you just sort of go through the… whatever they have in the book in the order in the book, and you talk about each one in turn. But, it’s a lot more fun if you can personify those.. and so that’s what I use the characters from Princess Bride to do… and so the main character, which is net present value, is personified as Wesley / (there’s a spoiler alert here – so you haven’t seen the movie Wesley is the Dread Pirate Roberts). I always show my kids my lecture slides and when I showed them that one my older daughter was mortified that I would ruin the Princess Bride movie for students by revealing that Wesley and the Dread Pirate Roberts are one and the same… and so from there, I personify the different rules based on the different characters in Princess Bride… and so internal rate of return which is sort of flashy and useful but has some flaws as Inigo Montoya; the payback period which is sort of a blunt instrument is Fezzak the Giant, and so forth… modified internal rate of return is the Cliffs of Insanity.

[LAUGHTER]

Rebecca: So, the important thing to ask then is do you have the spoiler alert at the beginning of your lecture.
AEX: I do now.

[LAUGHTER]

John: How old was your daughter at the time when she reacted to that?

Alex: About ten years old.

John: Okay. So, by the time students are in college, they’ve probably either seen it or they may be less likely to, so it probably doesn’t do quite as much damage.

Alex: I hope so, and in fact I even now encourage them to watch the movie before that lecture without really telling them why. So some of them do. Some of them ask around and figure out why, but that makes it more fun.

Rebecca: That was gonna be my follow-up… is that homework?

Alex: It’s the best homework they’ll ever have.

John: So, how have students responded to this? Do they remember this later?

Alex: Yes and no. for the Princess Bride lecture I believe that the students sort of uniformly enjoy that…. but I don’t have a good sense of how much that actually impacts the depth of their learning. For the marmots, it’s a completely mixed bag… some of the students really love it, and I really do lean hard on the fact that this is the mascot… this is the thing that ties it all together… and this is the visual cue so you will now recognize that when we do bond pricing “oh, here’s a marmot, that means it’s just time value of money…” “oh, when we do net present value, here’s a marmot showing up alongside our Princess Bride character it’s just time value of money” and so forth. Many of the students really enjoy that and grab on to it and some students, they’re not having any of it. They want boring… they want dry… and I’m afraid they’ve come to the wrong place. Been there and done that. I don’t do that anymore.

John: But that use of imagery is really common. People who work on developing memory… the memory palace type things where you tie specific concepts and bundles of concepts with chunks of item to key images, helps people remember things long term. I can see how it would be really effective.

Rebecca: I also could see that one of the things that students often struggle with when information is new to them is making those connections. So providing that visual cue like “here’s time to make a connection…” it’s actually really helpful, because those are the kinds of things that might seem really abstract and very separate if you’re not making an explicit. So, I like that you don’t even have to say explicitly like “this is the thing” you have by putting the image up there, and you’re prompting the students to predict what that connection is or challenging them to think of it on their own before you reveal what that connection is.

Alex: That’s right, and so what started off as just a fun way to get attention: “Hey, class is starting let’s all key in… Here’s a scary wolf.” So, now we’re all paying attention to the scary wolf because that’s how our brains are hard-wired. It’s now grown into this entire thing where throughout the entire class is a continuing callback.

Rebecca: It sounds to me like you must have some pressure every semester to have to have something new that you introduce into some sort of lecture so that there’s some anticipation.

Alex: I’ll be the first one to say that I shamelessly recycle all of my jokes…

[LAUGHTER]

… and so I’m always sort of terrified when I have a student who started the class last year but had to drop at the 6 week mark or the 8 week mark because whatever was going on their life, and then they’re back the next year. Like “oh man, you’re gonna get the exact same jokes with the exact same timing, the exact same patter.” It mostly seems to work fine, and I do continually try to incorporate new things. One of the nice things about teaching finance, although some of it is very static… the basic concept of time value of money is going to be there for basically forever and I won’t need to change the actual examples in those slides really ever… but a lot of the other material changes very rapidly. So, the notion of what does market efficiency mean? who can beat the market and win? what is the evidence? But for topics like market efficiency, for topics like financial markets, for topics like investment banking, those areas transform rapidly… and so I’m continually changing those lectures year after year because one of the big topics that I cover in my course that is not really traditional for an undergrad business finance class is how firms raise external capital. The reason why is because it’s one of my main research interests, and so I have lots of ideas of what I want the students to know, and lots of research that I can tie into the lectures that I give. For the past five or ten years financial markets, the regulation of financial markets, firms’ ability to raise external capital, has changed tremendously as regulation has changed and so I’m continually revamping that portion of the course basically every time that I’m breaking it out.

Rebecca: One thing that I can’t help but think is that you and John have some similar backgrounds in terms of content, so the time value of money that you keep referencing probably makes sense to you but I feel like our listeners might not know exactly what that is so maybe we should just take a minute and give a quick cap of that so people know.

Alex: Yeah, absolutely. So, if you have $100 and you invest it earning an interest rate of 5% per year, in a year how much money will you have?

Rebecca: One hundred and five?

Alex: One hundred and five dollars.

[LAUGHTER]

Rebecca: It’s not a trick question, right?

Alex: No, No, it’s not…

[LAUGHTER]
… and so, as you invest money it earns some rate of return and so then money in the future you have more, because it earns some positive rate of return, and conversely, would you rather have $100 today or $100 in a year? Well, the answer is $100 today, because you could invest that for that year and have the hundred dollars plus some additional return. So, you’d have a hundred and five dollars one year from now instead of the: “oh, I’m going to give you a hundred dollars a year from now” and that’s all the time value of money is.. and then it’s application of that over multiple cash flows and multiple periods where people start to lose track that it’s just math.

John: That’s the fun part.

Rebecca: Yeah. Yeah.

John: I also ask my students have they ever burned their mouth on a slice of pizza, and why don’t they just wait? ..and it’s one way of introducing the notion that we prefer things now to later. It’s a fairly important concept in economics and finance and it’s at the basis of finance.

Alex: Oh, absolutely… and I like the way of framing it as how patient you are. Are you willing to wait or are you impatient? …and so the way that I sometimes describe that in class is if you have a low discount rate that means that the future is worth about the same as the present and so you are patient. If you have a very high discount rate well then the future and the present are very different, so high discount rates → impatient, low discount rates → patient.

Rebecca: You realize that the irony of this whole thing is that I have these conversations with my husband all the time because he’s in finance, right? Yeah….

[LAUGHTER]

John: Ok, well, does that help explain it?

Rebecca: Yeah, yeah. I’ve had it explained to me many times.

[LAUGHTER]

I figured our listeners would need it.

Alex: Well, since we’re talking about the things that our students do, I’ll just share one example that I use in class that I continue to use even though it completely does not work with undergraduate students. It’s when I’m trying to introduce the concept of sunk costs… and so a sunk cost is some amount that you have paid… a cash flow that has happened in the past perhaps… and so once that is paid, you can’t get it back …and so it’s like the notion of “should he throw good money after bad” is another way of phrasing it… and so, what I used to say is, “Well, you go to a movie, you pay your fee to go in, and you decide it’s a terrible movie, should you keep watching to get your money’s worth or should you leave? and then a student pointed out to me one year that well, actually, if you go and you complain you can get your money back from the management.

[LAUGHTER]

Okay, different example. This is the one that never works on the undergrads. You go to an all-you-can-eat buffet and you have a choice of do you buy one plate for this amount or do you pay a little more to get the all-you-can-eat buffet and you decide to go for the all-you-can-eat. You pay the $10 for the all-you-can-eat instead of the $7 for the single plate… and you go and you fill up your plate and you eat… but you decide you’re full after just your first plate… but you’ve already paid that all-you-can-eat price. Should you go back for more? …and the intellectual scholarly answer is “Well, no, because the marginal benefit of eating more food is negative at this point because you’re full. Even though the marginal cost is zero.” But, for the students, the undergraduates, the marginal benefit of eating more food is always positive.

[LAUGHTER]

They view it as a sport. Dinner’s not over until they kick me out.

Rebecca: I think that’s really important to consider your audience and what works. So, I think that’s a really good demonstration of knowing your audience and why something might not work for a particular audience.

Alex: Yes, nonetheless, I still use that example… but I tell them ahead of time “I know this is not going to make sense to you. You’re going to push back on me. Nonetheless, I want to talk it through with you so that we can think about who has marginal benefits and marginal costs of what amount.”

John: Just last semester I used exactly the same thing of asking them how many of them would leave a movie theater if it was clear that they weren’t enjoying the movie and this time I had some people argue “I do that all the time.” So, yeah…

Rebecca: Movie theaters? Don’t you just have Netflix?

[LAUGHTER]

John: Okay, that’s actually a relevant point. In terms of the Princess Bride, we’re moving into a generation where many of your students might not have seen that. Do you have any other movie references that you might substitute in the future?

Alex: You know, I haven’t found the right one yet and, honestly, even if I found a good substitute, I’d really want to keep the Princess Bride, just to give them the incentive to watch that movie.

[LAUGHTER]

It is such a classic and such a gem. I really would like for them to all see that movie.

Rebecca: It’s an investment you’ve made.

[LAUGHTER]

Alex: That’s right.

Rebecca: How have faculty responded to the methods that you’ve been using?

Alex: Faculty are not surprised that I do the things that I do.

[LAUGHTER]

John: They’ve known you for a while.

Alex: They’ve known me for a while and I really lean into the corny pun bad jokes big time in class…. and I really I’m quite shameless about it.

John: That’s not new… I seem to remember that back at Duke when you were still in college.

Alex: …and it’s just a question of do I try to suppress that innate desire or not?

[LAUGHTER]

…and I’m at the stage in my career where “nope, not anymore.” You’re just gonna let it all out. My colleagues are not really surprised, but interestingly, one of my colleagues has adapted her lectures quite a bit to embracing this notion of adding images into the slide decks… and that’s my wife. She teaches business law and regulation of business. She was an attorney by training and I described her as a born-again economist because when she started teaching regulation of business she had to teach herself basically all of public choice economics. So, she got a couple of high-level textbooks and worked through them all but she and I regularly discuss teaching techniques and so she now has gone down the rabbit hole of finding that perfect image to highlight the point that you want to make… to have that really stretched metaphor that you can then call back throughout your course. So, it’s been a lot of fun to have her as a sounding board to go back and forth with.

Rebecca: I appreciate that you’ve brought design into the process.

Alex: Yeah.

John: Excellent. It’s something we all should probably do more of and think about more and certainly much more effective than those PowerPoint bulleted lists that are so common.

Alex: Yeah, a little bit of both actually goes a long way.

Rebecca: You’ve won some awards for your research. How do you maintain a balance between teaching and research?

Alex: It’s difficult. There are some ways in which research can feed directly into the teaching… and so my research that touches on market efficiency, some of that can come into the classroom. My research that touches on how firms raise external capital, some of that can come into the classroom. Now when I teach my PhD courses, those are heavily flavored by my research interests and preferences, but when I teach the undergraduate core finance class there’s not a whole lot that can flow back from that teaching into the research. So, that’s one of the downsides of teaching that class as opposed to perhaps a class that is more specialized or an elective or something that’s a little bit further downstream from the core class.

Rebecca: But at the same time, if it’s something that you teach routinely then the prep isn’t as difficult.

Alex: Absolutely. That’s true. It is hard to overstate how useful it is to teach a class a second time, or a third, or a fourth, or twentieth. Nonetheless, every single year, even though it’s the same class and mostly the same topics, I go through every slide, every lecture from the beginning every year.

Rebecca: But, I remember teaching a bigger selection of classes and one of the things that I’ve liked about my position at Oswego is that that suite of classes has gotten smaller… and then there’s a little less I’ll keep on top of to make sure that you have all that fresh information and what-have-you for classes.

Alex: Absolutely, and I’ve done the same thing. I’ve taught a variety of courses over the years and it’s been nice… as you describe it that suite of classes narrows, so that you have the same core group of classes that you’re teaching over and over again and you can start to specialize. You can really invest the time to get over the fixed costs of finding all those right images for the slides… to going through and taking time to invest in the design aspects of the lectures. That if you had four different courses every term, it would be incredibly difficult to find the time to do that effectively.

John: While you’re working with PhD students, you must do quite a bit of mentoring of them. How do you see the role of a faculty member as a mentor for graduate and undergraduate students?

Alex: Faculty vary widely on their views of how much mentoring PhD students should have… and so you have one model where it’s sink or swim… the PhD students are some of the smartest people in the world… they’re good students… they’ll figure stuff out… just point them in the right direction and let them go, and they’ll get there. That’s not the view that I subscribe to. Because I think we frequently overestimate just how much the doctoral students know, particularly about how the profession works. There’s no book for that. They can’t just go down to the library and find a textbook on how to be a good assistant professor or… there are books on how to write a dissertation but that only gets you so far, because it really needs to be very field specific. So, I tend to go very much the other way, which is a lot of sort of high-touch mentoring. I write co-authored papers with many PhD students, one of whom is now your colleague there at Oswego in the Finance Department in the School of Business, and it’s enjoyable for me. It is a good learning experience for the students and I think it helps them to learn how the profession works much more efficiently, because when it comes time to write a paper and they might put together some tables and say I want to structure the introduction this way. Oh, no, no, no, no, we can’t do that… because if you do that then it’ll make people be concerned about this issue here. So, instead, we need to twist it around this way and start with this… start with the big picture, not what your paper does but what your paper’s about. That sort of thing. That’s hard to learn on your own.
PhD students are PhD students because they are extraordinarily good students and they’re really good at learning. Though, that’s not the job for academics. The job is not the learning, the job is creating knowledge… and the transition from being a consumer to a producer of knowledge is scary, and it is the road that has very few signs or roadmaps to help them get down. It’s a transition of going from a consumer to a producer of knowledge… is very profound for a lot of people.

Rebecca: What’s interesting about what you’re hearing is you know my field the terminal degrees in an MFA a Master of Fine Arts, and the undergraduate degrees are really professional degrees. But, it’s the creators of cultural content ,and so that struggle happens at the undergraduate level too, of going from being that consumer of culture to a producer of culture. It’s really not that much different… just what they’re creating is a bit different.

Alex: That’s right.

John: I remember when I was working on my PhD, one time, where up until that point I had been meeting with my advisor every month to talk about my research, and at first I was just asking him questions… and then he was asking me questions and I realized suddenly that I knew more about the topic than he did… and that’s I think that sort of transition that’s sometimes difficult… because when you’re working on your research you’re mostly going out and finding all these earlier studies and so forth but you get to some point where suddenly you become the expert in the field and that’s a tough transition to make. It’s scary, as you said.

Alex: It is. It’s quite the watershed moment when you realize… when you are presenting your research to a room of 30 presumably learned scholars that maybe collectively they know more about the topic than you, but you know more than any individual person in that room… and becoming that expert and then owning it, so that you can write confidently is, I agree, a very tough transition

Rebecca: I like the emphasis on the owning it part. I think that’s key.

Alex: Yeah, imposter syndrome is real.
[LAUGHTER]

John: Everywhere.

Rebecca: So, we usually wrap up by asking “what are you gonna do next?”

Alex: Well, that’s a good question. I really wanna keep doing what I’m doing, making my class better year on year, teaching PhD students and training them year on year and working on research, hopefully research that people will actually find interesting and useful.
I have a sabbatical coming up.

Rebecca: When’s your sabbatical? What’s the countdown?

Alex: Well, the sabbatical is a year from now and in between then, one of my colleagues with whom I co-teach a doctoral seminar, he’s on his sabbatical. So that PhD course, that normally I teach half of, I’m now teaching the entirety of next fall. So, I think the first thing I’m gonna do is prep the rest of that class.

John: Excellent.

Rebecca: Deadlines make a difference.

Alex: Yes, indeed.

John: That procrastination thing… and that time preference…

Well, thank you!

Rebecca: Thank you so much.

Alex: Thanks.

[Music]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

27. Teaching big

You might think you have a heavy course load. Imagine being the instructor of record for approximately 5,000 students in a semester. In this episode, Dr. Kristina Mitchell, a faculty member and director of the online education program for the Political Science Department at Texas Tech, joins us again to discuss the design, organization, and management of high-enrollment online introductory political science courses.

Show Notes

  • Mitchell, Kristina M.W., and Robert Posteraro, MD. “Making Arrangements: Best Practice for Organizing Tools in Online Courses.” (manuscript)
  • Mitchell, Kristina M.W., and Whitney Ross Manzo (2018). “The Purpose & Perception of Learning Objectives.” Journal of Political Science Education (forthcoming)
  • Quality Enhancement Plan (QEP) at Texas Tech University
  • 2×2 Course Development Matrix
    Low engagment and low customization is like a MOOC style course. High engagement but low customization is akin to an Upper division course using ready-made content. Low engagement but high customization is what the introductory online political science courses at Texas Tech are like. High engagement and high customization would be an upper division course where the university writes their own content.

Transcript

John: You might think you have a heavy course load. Imagine being the instructor of record for approximately 5,000 students in a semester . In today’s episode, we’ll focus on one extreme teaching scenario.

Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Our guest today is Kristina Mitchell, a faculty member and director of the online education program for the Political Science Department at Texas Tech. At Texas Tech, she is the instructor of record for over five thousand students each semester. Welcome back, Kristina.

Kristina: Thank you.

Rebecca: Today our teas are…

John: Black raspberry green tea.

Rebecca: Kristina, what are you drinking?

Kristina: I’ve got my usual Diet Coke.

Rebecca: And I am drinking English Afternoon tea.

John: Could you tell us a little bit about this class of approximately 5000 students?

Kristina: Sure, we have two courses: we have Introduction to American government and then we have Introduction to Texas Government… both of which are required by our Higher Education Coordinating Board here in Texas. And it gives the students the opportunity to learn about our basic political system, and we throw in a little bit of political science theory, and it makes sure that our students leave their public university degree knowing something about how their government works and how they could participate in their government if they chose.

Rebecca: How is it 5,000 students large? [LAUGHTER]

Kristina: The State of Texas does require each of our students to take both of these before they can graduate with a public university degree, and with a large university like Texas Tech, we’re at about 35-36,000 students right now. What that means is that these massive freshman classes all need to get these two courses out of the way and it ends up being about 5,000 of our students every semester needing to take the two courses.

John: How do you manage a class so large… in terms of structure? What type of support do you have in terms of TA’s, co-instructors, or other assistants?

Kristina: We operate under what I call the “umbrella method.” So as an Economist (I’m a trained Economist), I definitely subscribe to David Ricardo’s idea about specializing and trading. So, rather than having each professor as an island where they’re handling every aspect of their 200 or 250 person course, instead we decided to specialize the roles that we needed, so that each person is handling one task for all of the students. So, I am at the little peak of the umbrella and I handle the course content. I make sure that we work with the publisher to get the students their materials and that it’s all delivered appropriately and I work with the various institutions on campus that need reporting data, assessment data, enrollment information. I have a co-instructor who handles the more day-to-day tasks of: students emailing… asking about course policies… asking for exceptions to course policies… questions about the content itself. He handles all of that for all the 5,000 students. We have two course assistants who just handle the mechanics of the course. So again, they get a lot of student emails as well. They deal with the settings in our learning management system… which we use Blackboard. So when a due date needs to be set or changed, they deal with that. And then we have TA’s, we supervise about 24 TA’s, each of them doing grading for the written work. It was really important for us that our students not only be doing multiple choice exams… we wanted to make sure that their they’re doing written work… because that’s, in my opinion, a better way to evaluate student learning. So each of the TA’s is grading one of the sections of the online course. So again, we’ve specialized out each task to make sure that one person isn’t required to do publisher and assessment and emails and content and grading for one section. It’s a lot more efficient, we think, to do it in a more specialized way.

John: Is it one large section or does it consist of multiple smaller sections?

Kristina: It’s multiple smaller sections. A lot of that is because we asked our students to do discussion questions with each other and it’s a little bit overwhelming to ask them to discuss with 2,500 other students. [Laughter] So, it ends up being usually 12 to 14 sections of each of the course. So 12 to 14 American Government, 12 to 14 Texas Government.

Rebecca: What’s the difference between how you teach your class and a MOOC?

Kristina: Yes. So, “MOOC” is definitely a dirty word these days. MOOCs really rose in popularity a few years ago. The idea was to create these just huge open online sections that would let anyone enroll and complete content as they pleased and either get credit or not, depending on how the course was set up. So we distinguish ourselves from a MOOC… first of all, because it’s not an open course. You have to be a student enrolled at the University to take this course, but also we don’t consider ourselves to be massive because we separate these courses into smaller sections, manageable chunks that our TA’s can grade. We also place a lot of value on the interactions that students have… so students interacting with each other (and often that’s done via discussion boards)…. students interacting with the content (and that’s done at their own pace asynchronously as they go through their course content)… and students interacting with the instructor… and that’s both to ask the instructor questions either on the course content or about the course mechanics… but also to provide feedback so that the instructor can take that information and then decide what to do with it. So, as we receive feedback, some of the feedback is, “your class is too hard” and we usually don’t really do much to address that… but if there are specific things: “the content is confusing, things are worded in different ways” then we can interact back with the content and the students to make sure that we are addressing these issues as they come along.

Rebecca: With such a complicated umbrella structure, how does the student know what person to communicate with?

Kristina: Well, sometimes they don’t, and we do have a really good system for navigating students to where they need to be… but we just try to include at the beginning of the semester both on their landing page (their home page when they enter in) and tell them, “Here’s the people that you can contact and here are the kinds of questions that they can help you with…” …and we also try to make sure to include a welcome video at the beginning that we ask students to watch… but give them an idea of how does this course work… who are you going to encounter… who you’re going to talk to… and what’s the most efficient way to get you through this course. I think a lot of times with these introductory mandatory courses, the students and the professor are at opposing goals. The student wants to get through this course as quickly as possible so they can check the box on their degree plan and move on. The professor wants the student to learn something about the discipline that they’ve devoted their lives to. So, I spent a lot of hours writing this content that matters to me a lot and and we tried to tailor the content to where we are. We’re in west Texas and when we talk about international trade, we talk about cotton because that’s what’s grown out here. So, I’m so passionate about the subject and if we aren’t aware that the student and the professor are at opposing goals, then it can lead to frustration for all of us. The students are frustrated because they think I’m asking too much and I’m frustrated because I don’t think they care enough.

So what we try to do is just make sure that we have a set of things that we want our students to accomplish, and some of those are content related: “Who’s your representative? Who are you going to contact if you have a problem?” …that’s something I want my students to know… how you go about registering to vote…. that’s something I want my students to know… political science concepts broadly about what influences judicial behavior… those are the things I want my students know. But I also value them learning things like meeting deadlines, and emailing professionally, and managing their own time, and that’s something that an online course is really uniquely situated to teach. So, we designed the course, knowing we want our students to learn some things. They may not want to engage with the content more than we require, so we give them a way to check the box… get it through their degree plan… while still making sure that they’re learning those concepts that are important to us.

John: What types of activities would students engage in each week in this course?

Kristina: It’s a very typical online course in the sense that the students are asked to do some readings and do some quizzes and write some short answer assignments. The one thing we’ve found to be most successful after doing this for 5 or 6 years… what we found to be most successful is chunking the content… and that’s a really popular phrase in online instructional design these days…. making the content into small manageable pieces. So I’m a millennial, I have to admit, and my attention span is about the attention span of a goldfish. So, I understand what it’s like to not want to watch a 25-minute video of someone lecturing to me. So, we really tried to take the concepts and distill them into manageable chunks. A 5-minute video talking about campaign financing rather than a 25-minute lecture… and at the end of the day, it facilitates that goal of getting the students to know what we need them to know without asking them to engage more than they’re willing to engage for a course that they just view as a checkbox.

Rebecca: I know that you wrote a paper on best practices for online courses. Can you summarize some of the key best practices that you’ve implemented in this class?

Kristina: Absolutely. The work that I’ve been doing on best practices… it really does stem from working with people who are trying to help us make our courses better. And I think a big problem in online education and instructional design is that it’s very new… and a lot of times when we think about what are the best practices, most of the evidence is just based on anecdote and experience. And so there’s something extremely useful about sitting down… I mean that’s what you’re doing with me right now… sitting down and listening to what I’ve done and what’s been successful for me…. and then, hopefully, listeners can take what they think will be helpful to them and apply it to their own experience. But I hesitate to say that just taking what has been a good experience for me, because it’s worked in my situation, could then be said to be a best practice for everyone else; that because our umbrella model is working really well for us, it is therefore the best practice. So what I’ve been doing with some co-authors is taking apart these ideas of best practices and empirically testing them to see whether they hold up to a quasi-experimental design.

The most recent work I have (that’s under review) looks at the organization of tools in a learning management system. So my co-author was teaching an online course and was told by his learning management system professionals that he wasn’t allowed to change the order of tools. So, the syllabus, the calendar, the grades– these sorts of things, he was supposed to leave them in a certain order because it was the best practice and he thought, “I don’t believe that… I don’t believe that that’s really a best practice…” So we tested it. We looked at when you manipulate the order, does it change the way students feel about their course? And does it change their performance? And we found it doesn’t and this is initial findings, of course, we do want to see these replicated before we were to say there is no best practice. But I think what most of my research is finding, when it comes to best practices in the order in which students experience content or the kinds of verbs we use when we describe what they’re going to do, it may not always be transferable across courses, across institutions so we need to be careful when we’re saying, “this is the best practice” as opposed to, “this is a practice that worked well for me and take what you will from that.”

John: Have you done any other research on best practices in terms of testing? With that large enrollment, you’ve got some nice possibilities of doing some randomized controlled experiments.

Kristina: Absolutely, and I think my IRB is getting tired of my requests to explore these differences in pedagogical practices. One paper that was recently accepted at the Journal of Political Science Education (and should be published this summer) looks at learning objectives. We’re often told as faculty members that we need to write learning objectives in a certain way. Bloom’s taxonomy, I’m sure you’ve heard of this, where we’re asked to use certain verbs to describe what we want our students to learn…. and my co-author, Whitney Manzo, who’s at Meredith College in Raleigh… we really wanted to dig into this. We argue that learning objectives are only useful to the extent that we all have a higher education community share an understanding of their purpose… of their definition… and to the extent that they help students learn better… learn more… perform better. And as we did interviews and surveys with students, faculty members, and assessment professionals, we found that there’s simply not a shared definition, and not a shared perception of why we have learning objectives…. and using learning objectives that are written and presented in a different way in a classroom didn’t seem to change the way the students perceive them or how they performed in the course. So, it’s not that my research is trying to say that, it doesn’t matter, it’s a free-for-all, you can do whatever you want. I think what my takeaway is… that we should be able to place a little bit more trust in our faculty members to know the nuances of their own classrooms… their own situations… their own institutions… and be able to listen to faculty members more in saying how they think something like Bloom’s taxonomy… or learning objectives.. or ordering of tools in an online course…. listening to faculty tell us what they think is the most useful in their situation.

Rebecca: In classes this large, cheating is likely to come up as a concern, I would imagine. A wide variety of web-based services have appeared to help facilitate academic dishonesty, including sites to facilitate plagiarism. What challenges have you had dealing with academic honesty and what tools have you implemented in your classes?

Kristina: Academic dishonesty is absolutely rampant and not only in an online course, I observe academic dishonesty in face-to-face courses as well. But sometimes, online courses just make it that much easier to get away with cheating. There are lots of tools that we’ve used, Turnitin.com and other plagiarism detection softwares. Sometimes I feel like your own eyes and experience as a faculty member are just as good, if not better, than these detection services because students have learned how to trick them. Students have learned that if you use a thesaurus and change the words, then the Turnitin won’t catch it.

John: And there’s even some websites that will do that automatically… where you can submit a paper and it will automatically change some of the words for you.

Kristina: Oh great… yeah, exactly. So it’s making it much easier to get away with cheating. Sometimes, if I just see a sentence that’s suspicious…with 5,000 students, I’ve seen the Wikipedia page on supply-side economics hundreds of time…. because now I’ve seen the words from it, either in those words or slightly altered in so many of my students writing. I think some level of academic dishonesty is inevitable whether we’re in a face-to-face course or not…. and in some ways, online courses make it easier for me to catch academic dishonesty. So in a written paper where a student hands me ten pages printed out from a printer, it’s a lot more effort for me to Google a sentence that looks suspicious than it would be if I just copied and pasted it out of the learning management system. So maybe it’s facilitated dishonesty, but it’s also made it easier for me to catch.

John: It’s a continuing arms race to some extent.

Kristina: [LAUGHTER] It is.

John: One thing I ran across a couple of years ago is that you can change individual letters, do a global search and replace with a different character set and that will defeat SafeAssign, Turnitin, and most of the other automatic detection services. But one of the clues there is that you end up with a plagiarism rating of zero percent, which is something you never see because of bibliography graphic references and similar things, it was interesting. I had one student do that in two different papers in two of my classes.

Kristina: Yes, the students are definitely ingenious in how they can come up with ways to cheat the system. Sometimes I think that if they would put as much work into their coursework as they did in trying to cheat, they would actually end up doing a lot better.

John: [LAUGHTER] Yes.

Rebecca: What steps have you taken to keep your graders and co-teachers and everybody in your big umbrella on the same page so that students have somewhat of a consistent experience?

Kristina: I think that this inter-rater reliability is always going to be a problem and it’s a problem no matter what. If we were to do this face to face and we had an instructor with a TA handling an individual section of 200 or 250 students and teaching it in a lecture hall, the content would be different… the expectations would be different… the tests would be different… the papers would be different… everything about the course would be different. So moving to this umbrella method has allowed us to ensure some consistency across sections. Now we still certainly allow faculty members who want to teach this course face-to-face to do so and it’s an academic freedom issue in that if they don’t want to use these materials, they certainly don’t have to. But it is nice for us to be able to generate some comparable assessment data that we can see are things changing over time and, if so, is that because the students are changing ?or is it because something about our course is changing? The ways that we try to ensure some consistency… what we can’t control… we do trainings with our graders… and we got this idea from the AP program where they essentially have retreats, where they sit around training these AP scorers on how to be consistent with these rubrics. So that’s one thing that we’ve done in trying to ensure that the graders are grading consistently. We also just monitor throughout. So, not only do I look at the general averages… are some graders scoring more or less strictly… but also spot-checking assignments and trying to do a systematic sample where we check every certain assignment to see if it’s consistent with what I would expect and what we train them to do at the beginning. And if we notice inconsistency it’s not punitive… we don’t threaten to fire TA’s who aren’t grading consistently… but it does give us an opportunity to say, “Hey, I think you’re grading these a little bit too harshly” or “a little bit too generously” or “I think you’re focusing on the wrong aspects of this assignment, you should be focusing on something different.”

Rebecca: One of the other things that you’ve presented on is online accessibility. We’re certainly working on that here. I’m working with faculty and other colleagues in other departments on some accessibility policies and things like that. How have you encouraged this to happen in your program and in your courses?

Kristina: It’s always difficult to tell faculty what they have to do. I don’t know if you’ve had much success going to a faculty member’s office and saying, “Guess what, you have to do this now” and them being like, “Great! I’m in, I want to do it.” I haven’t had a whole lot of success at that approach. Some of what I’ve been doing with my online course…. it’s kind of just accepting that this is how it is and this is what we have to do and so there’s no sense in trying to fight it. We’ve got to make our courses accessible to students with disabilities and there’s no reason to imagine that we wouldn’t want to do so. So we just make sure that all of our videos either have captions or transcripts. We make sure that all of our content that might not be easily accessible for a student with a disability to have an alternative method of accessing it if they need it. We try to avoid time limits on multiple choice questions because of students who might need extra time to complete their work. And so we just, in general, try to avoid the time limits altogether and make it equally accessible for all students, whether they have a disability or not. In terms of trying to encourage other faculty members in my department, who are either designing upper-division online courses or graduate-level online courses, making it less of a burden… providing resources and options for faculty… that’s been really successful. So if you just tell a faculty member you need to make your course accessible, that’s kind of an overwhelming request. There’s a lot to accessibility. So what we try to do is provide resources, ideas, support, ways to still do the same kind of content without running into accessibility problems. That’s been really successful… and taking just a more step-by-step approach. We know that the Americans with Disabilities Act requires everything to be compliant, but what we hope to always show is a good faith effort. So, whether we’ve achieved it perfectly every time or not, I guarantee if somebody combed through every course I’ve ever done, they’re gonna find something that’s not quite in compliance… but we’re making a good-faith effort and anytime a student brings to us an issue that they’re having, we work with the Student Disabilities Office to rectify it as quickly and as completely as possible.

John: With the umbrella framework for the course, it would seem that at least for these two courses, you have more control because you’re the one designing all the materials. So, you have at least some control over all the sections in terms of having the common materials compliant.

Kristina: Exactly, and it also helps that our Students with Disabilities Office, because of this umbrella model, knows exactly who they can call if there’s an issue that a student’s having. They know they can call me at any time, they know that I will always make a good-faith effort to keep things in compliance.

John: Do you teach any upper-level or graduate courses there?

Kristina: Yes I do, and others in our department also teach upper-division and graduate courses. What I envision when designing these courses is a political scientists favorite thing, which is a two-by-two matrix.

[LAUGHTER]

We love our two-by-two matrices and I’m happy to draw out this two-by-two matrix so that you guys can have it in your supplementary materials. But when I think about the purpose of the course and the customization of the course… so we can have a course that’s intended to be very in-depth and a deeper understanding and engagement with content (which we would want in a graduate-level course or upper division) versus something that, as I mentioned with our lower division courses., we’re seeking some basic understanding and some basic behaviors to help students learn what we want them to know, while they’re able to still satisfy those requirements. And so that’s one dimension. We can also think about the dimension of: “How customized do we want our content to be. So do we want to use something that’s sort of off-the-shelf or canned?” Or do we want to use something that we’ve written every word… we’ve written all of the quiz questions… all of the readings. And so I think every online class can kind of be placed somewhere in this 2×2 matrix. When we’re doing our introductory courses, we’re looking at maybe a lower level of student engagement but we still have a really high level of customization. So, as I mentioned, we’re in West Texas, I make sure we talk about cotton disputes when we talk about international agreements and international negotiations. In our graduate level courses, we move over to the to the high level of engagement, to the in-depth understanding… and then depending on what the faculty member wants, we can either do a fully customized version of a course where we write most or all of the content. Or we can use some existing resources… and there are publishing companies out there that have a lot of content that’s really good and that’s really ready to be used and it’s really highly interactive and engaging. So, we try to establish what’s the purpose of our course and what is our content that we want to create versus we want to use existing content… and then where does this course fit in. And after we’ve placed it in that matrix, then we can decide how to move forward in designing the course.

Rebecca: Do you work with instructional designers as you’re designing these courses or is that the function that you provide?

Kristina: It’s somewhere in the middle. So I’ve worked with our instructional designers here at Texas Tech and they are very supportive. They have a great way to make sure that they’re providing the level of help that you want and need. There are plenty of faculty members (and I’ve worked with them) who want to assign some readings and then have a final exam and call it an online course. And our instructional designers exist because we all know that that’s not going to work. SACSCOC is going to shut us down if that’s what our online courses look like. And so when I worked with instructional designers, usually once they see where we are in our course development… the fact that we’ve developed a lot of courses before… they provide us a high level overview… they give us some suggestions… compliance issues… and they let us go on our way. But when we have faculty members that have less experience, they are a lot more hands-on. Now I’m not sure they really liked all my learning objectives research… I don’t think they liked that research very much… but they’ve definitely been supportive and interested in hearing what it is that we’re doing here in Political Science.

John: Well they may not like it, but it could become part of their future dissemination that if you find significant results, it can help them improve all the courses there and more broadly.

Kristina: Absolutely.

John: Now you’ve received, I think, an internal grant for “engaging students in global governance and communication, hosting a lecture series and sponsoring in-depth, archival, and undergraduate research as a part of the university’s quality enhancement plan.” Could you tell us a little bit about that?

Kristina: Sure, so our quality enhancement plan, that’s QEP, it’s a part of SACSCOC program. The goal is to identify a way to enhance student experience and then dig in and make the student experience at our university better. So our QEP plan is about communicating in a global society and our Center for Global Communication offered a grant to programs that were willing to engage in something that would touch a lot of students and would encourage engagement with this global communication. So we have several levels of engagement that our students can participate in. In our introductory course, the students are asked to watch a lecture about communicating in a global society, about having relationships with those who are different than you — from a different country or background. And all of the students who take these introductory courses are exposed to that content, so that’s a great way that… even though maybe it’s in that lower level of engagement in the 2×2 matrix, it’s reaching a lot of students. But we also sponsor a lecture series and this is where we’ve brought in, typically political scientists, but generally social scientists have all been considered to speak as part of our lecture series. And so we’ve had people talk about terrorism and how social media can facilitate terrorist recruitment. We’ve had people talk about presidential travel as a form of global communication. So, when President Trump goes to Saudi Arabia first, as his first international visit, what kind of communication does that send to the world? This lecture series invites our students to come in and get a little bit more in-depth experience with global communication and that’s the way our online students can feel like they’re not just isolated at home… that they’re not participating in a university experience. We invite our online students and we usually get two to four hundred students at each of these events… so they’re wildly successful. In addition, after the lecture series, we’ve offered students an opportunity to participate in undergraduate research and we’re even able to sponsor two of our students to attend a national social science conference under the supervision of a faculty member to present their own research or at least be exposed to other research, related to global communication. As political scientists, we’re very well situated to expose students to the idea of global communication and as a large online program that can reach a lot of students, it also helps us to get the message out about the fact that this is an important part of their education.

Rebecca: Sounds great! It’s a really unique case study.

John: It seems like you’re reaching quite a few students who might not be reached by traditional educational programs.

Kristina: Yeah, I think that this kind of program might be new and different right now, but this is where education is going to be going, as we have the need to educate more and more students, college isn’t something that’s only for wealthy upper-class men and and women. It’s for everyone now… and we’re going to try to meet students where they are. Online education is where the future is and when we see students who are non-traditional trying to come back and get degrees… they have full-time jobs… they have families… they’re not able to go to class like a traditional 18- or 19-year old student is going to be able to attend classes during the day. I really think that this could be a potential equalizer for students. Not only in accessing content and getting degrees, but in learning some of these really valuable skills in how to interact in an increasingly online world.

John: And the rate of return to a college degree is the highest we’ve ever observed. We’re seeing not a lot of jobs out there and not a lot of job growth for people with high school degrees and high school dropouts. We need to do more to get more people to college and certainly the approach you’re taking there is a very efficient way of bringing education to a large number of people.

Rebecca: Well thanks so much for joining us Kristina, it was nice to have you back and it’s always nice to hear what you’re working on.

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

16. Student attention span

Have you ever been told that to keep students engaged you should chunk lectures into ten minute segments? Neil Bradbury, a Professor of Physiology and Biophysics at the School of Graduate and Postdoctoral studies at the Rosalind Franklin University of Science and Medicine, investigated the origins of this myth. In this episode, Neil joins us to discuss his review of the research on student attention spans.

Show Notes

  • Bain, K. (2011). What the best college teachers do. Harvard University Press.
  • Bradbury, N. A. (2016). Attention span during lectures: 8 seconds, 10 minutes, or more?. Adv Physiol Educ, 40, 509-513.

Transcript

John: Have you ever been told that to keep students engaged you should chunk lectures into ten minute segments? In this episode, we examine the origins of this myth about student attention.

Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist,

John: and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Our guest today is Neil Bradbury. Neil is a Professor of Physiology and Biophysics at the School of Graduate and Postdoctoral studies at the Rosalind Franklin University of Science and Medicine. Welcome, Neil.

Neil: Thank you. Nice to meet you.

John: Nice meeting you.

Rebecca: Yeah, thank you for joining us. Today our teas are

Neil: Today I’m going to go with Lapsang Souchong.

Rebecca: Sounds like a great choice.

John: Very nice, I got some of that, too. I have to keep it separate though because it has a smoky smell and it blends with the rest.

Neil: Yes it does have a strong, smoky smell.

Rebecca: And today I have vanilla chai.

John: I have Yorkshire Gold tea.
Your two thousand and sixteen paper on student attention span has gotten quite a bit of interest on a number of email lists and professional development groups and that’s where I first saw it. What prompted your interest in this topic? It’s a bit different than your usual research.

Neil: It is a little different. I have gone to various teaching institutional days and I had been told of this ten-minute rule, and naturally I accepted it because that was what the elite was telling me. And then shortly before I started looking into this, our medical school, like many medical schools in the country, was revising its curriculum entirely over the entire four years… and it was in a meeting with the Dean going through the new curriculum where he mentioned again, “I now have to stop because I’ve been talking for 10 minutes and so I need to do something different…. and I thought to myself “Well, that’s interesting, I wonder where this notion comes from?” …and I decided I was going to find out, because everyone told me ten minutes is the rule, ten minutes is the rule, but no one knew where it came from… and so I decided that I would have a look at this and if indeed it was the case I should be taking notice of this when I’m teaching… if it’s not the case, where did it come from? Where did this apparent educational myth arise? So that was really what started my interest in this… looking for where the source of this myth was.

John: I have the same sort of thing… I just took it as given, I’ve seen that said so many times and seen it in so many books and papers and recommendations, that I also took this as given. One of the things I’ve always used to judge the quality of someone’s presentation on teaching and learning is whether they start mentioning learning styles, or they put up a picture of Dale’s Cone of Learning, because those are myths that are pretty well known and pretty well established, but this one I think a lot of us had taken as a given…. there must be some research on… or we wouldn’t keep hearing it so much, and we should have known better. So what did you find when you began to investigate this? Where did this rule come from?

Neil: When I was looking through this, as you say, it’s often repeated, and people make the statements and then refer to a paper that was published many years ago… and when I eventually tracked down all the references, I did eventually come to a paper that was published in the fifties… and much to my surprise, the paper actually didn’t mention attention span at all in any of the words, which was a little curious since this is the basis that everyone uses for the ten-minute attention span. And what I found was that, in actual fact, the paper does not describe attention at all, but rather note taking…. and even more curious is that a subsequent publication by these authors even stated that they felt that note taking was no basis for discerning attention span anyway. So I think the whole propagation… and to be fair it’s the original authors, they did not say that this was looking at attention span… but somehow it’s got changed over the years… saying that they were looking at attention span of ten minutes, and that got propagated through the literature, and propagated by people who were looking not at the primary literature, but at someone else’s interpretation of the literature and it just propagates without anyone actually going back to the primary literature.

John: It’s nice that you did. It’s about time someone did and thank you for doing that.

Rebecca: I think it’s so important in the day we’re really focused on the idea of evidence-based teaching that we do remind ourselves that we should be looking at the evidence and not just taking things for granted.

John: And in your paper you also mention that there were some potential flaws in some of the early studies on note taking. What were the major flaws in some of that early work?

Neil: I think that some of the major flaws were really in experimental design. As a scientist, I spend a lot of effort pre-planning my experiments, often taking more time to plan the experiment than to do the experiment, and I found that a lot of these lacked really rigorous planning. So, for example, one study involved two lecturers that went in to observe a class, and in the paper, they stated that more often than not at least one person turned up to investigate attention. Well, if you’ve only got two people looking at student attention and only one of them occasionally turns up, it’s hard to take anything seriously that comes out of that study. So, a lot of things like that. One of the things that, as a scientist, I particularly have to take notice of is statistical rigor and we have to take care to provide instruction in what statistics we’re using and whether or not our statistics are valid. Many of the studies just state categorically, “our results were statistically significant” with no indication as to what those statistics were but then that gets repeated by the next person who just blithely states “oh, well this person states that it was significant” and just the takes that for granted, and so it gets propagated.

John: …and it might have been significant at the sixty percent level or something similar, if they don’t specify…

Neil: We don’t know because, it’s never mentioned in the papers.

John: …and one of the things you mention, in one of those early studies of note taking was that it may have been perhaps more of a measure, I think, of the content of the presentation rather than student attention.

Neil: Yes, I think it’s important when we’re considering lectures as to really fundamentally what the purpose of a lecture is, and I clearly don’t think that the purpose of a lecture is to have students take notes as the end result of a lecture. The notes should be something that the student refers to at the end of the lecture to remind them of what was covered. Now certainly note taking is important, but we don’t take notes on everything. So, for example, when I’m giving lectures I’m trying to convey a certain concept that may be difficult or it may be easy, but I try and give some illustrations of how that concept can be applied to real-life situations. Often, as it turns out, one of the things I like to do is discuss how my concepts can be applied to understanding how people use different drugs to murder people.

[Laughter]

Neil:

Now I don’t really expect students to take copious notes on how to murder someone they don’t like.

[Laughter]

Neil:So, that may be… the students would not take notes on, that’s just a little bit of fun and interest. The physiology underlying that? Then yes, that’s important to take notes on. So, you need to think about what’s being discussed. Is it important to take notes on that? Other things? No, it’s not important to take notes on. So I think there’s a balance there… that you can’t just take notes across the entire lecture, it’s really what’s being conveyed by the teacher at that particular time as to whether the notes are worth taking or not.

John: …and students could be very engaged, but not taking notes, because they are actively processing the information and making connections. And there may be no need for notes if that part of the presentation is very clear and doesn’t give them new information they need to transcribe somehow.

Neil: Absolutely agree… and just staring at a teacher or writing notes doesn’t necessarily mean that they’re focused on what they’re doing.

Rebecca: …especially if the content of the notes aren’t really being evaluated, right? It could be notes on anything!

Neil: Well, I think we’ve all done that at some point. I think we’re all guilty of that, but you’re absolutely right, just taking notes for the sake of taking note has no intrinsic value either.

John: …and in one of the studies, you mention that the students were keeping track of their level of attention where they were periodically polled and they had to indicate their level of engagement, but that was done with two different classes: one with first-year students and the other with fifth-year students, but they didn’t hold constant the instructor. So, the results were very different across instructors.

Neil: I think that that’s true broadly ,that we have all experienced instructors in our own life that have been really wonderful and engaging, and we really enjoy going to. We’ve also had instructors that we really did not want to go and listen to. So, I think there is certainly a large component of the instruction that makes a big difference in how they present. I think the other difference is that when comparing a first year with a fifth year is a huge difference. A first year really does not have any basis for knowledge, they have very little understanding of what’s going on, and so any new knowledge that they get is a huge increase. Compared to someone in the fifth year, where they may be really not learning new things, but they’re adding to what they already know, and it’s a lot easier to learn when you already know things.

John: You have more connections there.

Neil: Trying to fit things into a model that you already have to further strengthen that, that’s really not that hard to do, compared to learning a concept from the start. So, I think there’s a big difference there, between first and fifth years on a lot of levels.

Rebecca: Why do you think note taking was conflated with attention in the first place?

Neil: I think it’s conflated because it is really hard to assess attention, and we all know that when we go to lectures, we’re supposed to take notes. I think that was conflated because note taking is something obvious that we can see. We think of attention and people paying attention to what’s going on… that goes on inside the brain and you can’t really see what’s going on inside the brain; whereas you can see people writing notes. I think that was just used as a surrogate because it was something that could be measured, not necessarily because it was a valid measure of something that was going on.

Rebecca: What were your biggest takeaway is from the survey of literature that you did?

Neil: I think the biggest thing that I’ve taken away is that, as a teacher, I don’t need to worry about keeping things as ten minutes. I can focus on a concept. I was thinking about this as I went with my family to the latest Star Wars movie. I was thinking “What if I watch Star Wars only in ten-minute segments? and that those segments were not connected with each other?” I don’t think anybody would go and see that movie. So the ten minutes attention span really doesn’t hold up for that… no other experience in life do we have or do anything that’s only ten minutes. Why would it be any different from a lecture? And so I have freed myself from having to worry about going in ten-minute blocks, and I can focus more on providing a conceptual framework for what I’m trying to cover.

John: Students don’t have any trouble watching an hour and a half or two and a half or three hour movie or sitting at an engaging video game for hours at a time, and they often require lots of learning and so forth… and we don’t really worry about the amount of time it takes on those activities. When I’ve gone to talks, there have been some talks I’ve been at that, within thirty seconds, I’d like to bolt for the door (or fall asleep) and there’s been others where I’ve been fascinated for an hour or more at a time.

Neil: I think it’s the latter that we’re trying to go for… and it would be nice if a student suddenly looks at that watch and says “Oh I’ve been here for two hours …where’s the time gone? It’s been so engaging.” That’s what I, as an instructor, am really trying to aim for… is not to have a student worrying about what they’re going to be doing in the next two minutes but to realize Wow, the lecture is over. It’s been worthwhile. I’ve learned something and I’m not really bothered what the time is.”

John: That’s always great when that happens, I wish it happened more often, but it’s great when students want to stick around and find out more and they’re not ready to leave at the end of the class.

Rebecca: Sounds to me like a lot of faculty probably need to spend more time figuring out how to make their lectures more engaging and to captivate their audience, rather than simply trying to break things up into smaller pieces, right? So you might not take communication style or things like that as seriously, but perhaps something that we need to invest more time in.

Neil: I agree, I think one of the questions that I always try and ask myself is “Why is what I’m teaching important? Am I teaching it because it’s there in the textbook or am I teaching it because it really is important?” …and if it’s important, I should be able to describe why it’s important… why you need to learn this…. and it shouldn’t be abstract, it should have application to what the students are to be doing. You need to learn this because it means that you can understand what’s coming next. Since I’m teaching a lot of medical students, why it’s going to make an impact upon health care of the patients you are going to be seeing. If I’m talking to chemistry students, you need to know this is what’s important because it’s going to dictate how you design your chemistry experiments. So, no matter what you’re teaching, I think you have to come up with reasons for why what you’re teaching is important, convince yourself it’s important, and then try to convince the students it’s important.

John: That’s a point that Ken Bain made in his book What The Best College Teachers Do. He suggests that you should start with the key concepts in each class, explain to students why those are really important… you explain to them they need to be able to do these things to be able to answer those big important questions that matter to them in some way. You mentioned before that first-year students often don’t have as rich of a network of concepts, and as a result there’s perhaps a bit more cognitive load that they have to deal with. Might there be some advantage of breaking up a presentation though into small chunks and then having them actively engage with the material before moving on to the next concept — in terms of keeping the cognitive load manageable?

Neil: I think there is some merit to that. I think it should be dictated by the material, rather than a clock, and so we can look at things and see where are the boundaries that make a unit of knowledge a reasonable unit… whether you can cover that in six minutes or fifteen minutes, I don’t think that matters, but can you have a coherent unit that can stand on its own that you can put together with other things. So, content is more important than the time allotted to it.

Rebecca: I think what you said about content is really important and as a designer there’s a methodology called “content first,” and it makes sense in a classroom setting too… where you decide what the content is and then design around that, right? There’s some things are going to make more sense to do hands on, some things that are in going to make more sense to provide a lecture on… and if you let the content dictate what it is that you do, it makes a lot of sense.

Neil: …and certainly, when you’re thinking about how you’re going to organize a lecture, you have certain content that you need to cover. But the order of that content is really important. You need to start out with the foundations before you can go to the building, and to start with those foundations, that may be small chunks. I agree that can be time limited, and that you could work on, and then as you getting more knowledge more content, you start building that up into a logical, coherent molecule. But, you’ve got to start with the basics first and then you can build onto larger structures.

Rebecca: It gets funny that, as experts in a particular topic, we forget that there’s building blocks and foundations because our mental models are so much more complex. So, I think although sometimes these things seem obvious, we need the reminders to take a step back and remember what it’s like to be a novice at something.

Neil: I agree. Most of my research is focused on cystic fibrosis and I’ve been doing that for many years, but I realize when I’m teaching that to students, they don’t have the decades of experience that I have. They are not going to become experts in an hour. What I’m trying to convey to them is the broad concepts and if they get those broad concepts, they don’t need to know the minutiae that intrigues me on a daily basis. But, they have to have the broader picture and so I have to put myself in a student’s position and understand from their position what they need to learn and even what they’re capable of learning at an early stage

John: Now, have you shared these results with your colleagues and how have they responded? Has this affected how they’re teaching?

Neil: As you might expect, it’s been somewhat of a mixed reaction. Some have really liked it, some have not really been that embracing of it. I remember we had a speaker from another institution who were discussing their new curriculum and they had designed their entire curriculum around 10 minutes… and I had the temerity to ask why that was the case and the response I got was “Well, everybody knows the attention span of students is only 10 minutes.” …and so… it happens, and it’s propagated, and I think most people really appreciate the fact that there really isn’t a lot of basis for this 10 minutes, but they don’t know that. They’ve heard and been told over and over again… it’s ten minutes. It turns out that’s not the case, so we don’t worry about, this is not something that you need to worry about. Plenty of other things to worry about when you’re teaching, but this is not one of them… So, don’t worry.

John: About ten years ago here, we had a guest speaker, whose name I won’t mention, but he gave a brilliant hour and fifteen minute talk on how a lecture is ineffective…. and later at a reception I went up to him and said “that was an incredibly good lecture… it was really engaging and dynamic and everyone seemed really interested,” and he did appreciate a little bit, I think, the irony of that comment.

Neil: Yes, I agree. I think the question is, “What is the point of a lecture?’ And I don’t think a lecture is really the place for student learning. That’s not where students learn. The point of a lecture is to convey information the students can learn later, but I think there also is an important point of inspiration. It should give students an understanding of why this is important and an appreciation for students to think “this is exciting, this is really fun stuff to learn.” If I can help students make their own decision that this is fun to learn, then I don’t have to worry about making it fun to learn. They’ve decided it’s fun and they’re going to invest effort in learning it themselves. And so part of my role as an instructor and a lecturer, is to get the students to appreciate something, that I think is true, is that learning is a lot of fun… it’s a fun thing to do and that it doesn’t matter what I’m learning… whether it’s the material I’m interested in or something completely different… it’s fun to learn… and if we can convey to students it’s fun to learn, they’ll be more than happy to learn things no matter what the instructor does.

Rebecca: Such a great point.

John: It is, and I also feel the same way about faculty. That we all got into this because we were among those students who found it fun to learn, but sometimes people forget that once they’ve been teaching for a while and it would be nice if we could also encourage each other to share that enthusiasm for learning.

Neil: Yeah, I think it’s important for faculty to go and visit each other’s lectures… how we can learn from each other. I’ve been working on my lecture style for many years, I doubt very much whether is the best lecture style going. It seems to be appreciated by the students, but I can always learn and improve. I only see it from one perspective, having faculty and colleagues come to my lectures, me going to their lectures can be of huge benefit in improving everybody’s teaching.

Rebecca: Is there something specific that you’re working on right now as a lecturer to improve?

Neil: With the new curriculum that we’re bringing in, there has been a large change in how we teach, and so that’s reflecting a lot on the content that we teach, and so I’m trying to come up with ways that integrate a lot of material across a lot of different disciplines. Which is proving to be a little bit of a challenge. As a physiologist, I’ve been teaching physiology. Trying to bring in other disciplines into the classes is proving to be a little bit of a challenge, but it’s also exciting and has a lot of opportunity that I can bring a lot of different aspects in. And I think that I’m going to learn a lot and hopefully that will be conveyed to the students… that we can all learn together something that we may not have covered before.

Rebecca: Sounds exciting but also very challenging.

Neil: That’s why we’re educators, we like challenge and those go together as educators.

Rebecca: Do you think after studying attention span in this way that it’s worth more study and to have others investigate attention or is it something that we’re spending too much time thinking about?

Neil: I think there is some basis for looking at other aspects. Most of what we’re focused on so far obviously is the lecture, but as we know now, the lecture is not the only teaching modality that we use. We have small groups, we also have lab practicals, we have discussions, and so far this attention span has only really focused on lectures. I think it would be informative to also look at other ways in which we teach, look at small group learning, look at peer learning, look at practical learning. I think that is going to be an interesting avenue to explore… is what is the attention span, by whatever criteria the define attention. Is that different from those modalities than a lecture? Is it the same? Is it different for each modality that we look at?

John: …and might it also vary by the topic you’re looking at? That some topics for many students would just be more interesting than others and that, I would assume, would vary quite a bit across students as well.

Neil: I think you’re right. Not everything that we cover is exciting and interesting. There are some things we just have to do because you need that knowledge, not because it’s exciting. But, that it allows you to get the exciting parts. So again coming back to knowing fundamentals and getting that basic groundwork that allows you to do the fun stuff later.

John: …and explaining to students why they need to know those basic, less fun, things helps provide them with a bit more motivation to work through it because they see where it’s going and how it’s connected.

Neil: Yes, you always have to have an endpoint… why this is important. It’s important because we’ve got to get to this position and this provides a pathway to get there, and once we’ve got there, a whole bunch of things will open up to you that you didn’t even realize.

Rebecca: Are you planning to do any more work on attention span or are you attention spanned-out?

Neil: Well I was thinking about it, but I got bored.

[LAUGHTER]

Neil:I’m really interested in looking at these different teaching modalities to see where that applies to different avenues, because we’ve only focused really on the lecture and that certainly has been a dominant component of education at the institution. We’re moving away from that… lectures are still important, but we’re also incorporating small group learning… peer learning…. and I think it’s going to be instructive to discover whether or not attention is really critical there. How can we get students, when they’re doing peer learning, to take this into account to make sure no one’s just falling off the edge and not learning anything. So I think it’s going to be exciting to increase our understanding of how students learn, how they’re attentive, how they’re focused on what they’re doing.

John: In your review of the literature, did you think of any good ways of addressing the question of attention span? Or is it, by its nature, impossible to measure well?

Neil: I think it’s a really difficult thing to measure because it is something that is going on inside people’s brains and that’s always a hard thing to measure. Certainly, we can put people into CAT machines and MRIs. But that’s probably not a good learning environment for anything. So I think it’s nebulous by its very nature. I think the important point is: “Are the students learning anything.” I don’t think necessarily what we’re covering in the lecture is the be all and end all. Some of the experiments that were performed, that I discussed in my paper, were evaluations of what students learned that were taken immediately after the lecture. But as I pointed out, no one ever does an examination immediately after the lecture… and so those kind of studies are really, to my mind, fairly meaningless. The question is, downstream a couple of weeks later, when we examine the students on the content of that material, have they learnt it then? And, I think, that’s when we really get to assess whether they were paying attention… not by looking at whether a student is taking notes during a particular lecture, not by asking them questions immediately after the lecture, but whether they’ve really spent time going over that material, and again, and again so that they can adequately answer questions two or three weeks later when we examine them on the material.

Rebecca: So it almost might be whether or not they’re engaged and motivated enough to want to continue pursuing that information, so that they can pass those exams and things a couple weeks later… and not really whether or not they’re paying attention in the moment that it’s introduced.

John: …and that ties back to the inspirational role of lectures that you suggested earlier.

Neil: Yes, we should inspire students to want to learn. We can never just force feed students information… it’s just gonna bounce back. We have to inspire students to want to learn for themselves, and that’s what effective teachers do. They don’t teach, they get the students to learn themselves because they’re excited about learning.

John: Very good. Well thank you, this was fascinating.

Rebecca: Yeah, thank you so much for spending some time with us today.

Neil: Well, I enjoyed it and thank you for this opportunity to discuss the paper, I really enjoyed it.

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

11. Mobile technology in the classroom

Smartphones, laptops and tablets can be useful learning tools in the classroom; they can also be a source of distraction. In this episode, we discuss alternative policies that faculty and students might adopt to facilitate learning. Recent research on the relative effectiveness of handwritten vs. digital note taking is also examined.

Show Notes

Coming Soon!

  • Aguilar-Roca, N. M., Williams, A. E., & O’Dowd, D. K. (2012). The impact of laptop-free zones on student performance and attitudes in large lectures. Computers & Education, 59(4), 1300-1308.
  • Artz, Benjamin and Johnson, Marianne and Robson, Denise and Taengnoi, Sarinda, Note-Taking in the Digital Age: Evidence from Classroom Random Control Trials (September 13, 2017).
  • Bain, Ken (2011). What the best college teachers do. Harvard University Press.
  • Bui, D. C., Myerson, J., & Hale, S. (2013). Note-taking with computers: Exploring alternative strategies for improved recall. Journal of Educational Psychology, 105(2), 299.
  • Carter, S. P., Greenberg, K., & Walker, M. S. (2017). The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy. Economics of Education Review, 56, 118-132.
  • Brooks, D. Christopher and Pomerantz, Jeffrey (2017). ECAR Study of Undergraduate Students and Information Technology, 2017.
  • Hembrooke, H., & Gay, G. (2003). The laptop and the lecture: The effects of multitasking in learning environments. Journal of computing in higher education, 15(1), 46-64.
  • Lang, James M. “The Distracted Classroom.” The Chronicle of Higher Education. March 13, 2017.
  • Miller, Michelle (2017). Addiction, Accommodation, and Better Solutions to the Laptop Problem. Dec. 8, 2017 blog post.
  • Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological science, 25(6), 1159-1168.
  • Patterson, R. W., & Patterson, R. M. (2017). Computers and productivity: Evidence from laptop use in the college classroom. Economics of Education Review, 57, 66-79.
  • Ravizza, S. M., Uitvlugt, M. G., & Fenn, K. M. (2017). Logged in and zoned out: How laptop internet use relates to classroom learning. Psychological science, 28(2), 171-180.
  • Sana, F., Weston, T., & Cepeda, N. J. (2013). Laptop multitasking hinders classroom learning for both users and nearby peers. Computers & Education, 62, 24-31.

Transcript

John: Today in Oswego, it is approximately six degrees and we’ve had about four feet of snow in the last couple of days. We’re going to talk about a topic that comes up often in many of our workshops at the Center for Excellence in Learning and Teaching here, which is student mobile device use in the classroom.

Rebecca: Today our teas are:

John: Blueberry green tea

Rebecca: …. and Jasmine green.

Rebecca: So, in 2017 an ECAR Study of Undergraduate Students and Information Technology report came out and indicated that laptop ownership is up to 95 percent amongst undergraduate students and smartphones device use is up to 97 percent. I’m sure Chromebooks and other inexpensive devices have certainly increased the amount of technology… but they’re everywhere… and I definitely feel that in my classroom I don’t think I’ve had a semester yet where a student doesn’t have a smartphone or a laptop with them.

John: Another thing that the study noted is that tablet use is actually declining a little bit over the last couple of years. Still, though, about 50 percent of students that have a tablet computer as well… and most students report owning multiple mobile devices. I’ve noticed the same thing, that over the last three or four years more and more of my students in my large intro class (with about 360 or 400 students) are coming with computers. I’d say probably about 60 to 70% of them show up in class with them now.

Rebecca: Do you have any sort of policy on using these sorts of devices in your classroom?

John: I actually do encourage the use of them for specific purposes. One of the main concerns that faculty express is that computers provide a very convenient source of distractions, and I do observe that…. because I wander around the classroom quite a bit and it’s not uncommon to see students doing things other than what we’re working on in class. So that’s an issue, I think, which is at the heart of most people’s concern.

Rebecca: How do you discourage that distraction in such a large class?

John: I remind them that their time in class is limited and that it perhaps might be more effective for them if they were to use them for class related activities, which could include note-taking… or it also includes frequent use of polling. I use i>clickers in class and many of the students, probably a third to a half, are now doing the polling using either a laptop or their smartphones. What’s your policy on computers in class?

Rebecca: I teach web design courses, so obviously having multiple devices on any given student is convenient because we do need to test our work on multiple devices. So we certainly use them, in that respect. When we’re going over new material, I encourage students that want to use their devices to sit in the periphery of the room, so that they don’t necessarily distract other people who aren’t using devices… and then a lot of times we’re doing things that require diagramming and what-have-you… and so it’s just more convenient to take notes… not using a computer… unless they have a device where they can draw really easily, maybe a tablet that has a drawing application or something on it. So that’s usually my policy and most students are fine with that… and I also include in my syllabus that if you are distracting people around you then I might ask you to put it away… which I generally don’t have to do… but I have had to occasionally.

John: One of the concerns is that there’s a number of studies out there that find that when students are using computers in class, in a manner unrelated to the class, it not only harms their learning… but it also harms the learning of those around them. One of the studies on that was done by Sana, Weston, and Cepeda in a 2013 article in Computers & Education and there have been other studies that have found the same sort of results. So I use it, actually, as a learning experience because we talk about externalities in my introductory economics class, where externalities involve side effects of your actions that either provide benefits or harm to others…. and I know that when you have something that’s distracting others around you, that’s going to harm their learning as well… and perhaps we should discourage such negative externalities.

Rebecca: I think one of the things that also comes up is… distraction is one thing…. so if you’re distracted, how do you change from being distracted to having focused attention, right? ….and so… obviously the polling that you do in your classes is a great way to get students to refocus and regain attention. Are there other things that you do in your other classes that aren’t so large?

John: In my econometrics class, I will often have students work on problems in class where I’ll give them them a problem… they’ll pull up their regression software… run some results… and then we’ll talk about it in class… and generally there’s much less of an issue in upper-level classes, because the students tend to have more intrinsic motivation. They tend to be more focused on the topic in those classes as well.

Rebecca: I’d say my one class is an upper-level class but it’s still an introductory class. It’s an introduction to web design and I have a wide variety of students from first year students to graduate students that are all in the same room at the same time, but they’re all beginners and it’s an elective, so there’s a tendency to want to be there in the first place. They have some sort of motivation for wanting to learn that, so I don’t feel those same kinds of strains about focused attention that other faculty do in a general education class. But there is a lot of information out there, or strategies that you can use to start thinking about how those general education classes connect to individual majors, right? So the more you can help students find how it’s relevant to them, the more likely they will be to engage in the topic. So I’d recommend for faculty to really think about that at the start of the semester, and make that a good sell at the beginning… and connect back to that over and over again throughout the semester, so that students recognize that the general education courses really do matter to whatever it is that they’re gonna do as a professional.

John: …. and I do the same sort of thing the first day of class. I’ll generally ask my students why they’re taking this course and I’ll take responses from them sometimes I’ll do it on a poll and the results will come up… other times I’ll just have them say why they’re there and very quickly someone will say “Well, I’m here because I have to be here.” I said “ Well, why do you have to be here?” and they’ll respond “It’s required for my major.” So then the next logical question is “Why is it required for your major? What about this course is it that people think is so essential for your major that you all have to take it?” … and then that helps them think about why perhaps the things they’re learning here are relevant for their future careers…. and then I’ll tie it to that a little bit. I’ll ask them “So, what are you going to learn in this class that will help you in your future jobs that come out of this major?” and I remind them of that periodically as well… and try to make connections between what we’re doing and what their educational and career aspirations are.

Rebecca: ….This is really related to the idea of the “promising syllabus” that both Ken Bain and James Lang talk and write about quite a bit, where you’re really defining what are the big problems that the class is going to address…. and coming back to them over and over again… and that might be a strategy to really provide that focused attention.

John: mm-hmm. So, more broadly, in terms of policies, there’s three broad sets of policies that people seem to be using: one is to completely ban all mobile device use in the classroom (and we have quite a few faculty here that choose to do that), others have more of a laissez-faire policy, where students are encouraged to use computers productively… and that’s the approach I’ve generally tried to use. and others where mobile technology is is required for specific tasks… and that’s also true in my classes to a large extent.

Rebecca: Although that can be challenging, if you happen to have equity issues, right, at your university? ….and students, that three or five percent that don’t have devices, you need a way for those students to have devices.

John: Well, actually, I think the three to five percent would be with any one device that they don’t have. Virtually all students have at least one device. It’s been many years since I’ve had a student who didn’t have either a laptop or a smartphone. Most of the students come with at least one, and many of them come with both. When I teach in a summer program at Duke, for example, there are some students who may not have it. I generally bring some spare fifty-dollar Android tablets and pass them out if they forgot to bring their device. It’s not that they don’t have it, but sometimes they forget to bring it with them…. And you can do lots of activities where people share their device, as well. So the equity issue is much less of an issue than it would have been a decade or more ago. It may be in some other institutions. In lower-income areas or in some community colleges, device ownership is a bit lower, but we don’t really have much of that issue here.

Rebecca: I usually discover that the first day of class when I do a survey of my students to see what devices they have and then knowing that information, I can design activities and things that take advantage of those devices or avoid using certain devices if students really don’t have them. But I’ve also noticed, in keeping with the recent study, that my students have had more and more of those devices available to them in every class period. People who decide to have a ban on technology in the classroom… if they phrase it like that in the syllabus, and talk about that, I sometimes fear that that just sets up like rules and regulations sort of philosophy for the classroom. That sometimes doesn’t set up a classroom environment or climate that is welcoming and open. So I think if you want to discourage that use then you have to be really careful about the language that you use around that… so it doesn’t feel like a penalizing system.

John: One strategy that some faculty have used is…. you could invite students perhaps to refrain from distractive use of technology, because it could be in their own interest.

Rebecca: …and explaining that, and explaining why you might want them to put it away, I think, it can be helpful. So, rather than having a strict policy, maybe having those minor discussions here and there about when it becomes distracting, or not useful, it could be helpful.

John: … and they’re going to be living in a world where mobile technology is not going away. They’re going to be in a workplace using mobile technology and perhaps learning how to use it more productively might be useful.
One of the reasons faculty give for banning devices is that there have been some studies suggesting that students who take notes by hand perform better, or recall more information, than students who take notes on computers. One of the problems, though, with those studies (and there have been many of them) is that generally it was based on situations where students self selected, in terms of whether to take notes on computers or to take notes by hand… and those studies generally didn’t control for self selection. The most common explanation of the finding that note-taking by hand was more effective than note-taking by computers was that when students take notes by hand they don’t write as much… so when they looked at notes taken by hand and by computers, students who took notes by hand tended to write much more condensed notes, and the argument is there’s more cognitive processing going on in the note-taking… because they have to think about what the key concepts are and the additional learning is really coming from that part. The people who took notes by computers generally type much more and it appeared when you looked at the number of words typed, for example, that they were attempting to transcribe everything that was being said… or everything that was projected onto a screen (for people who were using PowerPoints, or Prezis, or Google slides, or Beamer).

Rebecca: That was one of the issues that I had with some of the studies… Specifically was that the way that they were measuring whether or not the notes were effective was whether or not they had everything that was provided copied down basically verbatim plus some as the highest score… and I would argue that that’s not very good note-taking in the first place.

John: Well, I think most of them… they actually focused on performance on some type of tests. In terms of recall, some of them were controlled experiments where there was an experimental lecture provided, and then students were tested on it immediately after, or a day or two later. But the analysis seemed to suggest that simply transcribing everything that was said is not very helpful, as you suggested there.

Rebecca: Yeah, and we know from evidence-based practices, that simply copying down, without doing any cognitive processing about what is important or what’s not important about that content…. and how does that content connect to other things, isn’t very useful. …and just reviewing your notes, in general, is also not a useful studying technique. So, if you’re only looking at those kinds of features, you’re not necessarily observing whether or not the laptop or device is inhibiting learning.

John: So, the argument for banning it is that it will reduce the number of distractions and it will force students to take notes perhaps in a more effective manner. The only problem with that logic, from my perspective, is that I remember when I was a student… I didn’t have a laptop or a mobile device, and I was perfectly able to distract myself when I found a class to be boring. There was always a notebook that I brought (I actually had only one notebook throughout my undergraduate career, I was not very good at taking notes back then) and I used it to play hangman, to write down notes to make plans for going out after class, and other things with friends around me, and I really didn’t need mobile technology to provide distractions.

Rebecca: The equivalent of texting, right? You’re still communicating with your friends by a notebook instead of through mobile devices, right? I had perfectly colored biology diagrams, I recall.
JOHN… hence, the art major thing…
….but one thing that these studies did not do is…. they found these very strong results… but they didn’t control for self selection. They observed that students who chose to take notes on computers did less well in remembering things than students who chose to take notes by hand.
There was an interesting study, though, released on September 13, 2017, by Artz, Johnson, Robson and Taengnoi (not sure of the pronunciation on those) in which there was a randomized controlled experiment conducted in, I believe, five economics classes… and they sorted students by their ID numbers. Students who had odd numbered student ID numbers were placed in one group, students with even numbers were placed in the other. A guest instructor was brought in who provided a very structured presentation in each of the classes. Half of the students, roughly, were required in the first sequence to take notes by computer and the other half were required to take notes by hand…. and then they reversed that a month later…. where they had the same groups but they switched the odd and even numbers. When they were tested on the material two days later, there was no significant difference. What this result seems to suggest is that the earlier studies were subject to sample selection bias, and the basic problem is that students who were more likely to choose to take notes on computers were students, on average, who would do less well whether they took notes by hand or by computer.

Rebecca: Students who weren’t as strong necessarily academically would be the ones who are more likely to choose to take notes on a computer… maybe that’s because they weren’t as interested in the subject matter…, maybe they didn’t really understand the subject matter and they were choosing that device, in particular, because they could “multitask.”

John: …and one of the things we do know, and this this has been found in many studies, is that none of us is very good at multitasking. The students dramatically overestimate their ability to multitask and they underestimate the amount of time they actually do multitask… and pulling away your attention to look at a text… to respond to a text… to jump to a website… to respond to a Facebook, or an Instagram ,or other notification… has a fairly strong adverse effect on your recall… on your ability to remember things… and students significantly underestimate how often they do that, as measured by studies where they’ve they were using a proxy server to actually track students’ use of mobile devices.
The main conclusion of the study is that it really doesn’t matter whether students take notes by hand or whether they take notes on a computer.

Rebecca: That leads me to kind of question whether or not students are just not very good note takers, right? We don’t spend a lot of time, especially at the college and university level, teaching students how to take good notes… and they think that copying verbatim whatever they see on the board, or what have you, is how to take notes, rather than taking the time to reflect, process, and figure out what things mean and putting it in their own words. So, I wonder how many faculty actually take any time to discuss note-taking or what might be effective in a particular class. I remember in Small Teaching that James Lang suggested using an outline and providing students with outlines that the students could fill in with blanks to follow along and focus their attention but also structure their notes in a way that might be useful. So, that’s one technique faculty could use.

John: Right, it could be an outline, or it could even be distributing… for those who use PowerPoint slides or other types of presentations …distributing those with some gaps, because if you provide students with too much detail then they tend not to use it very much… they tend not to annotate it quite as much. One of my former colleagues, Bill Goffe (who was on the show a few weeks ago), when he shares PowerPoint slides (or at least in the past when he shared PowerPoint slides) he’d leave the graphs off of it. So, he’d have some text there and some basic outlines, but then students would have to fill in part of it. So he’d share the notes in advance, but there would be things they’d have to fill in… which fills that same sort of purpose, of having a broad outline but then requiring students to make some connections on that as well.

Rebecca: Similarly, I think students generally don’t have very good metacognition, right? ….and so I find that using quizzes in class and going over those quizzes is a good way for students to fill in their notes. What I do is a series of review questions most class periods at the start of class that interleave different topics that we’ve had throughout the semester. Students take those quizzes and then I have them self grade it in a red pen… in a different color… and we go over it… and I require them to take notes about why their answer is incorrect and what the correct answer is. So I’m forcing them to do that reflective piece about what the thing means, and put it in their own words, essentially. If they just regurgitate something that I had said, and they can’t actually explain what that means, then we take that opportunity during the review questions… time to figure that out or take the time to do that.

John: In my class, probably more than half the time… sometimes 60 or 70% of the time, I’m giving students problems where they have to solve them, and I use Eric Mazur’s think-pair-share type of methodology with clickers, where they first answer the questions individually and they give their responses… and then if about half of them get it incorrect (I try to aim for questions where about half of them will get it correct the first time) then I have them discuss it and work on it and then they vote again… and generally there is some significant improvement and it’s that retrieval practice that’s probably much more effective than just simply lecturing to students. I do give some short lectures in class for 5 or 10 minutes at a time and students will often comment that they need more time to write things down and I’ll remind them that perhaps what they should be writing down is not every word that I say or every word that they may happen to see but the things that would be most helpful for them and making those connections… because I share any slides that I use with my class, so they don’t really need to try to transcribe everything. If they really want a transcription of everything, I record the class. They could watch it. We use Panopto here. They could watch it later if they need to go back and review something. They don’t really need to use the notes for that and they should be focusing on trying to use their time more effectively to help them understand the things that they don’t understand as well… and the retrieval practice that’s being done helps them… or at least the goal is, to help them understand what they don’t know… to improve that metacognition as you mentioned.

Rebecca: One of the things that I started doing after we’ve had our reading groups on campus, is spending more time in class talking about evidence-based practices, so that students are more aware of how they learn so that they can be more effective… But one of the things that this study is encouraging me to think more about, is to maybe spend a little bit of time in class to go over note-taking strategies… and when I’m finding that students are struggling with that, to kind of review those again. I do the evidence-based stuff frequently when we’re doing review questions because I often ask “Now, why do we do review questions?” before we do them to encourage students to take advantage of that opportunity… and remember that it’s not really a penalizing thing, but really about learning. So I think the note-taking is something that I want to take on maybe this spring and think about how to help students get better at that because every time I’ve asked a student who’s asked for help on something… and I say ”Well, where is that in your notes?” …their notes are like a disaster and they’re not organized. They’re in multiple notebooks, they’re in random scraps of paper or whatever. So I think finding methods to help students do that more effectively is worth the time and effort and I’d encourage other faculty to think about taking that on.

John: Normally, ask our guests what they’re going to do next, but now we’ll ask each other. What are you going to do next, Rebecca?

Rebecca: So, related to laptops and things in the classroom, I think that I’m going to be a little more cognizant about actively having students use their devices. So if they have them, that we’re taking advantage of the fact that they’re in class, and I’m definitely gonna find a way to spend some instruction time talking about note-taking. How about you, John?

John: I’ve been expanding, gradually, in talking more about why I’m doing things and to help motivate the methods that I’ve been using in class with retrieval practice, spaced practice and interleaved practice. It doesn’t always get through to students, but I’m gonna keep working on that… but I’m also going to be trying a couple of new things this spring semester. One is following up with the discussion we had with Jeffrey Riman, which came out in episode 10, I’m going to be using VoiceThread in at least one of my classes this spring, and I’m also going to flip my econometrics class a bit more than it has been flipped in the past… and I’m also going to write a couple new chapters for the book that we’re using. So I’ve got a lot of plans for this semester. I hope I manage to get them all together before the semester starts.
Thanks for joining us.

Rebecca: Catch you next time.

6: Evidence-based teaching in large classes

Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.

Show Notes

  • Walstad, W., & Allgood, S. (1999). What do college seniors know about economics?. The American Economic Review, 89(2), 350-354.
  • Cabane, O. F. (2012). The Charisma Myth. Master the Art of Personal Magnetism. New York: Penguin Publishing.
  • Chew, Stephen L. (2010). Improving classroom performance by
    challenging student misconceptions about learning. APS Observer, 23(4).
    https://www.psychologicalscience.org/observer/improving-classroom-performance-by-challenging-student-misconceptions-about-learning
  • Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The physics teacher, 30(3), 141-158.
  • Learning Scientists Blog
  • Resources for Economists on the Internet
  • Schwartz, Daniel L., Tsang, Jessica M., and Blair, Kristen P. (2016). The ABCs of How We Learn. New York: W.W. Norton and Co.
  • Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science, 323(5910), 122-124.
  • Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE-Life Sciences Education, 10(1), 55-63.
  • The Teaching Professor
  • Wieman, C. (2007). Why not try a scientific approach to science education?. Change: The Magazine of Higher Learning, 39(5), 9-15.

Transcript

John: Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego.Bill: is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.
Welcome, Bill!

Bill: Ah, thank you.

John: It’s good to talk to you again.

Rebecca: Yes.

Bill: It is.

Rebecca: Today our teas are are…

John: Are you having any tea today,

Bill:?

Bill: Any tea today? No, I am not. I just had some hot chocolate today, however.

John: it sounds like maybe that might be good for your cold… with some honey and so on.

Bill: I think it probably would be.

John: My tea today is ginger peach green tea.

Rebecca: … and I have Lady Grey.

Bill: Ah, enjoy.

John: You’ve been teaching large economics classes for quite some time, beginning at the University of Southern Mississippi, here at the State University of New York at Oswego and now at Penn State. How has your teaching changed and evolved over the years?

Bill: I think I’ve become much more reflective and thoughtful about what I do. It’s odd that the more you study this, the more you realize you do not know, which maybe is not surprising. I oftentimes now look at what’s going on in cognitive science research as well as in STEM education research, particularly physics education research, and apply to my classrooms.

John: …and I know you’re active on a number of listservs online in terms of following what’s going on in both economics teaching research and physics research. What are we doing differently in economics or what are we not doing?

Bill: I don’t think we’re very thoughtful if our students aren’t learning. I would guess that most economists would tend to blame the student rather than the instruction and careful thinking about things. There’s a paper by Hestenes, a physicist, where they develop a so-called concept inventory, looking at fundamental understanding of their students, and they very bluntly say that “our students didn’t learn very much.” The few that really did was by happenstance. Economics is not at that point.

John: I remember one time we were talking about a conversation you had coming back from a conference with someone from a military academy who was talking about their reaction to student learning where they found their students weren’t doing very well. Do you remember that? This was a few years ago.

Bill: Yeah. Yes, I do. It was fairly formative for me, He was flying from Colorado Springs, the Air Force Academy, back to Rhode Island for a conference and he was grading a test, his biology, genetics, or something. He noticed all his students missed a certain question, and I would tend to think (at least back then) that that’s the students’ problem, forget about it. …and it dawned on him that “oh, I mustn’t have taught this very well.” That was really an eye-opening story for me.

Rebecca: I think it’s really interesting that, you know, some of the best teachers are probably some of the most reflective teachers, and take the time to sit back and continue to try to see what they could do better, and take the time to learn new things, and read up on these things. So it’s it’s always exciting to hear other faculty members who spend the time to do this.
I’m wondering…. large classes are something that are pretty common in economics, but maybe aren’t in some other areas, but large large classes present certain kinds of issues. So, can you talk a little bit about how, some of the evidence-based practices that you’ve been studying, you’re implementing in these larger classes that might be a little intimidating for people to think about.

Bill: Several things, one is peer instruction, sometimes called think-pair-share where students will be presented with a difficult question. I tend to aim for questions that maybe half the students would get right, as students respond individually with their clickers. I then encourage them to talk to each other to: “convince them of your answer.” Typically, I see about a twenty, maybe thirty even, now and then a forty point improvement on number answering correctly, and then I actually talk out the answer as well. So they get correct answer oftentimes both from their peer as well as from me. There’s a paper by Michelle Smith. She is an educational researcher in biology. Actually you might want to look into her. She’s usually at Maine she’s on sabbatical this year at Cornell fairly close to you.

Rebecca: Wow, that is really close to us.

John: Speaking of Cornell there was a podcast issuing from Cornell, the Teach Better podcast…

Bill: Yes.

John: …with an economist actually as one of the co-hosts, and on this week’s episode they were talking about the use of concept tests and noting that we really don’t do that in economics, other than the Test of Understanding in College Economics (or the TUCE). Do you know if anyone’s been looking into, or developing, or thinking about developing concept tests for economics.

Bill: Yes, I know several. It might be worth taking a second setting back. In most of these concept tests, they have a good idea of students’ preconceptions and they test against that. We tend not to know what the students’ misconceptions or preconceptions are, but I’m not sure everything we do in economics would fit that model, if you will. Certainly students might have wrong preconceptions about how to calculate the unemployment rate, but something like aggregate supply and demand, I’m not sure they have many preconceptions there…. and I would add that Michelle Smith, who you listened to on that podcast, she’s the one I mentioned who is at Cornell. As far as people working on suc concept tests, I know of several. The projects are pretty not… not speeding along very quickly I think it’s fair to say.

John: One of the ways in which you can get some information from students on preconceptions is with clickers. Is that part of the way you use it? To test for common misunderstandings and help them get past that?

Bill: Yes, indeed it is. I sometimes test for misunderstandings first with so-called JITTs where students answer ahead of time in our LMS. We happen to use Canvas. I’ll give some essay questions on a given topic and I also ask them what they find confusing or puzzling in that set of readings, and from that I’ll base a clicker question on it. One example is students tend to think that roughly a third of workers earn the minimum wage, when in fact it’s maybe not even a tenth of that, and that’s a sort of misconception I actually do address with clickers with data from the so-called JITTs.

Rebecca: In a… in a large class setting like that where you’re using little essays in your course management system, how many how many students are you doing that technique with.

Bill: Quite a few. I have two sections of about 300 students, give or take, and in both those sections I have six graders. So each graders grading about 50 different students’ material. Works out fairly well. They typically have about one thing to grade per week.

John: So, JITTs stands for just-in-time teaching, I’m assuming.

Bill: Yes.

John: How do you go about doing that?

Bill: So, it’s assigned before class, thus the just-in-time. However, just before class and then base class upon that. To be truthful, though, now teaching this same cohort ,the same group, I don’t see that many surprises anymore… but I do review them just to remind myself of where they stand.

Rebecca: So you mentioned graders and also teaching assistants. Can you talk a little bit about, kind of, the helpers you have in the classroom and how that helps facilitate learning, and some of these techniques that you’re implementing in your classes?

Bill: Sure, so for each of these 300 student classes I have one graduate TA, he or she does administrative things. I have about four in-class learning assistants. Penn State’s a hotbed, if you will, for learning assistants undergraduates. Our College of Science used about five hundred this semester in a number of different science courses. It’s a real showcase for Penn State science. Then I also have about six undergraduate graders per class who do the grading. I joke I become a middle manager.

Rebecca: Do these student helpers help with class motivation or anything during the actual class period? or just kind of this… these outside more administrative activities?

Bill: Huh, good point The learning assistants, they go to every class. They write up a summary for me about things that went well… things that did not. You know it’s hard to get feedback here teaching and they’re a natural. They know the material, but they’re also students… and then we do a peer instruction, something like think-pair-share. We all wander around the room to get a sense of students’ understanding or exactly what they’re not understanding with these challenging clicker questions and they report back to me both during class as well as after class. It’s a very common sort of a STEM education technique.

John: How do you keep that many students’ focus? I know when I teach large classes there’s always a few students who want to chat a little bit in the back and that can be distracting.

Bill: Sure, there are several things. One is the first day of class, I do a quick assignment, I guess you could call it. I ask the students to take out a piece of paper and write on that paper what should be the classroom rules so this class runs smoothly and then they develop the for coming to class on time, not talking, and so forth. So when the student is talking, I can say: “Look the people around you don’t want to hear you talk,” because we’ve gone over that data from that first day survey. About 1/3 of the students say talking is disruptive, so I can use a student’s voice channelled through me to get quiet. I actually work on my public speaking quite a bit. I’m a member of Toastmasters. You might remember when Ken Bain came to talk at Oswego, this was maybe eight years or so ago. I’ve never seen someone command a room like that. I’ve been in rooms with two U.S. presidents and he used a lot of public speaking techniques to command the room, and I do that as well. Not as good but still, it’s helpful. Finally, I have questions every day or two that hopefully are puzzling, that will challenge students with the current material, to make it as interesting as macroeconomics can be.

John: ….and going back to Ken Bain, one of the main thing he’s argues for is for having these big questions that you organize a course, as well as individual class sessions on, and I believe you do that too in your intro macro class.

Bill: I do indeed. I ask two questions that we answer throughout the course: one is “why are some countries rich… other ones poor.” The very first day of class, I show a pair of videos: life in modern-day Australia, people living the good life (food, fashion, sports, nice cars, great life) and then I pair it with a clip from Port-au-Prince Haiti (an extremely poor place, shockingly poor) so it frames that question. The other question this semester is why are some people doing poorly an economy in spite of record low employment rates and GDP growth since 2009; in other words, why are some people being left behind? One of the things about that course is the current slow expansion, so thus we talk about the standard topics with aggregate supply and aggregate demand.

Rebecca: I know you’re a big advocate of evidence-based teaching methods, so I’m curious about how you help students make sure that they’re doing spaced practice and interleaved practice.

Bill: Sure. So rather than giving a couple of midterms and a final here’s how I break-up would be midterms into a series of seven quizzes. The first was in the third week of the semester, the seventh one is the very last week. they’re exam caliber questions, but they’re just split up, and so for any given quiz I can ask material (and indeed I do) from earlier in the semester. So that gets the spacing down. They cover a given topic: What is GDP? capital? unemployment? so forth, several times and then I always make sure to mix up all the questions so they’re actually interleaved.

Rebecca: Interesting, and that sounds like an interesting…. It’s always hard to balance, I would imagine in such a big class, like how to find that balance and communicate that to students. Do you take time in classes to explain to students why you do some of the methods that you do? To help them understand?

Bill: I do and actually what I do for that is I use a student’s voice.On the first day class ,I show student comments from my student evaluations from the previous semester, about using clickers like I do, about quizzes, and I let past students tell my new students on the first day of class about why these methods work. I think that probably helps with the buy in, in having a student’s voice rather than my own voice. I should add on the quizzes, we also go over them right then and there, so it’s example of so-called deliberate practice where someone’s trying something challenging, and then they’re getting feedback on their work, and hopefully that feedback helps them develop a richer mental model of the topic at hand.

John: …and the same thing occurs when you’re using the clicker questions, when you’re doing the think-pair-share type activity…. that you’re giving them immediate feedback, because sometimes some of them will go astray even though you see that improvement, that immediate feedback is helpful.

Bill: Yes, indeed, and that was prior to Michelle Smith’s paper where she found… they looked at peer instruction (just students talk with each other) or a clicker question (instructor explaining it} and she found much more learning when both things happened. I think what’s happening is two explanations probably come at a given topic from different directions, and help you understand things even better. There’s a fairly new book out by Schwartz and co-authors… I forget their names…. The ABCs of How We Learn. Schwartz is at Stanford, and they find when you’re doing analogies, two analogies is much more effective in teaching than one. I’m thinking that several explanations, as long as they’re brief, are more efficient, in terms of more learning, than just one.

John: Because students come with different backgrounds and different explanations can connect with some students better than others, and the more hooks they have the more elaborate the models they can develop.

Rebecca: Especially when you’re, kind of, meeting students where they’re at, and it sounds, it sounds like the opportunities that you’re providing for student… the student voice to come out is also helping the students connect and feel related to the material because if it comes from a peer it’s going to help them feel a little more related as well, I would imagine.

Bill: I think that’s the case. They feel connected, for example, I’ll put up some of their JITT responses in class when we’re just starting a section or I’ll say someone over here’s puzzled about this, let me address that… and I also get a lot of questions in class, which I think it’s fairly uncommon for large classes

John: I’ve noticed the same thing in the large class when you start giving them more interesting questions and you get them more actively engaged they’re much more likely to ask questions and participate in general.

Bill: Ah, good.

Rebecca: I’d like to shift the conversation a little bit to some faculty professional development stuff, because I think there’s a lot that you’ve already mentioned but also other things that I know that you’re doing. I’m really curious in your mention of Toastmasters and the public speaking and developing those skills. So do you have some advice for faculty about how to command the room or how to develop those skills to command the room?

Bill: Several thoughts there, one is there are Toastmasters clubs pretty much everywhere. They’re easy to join and almost every long-term member of my club who’s been there some years speaks better than almost any academic I’ve ever seen and part of it is that it’s really helpful to see really good speakers speak, and then it quickly forces you to get better on the easy things to change: cut out the UM the AHS, nervous tics and things like that. It sounds a bit odd, but I just happened to pick up a book one day. It’s called The Charisma Myth, and it talks about how one can generate charisma in different situations, and a couple of chapters deal directly with the sorts of techniques that Ken Bain used. I could literally check off the things I saw him doing. I was just quite impressed with that book, The Charisma Myth.

John: So just like anything else, more deliberate practice can improve your skills.

Bill: Yes, very much so. Most things were pretty content with being kind of average. I suspect most of us are fairly average drivers. We don’t go over curbs, but we’re not driving stunt cars in Hollywood, but when you teach I think it makes sense to ask for more effort there and to actually try to get better at it. It’s quite rewarding when you do.

Rebecca: How did you get involved in studying teaching and learning? and what advice do you offer others who might be interested in doing their own studies?

Bill: Sure, it was about 8 years or so ago. Some friends of mine, Scott Simkins and Mark Maier sent me a paper by Carl Wieman and it had the title: “Why Not Take a Scientific Approach in Science Teaching” and it just opened up a whole new world to me. I don’t know if

John: knows this story, but I was thinking about perhaps leaving the profession at that point. I was kinda bored… research didn’t interest me that much…. and teaching was okay…. and this just opened my eyes to how we could teach better using principles from cognitive science. So that was the hook for me… was these methods other disciplines are using to improve learning for their students

John: might be familiar with the paper by Walston and Allgood in 1999, that it seems economic students really aren’t learning all that much. I saw a problem that potentially could be solved.
To get others interested then, maybe look at your local STEM discipline based education research group, a lot of universities have those. They’d probably be glad to have other people involved.

Rebecca: What resources do you use to stay up on the scholarship of teaching and learning.

Bill: I used Twitter quite a bit. I follow a fair number people on Twitter. I mainly use it for professional things. I saw a real interesting paper the day on metacognition. It was in the chemistry education research, on how with repeated practice on metacognition, that is students thinking about thinking and specifically thinking about performance on homeworks and exams, they can get better at it. So, I’m a big follower on Twitter…. also use the Learning Scientists’ blog. They have a very nice blog and Magna publications has a number of different publications, like The Teaching Professor. Those are mainly my main things that I follow.

John: Along those lines, we had a podcast that came out on Wednesday involving the online metacognitive cafe discussion forum, which is designed exactly to help students improve their metacognition. It probably wouldn’t work too well in a very large class, but in a smaller setting it can help students, and in fact when I use this, I use some of the Learning Sciences videos and some of their other materials. The developer of it was Judie Little

John: at Genesee Community College, and she used some other resources and she had some different issues that she wanted to focus on. Particularly she wanted to help students understand the importance of doing reading and providing more effort into their work. Then it evolved into more broadly an examination of how students could improve their learning skills, and that’s useful.

Bill: Oh, indeed it is, that’s very nice to hear.

Rebecca: Not surprising that in metacognition, like in other areas, practice is what gets us good at things, right? So, the more you practice being aware of what, you know, the better you’re going to be at it… and I think we can all be better at helping our students and guiding our students through those processes. You know, when I’m teaching Web Design I have those same issues that students need to troubleshoot and problem-solve and a lot of the times that they don’t really even understand what the problem is or where they where they’re connecting into or where they’re getting hung up…. and so coaching them through problem-solving strategies and helping them recognize strategies to do those sorts of things can be really helpful. So, I think the more we can all do more things that help… help students focus on that is… is a good thing to do?

John: Right before we started this I was in the midst of grading last week’s metacognitive cafe discussion forum and one of the prompts I used was asking students to reflect back on what we’ve done so far this semester and connecting it to their future career and their lives, and so forth, and asking what they’ve learned that was surprising or what they’ve learned that will be most useful for them in the future… and I was really surprised to see that probably about 90% of them listed the metacognitive cafe and learning how to learn more efficiently as the most useful thing they’ve taken out of the course so far. Many of them also mentioned economics, but it was nice to see that that is having, or at least they perceive it as having, an impact.

Bill: Oh, that’s very impressive. Congratulations.

Rebecca: Do you do any sort of mid-semester feedback to kind of know what students are still getting hung up with or how they can move forward.

Bill: Sometimes I did a mid-semester revaluation. I’m sorry to say this semester I did not. I’ve taken up a new online class that I had not taught before. It’s taking up a lot of time, so that was one of the things that just fell away.

John: But you do have the student learning assistants out there who are giving you feedback from their perspective, so you’ve got a near current student perspective, right?

Bill: Yes I do, it’s very handy. I just… as you get older you don’t understand students as well, I think. It’s very handy to have a translator,if you will.

Rebecca: It’s funny how they stayed the same age and you don’t, right?

Bill: Yes. Yes, this is really not fair.

John: Well, I’m not sure it’s just due to getting older because when we talked to faculty a lot of them are talking about how…. even new faculty were saying “I was never like that as a student” and in fact they’re probably right because they… the people who choose to become faculty are not random drawings from the population of students…. and if they think back carefully, many of their fellow students were probably very much like the students that we deal with.

Bill: Yes.

John: You’ve mentioned many times that you like to try new things every year. What are some of the things you’d like to try in the future to improve student learning?

Bill: I want to work on having students use better study skills. I think that’s one of those things I suspect, and part of this project actually collecting data, is improving student study skills. I’m thinking about using some of the materials from The Learning Scientists, the blog we mentioned earlier, and also do an in-class exercise a bit like Steve Cho does on illustrating to students how deeper processing leads to more retention. You might remember Steve Cho’s project. He has students do a word list where students either study it intently, determining if words either have a positive or negative aspect, or another group studies this words more lightly, if you will, they just count letters and those that process this more deeply remember their words more fully. That’s true, but doesn’t have much resonance with our students, because we don’t do word lists in what we teach. I want to do a similar project with reading economic content and then actually see if those who use better techniques, directed by me in this case, actually learn more than students who did not. This all take place in the classroom.

Rebecca: Sounds really exciting. I’m looking forward to hearing what those results will be.

Bill: Ah, well, thank you.

John: When Michelle Miller visited a couple years ago, she talked about how they redesigned the first-year program to include a component on improving student learning and… as an online blackboard module. I’ve often argued that perhaps that would be useful if we could do it for all freshmen here in some way so that students all get exposed to some improved learning methods. Not only are faculty sometimes unaware of effective practices, but students often use practices that are not particularly useful.

Bill: Yes that’s certainly true and it actually even after students who use these more effective practices they think they’re less effective than just rereading the highlighting.

John: Part of the reason is that the practices they like to use like cramming gives them some really good immediate effects but they’re not very useful in terms of long-term learning. So they get positive feedback when they use it but they don’t see the connection between that and not remembering the stuff when they get to the next course… and how can we break that down?

Bill: That would be hard. What I’m trying to do in this project I mentioned is show to students there in the classroom that these different methods actually do lead to more learning with real actual content so that’s like my goal and hope for this project.

John: We’re looking forward to hearing more about that.

Rebecca: Yeah, thanks for joining us! As always, it’s a pleasure to hear what you’re doing in the classroom and what you’re researching.

Bill: Well, thank you so much. It was great fun and I’m probably going to go buy some tea now, to be honest with you.

John: Try some honey, it’ll probably help you feel a little bit better.

Bill: Yes, I think so.

John: It’s great talking to you,

Bill: and I look forward to talking to you again

Bill: Great. Great talking to y’all. Thank you.

Rebecca: Thanks.

John: Thank you.