6: Evidence-based teaching in large classes

Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.

Show Notes

  • Walstad, W., & Allgood, S. (1999). What do college seniors know about economics?. The American Economic Review, 89(2), 350-354.
  • Cabane, O. F. (2012). The Charisma Myth. Master the Art of Personal Magnetism. New York: Penguin Publishing.
  • Chew, Stephen L. (2010). Improving classroom performance by
    challenging student misconceptions about learning. APS Observer, 23(4).
  • Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The physics teacher, 30(3), 141-158.
  • Learning Scientists Blog
  • Resources for Economists on the Internet
  • Schwartz, Daniel L., Tsang, Jessica M., and Blair, Kristen P. (2016). The ABCs of How We Learn. New York: W.W. Norton and Co.
  • Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science, 323(5910), 122-124.
  • Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE-Life Sciences Education, 10(1), 55-63.
  • The Teaching Professor
  • Wieman, C. (2007). Why not try a scientific approach to science education?. Change: The Magazine of Higher Learning, 39(5), 9-15.


John: Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego.Bill: is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.
Welcome, Bill!

Bill: Ah, thank you.

John: It’s good to talk to you again.

Rebecca: Yes.

Bill: It is.

Rebecca: Today our teas are are…

John: Are you having any tea today,


Bill: Any tea today? No, I am not. I just had some hot chocolate today, however.

John: it sounds like maybe that might be good for your cold… with some honey and so on.

Bill: I think it probably would be.

John: My tea today is ginger peach green tea.

Rebecca: … and I have Lady Grey.

Bill: Ah, enjoy.

John: You’ve been teaching large economics classes for quite some time, beginning at the University of Southern Mississippi, here at the State University of New York at Oswego and now at Penn State. How has your teaching changed and evolved over the years?

Bill: I think I’ve become much more reflective and thoughtful about what I do. It’s odd that the more you study this, the more you realize you do not know, which maybe is not surprising. I oftentimes now look at what’s going on in cognitive science research as well as in STEM education research, particularly physics education research, and apply to my classrooms.

John: …and I know you’re active on a number of listservs online in terms of following what’s going on in both economics teaching research and physics research. What are we doing differently in economics or what are we not doing?

Bill: I don’t think we’re very thoughtful if our students aren’t learning. I would guess that most economists would tend to blame the student rather than the instruction and careful thinking about things. There’s a paper by Hestenes, a physicist, where they develop a so-called concept inventory, looking at fundamental understanding of their students, and they very bluntly say that “our students didn’t learn very much.” The few that really did was by happenstance. Economics is not at that point.

John: I remember one time we were talking about a conversation you had coming back from a conference with someone from a military academy who was talking about their reaction to student learning where they found their students weren’t doing very well. Do you remember that? This was a few years ago.

Bill: Yeah. Yes, I do. It was fairly formative for me, He was flying from Colorado Springs, the Air Force Academy, back to Rhode Island for a conference and he was grading a test, his biology, genetics, or something. He noticed all his students missed a certain question, and I would tend to think (at least back then) that that’s the students’ problem, forget about it. …and it dawned on him that “oh, I mustn’t have taught this very well.” That was really an eye-opening story for me.

Rebecca: I think it’s really interesting that, you know, some of the best teachers are probably some of the most reflective teachers, and take the time to sit back and continue to try to see what they could do better, and take the time to learn new things, and read up on these things. So it’s it’s always exciting to hear other faculty members who spend the time to do this.
I’m wondering…. large classes are something that are pretty common in economics, but maybe aren’t in some other areas, but large large classes present certain kinds of issues. So, can you talk a little bit about how, some of the evidence-based practices that you’ve been studying, you’re implementing in these larger classes that might be a little intimidating for people to think about.

Bill: Several things, one is peer instruction, sometimes called think-pair-share where students will be presented with a difficult question. I tend to aim for questions that maybe half the students would get right, as students respond individually with their clickers. I then encourage them to talk to each other to: “convince them of your answer.” Typically, I see about a twenty, maybe thirty even, now and then a forty point improvement on number answering correctly, and then I actually talk out the answer as well. So they get correct answer oftentimes both from their peer as well as from me. There’s a paper by Michelle Smith. She is an educational researcher in biology. Actually you might want to look into her. She’s usually at Maine she’s on sabbatical this year at Cornell fairly close to you.

Rebecca: Wow, that is really close to us.

John: Speaking of Cornell there was a podcast issuing from Cornell, the Teach Better podcast…

Bill: Yes.

John: …with an economist actually as one of the co-hosts, and on this week’s episode they were talking about the use of concept tests and noting that we really don’t do that in economics, other than the Test of Understanding in College Economics (or the TUCE). Do you know if anyone’s been looking into, or developing, or thinking about developing concept tests for economics.

Bill: Yes, I know several. It might be worth taking a second setting back. In most of these concept tests, they have a good idea of students’ preconceptions and they test against that. We tend not to know what the students’ misconceptions or preconceptions are, but I’m not sure everything we do in economics would fit that model, if you will. Certainly students might have wrong preconceptions about how to calculate the unemployment rate, but something like aggregate supply and demand, I’m not sure they have many preconceptions there…. and I would add that Michelle Smith, who you listened to on that podcast, she’s the one I mentioned who is at Cornell. As far as people working on suc concept tests, I know of several. The projects are pretty not… not speeding along very quickly I think it’s fair to say.

John: One of the ways in which you can get some information from students on preconceptions is with clickers. Is that part of the way you use it? To test for common misunderstandings and help them get past that?

Bill: Yes, indeed it is. I sometimes test for misunderstandings first with so-called JITTs where students answer ahead of time in our LMS. We happen to use Canvas. I’ll give some essay questions on a given topic and I also ask them what they find confusing or puzzling in that set of readings, and from that I’ll base a clicker question on it. One example is students tend to think that roughly a third of workers earn the minimum wage, when in fact it’s maybe not even a tenth of that, and that’s a sort of misconception I actually do address with clickers with data from the so-called JITTs.

Rebecca: In a… in a large class setting like that where you’re using little essays in your course management system, how many how many students are you doing that technique with.

Bill: Quite a few. I have two sections of about 300 students, give or take, and in both those sections I have six graders. So each graders grading about 50 different students’ material. Works out fairly well. They typically have about one thing to grade per week.

John: So, JITTs stands for just-in-time teaching, I’m assuming.

Bill: Yes.

John: How do you go about doing that?

Bill: So, it’s assigned before class, thus the just-in-time. However, just before class and then base class upon that. To be truthful, though, now teaching this same cohort ,the same group, I don’t see that many surprises anymore… but I do review them just to remind myself of where they stand.

Rebecca: So you mentioned graders and also teaching assistants. Can you talk a little bit about, kind of, the helpers you have in the classroom and how that helps facilitate learning, and some of these techniques that you’re implementing in your classes?

Bill: Sure, so for each of these 300 student classes I have one graduate TA, he or she does administrative things. I have about four in-class learning assistants. Penn State’s a hotbed, if you will, for learning assistants undergraduates. Our College of Science used about five hundred this semester in a number of different science courses. It’s a real showcase for Penn State science. Then I also have about six undergraduate graders per class who do the grading. I joke I become a middle manager.

Rebecca: Do these student helpers help with class motivation or anything during the actual class period? or just kind of this… these outside more administrative activities?

Bill: Huh, good point The learning assistants, they go to every class. They write up a summary for me about things that went well… things that did not. You know it’s hard to get feedback here teaching and they’re a natural. They know the material, but they’re also students… and then we do a peer instruction, something like think-pair-share. We all wander around the room to get a sense of students’ understanding or exactly what they’re not understanding with these challenging clicker questions and they report back to me both during class as well as after class. It’s a very common sort of a STEM education technique.

John: How do you keep that many students’ focus? I know when I teach large classes there’s always a few students who want to chat a little bit in the back and that can be distracting.

Bill: Sure, there are several things. One is the first day of class, I do a quick assignment, I guess you could call it. I ask the students to take out a piece of paper and write on that paper what should be the classroom rules so this class runs smoothly and then they develop the for coming to class on time, not talking, and so forth. So when the student is talking, I can say: “Look the people around you don’t want to hear you talk,” because we’ve gone over that data from that first day survey. About 1/3 of the students say talking is disruptive, so I can use a student’s voice channelled through me to get quiet. I actually work on my public speaking quite a bit. I’m a member of Toastmasters. You might remember when Ken Bain came to talk at Oswego, this was maybe eight years or so ago. I’ve never seen someone command a room like that. I’ve been in rooms with two U.S. presidents and he used a lot of public speaking techniques to command the room, and I do that as well. Not as good but still, it’s helpful. Finally, I have questions every day or two that hopefully are puzzling, that will challenge students with the current material, to make it as interesting as macroeconomics can be.

John: ….and going back to Ken Bain, one of the main thing he’s argues for is for having these big questions that you organize a course, as well as individual class sessions on, and I believe you do that too in your intro macro class.

Bill: I do indeed. I ask two questions that we answer throughout the course: one is “why are some countries rich… other ones poor.” The very first day of class, I show a pair of videos: life in modern-day Australia, people living the good life (food, fashion, sports, nice cars, great life) and then I pair it with a clip from Port-au-Prince Haiti (an extremely poor place, shockingly poor) so it frames that question. The other question this semester is why are some people doing poorly an economy in spite of record low employment rates and GDP growth since 2009; in other words, why are some people being left behind? One of the things about that course is the current slow expansion, so thus we talk about the standard topics with aggregate supply and aggregate demand.

Rebecca: I know you’re a big advocate of evidence-based teaching methods, so I’m curious about how you help students make sure that they’re doing spaced practice and interleaved practice.

Bill: Sure. So rather than giving a couple of midterms and a final here’s how I break-up would be midterms into a series of seven quizzes. The first was in the third week of the semester, the seventh one is the very last week. they’re exam caliber questions, but they’re just split up, and so for any given quiz I can ask material (and indeed I do) from earlier in the semester. So that gets the spacing down. They cover a given topic: What is GDP? capital? unemployment? so forth, several times and then I always make sure to mix up all the questions so they’re actually interleaved.

Rebecca: Interesting, and that sounds like an interesting…. It’s always hard to balance, I would imagine in such a big class, like how to find that balance and communicate that to students. Do you take time in classes to explain to students why you do some of the methods that you do? To help them understand?

Bill: I do and actually what I do for that is I use a student’s voice.On the first day class ,I show student comments from my student evaluations from the previous semester, about using clickers like I do, about quizzes, and I let past students tell my new students on the first day of class about why these methods work. I think that probably helps with the buy in, in having a student’s voice rather than my own voice. I should add on the quizzes, we also go over them right then and there, so it’s example of so-called deliberate practice where someone’s trying something challenging, and then they’re getting feedback on their work, and hopefully that feedback helps them develop a richer mental model of the topic at hand.

John: …and the same thing occurs when you’re using the clicker questions, when you’re doing the think-pair-share type activity…. that you’re giving them immediate feedback, because sometimes some of them will go astray even though you see that improvement, that immediate feedback is helpful.

Bill: Yes, indeed, and that was prior to Michelle Smith’s paper where she found… they looked at peer instruction (just students talk with each other) or a clicker question (instructor explaining it} and she found much more learning when both things happened. I think what’s happening is two explanations probably come at a given topic from different directions, and help you understand things even better. There’s a fairly new book out by Schwartz and co-authors… I forget their names…. The ABCs of How We Learn. Schwartz is at Stanford, and they find when you’re doing analogies, two analogies is much more effective in teaching than one. I’m thinking that several explanations, as long as they’re brief, are more efficient, in terms of more learning, than just one.

John: Because students come with different backgrounds and different explanations can connect with some students better than others, and the more hooks they have the more elaborate the models they can develop.

Rebecca: Especially when you’re, kind of, meeting students where they’re at, and it sounds, it sounds like the opportunities that you’re providing for student… the student voice to come out is also helping the students connect and feel related to the material because if it comes from a peer it’s going to help them feel a little more related as well, I would imagine.

Bill: I think that’s the case. They feel connected, for example, I’ll put up some of their JITT responses in class when we’re just starting a section or I’ll say someone over here’s puzzled about this, let me address that… and I also get a lot of questions in class, which I think it’s fairly uncommon for large classes

John: I’ve noticed the same thing in the large class when you start giving them more interesting questions and you get them more actively engaged they’re much more likely to ask questions and participate in general.

Bill: Ah, good.

Rebecca: I’d like to shift the conversation a little bit to some faculty professional development stuff, because I think there’s a lot that you’ve already mentioned but also other things that I know that you’re doing. I’m really curious in your mention of Toastmasters and the public speaking and developing those skills. So do you have some advice for faculty about how to command the room or how to develop those skills to command the room?

Bill: Several thoughts there, one is there are Toastmasters clubs pretty much everywhere. They’re easy to join and almost every long-term member of my club who’s been there some years speaks better than almost any academic I’ve ever seen and part of it is that it’s really helpful to see really good speakers speak, and then it quickly forces you to get better on the easy things to change: cut out the UM the AHS, nervous tics and things like that. It sounds a bit odd, but I just happened to pick up a book one day. It’s called The Charisma Myth, and it talks about how one can generate charisma in different situations, and a couple of chapters deal directly with the sorts of techniques that Ken Bain used. I could literally check off the things I saw him doing. I was just quite impressed with that book, The Charisma Myth.

John: So just like anything else, more deliberate practice can improve your skills.

Bill: Yes, very much so. Most things were pretty content with being kind of average. I suspect most of us are fairly average drivers. We don’t go over curbs, but we’re not driving stunt cars in Hollywood, but when you teach I think it makes sense to ask for more effort there and to actually try to get better at it. It’s quite rewarding when you do.

Rebecca: How did you get involved in studying teaching and learning? and what advice do you offer others who might be interested in doing their own studies?

Bill: Sure, it was about 8 years or so ago. Some friends of mine, Scott Simkins and Mark Maier sent me a paper by Carl Wieman and it had the title: “Why Not Take a Scientific Approach in Science Teaching” and it just opened up a whole new world to me. I don’t know if

John: knows this story, but I was thinking about perhaps leaving the profession at that point. I was kinda bored… research didn’t interest me that much…. and teaching was okay…. and this just opened my eyes to how we could teach better using principles from cognitive science. So that was the hook for me… was these methods other disciplines are using to improve learning for their students

John: might be familiar with the paper by Walston and Allgood in 1999, that it seems economic students really aren’t learning all that much. I saw a problem that potentially could be solved.
To get others interested then, maybe look at your local STEM discipline based education research group, a lot of universities have those. They’d probably be glad to have other people involved.

Rebecca: What resources do you use to stay up on the scholarship of teaching and learning.

Bill: I used Twitter quite a bit. I follow a fair number people on Twitter. I mainly use it for professional things. I saw a real interesting paper the day on metacognition. It was in the chemistry education research, on how with repeated practice on metacognition, that is students thinking about thinking and specifically thinking about performance on homeworks and exams, they can get better at it. So, I’m a big follower on Twitter…. also use the Learning Scientists’ blog. They have a very nice blog and Magna publications has a number of different publications, like The Teaching Professor. Those are mainly my main things that I follow.

John: Along those lines, we had a podcast that came out on Wednesday involving the online metacognitive cafe discussion forum, which is designed exactly to help students improve their metacognition. It probably wouldn’t work too well in a very large class, but in a smaller setting it can help students, and in fact when I use this, I use some of the Learning Sciences videos and some of their other materials. The developer of it was Judie Little

John: at Genesee Community College, and she used some other resources and she had some different issues that she wanted to focus on. Particularly she wanted to help students understand the importance of doing reading and providing more effort into their work. Then it evolved into more broadly an examination of how students could improve their learning skills, and that’s useful.

Bill: Oh, indeed it is, that’s very nice to hear.

Rebecca: Not surprising that in metacognition, like in other areas, practice is what gets us good at things, right? So, the more you practice being aware of what, you know, the better you’re going to be at it… and I think we can all be better at helping our students and guiding our students through those processes. You know, when I’m teaching Web Design I have those same issues that students need to troubleshoot and problem-solve and a lot of the times that they don’t really even understand what the problem is or where they where they’re connecting into or where they’re getting hung up…. and so coaching them through problem-solving strategies and helping them recognize strategies to do those sorts of things can be really helpful. So, I think the more we can all do more things that help… help students focus on that is… is a good thing to do?

John: Right before we started this I was in the midst of grading last week’s metacognitive cafe discussion forum and one of the prompts I used was asking students to reflect back on what we’ve done so far this semester and connecting it to their future career and their lives, and so forth, and asking what they’ve learned that was surprising or what they’ve learned that will be most useful for them in the future… and I was really surprised to see that probably about 90% of them listed the metacognitive cafe and learning how to learn more efficiently as the most useful thing they’ve taken out of the course so far. Many of them also mentioned economics, but it was nice to see that that is having, or at least they perceive it as having, an impact.

Bill: Oh, that’s very impressive. Congratulations.

Rebecca: Do you do any sort of mid-semester feedback to kind of know what students are still getting hung up with or how they can move forward.

Bill: Sometimes I did a mid-semester revaluation. I’m sorry to say this semester I did not. I’ve taken up a new online class that I had not taught before. It’s taking up a lot of time, so that was one of the things that just fell away.

John: But you do have the student learning assistants out there who are giving you feedback from their perspective, so you’ve got a near current student perspective, right?

Bill: Yes I do, it’s very handy. I just… as you get older you don’t understand students as well, I think. It’s very handy to have a translator,if you will.

Rebecca: It’s funny how they stayed the same age and you don’t, right?

Bill: Yes. Yes, this is really not fair.

John: Well, I’m not sure it’s just due to getting older because when we talked to faculty a lot of them are talking about how…. even new faculty were saying “I was never like that as a student” and in fact they’re probably right because they… the people who choose to become faculty are not random drawings from the population of students…. and if they think back carefully, many of their fellow students were probably very much like the students that we deal with.

Bill: Yes.

John: You’ve mentioned many times that you like to try new things every year. What are some of the things you’d like to try in the future to improve student learning?

Bill: I want to work on having students use better study skills. I think that’s one of those things I suspect, and part of this project actually collecting data, is improving student study skills. I’m thinking about using some of the materials from The Learning Scientists, the blog we mentioned earlier, and also do an in-class exercise a bit like Steve Cho does on illustrating to students how deeper processing leads to more retention. You might remember Steve Cho’s project. He has students do a word list where students either study it intently, determining if words either have a positive or negative aspect, or another group studies this words more lightly, if you will, they just count letters and those that process this more deeply remember their words more fully. That’s true, but doesn’t have much resonance with our students, because we don’t do word lists in what we teach. I want to do a similar project with reading economic content and then actually see if those who use better techniques, directed by me in this case, actually learn more than students who did not. This all take place in the classroom.

Rebecca: Sounds really exciting. I’m looking forward to hearing what those results will be.

Bill: Ah, well, thank you.

John: When Michelle Miller visited a couple years ago, she talked about how they redesigned the first-year program to include a component on improving student learning and… as an online blackboard module. I’ve often argued that perhaps that would be useful if we could do it for all freshmen here in some way so that students all get exposed to some improved learning methods. Not only are faculty sometimes unaware of effective practices, but students often use practices that are not particularly useful.

Bill: Yes that’s certainly true and it actually even after students who use these more effective practices they think they’re less effective than just rereading the highlighting.

John: Part of the reason is that the practices they like to use like cramming gives them some really good immediate effects but they’re not very useful in terms of long-term learning. So they get positive feedback when they use it but they don’t see the connection between that and not remembering the stuff when they get to the next course… and how can we break that down?

Bill: That would be hard. What I’m trying to do in this project I mentioned is show to students there in the classroom that these different methods actually do lead to more learning with real actual content so that’s like my goal and hope for this project.

John: We’re looking forward to hearing more about that.

Rebecca: Yeah, thanks for joining us! As always, it’s a pleasure to hear what you’re doing in the classroom and what you’re researching.

Bill: Well, thank you so much. It was great fun and I’m probably going to go buy some tea now, to be honest with you.

John: Try some honey, it’ll probably help you feel a little bit better.

Bill: Yes, I think so.

John: It’s great talking to you,

Bill: and I look forward to talking to you again

Bill: Great. Great talking to y’all. Thank you.

Rebecca: Thanks.

John: Thank you.