55. Open pedagogy

Imagine an academy that values a public knowledge commons and supports and recognizes the academic labor required to develop, maintain, build and evolve that commons. Imagine your students actively contributing to that commons. In this episode, Robin DeRosa joins us to discuss open pedagogy, free textbooks, and the building of such  a commons.

Show Notes

Transcript

John: Imagine an academy that values a public knowledge commons and supports and recognizes the academic labor required to develop, maintain, build, and evolve that commons. Imagine your students actively contributing to that commons. In this episode, we discuss open pedagogy, free textbooks, and the building of such a commons.

[MUSIC]

Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Robin DeRosa…

Robin: That’s me.

John: …Professor of Interdisciplinary Studies and Director of the Interdisciplinary Studies Program at Plymouth State University. Robin is an editor of Hybrid Pedagogy and is a co-founder of the Open Pedagogy Notebook. She has also published on a wide variety of topics, including the Salem witch trials. Welcome, Robin.

Robin: Thank you.

Rebecca: Today our teas are…

Robin: Oh, I thought we were talking about teasing people for a second and I was like, I don’t have a tease. [LAUGHING] What am I teasing?

Robin: No, I actually have two cups of tea in front of me, which is how I like it. One is a ginger tea and one is a sunny orange because I have to stay away from the caffeine at a certain hour of the day, so I’m all herbal.

Rebecca: Sounds like a nice combination.

Robin: I know, I’m just taking one and then the other; it’s perfect.

John: And I have a Tea Forte Black Currant Tea.

Rebecca: I have a Jasmine Earl Grey, that wasn’t there before, so I gave it a try.

John: It’s been there for a year.

Rebecca: Wow, it’s been hiding under the big pile of tea that we have.

John: In exactly the same tray…

Rebecca: No…

John: I just refilled it today.

Rebecca: Oh, you know what, it was the box sitting on top that you didn’t take back after we refilled the tray.

John: Okay. [LAUGHTER]

Rebecca: That’s where I got it from.

Robin: But your listeners are probably like, how could you not see a box, but if they saw this table, my jaw was on the ground; it is a really quite an impressive tea table that you’ve assembled here. You should be very proud.

Rebecca: We don’t mess around.

Robin: You do not mess around. I would use even other words but I know… public… this is some serious hardcore tea happening here. [LAUGHTER]

John: We invited you here to talk about your work with Open Pedagogy. For those that are unfamiliar, can you explain what is meant by Open Pedagogy?

Robin: Sure, which is such a funny question really because if there is a thing—and I could just say it—because there’s a lot of productive disagreement in the community about what Open Ped is; it’s one of the reasons that my colleague Rajiv Jhangiani and I started the Open Pedagogy Notebook because it’s more of a collage approach to defining Open Ped by people doing and practicing in different ways and then sharing that, but if I had to boil it down I would say it’s really about access, both to knowledge and to knowledge creation… so, the idea that we remove barriers to sharing resources and helping people access conversations and find pathways into education, but then we also try to find ways to amplify student voices to make them contributors to the Knowledge Commons and not just consumers, and I think it’s pretty salient right now as students are really in some ways maybe being pushed into these kind of training and competency models that are really about kind of downloading information and instead Open Ped suggests that we really want students to interact with knowledge and shape the world that they’re going to graduate into, not just train for it.

John: So they’re more actively engaged in the academic conversation?

Robin: That’s the idea. Right. In some ways there’s a lot of kinship, I think, with connected learning and with the idea of involving students in their academic and professional networks. Right from the beginning, because even as people who are new to our particular field, they have so much to offer and as an interdisciplinarian, we talk about that all the time that the outsider’s perspective is beneficial—it’s part of the reason you assemble an interdisciplinary team to tackle a problem and newcomers to a field ask sometimes questions that really can illuminate the challenges that a field is facing in new ways, so I have found that even the most beginning introductory students in a particular area have something to contribute both to the field itself and also in terms of helping their peers in terms of, for example, making educational materials. Students are really well positioned to make great educational materials ‘cause they understand better than anybody what’s hard to understand about a certain area.

John: They’re not subject to the curse of knowledge…

Robin: The curse of knowledge. [LAUGHTER] I have that curse, John.

John: We all do to some extent.

Robin: But it is true like when especially when I was teaching Early American Lit and you just finished your PhD and you start teaching and then you teach something for 15 years and no matter how gifted of a teacher you are, sometimes you’re like, “I don’t understand how they don’t understand this,” like “what’s hard about this?” and of course they really understand what’s hard about it, so when we did student projects where students were working on a textbook that we were crafting together, they really made some great materials for each other that I think were far better than some of the lectures I would have prepared or had prepared over the years.

John: So tell us a little bit about how you got involved in that first project you had?

Robin: I was at, ya know, one of those faculty development events that you guys might be aware of and they had brought in, ya know, a keynote speaker, and I don’t want to say I wasn’t prepared to be impressed, but it was a technology oriented conference and I was definitely one of those curmudgeons that was highly skeptical about how useful… or actually more skeptical about the ways technology was being deployed, so I was prepared to be mad, that was how I came in the room, and I actually still think that’s generally my positionality with technology is like, I’m pretty prepared to be skeptical at all times. But the keynote speaker happened to be Cable Green from Creative Commons and I had this really just pivotal “aha” moment when he was talking about the Creative Commons licenses where I realized that my students were paying 90 bucks every semester, each student, for access to public domain Early American Literature and my heart just sort of fell on the floor. It’s like why are we paying commercial publishers when all of this stuff is out of copyright. So, some students and I spent that summer before the next fall’s class rebuilding the Heath Anthology of American Literature out of public domain texts that we found online and we did not build a hearty replacement for the Heath, but we built a sufficient replacement and when we got into the class the students were super psyched that I had saved them 90 dollars, which is totally real money to my students and makes a big difference. They were grateful but they did not like the book, because it had nothing except public domain literature, so there were no introductions, no maps, no footnotes, no glossaries, no “Don’t worry, I know this doesn’t make sense to you, but let me walk you through it” kind of ancillary scaffolds. So, of course, it seems obvious now but at the time we thought we were rock star smart when we figured out like, hey, the students can make this stuff for the book, and so the students worked ahead, different pods of students would work ahead a week or two and build wrap around materials for that chapter and we got there the rest of the students would use it and, of course, it just ended up being 500 times better than the Heath Anthology of American Literature, partially because they were excited that their classmates were reading their work instead of putting it in Moodle or Canvas or Blackboard, where things go to die. So, they got excited about doing what David Wiley and others have called these non-disposable assignments and then they start getting creative, they start making little videos. Ee’d drop those in… two-minute intro to the Haitian Revolution or whatever. I put a little app in the sidebar called Hypothesis where students can annotate and so they liked that and then at the end of the semester people are like, “The best part of this class was the textbook,” which…

Rebecca: Which they made.

Robin: … they made… and they never said about the Heath anthology. So, that transformed my pedagogy partially because I was excited about making all sorts of access oriented changes in our program; we opened a food pantry at the same time as we did this, so we were thinking about lots of things in terms of accessibility of resources for students. But, in terms of thinking about not dumping my student’s work down the digital toilet every semester, it gave me stomach cramps when I first thought about what I had been doing. Every time I taught the course it was the same course, the student’s contributions weren’t transforming anything—it was no wonder that some of it felt dry to them. There was a lot of hoop jumping, and I still think I was a good teacher, that wasn’t like I was bad, but this idea of really empowering students to really, truly engage with the fields and the materials and shaping how the course runs has really changed everything for me.

Rebecca: There’s something that you’re talking about… the idea of building the textbook together, but then the course material is sort of the same from semester to semester and the materials are created by the students, so how does that continue to transform semester to semester so it doesn’t feel like it’s a one-off project… that it continues to evolve and it continues to be a value and that students continue to learn new things?

Robin: Such a great question and I get it a lot because people—I think in some ways mistakenly assume—that that first build was the exciting build, but totally that first build was the annoying build, right, because we got to do a lot of legwork tracking down these texts. We had to do a lot of copyright discovery, like “Can we use this version of Thomas Jefferson or not” and it wasn’t all that exciting, and that’s still ongoing. So Rebus community, who’s working on that textbook now, they took our version and they’re building it out; they’re still doing that kind of discovery in that initial work, and to be frank, that will never end. Even with Early American Literature you’re constantly discovering, changing excerpts, building things out, but to me what’s even more exciting is, for example, there’s a whole unit at the beginning of Native American Oral Tradition and asking critical questions about what’s the genesis of American literature. So, at the same time as I was teaching that that semester we had the Dakota Pipeline stuff happening and lots of stuff about water protectors and about native history so you can relate things to current events that way, but also think about when I finished the course, my colleague Abby Goode, who teaches Early American Literature, but very much from an environmental and sustainability perspective, she kind of remixed the whole book so that now it was about the environment in Early American Lit; they chose some different texts, they reframed the introductions. Her book, in my opinion, is quite a bit more coherent than ours was, which was more of just a collage of things. So, there’s all sorts of opportunities for how you shape and reshape, and, of course, what students are learning there, which is really the point of those survey classes in English is that there is no American literature; there is only the canon that you intentionally assemble. There’s a lot of politics and a lot of editorial choices and each semester it could look different and that’s an important lesson because the Heath anthology was not a neutral objective collection either and I think that’s been a helpful way for them to encounter the building of canon.

Rebecca: I think it’s really easy for people to imagine how this works in the humanities. Can you provide some examples or ways of thinking about being open in the same way in STEM or in business or other disciplines that might not latch on quite as quickly?

Robin: Yeah, and my colleague at Keene State, which is one of our sister institutions, Karen Cangialosi, published a wonderful article recently called “You Can’t Do That in a STEM Class,” which is basically the answer to your questions. We should just stop answering questions and you should go read that article now. But really, the open dcience movement is a huge movement, in some ways dwarfs open education and I think climate change is a really good example of this, but also just open access publishing, the idea that in order to have scientific breakthroughs we need to have the public sharing of scientific knowledge and collaboration in science and so bringing our students into that early as opposed to saying, “Here when you’re a student you’ll be confined to this one class and stuck in this one book, but then when you become a scientist we assume you’ll just understand how to become part of this larger, more public scientific community.” That makes really little sense. So, what you’ll see in classes by people who work this way in STEM, and Karen’s a good example, is that their students are working on issues that are of critical importance and they’re putting their research and ideas into the commons and asking mentor scientists to engage with them. So, we understand that our students are not always going to be doing top-level research; the next breakthrough in diabetes research is not necessarily gonna come from a sophomore. Although occasionally you hear those stories, right, but really what they’re gonna do is they’re gonna ask for guidance, they’re gonna ask for help, they’re going to amplify other scientists’ work and translate them for their communities so that a new generation of scientists can get access to the issues and that’s how we’re going to assure that our scientists are working for what I might call the public goods. So, we’re seeing lots of people using blogs for this purpose instead of just doing their labs in a vacuum or whatever, sharing some of that work and creating sites together, or working in experiential ways through internships. This is why I sometimes bristle about OER being kind of like a cheap or free textbook movement. It’s really very much about a public knowledge commons and how we bring students authentically into that, so when my students are out in the field… and I teach interdisciplinary studies now, so my students are pretty much not in the humanities, to be honest. I have lots of business students, lots of students working in marketing, lots of students in allied health, physical therapy; they’re all in my program, and the work that they do in our open textbook and with OERs is one thing and we do create all of our own materials for the program, but beyond that they’re also creating capstone projects that are generally online, often openly licensed; of course they hold the autonomy to make those decisions for themselves. At that point they’re pretty educated about how open they want to be and then they’re also working on applied projects out in their fields and I see all that stuff as part of the same way that we engage our students in the public world of knowledge.

John: And science is often taught, or at least it’s perceived by students, as this body of knowledge they have to memorize rather than this ongoing dialogue and a series of active research projects, and when they are more engaged in the process of making those connections it’s likely they’ll learn it better and they’re likely to become much more interested in the subject, because one of the main problems in the STEM fields is that students give up early on. But if they can see the relevance, I would think they’d be much more likely to continue onward.

Robin: Yeah, I think it’s a great point. I’m also the mom of a teenager right now who—is this gonna make my whole family sound bizarre? I’m not sure… but she’s got an obsession with taxidermy, so if you go into her room it’s all—I’m not joking; we’ve got boars, we’ve got bison, we’ve got deer, we’ve got every pelt you could imagine, and then her bookshelves are filled with skulls and bones…loves it. She’s out there digging for bones from the time she was little, researching which skull is this, what skull is that; she doesn’t like science, though; that’s what she tells me: she doesn’t like science. I’m like, you do like science, you nutty kid.

John: You’re kind of doing it; this is where a lot of science started.

Robin: That’s right, and so I’ve been waiting for her and she’s had great teachers here and there, but she really did finally have a biology teacher last year who helped her understand that she does love science, but before that she thought, I don’t really like these worksheets and I don’t really like memorizing these tables, and she’s an interactive person. So, I think there’s a lot of compatibility between open and active learning and experiential learning and high-impact practice and all these buzzwords. People call me an advocate for open, which I am, but really I’m an advocate for learners, like paying attention to the kinds of things they are constantly telling us that they need in order to be successful. While we’re over here shopping for some kind of software program, they’re sitting right here telling us, I’m hungry or I can’t afford my materials or I don’t feel like my voice matters or I don’t know enough to be useful here, so you just tell me what I need to know.

Rebecca: Or I’m not represented.

Robin: I’m not represented. That’s a huge one because when you transfer to this mode of learning, it’s a little bit the sort of Wikipedia model, although Wikipedia is a horrible example because of representation in Wikipedia and the stats we have on that, but the idea that you can pay attention to all the voices in your community but the open movement is really wrestling with this right now to figure out how much is about open and licenses and sharing and how much is about creating an ecosystem of inclusivity, access… the kinds of things that truly do shape a commons, which we mostly don’t have in education, so the commitment, I think, is for me is less to the technicalities of open and more towards the long-game vision, which is really about how do we bring more voices to the table to engage in the community for whatever the community’s needs are.

John: I wanted to go back to a point you made earlier; it reminded me a little bit about some behavioral economic studies and I haven’t thought about this before, but I think it’s relevant. There was some interesting experiments done by Dan Ariely a while back. Dan Ariely calls this the IKEA Effect and he notes that when Duncan Hines first started selling cake mix they sold horribly and the reason was you just added water, you stirred and you baked and people didn’t feel that they had created something, so they changed the mix so you had to add an egg to it and stirred and mixed in and baked it, but by the simple act of breaking an egg and mixing it in, it felt like they had created something in a way it wasn’t where they just added water. One of the experiments he did was he had people create these origami by following directions… of paper cranes and he asked them to evaluate how much they thought their creations were worth and then he swapped them and he asked them to evaluate someone else’s creation and people valued their own at roughly twice as high as the others across the board, and then he changed it in another iteration of this and he took some of the directions out… so it was really hard to replicate and objectively, when people evaluated the other people’s that time they rated the value of them much lower in terms of how much they were willing to pay, but because they put more work into building these things themselves, they rated their own creations much higher, and the simple act of creating something gives you this feeling of ownership and value that I think would be a useful part of this in terms of getting students much more engaged with the process and more engaged with their own learning.

Robin: Yeah, so I would say two things about that super fascinating set of stories that I’m totally going to use all the time, maybe tomorrow, when I speak with your faculty. So the first is that we run a customized major program where students create their own majors and the cake mix effect is enormous in our program; we have almost a hundred percent retention in our program, which I think is so very much attached to this idea that if you create it yourself that sense of ownership is huge. So, the one way I would revise your stories is the Duncan Hines model is kind of a slight of hand, right, you know, we could of just put the egg in there, and so that’s the part that I’m always wrestling with is this no hoops and mean it, so it’s actually kind of a big leap because in education we’ve known for a long time that we want to build student ownership, but we do a lot of tricks to kind of say, doesn’t this look like ownership, so how do we authentically do that instead of just sort of fake removing the egg and that’s where I think you see a lot of institutions push back—they are happy to make their students feel empowered, but they are not happy to empower their students.

John: That’s scary.

Robin: Yeah, it’s very scary and we talked about student-centered learning; when I started doing student-centered learning I was like, “Chumps, you are not doing no student-centered learning because once you center your students the whole course changes, falls apart a little bit.” It’s also the magic of tenure; it’s very risky to do a lot of the things that I’ve been blessed enough to be able to experiment with it, which have paid off, I think, hugely for our students, but there’s a lot of pushback sometimes from students and oftentimes just from institutional structures that can’t really accommodate learning that looks like this very easily, so that egg is kind of the whole thing there, right, and I love your metaphors; I’m gonna borrow them.

Rebecca: When you want to be authentic that also means that you have to be ready to completely change any plans; it’s like, oh, now we’re going down this rabbit hole that I didn’t know we were gonna go down, but I guess we’re going there and we’re all gonna go together and be open to that.

Robin: Some people are like wired for that, like my husband is a sculptor and that’s kind of the studio ethos, but so not how I came to teaching. I mean, I didn’t have my lectures written out but just super organized and when I would come to an epiphany it was always an epiphany that I had planned for many weeks… I think, “Oh my gosh, this reminds me of this thing, can you believe it?” And of course I knew all along that we were gonna be arriving at that epiphany, so when students would move in a different direction, even if I could tell at the time it was brilliant, you would have to pull them back to the place that you were going, but I have definitely changed my mode of operation because the content, really whatever you teach, it doesn’t exist in the world in 14-week packages, so the idea that you can’t do such-and-such because you’re going to miss this key fundamental thing is just bizarre when you think of the scope of knowledge, so I understand people wrestle with accreditation and we wrestle with standards and all these things are realities. But, for the most part, I think really radically meaning a lot of those buzzwords that we use is revolutionary. If you read your mission statement for your university and then you actually do some of that stuff, it’s gonna be crazy; nobody’s actually doing the things they say that they do, in my opinion.

Rebecca: A couple of weeks ago we had an episode about metaliteracy, which expands the idea of information literacy to include the idea of creation, so the idea of becoming more literate in the making of things as part of that information literacy process, which is clearly very connected to the idea of being open, especially when your students are creating this content and creating knowledge. The question that I have is one that I’m wrestling with currently as an educator who’s really about access as well, but I’ve been focusing a lot on access for people with disabilities in thinking about accessibility in that way, digital accessibility and learning those skill sets and where those come in and how do we make sure that things are visually organized and consistent so that an experience through these things that students are making is a good one for everyone who comes afterwards as well?

Robin: I am so happy that you asked that question because this has been my last three weeks; I have barely slept because I’ve started getting so excited. So, for probably the last year or so my own personal challenge has been to think about accessibility in terms of making our materials more accessible, so I’ve been learning about how screen readers actually work in order to fix my own syllabus to redo a lot of annoying things because I didn’t realize you had to use the headings to make things easier. So, I’ve just been learning that basic stuff and that’s been just a long, slow and interesting process. One of the last things that I really hadn’t learned about at all or hadn’t even really thought about was in giving presentations, which I give a lot, I had to think about slides… and so at Open Ed ‘18 in Niagara Falls… I wasn’t there, which is actually an important part of the story because one of the keynoters was Jess Mitchell, who is kind of a mentor of mine in terms of accessibility stuff—she’d be a great guest—and Jess gave a really moving and powerful keynote focusing mostly on inclusion in open and she is very much an accessibility advocate and what was amazing to me as someone who was not there and didn’t see a recording was when I looked at her slides afterwards I was able to experience really the whole keynote because they were designed to be accessible to folks who were in the room, text was organized in a certain ways and things were very clear and I came away really grateful for how she had set up these slides, which was interesting because they were really different than the kinds of slides I make. I had always prided myself on like “Robin made some fancy slides,” you know, they’re like just pretty and like visual impact and bold images, but because they were, I think, graphically designed in a lovely way, I mistakenly thought that that meant that they were actually accessible because they were clear in certain kinds of ways, but they weren’t, they weren’t set up well, so what happened was when I learned and saw in action some of the techniques that she was using, I started to look at this keynote that I was giving the other day—I had like two days left—and now the keynote was ready to go, all the slides were made; I looked at the slides and I was like, oh, crap, you know, no… So, I thought I’ll just redo these slides real quick, but what I ended up doing was really learning about the accessibility changed everything about how I approached the making, which actually ended up changing all of the ideas in the keynote in this dramatically productive way. So accessibility for me, of course, is not really just about like, oh, you have low vision or whatever; it’s very much part of this access broadly-writ idea… that openness… But beyond that because it’s built into how we build; it’s really about how we’re gonna design infrastructure and that is actually my passion right now; it’s less about making these materials—okay, so great, here’s an accessible material, great, they should be—but beyond that it’s about let’s just design an ecosystem now with access at the heart. So, in terms of accessibility, none of that sort of retrofitting one-by-one whatever, but also just what would happen with everything if access for the broadest array of learners was key. I was recently in Providence, Rhode Island at College Unbound, which is just a very cool program for adult learners completing college—they have to have least nine credits to start—but many of them have many more… and there are mostly students of color, mostly poorer students and mostly, I’d say, they seem like over 30 in age and they start their seminars with a hot meal and then after the hot meal they go into their different cohorted seminar rooms and tons of those students have their kids with them and the kids are just a normal part of the learning environment there and the whole place is designed around what kinds of access people needed, what times of day and what services in order to come here to learn and I just feel like everything about the content that we’ll produce and the ways we’ll set up schools and just everything will change if that’s how we build—we build around what I might call human beings, right, which is like the most innovative idea of all, right, it was not technology, it was humans.

John: Audience matters, as Rebecca is fond of mentioning on this podcast.

Rebecca: I almost did it earlier but I…

John: Well, I did it for you this time.

Rebecca: …I contained myself. [LAUGHTER]

John: But, It is important.

Robin: And it’s exciting, I mean, honestly, it’s just exciting because you do realize when you start thinking this way that it is again gonna change everything, right, you’re not just gonna put a caption on your video, it’s gonna be like every single thing is gonna change and that’s why it’s also important to say like, “Here’s how I still suck,” because you can’t just decide to do this and then be done. I’m just learning every single day, I’m messing up every single day and I think it’s better to kind of own that and think of it as a process, which is really invigorating.

Rebecca: To speaking about the process, how would someone get started? What advice would you give someone who is inspired to be more open in their process and the way that they teach and what they put out in the world? What’s the first step?

Robin: Well, the first thing I might encourage people to think about is what excites or interests you here? I think starting with a thing is not really the way to start. So, for example, a lot of times people will come down into our teaching and learning center (where my office is co-located—in the teaching and learning center). So, people will come down—“I need to start a blog with my students”—“Oh, okay, we can help you with that, why do you want to do a blog? “I don’t know; everybody’s blogging.” “Okay, we got to blog.” You really don’t have to blog; you could blog, we could help you, but I think having a sense of the goal: do you want to connect your students out to their communities? Do you feel like that would be valuable for your students? Would you like to lower some access barriers for your students? For me, there’s a lot of excitement that happens when I think about the hardships that we face in public education and trying to make a case for working in more public ways and what public work looks like, so I tried to start with what might excite faculty. So, you can do that on a one-on-one level or when I talk to large groups of faculty I usually start by helping them understand some of the implications of the high cost of textbooks, so if you just say to a faculty member, “That textbook costs a lot,” it’s too abstract. Usually they’ll just say, okay, this was 200 dollars and this one is $180; I’ve picked the 180 dollar textbook, I’m a good person… and they are… but showing them some of the data on what happens to students who can’t afford textbooks, and we have that data collected now and you can reach out to your librarians to access that data really quickly, talking about that with faculty and helping them see this as a social justice issue that impacts whether their students will pass classes, take credits, graduate from college, that I have found is persuasive, but then also talking about engaging their students in the world, really helping them to contribute rather than just consume, become better critical thinkers, all of those things are persuasive. Saying faculty don’t care about cost… I think first of all is not super true, but it’s also like we’re told all the time as faculty, cut costs. Cutting costs does not do wonderful things for learning most of the time. The things we’re asked to cut, especially in our public institutions right now. The age of austerity is decimating to innovation, in my opinion.

John: And the cost of textbooks has been rising at three to four times the rate of inflation for the last several decades?

Robin: Yeah, If you graph it out, I think the thing that I found most shocking was there’s the Consumer Price Index, you know, down below and then there’s the spiky line of the textbook cost and then if you map healthcare—it’s actually in between—it hasn’t been rising as fast as textbook costs, so I think people sometimes find that alarming.

John: Shocking, because that was also rising much faster than the inflation rate.

Robin: Exactly.

John: Going back to the issue of access, the students who have the most trouble affording textbooks often come from households where the parents have less education. Because there’s less early human capital development in those households, those students are already often starting at a bit of a disadvantage and many of them will choose either not to buy the book or wait as long as they can before buying the book. So, they’re far behind when they’re starting their classes and that would be a major factor in their retention on campus.

Robin: Yeah, actually some of this data that you’re talking about comes from the Florida Textbook Study in 2016, which is very persuasive for faculty, I think, but there’s some really new data—Eddie Watson out of Georgia, I believe, that just came out that shows that the benefits of switching to OER in terms of things like course throughput rates, grades and passing and…

John: the drop, fail, withdrawal rate, yeah.

Robin: …that the benefits are especially pronounced for students of color and for Pell eligible students. Some of our most vulnerable learners stand to make the biggest gains when they have access right from day one, and faculty recognize this when you talk to them about it because they are very used to having the small number of students in this side of the room saying, “My check isn’t in yet; I need to wait two weeks until I get paid,” or “I ordered a cheaper version that’s gonna be here in six to eight weeks” or whatever, so nobody’s surprised by it, but to realize that you are actually empowered to solve a problem in higher education is surprising to people and OER actually solves a pretty concrete problem and pretty quickly and the data shows us it solves it pretty well.

Rebecca: So what you’re saying is that OER is the gateway to open pedagogy?

Robin: Well, it’s so funny… [LAUGHTER] I have actually become maybe more famous in the community for saying the opposite because that is actually the party line: catch them with the OER and then show them the pedagogy, but as you’ll see in the faculty development talk that I’ll do here at SUNY Oswego tomorrow, I do that a little bit but definitely I think people are kinda like, “Okay, I’m in, yeah, sure,” and then you start talking about the teaching and learning and that’s when people really kind of come alive and then they shrink back again because they say, “Well, that’s you, because you’re techie and you’ve been doing this forever” because it looks overwhelming and I just want to tell them, first of all, I’m an early Americanist; there is nobody less oriented to this work than I was when I started, but I only heard about Creative Commons maybe like four years ago, like that was the first time I heard of it and now every single thing I do is related to this stuff. The learning curve is overwhelming at the very, very beginning, but the tools that you use and the ability to make these kinds of changes, especially if you do them incrementally. It is really within anybody’s ability and people should trust me when I say that because my husband is a sculptor—he’s a studio sculptor, teaches welding and that kind of stuff and he’s doing all of this now. So, he does OER, but he’s also doing lots of connected learning and his students have their own domains and he is somebody who for the most part does not really even enjoy email, so anybody can engage and I think we need good librarians and good instructional designers and we need to keep funding teaching and learning centers because paying big money to fancy software programs and outside contractors, these are sort of Hail Mary passes to save education. But, in my opinion, teaching and learning and instruction shows real benefits, but we don’t invest in it and we therefore can’t expect to get the full rewards that we could get if we were really focused on working with our faculty.

John: And a lot of the really powerful tools used in these courses are free, like Hypothesis, as you mentioned before. Do you recommend, for example, the use of Pressbooks for OER materials?

Robin: Yes, I’ve been very inspired by the Critical Digital Pedagogy folks out of Hybrid Ped and one of the things they talk about is analyzing your tools and I’ve been really trying, along with my work in accessibility—the other kind of learning curve for me right now has been trying to go through my own tools and gravitate towards not just free but open tools and that’s challenging in some ways; in other ways we’re all ready to go. I favor nonprofit companies like Hypothesis and Pressbooks is Open-source software; I use it through Rebus Community, which is a non-profit OER publishing community that’s developing now under the direction of Hugh McGuire, who was previously with Pressbooks and developed Pressbooks. So, I think the tools should not stress anybody out because the tools will be different next week, right? So, it’s not worth getting too worried if you’re like, this tool it makes no sense, okay, well wait till next week; they’ll be another tool, but it’s good to ask critical questions about if we’re really trying to not just save some cash but to maybe transform into more of a learning ecosystem that focuses on the public good, then we need to build infrastructure that has similar commitments to the kinds of content we might look at or the kinds of processes we might use in our pedagogy. That’s my goal now, is to transfer whatever I’m using into tools that have the same sort of investments that I do.

Rebecca: Speaking of infrastructure… We have infrastructure for students in teaching and learning the classroom kind of side of things, but we also need infrastructure to support faculty who want to be open and do open publishing and do this public good or public discourse methods in general. So what recommendations do you have for helping us move in that direction for public scholarship?

Robin: Yeah, but there’s a lot of myth-busting that needs to happen around open access publishing. Mostly faculty do have some pretty good autonomy, so the promotion and tenure processes that faculty will tell you, “I can’t publish in this journal because it doesn’t meet the impact factor regulations for my field.” Well, those are mostly coming from, like that old joke, “It’s coming from inside the house,” right? Really what this is is about faculty education to help faculty understand that it’s not in the best interest of faculty or knowledge to have the commercial publishing industry stranglehold on academic publishing, but of course faculty are concerned that there are quality issues, they think open access publishing sometimes is like, “I self-published this on Amazon” or whatever, so helping them understand that there are definitely low quality, predatory open access presses just like there are low quality, predatory commercial presses and helping people understand that what you’re really talking about is not whether it’s open or closed but what’s the peer review and what are you looking for in peer review. I think we’re seeing lots of institutions move towards open access policies that give faculty lots of autonomy in how they control their materials, but we need to do a better job educating ourselves about what’s wrong and broken in academic publishing right now.

John: SUNY has just introduced an open-access policy for the whole SUNY system very recently, and (at least at our institution) the upper administration, including the President, the Provost, and the Deans, have generally been very supportive, but it doesn’t always make it down to the departmental chairs and personnel committees and that’s a barrier that, as you said, we’re imposing on ourselves and it’s tough to get through, especially if you’re a junior faculty member coming up for tenure.

Robin: That’s right, and usually I tell administrators that I work with, it’s great that you’re supportive; please don’t tell anyone, you know, because we don’t want these to be top-down initiatives, they have to grow from the faculty and I don’t mean that again in the kind of Duncan Hines egg sense—like a fake way; it’s important that faculty steward the new era of academic publishing—that matters; that should not come from administration, should not come from state legislators. The state legislators are only too excited really to say, everyone must use OER. I went to our board of trustees and they were really happy to give us money for open and they said, we’re gonna pass a resolution that everybody needs to consider an open textbook—I said, thank you, I don’t want your resolution; I’m very grateful… Because it really is important that we do the education at the source which really is for the most part with faculty and actually with students, I think, is where it matters and we’ll grow it that way and the reason I have hope is that I’ve never talked with anyone for any length of time and had them say at the end, well that’s horrible and stupid. There’s lots of nitty-gritty problems to iron out and the open access community does not yet know exactly what the best path is for funding open access presses or all sorts of issues, but it’s very hard to find someone to say to you what you’re saying is horrible, so I think that we will see huge transformation in both OER and open access publishing in the next five to ten years, but we need to grow it with our people.

John: And some of the STEM fields have led the way there; the National Institute of Health and all their grants require

Robin: NASA, the White House….

John: …that things be publicly available and in public access.

Robin: Yeah, and of course those publishing models are a little bit different because there’s so much grant funding in science and the public has a right and that might be a little bit different than, say, a monograph by a historian. When we talk about open we always want to talk contextually, I think, and specifically about what makes sense for, I think, two groups: the public and the knowledge—thinking about both knowledge and users for every different example.

John: Where do you see open pedagogy as going in the future? It’s a relatively recent area and you’ve been very actively involved in this, but where do you see things going in terms of new and interesting directions?

Robin: I cannot answer that question because it boggles my mind… the question you’ve asked and I don’t think I can answer it and it wouldn’t help me to go away and think about it. I think what I could answer is where I hope things are going, and I feel very strongly that there needs to be a robust connection between open education and public education, and I feel like we are in a very dark time where our public education channels are being insidiously co-opted for private profits, and even in some of our public institutions you’re seeing the language of public just dissolve, so we’re seeing college presidents saying, “Yeah, that’s over, that era is over; we’re not going to get any more public dollars; it’s dried up; we need to get corporate money, we need to do partnerships, we’re gonna fund ourselves in these new private ways.” I think this is our chance to intercede in what I see is a very downward trend and I think open education has some really incredible possibilities for helping us articulate what public practice looks like and if we can articulate what public practice looks like, the fact of the matter is, and I do not think this is an argument, I think it’s a fact;—it’s true—I really think it’s true that it is in the public’s benefit for the public to fund the kinds of public work that we’re talking about here for both students and researchers and if we could help explain why by being a little more coherent for ourselves with what it means to work in and at public education and research, I think we’d have a better chance at making that case for the public. So that’s where I’m hoping to take all of this is to say really what we’re talking about here is a resurgence for public education at the K-12 level, resistance to the charter takeover and higher ed to say it’s time to reclaim a public mission for our public universities and fund them appropriately and realize that innovation comes from people and not from private, gated… Right now, the idea is that all things innovative thrive in the market and I think that’s because we’ve been intentionally starving and strangling our publics. Wow, that was like really radical.

Rebecca: That was good, yeah.

John: And it’s an important message.

Robin: I think it’s coherent and I think it’s persuasive; I feel people come alive when I talk about it, but we need also some national leadership on this both from inside education and inside government and I don’t just mean nationally,—the United States is in a squalid mess right now, which it is—but just even in the Obama years and whatever like who are our champions for public, where is that coming from? I’d like to see more mentors and like to see our college presidents use this kind of language—if you’d like to hire me to be your college president… [LAUGHTER] Call me. [LAUGHTER]

Rebecca: Maybe that’s a “what’s next.” [LAUGHTER]

John: So, we usually end these podcasts by asking, what are you doing next?

Robin: Tomorrow, I will be here at SUNY Oswego. You know, the question of next is a really hard one; My own personal life has been changed so radically by this. I never saw myself leaving the English department, I never saw myself having a whole in some ways second career. I used to be asked to be department chair because it was your turn, not because I was anything special, and I would go under the table, you can’t make me do it. I declined everything. I really think, though, people with a grassroots passion for doing this work need support at higher levels in higher education. In terms of me personally, I started thinking about trying to step into some of those roles and I can’t say I feel sort of super personally excited about some of the aspects of that work, but I know that even though I see this as a grassroots movement,—and I do use that word—it’s really hard to change institutions, and in order to do it we’re gonna need to get people at every level to care about these kinds of things and so I’m inspired by people like Tressie McMillan Cottom and Sara Goldrick-Rab and they’re faculty, but they step out to set a national example, and I’d like to maybe think about trying to move this stuff a little bit more institutionally, as opposed to just inside of programs or with particular faculty development events. I’d like to see some institutions really step out and lead. SUNY is doing a great job. You guys have about 48 of your 64 institutions, I think, actively engaged and you are careening towards some system-wide impacts, partnerships with CUNY, statewide conversations; this is where I think things really get exciting to me.

John: The community college and SUNY have really been leading and they’ve been very active in doing this. The four-year colleges have been moving, but not quite as quickly and the university centers have a bit more inertia. So, SUNY has been making some really great efforts in providing incentives and doing a lot of encouragement and the workshops they’ve been funding have helped to try to get more grassroots movement, but it’s not as quick as many of us would like, but it’s much faster than it was a few years ago.

Rebecca: Incremental change is still change?

John: It is.

Robin: It absolutely is. Someone was telling me… Is this an economics thing about the parable of the ant, that ants are going up a hill? Okay, somebody on Twitter, you just sent me this,—I’m losing my brain now—but anyway, an ant is going up a hill and when because of the position of the ants eyes they can’t assess the whole hill, so all they do is at every point they could assess, I want to get to the top of the hill, and all they can assess is, okay, this is the next step that I take, so then the ant gets to the next step and it assesses again and that’s the kind of way incrementally the ant will get up. In that sense the ant doesn’t really even have to know where it’s going; it is just able to constantly resurvey and take one more step and I found that really reassuring when someone sent that over to me today. That’s kind of a metaphor for how you can keep going when you don’t always know exactly where you’re going, and also to your other point, community colleges are clearly the national leaders in this work and I find that really good for education because community colleges are actually really good at teaching and learning and it’s important to look at what they’re doing with open to learn our lessons, but also we can just learn a lot from partnering with our community colleges more effectively.

John: And they’re often the first point of access for first-generation students who may find it difficult to go directly into a four-year college, and they have many of the students who most need that sort of access.

Rebecca: Well, thank you so much for spending time with us and engaging us in this really great conversation; I hope that incremental change becomes much bigger increments as we hear more people and more people get on board.

John: And if you get one person in department doing it, it’s a whole lot easier to convince others to try.

Robin: There’s no secret trick or no secret sauce, it’s just people, so every time somebody as a human gets invested you actually get a lot closer to where you’re going, I think. It’s exciting, it’s exciting, and thank you guys for having me because this like fancy stuff and I feel very listened to and I’m gonna put all sorts of pictures on Twitter of myself in front of these microphones. [LAUGHTER]

John: Okay, well thank you.

Rebecca: Yeah, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast please subscribe and leave review on iTunes or your favorite podcast service. To continue the conversation join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kim Fischer, Brittany Jones, Gabriella Perez, Joseph Santarelli-Hansen, and Dante Perez.

[MUSIC]

49. Closing the performance gap

Sometimes, as faculty, we are quick to assume that performance gaps in our courses are due to the level of preparedness of students rather than what we do or do not do in our departments. In this episode, Dr. Angela Bauer, the chair of the Biology Department at High Point University, joins us to discuss how community building activities and growth mindset messaging combined with active learning strategies can help close the gap.

Show Notes

  • “Success for all Students: TOSS workshops” – Inside UW-Green Bay News (This includes a short video clip in which Dr. Bauer describes TOSS workshops)
  • Dweck, C. S. (2008). Mindset: The new psychology of success. Random House Digital, Inc.
  • Barkley, E. F., Cross, K. P., & Major, C. H. (2014). Collaborative learning techniques: A handbook for college faculty. John Wiley & Sons.
  • Life Sciences Education
  • Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of personality and social psychology, 69(5), 797.
  • Steele, C. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American psychologist, 52(6), 613.
  • The Teaching Lab Podcast – Angela Bauer’s new podcast series. (Coming soon to iTunes and other podcast services)

Transcript

Coming Soon!

36. Peer instruction

Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know (and don’t know). Did you picture a lecture class, students taking a test, or students writing? In this episode, John discusses three ways in which he has been using peer-instruction in his classes: classroom polling, calibrated peer review writing assignments, and two-stage exams.

Show Notes

Transcript

Rebecca: Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know. Did you picture a lecture class, students taking a test, or students writing? If not, stay tuned, this episode explores ways to use peer-instruction to transform the learning experience.

[Music]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[Music]

Rebecca: Today’s guest is my co-host John Kane. John is the director of the Center for Excellence in Teaching—that’s not even right…

[LAUGHTER]

John: …Center for Excellence in Learning and Teaching at SUNY Oswego.

Rebecca: Yeah, woops! Welcome to your own show, John!

John: Thanks, Rebecca.

Rebecca: Today our teas are…

John: Prince of Wales.

Rebecca: Oh, that’s a good one.

Rebecca: I have Golden Tipped English Breakfast today.

John: Excellent.

Rebecca: One of the areas you’ve been teaching experimenting in, and that I’m fascinated in, is peer instruction. Can you tell us a little bit about what peer instruction is and why you’re drawn to using this methodology in your courses?

John: Peer instruction involves using peers to assist with instruction, where students explain….

Rebecca: Thanks John.

[LAUGHTER]

John: …where students explain things to each other. One of the issues that we have is that, once we become experts in the field, it’s very hard for us to express things in terms that are easily understood by students. There’s a “curse of knowledge;” once you become adept at something, it’s really hard to explain things at a level that’s appropriate to the level of understanding that students may have. There was a classic study done in which a researcher gave people a list of songs, very well-known popular songs, and asked them to tap out the beats from that song.

Rebecca: Oh, I would fail…

John: …and then before actually seeing if people would recognize it (who had the same list), she asked them to make a prediction of what proportion of people would understand it based on their tapping… and they overestimated that by a factor of I believe, somewhere around 20 times. Basically, it was purely random if people happen to guess it. But the issue is, once you hear something in your own mind, it’s clear to you, but it may not always be clear to the people who don’t have the same rich net of connections. When students are explaining things to each other, they benefit from taking a position, arguing that position, trying to filling gaps and they’re also explaining in terms that are appropriate for people at their level of cognitive development for people who have a similar background in terms of what they know and their prior knowledge.

Rebecca: Sounds like a really good way to expand and refine mental models and also just develop better metacognition. Because, as soon as you go to explain it, you realize what you don’t understand.

John: …and if you don’t understand it yourself, your peers will often help you understand. they’ll say: “Well, you haven’t considered this…” and that sort of interaction is one that doesn’t work as well when it’s instructor to a large group of students. But, it does work very well one-on-one.

Rebecca: You’re known on our campus for teaching really large lecture sections. Implementing peer instruction in a large setting can seem pretty daunting, especially to someone who teaches smaller classes like I do. What strategies do you use?

John: The most commonly used one is to use clicker quizzes… and I use a methodology that Eric Mazur developed slightly over 20 years ago, where you ask the students a challenging question… you try to find questions that about half of them will get wrong… and over time you can develop that, you can come up with a pool of questions that fit somewhere in that range… and you let students first vote on the response themselves after they’ve had a little bit of time to process it, and then you look at the results. If you see that 90% or more of them got it correct or even 80% or more, you can just go over it and move on to the next topic, because most students understand it. But, if you see that somewhere around half of them get it right and somewhere around half of them get it wrong (plus or minus 20% or so), then the next stage is to let them explain it to each other, and that’s where the peer instruction comes in. When you have students argue it and take a stand and a position on it, we get a very significant gain and improvement when we then let them vote on it a second time… and the usual practice is not to reveal the poll results or the answer until after they’ve had that opportunity to engage in that discussion.

Rebecca: Just make sure, to make sure I understand correctly: you do the poll, you see the results as students don’t see the results…

John: Right.

Rebecca: …based on their answers or their responses when you decide whether or not they do the peer instruction piece. How long do they usually talk to each other about the topic?

John: It depends on the problem and normally I will have some undergraduate TAs and I’ll wander around the class and see what they’re talking about, listen in, answer some questions from them and the TAs will be doing the same thing…. and it’s usually pretty clear when they’re coming to a consensus. You can see them reaching for their clickers or their phones and getting ready to vote, so generally it may only be a minute or two, it could be longer… it depends on the complexity of the problems. Some of the problems require a bit of effort and require some calculations, but normally they’ve already done that… so, the second stage, where there’s a discussion, you can hear the volume build-up and then as they’re approaching solutions and consensus, it tends to drop back down again. It’s fairly easy to get a pretty good read on where they are and when they’re ready to vote again.

Rebecca: I imagine that you would really need to keep your ear to the ground, otherwise chaos could ensue. Because now, if they’re finished talking about the problem and there’s still time, then they could easily derail if you’re not quick to get back to the clicker question.

John: Right, and normally the time is generally held fairly tight. I suspect sometimes it’s only 30 seconds to a minute, other times it may go up to a couple minutes, but if I see them getting distracted and doing other things, the polling starts immediately.

Rebecca: Obviously technology is your friend in this particular situation. Can you talk a little bit about the technology you’re using to manage this many students all at once?

John: Here, we’ve adopted iClicker as a campus standard, so we use that in pretty much all of the classes where we’re doing polling and there’s both a physical radio frequency clicker that students may buy or they can buy an app and pay by the semester or over four years for the use of the app.

Rebecca: How do you make sure that the cost doesn’t get prohibitive to students?

John: That’s an issue, and it’s been a major source of concern…

Rebecca: They’re not very expensive, right?

John: Well, they can be expensive. A new clicker costs somewhere around $40. A used one can often be purchased for $15 to $20, sometimes less… and the apps I think, are somewhere around $12 to $15 for a semester and I think about $35 for four years.

Rebecca: …and you can use the clickers in all of the classes, right? So if multiple faculty member(s) are using all the same system, then the investment is a good one for students.

John: …and that’s why we have a campus adoption because in places where you don’t have that, students might have to buy two or three or four different clicker systems in different classes. So, once they buy the clicker for one as long as they hold on to it, they can use it in classes for the rest of their career. Almost everyone in the economics department, for example, now uses clickers, so if they’re economics majors or business majors, it’s very likely they’ll use them in multiple courses. The cost is much more tolerable when it’s spread out over multiple classes.

Rebecca: The other area where you do some peer instruction in these large classes is in writing. Which seems kind of crazy. You have all these students in this big classroom and somehow you manage to do writing assignments.

John: Yeah, my large class generally is somewhere between 350 and 420 students. At one time, for actually about a decade or so, I was giving weekly online discussion forums. But grading that or evaluating that and providing feedback was taking an awful lot of time…probably 30 to 40 hours a week. So, I pretty much…

Rebecca: A full-time job in and of itself…

John: I stopped that a few years ago and, a few years back, I replaced that with calibrated peer review assignments. The calibrated peer review system is something that Eric Mazur talked about while he was here… a visit in 2014… and when he mentioned it, a lot of people got excited. The way the system works is that you create an assignment, you store it on a central server at UCLA, and then it’s something that other people can adapt and use and modify—it’s released under a license, which is similar to a creative common license within the system… and you create the assignment… you create an evaluation rubric for the assignment… and you have to be really careful in designing that to make sure it’s one that students will be able to apply, because other ones that do that… and then you create three sample assignments yourself: a low-quality one, a medium quality one, and a high quality assignment… and you have students submit their own assignments first (according to the rubric and guidelines you provide to them)… then they go in and they evaluate the three that you’ve done. They’re given in random order, and they’re assessed in terms of how closely their evaluations match yours. That’s the calibration part. Students receive a calibration score based on how similar their evaluations are to the ones that you assigned to the sample responses. Then after they complete that stage, they evaluate each other, using the same rubric, and a weighted average of those scores is assigned as a component of the grade. They’re graded in a number of dimensions. One is based on the weighted average of the peers, where students who had a high calibration score will have evaluations that rate more highly in evaluating other students. They’re also rated in terms of how closely their evaluations match the others during that stage. So, if their evaluation is an outlier… much higher or much lower than other students… they lose some points on that… and then after they evaluate the other three students, they rate their own work… and one of the goals of that is so that they have improved metacognition. That, by the time they go back and look at their work again, they’ve rated three works by the instructor and three assignments done by their peers and then they’re asked to evaluate their own using the same criteria. What’s really interesting about the calibrated peer review process is their grade on this is tied not to whether they give themselves a high or low score on this, but it’s how close their self evaluation comes to the weighted average of their peer evaluations. So, they have an incentive to try to look at their work more objectively, and not try to game the system… because if they score their work too high or too low, they could end up with no points on the self-evaluation stage. So, the closer they get to the weighted average of their peer evaluations, the higher the score will be on that component.

Rebecca: I think that’s an area that we often see students struggling, is being able to effectively evaluate their work or other work. So, really training them to use a rubric and understand and think about what’s important or what’s not important about particular kinds of assignments or particular kinds of work could be really valuable to students in a way that we don’t really have other systems to do that.

John: The nice thing about this is it scales really easily. There’s a lot of upfront work in creating the assignments, creating the rubric, and a really good practice is to test them thoroughly before you give them out the first time. What I normally had done is asked some of my peers to look at that, some of my colleagues to look at it, and sometimes I’d have some upper-level students were…. and this does give students a little bit more reflective practice, where they get to look at their own work a bit more critically, perhaps, and reflect on it and see how they’re doing compared to how other students are doing in the course… and I think that’s helpful.

Rebecca: I think that the rubric would probably be a challenge to make but I think what would be more challenging is putting up those different assignments that are scored at different levels at the very beginning as your calibration tool. What strategies have you developed to make those in a way that it doesn’t take forever?

John: Well, I only do this three times in a semester, and once you’ve done it once, if you design it in a way so that it won’t go stale… and I generally have students, for example, find some articles in the news in the last six months that relate to a topic that we’ve talked about, or I ask students to find some examples in their own life to illustrate behavioral economics concepts in one of the assignments, for example…. where it’s not something that they could easily copy and paste from other people’s work. Because, there is always a concern with academic dishonesty and so forth. You don’t want these things showing up on Chegg or any of those other systems, where it would be easy to copy and paste good responses. So, I’ve tried to design assignments where once they’re done, they can be used for multiple years in one form or another. I modify them each year based on how they work. But perhaps a more serious problem is what happens when students really don’t like the evaluations. One of the things I’ve done when I’ve used this is to have three of these assignments, but I drop the lowest score… because, sometimes people will get some scores back that they didn’t expect or they may have neglected to look at the rubric I sent them and they may have omitted a major part of the assignment and ended up losing quite a bit of points all the way through that. But, as long as one of the scores is dropped, they have the opportunity to learn from their mistakes and do a little bit better. But, there are procedures built-in that make it easier to catch any outliers when you have someone who is just rating everyone extremely highly or rating everyone really poorly—inappropriately highly or poorly. There are tools in it which will give you a list of all the cases where there’s a high variance across reviewers or where someone happened to be evaluated by people who had very low calibration scores… so, if you end up with two out of the three peer reviews with low scores, that’s something that’s flagged by the system. I check all the cases where it’s flagged and I tell the students if they’re unhappy with their score or if they have any questions about it, to contact me, explain why they’re dissatisfied with their score, and then I’ll go in and look at it. In nearly all cases, it’s been an issue with the students submission and not with the peer reviews. Because, while some people tend to overrate things and some tend to underestimate some of it, compared to where I would evaluate the wok… on average, it’s been very close, typically, to what I would have scored or what I would have assigned as a score. But I do make, in rare cases, some adjustments when I see that something went wrong in the process.

Rebecca: Do you prevent students from seeing the score then, until you’ve reviewed all of the scores to make sure that you’re okay with what has happened before they have access or…?

John: In this system, that really can’t be done easily…

Rebecca: ok.

John:… because what happens is they get the results as soon as the last stage is completed. I’ll send a note out saying, “Now that the stage is completed, you can review your scores, you can read all the comments that your peers have provided, and you can see what your grade is at each component…” and we have gone over that in class so they know what they’ll be seeing.

Rebecca: What kind of workload do you end up with, dealing with problems?

John: In general, when I’ve used this in the class of 360 to 420 students, there’s usually 3 to 5 students who find their grade unreasonable, and sometimes, I found the grades perfectly fine. Occasionally one or two of those, I’ll make some minor adjustments to—if something went wrong where one of their peer reviewers didn’t show up, for example, one or two of them didn’t complete that stage of the assignment, and someone was overly harsh or perhaps overly harsh in their grading, but it’s rare.

Rebecca: Can that system be used for things other than writing? Like other kinds of documents?

John: It could be used for any type of document because basically students will either write something up or they’ll submit something and it could be an image, it could be used for peer review, or calibrated peer review, on pretty much anything as long as it can be disseminated in digital format. It could be used for websites, for example.

Rebecca: Well, that’s what I was getting at when I was asking.
You also teach some upper-level seminar courses with 30 or so students. This semester, you tried a two-stage exam after talking with Doug McKee when he was on campus about it. What is a two-stage exam and how did it work?

John: Backing up a bit, I was considering it even before Doug came here because I heard the episode of the Teach Better podcast where they discussed a two-stage exam and then when we were talking here and he was in one of our earlier podcasts and we discussed this very issue, I became more interested after we talked with Doug. A two-stage exam is one where in the first stage of the process, students take the exam by themselves and then in the second stage, they do some group work– either on a subset of the questions or on some very closely related questions. It’s being used quite a bit in the sciences and there’s a growing amount of research indicating that it has been successful. Some studies have found weak results, others are finding stronger results, but it’s still fairly early in the exploration of this. The Carl Wieman Science Education Initiative has quite a few resources associated with two-stage exams. This leverages peer instruction in the second stage.

The usual process, or the most common practice, is to take the exam period and have students work on this for the first two-thirds or so of the exam time slot and then they work in a group in the last third. I did it a little bit differently than this. In my case, I gave the exam on a Wednesday and I graded the exam but didn’t get them back to the students and then I selected a subset of the questions and I had them work on them in groups on that Friday… and that worked pretty well too, they had a little chance to review in between, they didn’t get to keep the exams, but there were only seven questions on it. They could go back and review things. I didn’t tell them which questions would be on the second stage in large part because I didn’t know. I told them that two of the questions would definitely be on it, but it would depend on how they did on the other part. So, I was able to look at the exam, find the parts where they had the most trouble, and assigned those as ones for the second stage… and in general, it was a remarkable experience. It was really nice to be giving an exam and to see students working in groups of three or four, actively discussing the issues, arguing over them, trying to explain things to each other and it was a really fun experience. It was very energizing to see that much effort being devoted to try to understand concepts that students had some difficulty with.

Rebecca: I remember seeing an image of your class being really actively engaged, really talking about the core class material that you shared during your test and I think the caption was: “This is during a test!”
[LAUGHTER]

John: Yes, I took a picture of it from my phone and I think I sent that to you during the exam because it was just so exciting to see that… and it was also a reminder for myself just how well this was working. I wandered around the room and listened in on the discussions and they were all very focused and coming up with much better explanations of these things then they would have likely been able to see if it was a whole class discussion… because they were very focused, they were arguing over what was the best approach to deal with some of these problems. I could see people making connections and suddenly understanding how things they had done before fit in and pulling together a lot of concepts that they might not have done as effectively if it had not been for those small group discussions.

Rebecca: Were you tempted to join in on those conversations because they were so lively?

John: I was, but I mostly just listened in and let them work it out themselves… and in general, they did quite a bit better… and what I should have mentioned before is that the overall grade for the exam is a weighted average of the first part and the second with most of the weight being on the individual part. One of the things that really appealed to me is that typically, when we give an exam and then grade it and return it, the students who did well generally just put it away and are happy with the results and they may glance at some of the things they got wrong (if they got many things wrong), but they’re not going to spend a lot of time actively processing it. The students who did poorly tend to get discouraged, some of them may give up a bit, but rarely are they likely to go back and try to put in the effort to correct their mistakes and to see where they went wrong. It was really nice to see that processing taking place by both groups. The students who did really well the first time deepen their understanding by explaining it to others and I suspect that should increase their long-term recall of this. The act of explaining it to others in some studies seems to be really helpful in encouraging transfer, where you can take concepts and apply them to other circumstances and when you’re in a course like econometrics, you have to be able to apply the same concepts in a wide variety of topics and areas. I think it was a very useful experience.

Rebecca: I think it’s a great method to allow some time and space for a reflective practice, because students tend not to do that on their own unless they’re asked to do it and if you do it as a homework assignment, I suspect that students don’t really spend that much time doing it, but this time they spent the whole class period doing the reflection. So, that seems really valuable.

John: Because I know a lot of people will do that. They’ll have an exam, they’ll give it back to students, and they’ll tell them they can make up part of the grade if they turn it in with corrections… and many students would do that, but I don’t think that would be as effective as having the group discussion on this. Some of them were able to make very clear what they didn’t understand and then they were able to get explanations from others and sometimes the explanations were right, sometimes they were wrong, but they had to process it much more actively and that’s always helpful, I think.

Rebecca: The grade weights is what seems most compelling to me in this situation because I’ve offered quizzes in my classes, more low-stakes assignments where I let students work on it for a while and I don’t tell them that they’re gonna get to do some peer instruction as part of it, but then they’re struggling with what they’re doing and then I say, “Oh, well, you have five minutes to work with your peers to revise anything you want to do before you turn it in.” And those generally result in some pretty active conversations as well, but there still are those few students who just copy down the answer and don’t engage in the conversation… but I think if there was that wait between before and after, that would really change that dynamic. So, I think that that’s a really compelling opportunity.

John: I thought it was useful and another reason why I didn’t do it all at one stage in one day is because I’m teaching on a Monday-Wednesday-Friday schedule and we only have 55 minutes and I have quite a few students in the class who are not native English speakers and they always take more time or they need more time to process and write information in a second language. So, I didn’t want to constrain the time and make both parts of it much shorter.

Rebecca: If you encourage people to practice and retrieve that information in extra time outside of class, there’s nothing wrong with that either.

John: Exactly.

Rebecca: I’d rather the students learn the material rather than just panic about a test. What do you recommend to our listeners to read to learn more about this evidence-based practice?

John: In terms of peer instruction, Derek Bruff has a really good book on using clickers. Eric Mazur’s original book on this, which is now slightly over 20 years old, is still very good… where he describes a process of developing this peer instruction technique. Eric Mazur also gave a talk here a few years ago and we have a recording of his presentation on this. There’s a really great example in there where he used peer instruction and what was most compelling about it, and Rebecca’s heard this before, but…

Rebecca: I was there!

John: …and Rebecca was there, was he used this example where he gave a really short presentation on what happens to the hole in a plate of metal if you heat it up… and people were asked to vote on that and then they had a chance to discuss it.

Rebecca: …and he never told us the answer!

John: …and then he noted how energized people were and he said, “You were so actively discussing these things…” When he tried to go on after making a point about how they suddenly were interested in something they normally wouldn’t have been interested in… he started to go on to the next topic. People were really upset, because they wanted the answer and he finally gave the answer, but he did that deliberately to show that this sort of thing… where the students don’t know the answer but they committed to a position and they want to know if they’re right… builds a sort of interest in learning that might not intrinsically be there otherwise.

…and that’s exactly what I saw, by the way, in my exam. They were so actively discussing things that normally they’d be bored out of their minds with. So, that environment can be very supportive of learning.

Rebecca: Yeah, it really gets people curious. I remember being in that room… dying to know what version was right? People had such compelling arguments.

[LAUGHTER]

John: Exactly, and that’s why it’s really good to pick questions, with any of these things, where it’s not going to be clearly obvious, where they have to process it, and they have to make connections, and you could build a case, correctly or wrongly, for different answers, and people want to know what the answers are.

Rebecca: I mean it was key that he finally gave the answer, right? So there was some corrective feedback there, so that people didn’t continue to mislearn the information.

John: And that was nearly four years ago, and we remember that very vividly. If that was just a point in a class that was given… say, four years ago, we probably wouldn’t be talking about that now.

Rebecca: I can’t believe it was that long ago.

John: I think it was.

Rebecca: It was a while ago.

John: Yeah.

Rebecca: …and so I’m dying to know, what are you gonna do next?

John: One of the next things I’m going to do is a follow-up to something we talked about in an earlier episode, when we talked to Judie Littlejohn about the metacognitive cafe. One of the things I’ve been observing is that the use of this process by having students work to improve their metacognition about how they learn and what they’re learning… Students, at least, perceive there is being some significant learning gains from that. That’s convinced me that I’d like to do something similar in a large class, but an online discussion forum for 400 students again doesn’t scale quite as well. So, I’m going to be doing some weekly activities and I’m working with Liz Dunne Schmitt who teaches our large macro class in the spring semester, and a couple of other people: Kris Munger, and Michelle Miller, who also who’s the author of Minds Online: Teaching Effectively with Technology (and was a guest here a while back). We’re going to try to put together an experiment where we use some evidence-based methods as weekly assignments, say for ten weeks in a semester…. that’s our current plan at least)… and students will be exposed to this… and they’ll engage in some sort of reflection or some practice with one of these activities… and then in terms of evidence-based methods of learning, such as retrieval practice, spaced practice, and interleaved practice, and similar things… and then we’re going to see how that exposure along with some reasonably easily assessed activity, which could be just some short responses in a forum or it could be perhaps some online quizzes, evaluating whether that impacts their actual behavior in the class, and their actual performance in the class. One-half of the group will be exposed to those types of interventions, and the other half will be exposed to some form of standard study skills module, because most of the students in this class of freshmen and basically what we’re looking at is, if we present students with evidence on what really increases our ability to learn, whether that will result in significant change in either their behavior, or in their performance. So, we’re going to try, at least the plan, is to try to see whether that affects the number of times they take quizzes that can be taken repeatedly, whether it affects the number of times they log in and view other materials, and whether it changes a perception of how we learn. so right now we’re at the…

Rebecca: And performance too, right?

John: …and their performance.

Rebecca: And is the plan to start collecting that data in the fall?

John: The plan is to put all this together the spring, I’m hoping and then to submit a proposal to the IRB, and then to conduct the study and the fall and the spring, at least for a first stage and then we’re hoping to be able to follow these students up, to see if this has a significant effect later in terms of their grades or their persistence.

Rebecca: Sounds pretty exciting. I’m looking forward to hearing how that goes.

John: It is. I’m looking forward to it being all together and actually being implemented. I think it’s an interesting study.

Rebecca: We’ll have to have you back, John.

[LAUGHTER]

John: I think we can manage that.

Rebecca: Well, thanks so much for sharing all this information about peer instruction. I know it’s something that I’m always kind of asking you about and like to hear about, and I’m sure others will too.

John: Well, thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.