37. Evidence is Trending

Faculty are increasingly looking to research on teaching and learning to make informed decisions about their practice as a teacher and the policies their institutions put into place. In today’s episode, Michelle Miller joins us to discuss recent research that will likely shape the future of higher education.

Michelle is Director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general interest publications.

Show Notes

Rebecca: Faculty are increasingly looking to research on teaching and learning to make informed decisions about their practice as a teacher and the policies their institutions put into place. In today’s episode we talk to a cognitive psychologist about recent research that will likely shape the future of higher education.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guest today is Michelle Miller. Michelle is Director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general interest publications.
Welcome, Michelle!

Michelle: Hi, I’m so glad to be here.

Rebecca: Thanks for joining us.
Today’s teas are:

Michelle: I’m drinking a fresh peppermint infused tea, and it’s my favorite afternoon pick-me-up.

Rebecca: …and it looks like it’s in a really wonderfully designed teapot.

Michelle: Well, thank you… and this is a thrift store find… one of my favorite things to do. Yeah, so I’m enjoying it.

John: I have Twinings Blackcurrant Breeze.

Rebecca: …and I’m drinking chai today.

Michelle: Pretty rough.

John: We invited you here to talk a little bit about things that you’ve been observing in terms of what’s catching on in higher education in terms of new and interesting innovations in teaching.

Michelle: Right, that’s one of things that I really had the luxury of being able to step back and look at over this last semester and over this last spring when I was on sabbatical… One of the really neat things about my book Minds Online, especially now that it’s been out for a few years, is that it does open up all these opportunities to speak with really engaged faculty and others, such as: instructional designers, librarians, academic leadership, educational technology coordinators… all these individuals around the country who are really, really involved in these issues. It’s a great opportunity to see how these trends, how these ideas, how these innovations are rolling out, and these can be some things that have been around for quite some time and just continue to rock along and even pickup steam, and some newer things that are on the horizon.

John: You’ve been doing quite a bit of traveling. You just got back from China recently, I believe.

Michelle: I sure did. It was a short visit and I do hope to go back, both to keep getting involved in educational innovations there and, hopefully, as a tourist as well. So, I was not there for very long but I had the opportunity to speak at Tsinghua University in Beijing, which is a really dynamic institution that’s been around for about a hundred years. For a while in its history it specialized in things like engineering education polytechnic, but now it’s really a selective comprehensive university with very vibrant graduate and undergraduate programs that are really very relatable for those of us in the United States working in similar contexts. My invitation was to be one of the featured speakers at the Future Education, Future Learning Conference, which was a very interdisciplinary gathering of doctoral students, faculty, even others from the community, who were all interested in the intersection of things like technology, online learning, MOOCs even, and educational research (including research into the brain and cognitive psychology), and bringing all of those together… and it was a multilingual conference. I do not speak Chinese but much of the conference was in both English and Chinese and so I was also able to really absorb a lot of these new ideas. So yes, that was a real highlight of my sabbatical semester and one that I’m going to be thinking about for quite some time.

I should say that part of what tied in there as well is that Minds Online, I’ve just learned, is going to be translated into Chinese and that’s going to come out in May 2019. So, I also got to meet with some of the people who were involved in the translation… start to put together some promotional materials such as videos and things like that.

Rebecca: Cool.

John: Excellent.

Rebecca: So, you’ve had a good opportunity, as you’ve been traveling, to almost do a scavenger hunt of what faculty are doing with evidence-based practices related to your book. Can you share some of what you’ve found or heard?

Michelle: This theme of evidence-based practice, and really tying into the findings that have been coming out of cognitive psychology for quite some time, that really is one of the exciting trends and things that I was really excited to see and hear for so many different quarters I visited in different institutions… and so I would say definitely, this is a trend that is continuing and is increasing. There really does continue to be a lot of wonderful interest and wonderful activity around these real cognitively informed approaches to teaching, and what I think we could call scientifically based and evidence-based strategies. One form this has taken is Josh Eyler’s new book, called How Humans Learn: The Science and Stories behind Effective College Teaching. This is a brand new book by a faculty development professional, and a person coming out of the humanities, actually, who’s weaving together even from his humanities background everything from evolutionary biology to classical research in early childhood education to the latest brain-based research. He’s weaving this together into this new book for faculty. So, that’s one of the things that I’ve noticed and then there’s the issue which i think is another great illustration of best-known practice which is the testing effect and retrieval practice.

John: One of the nice things is how so many branches of research are converging… testing in the classroom, brain-based research, and so forth, are all finding those same basic effects. It’s nice to see such robust results, which we don’t always see in all research in all disciplines.

Rebecca: …and just breaking down the silos in general. The things are all related and finding out what those relationships are… exploring those relationships… is really important and it’s nice to see that it’s starting to open up.

John: We should also note that when you visited here, we had a reading group and we had faculty working on trying to apply some of these concepts, and they’re still doing that… and they still keep making references back to your visit. So, it’s had quite a big impact on our campus.

Michelle: This wasn’t true, I don’t think, when I first entered the teaching profession… and even to the extent when I first started getting interested in applied work in course redesign and in faculty professional development. you would get kind of this pushback or just strange looks when you said “Oh, how about we bring in something from cognitive psychology” and now that is just highly normalized and something that people are really speaking across the curriculum… and taking it and running with it in a lasting ongoing way, not just as a “Oh, well that was an interesting idea. I’m going to keep doing what I’m doing” but really people making some deep changes as you mentioned. This theme of breaking down silos… I mean I think if there’s kind of one umbrella trend that all of these things fits under it’s that breakdown of boundaries. So, that’s one that I keep coming back to, I know, in my work.

So, the idea of retrieval practice, drilling down on that one key finding which goes back a very long ways in cognitive psychology. I think of that as such a good example of what we’re talking about here… about how this very detailed effect in cognition and yet it does have these applications across disciplinary silos. Now when I go to conferences and I say “Okay, raise your hand. How many people have ever heard of retrieval practice? How many people have ever heard of the testing effect? How many people have heard of the book Make it Stick (which really places this phenomena at its center)?” and I’m seeing more hands raising.

With retrieval practice, by the way, we’re talking about that principle that taking a test on something, that retrieving something from memory actively, has this huge impact on future memorability of that information. As its proponents like to say, tests are not neutral from a memory or from a learning standpoint… and while some of the research has focused on very kind of stripped-down laboratory style tasks like memorizing words pairs, there are also some other research projects showing that it does flow out to more realistic learning situations.

So, more people simply know about this, and that’s really the first hurdle, oftentimes, with getting this involved disciplinary sometimes jargon riddled research out there to practitioners and getting it into their hands. So, people heard of it and they’re starting to build this into their teaching. As I’ve traveled around I love to hear some of the specific examples and to see it as well crop up in scholarship of teaching and learning.

Just recently, for example, I ran across and really got into the work of Bruce Kirchhoff who is at University of North Carolina – Greensboro and his area is botany and plant identification. He has actually put together some different really technology-based apps and tools that students and teachers can use in something like a botany course to rehearse and review plant identification. He says in one of his articles, for example, that there just isn’t time in class to really adequately master plant identification. It’s just too complex of a perceptual and cognitive and memory test to do that. So, he really built in from the get-go very specific principles drawn from cognitive psychology… so, the testing effect is in there… there’s different varieties of quizzing and it all is about just getting students to retrieve and identify example after example. It brings in also principles such as interleaving, which we could return to in a little bit, but has to do with the sequencing of different examples… their spacing… So, that’s even planned out exactly how and when students encounter different things that they’re studying. It’s really wonderful. So, for example he and his colleagues put out a scholarship of teaching and learning article talking about how this approach was used effectively in veterinary medicine students who have to learn to identify poisonous plants that they’ll see around their practice. This is something that can be time-consuming and very tough, but they have some good data showing that this technology enhanced cognitively based approach really does work. That’s one example. Coincidentally, I’ve seen some other work in the literature, also on plant identification, where the instructors tagged plants in an arboretum… they went around and tagged them with QR codes… that students can walk up to a plant in the real environment with an iPad… hold the iPad over it… and it would immediately start producing quiz questions that were are specific to exactly the plants they were looking at.
So, those are some of the exciting things that people are taking and running with now that this principle is out there.

Rebecca: What I really love about the two stories that you just shared was the faculty are really designing their curriculum and designing the learning experiences with the students in mind… and what students need and when they need it. So, not only is it employing these cognitive science principles, but it’s actually applying design principles as well. It’s really designing for a user experience and thinking about the idea that if I need to identify a plant, being able to identify it in this situation in which I would need to identify it in makes it much more dynamic I think for a student… but also really meets them where they’re at and where they need it.

John: …and there’s so many apps out there now that will do the plant identification just from imagery without the QR code, that I can see it taking it one step further where they can do it in the wild without having that… so they can build it in for plants that are in the region without needing to encode that specifically for the application.

Michelle: I think you’re absolutely right once we put the technology in the hands of faculties who, as I said, they’re the one to know: “Where are my students at? Where are the weak points? Where are the gaps that they really need to bridge?” and that’s where their creativity is giving rise to all these new applications… and sometimes these can be low-tech as well… or also things that we can put in a face-to-face environment… and I’d like to to share just some experiences that I’ve had with this over the last few semesters.

In addition to trying to teach online with a lot of technology, I also have in my teaching rotation a small required course in research methods in psychology which can be a real stumbling block… the big challenge course… it’s kind of a gateway course to continued progress in our major. So, in this research methods course, some of the things that I’ve done around assessment and testing to really try again to stretch that retrieval practice idea… to make assessments really a more dynamic part of the course and more central part of the course… to move away from that idea that tests are just this kind of every now and again this panic mode opportunity for me to kind of measure in sorts of students and judge them… to make good on that idea that tests are part of learning. So, here’s some of the things that I try to do. For one thing, I took time out of the class almost every single class meeting as part of the routine to have students first of all generate quiz questions out of their textbook. So, we do have a certain amount of foundational material in that course as well as a project and a whole lot of other stuff is going on. So they need to get that foundational stuff.

Every Tuesday they would come in and they knew their routine: you get index cards and you crack your textbook and you generate for me three quiz questions. Everybody does it. I’m not policing whether you read the chapter or not. It’s active… they’re generating it… and also that makes it something like frequent quizzing. That’s a great practical advantage for me since I’m not writing everything. They would turn those in and I would select some of my favorites I would turn those into a traditional looking paper quiz and hand that out on Thursday. I said “Hey, take this like a realistic quiz.” I had explained to them that quizzes can really boost their learning, so that was the justification for spending time on it and then I said: “You know what? I’m not going to grade it either. You take it home because this is a learning experience for you. It’s a learning activity.” so we did that every single week as those students got into that routine.

The second thing that I did to really re-envision how assessment testing and quizzing worked in this particular course, was something inspired by different kinds of group testing and exam wrapper activities I’ve seen, particularly coming out of the STEM field, where there’s been a lot of innovation in this area. What I would do is… we had these high stakes exams at a few points during the semester. But, the class day after the exam, we didn’t do the traditional “Let’s go over the exam.” [LAUGHTER] That’s kind of deadly dull, and it just tends to generate a lot of pushback from students… and as we know from the research, simply reviewing… passing your eyes over the information… is not going to do much to advance your learning. So, what I would do is… I would photocopy all those exams, so it has a secure copy. They were not graded. I would not look at this before we did this… and I would pass everybody’s exams back to them along with a blank copy of that same exam. I assigned them to small groups and I said “Okay, here’s your job. Go back over this exam, fill it out as perfectly as you can as a group, and to make it interesting I said I will grade that exam as well, the one you do with your group, and anything you get over 90% gets added to everybody’s grade. This time it was open book, it was open Google, it was everything except you can’t ask me questions. So, you have each other and that’s where these great conversations started to happen. The things that we always want students to say. So, I would eavesdrop and hear students say “Oh, well you know what, I think on this question she was really talking about validity because reliability is this other thing…” and they’d have a deep conversation about it. I’m still kind of going back through the numbers to see what are the impacts of learning? Are there any trends that I can identify? But, I will say this: in the semesters that I did this, I didn’t have a single question ever come back to me along the lines of “Well, this question was unclear. I didn’t understand it. I think I was graded unfairly.” it really did shut all that down and again extended the learning that I feel students got out of that. Now it meant a big sacrifice of class time, but I feel strongly enough about these principles that I’m always going to do this in one form or another anytime I can can in face-to-face classes.

Rebecca: This sounds really familiar, John.

John: I’ve just done the same, or something remarkably similar, this semester, in my econometrics class which is very similar to the psych research methods class. I actually picked it up following a discussion with Doug McKee. He actually was doing it this semester too. He had a podcast episode on it. It sounded so exciting, I did something… a little bit different. I actually graded it but I didn’t give it back to them because I wanted to see what they had the most trouble with, and then I was going to have them only answer the ones in a group that they struggled with… and it turned out that that was pretty much all them anyway. So, it’s very similar to what you did except I gave them a weighted average of their original grade and the group grade and all except one person improved and the one person’s score went down by two points because the group grade was just slightly lower… but he did extremely well and he wasn’t that confident. The benefits to them of that peer explanation and explaining was just tremendous and it was so much more fun for them and for me and, as you said, it just completely wiped out all those things like “Well, that was tricky” because when they hear their peers explaining it to them the students were much more likely to respond by saying “Oh yeah, I remember that now” and it was a wonderful experience and I’m gonna do that everywhere I can.

In fact. I was talking about it with my TA just this morning here at Duke and we’re planning to do something like that in our classes here at TIP this summer, which i think is somewhat familiar to you from earlier in your academic career.

Michelle: That is right we do have this connection. I was among, not the very first year, but I believe the second cohort of Talent Identification Program students who came in, I guess you would call it now, middle school (back then, it was called junior high) and what a life-transforming experience. We’ve had even more opportunities to talk about the development of all these educational ideas through that experience.

John: That two-stage exam is wonderful and it’s so much more positive… because it didn’t really take, in my class, much more time, because I would have spent most of that class period going over the exam and problems they had. But the students who did well would have been bored and not paying much attention to it; the students who did poorly would just be depressed and upset that they did so poorly… and here, they were actively processing the information and it was so positive.

Michelle: That’s a big shift. We really have to step back and acknowledge that, I think. that is a huge shift in how we look at assessment, and how we think about the use of class time… and it’s not just “Oh my gosh, I have to use every minute to put such content in front of the students.” Just the fact that more of us are making that leap, I think, really is evidence this progress is happening… and we see also a lot of raised consciousness around issues such as learning styles. That’s another one that, when I go out and speak to faculty audiences, 10 years ago you would get these shocked looks or even very indignant commentary when you say “Ok, this idea of learning styles, in the sense that say there are visual learners, auditory learners, what I call sensory learning styles (VAK is another name it sometimes goes by). The idea that that just holds no water from a cognitive point of view…” People were not good with that, and now when I mentioned that at a conference, I get the knowing nods and even a few groans… people like “Oh, yeah. we get that. Now, K-12, which I want to acknowledge it’s not my area, but I’m constantly reminded by people across the spectrum that it’s a very different story in K-12. So, setting that aside… but this is what I’m seeing… that faculty are realizing… they’re saying “Oh, this is what the evidence says…” and maybe they even take the time to look at some of the really great thinkers and writers who put together the facts on this. They say “You know what? I’m not going to take my limited time and resources and spend that on this matching to styles when the styles can’t even be accurately diagnosed and are of no use in a learning situation. So, that’s another area of real progress.

Rebecca: What I am hearing is not just progress here in terms of cognitive science, but a real shift towards really thinking about how students learn and designing for that rather than something that would sound more like a penalty for grade like “Oh, did you achieve? Yes or no…” but, rather here’s an opportunity if you didn’t achieve to now actually learn it… and recognize that you haven’t learned it, even though it might seem really familiar.

John: Going back to that point about learning styles. It is spreading in colleges. I wish it was true at all the departments at our institution, but it’s getting there gradually… and whenever people bring it up, we generally remind them that there’s a whole body of research on this and I’ll send them references but what’s really troubling is in my classes the last couple years now, I’ve been using this metacognitive cafe discussion forum to focus on student learning… and one of the week’s discussions is on learning styles and generally about 95 percent of the students who are freshmen or sophomores (typically) come in with a strong belief in learning styles… where they’ve been tested multiple times in elementary or middle school… they’ve been told what their learning styles are… they’ve been told they can only learn that way… It discourages them from trying to learn in other ways and it does a lot of damage… and I hope we eventually reach out further so that it just goes away throughout the educational system.

Rebecca: You’ve worked in your classes, Michelle, haven’t you to help students understand the science of learning and use that to help students understand the methods and things that you’re doing>

Michelle: Yes, I have. I’ve done this in a couple of different ways. Now, partly, I get a little bit of a free pass in some of my teaching because I’m teaching the introduction to psychology or I’m teaching research methods where I just happen to sneak in as the research example will be some work on say attention or distraction or the testing effect. So, I get to do it in those ways covertly. I’ve also had the chance, although it’s not on my current teaching rotation… I’ve had the chance to also take it on as in freestanding courses. As many institutions are doing these days… it’s another trend… and what Northern Arizona University, where I work, has different kinds of freshmen or first-year student offering for courses they can take, not in a specific disciplinary area, but that really crossed some different areas of the student success or even wellbeing. So, I taught a class for awhile called Maximizing Brain Power that was about a lot of these different topics. Not just the kind of very generic study skills tip… “get a good night’s sleep…” that kind of thing… but really some again more evidence-based things that we can tell students and you can really kind of market it… and I think that we do sometimes have to play marketers to say “Hey, I’m going to give you some inside information here. This is sort of gonna be your secret weapon. So, let me tell you what the research has found.”

So, those are some of the things that I share with students… as well as when the right moment arises, say after an exam or before their first round of small stakes assessments, where they’re taking a lot of quizzes… to really explain the difference between this and high stakes or standardized tests they may have taken in the past. So, I do it on a continuing basis. I try to weave it into the disciplinary aspect and I do it in these free-standing ways as well… and I think here’s another area where I’m seeing this take hold in some different places… which is to have these free-standing resources that also just live outside of a traditional class that people can even incorporate into their courses… if say cognitive psychology or learning science isn’t their area… that they can bring in, because faculty really do care about these things. We just don’t always have the means to bring them in in as many ways as we would like.

John: …and your Attention Matters project was an example of that wasn’t it? Could you tell us a little bit about that?

Michelle: Oh, I’d love to… and you know this connects to what it seems to be kind of an evergreen topic in the teaching and learning community these days, which is the role of distracted students… and I know this past year there just have been these one op-ed versus another. There’s been some really good blog posts by some people I really like to follow in the teaching and learning community such as Kevin Gannon talking about “Okay, do you have laptops in the classroom? and what happens when you do?” and so I don’t think that this is just a fad that’s going away. This is something that the people do continue to care about, and this is where the attention matters project comes in.

This was something that we conceptualized and put together a couple years ago at Northern Arizona University with myself, and primarily I collaborated with a wonderful instructional designer who also teaches a great deal… John Doherty. So, how this came about is I was seeing all the information on distraction… I’m really getting into this as a cognitive psychologist and going “Wow, students need to know that if they’re texting five friends and watching a video in their class. It’s not going to happen for them.” I was really concerned about “What can I actually do to change students minds?” So, my way of doing this was to go around giving guests presentations in every classes where people would let me burn an hour of their class time… and not a very scalable model… and John Doherty respectfully sat through one my presentations on this and then he approached me and said “Look, you know, we could make a module and put this online… and it could be an open access within the institution module, so that anybody at my school can just click in and they’re signed up. We could put this together. We could use some really great instructional design principles and we could just see what happens… and I bet more people would take that if it were done in that format. We did this with no resources. We just were passionate about the project and that’s what we did. We had no grant backing or anything. We got behind it. So, what this is is about a one- to two-hour module that, it’s a lot like a MOOC in that it there’s not a whole lot of interaction or feedback, but there are discussion forums and it’s very self-paced in that way… so one- to two-hour mini MOOCs that really puts at the forefront demonstrations and activities… so we don’t try to convince students about problems with distraction and multitasking… we don’t try to address that just by laying a bunch of research articles on them… I think that’s great if this were a psychology course, but it’s not. So, we come at it by linking them out to videos, for example, that we were able to choose, that we feel really demonstrate in some memorable ways what gets by us when we aren’t paying attention… and we also give students some research-based tips on how to set a behavioral plan and stick to it… because just like with so many areas of life, just knowing that something is bad for you is not enough to really change your behavior and get you not to do that thing. so we have students talking about their own plans and what they do when, say, they’re having a boring moment in class, or they’re really really tempted to go online while they’re doing homework at home. What kinds of resolutions can they set or what kind of conditions can make that that will help them accomplish that. Things like the software blockers… you set a timer on your computer and it can lock you out of problematic sites… or we learned about a great app called Pocket Points where you actually earn spendable coupon points for keeping your phone off during certain hours. This is students talking to students about things that really concern them and really concern us all because I think a lot of us struggle with that.

So, we try to do that… and the bigger frame for this as well is this is, I feel, a life skill for the 21st century… thinking about how technology is going to be an asset to you and not detract from what you accomplish in your life. What a great time to be reflecting on that, when you’re in this early college career. so that’s what we try to do with the project…and we’ve had over a thousand students come through. They oftentimes earn extra credit. Our faculty are great about offering small amounts of extra credit for completing this and we’re just starting to roll out some research showing some of the impacts… and showing it in a bigger way just how you can go about setting up something like this.

Rebecca: I like that the focus seems to be on helping students with a life skill rather than using technology is just a blame or an excuse. We’re in control of our own behaviors and taking ownership over our behaviors is important rather than just kind of object blaming.

Michelle: So, looking at future trends, I would like to see more faculty looking at it in the way that you just described, Rebecca, as this is a life skill and it’s something that we collaborate on with our students… not lay down the law… because, after all, students are in online environments where we’re not there policing that and they do need to go out into work environments and further study and things like that. So, that’s what I feel is the best value. For faculty who are looking at this, if they don’t want to do… or don’t have the means to do something really formal like our Attention Matters approach, just thinking about it ahead of time… I think nobody can afford to ignore this issue anymore and whether you go the route of “No tech in my classroom” or “We’re going to use the technology in my classroom“ or something in between… just reading over, in a very mindful way, not just the opinion pieces, but hopefully also a bit of the research, I think, can help faculty as they go in to deal with this… and really to look at it in another way, just to be honest, we also have to consider how much of this is driven by our egos as teachers and how much of it is driven by a real concern for student learning and those student life skills. I think that’s where we can really take this on effectively and make some progress when we are de-emphasizing that ego aspect and making sure that it really is about the students.

John: We should note there’s a really nice chapter in this book called Minds Online: Teaching Effectively with Technology that deals with these types of issues. It was one of the chapters that got our faculty particularly interested in these issues… on to what extent technology should be used in the classroom… and to what extent it serves as a distraction.

Michelle: I think that really speaks to another thing which I think is an enduring trend… which is the emphasis on really supporting the whole student in success and what we’ve come to call academic persistence… kind of a big umbrella term that has to do with, not just succeeding in a given class, but also being retained… coming back after the first year. As many leaders in higher education point out, this is as a financial issue. As someone pointed out, it does cost a lot less to hang on to the students you have instead of recruiting more students to replace ones who are lost. This is, of course, yet another really big shift in mindset of our own, because after all we did used to measure our success by “Hey, I flunked this many students out of this course” or” Look at how many people have to switch into different majors…our major is so challenging…”

So, we really have turned that thinking around and this does include faculty now. I think that we did used to see those silos. We had that very narrow view of “I’m here to convey content. I’m here to be an expert in this discipline, and that’s what I’m gonna do…” and sure, we want to think about things like do students have learning skills? Do they have metacognition? Are they happy and socially connected at the school? Are they likely to be retained so that we can have this robust university environment?

We had people for that, right? It used to be somebody else’s job… student services or upper administration. They were the ones who heard about that and now I think on both sides we really are changing our vision. More and more forward-thinking faculty are saying “You know what? Besides being a disciplinary expert, I want to become at least conversant with learning science. I want to become at least conversant with the science of academic persistence…” There is a robust early literature on this and that’s something that we’ve been working on at NAU over this past year as well… kind of an exciting newer project that I like very much. We’ve started to engage faculty in a new faculty development program called Persistence Scholars and this is there to really speak to people’s academic and evidence-based side, as well as get them to engage in some perspective-taking around things like the challenges that students face and what it is like to be a student at our institution. We do some really selected readings in the area we look at things like mindset… belongingness… these are really hot areas in that science of persistence… in that emerging field. But, we have to look at it in a really integrated way.

It’s easy for people to say just go to a workshop on mindset and that’s a nice concept, but we wanted to think about it in this bigger picture… really know what are some of the strengths of that and why? Where do these concepts come from? What’s the evidence? That’s something that I think is another real trend and I think as well we will see more academic leaders and people in staff and support roles all over universities needing to know more about learning science. There are still some misconceptions that persist, as we’ve talked about. We’re making progress in getting rid of some of these myths around learning, but I will say… I’m not gonna name any names… but, every now and again I will hear from somebody who says “Oh well, we need to match student learning styles” or “Digital natives think differently, don’t you know?” and I have to wonder whether that’s a great thing. I mean, these are oftentimes individuals that have the power to set the agenda for learning all over a campus. Faculty need to be in the retention arena and I think that leaders need to be in the learning science arena. The boundaries is breaking down and it’s about time.

Rebecca: One of the things that I thought was really exciting with the reading groups that we’ve been having on our campus… that we started with your book, but then we’ve read Make it Stick and Small Teaching since… is that a lot of administrators in a lot of different kinds of roles engaged with us in those reading groups, it wasn’t just faculty. There was a mix of faculty, staff, and some administrators, and I think that that was really exciting. For people who don’t have the luxury of being in your persistence scholar program, what would you recommend they read to get started to learn more about the science of persistence?

Michelle: I really, even after working with this for quite some time, I loved the core text that we have in that program, which is Completing College by Vincent Tinto. It’s just got a great combination of passionate and very direct writing style. So, there’s no ambiguity, there’s not a whole lot of “on the one hand this and on the other hand that.” It’s got an absolutely stellar research base, which faculty of course appreciate… and it has a great deal of concrete examples. So, in that book they talk about “okay, what does it mean to give really good support to first semester college students? What does that look like?” and they’ll go out and they’ll cite very specific “Here’s a school and here’s what they’re doing… here’s what their program looks like… here’s another example that looks very different but gets at the same thing.” So, that’s one of the things that really speak to our faculty… that they really appreciated and enjoyed.

I think that as well we tested good feedback about work that’s come out of the David Yeager and his research group on belongingness and lay theories, and lay theories is maybe a counterintuitive term for kind of a body of ideas about what students believe about academic success and why some people are successful and others are not and how those beliefs can be changed sometimes through relatively simple interventions and when it happens we see great effects such as the narrowing of achievement gaps among students who have more privilege or less privileged backgrounds… and that’s something that, philosophically, many faculty really really care about but they’ve never had the chance to really learn “Okay, how can I actually address something like that with what I’m doing in my classroom, and how can I really know that the things that I’m choosing do have that great evidence base…”

John: …and I think that whole issue is more important now and is very much a social justice issue because, with the rate of increase we’ve seen in college cost inflation, people who start college and don’t finish it are saddled with an awfully high burden of debt. The rate of return to a college degree is the highest that we’ve ever seen and college graduates end up not only getting paid a lot more but they end up with more comfortable jobs and so forth… and if we really want to move people out of poverty and try to reduce income inequality, getting more people into higher education and successfully completing higher education is a really important issue. I’m glad to see that your institution is doing this so heavily and I know a lot of SUNY schools have been hiring Student Success specialists. At our institution they’ve been very actively involved in the reading group, so that message is spreading and I think some of them started with your book and then moved to each of the others. So, they are working with students in trying to help the students who are struggling the most with evidence-based practices …and I think that’s becoming more and more common and it’s a wonderful thing.

Rebecca: So, I really liked Michelle that you were talking about faculty getting involved in retention and this idea of helping students develop persistence skills, and also administrators learning more about evidence-based practices. There’s these grassroots movements happening in both of these areas. Can you talk about some of the other grassroots movements that are working toward, or efforts that faculty are making to engage students and capture their attention and their excitement for education?

Michelle: Right, and here I think a neat thing to think about too is just it’s the big ambitious projects… the big textbook replacement projects or the artificial intelligence informed adaptive learning systems… those are the things that get a lot of the press and end up in The Chronicle of Higher Education that we read about… But, outside of that, there is this very vibrant community and grassroots led scene of developing different technologies and approaches. So, it really goes back for a while. I mean, the MERLOT database that I do talk about in Minds Online has been trove for years of well hidden gems that take on one thing in a discipline and come at it from a way that’s not just great from a subject-matter perspective but brings up the new creative approaches. In the MERLOT database, for example, there’s a great tutorial on statistical significance and the interrelationship between statistical significance and issues like simple sizes. You know, that’s a tough one for students, but it has a little animation involving a horse and a rider that really turns it into something that’s very visual… that’s very tangible… and it really actually tying into analogies, which is a well-known cognitive process that can support the advancement of learning something new. There is something on fluid pressures in the body that was treated for nursing students by nurses, and it’s got an analogy of a soaker hose that this is really fun and is actually interactive. So, those are the kinds of things. The PhET project, P-h-E-T which comes out of University of Colorado, that has been around for a while… again, faculty-led and a way to have these very useful interactive simulations for concepts in physics and chemistry. So, that’s one. CogLab, that’s an auxiliary product that I’ve used for some time in like hundred psychology courses that simulates very famous experimental paradigms which are notoriously difficult to describe on stage for cognitive psychology students. That started out many years ago as a project that very much has this flavor of “We have this need in our classroom. We need something interactive. There’s nothing out there. Let’s see what we can build.” It has since then picked up and turned into a commercial product, but that’s the type of thing that I’m seeing out there.

Another thing that you’ll definitely hear about if you’re circulating and hearing about the latest project is virtual reality for education. So, with this it seems like, unlike just a few years ago, almost everywhere you visit you’re going to hear that “Oh, we’ve just set up a facility. We’re trying out some new things.” This is something that I also heard about when I was talking to people when I was over in China. So, this is an international phenomenon. It’s going to pick up steam and definitely go some places.

What also strikes me about that is just how many different projects there are. Just when you’re worried that you’re going to be scooped because somebody else is going to get there first with their virtual reality project you realize you’re doing very very different things. So, I’ve seen, for example, it used in a medical application to increase empathy among medical students… and I took a six or seven minute demonstration that just was really heart-rending, simulating the patient experience with a particular set of sensory disorders… and at Northern Arizona University we have a lab that is just going full-steam in coming up with educational applications such as interactive organic chemistry tutorial that is is just fascinating. We actually completed a pilot project and are planning to gear up a much larger study next semester looking at the impacts of this. So, this is really taking off for sure.

But, I think there are some caveats here. We still really need some basic research on this… not just what should we be setting up and what the impacts are but how does this even work? In particular, what I would like to research in the future, or at least see some research on, is what kinds of students… what sort of student profile… really gets the most out of virtual reality for education. Because amidst all the very breathless press that’s going on about this now and all the excitement, we do have to remember this is a very, very labor intensive type of resource to set up. You’re not just going to go home and throw something together for the next week. It takes a team to build these things and to complete them as well. If you have, say, a 300 student chemistry course (which is not atypical at all… these large courses), you’re not going to just have all of them spend hours and hours and hours doing this even with a fairly large facility. It’s a very hands-on thing to guide them through this process, to provide the tech support, and everything else.

So, I think really knowing how we can best target our efforts in this area, so that we can build the absolute best, with the resources we have, and maybe even target and ask the students who are most likely to benefit. I think those are some of the things that we just need to know about this. So, it’s exciting for somebody like me who’s in the research area. I see this as a wonderful open opportunity… but those are some of the real crossroads we’re at with virtual reality right now.

Rebecca: I can imagine there’s a big weighing that would have to happen in terms of expense and time and resources needed to startup versus what that might be saving in the long run. I can imagine if it’s a safety thing that you want to do a virtual reality experience, like saving people’s lives and making sure that they’re not going to be in danger as they practice particular skills, could be a really good investment in these… spending the resources to make that investment… or if it’s a lot of travel that would just be way too expensive to bring a bunch of students to a particular location… but you could virtually… it seems like it would be worth the start-up costs and those are just two ideas off the top of my head where it would make sense to bend all of that resource and time.

John: …and equipment will get cheaper. Right now, it’s really expensive for computers that have sufficient speed and graphics processing capability and the headsets are expensive, but they will come down in price, but as you said, it’s still one person typically and one device… so it doesn’t scale quite as well as a lot of other tools or at least not at this stage.

Rebecca: From what I remember, Michelle, you wrote a blog post about [a] virtual reality experience that you had. Can you share that experience, and maybe what stuck with you from that experience?

Michelle: Right, so I had the opportunity, just as I was getting to collaborate with our incredible team at the immersive virtual reality lab at NAU… one of the things I was treated to was about an hour and a half in the virtual reality setup that they have to explore some of the things that they had… Giovanni Castillo, by the way, is creative director of the lab and he’s the one who was so patient with me through all this. We tried a couple of different things and of course there’s such a huge variety of different things that you can do.
There’s a few things out there like driving simulators that are kind of educational… they’re kind of an entertainment… but he was just trying to give me, first of all, just a view of those… and I had to reject a few of them… I will say, initially, because I am one of the individuals who tends to be prone to motion sickness. So, that limits what I can personally do in VR and that is yet another thing that we’re gonna have to figure out. At least informally, what we hear is that women in particular tend to experience more of this. So, I needed, first of all, to go to a very low motion VR. I wasn’t gonna be whizzing through these environments. That was not going to happen for me. So, we did something that probably sounds incredibly simplistic, but it just touched me to my core… which is getting to play with Google Earth. You can spin the globe and either just pick a place at random or what Giovanni told me is… “You know, I’ve observed that when people do this, when we have an opportunity to interact with Google Earth, they all either go to where they grew up or they’ll go to someplace that they have visited recently or they plan to visit. So, I went to a place that is very special to me and maybe it doesn’t fit into either one of those categories neatly, but it’s my daughter’s University… her school… and I should say that this is also a different thing for me because my daughter goes to school in Frankfort, Germany… an institute that is connected to a Museum. So, I had only been to part of the physical facility… the museum itself… and it was a long time ago… and part of it was closer to the holiday. So, this is my opportunity to go there and explore what it looks like all over… and so, that was an emotional experience for me. It was a sensory experience… it was a social one… because we were talking the whole time… and he’s asking me questions and what kinds of exhibits do they have here… and what’s this part of it. So, that was wonderful. it really did give me a feel for alright, what is it actually like to be in this sort of environment?

I’m not a gamer. I don’t have that same background that many of our students have. So, it got me up to speed on that… and it did show me how just exploring something that is relatively simple can really acquire a whole new dimension in this kind of immersive environment. Now the postscript that I talked about in that blog post was what happened when I actually visited there earlier in the year. So, I had this very strange experience that human beings have never had before… which is from this… I don’t know whether to call it deja vu or what… of going to the settings and walking around the same environment and seeing the same lighting and all that sort of stuff that was there in that virtual reality environment… but this time, of course, with real human beings in it and the changes… the little subtle changes that take place over time, and so forth.

So, how does it translate into learning? What’s it going to do for our students? I just think that time is going to tell. It won’t take too long, but I think that these are things we need to know. But, sometimes just getting in and being able to explore something like this can really put you back in touch with the things you love about educational technology.

Rebecca: I think one of the things that I’m hearing in your voice is the excitement of experimenting and trying something… and that’s, I think, encouragement for faculty in general… is to just put yourself out there and try something out even if you don’t have something specific in mind with what you might do with it. Experiencing it might give you some insight later on. it might take some time to have an idea of what you might do with it, but having that experience, you understand it better… it could be really useful.

John: …and that’s something that could be experienced on a fairly low budget with just your smartphone and a pair of Google cardboard or something similar. Basically, it’s a seven to twelve dollar addition to your phone and you can have that experience… because there’s a lot of 3D videos and 3D images out there on Google Earth as well as on YouTube. So, you can experience other parts of the world and cultures before visiting… and I could see that being useful in quite a few disciplines.

Rebecca: So, we always wrap up with asking what are you going to do next?

Michelle: I continue to be really excited about getting the word out about cognitive principles and how we can flow those in to teaching face-to-face with technology… everything else in between. So, that’s what I continue to be excited about… leveraging cognitive principles with technology and with just rethinking our teaching techniques. I’m going to be speaking at the Magna Teaching with Technology Conference in October, and so I’m continuing to develop some of these themes… and I’m very excited to be able to do that. I’m right now also… we’re in the early stages of another really exciting project that has to do with what we will call neuromyth… So, that may be a term that you’ve turn across in some of your reading. It’s something that we touched on a few times, I think, in our conversation today… the misconceptions that people have about teaching and learning and how those can potentially impact the choices we make in our teaching. So, I’ve had the opportunity to collaborate with this amazing international group of researchers who’s headed up by Dr. Kristen Betts of Drexel University… and I won’t say too much more about it other than we have a very robust crop of survey responses that have come in from, not just instructors, but also instructional designers and administrators from around the world. So, we’re going to be breaking those survey results down and coming up with some results to roll out probably early in the academic year and we’ll be speaking about that at the Accelerate conference, most likely in November. That’s put out by the Online Learning Consortium. So, we’re right in the midst of that project and it’s going to be so interesting to see what has the progress been? What neuromyths are still out there and how can they be addressed by different professional development experiences. We’re continuing to work on the Persistence Scholars Program on academic persistence. So, we’ll be recruiting another cohort of willing faculty to take that on in the fall at Northern Arizona University. I am going to be continuing to collaborate and really work with and hear from John and his research group with respect to the metacognitive material that they’re flowing into foundational coursework and ways to get students up to speed with a lot of critical metacognitive knowledge. So, we’re going to work on that too… and I like to keep up my blog and work on shall we say longer writing project but we’ll have to stay tuned for that.

Rebecca: Sounds like you need to plan some sleep in there too.


John: Well, it’s wonderful talking to you, and you’ve given us a lot of great things to reflect on and to share with people.

Rebecca: Yeah. Thank you for being so generous with your time.

John: Thank you.

Michelle: Oh, thank you. Thanks so much. It’s a pleasure, an absolute pleasure. Thank you.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.

36. Peer instruction

Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know (and don’t know). Did you picture a lecture class, students taking a test, or students writing? In this episode, John discusses three ways in which he has been using peer-instruction in his classes: classroom polling, calibrated peer review writing assignments, and two-stage exams.

Show Notes


Rebecca: Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know. Did you picture a lecture class, students taking a test, or students writing? If not, stay tuned, this episode explores ways to use peer-instruction to transform the learning experience.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


Rebecca: Today’s guest is my co-host John Kane. John is the director of the Center for Excellence in Teaching—that’s not even right…


John: …Center for Excellence in Learning and Teaching at SUNY Oswego.

Rebecca: Yeah, woops! Welcome to your own show, John!

John: Thanks, Rebecca.

Rebecca: Today our teas are…

John: Prince of Wales.

Rebecca: Oh, that’s a good one.

Rebecca: I have Golden Tipped English Breakfast today.

John: Excellent.

Rebecca: One of the areas you’ve been teaching experimenting in, and that I’m fascinated in, is peer instruction. Can you tell us a little bit about what peer instruction is and why you’re drawn to using this methodology in your courses?

John: Peer instruction involves using peers to assist with instruction, where students explain….

Rebecca: Thanks John.


John: …where students explain things to each other. One of the issues that we have is that, once we become experts in the field, it’s very hard for us to express things in terms that are easily understood by students. There’s a “curse of knowledge;” once you become adept at something, it’s really hard to explain things at a level that’s appropriate to the level of understanding that students may have. There was a classic study done in which a researcher gave people a list of songs, very well-known popular songs, and asked them to tap out the beats from that song.

Rebecca: Oh, I would fail…

John: …and then before actually seeing if people would recognize it (who had the same list), she asked them to make a prediction of what proportion of people would understand it based on their tapping… and they overestimated that by a factor of I believe, somewhere around 20 times. Basically, it was purely random if people happen to guess it. But the issue is, once you hear something in your own mind, it’s clear to you, but it may not always be clear to the people who don’t have the same rich net of connections. When students are explaining things to each other, they benefit from taking a position, arguing that position, trying to filling gaps and they’re also explaining in terms that are appropriate for people at their level of cognitive development for people who have a similar background in terms of what they know and their prior knowledge.

Rebecca: Sounds like a really good way to expand and refine mental models and also just develop better metacognition. Because, as soon as you go to explain it, you realize what you don’t understand.

John: …and if you don’t understand it yourself, your peers will often help you understand. they’ll say: “Well, you haven’t considered this…” and that sort of interaction is one that doesn’t work as well when it’s instructor to a large group of students. But, it does work very well one-on-one.

Rebecca: You’re known on our campus for teaching really large lecture sections. Implementing peer instruction in a large setting can seem pretty daunting, especially to someone who teaches smaller classes like I do. What strategies do you use?

John: The most commonly used one is to use clicker quizzes… and I use a methodology that Eric Mazur developed slightly over 20 years ago, where you ask the students a challenging question… you try to find questions that about half of them will get wrong… and over time you can develop that, you can come up with a pool of questions that fit somewhere in that range… and you let students first vote on the response themselves after they’ve had a little bit of time to process it, and then you look at the results. If you see that 90% or more of them got it correct or even 80% or more, you can just go over it and move on to the next topic, because most students understand it. But, if you see that somewhere around half of them get it right and somewhere around half of them get it wrong (plus or minus 20% or so), then the next stage is to let them explain it to each other, and that’s where the peer instruction comes in. When you have students argue it and take a stand and a position on it, we get a very significant gain and improvement when we then let them vote on it a second time… and the usual practice is not to reveal the poll results or the answer until after they’ve had that opportunity to engage in that discussion.

Rebecca: Just make sure, to make sure I understand correctly: you do the poll, you see the results as students don’t see the results…

John: Right.

Rebecca: …based on their answers or their responses when you decide whether or not they do the peer instruction piece. How long do they usually talk to each other about the topic?

John: It depends on the problem and normally I will have some undergraduate TAs and I’ll wander around the class and see what they’re talking about, listen in, answer some questions from them and the TAs will be doing the same thing…. and it’s usually pretty clear when they’re coming to a consensus. You can see them reaching for their clickers or their phones and getting ready to vote, so generally it may only be a minute or two, it could be longer… it depends on the complexity of the problems. Some of the problems require a bit of effort and require some calculations, but normally they’ve already done that… so, the second stage, where there’s a discussion, you can hear the volume build-up and then as they’re approaching solutions and consensus, it tends to drop back down again. It’s fairly easy to get a pretty good read on where they are and when they’re ready to vote again.

Rebecca: I imagine that you would really need to keep your ear to the ground, otherwise chaos could ensue. Because now, if they’re finished talking about the problem and there’s still time, then they could easily derail if you’re not quick to get back to the clicker question.

John: Right, and normally the time is generally held fairly tight. I suspect sometimes it’s only 30 seconds to a minute, other times it may go up to a couple minutes, but if I see them getting distracted and doing other things, the polling starts immediately.

Rebecca: Obviously technology is your friend in this particular situation. Can you talk a little bit about the technology you’re using to manage this many students all at once?

John: Here, we’ve adopted iClicker as a campus standard, so we use that in pretty much all of the classes where we’re doing polling and there’s both a physical radio frequency clicker that students may buy or they can buy an app and pay by the semester or over four years for the use of the app.

Rebecca: How do you make sure that the cost doesn’t get prohibitive to students?

John: That’s an issue, and it’s been a major source of concern…

Rebecca: They’re not very expensive, right?

John: Well, they can be expensive. A new clicker costs somewhere around $40. A used one can often be purchased for $15 to $20, sometimes less… and the apps I think, are somewhere around $12 to $15 for a semester and I think about $35 for four years.

Rebecca: …and you can use the clickers in all of the classes, right? So if multiple faculty member(s) are using all the same system, then the investment is a good one for students.

John: …and that’s why we have a campus adoption because in places where you don’t have that, students might have to buy two or three or four different clicker systems in different classes. So, once they buy the clicker for one as long as they hold on to it, they can use it in classes for the rest of their career. Almost everyone in the economics department, for example, now uses clickers, so if they’re economics majors or business majors, it’s very likely they’ll use them in multiple courses. The cost is much more tolerable when it’s spread out over multiple classes.

Rebecca: The other area where you do some peer instruction in these large classes is in writing. Which seems kind of crazy. You have all these students in this big classroom and somehow you manage to do writing assignments.

John: Yeah, my large class generally is somewhere between 350 and 420 students. At one time, for actually about a decade or so, I was giving weekly online discussion forums. But grading that or evaluating that and providing feedback was taking an awful lot of time…probably 30 to 40 hours a week. So, I pretty much…

Rebecca: A full-time job in and of itself…

John: I stopped that a few years ago and, a few years back, I replaced that with calibrated peer review assignments. The calibrated peer review system is something that Eric Mazur talked about while he was here… a visit in 2014… and when he mentioned it, a lot of people got excited. The way the system works is that you create an assignment, you store it on a central server at UCLA, and then it’s something that other people can adapt and use and modify—it’s released under a license, which is similar to a creative common license within the system… and you create the assignment… you create an evaluation rubric for the assignment… and you have to be really careful in designing that to make sure it’s one that students will be able to apply, because other ones that do that… and then you create three sample assignments yourself: a low-quality one, a medium quality one, and a high quality assignment… and you have students submit their own assignments first (according to the rubric and guidelines you provide to them)… then they go in and they evaluate the three that you’ve done. They’re given in random order, and they’re assessed in terms of how closely their evaluations match yours. That’s the calibration part. Students receive a calibration score based on how similar their evaluations are to the ones that you assigned to the sample responses. Then after they complete that stage, they evaluate each other, using the same rubric, and a weighted average of those scores is assigned as a component of the grade. They’re graded in a number of dimensions. One is based on the weighted average of the peers, where students who had a high calibration score will have evaluations that rate more highly in evaluating other students. They’re also rated in terms of how closely their evaluations match the others during that stage. So, if their evaluation is an outlier… much higher or much lower than other students… they lose some points on that… and then after they evaluate the other three students, they rate their own work… and one of the goals of that is so that they have improved metacognition. That, by the time they go back and look at their work again, they’ve rated three works by the instructor and three assignments done by their peers and then they’re asked to evaluate their own using the same criteria. What’s really interesting about the calibrated peer review process is their grade on this is tied not to whether they give themselves a high or low score on this, but it’s how close their self evaluation comes to the weighted average of their peer evaluations. So, they have an incentive to try to look at their work more objectively, and not try to game the system… because if they score their work too high or too low, they could end up with no points on the self-evaluation stage. So, the closer they get to the weighted average of their peer evaluations, the higher the score will be on that component.

Rebecca: I think that’s an area that we often see students struggling, is being able to effectively evaluate their work or other work. So, really training them to use a rubric and understand and think about what’s important or what’s not important about particular kinds of assignments or particular kinds of work could be really valuable to students in a way that we don’t really have other systems to do that.

John: The nice thing about this is it scales really easily. There’s a lot of upfront work in creating the assignments, creating the rubric, and a really good practice is to test them thoroughly before you give them out the first time. What I normally had done is asked some of my peers to look at that, some of my colleagues to look at it, and sometimes I’d have some upper-level students were…. and this does give students a little bit more reflective practice, where they get to look at their own work a bit more critically, perhaps, and reflect on it and see how they’re doing compared to how other students are doing in the course… and I think that’s helpful.

Rebecca: I think that the rubric would probably be a challenge to make but I think what would be more challenging is putting up those different assignments that are scored at different levels at the very beginning as your calibration tool. What strategies have you developed to make those in a way that it doesn’t take forever?

John: Well, I only do this three times in a semester, and once you’ve done it once, if you design it in a way so that it won’t go stale… and I generally have students, for example, find some articles in the news in the last six months that relate to a topic that we’ve talked about, or I ask students to find some examples in their own life to illustrate behavioral economics concepts in one of the assignments, for example…. where it’s not something that they could easily copy and paste from other people’s work. Because, there is always a concern with academic dishonesty and so forth. You don’t want these things showing up on Chegg or any of those other systems, where it would be easy to copy and paste good responses. So, I’ve tried to design assignments where once they’re done, they can be used for multiple years in one form or another. I modify them each year based on how they work. But perhaps a more serious problem is what happens when students really don’t like the evaluations. One of the things I’ve done when I’ve used this is to have three of these assignments, but I drop the lowest score… because, sometimes people will get some scores back that they didn’t expect or they may have neglected to look at the rubric I sent them and they may have omitted a major part of the assignment and ended up losing quite a bit of points all the way through that. But, as long as one of the scores is dropped, they have the opportunity to learn from their mistakes and do a little bit better. But, there are procedures built-in that make it easier to catch any outliers when you have someone who is just rating everyone extremely highly or rating everyone really poorly—inappropriately highly or poorly. There are tools in it which will give you a list of all the cases where there’s a high variance across reviewers or where someone happened to be evaluated by people who had very low calibration scores… so, if you end up with two out of the three peer reviews with low scores, that’s something that’s flagged by the system. I check all the cases where it’s flagged and I tell the students if they’re unhappy with their score or if they have any questions about it, to contact me, explain why they’re dissatisfied with their score, and then I’ll go in and look at it. In nearly all cases, it’s been an issue with the students submission and not with the peer reviews. Because, while some people tend to overrate things and some tend to underestimate some of it, compared to where I would evaluate the wok… on average, it’s been very close, typically, to what I would have scored or what I would have assigned as a score. But I do make, in rare cases, some adjustments when I see that something went wrong in the process.

Rebecca: Do you prevent students from seeing the score then, until you’ve reviewed all of the scores to make sure that you’re okay with what has happened before they have access or…?

John: In this system, that really can’t be done easily…

Rebecca: ok.

John:… because what happens is they get the results as soon as the last stage is completed. I’ll send a note out saying, “Now that the stage is completed, you can review your scores, you can read all the comments that your peers have provided, and you can see what your grade is at each component…” and we have gone over that in class so they know what they’ll be seeing.

Rebecca: What kind of workload do you end up with, dealing with problems?

John: In general, when I’ve used this in the class of 360 to 420 students, there’s usually 3 to 5 students who find their grade unreasonable, and sometimes, I found the grades perfectly fine. Occasionally one or two of those, I’ll make some minor adjustments to—if something went wrong where one of their peer reviewers didn’t show up, for example, one or two of them didn’t complete that stage of the assignment, and someone was overly harsh or perhaps overly harsh in their grading, but it’s rare.

Rebecca: Can that system be used for things other than writing? Like other kinds of documents?

John: It could be used for any type of document because basically students will either write something up or they’ll submit something and it could be an image, it could be used for peer review, or calibrated peer review, on pretty much anything as long as it can be disseminated in digital format. It could be used for websites, for example.

Rebecca: Well, that’s what I was getting at when I was asking.
You also teach some upper-level seminar courses with 30 or so students. This semester, you tried a two-stage exam after talking with Doug McKee when he was on campus about it. What is a two-stage exam and how did it work?

John: Backing up a bit, I was considering it even before Doug came here because I heard the episode of the Teach Better podcast where they discussed a two-stage exam and then when we were talking here and he was in one of our earlier podcasts and we discussed this very issue, I became more interested after we talked with Doug. A two-stage exam is one where in the first stage of the process, students take the exam by themselves and then in the second stage, they do some group work– either on a subset of the questions or on some very closely related questions. It’s being used quite a bit in the sciences and there’s a growing amount of research indicating that it has been successful. Some studies have found weak results, others are finding stronger results, but it’s still fairly early in the exploration of this. The Carl Wieman Science Education Initiative has quite a few resources associated with two-stage exams. This leverages peer instruction in the second stage.

The usual process, or the most common practice, is to take the exam period and have students work on this for the first two-thirds or so of the exam time slot and then they work in a group in the last third. I did it a little bit differently than this. In my case, I gave the exam on a Wednesday and I graded the exam but didn’t get them back to the students and then I selected a subset of the questions and I had them work on them in groups on that Friday… and that worked pretty well too, they had a little chance to review in between, they didn’t get to keep the exams, but there were only seven questions on it. They could go back and review things. I didn’t tell them which questions would be on the second stage in large part because I didn’t know. I told them that two of the questions would definitely be on it, but it would depend on how they did on the other part. So, I was able to look at the exam, find the parts where they had the most trouble, and assigned those as ones for the second stage… and in general, it was a remarkable experience. It was really nice to be giving an exam and to see students working in groups of three or four, actively discussing the issues, arguing over them, trying to explain things to each other and it was a really fun experience. It was very energizing to see that much effort being devoted to try to understand concepts that students had some difficulty with.

Rebecca: I remember seeing an image of your class being really actively engaged, really talking about the core class material that you shared during your test and I think the caption was: “This is during a test!”

John: Yes, I took a picture of it from my phone and I think I sent that to you during the exam because it was just so exciting to see that… and it was also a reminder for myself just how well this was working. I wandered around the room and listened in on the discussions and they were all very focused and coming up with much better explanations of these things then they would have likely been able to see if it was a whole class discussion… because they were very focused, they were arguing over what was the best approach to deal with some of these problems. I could see people making connections and suddenly understanding how things they had done before fit in and pulling together a lot of concepts that they might not have done as effectively if it had not been for those small group discussions.

Rebecca: Were you tempted to join in on those conversations because they were so lively?

John: I was, but I mostly just listened in and let them work it out themselves… and in general, they did quite a bit better… and what I should have mentioned before is that the overall grade for the exam is a weighted average of the first part and the second with most of the weight being on the individual part. One of the things that really appealed to me is that typically, when we give an exam and then grade it and return it, the students who did well generally just put it away and are happy with the results and they may glance at some of the things they got wrong (if they got many things wrong), but they’re not going to spend a lot of time actively processing it. The students who did poorly tend to get discouraged, some of them may give up a bit, but rarely are they likely to go back and try to put in the effort to correct their mistakes and to see where they went wrong. It was really nice to see that processing taking place by both groups. The students who did really well the first time deepen their understanding by explaining it to others and I suspect that should increase their long-term recall of this. The act of explaining it to others in some studies seems to be really helpful in encouraging transfer, where you can take concepts and apply them to other circumstances and when you’re in a course like econometrics, you have to be able to apply the same concepts in a wide variety of topics and areas. I think it was a very useful experience.

Rebecca: I think it’s a great method to allow some time and space for a reflective practice, because students tend not to do that on their own unless they’re asked to do it and if you do it as a homework assignment, I suspect that students don’t really spend that much time doing it, but this time they spent the whole class period doing the reflection. So, that seems really valuable.

John: Because I know a lot of people will do that. They’ll have an exam, they’ll give it back to students, and they’ll tell them they can make up part of the grade if they turn it in with corrections… and many students would do that, but I don’t think that would be as effective as having the group discussion on this. Some of them were able to make very clear what they didn’t understand and then they were able to get explanations from others and sometimes the explanations were right, sometimes they were wrong, but they had to process it much more actively and that’s always helpful, I think.

Rebecca: The grade weights is what seems most compelling to me in this situation because I’ve offered quizzes in my classes, more low-stakes assignments where I let students work on it for a while and I don’t tell them that they’re gonna get to do some peer instruction as part of it, but then they’re struggling with what they’re doing and then I say, “Oh, well, you have five minutes to work with your peers to revise anything you want to do before you turn it in.” And those generally result in some pretty active conversations as well, but there still are those few students who just copy down the answer and don’t engage in the conversation… but I think if there was that wait between before and after, that would really change that dynamic. So, I think that that’s a really compelling opportunity.

John: I thought it was useful and another reason why I didn’t do it all at one stage in one day is because I’m teaching on a Monday-Wednesday-Friday schedule and we only have 55 minutes and I have quite a few students in the class who are not native English speakers and they always take more time or they need more time to process and write information in a second language. So, I didn’t want to constrain the time and make both parts of it much shorter.

Rebecca: If you encourage people to practice and retrieve that information in extra time outside of class, there’s nothing wrong with that either.

John: Exactly.

Rebecca: I’d rather the students learn the material rather than just panic about a test. What do you recommend to our listeners to read to learn more about this evidence-based practice?

John: In terms of peer instruction, Derek Bruff has a really good book on using clickers. Eric Mazur’s original book on this, which is now slightly over 20 years old, is still very good… where he describes a process of developing this peer instruction technique. Eric Mazur also gave a talk here a few years ago and we have a recording of his presentation on this. There’s a really great example in there where he used peer instruction and what was most compelling about it, and Rebecca’s heard this before, but…

Rebecca: I was there!

John: …and Rebecca was there, was he used this example where he gave a really short presentation on what happens to the hole in a plate of metal if you heat it up… and people were asked to vote on that and then they had a chance to discuss it.

Rebecca: …and he never told us the answer!

John: …and then he noted how energized people were and he said, “You were so actively discussing these things…” When he tried to go on after making a point about how they suddenly were interested in something they normally wouldn’t have been interested in… he started to go on to the next topic. People were really upset, because they wanted the answer and he finally gave the answer, but he did that deliberately to show that this sort of thing… where the students don’t know the answer but they committed to a position and they want to know if they’re right… builds a sort of interest in learning that might not intrinsically be there otherwise.

…and that’s exactly what I saw, by the way, in my exam. They were so actively discussing things that normally they’d be bored out of their minds with. So, that environment can be very supportive of learning.

Rebecca: Yeah, it really gets people curious. I remember being in that room… dying to know what version was right? People had such compelling arguments.


John: Exactly, and that’s why it’s really good to pick questions, with any of these things, where it’s not going to be clearly obvious, where they have to process it, and they have to make connections, and you could build a case, correctly or wrongly, for different answers, and people want to know what the answers are.

Rebecca: I mean it was key that he finally gave the answer, right? So there was some corrective feedback there, so that people didn’t continue to mislearn the information.

John: And that was nearly four years ago, and we remember that very vividly. If that was just a point in a class that was given… say, four years ago, we probably wouldn’t be talking about that now.

Rebecca: I can’t believe it was that long ago.

John: I think it was.

Rebecca: It was a while ago.

John: Yeah.

Rebecca: …and so I’m dying to know, what are you gonna do next?

John: One of the next things I’m going to do is a follow-up to something we talked about in an earlier episode, when we talked to Judie Littlejohn about the metacognitive cafe. One of the things I’ve been observing is that the use of this process by having students work to improve their metacognition about how they learn and what they’re learning… Students, at least, perceive there is being some significant learning gains from that. That’s convinced me that I’d like to do something similar in a large class, but an online discussion forum for 400 students again doesn’t scale quite as well. So, I’m going to be doing some weekly activities and I’m working with Liz Dunne Schmitt who teaches our large macro class in the spring semester, and a couple of other people: Kris Munger, and Michelle Miller, who also who’s the author of Minds Online: Teaching Effectively with Technology (and was a guest here a while back). We’re going to try to put together an experiment where we use some evidence-based methods as weekly assignments, say for ten weeks in a semester…. that’s our current plan at least)… and students will be exposed to this… and they’ll engage in some sort of reflection or some practice with one of these activities… and then in terms of evidence-based methods of learning, such as retrieval practice, spaced practice, and interleaved practice, and similar things… and then we’re going to see how that exposure along with some reasonably easily assessed activity, which could be just some short responses in a forum or it could be perhaps some online quizzes, evaluating whether that impacts their actual behavior in the class, and their actual performance in the class. One-half of the group will be exposed to those types of interventions, and the other half will be exposed to some form of standard study skills module, because most of the students in this class of freshmen and basically what we’re looking at is, if we present students with evidence on what really increases our ability to learn, whether that will result in significant change in either their behavior, or in their performance. So, we’re going to try, at least the plan, is to try to see whether that affects the number of times they take quizzes that can be taken repeatedly, whether it affects the number of times they log in and view other materials, and whether it changes a perception of how we learn. so right now we’re at the…

Rebecca: And performance too, right?

John: …and their performance.

Rebecca: And is the plan to start collecting that data in the fall?

John: The plan is to put all this together the spring, I’m hoping and then to submit a proposal to the IRB, and then to conduct the study and the fall and the spring, at least for a first stage and then we’re hoping to be able to follow these students up, to see if this has a significant effect later in terms of their grades or their persistence.

Rebecca: Sounds pretty exciting. I’m looking forward to hearing how that goes.

John: It is. I’m looking forward to it being all together and actually being implemented. I think it’s an interesting study.

Rebecca: We’ll have to have you back, John.


John: I think we can manage that.

Rebecca: Well, thanks so much for sharing all this information about peer instruction. I know it’s something that I’m always kind of asking you about and like to hear about, and I’m sure others will too.

John: Well, thank you.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.

35. FLEXspace

Learning spaces greatly influence the teaching methods and learning activities we can employ. In this episode, Lisa Stephens and Rebecca Frazee join us to discuss the Flexible Learning Environments eXchange, an international platform for archiving, exploring and planning informal and formal learning spaces.

Lisa serves as the Assistant Dean for Digital Education in the UB School of Engineering and is a Senior Strategist for Academic Innovation in the Office of the SUNY Provost. Her work at SUNY includes serving as the Interim Director of FLEXspace. Rebecca Frazee is a member of the San Diego State University faculty in the Learning Design and Technology Program. She serves as the FLEXspace Manager.

Show Notes


Rebecca M: Learning spaces greatly influence the teaching methods and learning activities we can employ. In this episode, we discuss the Flexible Learning Environments eXchange, an international platform for archiving, exploring and planning informal and formal learning spaces.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca M: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca M: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guests today are Lisa Stephens and Rebecca Frazee. Lisa serves as the Assistant Dean for Digital Education in the UB School of Engineering and is a Senior Strategist for Academic Innovation in the Office of the SUNY Provost. Her work at SUNY includes serving as the Interim Director of FLEXspace. Rebecca Frazee is a member of the San Diego State University faculty in the Learning Design and Technology Program. She serves as the FLEXspace Manager.

Rebecca M: Welcome.
Our teas for today are:

John: Are either of you drinking tea?

Lisa: No, but I really am jonesing for my gunpowder green at home. It’s highly caffeinated.

Rebecca F: Yes, my tea was a cinnamon vanilla yogi tea.

Rebecca M: That sounds tasty. How about you, John?

John: I have a blackberry green tea.

Rebecca M: …and I’m trying the berry blush black tea which I’m finding smells good not warm, but warm… I’m not sure about. It’s kind of weird.


John: We’ve invited you here to talk a bit about the FLEXspace project. Could you tell us a little bit about what FLEXspace does and how it’s used?

Lisa: FLEXspace is an open repository, free for use to all of higher education, and now K-12. The idea is to make exemplars of learning environments available for people to ideate from, to share best examples with each other, and to benchmark against peer institutions, and also to just use it as a general reference on how are we doing? How can we think about improving our learning spaces in general? Money isn’t falling off trees for higher education right now and we’re certainly all under a great deal of scrutiny in terms of how those precious resources are spent…. and if you go back 20 years or so it was not uncommon to go visit a peer institution by driving to the next state or perhaps flying a small team of people to another institution to really understand how they developed one of their spaces. One of the goals was to create, essentially, a virtual field trip to enable people to visit each other’s campuses and look at those spaces, not just by the visual cues, but also to dive into the details of those rooms in terms of: the type of equipment that’s used, the fit and finish of a particular space, how something may have been planned for. We don’t include detailed budgets but we include aggregate budgets if people are willing to share that information. So, for example you wouldn’t drill down to the level of the cost of a particular piece of equipment, but sometimes people share the aggregate cost of all of the A/V and information technology resources in a room… or the cost of the furniture… or the cost of the entire rehab… things of that nature. FLEXspace is actually an acronym, it stands for the Flexible Learning Environments eXchange, and we have found that people think of FLEXspace as flexible furniture… and interestingly enough we always thought of the emphasis being on the exchange, small e, large X,because it’s all about sharing spaces and best practices within an online community of practice. It was created by educators for educators.

Rebecca F: … and when I think of FLEXspace, I have come to realize, talking to different users over the the years now, that it really centers on the community. So, when I think of FLEXspace, I think of it as, from the users’ perspective, why does somebody come to FLEXspace? …and mainly it’s to get ideas… advice… to get assistance… to help make decisions about a learning space project. FLEXspace provides that community and provides that support through detailed examples of learning spaces, other resources, best practices, and so forth… and also that community network that help connect people who have questions with those who have experience and advice and answers.

Lisa: Yeah, FLEXspace is a large global community. Think of it as a big portal that serves as a one-stop shop for all those best practices. It’s considered an open resource. It was always developed as an OER in mind but we also wanted to make sure it was password-protected, so everybody within the community felt safe sharing within the community… especially if you’re going to share examples of rooms that might not be ideal rooms.

Rebecca M: You mentioned it’s a global community. Can you talk a little bit about who is a member and how one becomes a member?

Rebecca F: We’re super excited that we continue to watch our user membership grow week by week. Right now, we’re proud to say we have over 3,000 registered users and those individuals are from about 1400 different unique institutions from 45 countries. So, we’re all pretty proud of that.

Lisa: You become a member just by going to the website flexspace.org and once you receive a login you can go in and browse the collection… It’s a really neat tool. It really is designed to be the one-stop shop for people that are either designing… envisioning spaces.

Rebecca M: How did FLEXspace get started? It’s such a big community now.

Lisa: This is such a neat story. We were part of a task group… the Faculty Advisory Council on Teaching and Technology… when our Provost walked in and said: “For goodness sake, we are spending millions and millions of dollars each year at campuses all across the state.” We have 64 campuses… we’re the largest most comprehensive system in the United States. So, when you think of everything in the range from community colleges all the way through research universities, we’re all under the governance of SUNY… and Provost Lavallee was looking for more efficient ways for us to share ideas with each other… and he said “if we could even share pictures with one another, that would be helpful.” So, a small team of us got together and said: “Well, how the heck are we going to share information with each other? Nobody wants ads pushed to them… nobody wants to feel like their data is being collected…” and we were talking about that long before the Facebook challenges that have recently come to light. So, we ruled out the common platforms like Facebook or Flickr or any of the open tools that were available back when this was starting up. So, that was the impetus to get everything started. The first thing we thought through is what is it we need to share. Well, we knew we needed to share photographs of rooms, but we also figured out it would only have value if there was a lot of data and interesting details about the spaces… and then that only goes so far, because you really need a point of contact to call someone that was involved in the space… to ask those more detailed questions: “Would you use this particular piece of equipment again? Would you work with this particular group again? Tell us more about why you chose a particular path forward.”

We decided to just collect the details, figure out the attributes, and it happened that we stumbled into a partnership with Artstor… and at the time they were interested in that OER element as well. They were looking to empower campuses with the ability to create and curate their own collections. So that’s how we got started.

Rebecca M: It’s a neat story. As a designer and an educator I can see so many different ways that a system like this could be used, but I have some questions. As an instructor I know that spaces can impede certain kinds of learning activities and then spaces can also enable certain kinds of activities. So, I would imagine that instructors coming to the table with the kinds of things that they’d like to be able to do would be an important way that they’d be able to search images and then ask those same questions that you’re mentioning about how effective has it been for those kinds of learning opportunities. Can that be done?

Lisa: Yeah, thanks for setting that question up so nicely Rebecca and I’m sure my FLEXspace partner Rebecca will dive into more detail on this… but when we were talking with people throughout the community, we realized that the biggest challenge people have in spaces are looking at space development through the lens of their own expertise. Now, you just laid out the faculty expertise, but a facilities planner or an architect is probably going to view how to design and build a space differently and, of course, the A/V and IT integrators are certainly going to have a different set of concerns when they’re thinking about the integration of the technology tools in a particular classroom. For example, an architect may implement a more cost-effective air handling system, but the a/v integrator will be concerned that a particular space is going to be used for video conferencing and they don’t want any air handling noise to come through the microphones. So, you need to have that communication to say: “Alright, are we going to use ceiling mounted microphones? Are we going to use desktop table microphones? Are we going to use a different type of system? Does everybody in the classroom need to be heard?” So, there’s all sorts of opportunity to think through from the perspective of your expertise if you’re a teacher or a technology integrator or an architect.

Rebecca F: Yeah, and I’ll add to that. We’ve been hearing different user case scenarios when we go out and talk to folks. We show them FLEXspace and how we intended it, and then they tell us: “Oh, wouldn’t it be cool if we could use it like this?” …and so we’ve been really excited to get these examples. For example, there are some campuses that are saying they want to provide a resource for faculty. Let’s say you’re a faculty member and you are going to be teaching in an active learning classroom. You’ve never been in the room and from the comfort of your own laptop you can actually log into FLEXspace and look at detailed photographs, even a 360-degree view of the room… details about exactly what equipment is in the room… moveable tables and chairs… and whiteboards, and so forth… and so you can see what’s in the room. It kind of sets expectations, and then the campus can also upload instructor support materials. So maybe there’s tips for teaching in an active learning classroom, or a job aid, or instructional how-to quick sheet on how to use the interactive display or set up some of the equipment in the room. So, some campuses want to use FLEXspace in that way. One person suggested that they would want to use FLEXspace to help their own classroom support technicians. Let’s say you’ve got your technician and they need to go out into the field, because there’s some sort of piece of equipment that’s broken. They can log into FLEXspace… they can pull up all the specs that are in that room… they know exactly what model of projector is in there before they even walk across campus. They know what to take… what’s in the room… and so forth. So, we’re providing the resource and then we’re excited to see how people will use it. S

John: Some campuses are using it then for their own internal purposes as well as for sharing ideas… in terms of providing information about what’s available in each room.

Rebecca F: Exactly.

Lisa: We’ve learned a lot since 2011. When the task group launched in 2011, we were able to pilot in 2012 and then we actually had something into beta by 2013. and by that time we had received some attention from Herman Miller. We had talked with them at a couple of conferences and asked if they’d be interested in helping us to develop this tool. So, they gave us good ideas and as we went to the Consortium of College and University Media Centers, the CCUMC conference, they actually invested in the platform and helped us get it off the ground because they realized the benefit this would have for their members. Then ELI got involved from EDUCAUSE. …and SCUP, the Society for College and University Planning… and each one of those groups has their own tribe, if you will. So, SCUP tends to have a lot of high-level campus planners, people that are working on the financial resources side… and the architects. Of course, EDUCAUSE tends to have more teaching and enterprise level IT support people. All these different voices coming together really helped refine the taxonomies and we had over a hundred volunteers across the country pop in and help us to create and populate these room attributes that describe these rooms in great detail through the lens of each of those expertise areas… be it the faculty and the pedagogical perspective… or the architects and facility planning perspective… or again from the IT and A/V and academic technology perspective.

Rebecca M: I’m really familiar with the Artstor platform given my my background as an artist. I’m envisioning images and a database and all these different ways that you can search metadata…and you just mentioned taxonomy. Can you talk a little bit about some of the different ways you can search and find things? What are some of the attributes that are available to search?

Lisa: Historically, it was all focused on peer benchmarking. If you wanted to look up a classroom you would go naturally to one of your peers and see if they have any information uploaded and then you’d start to look at the type of teaching that was taking place in the room or the type of curriculum. Rebecca why don’t you take that one because you can speak more directly to the new portal under development.

Rebecca F: Before I even came on board the project there were a lot of smart people on this that came up with this taxonomy to figure out what were the different dimensions… aspects of the space so we have these main buckets if you will of descriptors that are looking at the layout of the space: the furniture, technology equipment, other kinds of furnishings in the space (like whiteboards)… different kinds of equipment in the space, even the facilities aspect. so for instance the ceiling… floor… wall… color… maybe it’s a certain kind of LED lighting in the space. We have all of those aspects and then, in addition to that, the types of learning activities that are supported in the space. That might be… if this space is used for active learning, a small group collaboration performance space, and so forth. As we evolved the design into this what we’re calling FLEXspace 2.0, after doing even more user experience research, we discovered that there were a handful of data descriptors that we could require and that way we provide some consistency for the data that’s in there… and then users can open text search the entire collection. They can also use filters and that’s for that required data came in so they can filter on the type of space. So, for instance, it might be an active learning classroom… general purpose classroom… media or computer labs… science lab… makerspace… even informal spaces like a Learning Commons… transitional area… performance space… that sort of thing… and then we can also search on seating capacity, because we found that was very important for users. If they wanted to search on examples, but they’re looking for a space that would accommodate a hundred students… or they might be looking for a space for 30 students… so we have filtering capabilities in that way. You can also filter on “what’s the project scope?” Is this a completely new construction project? Is it a renovation? Is it a room refresh? You can search on that and also, as we’re expanding beyond higher ed examples, we’re now building the community in k-12… you can filter on the institution type… and so we learned that this was important. Is it a private university? public university? is it a community college? a k-12 environment? and so forth. So, there are a lot of different ways you can filter and search the collection.

John: It might be helpful if you talk about perhaps some case studies of how some specific campuses have used this to help move forward… moving away from the old traditional classroom structures.

Lisa: Yeah, that really is the easiest way to explain the value of FLEXspace, John. Thank you. One of our early core team members was working at Iowa State at the time. They had a very large project underway, and in addition to FLEXspace being a very useful tool, another tool was under development at the time out of a constituent group out of EDUCAUSE – called the learning space rating system. So, their concept was to imagine the LEED environmental credit system and if you could apply that in a quantitative manner to the learning potential of different types of spaces you could actually measure the potential effectiveness of a space… and it’s nice because it’s quantitative… because then you can rank the condition of the space across a number of different measures. So, that’s exactly what they did at Iowa State, they went through a large audit of a number of classrooms with the LSRS and then gathered a very diverse group of people around the table representing faculty, and the facilities planners, A/V IT integrators, I think they even had a couple of students, and of course the financial people were represented. They said “Alright, how are we going to work together to have a conversation about the type of space that we need to build. We have an opportunity here.” So, they parsed out groups within the main group and said “everybody get a FLEXspace account” …and you know what’s going to happen is the faculty member knows of a very cool space at another campus, so they’re gonna go in and immediately search through FLEXspace to find a really cool classroom… and meanwhile the biology professor is going to do the same thing… and you might have a GIS teacher going after a similar type of search… They found, when they came back together, that a number of the rooms they were looking at may be different, but the room attributes were very similar. Now, you can start to have a real discussion. Why did you like that particular space? or better yet “Wow, we each picked the same space, why? …and they started to narrow the selection of the spaces they were looking at in order to put in front of their executive leadership. These are the types of exemplars. These are the types of spaces that we’d like to build here on our campus… and this is why we think they’re valuable.
Another good case study would be from SUNY Geneseo. SUNY Geneseo went through a very similar process where they were looking at recommendations from faculty. They started by saying “Gee, these 17 rooms on campus are in need of refresh…” and the more they thought about it they said “Well, let’s do something similar to what Iowa State had done. They took the same approach as Iowa State where they did an audit with the learning space rating system and then went through, ranked their classes, considered other metrics like enrollment, and decided to invest in three or four different classrooms to bring them up to beautiful new classrooms… active learning spaces… things of that nature… but they found a very similar process… where a group of people around the table using FLEXspace to serve as the glue of the discussion about why they like certain classrooms and what the attributes were that they felt were important to have on their campus.

Rebecca F: …and I can add a couple of short examples. When I started working on the FLEXspace project, I was focused also in my role… I teach at San Diego State… and so I was working with some of the Directors of Academic Technology at some of the different California State University’s campuses. I had one instance where the new Chief Learning Officer there was responsible for designing and building and planning some new active learning classrooms, and he found himself in a position of not being so familiar with what was the latest and greatest going on in this arena. So he reached to FLEXspace to get prepared… sort of get up to speed on what are other campuses doing in this area before he met with the architects, so that he didn’t go into that meeting with a blank slate. He could go into FLEXspace, look for some examples, and then bring those examples to the meeting, and say “Okay, we’d like to do something like this. How much would that cost?” or “Can we do that here? Is it feasible?” So he was able to get some examples and feel a little more informed going into those discussions with the architects and the facilities planners.

…and another example… There was a meeting of Library Directors that all met here in San Diego State on campus, and they were going to be discussing how to convert some of their library spaces into group study spaces… maybe maker spaces… and so forth. …and so the Director of Academic Technology there at San Diego State, he went to FLEXspace and started searching through examples of how different libraries across the country had converted their spaces. You can go in there and you can tag your favorites, group them into a collection, and then he took that collection of maybe a dozen examples and brought that to the meeting so that he could show those to this library directors’ conference and say “Okay, here are some examples…” and then used as a conversation starter to say “What are some things that you like that you see here? How does that spark other ideas for your campus? and so forth.

Lisa: We’ve been really lucky, because we’ve had experts all over the country volunteer time to help us refine this system. I think we’ve got a bit of a tiger by the tail because it’s clearly meeting a need. The combination of the learning space rating system with FLEXspace is invaluable in terms of having two free tools to assist people at a number of levels on campus to have an effective learning environment.

John: …and it’s won some awards, hasn’t it?

Lisa: Oh, thank you, yes. We did receive the Innovators Award in 2016 from Campus Technology, which was really cool, because it was a nice way to acknowledge all of the efforts, not just of the people who participated in developing the system, but at the end of the day if you want to keep a tool like this free to all of higher education or K through 12 or whoever wants to create an account to use it, you have to find a way to support it…. and we have some very generous sponsors that have been helping right from the get-go. Certainly Herman Miller played a huge role in getting us off the ground and there’s been a number of other sponsors that have stepped up and made it possible to keep this free… and we’re challenged to continue to keep it as a free service moving forward because this new portal development is pretty pricey… but we think we’re going to have features that are really really going to surprise people. Do you want to talk more about the sponsor relations, Rebecca?

Rebecca F: Yes, so we’re really grateful for all of our sponsors and we’re excited to be able to even recently announce that Herman Miller, in addition to being our founding sponsor, they are going to continue as our Platinum premium sponsor moving forward with the new FLEXspace portal. So, we’re really excited about that. What’s been great is that our community of academic users welcomes our industry partners. They see them as trusted partners. They want them to be involved in the community, so they don’t see it as some sort of direct advertising… and it’s not that at all. It’s really that this is community of experts from industry and from academia who are coming together dedicated to improving learning spaces and sharing resources that way. We feel like we couldn’t do it without our partners like Herman Miller. We also have our Gold Partner Computer Comforts who is another furniture manufacturer who’s come in this year and they’ve really taken a leap of faith because, as we build the new portal, they are supporting us and they see the value of FLEXspace. They see the value for the education community and they want to be a part of that. So, other sponsors we have including Shaw Contract, some AV companies (AVI-SPL, Crestron, Wolfvision, MediaSite, FSR, Sony) and another furniture manufacturer AvinEd, they’ve all stepped up and given their support to FLEXspace to make sure that we are continuing to thrive and grow and provide that free resource for the academic community.

John: You started on Artstor. Are you still using the Artstor platform or has it moved to a new site. You mentioned the transition.

Rebecca F: We started with Artstor and we were so grateful for their support. They had this wonderful platform that really focused on images and the descriptive data. We were able to start with Artstor and use it for FLEXspace and then over time we realized that it wasn’t necessarily built specifically for the needs of our users… and so we migrated away from Artstor because we did this user experience research. We found out that our users, first and foremost, needed the platform to be mobile. They needed some other collaborative features in there as well, and so now we’re in web development with a different platform. We’re really building it from the ground up at this point. We’ve got some new features in there that support collaborations. So, for instance, let’s say I’m at San Diego State and I want to upload an example active learning classroom at our campus. I might not have time or I might not have all the details to describe the space. so one of the new features in the platform is that you can add collaborators or co-editors when you’re uploading a space. I can start it. I could be out in the field with my mobile phone or tablet. I could snap some photos, upload it, and create a FLEXspace entry, and then I can tell somebody from my team “Hey, will you go in there and add the details because you’re really familiar with all the A/V that’s in the room…” and then I can add somebody else from my team as a co-editor and say “Hey, can you describe the furniture that’s in the space?” I can even ask a faculty member who’s teaching in the space and add them as a co-editor and and say “Hey, will you go into this record for the active learning classroom, room 101, and tell a little story about how you’re using the space? What’s working? What would you do differently? How are the students responding to the activities?” and so forth. So, you can really have more collaboration and, like Lisa was saying earlier, have many more perspectives to describe the space. it makes it even more useful when you’re reading the examples.

Rebecca M: I have to say as a web designer I got really excited because you’re using all the best practices for designing a platform like this so kudos for her user centered design it makes it better for everybody.

Rebecca F: Yeah, absolutely. We thought that was really important because it took on a life of its own and it really started gaining momentum and growing but it really hadn’t been designed with those user needs in mind. We started the deep dive into the user experience research with the higher ed community and, really, it was more focused on users in the United States. We also have an International Committee as part of our core team, and that’s one of our next steps. We want to do a little more research into what are the international users needs as well. Obviously, there’s a lot of overlap, but there are some differences in what their needs are… and then we also want to take a deeper dive into what are the needs for the K-12 community. Again, there’s overlap, but there are some subtle nuances there. We’ve been really fortunate in this dedicated community, like Lisa was saying, people who are contributing their time. We have some new representatives on our core team from the K-12 community as well… and we formed a partnership with ISTE.

Lisa:Now we have a custom portal and we’re so fortunate that Xennial Digital has really gotten behind this project with their expertise and their web portal development because they have experts and they’re able to turn this into a very dynamic community of practice in a way that, frankly, I don’t think we would have been able to afford any of this without their support. So, we hope that this is serving their purposes as well and showcasing what they’re able to do with a portal like this.

Rebecca F: FLEXspace started as a place to showcase exemplar spaces, but now we’re hearing more and more that in higher ed and K-12 they want to use FLEXspace to document and catalogue all of their spaces… so not only the shiny new innovative spaces but also the more typical spaces on their campuses so that they can have this classroom directory and so they don’t have to build their own. There are some campuses that have built their own directories and they have the support to do that, but we’ve heard time and time again… there are many smaller campuses or those that just don’t have the resources and they don’t want to build their own directories, so they can they can use FLEXspace…

When we’re presenting about FLEXspace, hands go up and they say: “When is the upload feature going to be ready? Because right now I have a Google spreadsheet with all of the data on our classrooms and I want to get it up into FLEXspace.” We’re really excited about that, and that takes it in a new direction as well.

So, you can search for exemplars, but you could also use it in the in this other way… to show these documented spaces to your faculty who are teaching in them, to your tech support folks… maybe you’re looking at this as sort of an audit of your classrooms – and then in conjunction with using the LSRS you might rate your different spaces… and some of them are in dire need of improvement… but you’ve documented those… you’ve got photos… you’ve got details… you can do the LSRS rating… and then you can go in and improve the space… and then upload the example of the renovated space… so, sort of the before and after photos as well…

Lisa: One of the really nice features of the learning space rating system is that it quantifies the value of what spaces need attention… and once you have that ranking it tends to take some of the politics out of the equation. We’ve heard from a number of people that once they go through that quantitative exercise, then you can dive into FLEXspace and start playing with the feature sets.

Rebecca M: You’ve been talking about a new system, and I was wondering whether or not that was actually implemented or not. Is that what’s up and live now or is that something that’s in progress?

Rebecca F: We have a phased rollout of the development of the new portal. Right now, it is indeed live and you can request an account. Go to FLEXSpace.org, request an account, you can log in and you can see you can filter and see all of the examples of learning spaces that we migrated over from the old system… and then, even as we speak, we are in development of the next feature set… which is the case study template feature that allows you to upload examples. Right now, you can look at examples and in June of this year you will be able to upload your own examples. and we’re also launching the membership directory and community forum. Prioritizing all the different features that we have on our roadmap, we felt like the community aspect and then obviously being able to upload spaces were the biggest priorities and so that’s what is coming soon.

Lisa: A lot of people are familiar with the website Houzz – H-O-U-Z-Z….

Rebecca F: Houzz or Pinterest… those kinds of tools.

John: I’m one of those people who is not familiar with Houzz. Pinterest, I’ve seen.

Rebecca M: Yeah.

Lisa: Well, we think that this portal that’s under development, FLEXspace 2.0, is going to rival the features that people are already used to seeing in applications like Pinterest…. and Rebecca can speak to the idea boards and the tool kits. It’s going to be a really nice way to keep track of your own ideas… pull together a collection… and then perhaps invite others to come in and ideate in that space with you.

Rebecca F: Yeah, so you were asking what are some of the things that you might find in FLEXspace. Originally, the collection consisted of examples of spaces and so that meant a lot of photographs, a lot of detailed tags and descriptions, and then also users can upload, let’s say, a floor plan, a detail spec sheet, maybe details about the wiring layout of the room, or the wall colors, and so forth… so, lots of details and lots of images. But, now we are expanding those kinds of resources. We are including more kinds of spaces, not just formal learning spaces but also informal learning spaces and different types of settings… so, more k-12 examples, museums, libraries, more kinds of details. You can upload video, 360 images, more details about the case study of the space (including evidence of impact and efficacy). We’re also encouraging the upload of more kinds of support resources… research papers, white papers, things that don’t necessarily have to be tied to one particular space but more broadly useful resources.

We’re also building in more ways to connect. You can add collaborators or co-editors to your space. We’re offering the member profile and membership directory, so you can go in there and see who else at another campus is working on learning spaces. You can connect with them. We are starting the community discussion forum. If you have questions about a particular challenge that you’re having you can participate in that discussion forum. We’re also starting to use this new tool called an idea board. It’s sort of like that Pinterest board. You can tag your favorites, but you can also create folders or collections. Let’s say you’re thinking about creating a new makerspace or a STEM lab. You can start gathering inspiration and create this idea board called “ideas for makerspaces” and you can keep your idea board private… you can add your team members to it so that we could all be adding to the same idea board. Let’s say we’re working at San Diego State and we want faculty to be giving input… we want the technology folks… the facilities folks… maybe it’s housed in the library… we want somebody from the library staff. We can all be added as collaborators and all be adding ideas to this idea board. so that when we talk about planning our makerspace we can see what everybody else is finding is inspiration.

Lisa: One of the guide posts with our sponsors and they both brought this up and agreed to it is we know that if anyone were to use the portal to pull data or use this in some fashion to solicit to the members that we’d all be dead in the water. This is a safe community forum place where you can interact with people who understand the details of new products or new innovations that they’re offering, but you aren’t going to be hounded inside the portal in any way. We know it’s a big concern these days about the amount of email everyone’s getting. Our sponsors get that people don’t want to be hounded. We’re trying to find that fine line between “Look folks, if you want it to be free we have to provide something of value back to the sponsors…” and I think everybody understands the trade-off in that environment. So, the sponsors are going to have access to the tracking and the analytics and the things that are valuable to them, but we want to keep everything upbeat and positive around that community.

John: So, both groups benefit… the sponsors as well as the participants because the people who are looking at these spaces are going to be designing things and getting access to information about the sponsors and information from other users, I think, could be quite useful.

Lisa: The sponsors actually, depending on the level of sponsorship that they participate in have a space within the portal to talk about their new products and to be able to respond and interact with people on the platform within the portal. We think it’s a useful communication tool and the vendors are welcome to be a part of that community of practice. That’s what’s made CCUMC so successful over the years, is having that relationship with the vendors in a safe place where people can exchange ideas freely without feeling pressured.

John:okay so this started off as an initiative from a small committee and it’s grown into this huge collaboration space. How have you been able to maintain momentum of keeping this going?

Lisa: Adrenalin?
The momentum has been carried on by the community. I think the most exciting thing that both Rebecca and I experienced when we were at EDUCAUSE this past year was walking into a room and hearing other people talk about FLEXspace and talk about the value of FLEXspace and how they want to be a part of it without us initiating the conversation. That was a groundbreaking time for us. It was very energizing.

Rebecca F: Yeah, I agree. We’ve been on the road for the last couple of years. You mentioned we did get the Campus Technology Innovators Award… I believe that was in 2016. We’ve had a lot of support from affiliate organizations like UBTech this year. We’re a program sponsor at that conference and we were a program sponsor last year. We’ve gotten a lot of support and encouragement from Infocomm, Campus Technology, the folks at EDUCAUSE. We’ve been doing a lot of outreach by giving conference presentations, workshops, webinars, and really it also comes from I’d say a very dedicated network of our colleagues who are all committed to improving learning spaces.

We regularly meet with architects, A/V integrators, faculty members, researchers, leaders in academic technology and information technology. We have our core team meetings. They are out there in the field being FLEXspace champions. They spread the word, they encourage others to take advantage of FLEXspace and so forth… and to echo what Lisa said even this year just recently we were excited that somebody from the ISTE organization (that’s the International Society for Technology and Education, they’re a big professional organization for the K-12 community) they reached out to FLEXspace and said “Hey, we want to partner with FLEXspace. We see this as a valuable resource. We don’t want to reinvent the wheel for the K-12 community. Can we partner?” …and so we’re planning to do some sort of train-the-trainer activities with some of the ISTE Learning Spaces Professional Learning Network Committee. They will learn more about FLEXspace and how the K-12 community could benefit from it… and then they are going to go out at their annual conference, and through their local chapter meetings, and really be proponents of FLEXspace. So, it really has garnered the support from the community and that’s why we want to keep listening to the community’s needs to make sure that we’re providing a very valuable resource that they want to keep going back to again and again.

Lisa: …and a shout out to the vision of the people at the systems as well because SUNY has been very generous with allowing this to grow… enabling us to put time into it… which is of course the most valuable resource and to the Cal State University system which has provided generous support to enable this partnership between SUNY and Cal State and Rebecca and myself.

Rebecca M: Well, this has been really great. I find it really informative and I look forward to jumping in and trying to use this FLEXspace platform for some of our upcoming renovations on our campus.

John: I know a number of people here do have accounts and have been using it.

One thing we always ask is: “What are you going to do next? In this case, where is FLEXspace going next?

Lisa: We think that the continuous quality improvement cycle is going to be at play in a big way here, because once people get in to the new feature sets and see all that they can do, we’ll probably spend about a year spreading the word about the new features… and then I have no doubt that people are going to have fresh ideas about how to improve it. So, as long as our sponsor community continues to support us, we will continue to improve it in any way that the imagination seeks.

Rebecca F: Yes, and I will leave you with this one statement here. Our vision is that we want FLEXspace to become this one-stop-shop for best practices, detailed examples, and a community dedicated to improving learning spaces around the world. So, that is where FLEXspace is going in my mind.

Rebecca M: Well, thank you both for joining us and taking some time out to talk to us about this great endeavor and all the time you spend working on it.

John: Yes, thank you.

Rebecca F: Thank you

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca M: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer. Audio editing assistance provided by Nicky Radford.

34. Flex courses

Working towards a degree for some students can be a struggle as they balance full-time work, families and coursework. In this episode, Marela Fiacco, a Healthcare Management Instructor and Curriculum Coordinator at SUNY Canton joins us to explore options that give students greater access to courses and co-curricular activities. Dr. Fiacco is the first instructor at her institution to teach a flex course, a modality in which students may participate either in person or remotely.

Show Notes


Rebecca: Working towards a degree for some students can be a struggle as they balance full-time work, families and coursework. In this episode we’ll explore options that give students greater access to courses and co-curricular activities.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


Rebecca: Today our guest is Marela Fiacco. Marela is a Healthcare Management Instructor and Curriculum Coordinator for the Healthcare Management Program at SUNY Canton. She is also the first instructor in this program to teach a flex course. Welcome Marela.

Marela: Thank you, thank you for having me.

John: We’re happy to have you here. Are you drinking tea?

Marela: I’m drinking water.

Rebecca: It’s a nice, healthy choice.

John: Our teas today are…

Rebecca: I’m drinking English afternoon tea… again.

John: And I have Harry and David’s Bing Cherry Black Tea.

Marela: Oh, that sounds yummy.

John: It really is. It’s hard to find– you have to go to a Harry and David store or order from them online, but it’s a Republic of Tea tea that’s custom made for them.

You were the first instructor at your institution and one of the first in SUNY, I believe, to teach a flex course. Could you tell us a little bit about what a flex course is?

Marela: Certainly, at SUNY Canton we received a grant from SUNY system, I believe, and this is by our Dean of Instructional Technologies and the idea was born to create what we called at the time, probably about this time last year, we called it a converged modality classroom delivery. We can call it a flex class or a converged modality. To us at the time, it was a mix of face-to-face and an online class, so we have students in a face-to-face class and students in an online class. Students in an online class can watch the video live or they can watch a recording later. Face-to-face students, if they are not in attendance, could watch a recording later. It’s really a mix, and when you say a flex course that’s what it is. It provides flexibility.

John: A flex course, then, is a combination of a face-to-face and an online class. Are students free to choose the modality, or are they enrolled in one section or the other?

Marela: Since this was a pilot, it was really difficult for students to understand and determine when they were first registering for a class. We have since gotten a little bit better and created two different sections and they understand what it is and they understand what they’re signing up for. At the time, we just provided a face-to-face and an online section. Those students who were purely online and couldn’t come to class physically, they chose an online version. Little did they know that they were going to be in a converged class, or a flex course, but at the very beginning I had a video done where I was explaining what it is and should they choose not to participate they could have picked a different section… but, this was just something added that they could benefit from. It wasn’t anything that we were taking away from their online experience.

John: So, if they were in the online course, could they attend face-to-face classes?

Marela: They absolutely had an opportunity to participate in a face-to-face class if they chose to do that. But, I think, for the health care management program, we advertised it as a hundred percent online program, and so most of our online students choose online because of the convenience. But we also do have students on campus and commuters who come to class face-to-face.

Rebecca: What was it like teaching a flex class?

Marela: Quite frankly, it was a lot of prep work at the very beginning. Well, first I thought: “How am I gonna do this? I’ve never done anything like this. It’s going to be a lot of work and then I thought “Well I have to start from somewhere.” I taught this class both online and face-to-face in the past, so that helped. We teach in Blackboard. I took the online class and just really took a hard look at it, and thought to myself “How do I make this class more user-friendly for both groups. I modified my online class. I of course added both on converged modality. We have videos for each lecture capture, so each time I’m in class and I’m lecturing I am using those to upload into those weekly modules for those online students to watch and for livestream. The thing I really wanted to achieve is I didn’t want to have it separate. The face-to-face students, I think they had more work than anybody else. They were asked to participate in discussion posts online together with online students. So, for them, it was almost a hybrid. They were supposed to upload assignments in Blackboard and also discussed whatever topic it is… whatever questions that we had for those weeks in Blackboard.

I also created a group assignment that I think was a bit of a challenge for all of us, because I intentionally picked the groups and they didn’t have any say in that and I picked online students and face-to-face students, and provided him with links in Blackboard Collaborate to get together and work on their assignment. That was a challenge for all of us.

John: From your perspective, are you compensated for teaching two courses or is it treated as if it’s one course, in terms of your workload.

Marela: I did get an extra that an adjunct would be paid or that a faculty member would be paid for teaching an extra course, because of the workload. Because this was a pilot because we capped both of the sections at 15 students normally our caps are at 30. So, I did get an additional pay last semester for teaching this converged modality or flex class. Moving forward, I don’t think that they’re actually doing that with other faculty.

John: To get things started, it often helps to give a stipend to encourage people to experiment. In the future, would the combined sections be capped at the same level as a single section would have been?

Marela: They are capped at 30 and it’s just one section.

John: …and then students are free to either attend in person or online. That’s what I was thinking..

Marela: Yeah.

John: …because that’s what I’ve generally heard about flex courses. I just haven’t seen many examples of them in practice yet.

Rebecca: What was one of the biggest challenges you had as an instructor? and what might you do differently next time you teach a class like this?

Marela: One of the biggest challenges was technology, to be honest. Just working out the kinks… because this was new for all of us, including the online support staff. You come in and the camera is not working… or the sound’s not working… videos not working. We might be missing a livestream or we might be missing a recording. It was the technical difficulties that were really the hardest. I struggled with attendance in the face-to-face class, because now students are thinking: “This is super flexible, I don’t have to show up.” So, I struggled with the balance between allowing them to have flexibility and the fact that you signed up for this class. So, you want to provide students that flexibility if there is a snowstorm and we lose power and whatever it might be and if they’re traveling a distance. So that was kind of the fine line, but I think each instructor determines their own attendance policy, so every one of us will approach it differently, I suppose.

John: Did you use any tools such as polling or quizzing in class where students had to participate either virtually or physically?

Marela: No, not this time. Just because the technology was new and I think next time I do it I definitely would want to do that. But at the same time, that is another hard one, because the online students, the reason that they take online classes is for convenience. Most of them are working professionals. In my program, 85% of the students are working professionals. The classes… let’s say nine o’clock in the morning… well, they can’t exactly participate live. They do appreciate the recording later at night working on their homework or whatever it might be.

John: So, for online students synchronous attendance isn’t required? It’s an option but not required?

Marela: Correct.

John: What would you say would be the major advantages that students get from this sort of offering?

Marela: I think the greatest advantage for online students specifically is the lecture capture. If there are any misunderstandings about the assignments, whatever it might be they, they actually get a lecture instead of being self taught if, you think about a purely online class. Another thing too, is it provided greater connectivity with students. I really can’t stress that enough, that it really gave me an opportunity to connect with the online students… one that I wouldn’t have otherwise… because it often feels disconnected from the campus. With this grant, it wasn’t just the converged modality, it was also to connect with online students. We invested money in live streaming our Excellence in Leadership lectures and speakers so that we can bring online students and have them participate in different things on campus. We really wanted to create that connection and that’s one thing that they really appreciated the most… that was their feedback. They want to see us, that we exist… we’re here… and that we care enough that we want to do this for them. So, I think that was one thing that they really appreciated the most.

John: It created more sense of instructor presence and more of a connection to the institution.

MAREA: Yes, and they feel a sense of community… they feel a sense of belonging.

Rebecca: Did you find that a lot of the students took advantage of some of the extra things that the college invested in, so they could take advantage of those extracurricular opportunities?

Marela: They did. They really did… and we were really pleased with that… and actually the Dean of Students, myself, the Assistant to the Provost, and the Dean of Instructional Technologies we are presenting at the CIT conference and we are reporting our findings, not just on academic side but also the extracurricular and non-academic piece of it… and what is it that students took part in, what they enjoyed the most, what is it that we should continue doing for our online students. We have a large population, and let’s face it, we are all looking to online to look outside of local and geographic area because our enrollments here are really declining because of the graduation rates in high schools. We are all looking for ways to connect with online students on different levels to make them feel part of our campus and our community.

Rebecca: The extracurricular piece seems like it’s one of the most powerful additions to this particular opportunity because I think you’re right that the students, when they’re taking a class online and they’re not coming to campus, they miss out on a lot of that intellectual development from these other points of view that we don’t always offer just in the class… having the opportunity to get involved just seems like it would be really exciting for some students.

Marela: It is. It is. It’s very exciting for them. Actually we have had, for instance, one time we had a CEO of one of the local hospitals come and speak… particularly to the networking and career goals and things like that… and the students were very interested… and actually emailing and asking when is the live stream going on? Usually, these are in the evenings… part of our Excellence in Leadership series. They really wanted to take part in that… listen and understand their career options. or whatever it might be. So, those are some of the things of value to them. Actually, I was just talking to the President the other day… we had our scholarly activities, where students come and faculty come… present their poster presentations… and present their research and such… and I was just talking to him and I said: “You know, it would be great to involve our online students in these scholarly activities a little bit more, not only our engineering students and nursing students who are here on campus, but because our online students are getting involved a lot and some of them are lobbying. They’re involved in so many different activities… in their own communities and some of them do research, so it would be really great for them to present… to have their posters there and have them on Skype or somehow live streaming, where they can be present or invest in bringing them here or whatever it might be.

John: For the synchronous sessions what are you using as a platform for live streaming the classes? are you using interactive video?

Marela: We were using Panopto. This was purchased by our online programs. So they were using Panopto, which is embedded in Blackboard and then that’s how we’re doing that.

John: We use that here too. We have it in pretty much all of the classrooms and we have a site license for it. It works really well. The only limitation I could see for it in this context, and I’ve experienced the same thing, is it doesn’t work as well for two-way communication. The remote students who are viewing synchronously only have the option of typing in little text messages. They’re not able to interact in real time other than with text.

Marela: That is correct, and we are looking at different ways to fix that. I don’t know if they were able to accomplish that this semester, but you’re absolutely right. Students would have to type in their question. I would have to come back to the computer to check every now and then to see if there are messages there.

John: Because when it pops up, it’s only there for a few seconds, so it’s really hard to see unless you check on the screen itself.

Marela: That’s correct.

John: Just a thought… it might work better if you use something like Google Hangouts or Zoom or something similar for real-time sessions where students who are viewing in real time could actually communicate with voice without having that barrier. We used Panopto for many years here for our workshops with remote participants, and it wasn’t quite the same experience. We switched over a few of the workshops about a year and a half ago, and we switched all of them in the past year to using Zoom and it’s been a much better experience for people who are participating from other cities, countries or just from their offices even.

Rebecca: It’s a better experience for the person doing the presentation or teaching as well, because then you can see what the students are doing. You can see who those other participants are, when they might have questions, if they’re bored, or whatever, just like you can students in the class.

John: Just having those recordings can be useful. We have a lot of ESL students, students who are foreign who sometimes struggle with English, and having the ability to go back and replay parts of the course or look things up while they’re watching it and pausing, or slowing it down to half speed sometimes, is something they find really helpful, and in Panopto you can go back and look to see what portions of the video people were watching, and it’s also could be useful to see “well, maybe I need to explain this a little bit better, next time” if you see that there’s areas where students were going back more often.

Marela: Exactly.

Rebecca: If someone wanted to pursue a flex class what advice would you give them?

Marela: Be flexible.


John: I think that’s good advice with any form of teaching.

Marela: Really… be flexible. Actually, at the beginning of this semester, the Dean asked me to do a bit of a SWOT analysis for the instructors who are teaching it this semester… and I said: “Be ready to have technical difficulties. Be ready to laugh it off and not get caught up in it. Be ready for students to not show up. Be ready for multiple things.” So really, be flexible and allow for it to take its own course, I suppose… and just try to be as accommodating with students as possible. They don’t always understand it. They like it and I just keep going back to this connection with students. Just keep that in mind. It’s worth it. It does take a little bit more work outside of class and you have to be ready to be there a little bit early to get set up and stay after the class is over to make sure it’s all uploaded. Make sure that you give a lot of instructions. Revisit your attendance policy, those types of things, but really be flexible for a flex class.

John: You mentioned that you started with your online course and then you thought carefully about how to restructure it to work in this environment. What were some of the things that you did more of? Some things that you trimmed back? How would you characterize the main changes you made so that things would work better in this environment?

Marela: To be honest with you, I did not alter the online course as much as I thought I would have to. The biggest change was capturing the lectures. The one thing that was challenging was the discussions. The online students were not participating, so they are participating afterwards when the discussion post is due. Face-to-face students may be having a discussion in class. My biggest worry was how do I now capture that and make it fair because they are participating, there is a classroom participation. So, I then additionally was asking my face-to-face students to almost hold on to those discussions. Go back after class… go into the blackboard… and post those, and write about it. So, I had to think about the grading policy actually the most… and make sure that they receive credit for their thoughts and discussions in addition to what they were providing in class. The assignments I altered a little bit more. I wanted him to know that I valued their patience with the group assignment and so I gave more weight to that assignment… and I know some of them really saw a benefit to it. There was a student or two who complained about it, but I said: “Well, think of it this way…. If you are in the real world and you have to work with a facility that’s hundred miles away and you still have to connect and you’re working on this project. There is logic behind my madness, and why I’m asking you to do this.” That was one of the things. But, in terms of the material, the content what was being taught wasn’t anything different. I would say probably the assessment piece was a bit different and then capturing the lectures for students.

Rebecca: I’ve been thinking this whole time as you’ve been talking about what the classroom experience is like versus an online experience and so I was thinking about my own classes…. thinking about: “Well, how would I change a hands-on activity that’s in class so that people could participate in a different time and space, and then make sure that everyone can come back together and see what the results were and share out.” One of the kinds of activities that I do a lot in my web design classes is little code examples where they’re practicing putting code into play… and I guess what I’ve discovered is, over the course of this semester, I started doing things in a more flexible way because I realized that my mix of students was a lot more diverse than I had been in the past. I had students from different majors that I hadn’t had in the past before and to accommodate that I started giving out little exercises… giving some time in class… giving some tips out in a lecture… that I could easily have shared out to online students and then having students finish the exercise for homework, taking the tips into place, and then coming back and going over the example or going over the results the following class period. I think that over time I ended up having to implement something that was flex-like just because my students were a lot more different from one another than they had been in the past… and so, although it wasn’t because they were in different places, I think this strategy might work in other scenarios.

John: …and actually I’ve done a couple of things over the last five or six years too, partly because of the availability of things like Panopto. I teach a class in the fall with generally 360 to 420 students in it, and it’s always offered on Tuesday-Thursday and we have this Thursday Thanksgiving holiday, and a lot of the buses leave late afternoon on Tuesday and often there aren’t a lot of students in class… and a lot of classes on campus end up being canceled effectively or not covering anything substantive on that day… and I never wanted to miss that class. So, what I’ve been doing is in my class we use clicker questions, but now that over the last several years over half of the students use mobile apps for it, they don’t have to physically be there. So, each Tuesday before Thanksgiving when I have class I may have half or two-thirds of my students sometimes even three-quarters of them not physically present but I’ve had up to 150 students who’ve been watching the video stream on Panopto and participating in the clicker quizzes all through the class. They don’t have to miss the class… and I’ve had students who are on vacation… I’ve had students on cruises… I had students participating on that Tuesday class while they were on a family vacation in Florida, for example.

Rebecca: Or on the bus somewhere…

John: …and it’s worked pretty well… and actually, more generally, I have students who are at various sports events in my other classes, where they’re going to be away traveling on a bus or they’re going to be out of town and if they tell me in advance, if it’s not a class that I regularly livestream, I’ll just click the little button in Panopto to set up the live stream, especially in classes we they have clicker options… and they can participate and even though some of them use a physical radio frequency clicker, they have the option of getting two weeks of free use of the app version of it. So, that allows them to participate from wherever they are, and it’s essentially a mini version of the Flex course. Going back to Rebecca’s comment, the other case where I did something very similar, in some ways to yours, is I taught a COIL course which was jointly offered with an instructor in Mexico. My class was an online class, her class was a face-to-face class, but we had some common components where they were working in groups. Most of their work was done asynchronously. But they worked in small groups synchronously with each other and there was a lot of benefits from that… and the students really enjoyed that community, especially the cross-cultural community where they were working with students from another country. There’s a lot of advantage of this modality that makes our offerings more available to a wider range of students who wouldn’t otherwise be as much a part of the college communities. I think it’s great.

Rebecca: I’m appreciating your advice to be more flexible. As you were talking I was thinking like “What else could I do that would be more flexible in general.”

John: That’s really good advice.

Marela: Thank you.

John: How many classes are now being offered? Your’s was the first class… that was in the fall wasn’t it?

Marela: That was in the fall, yes. Right now we have, I believe, two classes from the School of Business and Liberal Arts. One is a finance course, the other one is economics… and we also have classes in the criminal justice and law enforcement leadership from the School of Health and Sciences and they are using the classroom… and interestingly enough I think some instructors are using the technology in the classroom similar to some of the things that you described… whether it’s clicker, whether it’s using it for different group projects and things like that. My class was just Intro to Healthcare Management, it wasn’t anything super exciting.

John: …as opposed to economics, yes. [LAUGHTER] My students might disagree with that.

Marela: As I was going through the semester, I thought to myself that perhaps greater value in this type of delivery lies in courses such as finance and math and economics where students may struggle with the concept. They really need to pay attention and they really need to be tuned in to the videos and watching it and rewinding it and whatever they have to do to get it and to understand the concept. I think, for them, it may be a little bit more of value versus teaching yourself some concepts that may not be as abstract or as hard to understand. I think even toward the end of the semester it was thinking there are so many things that I would do differently… provided that we might alter the technology and have it where students can be interacting and asking questions. I really wanted that classroom interaction to be better, or to foster more of it, rather than just online students watching it later and listening to other students having a discussion… and then I was thinking “well, how can they participate?” So, there are a lot of ideas. Of course, everything in hindsight is different… some of the things that you might do differently and have them build more of a connection, I suppose, between the face-to-face and online students.

John: But, having your group projects, I think, is a good way of doing that because then it does provide those connections for both groups of students.

Marela: Yeah, they really enjoyed that and I think they learned a lot from each other. Online students are professionals, non-traditional students. About 80-85% of them are already working in the field. A majority of the students on campus are first-time freshmen. I think it was a great way for them to bridge, not only the age gap, but also the knowledge and skills type of gap so students here could learn from the online students.

John: …and we’re moving into a world where people will often be working with people locally but also be working with people remotely, and this type of experience is good preparation for those future work skills.

Marela: Absolutely.

Rebecca: I think the more we emphasize that for students the more adaptable they are and the more likely that they are to appreciate the platform or the methods.

Marela: Yes, you almost need their buy-in and because there are other sections of the same course being offered, you have to sell it to them: “What’s in it for you in this class?” and so, if they really appreciate what they can get out of it, they might be more willing to participate and be vested in that class.

John: How are your colleagues responding? Do they generally enjoy this format or is there opposition to it? What’s the general reaction to this modality?

Marela: I think you’ll hear a mix of both.


Some of us are more technologically challenged than others and some of the facultyu members when they hear the technology and when they hear what it’s like and all of the little things that can go wrong, they shy away from it and are probably unwilling to try it. Some are embracing it and saying “This is awesome, I can do all these different things with my students now.” You will have the early adopters and then…

John: Yeah, even those people who are reluctant to try new technology often drive their vehicles. They’re not riding a horse. So people come around eventually.

Marela: Yes.

Rebecca: So, we usually wrap up our interviews by asking: “What’s next?” What’s next for you?

Marela: I would like to do a converged modality or a flex class model in some of my upper-level courses and try to get our other faculty in the program to use this modality… especially, for instance, healthcare finance courses where we use simulation. I think those would be some of the things that I would like to experiment with and try that and see how students respond to it and whether we have a good response.

Rebecca: I can imagine a flex class at a lower level being quite different from a flex class at an upper level so it’ll be interesting to see how your experiments go and how that experience for you and for the students might be a bit different.

Marela: I believe that it would be different, and yes, I just don’t know how until I do it.


Rebecca: Well, thank you so much for joining us today. It was really interesting and definitely got both of our heads buzzing about ideas for our own classes I think.

John: Yes.

Marela: Thank you so much for having me.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

33. The Marmots of Finance

In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, Alex Butler, a Professor of Finance at Rice University, joins us to discuss how rich imagery can be used to help students make connections and deepen their understanding.

Alex received the Rice University Presidential Mentoring Award and the George R. Brown Award for Superior Teaching in 2018. He is also the recipient of the Jones School’s Award for Scholarship Excellence in 2011 and 2012.

Show Notes

  • Duke Talent Identification Program (TIP)
  • Medina, J. (2011). Brain rules: 12 principles for surviving and thriving at work, home, and school. ReadHowYouWant.com.
  • Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185-205). Cambridge, MA: MIT Press.
  • Alex’s web site at Rice


Rebecca: In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, we’ll discuss how rich imagery can be used to help students make connections and deepen their understanding.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guest today is Alex Butler, a professor of finance at Rice University. Alex received the Jones School’s Award for Scholarship Excellence in 2011 and 2012. Alex teaches financial markets and corporate finance in the undergraduate MBA, MBA, and PhD programs. Sometime in the latter part of the last century, Alex and I spent three summers teaching introductory economics to highly gifted middle school and high school students at the Talent Identification Program at Duke University.
Welcome, Alex.

Alex: Oh, thank you. Thank you for having me, John.

Rebecca: Today’s teas are:

Alex: I am not drinking tea…. just water at this point.

Rebecca: Another one… epidemic.

John: My tea is ginger peach white tea.

Rebecca: I’ve Prince of Wales today.

John: Could you tell us a little bit about the courses that you normally teach?

Alex: Sure. I have, over the years, taught almost every course that we have… either here or at other schools. Right now what I teach (and I’m glad to finally have settled into a group of classes that are my classes year in and year out) and those classes are the undergraduate business finance class and PhD courses – one in corporate finance, and one that’s a topics class on causal inference. I really enjoy teaching the undergrad business finance class in particular because I was an undergraduate student here at Rice many, many years ago, and so it’s fun to be back and be on the other side of the podium teaching the students. One of the things that I really like about teaching the undergraduates here is that they are able to appreciate my dorky sense of humor.

Rebecca: …which is indeed a very important thing.

Alex: Agreed. For years, I tried to suppress my dorky sense of humor in the classroom… ad tried to teach the course sort of straight up and dry, and it made everybody miserable… the students…. me… and so, eventually over time, as I grew more and more confident teaching, I started incorporating more and more jokes here and there… and then I allowed them to become more elaborate, and then before you know it, I’ve developed a full-blown dad sense of humor and full-on dork mode.

John: That’s actually why we invited you here. We read a little bit about that in a teaching award you just received. What do students expect the course to be about when they take an introductory finance course?

Alex: At Rice, the students have a good grapevine of information about what courses are about and so, at this point, the students come in with pretty solid expectations that line up with what the class actually is… and whereas some students are hoping that I’m going to teach them how to become millionaires in the stock market, what the course mostly is about is that’s very hard to become a millionaire investing in the stock market and how to make decisions in a corporate setting that will maximize firm value.

Rebecca: So you mentioned that some students come in with this misperception of becoming a millionaire. What are some of the strategies that you use to dispel that myth?

Alex: I should say this class is one that is a fairly standard course that’s taught in lots of different business schools all over the country, all over the world. One way that I do it differently is in the order of material that I cover… and so the very first week of class, I talk about market efficiency. That’s the notion that it’s very difficult to earn abnormal returns in the stock market. In other words, you can’t beat the stock market very easily, unless it just happens to be by luck… and so I come in the first week talking about the reasons why it’s very difficult to beat the stock market… and the reason why is because there are literally tens of thousands of people who have more money and more resources and who are faster and are doing this as a full-time job who are also trying to find the stocks that are mispriced… and so unless you are the investor who is faster, and smarter, and has more money to throw at the trading strategy that you think of, it’s very very difficult to beat the other 10,000 people… and so I spend the first week of class introducing this concept and then providing copious amounts of evidence… research that highlights how difficult it is for lay people to earn abnormal returns in the stock market. That sets the setting for the rest of the course, which is this notion of how competition affects prices and how that feeds through to other applications and the corporate domain as opposed to the financial markets domain.

Rebecca: What made you switch the order?

Alex: What I realized after teaching this class for a long time is that most of corporate finance, most of business finance, is about discounting cash flows back to the present at some appropriate discount rate, and I found I was having a hard time getting the students to understand the notion of what interest rate, what discount rate, should be used to make these cash flows that are spread through time to get a present value equivalent. I figured if I started with some aspect of that, where that rate comes from, where those prices come from. but that would make the rest of the course easier for them to understand as we go through. So, that’s why. Most people wait until after they’ve introduced things like: “oh, portfolio theory” and “capital asset pricing model” and other asset pricing concepts before they talk about market efficiency… and so I just sort of turned that around backwards… and I open the course with that. So, point number one is pedagogical, and point number two is that the lectures that I do on market efficiency are really fun, and so I really like starting the course off with something that’s really fun. So, we can talk about stories of insider trading, and we can talk about stories of surprise announcements and how that affects stock returns, and we can talk about “oh, so you think you’re going to beat the market, well let me explain to you how hard it is and the reasons why …” also were very fun. So, we can spend the first week talking about fun stuff. It’s a giant bait and switch.


I lure them in with fun stuff and then beat them to death for the next fourteen weeks with discounted cash flow analysis.

Rebecca: It also sounds like it’s a good way to motivate students. Not only is it fun, but it gets students motivated and interested and they buy into the class, which I wouldn’t discount that. I think that’s an important task.

Alex: Oh, absolutely.

John: …discounting in a different sense, but…


Rebecca: Yeah, sorry… I’m a designer

John: …but it also starts a class by dispelling that myth that they’re going to learn tools that will allow them to become really wealthy in financial investments and so forth. So, you’re setting it up by getting rid of that myth and they’re ready to start actually learning without having that at the background.

Alex: Correct.

Rebecca: I really like the idea of just meeting students where they’re at. If this is where some students are coming with, and maybe they’re super motivated in some ways but not in others, that you just tackle both of those in one week.

Alex: That’s right.

Rebecca: Cool.
You recently won a 2018 George R. Brown Award for superior teaching, and the Georgia R. Brown Awards are based on an interesting selection process. Can you tell us a little bit about that?

Alex: Yeah, so what the university does is they solicit feedback from recent alumni, people who graduated, I think, it’s two and five years ago. Now, they have graduated and they’re looking back what teachers would they want to see when these awards… and so, it’s really a neat honor, because once the students are two, three, four. five years out, they’re not responding to the short-term incentives that some professors used to gain teaching evaluations and things like that, they’re looking back and they’re actually remembering what the course was, what the professor was like, and whether it was meaningful to them. It’s really been just a phenomenal honor, and one that I honestly thought that I would never win. So, it’s been really special to have gotten that award.

John: It’s a really interesting idea to have an award given in that way, because it focuses on that long-term learning, that you’ve taught them something that’s going to benefit them later rather than, as you said, something that people do short term. I know some faculty, when we’re doing course evaluations, will give out cookies or other things just to boost their short-term course evaluations. But, that’s not going to pay off very much two to five years after graduation.

Alex: That’s correct. I went through the cookies and brownies phase myself years ago and decided that that just wasn’t who I wanted to be, so…. Now, we’re on to just just the pedagogy at this point.

Rebecca: I think there’s a couple of interesting things that I see about those awards, too, and one is that is more meaningful probably to win it because you can feel the impact but two it meshes really well with what we know about evidence-based practices in that students generally don’t like them while they’re happening, but they have longer-term effects… and that students tend to learn that material and transfer that material later on.

Alex: That’s absolutely correct, and I’m a big subscriber to that basic view that students often don’t like things that make them uncomfortable, and learning new and difficult things is uncomfortable. I think one could make students happier in the short run by giving them lots of little assignments where they feel like they’re making progress every day but they’re not actually being challenged, they’re not actually being pushed, and so instead holding them to very high standards for demonstrating their mastery of material while that is very uncomfortable for some of them, it is that that makes them better students and better scholars of that topic.

John: It reminds me of Bjork and Bjork, in their writings on “desirable difficulties.” that the most learning occurs when students are faced with feasible challenges… that if things are too easy, they get bored, and they may be happy with the course if they don’t have to struggle much… but they learn the most when they’re struggling but they see it’s possible.

We’ve heard that this award, though, based on some of the write-ups we’ve seen at your institution, may somehow be tied to marmots, wolves, and The Princess Bride. What do they all have to do with finance?

Alex: Several years ago, I was reading some books I thought would be…. some I thought it’d be helpful for my teaching, some that I just thought would be helpful for me, and one of the books was a book called Brain Rules by John Medina, and it’s basically a book that tries to take cognitive science, brain science, down to a level that lay people can understand and gives several rules of thumb of how the brain works and why the brain works the way it does… and as I’m reading this, reading it mostly for my own consumption so that I can be a better researcher and more thoughtful person, smarter, that sort of thing, I realized “Gosh, a lot of the rules here applied very directly to teaching, at least in the lecture format that I use in most of my courses…” and so one of the things that really stood out to me is how people learn better, remember better, I guess I should say, when they see images images that relate to whatever the topic at hand is. So, text maybe a PowerPoint slide with text, and you remember X percent but if you see an image, you remember much more of that material later on… and so this gave me just a license to, all of a sudden, start having fun on a completely new dimension. Reading this book and sort of embracing the notion that I could maybe help students remember the material better simply by infusing my lecture slides with some relevant images, was just eye opening for me… because now I could take my completely dry, boring slides with words and numbers and equations and now I can have fun with them…. and have this entire new dimension, a degree of freedom, to play around with what the slides are gonna look like and how the students are going to experience them. So, that’s sort of the extensive margin. The first part is: “Hey, I need images…” so the intrinsic margin is “what kind of images do I need?” “what will work best?” and Brain Rules comes to the rescue there again… and it says people respond to images that are faces, that are things that are scary, things that they can eat, things that might want to eat them, and things that they might want to mate with. Well, that last one’s kind of out for most of my lecture slides… [LAUGHTER] I can’t really incorporate that very directly. But it got me thinking “okay, what’s scary?” So, I started looking around for images of things that are scary to people… and I found this great image of this really, just terrifying, snarling wolf. I’m like “Ah, I’m gonna use that to get people’s attention…” But, as I started thinking about it, I wanted to lead into the wolf a little bit… and so the main prey of wolves in North America are marmots… yellow-bellied marmots… and so I found this great image of a little cute yellow-bellied marmot sitting on a rock somewhere in the mountain somewhere… and now when I come in to teach what could be the absolute driest lecture of the entire course which is time value of money and understanding how to discount cash flows (it’s the tool that everything else builds on, so it’s incredibly important to get it right, but it’s also potentially incredibly technical and boring)… and so I start that lecture not with an equation not with numbers but with a giant image of a marmot filling the entire computer’s projection screen…. and I just leave it up there… and the class is all looking at it… and I look back at them… and I pick someone at random. I cold call… and I say: “So, Charlotte, what do you think?”

“What do you mean what do I think?”

“What is it?”

… and we go through a series of guesses, and the guesses range from just ludicrous things: “it’s a gopher.”

“No, no, clearly it’s not a gopher. Gophers are a lowland creature. This is obviously in the mountains.”

“It’s an otter.”

“No, no, no. Otters prefer marine habitats, and this is clearly not there.”

… and go through this for a while until usually somebody recognizes it as a marmot. I say “Yes, very good. Alright..”

…and so then on to the next slide and the next slide is this picture of George Soros with no caption, no explanation, but again filling the entire screen, here’s this giant picture of George Soros… and so I go back to the first person: “Charlotte what do you think about this one? You didn’t get the marmot, how about this one?” …and so invariably somebody will eventually guess it’s a hedge fund manager.

I say: “Yes, very good. That’s George Soros, a famous hedge fund manager.” then the next slide is the wolf, the snarling wolf, and so at this point the captions on the slides read “This is a yellow-bellied marmot,” “This is a hedge fund manager” … and then it’s obvious what the wolf is. People get that right away. So, Charlotte gets to redeem herself at that point.


“This is a wolf, one of the main predators of the yellow-bellied marmot,” and then the next slide is another picture of a marmot but this one looking somewhat quizzical and the caption here is: “Why do wolves eat marmots, but not hedge fund managers?” …and the answer that I propose is because hedge fund managers understand the time value of money, but marmots do not. Now, that obviously doesn’t actually follow, but the correlation is there… that part is true… and so I just sort of lean into that and the ridiculousness behind that statement and from there on the marmot becomes our time value of money mascot… and so every time throughout the course I introduced a new application of time value of money then I bring in another image of a marmot to sort of tie all that together. iIt’s hard to see that the tie that binds everything together is time value of money and so the marmot, the mascot, is the visual cue, that “oh, this isn’t special, this isn’t different, this is just another application of the same technique we’ve been doing over and over again.

John: That notion of using visual cues goes back to the Greeks who use it to remember long stories before there was much printed word, and one of the arguments is that it’s because visual imagery developed much earlier than the use of language and the things you described in terms of things that might eat you may be tied back to our evolutionary adaptation, and so we’re tying into things that evolve fairly early in the evolution of human beings.

Rebecca: …and it also is just that emotional response.

John: Right, when you trigger an emotional response and fear is a major one…

Rebecca: Yeah. mm-hmm

John: Do the wolf’s chase the Princess Bride? I don’t remember that…

Alex: Well no…. So, the Princess Bride lecture is where I talk about decision rules using discounted cash flow techniques…. and so this sort of standard playbook for any finance professor anywhere as you talk about net present value and internal rate of return and payback period and maybe a few other rules as well, you just sort of go through the… whatever they have in the book in the order in the book, and you talk about each one in turn. But, it’s a lot more fun if you can personify those.. and so that’s what I use the characters from Princess Bride to do… and so the main character, which is net present value, is personified as Wesley / (there’s a spoiler alert here – so you haven’t seen the movie Wesley is the Dread Pirate Roberts). I always show my kids my lecture slides and when I showed them that one my older daughter was mortified that I would ruin the Princess Bride movie for students by revealing that Wesley and the Dread Pirate Roberts are one and the same… and so from there, I personify the different rules based on the different characters in Princess Bride… and so internal rate of return which is sort of flashy and useful but has some flaws as Inigo Montoya; the payback period which is sort of a blunt instrument is Fezzak the Giant, and so forth… modified internal rate of return is the Cliffs of Insanity.


Rebecca: So, the important thing to ask then is do you have the spoiler alert at the beginning of your lecture.
AEX: I do now.


John: How old was your daughter at the time when she reacted to that?

Alex: About ten years old.

John: Okay. So, by the time students are in college, they’ve probably either seen it or they may be less likely to, so it probably doesn’t do quite as much damage.

Alex: I hope so, and in fact I even now encourage them to watch the movie before that lecture without really telling them why. So some of them do. Some of them ask around and figure out why, but that makes it more fun.

Rebecca: That was gonna be my follow-up… is that homework?

Alex: It’s the best homework they’ll ever have.

John: So, how have students responded to this? Do they remember this later?

Alex: Yes and no. for the Princess Bride lecture I believe that the students sort of uniformly enjoy that…. but I don’t have a good sense of how much that actually impacts the depth of their learning. For the marmots, it’s a completely mixed bag… some of the students really love it, and I really do lean hard on the fact that this is the mascot… this is the thing that ties it all together… and this is the visual cue so you will now recognize that when we do bond pricing “oh, here’s a marmot, that means it’s just time value of money…” “oh, when we do net present value, here’s a marmot showing up alongside our Princess Bride character it’s just time value of money” and so forth. Many of the students really enjoy that and grab on to it and some students, they’re not having any of it. They want boring… they want dry… and I’m afraid they’ve come to the wrong place. Been there and done that. I don’t do that anymore.

John: But that use of imagery is really common. People who work on developing memory… the memory palace type things where you tie specific concepts and bundles of concepts with chunks of item to key images, helps people remember things long term. I can see how it would be really effective.

Rebecca: I also could see that one of the things that students often struggle with when information is new to them is making those connections. So providing that visual cue like “here’s time to make a connection…” it’s actually really helpful, because those are the kinds of things that might seem really abstract and very separate if you’re not making an explicit. So, I like that you don’t even have to say explicitly like “this is the thing” you have by putting the image up there, and you’re prompting the students to predict what that connection is or challenging them to think of it on their own before you reveal what that connection is.

Alex: That’s right, and so what started off as just a fun way to get attention: “Hey, class is starting let’s all key in… Here’s a scary wolf.” So, now we’re all paying attention to the scary wolf because that’s how our brains are hard-wired. It’s now grown into this entire thing where throughout the entire class is a continuing callback.

Rebecca: It sounds to me like you must have some pressure every semester to have to have something new that you introduce into some sort of lecture so that there’s some anticipation.

Alex: I’ll be the first one to say that I shamelessly recycle all of my jokes…


… and so I’m always sort of terrified when I have a student who started the class last year but had to drop at the 6 week mark or the 8 week mark because whatever was going on their life, and then they’re back the next year. Like “oh man, you’re gonna get the exact same jokes with the exact same timing, the exact same patter.” It mostly seems to work fine, and I do continually try to incorporate new things. One of the nice things about teaching finance, although some of it is very static… the basic concept of time value of money is going to be there for basically forever and I won’t need to change the actual examples in those slides really ever… but a lot of the other material changes very rapidly. So, the notion of what does market efficiency mean? who can beat the market and win? what is the evidence? But for topics like market efficiency, for topics like financial markets, for topics like investment banking, those areas transform rapidly… and so I’m continually changing those lectures year after year because one of the big topics that I cover in my course that is not really traditional for an undergrad business finance class is how firms raise external capital. The reason why is because it’s one of my main research interests, and so I have lots of ideas of what I want the students to know, and lots of research that I can tie into the lectures that I give. For the past five or ten years financial markets, the regulation of financial markets, firms’ ability to raise external capital, has changed tremendously as regulation has changed and so I’m continually revamping that portion of the course basically every time that I’m breaking it out.

Rebecca: One thing that I can’t help but think is that you and John have some similar backgrounds in terms of content, so the time value of money that you keep referencing probably makes sense to you but I feel like our listeners might not know exactly what that is so maybe we should just take a minute and give a quick cap of that so people know.

Alex: Yeah, absolutely. So, if you have $100 and you invest it earning an interest rate of 5% per year, in a year how much money will you have?

Rebecca: One hundred and five?

Alex: One hundred and five dollars.


Rebecca: It’s not a trick question, right?

Alex: No, No, it’s not…

… and so, as you invest money it earns some rate of return and so then money in the future you have more, because it earns some positive rate of return, and conversely, would you rather have $100 today or $100 in a year? Well, the answer is $100 today, because you could invest that for that year and have the hundred dollars plus some additional return. So, you’d have a hundred and five dollars one year from now instead of the: “oh, I’m going to give you a hundred dollars a year from now” and that’s all the time value of money is.. and then it’s application of that over multiple cash flows and multiple periods where people start to lose track that it’s just math.

John: That’s the fun part.

Rebecca: Yeah. Yeah.

John: I also ask my students have they ever burned their mouth on a slice of pizza, and why don’t they just wait? ..and it’s one way of introducing the notion that we prefer things now to later. It’s a fairly important concept in economics and finance and it’s at the basis of finance.

Alex: Oh, absolutely… and I like the way of framing it as how patient you are. Are you willing to wait or are you impatient? …and so the way that I sometimes describe that in class is if you have a low discount rate that means that the future is worth about the same as the present and so you are patient. If you have a very high discount rate well then the future and the present are very different, so high discount rates → impatient, low discount rates → patient.

Rebecca: You realize that the irony of this whole thing is that I have these conversations with my husband all the time because he’s in finance, right? Yeah….


John: Ok, well, does that help explain it?

Rebecca: Yeah, yeah. I’ve had it explained to me many times.


I figured our listeners would need it.

Alex: Well, since we’re talking about the things that our students do, I’ll just share one example that I use in class that I continue to use even though it completely does not work with undergraduate students. It’s when I’m trying to introduce the concept of sunk costs… and so a sunk cost is some amount that you have paid… a cash flow that has happened in the past perhaps… and so once that is paid, you can’t get it back …and so it’s like the notion of “should he throw good money after bad” is another way of phrasing it… and so, what I used to say is, “Well, you go to a movie, you pay your fee to go in, and you decide it’s a terrible movie, should you keep watching to get your money’s worth or should you leave? and then a student pointed out to me one year that well, actually, if you go and you complain you can get your money back from the management.


Okay, different example. This is the one that never works on the undergrads. You go to an all-you-can-eat buffet and you have a choice of do you buy one plate for this amount or do you pay a little more to get the all-you-can-eat buffet and you decide to go for the all-you-can-eat. You pay the $10 for the all-you-can-eat instead of the $7 for the single plate… and you go and you fill up your plate and you eat… but you decide you’re full after just your first plate… but you’ve already paid that all-you-can-eat price. Should you go back for more? …and the intellectual scholarly answer is “Well, no, because the marginal benefit of eating more food is negative at this point because you’re full. Even though the marginal cost is zero.” But, for the students, the undergraduates, the marginal benefit of eating more food is always positive.


They view it as a sport. Dinner’s not over until they kick me out.

Rebecca: I think that’s really important to consider your audience and what works. So, I think that’s a really good demonstration of knowing your audience and why something might not work for a particular audience.

Alex: Yes, nonetheless, I still use that example… but I tell them ahead of time “I know this is not going to make sense to you. You’re going to push back on me. Nonetheless, I want to talk it through with you so that we can think about who has marginal benefits and marginal costs of what amount.”

John: Just last semester I used exactly the same thing of asking them how many of them would leave a movie theater if it was clear that they weren’t enjoying the movie and this time I had some people argue “I do that all the time.” So, yeah…

Rebecca: Movie theaters? Don’t you just have Netflix?


John: Okay, that’s actually a relevant point. In terms of the Princess Bride, we’re moving into a generation where many of your students might not have seen that. Do you have any other movie references that you might substitute in the future?

Alex: You know, I haven’t found the right one yet and, honestly, even if I found a good substitute, I’d really want to keep the Princess Bride, just to give them the incentive to watch that movie.


It is such a classic and such a gem. I really would like for them to all see that movie.

Rebecca: It’s an investment you’ve made.


Alex: That’s right.

Rebecca: How have faculty responded to the methods that you’ve been using?

Alex: Faculty are not surprised that I do the things that I do.


John: They’ve known you for a while.

Alex: They’ve known me for a while and I really lean into the corny pun bad jokes big time in class…. and I really I’m quite shameless about it.

John: That’s not new… I seem to remember that back at Duke when you were still in college.

Alex: …and it’s just a question of do I try to suppress that innate desire or not?


…and I’m at the stage in my career where “nope, not anymore.” You’re just gonna let it all out. My colleagues are not really surprised, but interestingly, one of my colleagues has adapted her lectures quite a bit to embracing this notion of adding images into the slide decks… and that’s my wife. She teaches business law and regulation of business. She was an attorney by training and I described her as a born-again economist because when she started teaching regulation of business she had to teach herself basically all of public choice economics. So, she got a couple of high-level textbooks and worked through them all but she and I regularly discuss teaching techniques and so she now has gone down the rabbit hole of finding that perfect image to highlight the point that you want to make… to have that really stretched metaphor that you can then call back throughout your course. So, it’s been a lot of fun to have her as a sounding board to go back and forth with.

Rebecca: I appreciate that you’ve brought design into the process.

Alex: Yeah.

John: Excellent. It’s something we all should probably do more of and think about more and certainly much more effective than those PowerPoint bulleted lists that are so common.

Alex: Yeah, a little bit of both actually goes a long way.

Rebecca: You’ve won some awards for your research. How do you maintain a balance between teaching and research?

Alex: It’s difficult. There are some ways in which research can feed directly into the teaching… and so my research that touches on market efficiency, some of that can come into the classroom. My research that touches on how firms raise external capital, some of that can come into the classroom. Now when I teach my PhD courses, those are heavily flavored by my research interests and preferences, but when I teach the undergraduate core finance class there’s not a whole lot that can flow back from that teaching into the research. So, that’s one of the downsides of teaching that class as opposed to perhaps a class that is more specialized or an elective or something that’s a little bit further downstream from the core class.

Rebecca: But at the same time, if it’s something that you teach routinely then the prep isn’t as difficult.

Alex: Absolutely. That’s true. It is hard to overstate how useful it is to teach a class a second time, or a third, or a fourth, or twentieth. Nonetheless, every single year, even though it’s the same class and mostly the same topics, I go through every slide, every lecture from the beginning every year.

Rebecca: But, I remember teaching a bigger selection of classes and one of the things that I’ve liked about my position at Oswego is that that suite of classes has gotten smaller… and then there’s a little less I’ll keep on top of to make sure that you have all that fresh information and what-have-you for classes.

Alex: Absolutely, and I’ve done the same thing. I’ve taught a variety of courses over the years and it’s been nice… as you describe it that suite of classes narrows, so that you have the same core group of classes that you’re teaching over and over again and you can start to specialize. You can really invest the time to get over the fixed costs of finding all those right images for the slides… to going through and taking time to invest in the design aspects of the lectures. That if you had four different courses every term, it would be incredibly difficult to find the time to do that effectively.

John: While you’re working with PhD students, you must do quite a bit of mentoring of them. How do you see the role of a faculty member as a mentor for graduate and undergraduate students?

Alex: Faculty vary widely on their views of how much mentoring PhD students should have… and so you have one model where it’s sink or swim… the PhD students are some of the smartest people in the world… they’re good students… they’ll figure stuff out… just point them in the right direction and let them go, and they’ll get there. That’s not the view that I subscribe to. Because I think we frequently overestimate just how much the doctoral students know, particularly about how the profession works. There’s no book for that. They can’t just go down to the library and find a textbook on how to be a good assistant professor or… there are books on how to write a dissertation but that only gets you so far, because it really needs to be very field specific. So, I tend to go very much the other way, which is a lot of sort of high-touch mentoring. I write co-authored papers with many PhD students, one of whom is now your colleague there at Oswego in the Finance Department in the School of Business, and it’s enjoyable for me. It is a good learning experience for the students and I think it helps them to learn how the profession works much more efficiently, because when it comes time to write a paper and they might put together some tables and say I want to structure the introduction this way. Oh, no, no, no, no, we can’t do that… because if you do that then it’ll make people be concerned about this issue here. So, instead, we need to twist it around this way and start with this… start with the big picture, not what your paper does but what your paper’s about. That sort of thing. That’s hard to learn on your own.
PhD students are PhD students because they are extraordinarily good students and they’re really good at learning. Though, that’s not the job for academics. The job is not the learning, the job is creating knowledge… and the transition from being a consumer to a producer of knowledge is scary, and it is the road that has very few signs or roadmaps to help them get down. It’s a transition of going from a consumer to a producer of knowledge… is very profound for a lot of people.

Rebecca: What’s interesting about what you’re hearing is you know my field the terminal degrees in an MFA a Master of Fine Arts, and the undergraduate degrees are really professional degrees. But, it’s the creators of cultural content ,and so that struggle happens at the undergraduate level too, of going from being that consumer of culture to a producer of culture. It’s really not that much different… just what they’re creating is a bit different.

Alex: That’s right.

John: I remember when I was working on my PhD, one time, where up until that point I had been meeting with my advisor every month to talk about my research, and at first I was just asking him questions… and then he was asking me questions and I realized suddenly that I knew more about the topic than he did… and that’s I think that sort of transition that’s sometimes difficult… because when you’re working on your research you’re mostly going out and finding all these earlier studies and so forth but you get to some point where suddenly you become the expert in the field and that’s a tough transition to make. It’s scary, as you said.

Alex: It is. It’s quite the watershed moment when you realize… when you are presenting your research to a room of 30 presumably learned scholars that maybe collectively they know more about the topic than you, but you know more than any individual person in that room… and becoming that expert and then owning it, so that you can write confidently is, I agree, a very tough transition

Rebecca: I like the emphasis on the owning it part. I think that’s key.

Alex: Yeah, imposter syndrome is real.

John: Everywhere.

Rebecca: So, we usually wrap up by asking “what are you gonna do next?”

Alex: Well, that’s a good question. I really wanna keep doing what I’m doing, making my class better year on year, teaching PhD students and training them year on year and working on research, hopefully research that people will actually find interesting and useful.
I have a sabbatical coming up.

Rebecca: When’s your sabbatical? What’s the countdown?

Alex: Well, the sabbatical is a year from now and in between then, one of my colleagues with whom I co-teach a doctoral seminar, he’s on his sabbatical. So that PhD course, that normally I teach half of, I’m now teaching the entirety of next fall. So, I think the first thing I’m gonna do is prep the rest of that class.

John: Excellent.

Rebecca: Deadlines make a difference.

Alex: Yes, indeed.

John: That procrastination thing… and that time preference…

Well, thank you!

Rebecca: Thank you so much.

Alex: Thanks.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

32. The Three Little Pigs

What do the three little pigs, the big bad wolf, and dragons have to do with web design? More than you would think. Rebecca Mushtare discusses how a trip through fairy tales may open up the opportunity to develop empathy skills and conversations about race, disability and identity.

Allison Rank joins us again this week, this time as a guest host.

Show Notes


John: What do the three little pigs, the big bad wolf, and dragons have to do with web design? More than you would think. In this episode, we’ll explore how a trip through fairy tales opens up the opportunity to develop empathy skills in conversations about race, disability, and identity.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer. Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Allison Rank, a frequent guest on this podcast, joins us today as guest host. Our guest today is Rebecca Mushtare who, until this episode, had been the co-host of this podcast.

Allison: Nobody panic. She’ll be back in this chair next week.

John: Today our teas are:

Allison: English breakfast tea.

Rebecca: What?!?

Allison: …under duress. I’m highly under caffeinated.

Rebecca: I’m drinking my normal English afternoon tea.

John: …and I’m drinking pineapple ginger green tea.
We invited you here today… because you’re always here… but we’re asking you…

Rebecca: …it’s a matter of convenience….


John: A year ago your daughter was born… now the three little pigs have invaded your class. Could you tell us a little bit about how the three little pigs made their way into your web design class?

Rebecca: I’ve been looking for ways to help students develop more empathy for their audiences, and it’s been a struggle. Students (or anybody who’s new to anything) will immediately try to make things for themselves, because it’s the audience they know best. So, it’s the easiest way. If you’re working on technical things or other concepts you don’t have to worry about audience too, because you have that part figured out. But, I’ve been really wanting to challenge students to dive into audience and also deal with accessibility issues which doesn’t come intuitively to them. So, the three little pigs actually offers a really great opportunity to have different audience members to think about (and audience members that don’t really exist); it becomes a safe zone. In this scenario, I’m using three titles as ethnographies for the students to read to get to know their audience better. I spent some time reading about ten different versions of the “three little pigs” and I’ve identified the best three. They are: The True Story of the Three Little Pigs by John Scieszka and Lane Smith, and The Three Pigs by David Wiesner, and There’s a Wolf at the Door: Five Classic Tales Retold by Z.B. Alley and R.W. Alley.
They read those books and then we come into class and I ask them to help me understand who all the characters are, what’s important to them, and some of their characteristics or qualities that we need to think about in terms of design… and then (from the perspective of the characters) what’s going on in the community that they live in… and the frame that I’m giving my students is that they’re in this community called Dragon Town. Dragon Town has a mayor named Mayor Melanie McDonald, and she’s human, but there are talking animals and dragons and other creatures that live in this community together and there’s a clear creature divide going on. So, the humans seem to value themselves more than the other critters in town. The poor pigs, they’ve got houses that are falling down. They don’t even up stand the Wolf’s breath. So, we’ve got some issues going on here.
The students read the stories, came to class, brainstormed about these characters, and helped identify some really big issues that were happening in Dragon Town… and then my challenge to them was, in teams of three or four, to identify one of those 10 that we identified as a class…choose one that they were gonna use a web design to help raise awareness of or to start to tackle. Obviously they’re not gonna solve these big problems, but they could make a dent into it.

John: The purpose then is to have students look at a problem from another perspective, from the perspective of the intended audience of the webpage, rather than using their own biases.

Rebecca: Yeah, exactly, and it’s something that they really need to practice… and so, yeah, this is a good opportunity to do that. They’re characters that their familiar with, but the books actually challenged a lot of their initial remembrances of some of the stories. So, it’s a nice way to get them to revisit that in a different way.

Allison: How was this different than how you’ve tried to approach the same topic in earlier iterations of the class?

Rebecca: In a previous episode, I think I talked about my simulated client project where I had these big company scenarios with the audience members being Oswego (the community that we live in) and they worked okay… but the students had trouble aligning themselves with older adults or middle-aged individuals who they just don’t seem to find relevant to themselves and even though these are individuals that are readily available in our community that you could interview and get to know, it was a struggle. We did a project in the fall, “The Voices of Oswego Veterans” project that we had a guest (Stephanie Pritchard) on who talked about that project… and we did a web project with that as well… and that was another way to deal with the audience. This time the audience was members of the Oswego community (the SUNY Oswego community), so they had a little bit easier access to that community… but the community that they were representing was different from themselves. These were students, so the population that they were addressing or talking about was student veterans, which was an identity that nobody in the class happened to identify with. That got us closest to solving the problem… but it wasn’t quite where I wanted them to be yet. What’s nice about this is that you don’t have to worry about offending anybody, because they’re not real.

Allison: I can imagine how the fictional characters are really helpful in terms of giving students a lot of space to play and a lot of leverage, but I have to imagine that there are some real challenges associated with giving them that amount of space as well. I guess I sort of have a gut reaction that thinks that they will make up things that cause problems in and of themselves. They’ve got enough rope to get in some dangerous positions. What are some of the challenges that you faced?

Rebecca: That’s a really good question. What I found was, they were willing to talk about things that they were never willing to talk about before. That, first of all, was a good space to be in. That was things like: “oh, there’s species profiling going on…,” “oh, there’s accessibility issues because pigs have hooves so they can’t type and tap on the computer screen…” …the accessibility issues that just bubble up. There was also the concern that critters were eating other neighbors, so we needed to start a campaign to be vegetarian, for example. So, there’s a lot of different things that came up…. a lot of social issues… another one was stranger danger… and then they did these presentations to the mayor, and it was important because we brought someone from outside in and I think that helped prevent some of the issues that you were identifying could bubble up as being a problem, but there was someone that wasn’t me who was the audience but I didn’t tell him who it was gonna be (it was just a grad student I bribed) who came in and just sat and played the part and asked questions and what have you…. and they were taking notes and then we went away and had a meeting and I came back with notes to the students about what the client was concerned about. So, that helped resolve some issues. But, you know, in the presentations there were some crazy things that happened… like the one on stranger danger, for example, the students had still indicated that the stranger, the bad character, was the wolf and the whole point was that all of the animals, and all of the creatures, and all of the humans, also have children and they all need to be concerned about strangers. That we shouldn’t associate one population as the bad actor. We ended up having to have a conversation about that. You can’t perpetuate these stereotypes, but what happened was we could have that conversation safely.

Allison: The familiarity played in the same way that a stereotype would traditionally function in class, but in a much safer space to have the conversation that resolves it.

Rebecca: Exactly. We were having crazy conversations about racial bias, and all these sorts of things, but under this guise of “it’s about the species” and the species problem that’s going on. And now all of a sudden it became safe. When that one group was having issues getting their head around it, I said to them: “You realize that this is the exact same thing as racial bias, right?” and they just looked at me with deer-in-the-headlights look. The next time they came back, the whole project was fixed.

Allison: That was actually gonna be my next question. At what point did you pull out from playing in the sort of allegorical space to say: “Hey, here’s what we actually just did” or did you let the experience and the skill building stand on its own?

Rebecca: I let things unfold organically, and I prodded and probed as necessary. I didn’t want any projects to perpetuate stereotypes or to perpetuate lack of accessibility… those two key issues. I probed and invaded their team time a lot with those particular things to push them on that, but you know they’re not perfect. But, I think they did a lot more growing in that area than they would have otherwise. What I think is missing, that I want to do next time is allow for more of that reflection at the end, so that they could apply it to some other projects. What I’m thinking about doing is have them present the work as if they were in an interview, and so how would you explain this project and what you learned from this project to a potential employer who has no idea what Dragon Town is, so that it becomes something that’s valid and useful… and I think that’s going to take some effort on their part to make that leap. But I think it’s actually a really good project for them to talk about in an interview and most employers would see the value in that.
I already have them do portfolio documentation. I already have them thinking about that, but I need to coach them through that process a little bit more…. and maybe actually make them present that.

John: Yeah, I could see an employer looking at a webpage making a case on avoiding inter-species consumption and being perhaps a little bit puzzled….

Rebecca: The tagline was “don’t eat your neighbor.”

John: Yes.

Rebecca:… which I thought was right on.

Allison: Yeah.

Rebecca: Well, and that group actually was interesting too because they wanted to do something that was: “Don’t eat your neighbor.” They wanted to be vegetarian but I was like, “Well, dragons have a big appetite. What are you gonna do for them?” So they came up with this tree salad or whatever that has just bigger things. They had to adapt the recipes and things like don’t forget there’s small kids. You got to think about these different populations, and they adjusted their content accordingly, to rise to that occasion. I also found this really great article about whether or not pigs are colorblind that I used as a doorway into thinking about accessibility issues. Apparently, I learned, pigs don’t perceive color the same way that humans do. They can’t perceive as many colors, so we have to really be concerned about the spectrum of colors and the kind of contrast that colors have… so that they would be accessible to pigs,,, but that led into conversations about maybe the pigs have to use voice activation because their hooves won’t let them type on their devices… and then we also had to talk about a mobile device for a dragon is pretty large…. so we had certainly some fun playful conversations, but they were really meaningful. We started talking about those issues pretty deeply in a way that I’ve never had in my class before.

John: Were the students more open to addressing these issues when it was in this safe zone or this safe space?

Rebecca: Yeah, even when I called that one group out on being stereotypical and perpetuating bias, they just received… and were like: “Oh, okay” and then you try it again… “is this better?” “My god, could you push it a little bit more?” and gave them some ideas about how they could push it… and our first solution wasn’t great after that…. It was to put in a separate monster that didn’t exist in this world as being the stranger, and then I identified that like when someone the other, we shouldn’t just assume that they’re the bad person or the bad creature. We had to be careful. I tried to call them out on whether or not we were using the word person, because it didn’t apply to dragons. So, it was funny [in] their presentations they were really conscious about things like that and trying to be inclusive in their language. So, yeah we ended up trying to tackle some of those things, and I was pretty impressed with how far they got… but it took some pushing. That one group took four or five tries before they had something that was gonna work.

John: How did students respond when you first gave them the assignment?

Rebecca: Well, I should probably provide a little setup in that my class includes design students, marketing students, and graduate students in HCI. So, it’s a fairly diverse population in and of itself in terms of disciplinary background. So there’s that. There are a number of people in the class who may not be traditionally artsy or creative, so it’s a little risky, right? I think I’m also known for being very serious. Which if you know me personally, that might not be true, but in the classroom students perceive me as being very serious… and the semester just was not going great, to be honest. It’s like something’s got to give, the students were struggling with a lot of the technical things, and so I basically threw the syllabus out or revised it significantly. stopped and did just technical exercises so students get comfortable with some of the things that they were really struggling with… and then one day I just showed up and said this is what we’re doing… and they had a ton of fun…. and were shocked… they’re just like “Is she serious? She lost it?” There was definitely those looks, but then there was a couple of key students who just jumped in and ran with it… and I think that really helped. So, I’m hoping that that will happen again. I think if everyone in the class is a little too serious, I don’t know that it would work.

Allison: Would you plan on sticking with, in the future, the three little pigs as sort of the through line story or it sounds like the story with the five different ways that the wolf is at your door? Does that give you some entree into some other storytelling avenues?

Rebecca: There is some entree into some other avenues and I maybe need to read some more fairy tales to be up on that, but the reason why I stuck with the Three Little Pigs is actually the wolf is the character that carries through all of them. So, that the five stories that are connected are all based on the wolf and different stories. So there’s Little Red Riding Hood, the Boy who Called Wolf, those are some of the stories in that other one. So, maybe there’d be some versions? I also happen to know that there was like the version of the Three Little Pigs told from the wolf’s point of view, so I really like that because it’s in direct conflict with the Three Little Pigs version of the story. I liked that the ethnographies that they were collecting were realistic in that they conflicted with one another, that they had to deal with the fact that there was conflicting information, and that they had to resolve that or deal with the fact that a wolf’s perspective was different than the pigs perspective of what the wolves perspective was… and I think that was a healthy messiness about it that worked pretty well… and the particular version of the Three Little Pigs that I used pigs escaped getting eaten by the wolf because they jump out of the storybook. So, there’s some plot twists in there that the students wouldn’t necessarily expect. It’s not a traditional version of the story… plus, they all have really great illustrations and they’re beautifully designed.

Allison: Are there other classes where you’d be interested in trying the same type of fictional ethnography technique?

Rebecca: I think it could work in some other scenarios, but I like this because it’s in my intro class. It’s a nice doorway in. What I’m really interested in seeing is, when I have a couple of these students in the advanced class next time, if that impacts their ability to do some actual real audience research and use that research in context. I think I want to monitor that first before doing some of this other work. I like it in particular because it’s a beginning class even though it’s at the 300 level.

John: It sounds like a really fun project, and there’s nothing really wrong with making learning fun.

Rebecca: Yeah, I had a good time and we had some moments where you had to really practice the deadpan look, you know, be really serious about what it is that we’re doing… and that part was really fun.

Allison: …and that seems like an amazing turnaround on a class where you have to scrap the syllabus halfway through a semester.

Rebecca: Yeah, it was amazing… the community that was formed around the project… and the way that they were exchanging with one another and coming together was incredible, and I was so thankful.

There’s nothing worse than an off semester and you just want out. I think everybody wanted out and so I just said “We’re out. We’re gonna try something new” and it worked, so that was good.

John: I guess the next question is: “what are you going to do next?”

Rebecca: That’s a good question… I think that with this project I’m hoping to expand it a little bit… so I’m currently thinking through “are there things that I can eliminate that I was doing before that I could embed in this project or I just allow them to have the time and space to fully build things out?” They have really good ideas and pretty good plans and the execution is almost there and I’d like to be able to have them have that time for the “almost there” to be “there” and then also to do that reflection piece that I kind of half-assed.

John: Okay, well thank you for joining us and I guess we’ll see you again on our next episode… and back as a host.

Rebecca: I mean, that is, if you’ll have me back.



John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.


31. Writing Better Writing Assignments

Complaints about student writing are embedded in faculty conversations across disciplines. What if the issues with student writing, though, are not their fault, but ours instead? In this episode, Allison Rank and  Heather Pool join us to share suggestions about writing better writing prompts that provide student with explicit expectations.

Allison Rank is Assistant Professor of Political Science at the State University of New York at Oswego and Dr. Heather Pool is an Assistant Professor of Political Science at Denison University.

Show Notes

  • Rank, A., & Pool, H. (2014). Writing Better Writing Assignments. PS: Political Science & Politics, 47(3), 675-681. https://doi.org/10.1017/S1049096514000821
  • Bean, J. C. (2001). Engaging ideas. The Professor’s Guide to Integrating Writing, Critical.
  • Rockmore, E. B. (2015). How Texas teaches history. The New York Times, A31.
  • Braver, Lee (2014). How I Mark Up Philosophy Texts. APA Newsletters, Fall, 14,1 Special section. p. 13


Rebecca: Complaints about student-writing are embedded in faculty conversations across disciplines. What if the issues with student-writing though are not their fault but ours instead? In this episode, we’ll talk about writing better prompts to make explicit what the expectations are and how to get there.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an Economist.

John: And Rebecca Mushtare, a Graphic Designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today, our guest are Dr. Allison Rank, Assistant Professor of Political Science at the State University of New York at Oswego and Dr. Heather Pool, Assistant Professor of Political Science at Denison University. Allison and Heather are co-authors for an article titled Writing Better Writing Assignments published in Political Science and Politics. Welcome, Allison and Heather.

Heather: Thank you.

Allison: Thanks.

Rebecca: So, welcome back to Allison, I think, right?

Allison: Yes.

John: Yes, welcome back, Allison.

Rebecca: So, today, our teas are?

John: Tea Forte, black currant black tea.

Allison: Water again.

Rebecca: It was coffee last time.

Allison: Okay.

Heather: I’m also water because I forgot that this was tea-oriented.

Rebecca: Yeah. Yeah. We have to send out those reminders ahead of time, I guess. Mine is Harney & Sons Paris tea.

John: What prompted your interest in writing about writing assignments?

Heather: I’ll start with that. I was director of a writing center at the University of Washington for a social science writing for a couple of years and then, Allison filled my seat after me. It was after we had seen numerous prompts that our students were coming in and asking for help with, and Allison, after she had completed her time at the writing center, came to me and was like, “I think we can do this. We can do some people feedback about how to do a better job at writing these.” We saw a lot of prompts that could have been more clear, let’s just say that.

Rebecca: Were there prompts that you didn’t understand?

Allison: I think usually we could figure out how to interpret them, but it was very easy to see why students couldn’t figure out how to interpret them.

Heather: Yeah. Right. And so, oftentimes, what happens is prompts are basically dissertations, right? Where you could literally write hundreds of pages on them or they’re so narrow that if you answer all of the questions, then, there’s no space for analysis or creativity or anything like that.

Allison: To add some details, so Heather had that job for two years and then, I had the job for two years. We’ve had four years between us of seeing these various prompts come in across the sub fields of political science and we’re actually seeing a lot of very similar problems and prompts on very different topics, which I think, for us, was part of being able to think about it’s the structure of how we write the prompts and how professors think about prompts is actually a place for an intervention and then, starting to teach our own classes sort of getting the sense that sometimes what comes back from students is on them, but also, we need to be a little bit more responsible around what it is we ask students to do because sometimes, some components of their poor writing may actually be more our fault than we’d like to admit.

Rebecca: I think we can all probably experience the idea that you get something back here like, “Yeah.” “Yeah, yeah, you answered that, yup.”

Allison: Yeah.

Heather: Right. Well, and part of it too, just to follow up on what Allison said, is we ourselves were early career and we’re just writing our own writing assignments for the first time. As a TA, you sort of inherit the assignments that people write and you’re like, “Okay, yeah. We can work with that.” But then when it comes to create your own, there’s no roadmap out there at all, and so, you stumble into stuff and you write assignments that the students have no idea how to interpret. And so, on the one hand, it was seeing some things that were out there that we thought, “Wow. There’s problems here. There’s commonalities,” and we can imagine how to get out of that problem and part of it was self help.

Allison: Yeah.

Heather: We’re looking for a resource that didn’t exist and Allison’s brilliant idea was like, “Ooh, we could create that resource.”

Allison: Yeah.

Heather: And so, that was a huge part of it.

Rebecca: Faculty definitely want students to be good writers …

Allison: Yeah.

Rebecca: … but we expect students to come in with those skillsets often and faculty often see themselves as content providers but not necessarily writing instructors. And I think that we hear that a lot even on campuses where writing across the curriculum exists. What role do you see faculty having in helping students develop their writing beyond just the prompt?

Allison: I think that faculty have a really important role to play on writing, but I think part of it comes from knowing what it is that you want to help students improve and having reasonable expectations for what the class that you have set up can actually help students do. In doing our research, when Heather’s saying we had a hard time finding roadmaps as we dug into a lot of the Bloom’s taxonomy literature and trying to figure out if we’re writing prompts that asks students to take particular steps, are we actually providing students a roadmap for those steps.

Allison: So, one of the things that I struggle with a lot is the way in which I don’t recognize that I’ve been disciplined. So, I’ve been disciplined as a political scientist. I ask questions in a way that political scientists ask questions, and then, get mad when my students don’t understand. That’s part of my expectations. But I also never make that explicit in content, even in the content-driven courses. That the way I’m approaching this content is about a political science perspective and here’s how that might be different and here’s how those expectations should then influence the way that you write a paper or approach a question.

Allison: And so, I think that it’s linking up the expectations for helping students with writing to the expectations we have around content-delivery is I think where a lot of faculty should spend more time.

Heather: I teach political theory, it’s not really a testable subject, and I could do a test but I don’t think that’s a particularly helpful way to evaluate people’s engagement with the content. On some level, I actually think it’s a cap out when faculty members say that they’re only content providers in part because I think we learn through writing and it’s not until we’re actually able to write about things that we grasp the kind of significance and the meaning and all of those things and we actually have some research. And I think in being … engaging ideas, I could be wrong, she suggests that we learn as we write. It’s only in the process of actually trying to put other people’s words into our own context that we actually grasp what’s going on.

Heather: And so, to be effective content teachers, I think we need to figure out how to be effective writing teachers as well and I think it’s important to be clear when we’re asking them to summarize and when we’re asking them to analyze and when we’re asking them to evaluate and those are all different things. And we need to give them opportunities to work on those things before we have them write big final papers or we ask them to do all of those without any scaffolding.

Rebecca: So, speaking of those nice keywords, I know that I’ve had conversations with students and they can’t actually tell me the difference between describe, analyze, reflect, things like that. So, can you share a little bit about how you might frame that for students, what those words mean and how you structure that?

Allison: Sure. Now, I’ll say off the top, I think that faculty, a lot of the time, don’t know what they mean when they use those specific terms. And so, part of what we would actually see in the writing center is prompts that said describe, but we read them a no, that if you actually just described, you are not going to get a good grade on this paper. That that was the word that was in the prompt, but I would bet money, if you follow those instructions, you would have problems. So, I think, I occasionally, for students to actually define the terms that are in the prompt, if I’m asking you to analyze let’s walk through in class one day, what would be the difference between summarizing this content and analyzing this content, so, actually walking them through what the terminology is.

Allison: I also think that that’s where having sub prompts after a prompt can be really helpful, where you break down for students that I expect you to summarize or describe a particular amount of the content and then, analyze something so that they have to distinguish for themselves what part of this assignment am I addressing in different components of my paper.

Heather: Yeah, I think that’s great. I do things like I have students do small stakes regular assignments where I have them summarize and then, reflect, and then, ask a question. And so, they’re already thinking about the difference between summary and reflection and then, I actually, in class, will talk about what’s the difference between describing something and analyzing something and one example that I use, because I went to grad school in Seattle is I’m getting off of a plane in Seattle. Seventy percent of the people on the plane are wearing super awesome Gore Tex water repellent gear and 20% of them are wearing wool and 10% of them aren’t wearing coats. So, that’s a description of the situation. But analysis is telling me why that’s the case. That’s trying to explain what we see and to make sense of it.

Heather: So, I then ask them to come up with reasons why, what that description says makes sense or what stories they can tell about why that’s what they see. There’s also a great piece, it’s the Netflix … the new Sherlock Holmes, it’s the lady in pink where he walks into a room and he sees a woman dead on the floor, and then, Sherlock Holmes goes through and comes up with all the stories about the particular things that he’s seeing are what he’s seeing. And it’s a really effective tool for students to be like, “Oh, summary is really different,” right? And many times, prior instructors may have asked them to summarize, and so, they’re relatively good at that, but it’s the analysis part that they really struggle with. Again, I think it’s our job to help them figure out what analysis actually is.

Rebecca: One of the things that I’ve noticed in my own department, we’ve been talking about writing in our department quite a bit lately. We had a conversation … My department is made up of art historians, designers and studio artists that all makes up like an art and design department. So, it seems like it’s all one discipline but we all have really different cultures within that discipline, and that we talk through what’s some of the kinds of writing that we do on our department and discovered that we didn’t really mean the same thing.

Rebecca: And so, we’re working on developing a common language and sharing that out within our own department to make sure that we can be consistent between levels because I think that’s some of the confusion that our students are experiencing.

Heather: Yeah, I think that’s absolutely right, and of course, writing across disciplines varies greatly, so we may put these statements in different places if we’re in the Humanities or if we’re in the Social Sciences, we may approach quotes differently and whether or not it’s appropriate to use them or not appropriate to use them, what counts as evidence differs from discipline to discipline. And the way I set that up for students is to say, “You’re going to end up in jobs where you don’t actually know what they want when they ask you to write something, and you’re going to need to be able to figure that out, and that’s actually what we’re trying to give you here is the ability to approach a writing practice and figure the rules out. And there’s different rules in different disciplines and your job is to develop the facility to be able to move between those things as needed.”

Allison: Yeah. I’ve done something in class with my intro class which tends to be … it’s very frequently a general education class. There tends to be students from a lot of different majors and actually, just asking them how do you think you’re supposed to write paper. And you’ll get all sorts of answers about …

Heather: Right.

Allison: … what a thesis statement is supposed to be, you should never use I. Which is a thing in political science, it’s like “No, I’ve got a correct that right now.”

Rebecca: Yeah.

Heather: Right.

Allison: You need to tell me I argue X, Y and Z and students are so taken aback, but it’s so much easier if you start, at least for me, by getting them to tell you what are all the rules you think you know so that I know where I need to tell you that at least for this class in this space, that’s not the right rule.

John: But part of it is just being more transparent with students …

Heather: Yes.

Allison: Yeah.

John: … and making sure they understand what you expect from them in terms of coming up with good writing prompt. You mentioned scaffolding a bit.

Allison: Yeah.

John: How do you scaffold it in terms of the stages of writing? How do you break it up for students or do you have them just submit it in whole draft or what?

Allison: Yeah. I think it really depends for me on different classes. So, for the intro class, before their four-page papers, they write a couple of four-page papers, they do something called reading reflections but it’s really a worksheet where they have to tell me the author or authors, the title, what type of source is it using the Chicago style guide. It’s essentially breaking out for them, everything they would need to know for citations, they have to tell me the research question, what they think the thesis statement would be in their own words, which again, is to get them in this format of saying like Madison argues X, Y, and Z. A couple of good quotations and then, their own initial impression of the piece. So then, when they sit down to write the paper, they already have the stack of material that’s like, “Oh, if I want to argue X, who would I go to as evidence to support that claim?”

Allison: In my advanced classes, I tend to break it down more in terms of the annotated bibliography, so before they would ever touch writing a longer paper, I first want an annotated bibliography and I do it slightly different than a “normal annotated bibliography” I ask for one paragraph of summary, and then, for every entry, I need one more paragraph that tells me the relationship between that piece and at least two other pieces in the annotated bibliography. So, getting them to think through what are the relationships that help them categorize where a literature review could go before throwing literature review on top of what it is that they have to write. And I may have stolen the annotated bibliography from Heather.

Heather: It’s possible, [inaudible 00:13:03] annotated bibliographies, yes. So, yeah, I do some similar things. I started to use Allison’s reading reflection assignment that I’m inching closer and closer to that mostly because I’m a little overwhelmed by grading. I have them do seven of these reading responses, I call them, where they do summary, reflection, and then, ask a discussion question. So, that is getting them to train to summarize stuff, and again, the point is they have to do one of those for each of the authors that we read, so they actually have a pretty decent summary and they have the other 24 summaries from people in the class that they can go to when it comes to writing their own papers.

Heather: And then, for my intro class, I hand a paper out and they need two and a half or three weeks before it’s due and then, I require a draft on say Tuesday, they then do peer review in class on Thursday, and then, the final draft of the paper is due the following Tuesday. So, they have to have a pretty decent working draft a week before the paper is due. And if you make a good effort, then, there’s no deductions from your final grade so it’s not a graded assignment but it is one that if you don’t do it will hurt you, and the same thing for sub [inaudible 00:14:06] good faith peer review. I like that a lot because the peer reviewers catch really irritating things that when I see them time after time after time, I get angry.

Heather: And so, the peer reviewers catch a lot of that where they say things like, “You seemed to have a problem with paragraph structure,” and somehow, when they’re hearing that from their peers and then, they hear the same thing from me when I give them feedback, I then ask them to do a reflection on the feedback that basically enforces them reading the comments, where one of the questions is, “Do you any commonalities in the feedback you’ve received from your peers and myself?” and surprisingly, there often is commonality there. And so, then, I start to get them thinking about what their patterns of error and what can they do to address those patterns of error.

Heather: So, in terms of scaffolding, I make it due early and I make a little stakes draft and then, they have a week where they can talk to their peers, they can come talk to me on office hours, et cetera, so that the paper that they turn in has been seen by at least two other pairs of eyes.

Allison: Yeah. I should say I do in my advanced classes, I have a version of that where there’s a draft due two weeks before finals week and then, students do not get evaluated on their drafts, they get evaluated on the quality of their feedback.

Heather: … what level?

Allison: Like a 5% grade. Yeah, I think it’s 2% you turned in a draft and then, after that, I have a sheet, I was doing it not graded, just sort of the participation points and I would get feedback that was like, “I really liked what you did here.” And I was like, “No.”

Heather: No.

Allison: “This is not going to work for me,” and so, changed it to where there’s an actual rubric for me to evaluate the feedback that they provide each other, and that has gotten students to give much more direct feedback to many students.

John: That was something I was just going to ask, have you used rubrics, and what do you see as the advantage of using a rubric for assessment?

Heather: Because we have writing-specific classes, and then, we have ones that aren’t but frankly, all of mine would qualify for the W overlay just because of the percentages. Teaching, I really care about writing, so that’s a simple part of the course but not all of them are Ws and if there are Ws, they have lower numbers of students and I can’t offer only Ws for curriculum reasons. And so, generally, all my classes are really heavy on writing, and so, I’ve moved more and more towards pretty specific rubrics where I basically highlight and bold stuff, and then, have a relatively short comment section. And I’ve just switched to a new rubric this semester and I actually think I like it.

Heather: I tend to over comment on their papers when I’m not constraint by a rubric and constraint by space, frankly. And so, for me, I’m a big fan, right? “This is what an A paper looks like,” “Here are four different categories that I’m assessing you on,” “This is what a B paper looks like,” “Here are four different categories for that as well.” And so, I’m tentatively enthusiastic about pretty specific rubrics.

Allison: I like very specific rubrics for intro classes. I have a hard time using very specific rubrics in a lot of my advanced classes, and I think it’s because I struggle to write rubrics that I think are balanced, aligned on being detailed enough to be a value but broad enough to where students can really sort of flex their muscles when it’s an open research question.

Allison: And then, a lot of my advanced classes, it’s an open question. And so, then, I find I have a rubric but it ends up being like on these criteria, would you be rated as excellent, good, fair, weak, poor. And so, it tells them where they are and then, with comments, but it’s nowhere near the level of sort of fine green value of the rubric that my intro classes have where everyone’s writing on the exact same question.

Rebecca: You are both hinting at differences in the role that a faculty member might play in different levels of courses between intro, intermediate and advanced. Can you explicitly address that and what the faculty member’s role is in each of those kinds of levels?

Heather: Yeah. I think I do something similar to what Allison does with my upper division classes, which I just taught at senior seminar. I have them do essentially two kind of shorter papers that are kind of lit reviewee where I’m asking them of pretty specific question about some segment of the course reading. And then, I have a big where like, you tell me what your research question is and then, they go through a proposal, an annotated bibliography, a draft with a clear pieces and then, a final draft. And that starts basically from the fall … the first one of those is the proposal was basically due at the beginning of November and the final paper isn’t due until the middle of December.

Heather: And so, I’m also a really big fan of if you don’t like the topics I wrote, then, you write one and tell me what you would like to write on, in part, because I think we say this in the paper, I actually am really interested in reading interesting papers. I would much rather read a paper that incorporates the material from the class in a way that you find compelling and that you want to write about, than I would read your [wrote 00:18:40] response to my question that you found really stupid. And so, I do give them that freedom but with a caveat that they do have to come talk to me. And I give students actually that freedom for intro, all the way up to seniors and my most advanced classes. But I do think it’s different when I’ve got a cinema major versus when I’ve got Political Science majors. I talk about writing and use examples in different ways across those levels.

Allison: Yeah. And I think that I, in the same way that you wouldn’t deliver the same content from intro to American government to an advanced American government class, I think it’s the same in terms of writing skills. And so, I tend to focus more in the intro class on these statements. Trying to lay bare some of the relatively, I would think in some ways, rudimentary, the thesis statements are deeply complicated space focusing on the building blocks of being in the discipline. These expectations of writing and then, in the more advanced levels, focusing on the types of writing that I think are in different forms more likely to be both of interest to them, but then, also, let them test skills that are more likely to be relevant.

Allison: So, for instance, I don’t have any full papers in my advanced classes usually outside of the big papers that are due during finals week, but for every book we read in the class, I have them do a critical analysis. It’s essentially a book review, but I found that if I call it a book review, I get book reports, which is not what I want so I call it a critical analysis. And the guidelines are I want no more than a half page of summary and then, up to two and a half pages of analysis. I don’t read anything over page three, with the idea of you need to be concise. I don’t want it to be summary and I give a set of prompts about what you can … here’s some places you might want to go but they’re very open in terms of talking about the content you know from your broadcast communication class that I haven’t read or how does this book help you think differently about some event that happened on campus or is happening in the news.

Allison: Where I think that that type of analytical skills sort of more what I want my advanced students to start being able to do, this thing I read in the classroom connects to some broader literature in political science or literature from another discipline or just the way I interpret the words. And that’s where I see the writing in the advanced classes outside of the research papers as more my responsibility.

Heather: I was just going to follow up with I’m teaching seniors again and they’re on the job market themselves and trying to figure out why they just did a major in Political Science and trying to actually have them answer a question that is meaningful to them as opposed to like I need to know that you know how to read a book and find a piece of statement. And so, really, trying to create space for more advanced students to do more advanced interesting things.

John: In the classes where you use rubric, do you share the rubrics with students? Because I would think that would give them a little bit more scaffolding in letting them know what you think is important and helping them determine how to structure the papers and things.

Allison: Yeah. I would say sometimes I do, and sometimes, I don’t. In introductory classes where I have students that are already very concerned about doing things “right”, I actually tend not to because I find that they then hew to the rubric in ways that are actually really counter-productive. I’ll give them more of what I would consider the left-hand column of the rubric. So, I’ll take into account, when I’m grading, your citations will be 10%. Your grammar and style will be up to 10%. Your thesis is going to be worth 20% so that they know how points are distributed. But I don’t actually like to give the specific boxes that are sort of it’s going to be an excellent if there are x criteria, because I found that that tends to lead to really I think counter-productive conversations about well, how do I meet the standard of that box, as opposed to what makes a good analytical argument.

Heather: I don’t put percentages on my rubrics. I’m a big fan of the visual rubric where I’m like there’s a lot of things in the C columns and what’s a C. There are a few things in the A column but there’s mostly things in the B column, that’s a B+. I’m a political theorist, we don’t really do quantitative things particularly well. I’m a big fan of not sharing that because I don’t actually know how to do that. Some rubrics, I share with them. I share rubrics about their participation with them, like here’s what I expect a good participant in this class to be able to do, and I assess them on that, but I just started using this new rubric so I didn’t share that with them at the beginning of this term, but now, I think maybe I should have. I don’t know if I think it would help them or hurt them, I wonder.

Rebecca: I have detailed rubrics that I use for grading and I just started using our learning management system to use the rubrics and I found that that actually can be really challenging because when they do that on paper, I sometimes circle the line between things.

Heather: Yes, right. Me to. Me to.

Rebecca: And then, you have to pick one …

John: But if you’re doing it in Blackboard or some other learning management system, you can always override the …

Rebecca: Yeah. Well, I was just going to …

John: … if someone works outside the box.

Rebecca: … which I have done. Yeah. And I sometimes will make a comment if I put it in the C column, the comment is to why it’s there if it wasn’t one of the criteria I had originally come up with, and so, it’s very clear. So, I’ve been experimenting with that a little bit. I tend to share the rubric, but I also find that students tend not to look at the rubric.

Heather: Until it comes back with a letter grade on it, and then, they’re like, “But why? What happened?”

Allison: Yeah. I will say that I use the point rubric in Blackboard for classes where the size or the amount of papers … and so, it’s basically just for intro where it speeds grading.

Heather: Yeah.

Allison: Right? That’s when I do a points rubric in Blackboard, but even then, the idea that Blackboard defaults you to having three categories and that I always go in and have like no, I definitely need a couple of more point variations, yeah.

John: I usually have four or five on that in mind.

Allison: [inaudible 00:24:20] Yeah, yeah. I have to go in and sort of add because I tried doing it with three ones, and I was like, “Why is everyone getting a 30?” It’s like …

John: Well, you can pick whatever categories or …

Allison: Yeah. Yes, and so, I had to go in. That was my first experience using it last year and I was like, “Well, that’s wrong.” Let’s go back and then, regrade it, and that changed all the rubrics. But okay now, yeah, it takes a little learning.

John: But it can be an iterative process …

Allison: Yes.

John: … where if there’s some work, you can modify it.

Rebecca: Yeah. That’s also why I often don’t put percentages for the categories upfront is because I sometimes see what I get back to see if I need to adjust what I thought the weights were going to be to make it more fair.

Heather: I struggle with the percentages because writing is hard to do in any sort of objective fashion and I worry about the kind of thesis is a percentage, because sometimes they write a not so great thesis and have a brilliant paper, right? And so, then, you’re like, “Well, okay. So, you got 75% on the way there on your thesis,” but your argument at the end was actually really good and so, my feedback is write a clearer thesis because your argument’s really interesting, but it’s hard for me to figure out how to do that.

Rebecca: One of the things that I think we haven’t addressed but hinted at a little bit is not only is there disciplinary ways of approaching writing but there’s cultural ways of approaching writing too. And so, when you’re talking, Allison, about needing to write really concisely, that’s something that’s popular in design as well.

John: And in economics.

Rebecca: Yeah.

John: Like economical writing.

Rebecca: Right. But I often have students who want to write with very flowery language or think the academic writing looks a particular way, and usually, it’s very convoluted, very complicated sentences that don’t make any sense.

Allison: Yeah. Yeah.

John: I wonder where they get that.

Allison: Often, I train political science but I always, in my intro classes and occasionally, in my advanced classes, depending on how many students it would be a repeat for pass out a piece called How Texas Teaches History from, I believe Ellen Rockmore. It was an op ed in the New York Times a couple of years ago about the high school textbooks that had gone out in Texas where all of the “benefits” of a slave-holding society, which is a deeply-problematic framing. Masters taught slaves Christianity has an active phrasing, and then, all of the brutalities of slavery are framed in passive ways. Slaves were beaten, slaves were assaulted. And so, you excuse any actors, and I passed that out to students before we do, sort of when I complained about your grammar, when I correct grammar, I’m not doing it because this is a pedantic exercise and I just want you to meet these standards. I do it because in political science, it is incredibly important that we are accountable for who the agents are that act, and the only way that I know who your agents are is when you tell me who the agents are.

Allison: And I think that sometimes, that tends to help ground at least conversations to about flowery language where it’s a slightly different point, but I can often say, “What you’re doing here is actually obscuring for me who is acting and what they’re doing,” and the most important thing that I need to know is who’s acting and why they’re acting and why it matters. And so, I found that piece actually really hit students in a way, it’s like I never thought about it before, I never thought about why it mattered before, and I found that to be really helpful.

Heather: We both teach in political science and I think that this is particularly true in politics. Instead of the something must be done, well, what needs to be done and who needs to do it, right? And in politics, I think that’s a pressing question in ways that it may be less pressing in other’s field of study.

Rebecca: I find that one of the comments that I read a lot for design students is like you haven’t said anything actually. “There’s only one sentence here that says anything and the rest can go.”

Heather: I do spend a fair bit of time talking about my own writing practice actually in class where when they’re working on the first drafts of their paper, I will tell the story of my first published article, I was like all done and, “Oh, yay, I’m about to send it out,” and then, I realized that the word count was 4,000 words less than the words that I had, so I needed to cut 4,000 words from my manuscript in order to send it in.

Allison: Oops.

Heather: Exactly. Geez, it was the first ignorance. And then, I tell them I got rid of all of the adjectives and all of the adverbs and I cut several paragraphs/pages in total and it made it better. You read the draft that I thought was finished and the draft that was submitted and the second one is way better because I had to be economical with my language, I had to be really clear, I had to be direct, I had to say what I wanted to say and move on as opposed of lingering, loving over the words because they are so pretty. Which is what we as people who write as a part of our job eventually realize that, but they haven’t had that drilled into them in the same way and like, “That’s my job.”

Allison: Yeah.

Heather: It’s like, “You kill your word babies.”

Allison: I definitely am a fan of showing students my writing process. So, for instance, when I teach the annotated bibliography, Heather, you may not know this, I actually showed the part of the annotated bibliography I sent you for the Bletchley Circle paper.

Heather: Oh, my God, that’s awesome.

Allison: When I was in charge of doing the lit review for a piece that we co-authored, what I showed to students on how to do a lit review is like, “So here’s the thing I sent to my co-writer. This is when I was doing work with someone else, this is how you do it,” so I showed that. I’ve actually taken to showing my annotated readings in class.

Heather: Me too.

Allison: So, in classes where I want students to annotate, I actually just put my work up on the dot cam, instead of doing like, “Everybody, to page 57,” and then, I just can only see my book. I want them to see that part of writing is also the annotating. So, getting as transparent in some ways about my process as possible I have increasingly done.

Heather: One of the first pieces I assigned in my intro class, it’s a four-page piece. I think his name is Lee Braver. It’s in the journal or the teaching journal of the American Philosophical Association or something, that’s how I mark up texts, right, how I mark up philosophy texts. And so, part of it is just getting them to pay attention to how they’re reading, and in many ways, that gets them to pay more attention to how they’re writing and to how … when they read a text, that they leave thinking, “Oh, I understood that,” it’s usually because the writing is really clear and you want everyone to leave, to finish reading your paper in the same way and have that sense of like, “Oh, I know how it was argued.” And if they don’t have that, then, it’s your job to actually fix it. In the same way that we can read authors and say, “Gosh, I wish Thomas Hobbes [used to 00:30:44] do our words.” But he’s dead, you’re not. You can do better.

Rebecca: What are some tips that you have for faculty who are running their … assignments for the semester or getting ready to write ones for the fall?

Heather: Right. Yeah. We’re definitely working …

Allison: Yeah.

Heather: … on our fall prompts. I think it’s really helpful to have other people look at them. I actually think it’s really helpful to have people map in your field and not even in your sub field, look at them, so I will occasionally ask my partner to read prompts and she knows how to do what I do. I’ve definitely sent assignments to Allison to just be, “Does this makes sense?” “Do you understand what I’m asking?”

Heather: I’ve had former students read prompts as well to see if it’s clear what I’m asking them to do. I think a huge part of it is time, like not writing it right before you hand it out, and then, getting other people’s eyes on it who you …

Allison: Yeah. So definitely, yeah, Heather and I send prompts back and forth before the semester starts. I’m also a huge fan of having all of my assignments done before the semester starts. I have everything loaded in Blackboard in the assignments, every assignment is in before the semester begins. And that helps me know, partially, it’s for me with planning a syllabus. If this is what I expect students to be able to do, where do I have to be. What do they actually have to have in order to do this assignment. So, for me, it’s just part of the planning process.

Allison: I also increasingly have a sort of stable rotation of assignments that I like, that I figured out packages for, and I, particularly in the advanced classes where it’s that more sort of open, I want them to be able to do what they want to do. I think figuring out assignment structures that get refined overtime and work well, and then, if they’re open enough, you can reuse them pretty frequently. And the thing I like about that a lot is that, then, the students start interpreting it for one another. It helps them become teachers for each other.

Allison: So, for instance, with those critical analysis assignments, occasionally, when it’s students that I’ve had for the first time, they’ll ask a question. I’ll try to answer it and then, another student will raise their hands and be like, “Dr. Rank, I got it,” and they’ll be like, “So, the thing she wants from you is this,” I’m like, “Great. Thank you for that.”

Heather: I’ve also have taken to, as I’m doing the grading, particularly at this point, for my intro class, I give them two or three options for which topic they want to answer and I will switch generally one of those topics each time because I’ve realized that it’s not actually asking something that’s important for them to think about for the course or it’s really poorly phrase or it’s not directing them to actually answer what I want. And so, if you’ve got something that’s worked relatively well, tweaking it as you’re grading it, you get your first five papers, and you’re like, “Oh, nobody answered the … I thought they were …” it may be me. Like maybe it’s not my students that are … maybe I misstated what I actually wanted.

Heather: And so, I will, as I am finishing grading something, if I realized that I wrote it wrong or that I wrote it unclearly, I immediately go in and fix it because I know I won’t actually remember the next time I use this that I did it badly.

Rebecca: Heather, I’ve also found that to be a really good procrastination technique during finals week.

Allison: Oh, my God. Oh, yeah.

Rebecca: So during finals week, I do so much planning for the following semester.

Allison: Oh, that’s when many assignments get written to go in Blackboard for the next semester, let’s be honest.

Heather: That’s true.

John: Going back to the thing about having a portfolio of assignments that you can rotate in, how do you deal with things like Chegg and Course Hero and other sites where students upload those materials?

Allison: Sure. I think because or what I’m talking about, the prompts are so broad that honestly, if students have my assignment for how to write an annotated bibliography ahead of time, bully for you. I’m really excited that you’re reading this annotated bibliography guideline …

John: Well, what I’m more concerned is …

Allison: … prior to being in my class.

John: … for the final projects and so forth, what prevents students from submitting mildly revised version of something that someone submitted two or three years ago?

Allison: I think that becomes about how often you teach the same class and using the same package for the same class. So, I’d say I have probably three packages of assignments that are scaffolded, that work for different types of classes. And I tend to not use the same package the second time I teach a class. So, you’re getting a good chunk of time between assignments, and again, I don’t teach large enough classes where I’m super concerned about not noticing, if that makes sense. Because they have to give me a research, for most of these, they require what your research topic going to be and then, I’m going to give you some feedback, and then, you’re going to turn in an annotated bibliography and I’m going to give you some feedback.

Allison: So, to some degree, if those are all coming back and you’re pulling them out of something like Chegg, I feel relatively-confident that I’d be able to tell.

John: You’ll recognize it.

Allison: Yeah.

John: Okay.

Heather: And I also, even for the classes that I recycle topics for regularly, I will often realize between iteration two and three that I asked for three sources from the course and actually, that was the un-doable and so, I lessen it to two. And I actually caught somebody who used the three sources prompt for the two sources assignment that I had given them, and then, I switched the readings out as well. And so, previously, it had been the full book and this time, it was an excerpt, and I don’t think you read all of [Espinoza 00:35:47] to write this paper.

Heather: And so, those minor adjustments that you’re making to your syllabus, it’s relatively easy to catch, and I’m not teaching 250 students a semester. I have maybe 50 or 60 or 70, and so, it’s relatively easy for me to catch the minor variations that I’ve made in my assignments or that I’ve made in the syllabus that they might not think I’ll catch.

Rebecca: What response have you gotten from students about your assignments in the way that your assignments are structured?

Allison: I’m not sure if this is about how the assignments are structured, though in the advanced classes, it might be. I get a lot of feedback from students that my classes are where they get the most feedback on their writing. That they never get as much feedback on writing as they get from my assignments. And I think that is partially because I think … I’m sure this is true for Heather too. If you’re a professor who cares a lot about writing, you give more feedback on writing. But I think it’s also because of that, so many of my assignments are staged. That I feel a real obligation to give a lot of feedback, and then, to give do overs in some ways, right, where it’s just the lower stake stuff first, and then, you can fix it.

Allison: And so, I hear a lot from students about appreciating the amount of feedback. I also hear a lot about appreciating the variable points that I assign for certain assignments. So, I have assignments that are structured so that if you improve more than more than one-letter grade between the two assignments, the point value of the second one goes up and the point value of the first one goes down, and students also talk about really appreciating that.

Heather: I certainly have students who say in my evaluations and I also ask them to do a final portfolio that reflects on their learning as a writer, what are your greatest strengths as a writer, what are your greatest weaknesses, or I don’t say that, I say what are your areas to work on improvement. And so, I ask them at multiple points throughout the semester to do that kind of metacognitive reflective stuff because I actually think it makes them better learners, which means they become better writers. And so, my experience as an undergrad was I got a lot of papers that had a letter grade and like the occasional “Good” or “What?” in the comments, and that was pretty much it. I had no idea what I needed to do to get an A and I wanted to get an A.

Heather: And no one actually took the time to tell me the steps that I needed to take and so, I try really hard to say, “Here are the four things you did really well on this paper,” like, “Oh my goodness, you used those sources really well. Great. Clear possible interpretations of the authors. You have a beautiful writing voice. Your citations were perfect,” and then, follow up with this sort of areas of growth and improvement.” And I also end up always being a cheerleader that I’m like, “You have one more of this,” “You can totally crush it.” And so, students, even if they get a grade that they’re not particularly excited about, I am on their side. I want them to succeed and they know that and they also know that because I didn’t just give them a letter grade.

Heather: The drawback, of course, is this is incredibly time-consuming, and I’m not sure how sustainable it is and I hope it is because I really care about it and it’s one of the places that I find the most satisfaction when I’ve had a student in two or three classes and I look at their first paper that they ever wrote for me and I look at the last paper that they wrote for me and there’s much difference, and I value that so much. But it also just takes so much time, and that’s why I rubrics, obviously. As I’m moving more towards rubrics that have less space for me to write, that becomes a little more feasible.

Allison: I feel like one of the things that I really like about and keep in my brain from the paper that we wrote is to always give that sort of a big question that lets students …

Heather: Yeah, yeah, it’s true.

Allison: … the difference between a prompt and sub prompts. So the prompt is the question that you could write a dissertation on, and then, the sub prompts are the space where you tell students, “To get a good grade on this paper, I’m going to need you to do the following three things. I need you to summarize the framer’s argument or justifications for x component of the constitution. I need you to analyze the differences between the interpretation of the constitution when it was put in place by the framers and the interpretation of it in the wake of the new deal. And I need you to interpret changes and who is allowed to be a citizen or considered to be a citizen in the United States.

Allison: So, there’s a really big question at the top that you could write a dissertation on and then, there are these cues that help students understand what are the important parts of answering that question. Because I often think that’s where students have a hard time distinguishing. They could give you lots of answers for the big question but those of us who are in the field would be like, “That is the least important thing you could say,” like, “That’s the least relevant way you can answer that question,” but it’s an answer to the question. And so, it’s maybe not fair to hold them accountable for that.

Allison: And so, giving the sub prompts helps cue them to really pay attention to the particular things that matter in a way that they might not have before. And then, again, having that ahead of time helps me, as the professor, know what I need to make sure they’re hitting in class. If they’re not bringing those things up on their own, I need to make sure I do it with them so that when the paper comes out, it’s not a surprise.

Rebecca: Sounds like there’s focus on scope and a focus …

Allison: Yes.

Rebecca: … on values.

Heather: Yeah. I think that’s absolutely right. That is one of the things that I continue to do when I’m writing assignments is that … The paper that just came in from my intro class was why is political obligation important for a community. That’s a huge topic and then, I’m like, “According to Socrates, you need to tell me what Socrates says and you need to tell me what Hobbes says and then, you need to make them talk to each other and then, you need to make an argument for why one is better than the other.” And that’s what those sub prompts do that I think is really helpful.

John: I guess our next question is for each of you, what are you going to do next?

Allison: Heather.

Heather: I will take that one. I’m really interested in writing a piece on what analysis is because I feel like we tell them all the time more analysis but we don’t actually clarify what we mean when we say analysis, and I don’t think they have any idea of what they mean when they say analysis. And so, I’ve started including an appendix in my ridiculously-wrong syllabus that is like what is analysis. At some point, I just need to write that up because I think most of us struggle with communicating to our students what we mean when we say that word, and I think being a little more clear about what we mean would actually help them learn to do it better.

Allison: In terms of teaching, the project that I am hoping to work on right now is something that I started maybe two years ago after I went to the Digital Humanities Summer Institute on using annotate with students as a way to help them with the group annotation. I’m trying to work through those annotating skills moving towards better writing. And so, that’s for my American politics classes that I am hoping to get better at. I get frustrated by the tech quickly, and then, drop it.

Allison: So, that’s one of the projects I’m hoping to work on this summer and I’m hoping to do some comparisons in different classes with how students do with group annotations versus annotating on their own.

Heather: Can you explain that? When you say group annotations, they’re all reading the same PDF and then, marking on it?

Allison: Yeah. So, it’s essentially, you load the PDF online and then, you can assign small groups of students to all work in the same version of the PDF.

Heather: Interesting.

Allison: And so, they actually can go through and put comments on each other’s annotations and say, “You could find someone else’s interpretation and let them either deepen it, disagree with it, link it to some other part of the text where they can start flagging for each other and having a conversation that is deeply in a relatively small section of a text.” Heather, I’m thinking about this for American political thought, an African-American political thought.

Heather: That’s wonderful. I love that idea.

Allison: Where it’s like beyond, I get some value out of collecting their annotations which I also do in American political thought where I show them how to annotate. I give them that same piece from Braver, and then, for the first couple of weeks, I actually collect their annotated readings and hand them back, and then, I’d like to start trying this group annotation as a way for them to start thinking of reading and working through texts is more of a collective exercise in conversation.

John: What was the software you use for that?

Allison: I believe it’s called Annotate.

Heather: Is it iAnnotate?

John: iAnnotate is an iOS app or an Android app, but is it-

Allison: It’s not an app. It’s a…

John: Web tool?

Allison: … a web tool.

John: Might be hypothesis.

Allison: Maybe it’s that.

John: I know a lot of people use that for …

Allison: Yeah.

John: We’ll check on that.

Allison: Yes.

John: We’ll add that to the notes.

Allison: Yes. You should. I found it very interesting to use and then, the one time I tried teaching it, students had all sorts of questions and I basically was at the front of the room like, “I don’t know,” and so, then, I scrapped it and just need some amount of time to go back in and play with it and anticipate more of what the questions would be.

Rebecca: Well, what you did was gather the questions.

Allison: Yes. Let’s …

Heather: That’s right.

Allison: … call it an …

John: A research exhibition.

Allison: Yes. Yes. Exactly.

Heather: Because it turns our much of teaching is not being successful.

Allison: Right.

Heather: Trying things that didn’t work very well, you’ll do it differently next time.

Allison: Yeah.

Rebecca: It’s an iterative process.

Heather: It’s exactly right. Exactly right.

Rebecca: Well, thank you so much for the insights that both of you have provided. I think that gives a lot of faculty food for thought.

Heather: Thanks so much for having me. I’m honored to be a part of the SUNY Oswego crew.

Allison: Yes, I was excited to be back.

John: If you’ve enjoyed this podcast, please subscribe and you may review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

30. Adaptive Learning

Do your students arrive in your classes with diverse educational backgrounds? Does a one-size-fits-all instructional strategy leave some students struggling and others bored? Charles Dziuban joins us in this episode to discuss how adaptive learning systems can help provide all of our students with a personalized educational path that is based on their own individual needs.

Show Notes

In order of appearance:


Coming soon!

29. Learning about learning

Learning is hard work. The most commonly used study techniques often provoke the illusion of knowing. David Parisian, a member of the Department of Curriculum and Instruction at SUNY-Oswego joins us in this episode to discuss how he helps students overcome their misperceptions by introducing them to the science of learning.

Show Notes


Rebecca: Learning is hard work. The most commonly used study techniques often provoke the illusion of knowing. In this episode, we discuss one faculty member’s success in helping students correct misperceptions by introducing them to the science of learning.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today our guest is Dr. David Parisian from the Department of Curriculum and Instruction at the State University of New York at Oswego.

Rebecca: Today our teas are…

John: Barry’s Irish tea.

David: I’m not drinking tea today.

Rebecca: No?

David: Nope, I’m trying to cut back. [LAUGHTER]

John: With some people, it’s an addiction.

Rebecca: I have a problem. I’m drinking Prince of Wales tea today. Could you tell us a little bit about what you teach and how you became interested in incorporating evidence-based practices in your classes?

David: Well, I teach a few different courses. When I’m on loan to the Computer Science Department for the CSC 103, which is “Computer Tools and Informational Literacy for Educators” and then in the teacher prep program, I do ADO 394, which is “Interdisciplinary Methods” and then EDU 303 and oversee the “Block One Practicum” students, and do the online EDU 430, which is taking concurrent with their student teaching.

Rebecca: What is the 303 and the last one you just mentioned?

David: 303 are the “Block One Practicums.” So, their first semester junior year, when they entered the the block sequence for education, they have to spend time in a classroom. So, Field Placement secures their placements and then they spend a semester observing and helping where they can, but they’re getting their feet wet of being in a classroom. And the EDU 430—the online course—is “Professionalism and Social Justice” and that’s an online course that they take concurrent with their student teaching. So, a lot of the modules that they’re based on actually helps them in preparation for the edTPA that they have to submit for certification.

Rebecca: So how does the evidence-based practices fit into these courses that you’ve been teaching?

David: Well, one of the things that happened is the CSC 103 is designed for freshmen or transfer students that come to us for teaching certifications. So, one of the things we look at ishellip; approach that class from how technology is integrated in education. And one of the thingshellip; there was the book-read we had a few years back by Peter Brown, Make It Stick, kind of reaffirmed my doctoral work as we went through it… and I created a little quiz… a little matched-pair quiz… so that if they had to learn effective means… “does repetition build memory or does retrieval build memory?” and taking the material from the book and organizing this quiz… and what I found out was that all my students failed. So when it came to the 12 Principles within the book that we were testing on, less than one percent passed… which meant that a lot of the students we see coming to us, even though they can come to college, don’t have the knowledge or foundation of what strategies are most effective to learning. And that, in a teacher preparation program, my thought was, “do I continue that propagation through the pipeline or do I try to stop this stop that flow of students who are not effective or aware of the most effective strategies as they go out to be teachers or do we want them to have those effective strategies so they can implement them once they start teaching?”

Rebecca: So you staged an intervention?

David: Yes, it really becomes an intervention at that point and it’s really interesting because they have so many misconceptions coming in. And what we try to do is… they change their relationship with how they view sleeping. We set it up within the course so they take Barbara Oakley’s MOOC Learning How to Learn. So, they’re taught how sleep flushes out the chemicals overnight that are built up in the brain. They talk about procrastination and the Pomodoro techniquehellip; the benefits of flashcards and how to integrate that in spaced practice… and that’s one of the interesting thingshellip; I was just talking students today because they’re preparing presentationshellip; but a hundred percent of the class knew about spaced practice but no one knew how to implement it, or what it might look like.

Rebecca: Right.

David: So we provide opportunities for them to explore those areas, whether it’s creating flashcards where they’re using a flashcard app or have them making flashcards to build in the spaced practice to know when to practice and how to interleave and all those components that research has shown to be most effective.

John: Many of our listeners are familiar with all of these, but could you go back and talk just a little bit about the Pomodoro Technique.

David: The Pomodoro Technique is really fascinating. What it is… is for those who procrastinate, you block out 25 minutes …and actually the Pomodoro Technique is based on a tomato kitchen timer… where you set it for 25 minutes and then at the end of 25 minutes, you take a five-minute break… and then you set the Pomodoro timer and you go through it again. And what happens is you can begin to measure your workload in terms of how many Pomodoros it’d take. For the students, what I notice is they start changing their perception of assignments from a product driven “what’s the end product?” to a process of “how to get to that end product?” So many have commented on that and you can get apps on your phone that are the Pomodoro Technique, that will set up a 25-minute block. There are apps that will stop any notifications for that time so you don’t have to be keep looking at their phone because that’s the distraction that I’ve noticed, is that people are attached to their devices and have to be aware of every beep and every notification and everything that comes as opposed to blocking time. So the Pomodoro Technique has been pretty effective and most students have implemented that in terms of working on their procrastination skills or adjusting… changing behaviors.

John: So why do students have these misperceptions?

David: Good question. I come from a K-12 environment. I was a secondary science teacher for 18 years. I was a district administrator for another 12. So, coming from the K-12 environment, what my message was to them is that you were taught by loving, caring educators who were passionate about what they did and did the best they could with what they had… based on the information. That might not have been entirely best practices of what we now know about neuroscience and how the brain learns and the effective study strategies on the materials from Make It Stick. So, I think they grew up with whatever progression they went through. One thing I did notice is that students who struggled… that worked hard in school but just got into collegehellip; still struggle with the workload. What surprised me was talking to some studentshellip; is the students who were the bright ones… the ones that got it quick in school… that went through high school with no challenges… come here and all of a sudden, they’re placed in a situation where now they have to study and do all those things and they don’t have the skill sets to study. Just coming up through, I think our assumption is we feel that students know how to study because they’re in college and really what’s taking place is students aren’t really taught how to study. I think the assumption that students know how to study probably backs all the way down into fifth, sixth grade. I think we had some earlier comments where we were talking about us being in school and teachers made us write flashcards. We didn’t understand why and they might not have understood why, but we made flashcards. Now, as that is an appropriate practice, or one that works that you know, that can be integrated into a spaced practice and it’s really just teaching them a little bit about neuroscience… how the brains learns… how that all we do is encode, consolidate, and retrieve… and how do you build those principles and practices using technology… using skillsets… managing their time and trying to put that into a package where they can begin to see it. Because once they see it, then they want to improve their studies so they may begin doing it… so they go to bed earlier, they get better sleep and they begin to change their behaviors. I didn’t expect that but it was an awesome sidebar from that.

John: Part of it is, as you said, they haven’t always had much practice or training in learning how to learn and partly that may be because many of the teachers didn’t have the same…

David: Exactly.

John: …people have been just doing the same things that seemed right for an awfully long time… and one of the problems, though, with some of the evidence-based practices is that it doesn’t feel quite as good… because when you try to work on retrieval practice, after you’ve been away from a topic for a while, it doesn’t feel as good as perhaps repeatedly rereading something until it looks so familiar that you think you understand it. So there is that fluency illusion that people get that feels really comfortable and when you do some low-stakes testing or when you do some attempts at retrieval, you realize you may not understand it quite as well and it doesn’t give you the same sort of reward immediately. So it takes some training, I would think, for students to be convinced that these methods really do work. It sounds like you’ve been able to achieve that in your classes.

David: One of the advantages is it’s a setup so that we use the class and the content within the class as a training mechanism. So, they’ll practice working on a flashcard app to learn the app and then create flashcards for the content. One of the things they do have to do because the modules I set up are in a worksheet form so the initial encoding part is our instructional part. The consolidate part is them now going into another class and applying it in another class and showing the evidence in mine. So that they’re forced to, not only just use it in mine, but now go into a Psych 100 or a LIT course or whatever they’re in to create flashcards, create mind maps, begin to incorporate that… and we map out, When are your tests,?” So, how far do you already have to back up. So it’s really just coaching them on how to utilize those strategies and to get them so they can begin to internalize that to meet with their success.

Rebecca: Do you find that the students are a little resistant at the beginning or do you think because you’re providing the evidence and the science behind… the reasons behind… why you might use a method, is that what’s helping?

David: Well I think it’s a combination of both. Part of it is they have to go through the MOOC… so it’s not me presenting the science. Then we have a series of videos. There’s a gentleman by the name of Thomas Frank, who has a series of videos on studying and using it, so he has a more modern twist to it, being late twenty-something or just through college so it’s a more animated video. And then I’m in class to give suggestions, those type of things, but once they saw that on the first day school, they take a quiz and fail it and then realize that they don’t know what they think they know, then they’re pretty open to “this is what we’re gonna do for the semester.”

Rebecca: Sounds pretty motivating.

John: It is a motivating technique, yeah.

David: Well, I had one student once say like, “first day in college and I’ve already failed a quiz.” I’m like, “rather fail it now then in 15 weeks.”

Rebecca: Yeah.

David: …that’s the poise. It’s really just redefining their tool sets and introducing new ones… and not everyone takes to everything. Like, mind mapping is a classic example. whether it’s Tony Buzan’s method, but I’m very mind map oriented, but a lot of students have a negative experience towards it and I said, “if you want to be a teacher you may never do a mind map but you will know how to do a mind map, because you may have a student that needs to know how to do it and you have to be able to teach it so even if it’s not for you and you don’t like it, you darn well better know how to be able to help someone else learn it.” It’s redefining a different role for the teacher as our candidates come through as be more knowledgeable base, more pragmatic, and more understanding on what effectively works because they can then share that information with their students.

Rebecca: So what do we do for all those students who aren’t in your class in their first year, right, who aren’t becoming teachers but they’re, you know, trying to exist in this system, who might also have the same exact struggles that your students have? What’s your recommendation?

David: Well again, Thomas Francas has great videos and whether it’s how to read a college textbook, how to take flashcards, how to do any of that. That’s one aspect– knowing that you need help is another aspect. We have the success tutors that that have those skill sets, are using the same information to help other students. So, I think there’s resources on campus to support that.

John: …and our Student Success specialists or I do are very familiar with all these techniques and they work with students that are struggling to help build that up.

David: Right, and I think looking at it from a instructor side of the house, just as people become familiar with the strategies… is making those known..making the strategies known. If you’re teaching an Economics course or teaching whatever, if you say there’s a set list of vocabulary or concepts that you need to know, you know, put them in flashcards, go to these links, it’s not like you have to prepare everything because you can’t make meaning for someone else… they have to make it on their own… but you have to provide those initial concepts or the information because the student has to be able to take it in, but telling them how to take it in or just pointing them to links can facilitate that process. I don’t think anyone really comes here with a desire to flunk out, you want to be successful, but some of the times, they get paralyzed by the paralysis of the whole situation.

John: Some of it can be done through course design too, so that they are designed to include some level of retrieval practice,

David: Absolutely.

John: … some amount of interleaved practice,

David: yeah

John: ….and some spaced practice.

David: Yup, yeah, absolutely. And those type of structures in the redesign isn’t that large of a jump to do for the professional side of the house. The biggest thing I can tell for any student, the simplest thing to do if you’re feeling that stressed is breathe. It’s kind of funny… whenever you’re upset, what do people tell you to do they tell you to breathe and just from a physiological standpoint, taking deep breaths… slow deep breaths… once you start breathing about six breaths a minute, there’s a branch of the vagus nerve of the parasympathetic system of our physiology that automatically slows the body down. So, when you’re taking a test, what do you do? Breathe slowly. If you get stressed out on it, draw a circle around every fifth question so that every fifth question, you’re just taking three deep breaths just to say, “this is all right, we can do this.” And you that will calm the physiology and reduce that stress response of seeing an exam… the fight, flight, or freeze… and we’ve all experienced those… with varying degrees of success. [LAUGHTER]

John: But we want to set up a system where students generally be successful.

David: Absolutely… Absolutely.

Rebecca: So you’ve talked about the Pomodoro Technique… you’ve talked about breathing… you’ve talked about flashcards. What are some other key things that instructors could help students think about as strategies to be successful?

John: Well, you mentioned sleep, too.

Rebecca: Sleeping’s good.

John: And that’s something students often have trouble with.

David: Well it was interesting because there was a presentation done once and one of my students… I commented that she would stay up till 2-3 in the morning texting her friends and then they got that part of the MOOC and talking about sleep and she’s the one that started to go to bed at a more reasonable hour to be able to sleep. I think if one hasn’t read Make It Stick… that’s a great book and I think for a lot of us on the professional side of the house… as instructors we’re like the great white shark of our content, we’re the rogue, we’re at the top of the food chain. But one of the things, I think, the book does is lays out for you this way in which people encode information, consolidate, and retrieve it and I think having that as a foundation and then, reaching out to the center here for like, “how would you design something? I’m struggling with this” and just say “I want to reach the students” because I think a lot of professors do want to make those connections with the students. I think there’s help and support on campus to help people in designing those strategies. Personally I start with trying to give the overview… try to give the gestalt of the course… “what’s the wholeness gonna look like” and then just lay out the clear objectives and then integrate and make the assignments, being those flashcards and studying and there’s just so many ways to communicate, I guess. Even taking Barbara Oakley’s MOOC Learning How to Learn and just learning some of the basis from the neuroscience, even from the students. Have students get together informally and go through it. It doesn’t really take a lot, all the information’s out there.

John: And if I recall, it’s offered every month isn’t it? It’s a four-week MOOC and it’s offered very regularly.

David: Well it’s a four week…, like every week now, I think. For those who aren’t aware of the MOOC through Coursera, a MOOC is a massively open online course, Coursera is a vendor for that and Barbara Oakley’s MOOC. I think last year i read in the New York Times where it was the most popular MOOC on the planet.

John: It is i believe still the most popular MOOC…

David: I know when I took it, I took it with two hundred and seventy-five thousand other people at the same time. That was a few years back. Integrating that type of information and layering it into a course just gives a different feel for the students. And as I tell them even though they’re freshmen and transfer, I said “if you’re going to be a teacher, down the road, some of your professional development is going to be delivered through these.” So, we’re just getting them ready orientating them. Well there’s just you know some really simple information that people can do to design, redesign, and be more effective, more engaging and have students be more successful.

Rebecca: I’ve had students respond very positively when you explain why you might do a particular technique or a method. At first it seems like “why are we doing this? why are we doing quizzes?” but if you explain why and how it helps they’re much less resistant and actually embrace the idea. I’ve had students ask for more quizzes because they’ve learned how it’s helpful…

David: That just reminds me of, in our lesson plan development, as the students are going through it… one of the parts of the lesson plan development I have after activating prior knowledge is setting the purpose… and setting the purpose is “so, why is this lesson important? why do you need to know it? How many courses have we ever been in that no one’s ever told us why we’re there and you’re absolutely correct. Once you say the why and give the explanation, it creates an association or connection with the student and that’s one of those underpinning things… as the more associations you can connect to them, then the more apt you’re gonna have their buy-in to engage you in the content.

Rebecca: I think a lot of times students see assessments like quizzes and tests as some sort of penalty…put you in the penalty box or whatever… and that’s a faculty member’s way of torture or something, but as soon as you say that the purpose of doing this is to help you recall information and to make sure that you have that foundational knowledge, you can continue building in these more advanced classes. They stop seeing it that way and it’s pretty amazing that it doesn’t take much of a conversation… it takes having the conversation.

John: And the more frequently you do that, the easier it is for them to get past that because high-stakes testing is stressful, but if you replace it with lower stakes testing and more regular testing, it’s easier for them to see that they’re learning from this experience and it doesn’t hurt them as much if they screw up… that it’s an opportunity for them to improve and continue…

Rebecca: However your students responded after your class like moving into these upper-level classes because you’ve been doing it long enough now that probably some of them are now in those advanced courses I would imagine.

David: Part of it is coming up through the pipeline and having viewed or patterned assessments or quizzes as the “gotcha” and getting hammered. Ken and Rita Dunne, one of the things they stressed that I always kind of stress is “high content, low threat.” So, whenever you can engage and have your material be of high content but a low threat, students are more apt to engage because they don’t have that fear reaction going on and I think it’s changing that patterning by having the conversations that these are beneficial for you. So, that’s my thought on the quizzes. Some of them continue that patterning and that’s one of the things where I try to keep it going for the semester in hopes that they’re gradually continuing to do those processes, those strategies to continue their learning. The one thing about teaching the freshmen… I’m probably one of the few faculty members that have this interesting purview because I see them as freshmen or transfers coming in, I’ll see them as their block one junior EDU 303 practicum. I’ll touch base with them again for the ADO students that take 394 as their block two, so I have them as a class and than in block 4 when they’re seniors going through the the end of the pipeline. So I can see their growth along the way from that and…

John: And ADO is adolescent

David: Yeah, the adolescent, yeah… and that one they’re constructing the lesson plans, they’re trying to see how this all fits together, trying to pull on multiple layers. We revisit Make it Stick because in all my courses that’s just part of it. So, they’ll read the book as a textbook in that section… so then they can begin to refresh like “Oh, I remember when we did this” and “how’s it going?” and then we have conversations on when they’ve been using it and how effective it’s been. I just keep trying and plugging away and stepping up to the plate…. every day you get to take a new swing

John: And the more people who do that in their classes and certainly the more people who are trained to do that at lower levels in elementary and secondary school, the better off students will be.

David: Yeah… especially nowadays when you you look at the various challenges in a k-12 environment. If you can begin to seed the ground with what works and just focus on that then we’ll be okay.

John: Do students go on and use these practices in other classes after they’ve been exposed to them in your intro level course?

David: Part of it, once they leave me is trying to continue that propagation based on the courses they’re in. Some of the strategies are more effective in some content areas than in others. Math is always a struggle in terms of looking at applying the flashcards while you can do color coding or dual coding where you’re including images or multicolored in the equations as you follow different variables through an equation sequence, those type of things. The other thing is that the strategies, and this might be one of the misconceptions students have, learning isn’t easy. Some people comment like “well if I just sit here I should get an A”…… but learning is messy… learning’s organic… learnings dynamic… and learning takes a lot of work and sometimes, depending on the student’s course load and what they’re taking… if you’re taking a 4-credit science course, you got three hours of lecture and a three hour lab somewhere in the week… having been a science major and having a lot of hours on a lab. So part of it is finding time to create the materials… the mind maps… the flashcards. Those students who have a better time management… work ethic, those are the underpinnings I think that makes this a successful student and that they put forth the time and schedule that to do all those things that are necessary, whether it’s creating a mind map or whether it’s creating flashcards or creating the time intervals for the spaced practice or when to do the spaced practice. I was talking to a friend who used flashcards and whenever she was grocery shopping for her family picked the longest line because it wasn’t about getting out of the grocery store fastest, it was about being in the line the longest as she pulled out her set of flashcards and reviewed them in the line at the grocery store. You can find intervals to do those type of things.

John: I always wondered why there were always more people in the longest lines. Maybe…Maybe they’ve been in your class [LAUGHTER].

Rebecca: Yeah… everybody’s doing flashcards.

David: Then again yeah, flashcards is just one aspect of that but you can integrate that from a quiz standpoint… from a retrieval…. and one thing that could be interesting is when you look at the research on flashcards, or how to create them, there isn’t the level of “how do you create” going down Bloom’s continuum of higher processing from a flashcard aspect. A lot of the information we see is low level…. vocabulary words, or those type of things… but how do you all of a sudden take two flashcards and compare them and say “compare and contrast these two concepts of something…” and so how do you get a bigger cognitive load going from using those… and the designing of flashcards… that’d be a great study for someone to do. I’lljust put that out for anyone who’d want to. [LAUGHTER]

John: You get to work with students a bit later in their academic careers that you worked with earlier. How do they respond when they come back in upper-level classes? What do they say about their experiences?

David: Usually, semesters later they forgot about me. [LAUGHTER]

Yeah, but one of the things we try and do at the end of the course is they have a five-minute presentation they have to do… and I give them the slides and so it’s like “how has this changed you?” and they go through and reflect on that. So, I give them a template of what the presentation is and it’s their five minutes of fame where they get to begin to find their teaching voice, and it’s the first time they’re in front of the class talking about it. So, you talk about how they how they did it. In terms of seeing them later on, the people who use flashcards and grew up continue using them and then you have various levels of people who took the buy-in to create those processes. The other thing is you try to encourage them to use it, so as they’re developing their mini unit you have them do the flashcards to go with the unit… you have them do the mind map. So, you have them go through it, and I think from that aspect, they recall fairly quickly what it is they needed to do to generate it… and then it reminds them like “oh, yeah.”

John: But, if we do this in more classes and we use them or we structure our classes so that students naturally adapt some of these practices, it’s going to help reinforce these things… and the more people are reminded of how important this is and how their usual practices may be really helpful in cramming for a test the next day but aren’t going to allow them to remember the things much past that day.

David: Right. In the real world you have to remember those things past that day.

Rebecca: A couple of the themes that you were mentioning most recently is about time management and the work ethic component fitting into this and so it seems like that’s the next discussion. How do you make sure that students know what is a good work ethic? Do they even know? Do these conversations happen? I’m not sure that they do.

David: You can see those students that have work ethic pretty quick …and I’m just trying to flashback through courses and images through my head of students… from a freshman those that are asking questions… those that are getting work done and turning things on time… those who are turning it in early… those who who show up to class earlier… and sitting there… they have a certain level of comfort within themselves… where they’ll ask a question. Truth be told, I didn’t feel comfortable asking a question in college till after seven years of college… not that it took me that long to get a bachelor’s degree… it wasn’t till after I did my student teaching a long time ago… and after I did my student teaching I learned how to study. I did the outlines before the chapter. The following semester I went for a neuroscience certification taking our geology courses. So, I outlined the chapters before I walked in there. I pre-taught myself the material. I laid out all my notes and stuff before the lecture… had the conversations because then it was internalized to me that this was important… and I think until someone gets that into their intrinsic fabric of themselves… where they want to take this as being important… and I still don’t know what I want to be when I grow up. I think that’s the big thing. People who are serious know to get the job done. They’ll do the work and they’ll do it to what they need to do… and if you give them criticism and feedback, they correct it.

Rebecca: I’m hearing a growth mindset described.

David: It is.

Rebecca: …you’re open to feedback… The the real challenge is how do you get students with the fixed mindset, who maybe don’t have that work ethic. or good time management skills. to get on board.

John: …and you mention that case about some students earlier who had done really well in high school and then suddenly struggle when they get here. Those, empirically, tend to be the students who have a fixed mindset… where they’ve been successful with the techniques they’ve used, but once they have to move into a new environment or they have to engage in more transfer than they had to earlier… suddenly face some struggle and it’s a shock to them and they often give up.

David: Well, and one of the interesting things, though, and this they’ll need to be reminded of, but within all of us we have growth and fixed mindsets.

JOHN. Yes.

David: So, I could have… I’m not picking on math but I’ll pick on that…. Most students… math is not a strong aspect, because really the only time you do math is in math class. No one asks you when you’re walking down a street to factor a quadratic equation, right?

John: Well, rarely… [LAUGHTER]

David: …rarely… but you consider a musician. They know with practice they get better. In sports, you know with practice you’ll get better. In math, if you practice you’ll get better. So, part of that is understanding the context of which it is. I may struggle in math but I might be a musician in the band, and so I know. So, you have to kind of transfer where they have been successful in showing them, in this part where they’re not being successful, how they can be. Because, if you can tap into a person’s growth mindset… and it could be in a K-12 environment. You have somebody I saw students riding their skateboards… phenomenal skateboarder… can do all these tricks… will spend hours learning a trick, right? That whole idea can be shifted to their studies as well. How long did it take you to learn this trick?

John: How many times did you fall you know along the way.

David: …and how many times did you fall? The culture is changing where, not only are we imparting the information to the students, but we’re also being their coach. We’re trying to nurture them. We look at them as adults coming here… without the parents for the first time. The baton we’re handed is actually trying to nurture them into the adult working life and understanding how we learn… how we process information… how we interact… the building of rapport. how do the rapport aspect is all part and parcel… I feel… what we need to do… or what we do… I know it’s what I do.

John: How did you integrate the MOOC into your class?

David: I build the class around modules that last two weeks. So, with the beginning of class, I took some of the material from the MOOC or what the topics were and then I created worksheets based around that. So, if it was procrastination then there’s articles that they were reading in, and picking up on, and getting their takeaways… So, that part was teaching the foundational aspects of of how these strategies work, and then giving them time to practice and doing them within the two weeks. The MOOC they can view offline. They can take the quizzes. The other thing I did, is with Barbara Oakley, she had Coursera set me up as an administrator for my course. So, then I could just upload my class list to it and then it would keep track of the quizzes…

John: Oh, nice.

David: …and then I could download the the grades or whatever. So, she was giving him the content on one side; we were building in related practiced and article support on the other side. Then the consolidation part… and I broke it down into it an encoding section of the worksheet, a consolidation, and a retrieval part basically patterning our learning process. So, watching videos and reading we’re encoding… applying the material was the consolidation… and then their reflection and the reflections based on making a video, responding to an interview question, or reflecting on their experience over the two weeks… and they were able to communicate those. So, part of it is just finding out what the MOOC is doing, getting materials that kind of pattern that (that’s where I brought in you know the Thomas Frank videos and other support materials).

John: The learning scientists also have some good ones.

David: Yeah, exactly, and I use a lot of the Learning Scientist’s material and McDaniel’s site deals with retrieval practice, so there’s a variety of things. We try to overlay the MOOC with Make it Stick and strategies there to create an environment that over the first four weeks they’re experiencing… they’re learning… and they’re beginning to apply… and then as we build out the other modules, we still keep repeating… For example this is module four we’re in and one of the things I’m training them in is advanced search strategies. What are the topics they’re going to be searching? Well, there’s eight setups within the lab so… elaboration… retrieval… spaced practice… those are the deep research things. So, each group has to now prepare a presentation, but they have to do the research. What’s the research that supports this? and what are strategies to help? So, now there’ll be eight presentations of the 8 strategies that they’ve learned. Trying to deepen and make touch points through the semester and keep reminding them… constantly reminding… constant… constant…[LAUGHTER]

John: So, what are you going to do next?

David: Well, next will be a continuation with the Computer Science…. Looking for more in-depth application across the content areas to help students… and then professionally, this summer, working with Educational Administration Department and their Project Blend Symposium. We’ll be doing the third installment of Resiliency and Leadership working with the Institute of HeartMath. So, outside of that, we’ll continue to work in those areas of heart-brain synchronicity and just working and having fun.

John: Thank you for joining us. It’s been a pleasure.

David: It’s been my pleasure. Thank you very much.

Rebecca: It’s always great to hear what you’re doing in your classes and the results and thanks for sharing that for everybody else.

David: Very good, thank you .

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

28. Augmented reality

Does reality sometimes fall short of your expectations? Perhaps it’s time to augment your reality. In this episode, Renee Stevens joins us to discuss the creation and use of augmented and virtual reality experiences that can increase our productivity, overcome cultural and language barriers, and provide a richer learning environment. Renee is an award-winning Interactive and Motion Designer and Assistant Professor and Associate Chair of Design at the Newhouse School at Syracuse University. In addition to teaching, Renee also runs her own design studio, is an exclusive designer for Minted and the co-director of education for the upstate New York Chapter of AIGA, the Professional Association for Design.

Show Notes

  • Tag AR
  • R Studio (Renée’s design studio)
  • Metaverse (referred to as Meta in the podcast)
  • Pokemon Go
  • Snapchat
  • Swift
  • Yelp
  • Zombies, Run!


John Does reality sometimes fall short of your expectations? Perhaps it’s time to augment your reality. In this episode, we discuss the creation and use of augmented and virtual reality experiences that can increase our productivity, overcome cultural and language barriers, and provide a richer learning environment.

Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca This podcast series is hosted by John Kane, an economist…

John …and Rebecca Mushtare, a graphic designer.

Rebecca Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

Rebecca Our guest today is Renée Stevens, an award-winning Interactive and Motion Designer and Assistant Professor and Associate Chair of Design at the Newhouse School at Syracuse University. In addition to teaching, Renée also runs her own design studio, is an exclusive designer for Minted and the co-director of education for the upstate New York Chapter of AIGA, the Professional Association for Design. Welcome, Renée!

Renée: Thank you.

John Today’s teas are…

Rebecca English afternoon.

Renée: Chai!

John Republic of Tea’s Emperor’s White Tea. So could you tell us a little bit about augmented reality? How does it compare to virtual reality and mixed reality and so forth?

Renée: Sure. The biggest difference between virtual and augmented reality is that virtual reality is a fully immersive experience that actually gives you a completely new view and a full inclusive view of another place. So, you could be fully immersed and you have 100% of your attention focused elsewhere, versus an augmented experience which is basically a layer of information that is applied onto the world around you. So, you are getting additional information, but yet you still have all of the things happening in your environment including your sights and sounds that you can then layer information on top of. And then of course you add that to your mixed reality which is kind of just a glorified augmented reality… where it’s a little more technical and a little bit more computer graphics based… a nice happy marriage between virtual and augmented reality.

Rebecca For those that maybe haven’t had an experience with augmented reality and can’t quite envision what you’re talking about, can you describe an augmented reality experience?

Renée: Well the one most people know would be Pokemon Go for better or worse, but that’s one that most people usually have a connection to. Snapchat also has some augmented experiences, with stickers and filters and things like that. Those are the ones that I think are the most mainstream that people understand. But, essentially, it could be something as simple as just adding navigation into the view where you’re driving… having it look like it’s in the road in front of you… or it could be something like using your mobile device to learn about something new in front of you… like a new device…. like how to turn a coffee machine on or something like that. So, it’lll apply an additional layer of information that makes the task at hand easier.

John And that information could be triggered by visual cues, by your phone’s camera, or by geospatial coordinates.

Renée: Yes, absolutely. So yeah, it depends on the function and obviously the users of the app. But yes, it could be based off of the camera… actually tracking a specific location in your environment… or your actual geolocation…. or visual cues or taps… or interactions with the user.

Rebecca What got you interested in augmented reality?

Renée: I was bored, no. [LAUGHTER] Not really.

Rebecca I’ve seen your agenda. I don’t think you are bored.

Renée: I wish. When I actually like sit down and think about it, it’s really been a perfect kind of combination of all my backgrounds. I started off with my undergraduate in graphic design, so I’ve always had this love for design. I obviously love to teach because I’m a professor, but I’ve also have a master’s degree in photography and specifically multimedia and storytelling. So, in my undergraduate, I was focused more on the foundations and the principles of good design practices and that led me in towards being more of a user experience designer and user interface designer. But my love for story and all that kind of got me into motion design, and so when you combine motion design and the user experience design… Merged together, that’s like the perfect marriage of augmented reality. So, I get to create mobile experiences and that kind of UI/UX experience, but with my knowledge and love of storytelling and designing for 3D space using time and interaction and all that good stuff. So I almost got into it by mistake, because I was just starting to do all these things and had all these ideas and I was trying to find a platform to make them come to reality. It turned out that’s in an augmented one.

Rebecca Nice… Nicely done. [LAUGHTER]…

John Nice segue.

Renée: I never said that before, it just kind of came out… so that was good.

Rebecca One of the barriers, I would imagine, in getting into this field as a designer, is having technology or packages available so that you can actually enter into this field and so the timing seems like it timed when… I think you and I talked about this previously… that it timed well when Apple released their AR kit.

Renée: Yes, so I actually had this concept for an app called “Tag AR” and it wasn’t called that at the time, but I had this idea and I was trying to make it come to fruition and I couldn’t get the technology right to do it. I had this idea, this concept… how is this going to work? And I didn’t know exactly how that was going to happen and I was actually talking to developers before AR kit came out from Apple and they were all like “we don’t have a solution for that, we don’t have a platform to release this.” So, I was kind of waiting for something to come along and that’s when ARKit came out. That day I reached out to all those developers and said “Ok, now we have the platform, who’s ready to do this?” and of course they all looked at me like “we don’t know how to do this” and I’m like “well, no one does, that’s the whole point.” So, yeah, the timing of that was great because I had this idea, this concept, and just needed the technology to back it and it all came within weeks, if not even days, looking for that perfect solution.

Rebecca How funny.

Renée: I know. It was meant to be, it’s kind of like one of those things where you don’t ask questions, you just go with the flow, like clearly that’s the path, so I just took it.

John And you’re doing the programming yourself?

Renée: Yes, so I ended up developing it myself, I couldn’t find a developer like I said. They said “you know, we don’t know how to do that, it just came out” and I said “yeah no one knows how to do it”. So I struggled trying to find the developer, and I’ll actually give my husband the credit, he said “you do what you always do” and I said “what’s that?” and he said “you do it yourself.” So, he may regret that now, seeing how many hours I put into development, but yeah that was like: “okay, I can do that” and he just gave me that extra confidence and I have been developing the whole thing independently since.

Rebecca Can you tell us a little bit about Tag AR?

Renée: Tag AR is an augmented, “hello, my name is nametag” essentially. So, what it does is it offers people’s names for you in augmented space. So, when you’re using the app, you actually can look around and see everyone’s names hovering over their heads.

Rebecca It’s not a little like Big Brother or anything [Laughter]

John But… it’s an opt-in program.

Renée: Right, it’s an opt-in program. You have to be signed in… and so the target audience is really specific for groups of people who are networking or working together. So, it’s really meant for educational platforms, for workshops, meetings, networking events where you actually want to be interacting and meeting new people. So, really any place where you would be wearing a nametag, this would be an augmented replacement for that, allowing you to see the people in the room from afar or up close, searching for people who maybe you want to make sure you network with, and then having that extra component where… you’re already on your phone… you have this device… you can then connect digitally, you have like this digital business card feature… where you could then connect with them via the app too.

Rebecca So if people want to get involved with Tag AR, what would they need to do?

Renée: Well, at the time of this recording, it will be launched very very soon, so by the time this comes out it should be in the Apple Store and, it is available for download on all iOS devices, on the iPhone. You just have iOS 11 installed and you have to have an A9 processing chip or higher on your phone to experience the augmented experience… so that’s iPhone 6s and above.

Rebecca Great!

Renée: And it’s free.

Rebecca Even better.. [Laughter] Have you designed any other or been involved in any other augmented reality experience development ?

Renée: Yes, so I’ve been actually working and collaborating with different people and different groups on some other projects at the same time as getting Tag AR up and running. I currently am teaching a class called immersive design, which is focused on augmented reality and I’ve been the creative director heading up the project that they’re creating, which is actually a translator app, but it translates… instead of language, it’s actually translating culture. It’s a 3D object recognition application that then translates that… and the target audience for that app is specifically refugees all over the world, who have been displaced from their homes for various reasons, but are trying to familiarize themselves in new culture… and so what it does is it helps actually scan 3D objects, identifies the name and that’s the augmented reality experience and then it uses resources on the web for them to learn how to use that within the culture and save collections of their words that they use most frequently to help them teach the language in the culture.

Rebecca I remember hearing a story about Syracuse, not that long ago… about refugees…. and one of the things that some refugees were struggling with was having electric stoves and knowing what they were and how they worked. So I can imagine… to someone who’s not a refugee or isn’t familiar with those communities, it would be like “I wouldn’t understand why that would be useful,” but I know of some of these really specific stories where “I don’t know what this device is, I have no idea how to use it because we were living in a tent, like we didn’t have an electric stove.”

Renée: Right, and actually the Syracuse community has a lot of refugees… and not a lot of support necessarily in some areas… and so one of those is obviously the cultural changes that they just need, that extra support that my students are helping to… at least help a little bit with some of that culture shock.

Rebecca Does that project have a timeline associated with it?

Renée: Yes, so by the end of the semester it has to be released, and it will be for iOS devices and then over the summer will be developed for Android as well.

Rebecca Great! Were the students doing the programming for that?

Renée: Yeah, students are doing everything. So we’ve taught them the entire experience, so from concept ideation all the way through designing, prototyping, developing and now they’re onto user testing. So, they’re getting the full experience… as they should.

John Are these undergraduates or graduate students?

Renée: It’s actually a mix. So, this is actually an experimental course. It’s the first time the course has been offered at the University. It’s a combination of undergraduate, graduate, and all different majors. So, we have some with programming backgrounds and some with absolutely none, and they’re all diving in and learning how to program and develop mobile apps and create AR experiences. So, it’s been pretty fun…

We’re also working on a few other ones based off of interest and also just some research projects that I have going on. I have a research project called “Augmented Learning” that I’m working on and it’s basically looking at how we can teach tools within the education platform using AR versus the traditional… like if you wanted to learn Adobe Illustrator for instance, you’d have to go from like a Lynda.com video frame to then going back to Illustrator and then going back and forth…and so what it does is… this research project that I’m implementing over the summer and then I’ll be testing and researching in the fall, having students compare student’s learning outcomes based off of augmented learning versus just your traditional platforms. We’re looking at time and how the timeliness is affected because of the Augmented learning experience, so I have that in the works.

Rebecca Sounds really interesting.

Renée: Yes, it should be very interesting. It’s really just waiting on the technology to catch up with my idea….[Laughter]……. Really… that’s what I’m waiting for.

Rebecca I’m noticing a pattern.

Renée:Yes, and so then the other idea that I’m working on is looking at almost like a closed captioning option for students… and the core of all my work is looking at how augmented reality can help overcome learning disabilities. So, Tag AR has an underlining assistance for those who specifically are dyslexic. So by offering a visual, to usually only an auditory component, it allows for additional resources for people (specifically with dyslexia) to have assistance that they need without really making it obvious that it’s specifically for people with learning disabilities… and so I also look to see how AR can help within the classroom setting for people with learning disabilities, but also people who… maybe English is their second language or additional other ways and so this is almost like a closed captioning option. So people could experience the same classroom setting, but they’ll almost like see your closed captioning, like you would see in a television but you would see that in your AR view…

John …in real time.

Renée: In real time, yup! So you could have that translate to a different language or it could just be English to English or whatever the case may be, based off of your need and the whole concept is that you’re getting assistance, but it’s discreet. And so a lot of people with learning disabilities they don’t even tell anyone they have learning disabilities. They don’t get the resources because it could come with a negative connotation… and myself considered. So I’ve never sought any assistance for any of my learning disabilities, including dyslexia, and it would basically empower those who have any challenges to get assistance without it being even known to others, even the person next to you.

John You gave a talk here earlier today, where you were mentioning that, while on the phone it’s not entirely discreet because you have to hold up the phone in front of your face…

Renée: Yes, it’s a little socially awkward.

John …but, you did mention the possibility of migrating this to various types of glasses that are in the very near horizon.

Renée: Yes, so I’ve been looking for partnerships with different companies that are creating the technology that would make this much more discreet and so obviously one of those things in the forefront is the design, right? You want it to look like normal glasses, it shouldn’t look like…

John Google glass…

Renée: Yeah, it shouldn’t look weird, right? Because then it’s very clear that you’re wearing something that’s an assistive device. Actually there’s a company in Rochester, who I’ve been working with that is really great: one, because of their proximity but also just because of the technology and their form and function of their product. Which will take away the awkwardness of holding up your mobile device and it looks like you’re just wearing glasses and you’re getting the additional assistance where needed.

Rebecca You have any initial research findings related to learning disabilities and augmented reality? Have other people done studies that you’ve been looking at or is this really kind of a new frontier?

Renée: Well, people get a lot of research on learning disabilities… and specifically design or typography for dyslexia and that kind of thing. I’ve been working on that and researching what’s available for those kinds of platforms and then seeing how I can then implement that into the AR space. Part of it has been a lot of research in those fields in what already exists and seeing how I can then take that and apply it to the AR component. A lot of things, specifically typography is a big thing, making it very clear what works best and is most readable, especially on a small mobile device or what will work within the optics, we know wearing glasses.

Rebecca and for an ever changing background that you have no control over…

Renée: Yes, so there’s lots of things you can’t control when it’s AR. Light, for instance, is a huge thing. You don’t know how much light will be in the area where people are using these… especially in an educational platform… it might go from really bright to really dark depending on what the professor is doing and then obviously that becomes harder to design for. So, you have to be prepared for the unexpected. The backgrounds could be really busy or they could be really simple and you really just don’t know. So, you have to have really clear separations between the foreground, background… and being very conscious of that design… especially when you’re dealing with accessibility.

Rebecca A lot of your work focuses on design for good, right and research specifically about learning disabilities. How do you see AR having a social impact?

Renée: Well, I think it’s almost a obligation as a designer to show the power of design for good… because it has so much power to do good. So, I almost see it as something that it’s like “yeah, I’m not gonna just design something for the sake of designing something, I’m going to design something that’s going to have a purpose, right?” and so that purpose could change, but I really see, especially within AR space, it’s this idea of practical augmented reality. You could make dinosaurs go across the street in front of you, right? but why? what’s the purpose? and so by adding that extra element of the why and answering the “why?” you actually can then solve a problem that exists within our society and it would offer additional assistance on top of being a really purposeful and helpful platform to design on. I don’t really necessarily look for the area of design for good, I think it almost is just something I gravitate towards because I am a problem solver and I look at things that I think could be improved through design, because design is that powerful and then finding the right platform to solve those specific issues or problems.

Rebecca Where have you seen students struggling as they’ve been designing for AR?

Renée: Development for sure is huge, especially if they’ve never developed before, but the first initial concept is really… if they’ve never designed within 3D space it’s kind of getting the idea of depth in their work, that’s kind of been the biggest challenge initially. Once we do some… just simple prototyping…. and actually I’ve been having them work in After Effects first, before they get into coding, just because that’s how I got into it and to see how the idea of Z space and depth applies from something that really is 2D to something that is 3D… so, taking what they know about 2D, applying 3D to it, and then making that fully immersive jump.

Rebecca So, in After Effects it allows the students to have video which simulates that regular field of vision and then you have your graphics or whatever layering on top.

Renée: Yes, and then adding cameras or giving that prototype feel, so that they can visualize the experience first, before they design it in a place where they have really no control over the environment…. just to give them that practice run.

John For someone who is interested in programming AR apps, what would they need to learn or what types of tools would they need to know?

Renée: So, it kind of depends on what kind of experience you’re creating. So if it’s going to be more of a 3D based object-oriented app, then right now it would be learning Unity…. it’s a little bit of a clunky program…

John But it’s free.

Renée: But it’s free… yeah, or at least parts of it are free… and so kind of like the industry standard I would say for creating those kinds of objects. But, what I’ve been teaching specifically in class, just because of the accessibility and the mainstream effect, would be just programming within Xcode, which is using a language called Swift and it’s actually the most approachable language I’ve ever had to learn… I guess because Apple created it.

John And that’s what most development is in apple.

Renée: Yes, right and so because Apple has their hand on it they usually try to make it a little bit more design friendly. You can definitely see the effect of Apple’s hand on that for sure. It makes it a little easier to teach and students usually can grasp it faster than other programming languages I’ve seen them try to tackle. So I think it really depends on your platform how you want to get the AR experience out. If you really just want to create an AR experience then Unity would work, but if you want it to be something that people can download and interact with on your phone then you need to have it out on a mobile device… so you could use Xcode specifically getting that out on the Apple Store, but Google just came out with their came out of beta for their ARCore so all the Android devices out there will be catching up too.

Rebecca Can you talk about other ways that augmented reality could be used to help aid the learning experience or any existing apps that you’re aware of that that already start moving forward in that direction?

Renée: Yes. So there is an app… I believe it’s called meta. It’s basically a really easy way for people to create AR experiences without knowing any code, and it’s specifically for educational purposes. It basically uses an application on your phone that you’re already clicking… dragging… all those kinds of things you’re used to doing on a phone to create an AR experience. Part of the hard thing with that is obviously the practicality of it. You’re limited to what you can do but I could see some platform resources where you could just very simply, especially for purposes of education, create a quick experience just to help people learn. Obviously the more immersive your teaching is the faster they’re gonna learn it right? So it’s more hands-on and that’s what their their goal is with that app as well, and it’s free. So that’s great too.

Rebecca So an example of using that platform might be if you’re taking students on a tour and you’re trying to get them to think about what it was like in history… a certain period of time… they could you know aim their phone at a particular location or something, right? …and it it could show a picture of what it looked like at a different time or something like that.

Renée: Yes, yep. Actually they have demos of that. So, yeah, that’s a great example.

John One of our colleagues at Fredonia who gave some workshops here. She hasn’t been on the podcast yet, had students in a Freshman Seminar do a Wikitude layer where they created information about various places on campus… where student reviews of them would pop up on Wikitude.

Renée: Great. Yeah, absolutely… and Yelp is kind of doing a similar thing now as well. So as you’re walking around of course you can see all their reviews right over the restaurants as you’re about to go into them, which could help….

Rebecca …which people who are herd people….

Renée: But yeah… It’s kind of a similar idea or concept, right? …of that immersive information layer that can be really helpful as you’re walking around navigating.

John A lot of apps use at least some level of augmented reality, so a lot of people aren’t really aware they’re doing it when they’re looking at Yelp or when they’re looking when they’re searching for things on maps or other things.

Renée: Yes, actually that’s a big thing when people will say, “I don’t know if I am ready for augmented reality or I don’t know how I feel about that.” Part of my response is a lot of people are already using augmented reality… they just don’t realize it… and actually that’s the best part about technology being used well, is if it’s invisible, right? and you don’t even notice that you’re using something and you wouldn’t even consider that AR because it’s just something that feels so natural, and that’s obviously a goal as a designer for sure.

John One less visual one that deals with sound something you had talked about this morning is the Zombie Run app I think it’s still out there I know some people who use that where you can hear zombies approaching spatially to encourage you to move faster or slower and so forth.

Rebecca That sounds terrifying.

John Well, there you go! But….

Renée: I think that’s what they want.

John I’ve heard some people find it motivating, especially fans of The Walking Dead I think.

Renée: But yes, absolutely… audio is a new component that is definitely going to add to the whole AR experience, right? Anything dealing with the senses and especially with exploration of auditory versus visual and how that sensory processing works. The audio component is very important and needs to be also at the forefront in consideration when you’re thinking about these things… especially with wearable devices. They’re going to be much more integrated in the technology to make the audio very clear in the direction and have control over that while at the same time being able to listen to the sounds in your environment… provides a lot of opportunity.

John And you mentioned that you had just looked at some demos of Bose.

Renée: Yes. I was just at South by Southwest… speaking down there, and got to team up with Bose AR and checked out all their 3d prototypes of their AR sunglasses in their wearables and they have some really cool things going on and looking forward to further conversations with them on that so.

John What might be some other applications of AR software for instructional use? Where assignments could be given and students work with AR materials or develop materials?

Renée: Well, the beauty of AR is that its hands-on and its immersive. So, anywhere that someone could in a normal situation where you wouldn’t necessarily be able to have an object in front of you, or you wouldn’t have an experience that you could have ever experienced, because of location or whatever the case maybe… AR provides that opportunity. So, there are some AR apps out there currently but even like thinking about youths and education… thinking about some of the STEM programs and trying to get people understanding how specific things work and how you could build specific things, I think there’s a huge opportunity for AR to help in a space where you get the information right where you need it.

John So, just-in-time instruction and assistance

Renée: Yes.

John Which is similar to the project you were thinking of working on or you’re planning to work on.

Renée: Yes, yep. In augmented learning. Yup.

Rebecca What are you gonna do next?

Renée: Sleep? [LAUGHTER] I have a couple things on the horizon… the biggest thing has been my augmented learning project. I’m really excited to see how I can implement that into my specific design curriculum and then once I see the benefits or things that need to be changed from that… seeing how that I could then have impact not just specific to design but curriculum and the way we learn and the hands-on learning, which I’m a huge advocate for. So, seeing how that can impact the future but also how we can make it a little bit more approachable and kind of getting over those learning curves of the technology to make it really something that can have impact on the way students learn

Rebecca Do you have collaborators? Do you have people that are gonna help you measure some of that?

Renée: Currently for that I have some grant funding that I’m working on to get me started but my hope is that as I keep working on it more I’ll get some more assistance. That is something I’m looking for: Is people who are interested in collaborating as well as making sure I have all the technology and everything needed to have the most user testing that we can have.

John Great.

Rebecca Well, thanks so much for spending some extra time on campus today with us.

Renée: Yes, thanks for having me, and thanks for the tea.

John If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.