57. Scalar

Imagine an online environment that makes the thought processes of a writer visible, including the loops they get stuck in, the relevant tangents they pursue, and the non-linear way in which their ideas evolve.  Now imagine that all of these features are easy to use and implement in the classroom. In this episode, Fiona Coll, an Assistant Professor of Technology and Literature at SUNY-Oswego, joins us to discuss how Scalar, a free open-source publishing platform, can help achieve these goals.

Show Notes


John: Imagine an online environment that makes the thought processes of a writer visible, including the loops they get stuck in, the relevant tangents they pursue, and the non-linear way in which their ideas evolve. Now imagine that all of these features are easy to use and implement in the classroom. In this episode, we examine how Scalar, a free open-source publishing platform, can help achieve these goals.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

Rebecca: Today our guest is Fiona Coll, an assistant professor of Literature and Technology at SUNY Oswego. Welcome, Fiona.

Fiona: I am very happy to be here, Rebecca.

John: We’re very happy to have you here. Our teas today are…

Fiona: Today I am drinking Cranberry Blood Orange Endless Sunshine tea, which is a very, very ambitious kind of tea by the Republic of Tea, and I just have to note that on the side, it proclaims that drinking this tea will “create social balance one sip at a time.”

Rebecca: So maybe that’s what we should all be drinking right now. [LAUGHTER] I’m drinking English Breakfast.

John: I’m drinking Bing Cherry Green tea.
We invited you here to talk about your work with Scalar. What is Scalar?

Fiona: Scalar is an online publishing platform designed for long-form, media-rich writing. In the words of Scalar’s creators, this means media-rich digital scholarship. It’s an open source platform created by a group called the Alliance for Networking Visual Culture, and the whole idea behind this platform is that it was built to serve scholars who were working on non-traditional, long-form academic writing, specifically projects that might involve visual culture or media culture. There are particular features of Scalar that have been geared towards this use case, but I would like to argue that Scalar is actually a fantastic tool for teaching because of some of its unusual features. Can I tell you about them?

John: Sure.

Rebecca: Yeah, please. [LAUGHTER]

John: We’d be asking anyway. [LAUGHTER]

Fiona: The best way to approach these unusual features is, I think, to describe how you use Scalar, and so I will. The basic unit of content in Scalar is called a page, and it seems fairly unremarkable when I begin talking about it in this way. When you’re creating a Scalar page there’s a text box where you enter a title, there’s a text box where you enter a description, and then a large text entry field where you can put in text and format it. You can choose from a few layout options. You can integrate media into that page. You can enter metadata. You can annotate. You can add comments. So far, so WordPress… fairly straightforward. However, things now get interesting once you create a Scalar page. Once you’ve created a series of Scalar pages, you can start building routes through that content. There are two ways to organize the pages that you create in Scalar: the first is to use tags, which create nonlinear clusters of organized content, or you can use the path feature in Scalar, which is, as it sounds like, a path—a linear, step-by-step progression through a sequence of Scalar pages that you determine. You can get very creative with this path structure; you can create branching paths or very complex forking paths; you can create recursive or looping paths that come back to steps you’ve already been through; you can create rabbit-hole paths that lead people away from the main branch of your content into an unretrievable nether place, but the point is Scalar does not impose any sort of order on the content that you created and indeed that’s why the platform is called Scalar—it comes from this reference to two ways we think of quantifying movement in the world, I suppose: Scalar versus vectors. Vectors are quantities that have both magnitude and direction to them and a scalar quantity is one that has only magnitude and so Scalar, the publishing platform Scalar, does not force you to do any sort of particular relationship between the things that you create. Again… doesn’t sound especially revolutionary, but remember how I mentioned you could add tags and comments and annotations to a piece of Scalar content? When you do that, when you create things like tags and annotations and comments, those all become Scalar pages themselves, and they can participate in this larger set of relationships. So a tag, which is also a page, can be tagged with something else, it can be a path of its own, a comment can be a tag, an annotation can also be a comment, can also be a tag, a path can be a tag on something. So, any piece of content in Scalar can be given any sort of relationship to any other piece of content, and what this means is that there’s a sort of radical, non-hierarchical organization to the way Scalar allows you to approach the products of your own creative work. So this becomes really, really interesting if you imagine what this means for creating something like an essay. We have a long tradition of thinking of an essay as an extraordinarily linear thing that begins at the beginning, that moves through a sequence, and that ends. But Scalar allows us to reimagine what an essay might be, not just what it might contain, so not just moving beyond text but moving beyond that linear structure, and when I first understood just how radically Scalar allowed the breaking down of this old-school essay model, I became very excited to imagine its possibilities in the classroom. So, I learned about Scalar and immediately thought that this would be a fantastic way to defamiliarize the writing process for students, and by “defamiliarize the writing process for students,” what I mean is I thought that this would be a fantastic tool to get students to reimagine the way that their thoughts unfold in writing. I wanted them to reimagine writing as actual making, as actual construction, and not just as a sort of tragic endpoint for a thinking process.

Rebecca: It’s active, basically.

Fiona: It’s active. It is a process; writing is a process and I always say writing is thinking, and students I don’t think quite understand what I mean by that, but what Scalar might allow me to do, I imagined upon first encountering the platform, would be to get students to think about how sections of their thoughts work; how ideas might connect to other ideas… not in linear ways… but in roundabout ways that might meander through other references or images or clips they came across on the internet or things from other classes… that thought is not linear no matter how much we try to get them to package it into straightforward, well-behaved writing.

Rebecca: So this is really exciting, I can imagine writing something like “I have this thought and now I’m in a loop and I can’t get out; I’m cycling through ideas and trying to get myself out and I just can’t, but sometimes that happens when you’re writing and it’s like, oh, this isn’t gonna work; I don’t have a conclusion.

Fiona: This is how the process of writing works; you do get in loops. It is a reiterative experience where you try something out and you might end up back where you started; you try it again, you come back where you started, but perhaps the loop needs to be there for a particular reason but there’s a little exit ramp you might find to some other form of thought and Scalar doesn’t force you to try and pretend that that is not happening, that that complexity is not happening. It allows you to in fact mark the way that your thought is moving and branching in non-linear ways and allows you to capitalize on those threads and those directions. One of the things that Scalar does that very useful in getting students to think about their writing process is that it timestamps every iteration of a particular page and it saves every iteration of a page. So there’s a sense in which students are free to revise or rethink. There’s a sense in which Scalar holds safe and secure all of the versions of their thought, so it works well in terms of allowing them the space to experiment and the space to make mistakes while also giving them a time-stamped chronology of the work that they’ve done. So, there are multiple ways in which Scalar allows for the thinking process to be represented.

Rebecca: It also seems like it’s a good model for students to know how long they’ve spent writing because their idea or conception of how much time they may have spent doing something might be really inaccurate.

Fiona: That’s a fantastic point because I think students do have a strange dislocation from the actual effort it takes, the actual labor that goes into producing something like a polished text. So, on the one hand there’s just an awareness of the sheer time that goes into that, but there’s also a sense in which Scalar allows students to really, really dig into the revision and the editing process, which often is hard. So, students sort of do the standard essay writing and I often find it difficult to convince them to let go of certain aspects of what they’ve written or to radically or drastically revise…

Rebecca: But it’s still there.

Fiona: But it’s still there; they don’t have to worry about losing it—they can try something completely different and perhaps see what happens when they release their hold on that idea that writing is just something you open up a word processor and do… start at the beginning and go until you’ve hit the word limit.

Rebecca: You’ve got pathway one like normal way, then it’s like here’s my cycle weird way, here’s my figure eight way.

Fiona: Yes!

John: When you have students work with this are they working individually or in groups?

Fiona: This is the next thing I wanted to mention, which is that Scalar allows for both. It is extraordinarily flexible in terms of this exact question. I usually begin by having students create content, create Scalar pages on an individual basis, but all of the students are creating within what I call one great big bag of Scalar content. Scalar uses the term “book” to describe one project in this way. So, the students can create their own content, tag it as their own content, organize it according to their own methods, but then I get students to interact with each other’s content, so they read each other’s content, they start to make tendrils of connections between their content and other students’ content, and then eventually I build up to students generating content collaboratively. So it works really well in allowing a wide range of writing collaboration, and the point I make as these networks of connection get more and more elaborate is that this is how knowledge works, this is how knowledge is created; it’s a collective, collaborative enterprise and nobody does best working in isolation.

John: When they do this is it something that’s shared just within a class or is it shared publicly?

Fiona: Again, Scalar has both options available. I discuss this issue of public versus private writing with the students and we usually make a decision together as to whether or not the students want to make their material, their writing, public or private. I’ve also had a class in which one student was very happy to make her Scalar project public, but all the rest of the students wanted to keep theirs private, so she was able to easily take her content, make a whole new Scalar book and proudly display it for everyone to see, so it is remarkably flexible in terms of what it allows you to do with what you create.

Rebecca: What about the converse, though, when it’s a one or two students that have a reason that it needs to be private?

Fiona: And there’s absolutely no problem in keeping a Scalar book entirely private. I also give students the opportunity to erase what they’ve done… so to remove it entirely. We do talk a lot about privacy, public writing, and issues of copyright is another angle that seems important to talk about in terms of the Scalar ecosystem. The group who has built Scalar is deeply invested in promoting open access and fair use of cultural resources as part of their commitment to generating very dynamic and free intellectual exchange, so they created something called the Critical Commons—it has a relation conceptually or figuratively to Creative Commons—but their Critical Commons is a place where copyrighted media is taken and transformed critically and then posted for fair use purposes. So you’ll see people who have taken clips of movies, for example, or a television shows but transform those clips through a critical apparatus that Critical Commons enables, and this allows students to really think about what they’re doing when they reference a piece of culture, whether that’s a photograph or a song or a video, that by adding their critical commentary to it, they are transforming it, they are generating ideas that are making that piece of content new, and Scalar’s link to Critical Commons allows them to really think about issues of copyright, issues of intellectual openness, what happens when something is locked down and is unavailable for access to them to write about, so it becomes a much, much broader discussion about the nature of knowledge, the nature of information in our 21st century.

Rebecca: It sounds like the emphasis then with this Critical Commons is the idea of fair use and understanding fair use and describing fair use and putting in a structure in place that embodies and enforces fair use.

Fiona: And that embodying and enforcing of fair use that you describe then becomes part of how the students think of themselves as creators, so what does it mean to take something that another student has written and to use it in some way in your own thinking? Where do the bounds of fair use lie? It’s often something students haven’t thought about and this actually relates to the labor-related facet of Scalar that I find really useful in terms of student learning. I often feel that students see the Internet as this place where disembodied text has just appeared and exists, but by generating it themselves they have to confront the fact that a lot of work or a lot of effort went into generating the things that they don’t think very much about, and so Scalar allows students to think about the writing process in new and interesting and productive ways, but it also allows students to think about the nature of information that they engage with on a daily basis.

Rebecca: It’s really funny that we’re talking about fair use today because I was talking to my students about fair use this morning. We had a visiting artist who uses fair use in her work and then there was like a thousand questions when she was here. I said, you know, “We’ll talk about fair use, I promise, on Monday, when we all get back and she’s not here and we’re not taking up her time to dig into it.” But, it’s funny because they have this commercial point of view and then also the cultural maker point of view and they conflate it as if it’s all the same and that is really different. Context matters… and that you need to be thinking about these things, so we tried to untangle that today, but you’re right, students don’t think about that at all; in fact, scholars don’t think about it very often either.

Fiona: It’s true, and I first used Scalar in a class that was comparing and contrasting 19th-century book technologies with 21st-century digital writing and publishing technologies and part of the reason that worked the way it did is that 19th century literature is, of course, out of copyright—it’s public domain—and so we were able to play very freely with the literature from that period, and then students had to stop and think and realize that the 21st century, again, literature in various interesting forms, was different, was fundamentally different because of this legal category that we use to distinguish between what is public domain and what isn’t, and students are fascinated by it, while also not understanding it or understanding its logic, necessarily. So, Scalar’s making visible of something that students just hadn’t thought about before is one of its many, many strengths or one of the many valuable ways in which it operates in a classroom.

Rebecca: Can you take us on an adventure through one of your classes to get us a better sense of how you’re actually putting it into play in a specific class with a specific group of students?

John: In terms of maybe the type of assignment that they might be working on?

Fiona: For sure. I first used Scalar in a class that contrasted 19th-century material book production with 21st-century digital publication technologies and I asked the class to really consider the ways in which genre, in particular, is affected by the shape of publishing possibilities. So students are used to thinking about genre as something that is an intellectual idea or an abstract idea informed by author influence or cultural anxieties, but they rarely think about genre as something that is shaped by the actual material affordances of publication, so we read 19th-century texts, we read 21st-century texts and then I asked the students to produce their own creative or critical response to the material in our classes, and what that meant was that some students wrote relatively traditional research essays that incorporated media, sound, video. It meant that some students created choose-your-own-adventure type creative stories that played with the notion of genre as Scalar allowed them to unpack conventions that were and were not possible in that electronic form. Students also used other sorts of technologies to play with the way that technology shaped the kinds of stories they could generate, but that’s a broad overview of how Scalar worked in one particular class. I am using Scalar currently in a class about digital literary studies and the students are making digital editions of 19th-century texts. So, students are in groups, they’ve each been assigned a story by an obscure local Oswego author and they are in groups deciding how they want to present these stories to the world… new and refreshed by their 21st century perspectives on the stories, so some of them are emphasizing maps and timelines. Some of them are emphasizing illustrating the stories. Some of them want to actually remix some of the stories and generate alternate routes through the stories. So, they’re able through Scalar to invent and create these approaches to literary interpretation—they’re making arguments about the text through their use of Scalar, and I should mention that one of Scalar’s appeals is that it’s possible to do a lot with minimal technical knowledge. It’s also possible to do a lot if you have maximal technical knowledge. There’s a lot of room for customization if you are fluent with CSS and that sort of business. It accommodates a really wide range of technical skill.

John: Could you tell us a little bit more about how the choose-your-own-adventure type things work? Is that difficult for the students to program the branches?

Fiona: The great news is the students don’t really have to program the branches. The student I’m thinking of in particular wanted to write a choose your own adventure story that turned into a different genre of story depending on which path you took through her story…

Rebecca: I love that idea.

Fiona: It was a fantastic idea and it really showed just how well she grasped the possibilities that Scalar offered. So she began—there was an introductory page that set up a scenario—it was a mystery, perhaps a murder mystery story at the beginning, and then she had a couple of options: you could choose to follow one character or one event; and as each choice branched a little bit further and a little bit further, so there were many, many iterations of the story, and again each arm of the story took on a slightly different generic set of conventions. It was relatively straightforward; literally in Scalar you simply mark, using a little sort of dialog box that you check or uncheck. you mark what pieces of content you want to attach to a page. So, there’s no encoding, there’s no high-level function that students need to worry about; they can simply imagine what they want to connect and they can make those connections relatively easily. I will say that one of the other things I love about Scalar is that it generates productive difficulty for the students, it generates a lot of intellectual uncertainty which is something that I find… [LAUGHTER] I enjoy producing in students in a constructive way, obviously. Because Scalar is this enormous bag into which students just throw pieces of content, it can get overwhelming really quickly—there can just be this amorphous, chaotic mass that they struggle to make sense of—but that’s part of the advantage, I would argue: it really, really makes them think about high order levels of structure and organization. So even though they can do multiple kinds of organization… even though they can be very creative about how they organize, they do have to really think about how they want their content to relate to one another. So Scalar has this ability to get students thinking at that high level of structure while also allowing students to pay very, very close attention on the level of annotation and close reading—it combines those two levels and sort of everything in between in a way that I find very, very useful for students to be doing. I haven’t even talked about the kind of media annotation that’s possible. But, you can annotate, on a pixel level, images. You can annotate in various time stamps on a video or a piece of audio. There’s an extraordinary level of very, very specific detail that you can attend to as well as dealing with these large high order or large-structure levels of organization.

Rebecca: How did you learn how to use Scalar and then also how do you help students learn how to use the platform?

Fiona: This is a fantastic question. I learned to use Scalar in a very short, informal lunchtime demonstration given by Cathy Kroll—who I believe is at Sonoma State University—at the 2015 Digital Humanities Summer Institute in beautiful Victoria, B.C. and Cathy Kroll simply went through the process of making a Scalar page and she simply explained—and there are all sorts of interesting, cool things you can do with this organizational system—and that was enough; that was enough to allow me to at least discover its possibilities. So, the barrier to entry is low, but then you can ramp up things an awful lot, and I do find that I’m learning more and more as I go. I first imagined using Scalar in my own scholarly work—I am working on this, again, obscure local Oswego author—and I was trying to imagine ways to experiment with bringing these stories back to digital life, but I found that I was almost more excited by the possibilities I was seeing in students and so I thought I would take the exact same approach. I tend to give students a very, very basic introduction to what Scalar can do and then just let them loose, so allow them [LAUGHTER]—again, productive frustration—they make mistakes, they lose pages, they can’t figure out if they’re tagging a page or if they’re making a page a tag. I allow this brief sort of beginning phase of crazy-making exploration and then I ramp up the features, so I introduce more and more features. I begin by, I suppose it’s the carrot versus the stick analogy, so I begin by showing some of the very cool things Scalar can do, so with a basic knowledge of how metadata works students can produce these very gorgeous timelines or maps; I show them how they can use iframes to pull in content from various places on the web and enliven their writing. But, I also then ask them to think very hard about how they’re engaging with other students’ work, and so it feels as though I start with one page and then just allow them to explore on their own while giving them pushes in certain directions to make sure that they are exploring as fully as possible.

John: Maintaining those desirable difficulties as they develop more skills.

Fiona: Maintaining the desirable difficulties, exactly. I’m still trying to figure out how much I should push them, so how much I should demand of the students. I know that other people have used Scalar simply as a writing tool, so just dealing with text and organization. I know that others have encouraged students to make use of the multimedia affordances of Scalar and I’m still figuring out what the balance is for my students who are mostly students of literature.

Rebecca: The first thing that comes to mind to me is how the heck do you grade that? [LAUGHTER] There’s a lot to keep track of and map and pay attention to, so how are you evaluating students in like what criteria and and how do you actually just sort through all of that content?

Fiona: This is a fantastic question and one I am still figuring out… [LAUGHTER] the answer to. As I’m introducing students to Scalar and as I’m letting them make a mess and generate multiple versions of a single page and get confused themselves, I do encourage them to keep in mind that ultimately they want to be imagining not just their own thought process and writing process but what it might feel like for a reader to come across their material, specifically a reader that is me; [LAUGHTER] specifically a reader that will be assigning them a grade. [LAUGHTER] At the very same time, I do try and emphasize process over product, and because students come with such a range of technical capabilities, I build into my rubrics how hard a student has worked to correct a deficiency or to overcome a limitation in their ability to understand Scalar. Ultimately I am interested in the argument that they’re making, but I do reward and encourage what I call bravery—willingness to try new things; willingness to fail; willingness to get things wrong but then to turn that failure into something useful or to meditate or reflect on it in a conscious way. So, there’s a metacognitive aspect to all of this, and essentially in every assignment I’m still trying to figure out what the balance between rigorous analysis and explorative risk-taking might be. I tend to err on the side of appreciating the risk-taking, I will say that.

Rebecca: So do they submit like a URL to you?

Fiona: No, what happens is they tell me where they want me to enter their work. I usually create an index page and I ask them to put their starting point on that index page, so they’re all contributing to one page that serves as my starting point and that’s the easiest way to wander through things. I can go hunting if I need to. I encourage the students to tag what they’re doing with their own names. If there’s a good search function, for example, if I’m looking for something that’s been lost. It definitely feels like hunting in a barn full of hay sometimes. That’s not quite the same as hunting in a haystack but it’s not quite not the same either. [LAUGHTER]

John: Have any other faculty in your department or on campus adopted Scalar yet?

Fiona: I don’t think anyone else in my department has adopted Scalar. I do think as my classes perhaps turn more towards public facing projects that might change, because I do think there are a number of faculty and approaches that could do very cool things with Scalar, but so far I have had to pull my examples from elsewhere… from other campuses. But hopefully soon there will be some robust Oswego examples.

John: Have you ever had students build upon the work of earlier classes?

Fiona: I have not, and that is something that I’m trying to figure out how to do successfully, for a couple of reasons. So it would be easy to do if I just kept one giant Scalar project and had students continually reiterate upon the work that had come before; I haven’t actually repeated a course yet that I’ve used Scalar in, so that in fact might be a next step for my work with Scalar—it would involve, of course, getting permission from the students to do this or to allow them to anonymize their work, but those are things we could work out—but I have not yet done so. I could imagine the Digital Archiving Project as being one that would lend itself towards that sort of semester after semester continuation.

John: How have students responded to this compared to more traditional writing classes?

Fiona: The great news is that students seem very, very excited by what feels like them to be freedom. They respond really well to the autonomy that Scalar offers them. They tend to respond in a slightly opposite direction when they realize that freedom comes at a price and that price is an awful lot of work and figuring out technical details, and some students truly do flounder—some students just find it absolutely maddening to try and understand what’s happening. But some students absolutely thrive and really run with the creative remixing possibilities and really embrace the radically democratic approach that Scalar allows them to take to their own writing and writing in groups. So, I would say that there’s a now predictable sort of curve: initial excitement as students think about the possibilities, then there’s an inevitable drop in enthusiasm as the students realize just how much work this involves and how much new thinking they have to do to wrap their minds around the defamiliarization that Scalar offers, and then perhaps two tails after that: one very enthusiastic skyrocketing of competence and then one more medium flavored… just sort of making peace with what I’ve asked them to do, and I do always offer students options, and if someone just feels absolutely unable to grapple with Scalar there’s always the possibility of doing a different sort of project, but I haven’t yet had a student who has completely resisted.

John: This is a nice follow-up to our earlier podcast with Robin DeRosa where she talked about open pedagogy and it seems like this would be a nice tool for students to create materials that can be widely shared, if they choose to.

Fiona: If they choose to, and I do think I’m gonna bring the concept of open pedagogy or open ed more and more explicitly into classes in which I use Scalar to make that a part of my justification, or something else to get students thinking about. It’s a growing and very exciting movement—the open pedagogy and open education movement—and I’m excited to see how Scalar can continue to be a part of that.

Rebecca: Does Scalar offer, by default, a way to license individual pieces of content using Creative Commons or is it more how you would traditionally license a website by copying and pasting the code from Creative Commons, for example, on an individual page?

John: Or is it just a Critical Commons option?

Fiona: That is a fantastic question. I think you would need to attach your own Creative Commons licensing; I don’t know that there’s a built-in feature. However, I should say that that doesn’t mean it doesn’t exist; it just means I don’t know about it at the moment, but I do think you, again, get to choose your own approach to that very issue. but I’m gonna look into it and see if I can figure out if there’s a built-in tool or aspect of Scalar.

Rebecca: We can make a note of that in our show notes, too, afterwards.

Fiona: I will follow up. [LAUGHTER]

John: And if we find any links we’ll include them.

Rebecca: ..[If] people wanted to get started, do you have a couple of examples that you might recommend for people to look at?

Fiona: I definitely have a few examples that I can recommend. I can add those to the show links, perhaps, and there are examples that range from student projects through elaborate library-based projects to very beautiful, customized versions of Scalar projects. I’d be very, very happy to share them and encourage people to try out the platform.

John: We always end our podcast with a question, what are you going to do next?

Fiona: To this point I have used Scalar in upper division literature courses where students come to the course already equipped with a certain set of writing and thinking skills that I can leverage and encouraging the curiosity and bravery I mentioned. So, next semester I’m gonna try using Scalar with a first-year composition course, and so I’m in the planning stages right now to see how that particular experiment might unfold.

Rebecca: That sounds really exciting.

Fiona: I’m super excited about it. As you might be able to tell, I really, really, am fascinated by the ways in which Scalar seems to activate student curiosity and student agency in their own intellectual work.

John: And if you reach freshmen with this they might perhaps suggest it to some other faculty as something they may wish to try.

Fiona: I like it, I like it as a plan.

Rebecca: Sounds like we’ll have to do a follow-up.

Fiona: I am here for it. [LAUGHTER]

John: Well thank you. This has been fascinating.

Rebecca: You’ve piqued my curiosity; I’m gonna go explore, so I can’t wait for those extra links so I can find a way in.

Fiona: If I’ve piqued your curiosity, I believe I have done my job.

John: And I did create an account a couple years ago when you gave a workshop and I kept meaning to go back, but now I’m more likely to. [LAUGHTER]

Fiona: Well let me know how you find it; let me know what you discover.

John: Thank you.

Rebecca: Yeah, thank you very much.

Fiona: Thank you so much; it was wonderful to talk with you.

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kim Fischer, Brittany Jones, Gabriella Perez, Joseph Santarelli-Hansen and Dante Perez.

52. Metaliteracy

Do your students create digital media in your courses or just consume it? Does the concept of information literacy seem too limited in this context? In this episode, Tom Mackey (Professor in the Department of Arts and Media at Empire State College) and Trudi Jacobson (Head of the Information Literacy Department and Distinguished Librarian at the State University of New York at Albany) join us to discuss metaliteracy as a framework for improving critical thinking and metacognition while students become active participants in the construction of knowledge in online communities.

Show Notes


Rebecca: Do your students create digital media in your courses or just consume it? Does the concept of information literacy seem too limited in this context? In this episode we discuss metaliteracy as a framework for improving critical thinking and metacognition while students become active participants in the construction of knowledge in online communities.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guests today are Tom Mackey, Professor in the Department of Arts and Media at Empire State College and Trudi Jacobson, the Head of the Information Literacy Department and Distinguished Librarian at the State University of New York at Albany. In fact, she is currently the only Distinguished Librarian in the SUNY system. Welcome, Tom and Trudi.

Trudi: Thank you.

Tom: Thank you. Very happy to be here. Thanks for the invitation.

Rebecca: Today our teas are…

Trudi: I am drinking highland blend.

Tom: I’m drinking sweet cinnamon spice.

John: And I am drinking chocolate mint tea, a Harney and Sons blend.

Rebecca: I’m back to my good ole English afternoon.

John: Such a surprise.

Rebecca: Sometimes you just need to have the dependable option.

John: Both of you have written very extensively and done a lot of research and workshops on metaliteracy, with three books, three MOOCs (with a fourth one under development), several articles, a badging system and the metaliteracy.org blog. Could you tell our listeners a bit about what metaliteracy is?

Tom: Sure. Thanks, John, I’ll start. Metaliteracy is a pedagogical framework that empowers learners to be active producers of information in collaborative environments. So that’s our elevator speech right there in terms of what it is. Basically, it is an approach to teaching and learning that prepares individuals to be reflective learners in addition to being critical thinkers, and we’ll talk a bit about how that reflection piece is especially critical for a metaliteracy, which, of course, applies metacognition. By doing so, learners are informed consumers of information, which means they ask good questions about the information they encounter in a variety of environments, and as you know, that’s important today with all the different environments and social media environments and access to different kinds of new sources that we have; it includes those especially mediated by technology. And we’ll talk as the idea was first introduced and developed why that was so important to the concept. When we first introduced it we really argued that because of the emergence of social media, online communities especially, think about web 2.0 and the change from the original web—what a critical moment that was—that what we really needed was a metaliteracy that promoted effective participation in these environments. As we know, these collaborative social environments have an engagement piece that is important; we build that into this metaliteracy framework; we thought there was a real need for that—how we were developing it. We also acknowledge that in addition to acquiring information and looking critically at information that individuals really needed to creatively create and share information in this connected network world. This idea of consuming information versus producing information, it’s an idea that’s been around for some time, but we really thought it was critical to develop it into a metaliteracy that also focused on reflection as a core concept. The idea of a metaliteracy is that we look at some of the common characteristics that unite different forms of literacy—that was the other piece of this. We introduced it as this comprehensive, unifying framework. The idea for that was that in this social media environment what we really needed was to try to better understand different competencies, different characteristics of literacies instead of just coming up with a new literacy every time there’s a new technology. We were trying to look at things in a more comprehensive way. As the idea developed in the first book, especially the meta in metaliteracy, intentionally invokes this idea of metacognition. Or thinking about your own thinking: this is really key to metaliteracy because metaliterate learners are reflective about their own learning experiences and they really take charge of their literacy and learning which is really where the empowerment piece comes in.

Trudi: Meta derived from the Greek… also means “after.” Metaliteracy is what happens after literacy. Basic reading and writing, what comes after that. Also what comes after information literacy, which is g enerally thought of as finding and locating information. The definition of information literacy has expanded since we started work on metaliteracy. In addition to reflecting on their own thinking, the metacognitive aspect of metaliteracy also means that individuals have the capacity to self-regulate their own learning, which means that they identify their own strengths and weaknesses and play a role in preparing themselves to adapt to new learning situations. Metaliteracy prepares learners to adapt new technology and to do so in a critical way, that is asking questions about how technologies are designed and the ways that technologies or platforms may impact how we access and create information as well as how we communicate with information. Originally we developed metaliteracy to emphasize how individuals participate in social media environments. And Tom, would you like to talk a little bit about that?

Tom: That piece is really essential to what we’re doing. We see this framework is relevant to a range of collaborative teaching and learning situations, but it is interesting that we saw a real need for emphasizing the social media aspect, online communities, this post web 2.0 environment that we are in, but we also don’t want it to be limited to that. We really see metaliteracy in all environments, all collaborative environments… communities of practice. This is something we should be thinking about beyond just the technology, but really how we engage with each other, how we participate and perhaps also how we blend the technology, how we mediate technology with those spaces as well.

Trudi: One of my favorite parts of metaliteracy is that it advances the idea that learners are teachers. We see this in collaborative environments where learners support and teach other learners, but what’s really important is that often students, for example, don’t think they have any particular expertise in something, and encouraging them to empower to teach others often leads to really interesting situations.

 Tom: That part is so key and that’s something that we saw in our own teaching experiences that when we had students in collaborative situations… group work… building technology tools together… building collaborative websites, for example, that the students themselves were as much a teacher as I was, and trying to foreground that so that they can see it, is critically important.

Rebecca: This is a really interesting framework and you’ve given us a lot to think about. Can you help us make it a little more concrete by providing an example of how you might emphasize metaliteracy in a class or what you mean by a student who might be metaliterate?

Trudi: One of the things that I would do in my classes is encourage students to be information creators and to explore the technology in doing so. So they don’t have a final paper that they have to write, but they may need to create a video or a tutorial or we’ll be talking about our badging system later, maybe creating content for that and doing it in small groups. If they’re doing something where they have to use a technology; I don’t teach them that technology; they sort of learn together and that “learner as teacher” often comes out in those situations because often there’ll be a student in a group who will have more expertise in that area or be more willing to just jump in and see what happens, and then the rest of the group will learn from that. One of the more interesting teams that I had when I’ve taught is one where none of the students felt they could do anything, but they actually accomplished it and their sense of pride and empowerment in doing that was wonderful.

Tom: I have an example: I’m currently teaching a course at Empire State College called “Digital Storytelling,” and the whole point of the course is that students learn about these resources, they locate them (with some prompts from me in the course), but it’s a fully online course and in many ways they have to figure this out on their own, they have to adapt to these new technologies, and I think that they’re looking at their own use of technology in a different way. So, for example, the very first assignment they have to create a selfie video with their cell phone. So they all have cell phones, they probably all done videos before, they probably all done selfies before, but this assignment is really designed for them to introduce themselves to everyone else in the class in a fully online course. From the very beginning they have to challenge themselves to present themselves a certain way to the class… to be themselves but to also think through that presentation, to really be the active producer of information in a collaborative setting where they’re doing something on their own but they’re sharing something about themselves to the other class. In an online course it allows us to get beyond just the text-based introduction and online discussion and to really seeing the students, to hearing from them. I posted a video of myself and it was great to see their response, so it was very much like a classroom situation but it happened asynchronously and online and it was a great way to get the class started, so from the very beginning they saw themselves as digital storytellers and they know that they now are starting their story and that we’re all going to participate and learn from them.

John: So it’s encouraging students not just to critically analyze information as consumers but to be active participants in social dialogues as producers as well. Is that a reasonable short summary?

Trudi: Yeah.

Tom: Absolutely. And what does that mean? …especially in today’s environment, which is very participatory but were divided and partisan in so many different ways. How do we get across those divides? What does it mean to be a responsible participant of information now? What does it mean to be an ethical contributor to these spaces? The whole idea is to really to get them to reflect on this, and not just to produce and share something, but now especially to think about the implications of that so that the informed consumer part is still important so that they’re thinking about these different sources that they’re encountering but also thinking about what they’re creating themselves and sharing.

Trudi: I think when they’re asked to be information producers in this way they think about themselves differently. They create information and share it on social media, but they don’t really think of themselves as information producers, and so I think it expands their horizon.

Tom: They may not have necessarily been asked to do so in an academic environment. This blurring of boundaries between informal learning and formal learning, I think it helps to push that a little bit. Not to say that they’re not beyond our classes, because they might be, but clearly they’re doing it in their everyday practice with their cell phones and the way they consume information now, but this really foregrounds, I think, in some of what the responsibilities are and what the empowerment of that is as well when they’re asked to construct something, so instead of a research paper maybe that is a collaborative media project with their peers—what kind of learning do you gain from that experience?

Trudi: Just one other point. The projects that I was talking about, they need to create them for public consumption. It’s not something that’s just directed at me as the professor of the course. They have to think about it a bit differently.

Tom: That’s a great point, because in the digital storytelling class they’re not just creating it even for the Moodle environment that we’re in; they have to actually upload their selfie videos to YouTube so that they’re thinking a bit about that public consumption piece even beyond the instructor and even beyond the class itself because now it’s up on YouTube and hopefully that’s having an impact on what they’re thinking about in terms of how they present themselves in the information that they’re producing.

Rebecca: I’m hearing two key things bubble up in what you’re talking about and one is audience and the second is reflection. Are those two key things to move up beyond traditional information literacy to this metaliteracy level?

Trudi: I think that those are two key pieces, but I think, well, there’s the old definition of information literacy and then there’s the newer one, which somewhat influenced by metaliteracy, but I think that often information literacy is thought of primarily as consuming and evaluating information, so not the responsible, creative production of it. It’s also too often, I think, seen in the academic setting as just related to research and not sort of life-wide. I think that that’s another element here.

Tom: In many ways that’s what I think we were really originally working against that original information literacy definition, the ALA definition and also the Association of College and Research Libraries, the original standards, b       which were very prescriptive in the way that they were designed, so that we were as a framework were really just trying to open this up and also take into account the technology piece—not make it all about technology, certainly, but in many ways the advance of web 2.0 and emerging technologies was kind of being, at the time, anyway, sort of avoided. We knew that there’s a real shift happening in our culture and I think that we’re sort of on the other side of that now, but I think that was important to bring that into the learning experience to have students really reflect on those environments and what they’re doing in those environments.

John: You both mentioned the new ACRL information literacy framework. How does metaliteracy relate to that?

Trudi: We developed metaliteracy in part because of a frustration, with this old definition as we were talking about and Tom mentioned the standards really were very prescriptive, very skills based, concentrated on behavioral and cognitive learning domains. Metacognition was not a part of it, so you identified metacognition so that reflection as something new and they didn’t explicitly address the affordances of web 2.0. So I was co-chair of the task force that was convened by the Association of College and Research Libraries and I brought the idea of metaliteracy to the group for consideration. There were a lot of forces at work in developing the structure of the framework and there were like 2000 people weighing in so it’s a very interesting process. Threshold concepts or core concepts was one of the primary features that we used with the framework. I sort of quote from the introduction to the frameworks; there are those ideas in any discipline that are passageways or portals to enlarged understanding or ways of thinking and practicing within that discipline. For example, in biology, evolution would be a threshold concept. That was one element and then the other was metaliteracy. The idea of learners as information creators as well as consumers—which we’ve talked about—definitely has a presence in the framework. There are four learning domains in metaliteracies: behavioral, cognitive, affective and metacognitive. These all have made their way into the framework, so there really is in part a close relationship between the two. For example, the affective domain maps to the whole sections on learner dispositions. I think that there really is a close relationship and I think metaliteracy has gotten additional notice from people because it is explicitly mentioned in the framework.

John: So it’s complementary that they fit well together, they link well together.

Trudi: That’s right.

Tom: I think that’s a good way to put it that they’re complementary, because that also allows each approach to still move forward because we see metaliteracy as this evolving concept and we’ve been working together—we’re working with a team of colleagues called the “Metaliteracy Learning Collaborative” on these ideas, we’re writing together and we’re developing this different MOOC and badging projects. Every time we do something new we’re learning something new and we’re trying to build that into the core ideas here. I think that this idea of complementarity is really important to these two; they’re not mutually exclusive, they work together, and as Trudi mentioned, when we go out and talk to different audiences on this they’re interested in both concepts and working with both. One interesting comment we often hear from people is that with metaliteracy they’ll say you found a way to talk about something that we were trying to do or that we were already doing but you found a name for that really made sense. We really like that: the fact that we were able to name something that really probably was in practice but maybe didn’t have as in-depth of a framework built around it and we like that dialogue with practitioners and something we try to do so this idea of theory and practice for metaliteracy is critically important and allows you to move forward.

Trudi: And the ACRL information literacy framework information literacy is not something that can be taught only by librarians so it’s really directed also towards faculty and administrators. It still seems to have a librarian focus to it, whereas metaliteracy, I think, extends beyond that. Librarians are interested in it but we’re also seeing all sorts of things that are being written or talked about by academics in a really broad range of disciplines.

Tom: And we’ve found that in the books we’ll talk about the two unedited books we’ve done in addition to the first metaliteracy book and we saw evidence of that when we do a call for proposal; it’s really from a wide area of academics. We definitely have librarians, but we also have faculty from many different disciplines, and also instructional designers. That piece of it has been really fascinating as well because we’ve been trying to really open it up to as many people as possible and not seeing it just within one particular discipline.

John: How have faculty and librarians responded to your work?

Trudi: There’s been a lot of interest in it to explore one of the collaborations. Somebody that I’m working with at the University at Albany is a political science professor. This will give you an indication how at least one person has responded to our work. She teaches a 200 level political science course that includes some of the general education competencies, one of which is information literacy, and she was developing this course from Pollock. She came to me to talk about information literacy. We ended up talking about metaliteracy and she was so excited by some of the things we’ve talked about that it would do for her students, so this idea of information creators, the empowerment that she has made metaliteracy sort of a key part of her course. She has the students do about 8 activities connected to metaliteracy. These activities come from a digital badging system that we can talk about a little bit later. She actually has students create an activity that would fit into this digital badging system, which is pretty exciting. This year she asked us to extend what we’re doing and we have been creating questions for the students about what it means to be an information creator, information producer, a teacher, a translator of information and we found this very exciting. It’s not just a collaboration in that she is using some of this material for her students, but her students are creating things for us and she’s giving us ideas. It’s just one example but it’s one where it has become a core part of this course, not only when she teaches it but when others teach it as well.

Tom: Collaboration has been key to what we’ve been doing from the very beginning. The first SUNY IITG we received was really to initiate to launch a metaliteracy learning collaborative and that first project led to the development of our first connectivist MOOC… b eginnings of the digital badging system, although it wasn’t part of the initial grant, but that’s something that we started working on, and also what was most important at the time was the development of the first metaliteracy goals and learning objectives which we’ve recently revised but it was important when we developed that that instead of just Trudi and I working on this together, we really opened it to a SUNY-wide audience that included faculty and librarians. Those goals and learning objectives are available via metaliteracy.org and we recently revised them as well. I think that collaboration with the metaliteracy learning collaborative also led to thinking about metaliteracy in a different way and thinking about those four domains of learning that Trudi mentioned previously; we would look at the metacognitive, which we’ve mentioned is key but also the behavioral, the cognitive and the affective domain so that what we’re really looking at is really the whole person. We’ve also through the metaliteracy learning collaborative we’ve been working on papers together, we’ve been working on these MOOCs; we were lucky enough to have the experience of working on a connectivist MOOC really early on and then I took Coursera MOOC and then a Canvas MOOC and now we’re working on open edX and all those projects involve faculty librarians from Empire State College, the University of Albany and other parts of SUNY, that’s really key. We’re very lucky that we’ve been invited to speak on this which also shows the level of interest and how people are responding to it and many different venues and last year we were lucky enough to present at a conference at the University of Guadalajara in Mexico for this literacy and learning conference and it was just a great experience to be there with international scholars who were talking about literacy in various ways and then we added something by talking about metaliteracy and there’s a lot of interest in what we were talking about. We appreciate those opportunities to have conversations that are both theoretical and practical; the response has just been really positive.

John: We should just clarify the IITG program you mentioned is a SUNY-wide competitive grant program for all of the colleges and universities within SUNY. You were one of the early recipients of that and have received some further funding from that, just to explain that to our listeners who are not as familiar with the SUNY system.

Trudi: John, since you mentioned the innovative instruction technology grant, just to show sort of interest from others, we did get one with School of Education faculty member, actually one from Albany and one from Empire State College because they were really interested in the digital badging, but also the idea of a digital citizen. The plan was and happened that graduate students in education who were going to be teachers would have an opportunity to learn about digital citizenship that’s important for them when they’re teaching, also what digital badging is, so there were a couple of different takeaways. We were able to move metaliteracy or an aspect of metaliteracy into graduate education for educators.

Rebecca: There’s been a lot of mention of metaliteracy badges so maybe we can talk about those?


Trudi: Yeah, certainly. This was something that developed out of one of those innovative instruction technology grants. We’ve been working on them ever since. What we did was we took the learning goals and objectives for metaliteracy and created open content, very ambitious scheme. There’s four digital badges in the system. Each one of which has anywhere from 12 to 20 activities, starting with lower level quests, moving up to challenges and ultimately you get to these four digital badges. They were written by members of the meta literacy learning collaborative. Tom has written some, I’ve written some. Students have written some, so undergraduate and graduate students they’re being used currently at Albany about 2,500 students have gone through parts of this badging system. The only ones so far who’ve actually earned badges are ones who have taken my courses. It’s content that can be used in classes across a range of disciplines. Also, adaptable to the disciplines. I mentioned earlier the political science professor and sometimes she sort of tweaks the assignments in there so it really relates to what she’s teaching in her political science course. The badge system itself at this point is restricted to University of Albany because there’s a single sign-on process, but we do have a website that has all of the content openly available. People are welcome to use this.

Tom: And from the perspective of someone who has developed some content for this it’s really a fascinating experience because you know that you’re somehow reaching learners that are not in your course but that it’s something that you’re opening and you’re sharing, so this idea of thinking about them as open educational resources that can be then adapted for different contexts. It’s really interesting and exciting to know that something I might create as a learning object could be used by a faculty member here at the University at Albany who’s having their students go through it. Some of them that I developed are based on learning activities I had created in some of my information science courses when I taught here at the university, but I’ve adapted them or updated them. That piece of it from a faculty perspective, as long as you’re open to it, is really engaging and interesting and a way to reach other learners who may not be students in your class but you’re sharing those ideas with them.

Trudi: And I don’t know if it’s ok if I plug a book that I just co-edited with Kelsey O’Brien… Just published this month, September 2018, Teaching with Digital Badges, which was published by Rowman & Littlefield. In that book there is a chapter written by Kelsey O’Brien on the metaliteracy badging system.

Rebecca: Great, you’re both working on a new book together, right?

Trudi: Yes.

Tom: Yes.

Rebecca: Can you talk a little bit about that new book and how it connects to your earlier work?

Tom: Sure, the new book is called “Metaliterate Learning for the Post Truth World.” We’ve shifted somewhat from I think what was a really optimistic view of the connected world and how great it is to be producers of information and be participatory to really trying to further emphasize some pieces that were there but I think needed to be fleshed out a bit more for the new environment we’re in post truth, which is based on confirmation bias and misinformation, false information and questions about new sources and all kinds of misleading facts that are being sent out. We really wanted to take that head-on because we saw metaliteracy in many ways even though it’s an idea that had developed previously as something that is a strong education response to some of the concerns and issues that we’re seeing today. Soon after the 2016 election we wrote a piece about fake news and that term is certainly changed even from the time that we originally wrote it. Wrote a piece for the conversation called “How to Reject Fake News in a Digital World,” so again taking a metaliteracy approach to looking at fake news in a critical way. Since that time even the term fake news, of course, has been weaponized, so we have conflicting thoughts about even using that term based on the research some educators think that it’s important to still keep using it and others want to reject it completely but I think we all generally know the narrative of that. The new book we decided to foreground metaliteracy in this environment and to make it an edited book so that we could engage other educators about this idea. Wasn’t just us but that it was other educators who were dealing with it. About half of the book is very theoretical and the other half of the book is more practical. When we did a call for proposals we tried to intentionally keep that open because we wanted different perspectives on this. I wrote the framing chapter to really talk about post truth, to reframe metaliteracy within this context and to also talk about a new figure that Trudi and I developed together based on the metaliteracy learning characteristics. The new book is going to present a new image, a new figure that further develops the metaliteracy idea from a theoretical perspective and talk about the importance of those characteristics in the post truth world. We’re joined by incredibly prestigious authors who from a theoretical standpoint look at things such as the importance of documentation in metaliteracy, and again, what they’re doing is they’re flushing out pieces of metaliteracy that we have not engaged with yet, so it was really exciting to see that. Another author talks about inoculation theory preparing learners to in many ways be resistant to some of the post truth issues that we’re currently engaged in. Scientific literacy, so there’s a whole chapter on the importance of scientific literacy and looking at it through the lens of metaliteracy. Also, looking at the synergy of word and image and photojournalism, Tom Palmer who teaches here in the journalism program at the University of Albany and it was also a journalist who works for the Times Union wrote that chapter. A few of the chapters do deal with the ACRL framework for information literacy for higher education, so we had that perspective. We were talking previously about both concepts are complementary and we have a few authors who really prove that. We also have a few authors who look at such topics as teaching students to be wrong, genre writing in the first year, writing instruction and the application of poetic ethnography in digital storytelling to create narratives in Philadelphia neighborhoods. I’m very interested in digital storytelling. I mentioned that previously and one of our authors also talks about digital storytelling to empower voices and to encourage students to really raise their voice in the current times that we’re in.

Trudi: And earlier you sort of asked how faculty, other educators, librarians have responded to metaliteracy. I think it’s really interesting. Tom and I did a workshop on metaliteracy at Temple University and a couple of these chapters actually came from people who were in that workshop. It was really sort of exciting to see the immediate impact that that had had.

Rebecca: That’s cool. So this sounds like a really great book; when can I get it?

Trudi: Next spring. [LAUGHTER]

John: Ok. Your current MOOC is a Coursera MOOC but you’re developing a new open edX MOOC. Could you tell us a little bit about that and how that new MOOC will differ from the prior MOOCs, because you’ve had more than one in the past?

Tom: This is part of a continuum of those three MOOCs. We actually wrote a paper in Open Praxis that talked about metaliteracy as a pedagogical framework that was applied in these different MOOCs, so we did a kind of compare and contrast of the different MOOC environments but also talked about our experiences and those different platforms and what it was like, and at the end of that paper one of our conclusions was that what we really needed to do next was create a kind of hybrid MOOC environment because what we had previously was the connectivist MOOC which was our first one and that Stephen Downes type approach. We actually used his grasshopper programming to run that MOOC, then we had the more structured xMOOCS, the Coursera and the Canvas. In many ways the paper was about that but what we decided at the end of the paper was we analyzed it was that we needed a hybrid version and it would it be possible to do that, is there a platform out there that has the learner-centered freeform approach of the connectivist MOOC with some of the structures that were valuable in the video that was really key to the xMOOCS. One of the ideas that propelled this idea forward… We also then, of course, had this shift to this thinking of a transition of kind of a connected world to a post truth world and what does that mean, and because we were working on this book “Metaliterate Learning for the Post Truth World,” we thought that’s a theme for a MOOC. We won’t go out there and call it the metaliteracy MOOC, but it’s a post truth MOOC that’s powered by metaliteracy that really applies the metaliteracy framework to each of the modules within the MOOC, so we’re really excited about that. We did apply for another SUNY IITG and we did receive funding for that, which allows us now to build a team—again it’s another Empire State College, University at Albany team—and we’re really excited about it, we’re developing it now, we’re exploring the open edX environment and as part of that too we’re working with the University of Buffalo because they’ve just launched an instance of open edX for their continuing education program and so they’ve already done a lot of the analysis and a lot of the footwork in terms of creating this instance of open edX on their campus, so they’re letting us experiment with what they’ve done and the idea is that our experience as one of the first two SUNY institutions beyond UV that are using open edX that we will hopefully pave the way for other SUNY faculty librarians that want to develop an open edX MOOC.

Trudi: One of the things that we’d like to do with this—Tom mentioned earlier—we’ve recently revised the metaliteracy learning goals and objectives. We are using those as the framework for this new MOOC. We would like to address issues such as confirmation bias, the role of expertise and authority in today’s environment, issues related to safety, security and personal privacy online, representations of reality in a virtual world and all the while sort of empowering participants to raise and share their voices while rebuilding communities of trust.

Rebecca: Who do you see is the audience for this particular MOOC?

Trudi: I think that we’re really hoping that it’s a very broad audience. We’ve had that, for example, with the Coursera MOOC where there were a lot of international participants everywhere from high school students to non-traditional types of students. We learned about their professions which just ran the gamut and I think that although we do hope to introduce this MOOC as part of courses both at Empire State College and at the University at Albany we’re really hoping that the participants are traditional learners and non-traditional learners. I think that what we’re going to be including in the way of content is something that needs to be broadly disseminated.

Tom: I think because that’s one of the advantages of MOOCs is that they do open up a potentially global audience, so we’re hoping for that international perspective as well, and as Trudi mentioned, we are developing courses so that we could on each of our campuses—I’m calling them wraparound courses—so that the courses that introduce students to the MOOC and they can then earn credit for doing so, because that’s been one of the big questions about MOOCs; can you learn credit, so what we’re doing is creating separate courses and in my version of the course I’m doing a full semester course so that the first half of the course will be introducing students to, well, what is a MOOC? What is post truth? What is metaliteracy? And I have a whole section on how to prepare for success in taking a MOOC, and then that will hopefully prepare them to be a successful learner in a MOOC environment so then they’ll take the six-week course and then there will be reflection piece at the end, which is very metaliteracy, and I actually think that a course about a course is very meta, so we’ve got that piece of it, and that idea to emerge from our very first connectivist experience where we tried to do it for credit and sure, you can talk about this experience at the University at Albany. In particular, in many ways the students were not prepared for the connectivist environment, so what we’re trying to do is in mind, since mine will be a full semester course, is invite students to take it but to really prepare them for being successful in MOOC because we know too that completion rates and MOOCs are not always great, but what if you offer it and prepare students for that environment. I think it is unique enough of an environment where that’s worth exploring.

Trudi: And Tom referred to our connectivist MOOC, which I did use as part of a course, essentially a blended course, and I was amazed when the students actually asked for more in-person class meetings because they couldn’t really grasp the idea of the MOOC and the fact that they were making decisions about their own learning. They were making decisions about which readings would be important. They needed to participate through a personal blog that was sort of elected and shared, and what they essentially did was doubt. I had about a 60% dropout rate in the course and the ones who were left were the ones who just wanted their hands held essentially through the rest of the course and that’s where we really learned that what Tom is going to be doing with his course, which is a full semester course, mine will be a quarter course again, is preparing them for this. This MOOC will be a more directed connectivist MOOC, but it was a very important takeaway.

Tom: And I’m hoping that by doing that it prepares them not only for our MOOC but it opens up the possibility of picking other MOOCs for lifelong learning. So that I think there are potential benefits, even beyond this experience. We’re hoping to launch the MOOC,—we’re developing it now—but we’re hoping to launch it for March 2019. It will be called “Empowering Yourself in a Post Truth World,” which is really important because we really want it to be a positive learning experience and one that provides resources for learners to be successful. You can imagine that talking about the post truth world could be a real downer, but what we really want it to be is a real positive focus of how to address the issues, look at these issues critically, but then to leave with some concrete ways of dealing with it. It also builds on some of the other MOOCs we had. The Coursera MOOC, for example, involved empowering yourself in a connected world and we’re running that now as an on-demand version. So when we first ran it in Coursera we were in the course and it was moving along and we were there in the discussions and following it but then Coursera changed its format a little bit and open up this possibility of on-demand and we actually like that because it allows us to have that content out there and to have learners engage with it in a self-paced way. Up to this point we’ve had, based on the stats we continue to receive from Coursera;—it’s running all the time—we’ve had 1,900 registrants and 900 active learners. We were really happy about that because it really gets some of these concepts out there, and I think it’s probably it’s been out there for a couple years now; it’s probably due for a revision, but that’s one of our projects that we’d like to do eventually, but I think that the post truth MOOC will in many ways build on that as well, so if someone wanted to go back they could look at that on-demand version, but as Trudi mentioned, the post truth MOOC is a six module, six-week learning experience on a very specific topic. I think it will be even more of a clearly-defined focused than even the other one.

John: Would be really nice to have all voters taking in the next couple of years. [LAUGHTER]

Trudi: We would like that.

Tom: Yes, yes.

Rebecca: So you’ve talked a lot about the learner side and some of the tools and materials and MOOCs and things that can help learners become more metaliterate. How do you help faculty coach students through this kind of process? What are the takeaways for faculty? They’ve listened to this episode, they’re really interested in the idea; where do they get started?

Trudi: I think not to just promote our books, but I think that perhaps if they took a look at the two edited volumes they might get a sense of how others are doing it and the range of disciplines is pretty broad, so they might find someone in their own or a related one. I think that that might be a good place to start. I think also taking a look at the learning goals and objectives might provide some ideas of things they’re already doing, but perhaps finding ways to highlight them or frame them slightly differently.

Tom: And not to promote our blog, but metaliteracy.org; everything is in there, including the goals and learning objectives. Summaries of all the books, because we’ve had the blog now for a few years, so it’s interesting even to kind of go back and look at some of the original postings, but it links to the books, it links to all the presentations. The presentations are available, and a few of the keynotes that were recorded are in there. I do think the metaliteracy goals and learning objectives are definitely key because those can be easily applied. Should we mention what we were just invited to write because that would actually address this audience as well?

Trudi: Yeah, we’re going to be writing a piece for higher education jobs. They have a couple of newsletters and going to be talking about the importance of teaching or emphasizing metaliteracy on campus for administrators and also what instructors can do. We think that those are going to be appearing in November.

Tom: Because we’ve had a commitment to making everything open—I know it’s a lot to look for, but we do have the metaliteracy YouTube channel, the blog, of course, the presentations and a lot of these resources were intentionally constructed that way so that other educators could use them, so go to “Empowering Yourself in a Connected World” on Coursera and access the videos, use the learning activities in any way you want. Go to the first module; there’s a PDF in there that has the metaliteracy learner roles and we’ve used them as learning activities in our own classes and it has some reflective questions, so you have this diagram that really explains the different roles a learner could take and then it has questions for learners to really think about those roles. So I think a lot of those resources can be adapted in any way that people want, and it’s really an open concept, so we want people to get involved and apply their own approaches to this.

Rebecca: We wrap up by always asking: what next? You’ve given us so much, but what else? [LAUGHTER]

Tom: That’s a really good question. The next book that we mentioned is coming out in the spring. We’re currently working on the open edX MOOC, “Empowering Yourself in a Post Truth World.” We also, of course, will be launching that in the spring.

Trudi: With the digital badging system we would like to if we can find some more funding have a learning pathway portion to it where instructors can really tailor the information or add components for their own disciplines. We’re also working on a metaliteracy module for another innovative instruction technology grant funded project called “I succeed,” which is being developed in western New York, and they’ve asked us to provide a module on metaliteracy and this is going to be directed to high school students who aspire to college or first year college students and can be used by instructors, so we are putting that together with four units.

Tom: We have a few upcoming panel presentations that OLC accelerate in Florida in November.

John: I may see you there.

Tom: Oh, great! I haven’t been there in a couple years so I’m looking forward to getting back and that’s such a great conference.

John: It is.

Tom: And of course there will be continued research and writing. I’m certain that the open edX experience that we’re currently immersed in will lead to a paper, and we’d like to do a research project that assesses the application of the metaliteracy goals and learning objectives. So much of what we’ve been doing is really theorizing and talking about practice and developing these environments, but we would like to delve into that a bit more. We might have an opportunity to work with an international scholar that we met last year at the University of Guadalajara, but we’re not sure about that if that’s going to happen, but that would allow us to really expand the metaliteracy concept: working with international scholars. So there’s a lot of possibilities. Perhaps a coil courses in our future, and that’s another SUNY resource; it’s a collaborative online international learning environment. I think that’s something that we would love to do with an international scholar, so we’ll see if that happens some day. A lot of ideas, got a lot going on, but we’ll see.

John: You got a nice track record of being really productive with us.

Rebecca: Thanks so much for joining us and spending time and giving us lots of things to think about.

John: Yeah, you’re doing some wonderful work.

Trudi:Thank you.

Tom: Thank you so much, we really enjoyed this.

Trudi: Yeah.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

37. Evidence is Trending

Faculty are increasingly looking to research on teaching and learning to make informed decisions about their practice as a teacher and the policies their institutions put into place. In today’s episode, Michelle Miller joins us to discuss recent research that will likely shape the future of higher education.

Michelle is Director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general interest publications.

Show Notes

Rebecca: Faculty are increasingly looking to research on teaching and learning to make informed decisions about their practice as a teacher and the policies their institutions put into place. In today’s episode we talk to a cognitive psychologist about recent research that will likely shape the future of higher education.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guest today is Michelle Miller. Michelle is Director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She is the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general interest publications.
Welcome, Michelle!

Michelle: Hi, I’m so glad to be here.

Rebecca: Thanks for joining us.
Today’s teas are:

Michelle: I’m drinking a fresh peppermint infused tea, and it’s my favorite afternoon pick-me-up.

Rebecca: …and it looks like it’s in a really wonderfully designed teapot.

Michelle: Well, thank you… and this is a thrift store find… one of my favorite things to do. Yeah, so I’m enjoying it.

John: I have Twinings Blackcurrant Breeze.

Rebecca: …and I’m drinking chai today.

Michelle: Pretty rough.

John: We invited you here to talk a little bit about things that you’ve been observing in terms of what’s catching on in higher education in terms of new and interesting innovations in teaching.

Michelle: Right, that’s one of things that I really had the luxury of being able to step back and look at over this last semester and over this last spring when I was on sabbatical… One of the really neat things about my book Minds Online, especially now that it’s been out for a few years, is that it does open up all these opportunities to speak with really engaged faculty and others, such as: instructional designers, librarians, academic leadership, educational technology coordinators… all these individuals around the country who are really, really involved in these issues. It’s a great opportunity to see how these trends, how these ideas, how these innovations are rolling out, and these can be some things that have been around for quite some time and just continue to rock along and even pickup steam, and some newer things that are on the horizon.

John: You’ve been doing quite a bit of traveling. You just got back from China recently, I believe.

Michelle: I sure did. It was a short visit and I do hope to go back, both to keep getting involved in educational innovations there and, hopefully, as a tourist as well. So, I was not there for very long but I had the opportunity to speak at Tsinghua University in Beijing, which is a really dynamic institution that’s been around for about a hundred years. For a while in its history it specialized in things like engineering education polytechnic, but now it’s really a selective comprehensive university with very vibrant graduate and undergraduate programs that are really very relatable for those of us in the United States working in similar contexts. My invitation was to be one of the featured speakers at the Future Education, Future Learning Conference, which was a very interdisciplinary gathering of doctoral students, faculty, even others from the community, who were all interested in the intersection of things like technology, online learning, MOOCs even, and educational research (including research into the brain and cognitive psychology), and bringing all of those together… and it was a multilingual conference. I do not speak Chinese but much of the conference was in both English and Chinese and so I was also able to really absorb a lot of these new ideas. So yes, that was a real highlight of my sabbatical semester and one that I’m going to be thinking about for quite some time.

I should say that part of what tied in there as well is that Minds Online, I’ve just learned, is going to be translated into Chinese and that’s going to come out in May 2019. So, I also got to meet with some of the people who were involved in the translation… start to put together some promotional materials such as videos and things like that.

Rebecca: Cool.

John: Excellent.

Rebecca: So, you’ve had a good opportunity, as you’ve been traveling, to almost do a scavenger hunt of what faculty are doing with evidence-based practices related to your book. Can you share some of what you’ve found or heard?

Michelle: This theme of evidence-based practice, and really tying into the findings that have been coming out of cognitive psychology for quite some time, that really is one of the exciting trends and things that I was really excited to see and hear for so many different quarters I visited in different institutions… and so I would say definitely, this is a trend that is continuing and is increasing. There really does continue to be a lot of wonderful interest and wonderful activity around these real cognitively informed approaches to teaching, and what I think we could call scientifically based and evidence-based strategies. One form this has taken is Josh Eyler’s new book, called How Humans Learn: The Science and Stories behind Effective College Teaching. This is a brand new book by a faculty development professional, and a person coming out of the humanities, actually, who’s weaving together even from his humanities background everything from evolutionary biology to classical research in early childhood education to the latest brain-based research. He’s weaving this together into this new book for faculty. So, that’s one of the things that I’ve noticed and then there’s the issue which i think is another great illustration of best-known practice which is the testing effect and retrieval practice.

John: One of the nice things is how so many branches of research are converging… testing in the classroom, brain-based research, and so forth, are all finding those same basic effects. It’s nice to see such robust results, which we don’t always see in all research in all disciplines.

Rebecca: …and just breaking down the silos in general. The things are all related and finding out what those relationships are… exploring those relationships… is really important and it’s nice to see that it’s starting to open up.

John: We should also note that when you visited here, we had a reading group and we had faculty working on trying to apply some of these concepts, and they’re still doing that… and they still keep making references back to your visit. So, it’s had quite a big impact on our campus.

Michelle: This wasn’t true, I don’t think, when I first entered the teaching profession… and even to the extent when I first started getting interested in applied work in course redesign and in faculty professional development. you would get kind of this pushback or just strange looks when you said “Oh, how about we bring in something from cognitive psychology” and now that is just highly normalized and something that people are really speaking across the curriculum… and taking it and running with it in a lasting ongoing way, not just as a “Oh, well that was an interesting idea. I’m going to keep doing what I’m doing” but really people making some deep changes as you mentioned. This theme of breaking down silos… I mean I think if there’s kind of one umbrella trend that all of these things fits under it’s that breakdown of boundaries. So, that’s one that I keep coming back to, I know, in my work.

So, the idea of retrieval practice, drilling down on that one key finding which goes back a very long ways in cognitive psychology. I think of that as such a good example of what we’re talking about here… about how this very detailed effect in cognition and yet it does have these applications across disciplinary silos. Now when I go to conferences and I say “Okay, raise your hand. How many people have ever heard of retrieval practice? How many people have ever heard of the testing effect? How many people have heard of the book Make it Stick (which really places this phenomena at its center)?” and I’m seeing more hands raising.

With retrieval practice, by the way, we’re talking about that principle that taking a test on something, that retrieving something from memory actively, has this huge impact on future memorability of that information. As its proponents like to say, tests are not neutral from a memory or from a learning standpoint… and while some of the research has focused on very kind of stripped-down laboratory style tasks like memorizing words pairs, there are also some other research projects showing that it does flow out to more realistic learning situations.

So, more people simply know about this, and that’s really the first hurdle, oftentimes, with getting this involved disciplinary sometimes jargon riddled research out there to practitioners and getting it into their hands. So, people heard of it and they’re starting to build this into their teaching. As I’ve traveled around I love to hear some of the specific examples and to see it as well crop up in scholarship of teaching and learning.

Just recently, for example, I ran across and really got into the work of Bruce Kirchhoff who is at University of North Carolina – Greensboro and his area is botany and plant identification. He has actually put together some different really technology-based apps and tools that students and teachers can use in something like a botany course to rehearse and review plant identification. He says in one of his articles, for example, that there just isn’t time in class to really adequately master plant identification. It’s just too complex of a perceptual and cognitive and memory test to do that. So, he really built in from the get-go very specific principles drawn from cognitive psychology… so, the testing effect is in there… there’s different varieties of quizzing and it all is about just getting students to retrieve and identify example after example. It brings in also principles such as interleaving, which we could return to in a little bit, but has to do with the sequencing of different examples… their spacing… So, that’s even planned out exactly how and when students encounter different things that they’re studying. It’s really wonderful. So, for example he and his colleagues put out a scholarship of teaching and learning article talking about how this approach was used effectively in veterinary medicine students who have to learn to identify poisonous plants that they’ll see around their practice. This is something that can be time-consuming and very tough, but they have some good data showing that this technology enhanced cognitively based approach really does work. That’s one example. Coincidentally, I’ve seen some other work in the literature, also on plant identification, where the instructors tagged plants in an arboretum… they went around and tagged them with QR codes… that students can walk up to a plant in the real environment with an iPad… hold the iPad over it… and it would immediately start producing quiz questions that were are specific to exactly the plants they were looking at.
So, those are some of the exciting things that people are taking and running with now that this principle is out there.

Rebecca: What I really love about the two stories that you just shared was the faculty are really designing their curriculum and designing the learning experiences with the students in mind… and what students need and when they need it. So, not only is it employing these cognitive science principles, but it’s actually applying design principles as well. It’s really designing for a user experience and thinking about the idea that if I need to identify a plant, being able to identify it in this situation in which I would need to identify it in makes it much more dynamic I think for a student… but also really meets them where they’re at and where they need it.

John: …and there’s so many apps out there now that will do the plant identification just from imagery without the QR code, that I can see it taking it one step further where they can do it in the wild without having that… so they can build it in for plants that are in the region without needing to encode that specifically for the application.

Michelle: I think you’re absolutely right once we put the technology in the hands of faculties who, as I said, they’re the one to know: “Where are my students at? Where are the weak points? Where are the gaps that they really need to bridge?” and that’s where their creativity is giving rise to all these new applications… and sometimes these can be low-tech as well… or also things that we can put in a face-to-face environment… and I’d like to to share just some experiences that I’ve had with this over the last few semesters.

In addition to trying to teach online with a lot of technology, I also have in my teaching rotation a small required course in research methods in psychology which can be a real stumbling block… the big challenge course… it’s kind of a gateway course to continued progress in our major. So, in this research methods course, some of the things that I’ve done around assessment and testing to really try again to stretch that retrieval practice idea… to make assessments really a more dynamic part of the course and more central part of the course… to move away from that idea that tests are just this kind of every now and again this panic mode opportunity for me to kind of measure in sorts of students and judge them… to make good on that idea that tests are part of learning. So, here’s some of the things that I try to do. For one thing, I took time out of the class almost every single class meeting as part of the routine to have students first of all generate quiz questions out of their textbook. So, we do have a certain amount of foundational material in that course as well as a project and a whole lot of other stuff is going on. So they need to get that foundational stuff.

Every Tuesday they would come in and they knew their routine: you get index cards and you crack your textbook and you generate for me three quiz questions. Everybody does it. I’m not policing whether you read the chapter or not. It’s active… they’re generating it… and also that makes it something like frequent quizzing. That’s a great practical advantage for me since I’m not writing everything. They would turn those in and I would select some of my favorites I would turn those into a traditional looking paper quiz and hand that out on Thursday. I said “Hey, take this like a realistic quiz.” I had explained to them that quizzes can really boost their learning, so that was the justification for spending time on it and then I said: “You know what? I’m not going to grade it either. You take it home because this is a learning experience for you. It’s a learning activity.” so we did that every single week as those students got into that routine.

The second thing that I did to really re-envision how assessment testing and quizzing worked in this particular course, was something inspired by different kinds of group testing and exam wrapper activities I’ve seen, particularly coming out of the STEM field, where there’s been a lot of innovation in this area. What I would do is… we had these high stakes exams at a few points during the semester. But, the class day after the exam, we didn’t do the traditional “Let’s go over the exam.” [LAUGHTER] That’s kind of deadly dull, and it just tends to generate a lot of pushback from students… and as we know from the research, simply reviewing… passing your eyes over the information… is not going to do much to advance your learning. So, what I would do is… I would photocopy all those exams, so it has a secure copy. They were not graded. I would not look at this before we did this… and I would pass everybody’s exams back to them along with a blank copy of that same exam. I assigned them to small groups and I said “Okay, here’s your job. Go back over this exam, fill it out as perfectly as you can as a group, and to make it interesting I said I will grade that exam as well, the one you do with your group, and anything you get over 90% gets added to everybody’s grade. This time it was open book, it was open Google, it was everything except you can’t ask me questions. So, you have each other and that’s where these great conversations started to happen. The things that we always want students to say. So, I would eavesdrop and hear students say “Oh, well you know what, I think on this question she was really talking about validity because reliability is this other thing…” and they’d have a deep conversation about it. I’m still kind of going back through the numbers to see what are the impacts of learning? Are there any trends that I can identify? But, I will say this: in the semesters that I did this, I didn’t have a single question ever come back to me along the lines of “Well, this question was unclear. I didn’t understand it. I think I was graded unfairly.” it really did shut all that down and again extended the learning that I feel students got out of that. Now it meant a big sacrifice of class time, but I feel strongly enough about these principles that I’m always going to do this in one form or another anytime I can can in face-to-face classes.

Rebecca: This sounds really familiar, John.

John: I’ve just done the same, or something remarkably similar, this semester, in my econometrics class which is very similar to the psych research methods class. I actually picked it up following a discussion with Doug McKee. He actually was doing it this semester too. He had a podcast episode on it. It sounded so exciting, I did something… a little bit different. I actually graded it but I didn’t give it back to them because I wanted to see what they had the most trouble with, and then I was going to have them only answer the ones in a group that they struggled with… and it turned out that that was pretty much all them anyway. So, it’s very similar to what you did except I gave them a weighted average of their original grade and the group grade and all except one person improved and the one person’s score went down by two points because the group grade was just slightly lower… but he did extremely well and he wasn’t that confident. The benefits to them of that peer explanation and explaining was just tremendous and it was so much more fun for them and for me and, as you said, it just completely wiped out all those things like “Well, that was tricky” because when they hear their peers explaining it to them the students were much more likely to respond by saying “Oh yeah, I remember that now” and it was a wonderful experience and I’m gonna do that everywhere I can.

In fact. I was talking about it with my TA just this morning here at Duke and we’re planning to do something like that in our classes here at TIP this summer, which i think is somewhat familiar to you from earlier in your academic career.

Michelle: That is right we do have this connection. I was among, not the very first year, but I believe the second cohort of Talent Identification Program students who came in, I guess you would call it now, middle school (back then, it was called junior high) and what a life-transforming experience. We’ve had even more opportunities to talk about the development of all these educational ideas through that experience.

John: That two-stage exam is wonderful and it’s so much more positive… because it didn’t really take, in my class, much more time, because I would have spent most of that class period going over the exam and problems they had. But the students who did well would have been bored and not paying much attention to it; the students who did poorly would just be depressed and upset that they did so poorly… and here, they were actively processing the information and it was so positive.

Michelle: That’s a big shift. We really have to step back and acknowledge that, I think. that is a huge shift in how we look at assessment, and how we think about the use of class time… and it’s not just “Oh my gosh, I have to use every minute to put such content in front of the students.” Just the fact that more of us are making that leap, I think, really is evidence this progress is happening… and we see also a lot of raised consciousness around issues such as learning styles. That’s another one that, when I go out and speak to faculty audiences, 10 years ago you would get these shocked looks or even very indignant commentary when you say “Ok, this idea of learning styles, in the sense that say there are visual learners, auditory learners, what I call sensory learning styles (VAK is another name it sometimes goes by). The idea that that just holds no water from a cognitive point of view…” People were not good with that, and now when I mentioned that at a conference, I get the knowing nods and even a few groans… people like “Oh, yeah. we get that. Now, K-12, which I want to acknowledge it’s not my area, but I’m constantly reminded by people across the spectrum that it’s a very different story in K-12. So, setting that aside… but this is what I’m seeing… that faculty are realizing… they’re saying “Oh, this is what the evidence says…” and maybe they even take the time to look at some of the really great thinkers and writers who put together the facts on this. They say “You know what? I’m not going to take my limited time and resources and spend that on this matching to styles when the styles can’t even be accurately diagnosed and are of no use in a learning situation. So, that’s another area of real progress.

Rebecca: What I am hearing is not just progress here in terms of cognitive science, but a real shift towards really thinking about how students learn and designing for that rather than something that would sound more like a penalty for grade like “Oh, did you achieve? Yes or no…” but, rather here’s an opportunity if you didn’t achieve to now actually learn it… and recognize that you haven’t learned it, even though it might seem really familiar.

John: Going back to that point about learning styles. It is spreading in colleges. I wish it was true at all the departments at our institution, but it’s getting there gradually… and whenever people bring it up, we generally remind them that there’s a whole body of research on this and I’ll send them references but what’s really troubling is in my classes the last couple years now, I’ve been using this metacognitive cafe discussion forum to focus on student learning… and one of the week’s discussions is on learning styles and generally about 95 percent of the students who are freshmen or sophomores (typically) come in with a strong belief in learning styles… where they’ve been tested multiple times in elementary or middle school… they’ve been told what their learning styles are… they’ve been told they can only learn that way… It discourages them from trying to learn in other ways and it does a lot of damage… and I hope we eventually reach out further so that it just goes away throughout the educational system.

Rebecca: You’ve worked in your classes, Michelle, haven’t you to help students understand the science of learning and use that to help students understand the methods and things that you’re doing>

Michelle: Yes, I have. I’ve done this in a couple of different ways. Now, partly, I get a little bit of a free pass in some of my teaching because I’m teaching the introduction to psychology or I’m teaching research methods where I just happen to sneak in as the research example will be some work on say attention or distraction or the testing effect. So, I get to do it in those ways covertly. I’ve also had the chance, although it’s not on my current teaching rotation… I’ve had the chance to also take it on as in freestanding courses. As many institutions are doing these days… it’s another trend… and what Northern Arizona University, where I work, has different kinds of freshmen or first-year student offering for courses they can take, not in a specific disciplinary area, but that really crossed some different areas of the student success or even wellbeing. So, I taught a class for awhile called Maximizing Brain Power that was about a lot of these different topics. Not just the kind of very generic study skills tip… “get a good night’s sleep…” that kind of thing… but really some again more evidence-based things that we can tell students and you can really kind of market it… and I think that we do sometimes have to play marketers to say “Hey, I’m going to give you some inside information here. This is sort of gonna be your secret weapon. So, let me tell you what the research has found.”

So, those are some of the things that I share with students… as well as when the right moment arises, say after an exam or before their first round of small stakes assessments, where they’re taking a lot of quizzes… to really explain the difference between this and high stakes or standardized tests they may have taken in the past. So, I do it on a continuing basis. I try to weave it into the disciplinary aspect and I do it in these free-standing ways as well… and I think here’s another area where I’m seeing this take hold in some different places… which is to have these free-standing resources that also just live outside of a traditional class that people can even incorporate into their courses… if say cognitive psychology or learning science isn’t their area… that they can bring in, because faculty really do care about these things. We just don’t always have the means to bring them in in as many ways as we would like.

John: …and your Attention Matters project was an example of that wasn’t it? Could you tell us a little bit about that?

Michelle: Oh, I’d love to… and you know this connects to what it seems to be kind of an evergreen topic in the teaching and learning community these days, which is the role of distracted students… and I know this past year there just have been these one op-ed versus another. There’s been some really good blog posts by some people I really like to follow in the teaching and learning community such as Kevin Gannon talking about “Okay, do you have laptops in the classroom? and what happens when you do?” and so I don’t think that this is just a fad that’s going away. This is something that the people do continue to care about, and this is where the attention matters project comes in.

This was something that we conceptualized and put together a couple years ago at Northern Arizona University with myself, and primarily I collaborated with a wonderful instructional designer who also teaches a great deal… John Doherty. So, how this came about is I was seeing all the information on distraction… I’m really getting into this as a cognitive psychologist and going “Wow, students need to know that if they’re texting five friends and watching a video in their class. It’s not going to happen for them.” I was really concerned about “What can I actually do to change students minds?” So, my way of doing this was to go around giving guests presentations in every classes where people would let me burn an hour of their class time… and not a very scalable model… and John Doherty respectfully sat through one my presentations on this and then he approached me and said “Look, you know, we could make a module and put this online… and it could be an open access within the institution module, so that anybody at my school can just click in and they’re signed up. We could put this together. We could use some really great instructional design principles and we could just see what happens… and I bet more people would take that if it were done in that format. We did this with no resources. We just were passionate about the project and that’s what we did. We had no grant backing or anything. We got behind it. So, what this is is about a one- to two-hour module that, it’s a lot like a MOOC in that it there’s not a whole lot of interaction or feedback, but there are discussion forums and it’s very self-paced in that way… so one- to two-hour mini MOOCs that really puts at the forefront demonstrations and activities… so we don’t try to convince students about problems with distraction and multitasking… we don’t try to address that just by laying a bunch of research articles on them… I think that’s great if this were a psychology course, but it’s not. So, we come at it by linking them out to videos, for example, that we were able to choose, that we feel really demonstrate in some memorable ways what gets by us when we aren’t paying attention… and we also give students some research-based tips on how to set a behavioral plan and stick to it… because just like with so many areas of life, just knowing that something is bad for you is not enough to really change your behavior and get you not to do that thing. so we have students talking about their own plans and what they do when, say, they’re having a boring moment in class, or they’re really really tempted to go online while they’re doing homework at home. What kinds of resolutions can they set or what kind of conditions can make that that will help them accomplish that. Things like the software blockers… you set a timer on your computer and it can lock you out of problematic sites… or we learned about a great app called Pocket Points where you actually earn spendable coupon points for keeping your phone off during certain hours. This is students talking to students about things that really concern them and really concern us all because I think a lot of us struggle with that.

So, we try to do that… and the bigger frame for this as well is this is, I feel, a life skill for the 21st century… thinking about how technology is going to be an asset to you and not detract from what you accomplish in your life. What a great time to be reflecting on that, when you’re in this early college career. so that’s what we try to do with the project…and we’ve had over a thousand students come through. They oftentimes earn extra credit. Our faculty are great about offering small amounts of extra credit for completing this and we’re just starting to roll out some research showing some of the impacts… and showing it in a bigger way just how you can go about setting up something like this.

Rebecca: I like that the focus seems to be on helping students with a life skill rather than using technology is just a blame or an excuse. We’re in control of our own behaviors and taking ownership over our behaviors is important rather than just kind of object blaming.

Michelle: So, looking at future trends, I would like to see more faculty looking at it in the way that you just described, Rebecca, as this is a life skill and it’s something that we collaborate on with our students… not lay down the law… because, after all, students are in online environments where we’re not there policing that and they do need to go out into work environments and further study and things like that. So, that’s what I feel is the best value. For faculty who are looking at this, if they don’t want to do… or don’t have the means to do something really formal like our Attention Matters approach, just thinking about it ahead of time… I think nobody can afford to ignore this issue anymore and whether you go the route of “No tech in my classroom” or “We’re going to use the technology in my classroom“ or something in between… just reading over, in a very mindful way, not just the opinion pieces, but hopefully also a bit of the research, I think, can help faculty as they go in to deal with this… and really to look at it in another way, just to be honest, we also have to consider how much of this is driven by our egos as teachers and how much of it is driven by a real concern for student learning and those student life skills. I think that’s where we can really take this on effectively and make some progress when we are de-emphasizing that ego aspect and making sure that it really is about the students.

John: We should note there’s a really nice chapter in this book called Minds Online: Teaching Effectively with Technology that deals with these types of issues. It was one of the chapters that got our faculty particularly interested in these issues… on to what extent technology should be used in the classroom… and to what extent it serves as a distraction.

Michelle: I think that really speaks to another thing which I think is an enduring trend… which is the emphasis on really supporting the whole student in success and what we’ve come to call academic persistence… kind of a big umbrella term that has to do with, not just succeeding in a given class, but also being retained… coming back after the first year. As many leaders in higher education point out, this is as a financial issue. As someone pointed out, it does cost a lot less to hang on to the students you have instead of recruiting more students to replace ones who are lost. This is, of course, yet another really big shift in mindset of our own, because after all we did used to measure our success by “Hey, I flunked this many students out of this course” or” Look at how many people have to switch into different majors…our major is so challenging…”

So, we really have turned that thinking around and this does include faculty now. I think that we did used to see those silos. We had that very narrow view of “I’m here to convey content. I’m here to be an expert in this discipline, and that’s what I’m gonna do…” and sure, we want to think about things like do students have learning skills? Do they have metacognition? Are they happy and socially connected at the school? Are they likely to be retained so that we can have this robust university environment?

We had people for that, right? It used to be somebody else’s job… student services or upper administration. They were the ones who heard about that and now I think on both sides we really are changing our vision. More and more forward-thinking faculty are saying “You know what? Besides being a disciplinary expert, I want to become at least conversant with learning science. I want to become at least conversant with the science of academic persistence…” There is a robust early literature on this and that’s something that we’ve been working on at NAU over this past year as well… kind of an exciting newer project that I like very much. We’ve started to engage faculty in a new faculty development program called Persistence Scholars and this is there to really speak to people’s academic and evidence-based side, as well as get them to engage in some perspective-taking around things like the challenges that students face and what it is like to be a student at our institution. We do some really selected readings in the area we look at things like mindset… belongingness… these are really hot areas in that science of persistence… in that emerging field. But, we have to look at it in a really integrated way.

It’s easy for people to say just go to a workshop on mindset and that’s a nice concept, but we wanted to think about it in this bigger picture… really know what are some of the strengths of that and why? Where do these concepts come from? What’s the evidence? That’s something that I think is another real trend and I think as well we will see more academic leaders and people in staff and support roles all over universities needing to know more about learning science. There are still some misconceptions that persist, as we’ve talked about. We’re making progress in getting rid of some of these myths around learning, but I will say… I’m not gonna name any names… but, every now and again I will hear from somebody who says “Oh well, we need to match student learning styles” or “Digital natives think differently, don’t you know?” and I have to wonder whether that’s a great thing. I mean, these are oftentimes individuals that have the power to set the agenda for learning all over a campus. Faculty need to be in the retention arena and I think that leaders need to be in the learning science arena. The boundaries is breaking down and it’s about time.

Rebecca: One of the things that I thought was really exciting with the reading groups that we’ve been having on our campus… that we started with your book, but then we’ve read Make it Stick and Small Teaching since… is that a lot of administrators in a lot of different kinds of roles engaged with us in those reading groups, it wasn’t just faculty. There was a mix of faculty, staff, and some administrators, and I think that that was really exciting. For people who don’t have the luxury of being in your persistence scholar program, what would you recommend they read to get started to learn more about the science of persistence?

Michelle: I really, even after working with this for quite some time, I loved the core text that we have in that program, which is Completing College by Vincent Tinto. It’s just got a great combination of passionate and very direct writing style. So, there’s no ambiguity, there’s not a whole lot of “on the one hand this and on the other hand that.” It’s got an absolutely stellar research base, which faculty of course appreciate… and it has a great deal of concrete examples. So, in that book they talk about “okay, what does it mean to give really good support to first semester college students? What does that look like?” and they’ll go out and they’ll cite very specific “Here’s a school and here’s what they’re doing… here’s what their program looks like… here’s another example that looks very different but gets at the same thing.” So, that’s one of the things that really speak to our faculty… that they really appreciated and enjoyed.

I think that as well we tested good feedback about work that’s come out of the David Yeager and his research group on belongingness and lay theories, and lay theories is maybe a counterintuitive term for kind of a body of ideas about what students believe about academic success and why some people are successful and others are not and how those beliefs can be changed sometimes through relatively simple interventions and when it happens we see great effects such as the narrowing of achievement gaps among students who have more privilege or less privileged backgrounds… and that’s something that, philosophically, many faculty really really care about but they’ve never had the chance to really learn “Okay, how can I actually address something like that with what I’m doing in my classroom, and how can I really know that the things that I’m choosing do have that great evidence base…”

John: …and I think that whole issue is more important now and is very much a social justice issue because, with the rate of increase we’ve seen in college cost inflation, people who start college and don’t finish it are saddled with an awfully high burden of debt. The rate of return to a college degree is the highest that we’ve ever seen and college graduates end up not only getting paid a lot more but they end up with more comfortable jobs and so forth… and if we really want to move people out of poverty and try to reduce income inequality, getting more people into higher education and successfully completing higher education is a really important issue. I’m glad to see that your institution is doing this so heavily and I know a lot of SUNY schools have been hiring Student Success specialists. At our institution they’ve been very actively involved in the reading group, so that message is spreading and I think some of them started with your book and then moved to each of the others. So, they are working with students in trying to help the students who are struggling the most with evidence-based practices …and I think that’s becoming more and more common and it’s a wonderful thing.

Rebecca: So, I really liked Michelle that you were talking about faculty getting involved in retention and this idea of helping students develop persistence skills, and also administrators learning more about evidence-based practices. There’s these grassroots movements happening in both of these areas. Can you talk about some of the other grassroots movements that are working toward, or efforts that faculty are making to engage students and capture their attention and their excitement for education?

Michelle: Right, and here I think a neat thing to think about too is just it’s the big ambitious projects… the big textbook replacement projects or the artificial intelligence informed adaptive learning systems… those are the things that get a lot of the press and end up in The Chronicle of Higher Education that we read about… But, outside of that, there is this very vibrant community and grassroots led scene of developing different technologies and approaches. So, it really goes back for a while. I mean, the MERLOT database that I do talk about in Minds Online has been trove for years of well hidden gems that take on one thing in a discipline and come at it from a way that’s not just great from a subject-matter perspective but brings up the new creative approaches. In the MERLOT database, for example, there’s a great tutorial on statistical significance and the interrelationship between statistical significance and issues like simple sizes. You know, that’s a tough one for students, but it has a little animation involving a horse and a rider that really turns it into something that’s very visual… that’s very tangible… and it really actually tying into analogies, which is a well-known cognitive process that can support the advancement of learning something new. There is something on fluid pressures in the body that was treated for nursing students by nurses, and it’s got an analogy of a soaker hose that this is really fun and is actually interactive. So, those are the kinds of things. The PhET project, P-h-E-T which comes out of University of Colorado, that has been around for a while… again, faculty-led and a way to have these very useful interactive simulations for concepts in physics and chemistry. So, that’s one. CogLab, that’s an auxiliary product that I’ve used for some time in like hundred psychology courses that simulates very famous experimental paradigms which are notoriously difficult to describe on stage for cognitive psychology students. That started out many years ago as a project that very much has this flavor of “We have this need in our classroom. We need something interactive. There’s nothing out there. Let’s see what we can build.” It has since then picked up and turned into a commercial product, but that’s the type of thing that I’m seeing out there.

Another thing that you’ll definitely hear about if you’re circulating and hearing about the latest project is virtual reality for education. So, with this it seems like, unlike just a few years ago, almost everywhere you visit you’re going to hear that “Oh, we’ve just set up a facility. We’re trying out some new things.” This is something that I also heard about when I was talking to people when I was over in China. So, this is an international phenomenon. It’s going to pick up steam and definitely go some places.

What also strikes me about that is just how many different projects there are. Just when you’re worried that you’re going to be scooped because somebody else is going to get there first with their virtual reality project you realize you’re doing very very different things. So, I’ve seen, for example, it used in a medical application to increase empathy among medical students… and I took a six or seven minute demonstration that just was really heart-rending, simulating the patient experience with a particular set of sensory disorders… and at Northern Arizona University we have a lab that is just going full-steam in coming up with educational applications such as interactive organic chemistry tutorial that is is just fascinating. We actually completed a pilot project and are planning to gear up a much larger study next semester looking at the impacts of this. So, this is really taking off for sure.

But, I think there are some caveats here. We still really need some basic research on this… not just what should we be setting up and what the impacts are but how does this even work? In particular, what I would like to research in the future, or at least see some research on, is what kinds of students… what sort of student profile… really gets the most out of virtual reality for education. Because amidst all the very breathless press that’s going on about this now and all the excitement, we do have to remember this is a very, very labor intensive type of resource to set up. You’re not just going to go home and throw something together for the next week. It takes a team to build these things and to complete them as well. If you have, say, a 300 student chemistry course (which is not atypical at all… these large courses), you’re not going to just have all of them spend hours and hours and hours doing this even with a fairly large facility. It’s a very hands-on thing to guide them through this process, to provide the tech support, and everything else.

So, I think really knowing how we can best target our efforts in this area, so that we can build the absolute best, with the resources we have, and maybe even target and ask the students who are most likely to benefit. I think those are some of the things that we just need to know about this. So, it’s exciting for somebody like me who’s in the research area. I see this as a wonderful open opportunity… but those are some of the real crossroads we’re at with virtual reality right now.

Rebecca: I can imagine there’s a big weighing that would have to happen in terms of expense and time and resources needed to startup versus what that might be saving in the long run. I can imagine if it’s a safety thing that you want to do a virtual reality experience, like saving people’s lives and making sure that they’re not going to be in danger as they practice particular skills, could be a really good investment in these… spending the resources to make that investment… or if it’s a lot of travel that would just be way too expensive to bring a bunch of students to a particular location… but you could virtually… it seems like it would be worth the start-up costs and those are just two ideas off the top of my head where it would make sense to bend all of that resource and time.

John: …and equipment will get cheaper. Right now, it’s really expensive for computers that have sufficient speed and graphics processing capability and the headsets are expensive, but they will come down in price, but as you said, it’s still one person typically and one device… so it doesn’t scale quite as well as a lot of other tools or at least not at this stage.

Rebecca: From what I remember, Michelle, you wrote a blog post about [a] virtual reality experience that you had. Can you share that experience, and maybe what stuck with you from that experience?

Michelle: Right, so I had the opportunity, just as I was getting to collaborate with our incredible team at the immersive virtual reality lab at NAU… one of the things I was treated to was about an hour and a half in the virtual reality setup that they have to explore some of the things that they had… Giovanni Castillo, by the way, is creative director of the lab and he’s the one who was so patient with me through all this. We tried a couple of different things and of course there’s such a huge variety of different things that you can do.
There’s a few things out there like driving simulators that are kind of educational… they’re kind of an entertainment… but he was just trying to give me, first of all, just a view of those… and I had to reject a few of them… I will say, initially, because I am one of the individuals who tends to be prone to motion sickness. So, that limits what I can personally do in VR and that is yet another thing that we’re gonna have to figure out. At least informally, what we hear is that women in particular tend to experience more of this. So, I needed, first of all, to go to a very low motion VR. I wasn’t gonna be whizzing through these environments. That was not going to happen for me. So, we did something that probably sounds incredibly simplistic, but it just touched me to my core… which is getting to play with Google Earth. You can spin the globe and either just pick a place at random or what Giovanni told me is… “You know, I’ve observed that when people do this, when we have an opportunity to interact with Google Earth, they all either go to where they grew up or they’ll go to someplace that they have visited recently or they plan to visit. So, I went to a place that is very special to me and maybe it doesn’t fit into either one of those categories neatly, but it’s my daughter’s University… her school… and I should say that this is also a different thing for me because my daughter goes to school in Frankfort, Germany… an institute that is connected to a Museum. So, I had only been to part of the physical facility… the museum itself… and it was a long time ago… and part of it was closer to the holiday. So, this is my opportunity to go there and explore what it looks like all over… and so, that was an emotional experience for me. It was a sensory experience… it was a social one… because we were talking the whole time… and he’s asking me questions and what kinds of exhibits do they have here… and what’s this part of it. So, that was wonderful. it really did give me a feel for alright, what is it actually like to be in this sort of environment?

I’m not a gamer. I don’t have that same background that many of our students have. So, it got me up to speed on that… and it did show me how just exploring something that is relatively simple can really acquire a whole new dimension in this kind of immersive environment. Now the postscript that I talked about in that blog post was what happened when I actually visited there earlier in the year. So, I had this very strange experience that human beings have never had before… which is from this… I don’t know whether to call it deja vu or what… of going to the settings and walking around the same environment and seeing the same lighting and all that sort of stuff that was there in that virtual reality environment… but this time, of course, with real human beings in it and the changes… the little subtle changes that take place over time, and so forth.

So, how does it translate into learning? What’s it going to do for our students? I just think that time is going to tell. It won’t take too long, but I think that these are things we need to know. But, sometimes just getting in and being able to explore something like this can really put you back in touch with the things you love about educational technology.

Rebecca: I think one of the things that I’m hearing in your voice is the excitement of experimenting and trying something… and that’s, I think, encouragement for faculty in general… is to just put yourself out there and try something out even if you don’t have something specific in mind with what you might do with it. Experiencing it might give you some insight later on. it might take some time to have an idea of what you might do with it, but having that experience, you understand it better… it could be really useful.

John: …and that’s something that could be experienced on a fairly low budget with just your smartphone and a pair of Google cardboard or something similar. Basically, it’s a seven to twelve dollar addition to your phone and you can have that experience… because there’s a lot of 3D videos and 3D images out there on Google Earth as well as on YouTube. So, you can experience other parts of the world and cultures before visiting… and I could see that being useful in quite a few disciplines.

Rebecca: So, we always wrap up with asking what are you going to do next?

Michelle: I continue to be really excited about getting the word out about cognitive principles and how we can flow those in to teaching face-to-face with technology… everything else in between. So, that’s what I continue to be excited about… leveraging cognitive principles with technology and with just rethinking our teaching techniques. I’m going to be speaking at the Magna Teaching with Technology Conference in October, and so I’m continuing to develop some of these themes… and I’m very excited to be able to do that. I’m right now also… we’re in the early stages of another really exciting project that has to do with what we will call neuromyth… So, that may be a term that you’ve turn across in some of your reading. It’s something that we touched on a few times, I think, in our conversation today… the misconceptions that people have about teaching and learning and how those can potentially impact the choices we make in our teaching. So, I’ve had the opportunity to collaborate with this amazing international group of researchers who’s headed up by Dr. Kristen Betts of Drexel University… and I won’t say too much more about it other than we have a very robust crop of survey responses that have come in from, not just instructors, but also instructional designers and administrators from around the world. So, we’re going to be breaking those survey results down and coming up with some results to roll out probably early in the academic year and we’ll be speaking about that at the Accelerate conference, most likely in November. That’s put out by the Online Learning Consortium. So, we’re right in the midst of that project and it’s going to be so interesting to see what has the progress been? What neuromyths are still out there and how can they be addressed by different professional development experiences. We’re continuing to work on the Persistence Scholars Program on academic persistence. So, we’ll be recruiting another cohort of willing faculty to take that on in the fall at Northern Arizona University. I am going to be continuing to collaborate and really work with and hear from John and his research group with respect to the metacognitive material that they’re flowing into foundational coursework and ways to get students up to speed with a lot of critical metacognitive knowledge. So, we’re going to work on that too… and I like to keep up my blog and work on shall we say longer writing project but we’ll have to stay tuned for that.

Rebecca: Sounds like you need to plan some sleep in there too.


John: Well, it’s wonderful talking to you, and you’ve given us a lot of great things to reflect on and to share with people.

Rebecca: Yeah. Thank you for being so generous with your time.

John: Thank you.

Michelle: Oh, thank you. Thanks so much. It’s a pleasure, an absolute pleasure. Thank you.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.

36. Peer instruction

Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know (and don’t know). Did you picture a lecture class, students taking a test, or students writing? In this episode, John discusses three ways in which he has been using peer-instruction in his classes: classroom polling, calibrated peer review writing assignments, and two-stage exams.

Show Notes


Rebecca: Imagine a scenario where students retain knowledge effectively and are active and engaged participants who are self-aware of what they know. Did you picture a lecture class, students taking a test, or students writing? If not, stay tuned, this episode explores ways to use peer-instruction to transform the learning experience.


John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


Rebecca: Today’s guest is my co-host John Kane. John is the director of the Center for Excellence in Teaching—that’s not even right…


John: …Center for Excellence in Learning and Teaching at SUNY Oswego.

Rebecca: Yeah, woops! Welcome to your own show, John!

John: Thanks, Rebecca.

Rebecca: Today our teas are…

John: Prince of Wales.

Rebecca: Oh, that’s a good one.

Rebecca: I have Golden Tipped English Breakfast today.

John: Excellent.

Rebecca: One of the areas you’ve been teaching experimenting in, and that I’m fascinated in, is peer instruction. Can you tell us a little bit about what peer instruction is and why you’re drawn to using this methodology in your courses?

John: Peer instruction involves using peers to assist with instruction, where students explain….

Rebecca: Thanks John.


John: …where students explain things to each other. One of the issues that we have is that, once we become experts in the field, it’s very hard for us to express things in terms that are easily understood by students. There’s a “curse of knowledge;” once you become adept at something, it’s really hard to explain things at a level that’s appropriate to the level of understanding that students may have. There was a classic study done in which a researcher gave people a list of songs, very well-known popular songs, and asked them to tap out the beats from that song.

Rebecca: Oh, I would fail…

John: …and then before actually seeing if people would recognize it (who had the same list), she asked them to make a prediction of what proportion of people would understand it based on their tapping… and they overestimated that by a factor of I believe, somewhere around 20 times. Basically, it was purely random if people happen to guess it. But the issue is, once you hear something in your own mind, it’s clear to you, but it may not always be clear to the people who don’t have the same rich net of connections. When students are explaining things to each other, they benefit from taking a position, arguing that position, trying to filling gaps and they’re also explaining in terms that are appropriate for people at their level of cognitive development for people who have a similar background in terms of what they know and their prior knowledge.

Rebecca: Sounds like a really good way to expand and refine mental models and also just develop better metacognition. Because, as soon as you go to explain it, you realize what you don’t understand.

John: …and if you don’t understand it yourself, your peers will often help you understand. they’ll say: “Well, you haven’t considered this…” and that sort of interaction is one that doesn’t work as well when it’s instructor to a large group of students. But, it does work very well one-on-one.

Rebecca: You’re known on our campus for teaching really large lecture sections. Implementing peer instruction in a large setting can seem pretty daunting, especially to someone who teaches smaller classes like I do. What strategies do you use?

John: The most commonly used one is to use clicker quizzes… and I use a methodology that Eric Mazur developed slightly over 20 years ago, where you ask the students a challenging question… you try to find questions that about half of them will get wrong… and over time you can develop that, you can come up with a pool of questions that fit somewhere in that range… and you let students first vote on the response themselves after they’ve had a little bit of time to process it, and then you look at the results. If you see that 90% or more of them got it correct or even 80% or more, you can just go over it and move on to the next topic, because most students understand it. But, if you see that somewhere around half of them get it right and somewhere around half of them get it wrong (plus or minus 20% or so), then the next stage is to let them explain it to each other, and that’s where the peer instruction comes in. When you have students argue it and take a stand and a position on it, we get a very significant gain and improvement when we then let them vote on it a second time… and the usual practice is not to reveal the poll results or the answer until after they’ve had that opportunity to engage in that discussion.

Rebecca: Just make sure, to make sure I understand correctly: you do the poll, you see the results as students don’t see the results…

John: Right.

Rebecca: …based on their answers or their responses when you decide whether or not they do the peer instruction piece. How long do they usually talk to each other about the topic?

John: It depends on the problem and normally I will have some undergraduate TAs and I’ll wander around the class and see what they’re talking about, listen in, answer some questions from them and the TAs will be doing the same thing…. and it’s usually pretty clear when they’re coming to a consensus. You can see them reaching for their clickers or their phones and getting ready to vote, so generally it may only be a minute or two, it could be longer… it depends on the complexity of the problems. Some of the problems require a bit of effort and require some calculations, but normally they’ve already done that… so, the second stage, where there’s a discussion, you can hear the volume build-up and then as they’re approaching solutions and consensus, it tends to drop back down again. It’s fairly easy to get a pretty good read on where they are and when they’re ready to vote again.

Rebecca: I imagine that you would really need to keep your ear to the ground, otherwise chaos could ensue. Because now, if they’re finished talking about the problem and there’s still time, then they could easily derail if you’re not quick to get back to the clicker question.

John: Right, and normally the time is generally held fairly tight. I suspect sometimes it’s only 30 seconds to a minute, other times it may go up to a couple minutes, but if I see them getting distracted and doing other things, the polling starts immediately.

Rebecca: Obviously technology is your friend in this particular situation. Can you talk a little bit about the technology you’re using to manage this many students all at once?

John: Here, we’ve adopted iClicker as a campus standard, so we use that in pretty much all of the classes where we’re doing polling and there’s both a physical radio frequency clicker that students may buy or they can buy an app and pay by the semester or over four years for the use of the app.

Rebecca: How do you make sure that the cost doesn’t get prohibitive to students?

John: That’s an issue, and it’s been a major source of concern…

Rebecca: They’re not very expensive, right?

John: Well, they can be expensive. A new clicker costs somewhere around $40. A used one can often be purchased for $15 to $20, sometimes less… and the apps I think, are somewhere around $12 to $15 for a semester and I think about $35 for four years.

Rebecca: …and you can use the clickers in all of the classes, right? So if multiple faculty member(s) are using all the same system, then the investment is a good one for students.

John: …and that’s why we have a campus adoption because in places where you don’t have that, students might have to buy two or three or four different clicker systems in different classes. So, once they buy the clicker for one as long as they hold on to it, they can use it in classes for the rest of their career. Almost everyone in the economics department, for example, now uses clickers, so if they’re economics majors or business majors, it’s very likely they’ll use them in multiple courses. The cost is much more tolerable when it’s spread out over multiple classes.

Rebecca: The other area where you do some peer instruction in these large classes is in writing. Which seems kind of crazy. You have all these students in this big classroom and somehow you manage to do writing assignments.

John: Yeah, my large class generally is somewhere between 350 and 420 students. At one time, for actually about a decade or so, I was giving weekly online discussion forums. But grading that or evaluating that and providing feedback was taking an awful lot of time…probably 30 to 40 hours a week. So, I pretty much…

Rebecca: A full-time job in and of itself…

John: I stopped that a few years ago and, a few years back, I replaced that with calibrated peer review assignments. The calibrated peer review system is something that Eric Mazur talked about while he was here… a visit in 2014… and when he mentioned it, a lot of people got excited. The way the system works is that you create an assignment, you store it on a central server at UCLA, and then it’s something that other people can adapt and use and modify—it’s released under a license, which is similar to a creative common license within the system… and you create the assignment… you create an evaluation rubric for the assignment… and you have to be really careful in designing that to make sure it’s one that students will be able to apply, because other ones that do that… and then you create three sample assignments yourself: a low-quality one, a medium quality one, and a high quality assignment… and you have students submit their own assignments first (according to the rubric and guidelines you provide to them)… then they go in and they evaluate the three that you’ve done. They’re given in random order, and they’re assessed in terms of how closely their evaluations match yours. That’s the calibration part. Students receive a calibration score based on how similar their evaluations are to the ones that you assigned to the sample responses. Then after they complete that stage, they evaluate each other, using the same rubric, and a weighted average of those scores is assigned as a component of the grade. They’re graded in a number of dimensions. One is based on the weighted average of the peers, where students who had a high calibration score will have evaluations that rate more highly in evaluating other students. They’re also rated in terms of how closely their evaluations match the others during that stage. So, if their evaluation is an outlier… much higher or much lower than other students… they lose some points on that… and then after they evaluate the other three students, they rate their own work… and one of the goals of that is so that they have improved metacognition. That, by the time they go back and look at their work again, they’ve rated three works by the instructor and three assignments done by their peers and then they’re asked to evaluate their own using the same criteria. What’s really interesting about the calibrated peer review process is their grade on this is tied not to whether they give themselves a high or low score on this, but it’s how close their self evaluation comes to the weighted average of their peer evaluations. So, they have an incentive to try to look at their work more objectively, and not try to game the system… because if they score their work too high or too low, they could end up with no points on the self-evaluation stage. So, the closer they get to the weighted average of their peer evaluations, the higher the score will be on that component.

Rebecca: I think that’s an area that we often see students struggling, is being able to effectively evaluate their work or other work. So, really training them to use a rubric and understand and think about what’s important or what’s not important about particular kinds of assignments or particular kinds of work could be really valuable to students in a way that we don’t really have other systems to do that.

John: The nice thing about this is it scales really easily. There’s a lot of upfront work in creating the assignments, creating the rubric, and a really good practice is to test them thoroughly before you give them out the first time. What I normally had done is asked some of my peers to look at that, some of my colleagues to look at it, and sometimes I’d have some upper-level students were…. and this does give students a little bit more reflective practice, where they get to look at their own work a bit more critically, perhaps, and reflect on it and see how they’re doing compared to how other students are doing in the course… and I think that’s helpful.

Rebecca: I think that the rubric would probably be a challenge to make but I think what would be more challenging is putting up those different assignments that are scored at different levels at the very beginning as your calibration tool. What strategies have you developed to make those in a way that it doesn’t take forever?

John: Well, I only do this three times in a semester, and once you’ve done it once, if you design it in a way so that it won’t go stale… and I generally have students, for example, find some articles in the news in the last six months that relate to a topic that we’ve talked about, or I ask students to find some examples in their own life to illustrate behavioral economics concepts in one of the assignments, for example…. where it’s not something that they could easily copy and paste from other people’s work. Because, there is always a concern with academic dishonesty and so forth. You don’t want these things showing up on Chegg or any of those other systems, where it would be easy to copy and paste good responses. So, I’ve tried to design assignments where once they’re done, they can be used for multiple years in one form or another. I modify them each year based on how they work. But perhaps a more serious problem is what happens when students really don’t like the evaluations. One of the things I’ve done when I’ve used this is to have three of these assignments, but I drop the lowest score… because, sometimes people will get some scores back that they didn’t expect or they may have neglected to look at the rubric I sent them and they may have omitted a major part of the assignment and ended up losing quite a bit of points all the way through that. But, as long as one of the scores is dropped, they have the opportunity to learn from their mistakes and do a little bit better. But, there are procedures built-in that make it easier to catch any outliers when you have someone who is just rating everyone extremely highly or rating everyone really poorly—inappropriately highly or poorly. There are tools in it which will give you a list of all the cases where there’s a high variance across reviewers or where someone happened to be evaluated by people who had very low calibration scores… so, if you end up with two out of the three peer reviews with low scores, that’s something that’s flagged by the system. I check all the cases where it’s flagged and I tell the students if they’re unhappy with their score or if they have any questions about it, to contact me, explain why they’re dissatisfied with their score, and then I’ll go in and look at it. In nearly all cases, it’s been an issue with the students submission and not with the peer reviews. Because, while some people tend to overrate things and some tend to underestimate some of it, compared to where I would evaluate the wok… on average, it’s been very close, typically, to what I would have scored or what I would have assigned as a score. But I do make, in rare cases, some adjustments when I see that something went wrong in the process.

Rebecca: Do you prevent students from seeing the score then, until you’ve reviewed all of the scores to make sure that you’re okay with what has happened before they have access or…?

John: In this system, that really can’t be done easily…

Rebecca: ok.

John:… because what happens is they get the results as soon as the last stage is completed. I’ll send a note out saying, “Now that the stage is completed, you can review your scores, you can read all the comments that your peers have provided, and you can see what your grade is at each component…” and we have gone over that in class so they know what they’ll be seeing.

Rebecca: What kind of workload do you end up with, dealing with problems?

John: In general, when I’ve used this in the class of 360 to 420 students, there’s usually 3 to 5 students who find their grade unreasonable, and sometimes, I found the grades perfectly fine. Occasionally one or two of those, I’ll make some minor adjustments to—if something went wrong where one of their peer reviewers didn’t show up, for example, one or two of them didn’t complete that stage of the assignment, and someone was overly harsh or perhaps overly harsh in their grading, but it’s rare.

Rebecca: Can that system be used for things other than writing? Like other kinds of documents?

John: It could be used for any type of document because basically students will either write something up or they’ll submit something and it could be an image, it could be used for peer review, or calibrated peer review, on pretty much anything as long as it can be disseminated in digital format. It could be used for websites, for example.

Rebecca: Well, that’s what I was getting at when I was asking.
You also teach some upper-level seminar courses with 30 or so students. This semester, you tried a two-stage exam after talking with Doug McKee when he was on campus about it. What is a two-stage exam and how did it work?

John: Backing up a bit, I was considering it even before Doug came here because I heard the episode of the Teach Better podcast where they discussed a two-stage exam and then when we were talking here and he was in one of our earlier podcasts and we discussed this very issue, I became more interested after we talked with Doug. A two-stage exam is one where in the first stage of the process, students take the exam by themselves and then in the second stage, they do some group work– either on a subset of the questions or on some very closely related questions. It’s being used quite a bit in the sciences and there’s a growing amount of research indicating that it has been successful. Some studies have found weak results, others are finding stronger results, but it’s still fairly early in the exploration of this. The Carl Wieman Science Education Initiative has quite a few resources associated with two-stage exams. This leverages peer instruction in the second stage.

The usual process, or the most common practice, is to take the exam period and have students work on this for the first two-thirds or so of the exam time slot and then they work in a group in the last third. I did it a little bit differently than this. In my case, I gave the exam on a Wednesday and I graded the exam but didn’t get them back to the students and then I selected a subset of the questions and I had them work on them in groups on that Friday… and that worked pretty well too, they had a little chance to review in between, they didn’t get to keep the exams, but there were only seven questions on it. They could go back and review things. I didn’t tell them which questions would be on the second stage in large part because I didn’t know. I told them that two of the questions would definitely be on it, but it would depend on how they did on the other part. So, I was able to look at the exam, find the parts where they had the most trouble, and assigned those as ones for the second stage… and in general, it was a remarkable experience. It was really nice to be giving an exam and to see students working in groups of three or four, actively discussing the issues, arguing over them, trying to explain things to each other and it was a really fun experience. It was very energizing to see that much effort being devoted to try to understand concepts that students had some difficulty with.

Rebecca: I remember seeing an image of your class being really actively engaged, really talking about the core class material that you shared during your test and I think the caption was: “This is during a test!”

John: Yes, I took a picture of it from my phone and I think I sent that to you during the exam because it was just so exciting to see that… and it was also a reminder for myself just how well this was working. I wandered around the room and listened in on the discussions and they were all very focused and coming up with much better explanations of these things then they would have likely been able to see if it was a whole class discussion… because they were very focused, they were arguing over what was the best approach to deal with some of these problems. I could see people making connections and suddenly understanding how things they had done before fit in and pulling together a lot of concepts that they might not have done as effectively if it had not been for those small group discussions.

Rebecca: Were you tempted to join in on those conversations because they were so lively?

John: I was, but I mostly just listened in and let them work it out themselves… and in general, they did quite a bit better… and what I should have mentioned before is that the overall grade for the exam is a weighted average of the first part and the second with most of the weight being on the individual part. One of the things that really appealed to me is that typically, when we give an exam and then grade it and return it, the students who did well generally just put it away and are happy with the results and they may glance at some of the things they got wrong (if they got many things wrong), but they’re not going to spend a lot of time actively processing it. The students who did poorly tend to get discouraged, some of them may give up a bit, but rarely are they likely to go back and try to put in the effort to correct their mistakes and to see where they went wrong. It was really nice to see that processing taking place by both groups. The students who did really well the first time deepen their understanding by explaining it to others and I suspect that should increase their long-term recall of this. The act of explaining it to others in some studies seems to be really helpful in encouraging transfer, where you can take concepts and apply them to other circumstances and when you’re in a course like econometrics, you have to be able to apply the same concepts in a wide variety of topics and areas. I think it was a very useful experience.

Rebecca: I think it’s a great method to allow some time and space for a reflective practice, because students tend not to do that on their own unless they’re asked to do it and if you do it as a homework assignment, I suspect that students don’t really spend that much time doing it, but this time they spent the whole class period doing the reflection. So, that seems really valuable.

John: Because I know a lot of people will do that. They’ll have an exam, they’ll give it back to students, and they’ll tell them they can make up part of the grade if they turn it in with corrections… and many students would do that, but I don’t think that would be as effective as having the group discussion on this. Some of them were able to make very clear what they didn’t understand and then they were able to get explanations from others and sometimes the explanations were right, sometimes they were wrong, but they had to process it much more actively and that’s always helpful, I think.

Rebecca: The grade weights is what seems most compelling to me in this situation because I’ve offered quizzes in my classes, more low-stakes assignments where I let students work on it for a while and I don’t tell them that they’re gonna get to do some peer instruction as part of it, but then they’re struggling with what they’re doing and then I say, “Oh, well, you have five minutes to work with your peers to revise anything you want to do before you turn it in.” And those generally result in some pretty active conversations as well, but there still are those few students who just copy down the answer and don’t engage in the conversation… but I think if there was that wait between before and after, that would really change that dynamic. So, I think that that’s a really compelling opportunity.

John: I thought it was useful and another reason why I didn’t do it all at one stage in one day is because I’m teaching on a Monday-Wednesday-Friday schedule and we only have 55 minutes and I have quite a few students in the class who are not native English speakers and they always take more time or they need more time to process and write information in a second language. So, I didn’t want to constrain the time and make both parts of it much shorter.

Rebecca: If you encourage people to practice and retrieve that information in extra time outside of class, there’s nothing wrong with that either.

John: Exactly.

Rebecca: I’d rather the students learn the material rather than just panic about a test. What do you recommend to our listeners to read to learn more about this evidence-based practice?

John: In terms of peer instruction, Derek Bruff has a really good book on using clickers. Eric Mazur’s original book on this, which is now slightly over 20 years old, is still very good… where he describes a process of developing this peer instruction technique. Eric Mazur also gave a talk here a few years ago and we have a recording of his presentation on this. There’s a really great example in there where he used peer instruction and what was most compelling about it, and Rebecca’s heard this before, but…

Rebecca: I was there!

John: …and Rebecca was there, was he used this example where he gave a really short presentation on what happens to the hole in a plate of metal if you heat it up… and people were asked to vote on that and then they had a chance to discuss it.

Rebecca: …and he never told us the answer!

John: …and then he noted how energized people were and he said, “You were so actively discussing these things…” When he tried to go on after making a point about how they suddenly were interested in something they normally wouldn’t have been interested in… he started to go on to the next topic. People were really upset, because they wanted the answer and he finally gave the answer, but he did that deliberately to show that this sort of thing… where the students don’t know the answer but they committed to a position and they want to know if they’re right… builds a sort of interest in learning that might not intrinsically be there otherwise.

…and that’s exactly what I saw, by the way, in my exam. They were so actively discussing things that normally they’d be bored out of their minds with. So, that environment can be very supportive of learning.

Rebecca: Yeah, it really gets people curious. I remember being in that room… dying to know what version was right? People had such compelling arguments.


John: Exactly, and that’s why it’s really good to pick questions, with any of these things, where it’s not going to be clearly obvious, where they have to process it, and they have to make connections, and you could build a case, correctly or wrongly, for different answers, and people want to know what the answers are.

Rebecca: I mean it was key that he finally gave the answer, right? So there was some corrective feedback there, so that people didn’t continue to mislearn the information.

John: And that was nearly four years ago, and we remember that very vividly. If that was just a point in a class that was given… say, four years ago, we probably wouldn’t be talking about that now.

Rebecca: I can’t believe it was that long ago.

John: I think it was.

Rebecca: It was a while ago.

John: Yeah.

Rebecca: …and so I’m dying to know, what are you gonna do next?

John: One of the next things I’m going to do is a follow-up to something we talked about in an earlier episode, when we talked to Judie Littlejohn about the metacognitive cafe. One of the things I’ve been observing is that the use of this process by having students work to improve their metacognition about how they learn and what they’re learning… Students, at least, perceive there is being some significant learning gains from that. That’s convinced me that I’d like to do something similar in a large class, but an online discussion forum for 400 students again doesn’t scale quite as well. So, I’m going to be doing some weekly activities and I’m working with Liz Dunne Schmitt who teaches our large macro class in the spring semester, and a couple of other people: Kris Munger, and Michelle Miller, who also who’s the author of Minds Online: Teaching Effectively with Technology (and was a guest here a while back). We’re going to try to put together an experiment where we use some evidence-based methods as weekly assignments, say for ten weeks in a semester…. that’s our current plan at least)… and students will be exposed to this… and they’ll engage in some sort of reflection or some practice with one of these activities… and then in terms of evidence-based methods of learning, such as retrieval practice, spaced practice, and interleaved practice, and similar things… and then we’re going to see how that exposure along with some reasonably easily assessed activity, which could be just some short responses in a forum or it could be perhaps some online quizzes, evaluating whether that impacts their actual behavior in the class, and their actual performance in the class. One-half of the group will be exposed to those types of interventions, and the other half will be exposed to some form of standard study skills module, because most of the students in this class of freshmen and basically what we’re looking at is, if we present students with evidence on what really increases our ability to learn, whether that will result in significant change in either their behavior, or in their performance. So, we’re going to try, at least the plan, is to try to see whether that affects the number of times they take quizzes that can be taken repeatedly, whether it affects the number of times they log in and view other materials, and whether it changes a perception of how we learn. so right now we’re at the…

Rebecca: And performance too, right?

John: …and their performance.

Rebecca: And is the plan to start collecting that data in the fall?

John: The plan is to put all this together the spring, I’m hoping and then to submit a proposal to the IRB, and then to conduct the study and the fall and the spring, at least for a first stage and then we’re hoping to be able to follow these students up, to see if this has a significant effect later in terms of their grades or their persistence.

Rebecca: Sounds pretty exciting. I’m looking forward to hearing how that goes.

John: It is. I’m looking forward to it being all together and actually being implemented. I think it’s an interesting study.

Rebecca: We’ll have to have you back, John.


John: I think we can manage that.

Rebecca: Well, thanks so much for sharing all this information about peer instruction. I know it’s something that I’m always kind of asking you about and like to hear about, and I’m sure others will too.

John: Well, thank you.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Theme music by Michael Gary Brewer. Editing assistance from Nicky Radford.

33. The Marmots of Finance

In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, Alex Butler, a Professor of Finance at Rice University, joins us to discuss how rich imagery can be used to help students make connections and deepen their understanding.

Alex received the Rice University Presidential Mentoring Award and the George R. Brown Award for Superior Teaching in 2018. He is also the recipient of the Jones School’s Award for Scholarship Excellence in 2011 and 2012.

Show Notes

  • Duke Talent Identification Program (TIP)
  • Medina, J. (2011). Brain rules: 12 principles for surviving and thriving at work, home, and school. ReadHowYouWant.com.
  • Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185-205). Cambridge, MA: MIT Press.
  • Alex’s web site at Rice


Rebecca: In our ongoing coverage of wildlife in the classroom, we can’t help but ask: How does a marmot become a mascot for a finance class? In this episode, we’ll discuss how rich imagery can be used to help students make connections and deepen their understanding.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.


John: Our guest today is Alex Butler, a professor of finance at Rice University. Alex received the Jones School’s Award for Scholarship Excellence in 2011 and 2012. Alex teaches financial markets and corporate finance in the undergraduate MBA, MBA, and PhD programs. Sometime in the latter part of the last century, Alex and I spent three summers teaching introductory economics to highly gifted middle school and high school students at the Talent Identification Program at Duke University.
Welcome, Alex.

Alex: Oh, thank you. Thank you for having me, John.

Rebecca: Today’s teas are:

Alex: I am not drinking tea…. just water at this point.

Rebecca: Another one… epidemic.

John: My tea is ginger peach white tea.

Rebecca: I’ve Prince of Wales today.

John: Could you tell us a little bit about the courses that you normally teach?

Alex: Sure. I have, over the years, taught almost every course that we have… either here or at other schools. Right now what I teach (and I’m glad to finally have settled into a group of classes that are my classes year in and year out) and those classes are the undergraduate business finance class and PhD courses – one in corporate finance, and one that’s a topics class on causal inference. I really enjoy teaching the undergrad business finance class in particular because I was an undergraduate student here at Rice many, many years ago, and so it’s fun to be back and be on the other side of the podium teaching the students. One of the things that I really like about teaching the undergraduates here is that they are able to appreciate my dorky sense of humor.

Rebecca: …which is indeed a very important thing.

Alex: Agreed. For years, I tried to suppress my dorky sense of humor in the classroom… ad tried to teach the course sort of straight up and dry, and it made everybody miserable… the students…. me… and so, eventually over time, as I grew more and more confident teaching, I started incorporating more and more jokes here and there… and then I allowed them to become more elaborate, and then before you know it, I’ve developed a full-blown dad sense of humor and full-on dork mode.

John: That’s actually why we invited you here. We read a little bit about that in a teaching award you just received. What do students expect the course to be about when they take an introductory finance course?

Alex: At Rice, the students have a good grapevine of information about what courses are about and so, at this point, the students come in with pretty solid expectations that line up with what the class actually is… and whereas some students are hoping that I’m going to teach them how to become millionaires in the stock market, what the course mostly is about is that’s very hard to become a millionaire investing in the stock market and how to make decisions in a corporate setting that will maximize firm value.

Rebecca: So you mentioned that some students come in with this misperception of becoming a millionaire. What are some of the strategies that you use to dispel that myth?

Alex: I should say this class is one that is a fairly standard course that’s taught in lots of different business schools all over the country, all over the world. One way that I do it differently is in the order of material that I cover… and so the very first week of class, I talk about market efficiency. That’s the notion that it’s very difficult to earn abnormal returns in the stock market. In other words, you can’t beat the stock market very easily, unless it just happens to be by luck… and so I come in the first week talking about the reasons why it’s very difficult to beat the stock market… and the reason why is because there are literally tens of thousands of people who have more money and more resources and who are faster and are doing this as a full-time job who are also trying to find the stocks that are mispriced… and so unless you are the investor who is faster, and smarter, and has more money to throw at the trading strategy that you think of, it’s very very difficult to beat the other 10,000 people… and so I spend the first week of class introducing this concept and then providing copious amounts of evidence… research that highlights how difficult it is for lay people to earn abnormal returns in the stock market. That sets the setting for the rest of the course, which is this notion of how competition affects prices and how that feeds through to other applications and the corporate domain as opposed to the financial markets domain.

Rebecca: What made you switch the order?

Alex: What I realized after teaching this class for a long time is that most of corporate finance, most of business finance, is about discounting cash flows back to the present at some appropriate discount rate, and I found I was having a hard time getting the students to understand the notion of what interest rate, what discount rate, should be used to make these cash flows that are spread through time to get a present value equivalent. I figured if I started with some aspect of that, where that rate comes from, where those prices come from. but that would make the rest of the course easier for them to understand as we go through. So, that’s why. Most people wait until after they’ve introduced things like: “oh, portfolio theory” and “capital asset pricing model” and other asset pricing concepts before they talk about market efficiency… and so I just sort of turned that around backwards… and I open the course with that. So, point number one is pedagogical, and point number two is that the lectures that I do on market efficiency are really fun, and so I really like starting the course off with something that’s really fun. So, we can talk about stories of insider trading, and we can talk about stories of surprise announcements and how that affects stock returns, and we can talk about “oh, so you think you’re going to beat the market, well let me explain to you how hard it is and the reasons why …” also were very fun. So, we can spend the first week talking about fun stuff. It’s a giant bait and switch.


I lure them in with fun stuff and then beat them to death for the next fourteen weeks with discounted cash flow analysis.

Rebecca: It also sounds like it’s a good way to motivate students. Not only is it fun, but it gets students motivated and interested and they buy into the class, which I wouldn’t discount that. I think that’s an important task.

Alex: Oh, absolutely.

John: …discounting in a different sense, but…


Rebecca: Yeah, sorry… I’m a designer

John: …but it also starts a class by dispelling that myth that they’re going to learn tools that will allow them to become really wealthy in financial investments and so forth. So, you’re setting it up by getting rid of that myth and they’re ready to start actually learning without having that at the background.

Alex: Correct.

Rebecca: I really like the idea of just meeting students where they’re at. If this is where some students are coming with, and maybe they’re super motivated in some ways but not in others, that you just tackle both of those in one week.

Alex: That’s right.

Rebecca: Cool.
You recently won a 2018 George R. Brown Award for superior teaching, and the Georgia R. Brown Awards are based on an interesting selection process. Can you tell us a little bit about that?

Alex: Yeah, so what the university does is they solicit feedback from recent alumni, people who graduated, I think, it’s two and five years ago. Now, they have graduated and they’re looking back what teachers would they want to see when these awards… and so, it’s really a neat honor, because once the students are two, three, four. five years out, they’re not responding to the short-term incentives that some professors used to gain teaching evaluations and things like that, they’re looking back and they’re actually remembering what the course was, what the professor was like, and whether it was meaningful to them. It’s really been just a phenomenal honor, and one that I honestly thought that I would never win. So, it’s been really special to have gotten that award.

John: It’s a really interesting idea to have an award given in that way, because it focuses on that long-term learning, that you’ve taught them something that’s going to benefit them later rather than, as you said, something that people do short term. I know some faculty, when we’re doing course evaluations, will give out cookies or other things just to boost their short-term course evaluations. But, that’s not going to pay off very much two to five years after graduation.

Alex: That’s correct. I went through the cookies and brownies phase myself years ago and decided that that just wasn’t who I wanted to be, so…. Now, we’re on to just just the pedagogy at this point.

Rebecca: I think there’s a couple of interesting things that I see about those awards, too, and one is that is more meaningful probably to win it because you can feel the impact but two it meshes really well with what we know about evidence-based practices in that students generally don’t like them while they’re happening, but they have longer-term effects… and that students tend to learn that material and transfer that material later on.

Alex: That’s absolutely correct, and I’m a big subscriber to that basic view that students often don’t like things that make them uncomfortable, and learning new and difficult things is uncomfortable. I think one could make students happier in the short run by giving them lots of little assignments where they feel like they’re making progress every day but they’re not actually being challenged, they’re not actually being pushed, and so instead holding them to very high standards for demonstrating their mastery of material while that is very uncomfortable for some of them, it is that that makes them better students and better scholars of that topic.

John: It reminds me of Bjork and Bjork, in their writings on “desirable difficulties.” that the most learning occurs when students are faced with feasible challenges… that if things are too easy, they get bored, and they may be happy with the course if they don’t have to struggle much… but they learn the most when they’re struggling but they see it’s possible.

We’ve heard that this award, though, based on some of the write-ups we’ve seen at your institution, may somehow be tied to marmots, wolves, and The Princess Bride. What do they all have to do with finance?

Alex: Several years ago, I was reading some books I thought would be…. some I thought it’d be helpful for my teaching, some that I just thought would be helpful for me, and one of the books was a book called Brain Rules by John Medina, and it’s basically a book that tries to take cognitive science, brain science, down to a level that lay people can understand and gives several rules of thumb of how the brain works and why the brain works the way it does… and as I’m reading this, reading it mostly for my own consumption so that I can be a better researcher and more thoughtful person, smarter, that sort of thing, I realized “Gosh, a lot of the rules here applied very directly to teaching, at least in the lecture format that I use in most of my courses…” and so one of the things that really stood out to me is how people learn better, remember better, I guess I should say, when they see images images that relate to whatever the topic at hand is. So, text maybe a PowerPoint slide with text, and you remember X percent but if you see an image, you remember much more of that material later on… and so this gave me just a license to, all of a sudden, start having fun on a completely new dimension. Reading this book and sort of embracing the notion that I could maybe help students remember the material better simply by infusing my lecture slides with some relevant images, was just eye opening for me… because now I could take my completely dry, boring slides with words and numbers and equations and now I can have fun with them…. and have this entire new dimension, a degree of freedom, to play around with what the slides are gonna look like and how the students are going to experience them. So, that’s sort of the extensive margin. The first part is: “Hey, I need images…” so the intrinsic margin is “what kind of images do I need?” “what will work best?” and Brain Rules comes to the rescue there again… and it says people respond to images that are faces, that are things that are scary, things that they can eat, things that might want to eat them, and things that they might want to mate with. Well, that last one’s kind of out for most of my lecture slides… [LAUGHTER] I can’t really incorporate that very directly. But it got me thinking “okay, what’s scary?” So, I started looking around for images of things that are scary to people… and I found this great image of this really, just terrifying, snarling wolf. I’m like “Ah, I’m gonna use that to get people’s attention…” But, as I started thinking about it, I wanted to lead into the wolf a little bit… and so the main prey of wolves in North America are marmots… yellow-bellied marmots… and so I found this great image of a little cute yellow-bellied marmot sitting on a rock somewhere in the mountain somewhere… and now when I come in to teach what could be the absolute driest lecture of the entire course which is time value of money and understanding how to discount cash flows (it’s the tool that everything else builds on, so it’s incredibly important to get it right, but it’s also potentially incredibly technical and boring)… and so I start that lecture not with an equation not with numbers but with a giant image of a marmot filling the entire computer’s projection screen…. and I just leave it up there… and the class is all looking at it… and I look back at them… and I pick someone at random. I cold call… and I say: “So, Charlotte, what do you think?”

“What do you mean what do I think?”

“What is it?”

… and we go through a series of guesses, and the guesses range from just ludicrous things: “it’s a gopher.”

“No, no, clearly it’s not a gopher. Gophers are a lowland creature. This is obviously in the mountains.”

“It’s an otter.”

“No, no, no. Otters prefer marine habitats, and this is clearly not there.”

… and go through this for a while until usually somebody recognizes it as a marmot. I say “Yes, very good. Alright..”

…and so then on to the next slide and the next slide is this picture of George Soros with no caption, no explanation, but again filling the entire screen, here’s this giant picture of George Soros… and so I go back to the first person: “Charlotte what do you think about this one? You didn’t get the marmot, how about this one?” …and so invariably somebody will eventually guess it’s a hedge fund manager.

I say: “Yes, very good. That’s George Soros, a famous hedge fund manager.” then the next slide is the wolf, the snarling wolf, and so at this point the captions on the slides read “This is a yellow-bellied marmot,” “This is a hedge fund manager” … and then it’s obvious what the wolf is. People get that right away. So, Charlotte gets to redeem herself at that point.


“This is a wolf, one of the main predators of the yellow-bellied marmot,” and then the next slide is another picture of a marmot but this one looking somewhat quizzical and the caption here is: “Why do wolves eat marmots, but not hedge fund managers?” …and the answer that I propose is because hedge fund managers understand the time value of money, but marmots do not. Now, that obviously doesn’t actually follow, but the correlation is there… that part is true… and so I just sort of lean into that and the ridiculousness behind that statement and from there on the marmot becomes our time value of money mascot… and so every time throughout the course I introduced a new application of time value of money then I bring in another image of a marmot to sort of tie all that together. iIt’s hard to see that the tie that binds everything together is time value of money and so the marmot, the mascot, is the visual cue, that “oh, this isn’t special, this isn’t different, this is just another application of the same technique we’ve been doing over and over again.

John: That notion of using visual cues goes back to the Greeks who use it to remember long stories before there was much printed word, and one of the arguments is that it’s because visual imagery developed much earlier than the use of language and the things you described in terms of things that might eat you may be tied back to our evolutionary adaptation, and so we’re tying into things that evolve fairly early in the evolution of human beings.

Rebecca: …and it also is just that emotional response.

John: Right, when you trigger an emotional response and fear is a major one…

Rebecca: Yeah. mm-hmm

John: Do the wolf’s chase the Princess Bride? I don’t remember that…

Alex: Well no…. So, the Princess Bride lecture is where I talk about decision rules using discounted cash flow techniques…. and so this sort of standard playbook for any finance professor anywhere as you talk about net present value and internal rate of return and payback period and maybe a few other rules as well, you just sort of go through the… whatever they have in the book in the order in the book, and you talk about each one in turn. But, it’s a lot more fun if you can personify those.. and so that’s what I use the characters from Princess Bride to do… and so the main character, which is net present value, is personified as Wesley / (there’s a spoiler alert here – so you haven’t seen the movie Wesley is the Dread Pirate Roberts). I always show my kids my lecture slides and when I showed them that one my older daughter was mortified that I would ruin the Princess Bride movie for students by revealing that Wesley and the Dread Pirate Roberts are one and the same… and so from there, I personify the different rules based on the different characters in Princess Bride… and so internal rate of return which is sort of flashy and useful but has some flaws as Inigo Montoya; the payback period which is sort of a blunt instrument is Fezzak the Giant, and so forth… modified internal rate of return is the Cliffs of Insanity.


Rebecca: So, the important thing to ask then is do you have the spoiler alert at the beginning of your lecture.
AEX: I do now.


John: How old was your daughter at the time when she reacted to that?

Alex: About ten years old.

John: Okay. So, by the time students are in college, they’ve probably either seen it or they may be less likely to, so it probably doesn’t do quite as much damage.

Alex: I hope so, and in fact I even now encourage them to watch the movie before that lecture without really telling them why. So some of them do. Some of them ask around and figure out why, but that makes it more fun.

Rebecca: That was gonna be my follow-up… is that homework?

Alex: It’s the best homework they’ll ever have.

John: So, how have students responded to this? Do they remember this later?

Alex: Yes and no. for the Princess Bride lecture I believe that the students sort of uniformly enjoy that…. but I don’t have a good sense of how much that actually impacts the depth of their learning. For the marmots, it’s a completely mixed bag… some of the students really love it, and I really do lean hard on the fact that this is the mascot… this is the thing that ties it all together… and this is the visual cue so you will now recognize that when we do bond pricing “oh, here’s a marmot, that means it’s just time value of money…” “oh, when we do net present value, here’s a marmot showing up alongside our Princess Bride character it’s just time value of money” and so forth. Many of the students really enjoy that and grab on to it and some students, they’re not having any of it. They want boring… they want dry… and I’m afraid they’ve come to the wrong place. Been there and done that. I don’t do that anymore.

John: But that use of imagery is really common. People who work on developing memory… the memory palace type things where you tie specific concepts and bundles of concepts with chunks of item to key images, helps people remember things long term. I can see how it would be really effective.

Rebecca: I also could see that one of the things that students often struggle with when information is new to them is making those connections. So providing that visual cue like “here’s time to make a connection…” it’s actually really helpful, because those are the kinds of things that might seem really abstract and very separate if you’re not making an explicit. So, I like that you don’t even have to say explicitly like “this is the thing” you have by putting the image up there, and you’re prompting the students to predict what that connection is or challenging them to think of it on their own before you reveal what that connection is.

Alex: That’s right, and so what started off as just a fun way to get attention: “Hey, class is starting let’s all key in… Here’s a scary wolf.” So, now we’re all paying attention to the scary wolf because that’s how our brains are hard-wired. It’s now grown into this entire thing where throughout the entire class is a continuing callback.

Rebecca: It sounds to me like you must have some pressure every semester to have to have something new that you introduce into some sort of lecture so that there’s some anticipation.

Alex: I’ll be the first one to say that I shamelessly recycle all of my jokes…


… and so I’m always sort of terrified when I have a student who started the class last year but had to drop at the 6 week mark or the 8 week mark because whatever was going on their life, and then they’re back the next year. Like “oh man, you’re gonna get the exact same jokes with the exact same timing, the exact same patter.” It mostly seems to work fine, and I do continually try to incorporate new things. One of the nice things about teaching finance, although some of it is very static… the basic concept of time value of money is going to be there for basically forever and I won’t need to change the actual examples in those slides really ever… but a lot of the other material changes very rapidly. So, the notion of what does market efficiency mean? who can beat the market and win? what is the evidence? But for topics like market efficiency, for topics like financial markets, for topics like investment banking, those areas transform rapidly… and so I’m continually changing those lectures year after year because one of the big topics that I cover in my course that is not really traditional for an undergrad business finance class is how firms raise external capital. The reason why is because it’s one of my main research interests, and so I have lots of ideas of what I want the students to know, and lots of research that I can tie into the lectures that I give. For the past five or ten years financial markets, the regulation of financial markets, firms’ ability to raise external capital, has changed tremendously as regulation has changed and so I’m continually revamping that portion of the course basically every time that I’m breaking it out.

Rebecca: One thing that I can’t help but think is that you and John have some similar backgrounds in terms of content, so the time value of money that you keep referencing probably makes sense to you but I feel like our listeners might not know exactly what that is so maybe we should just take a minute and give a quick cap of that so people know.

Alex: Yeah, absolutely. So, if you have $100 and you invest it earning an interest rate of 5% per year, in a year how much money will you have?

Rebecca: One hundred and five?

Alex: One hundred and five dollars.


Rebecca: It’s not a trick question, right?

Alex: No, No, it’s not…

… and so, as you invest money it earns some rate of return and so then money in the future you have more, because it earns some positive rate of return, and conversely, would you rather have $100 today or $100 in a year? Well, the answer is $100 today, because you could invest that for that year and have the hundred dollars plus some additional return. So, you’d have a hundred and five dollars one year from now instead of the: “oh, I’m going to give you a hundred dollars a year from now” and that’s all the time value of money is.. and then it’s application of that over multiple cash flows and multiple periods where people start to lose track that it’s just math.

John: That’s the fun part.

Rebecca: Yeah. Yeah.

John: I also ask my students have they ever burned their mouth on a slice of pizza, and why don’t they just wait? ..and it’s one way of introducing the notion that we prefer things now to later. It’s a fairly important concept in economics and finance and it’s at the basis of finance.

Alex: Oh, absolutely… and I like the way of framing it as how patient you are. Are you willing to wait or are you impatient? …and so the way that I sometimes describe that in class is if you have a low discount rate that means that the future is worth about the same as the present and so you are patient. If you have a very high discount rate well then the future and the present are very different, so high discount rates → impatient, low discount rates → patient.

Rebecca: You realize that the irony of this whole thing is that I have these conversations with my husband all the time because he’s in finance, right? Yeah….


John: Ok, well, does that help explain it?

Rebecca: Yeah, yeah. I’ve had it explained to me many times.


I figured our listeners would need it.

Alex: Well, since we’re talking about the things that our students do, I’ll just share one example that I use in class that I continue to use even though it completely does not work with undergraduate students. It’s when I’m trying to introduce the concept of sunk costs… and so a sunk cost is some amount that you have paid… a cash flow that has happened in the past perhaps… and so once that is paid, you can’t get it back …and so it’s like the notion of “should he throw good money after bad” is another way of phrasing it… and so, what I used to say is, “Well, you go to a movie, you pay your fee to go in, and you decide it’s a terrible movie, should you keep watching to get your money’s worth or should you leave? and then a student pointed out to me one year that well, actually, if you go and you complain you can get your money back from the management.


Okay, different example. This is the one that never works on the undergrads. You go to an all-you-can-eat buffet and you have a choice of do you buy one plate for this amount or do you pay a little more to get the all-you-can-eat buffet and you decide to go for the all-you-can-eat. You pay the $10 for the all-you-can-eat instead of the $7 for the single plate… and you go and you fill up your plate and you eat… but you decide you’re full after just your first plate… but you’ve already paid that all-you-can-eat price. Should you go back for more? …and the intellectual scholarly answer is “Well, no, because the marginal benefit of eating more food is negative at this point because you’re full. Even though the marginal cost is zero.” But, for the students, the undergraduates, the marginal benefit of eating more food is always positive.


They view it as a sport. Dinner’s not over until they kick me out.

Rebecca: I think that’s really important to consider your audience and what works. So, I think that’s a really good demonstration of knowing your audience and why something might not work for a particular audience.

Alex: Yes, nonetheless, I still use that example… but I tell them ahead of time “I know this is not going to make sense to you. You’re going to push back on me. Nonetheless, I want to talk it through with you so that we can think about who has marginal benefits and marginal costs of what amount.”

John: Just last semester I used exactly the same thing of asking them how many of them would leave a movie theater if it was clear that they weren’t enjoying the movie and this time I had some people argue “I do that all the time.” So, yeah…

Rebecca: Movie theaters? Don’t you just have Netflix?


John: Okay, that’s actually a relevant point. In terms of the Princess Bride, we’re moving into a generation where many of your students might not have seen that. Do you have any other movie references that you might substitute in the future?

Alex: You know, I haven’t found the right one yet and, honestly, even if I found a good substitute, I’d really want to keep the Princess Bride, just to give them the incentive to watch that movie.


It is such a classic and such a gem. I really would like for them to all see that movie.

Rebecca: It’s an investment you’ve made.


Alex: That’s right.

Rebecca: How have faculty responded to the methods that you’ve been using?

Alex: Faculty are not surprised that I do the things that I do.


John: They’ve known you for a while.

Alex: They’ve known me for a while and I really lean into the corny pun bad jokes big time in class…. and I really I’m quite shameless about it.

John: That’s not new… I seem to remember that back at Duke when you were still in college.

Alex: …and it’s just a question of do I try to suppress that innate desire or not?


…and I’m at the stage in my career where “nope, not anymore.” You’re just gonna let it all out. My colleagues are not really surprised, but interestingly, one of my colleagues has adapted her lectures quite a bit to embracing this notion of adding images into the slide decks… and that’s my wife. She teaches business law and regulation of business. She was an attorney by training and I described her as a born-again economist because when she started teaching regulation of business she had to teach herself basically all of public choice economics. So, she got a couple of high-level textbooks and worked through them all but she and I regularly discuss teaching techniques and so she now has gone down the rabbit hole of finding that perfect image to highlight the point that you want to make… to have that really stretched metaphor that you can then call back throughout your course. So, it’s been a lot of fun to have her as a sounding board to go back and forth with.

Rebecca: I appreciate that you’ve brought design into the process.

Alex: Yeah.

John: Excellent. It’s something we all should probably do more of and think about more and certainly much more effective than those PowerPoint bulleted lists that are so common.

Alex: Yeah, a little bit of both actually goes a long way.

Rebecca: You’ve won some awards for your research. How do you maintain a balance between teaching and research?

Alex: It’s difficult. There are some ways in which research can feed directly into the teaching… and so my research that touches on market efficiency, some of that can come into the classroom. My research that touches on how firms raise external capital, some of that can come into the classroom. Now when I teach my PhD courses, those are heavily flavored by my research interests and preferences, but when I teach the undergraduate core finance class there’s not a whole lot that can flow back from that teaching into the research. So, that’s one of the downsides of teaching that class as opposed to perhaps a class that is more specialized or an elective or something that’s a little bit further downstream from the core class.

Rebecca: But at the same time, if it’s something that you teach routinely then the prep isn’t as difficult.

Alex: Absolutely. That’s true. It is hard to overstate how useful it is to teach a class a second time, or a third, or a fourth, or twentieth. Nonetheless, every single year, even though it’s the same class and mostly the same topics, I go through every slide, every lecture from the beginning every year.

Rebecca: But, I remember teaching a bigger selection of classes and one of the things that I’ve liked about my position at Oswego is that that suite of classes has gotten smaller… and then there’s a little less I’ll keep on top of to make sure that you have all that fresh information and what-have-you for classes.

Alex: Absolutely, and I’ve done the same thing. I’ve taught a variety of courses over the years and it’s been nice… as you describe it that suite of classes narrows, so that you have the same core group of classes that you’re teaching over and over again and you can start to specialize. You can really invest the time to get over the fixed costs of finding all those right images for the slides… to going through and taking time to invest in the design aspects of the lectures. That if you had four different courses every term, it would be incredibly difficult to find the time to do that effectively.

John: While you’re working with PhD students, you must do quite a bit of mentoring of them. How do you see the role of a faculty member as a mentor for graduate and undergraduate students?

Alex: Faculty vary widely on their views of how much mentoring PhD students should have… and so you have one model where it’s sink or swim… the PhD students are some of the smartest people in the world… they’re good students… they’ll figure stuff out… just point them in the right direction and let them go, and they’ll get there. That’s not the view that I subscribe to. Because I think we frequently overestimate just how much the doctoral students know, particularly about how the profession works. There’s no book for that. They can’t just go down to the library and find a textbook on how to be a good assistant professor or… there are books on how to write a dissertation but that only gets you so far, because it really needs to be very field specific. So, I tend to go very much the other way, which is a lot of sort of high-touch mentoring. I write co-authored papers with many PhD students, one of whom is now your colleague there at Oswego in the Finance Department in the School of Business, and it’s enjoyable for me. It is a good learning experience for the students and I think it helps them to learn how the profession works much more efficiently, because when it comes time to write a paper and they might put together some tables and say I want to structure the introduction this way. Oh, no, no, no, no, we can’t do that… because if you do that then it’ll make people be concerned about this issue here. So, instead, we need to twist it around this way and start with this… start with the big picture, not what your paper does but what your paper’s about. That sort of thing. That’s hard to learn on your own.
PhD students are PhD students because they are extraordinarily good students and they’re really good at learning. Though, that’s not the job for academics. The job is not the learning, the job is creating knowledge… and the transition from being a consumer to a producer of knowledge is scary, and it is the road that has very few signs or roadmaps to help them get down. It’s a transition of going from a consumer to a producer of knowledge… is very profound for a lot of people.

Rebecca: What’s interesting about what you’re hearing is you know my field the terminal degrees in an MFA a Master of Fine Arts, and the undergraduate degrees are really professional degrees. But, it’s the creators of cultural content ,and so that struggle happens at the undergraduate level too, of going from being that consumer of culture to a producer of culture. It’s really not that much different… just what they’re creating is a bit different.

Alex: That’s right.

John: I remember when I was working on my PhD, one time, where up until that point I had been meeting with my advisor every month to talk about my research, and at first I was just asking him questions… and then he was asking me questions and I realized suddenly that I knew more about the topic than he did… and that’s I think that sort of transition that’s sometimes difficult… because when you’re working on your research you’re mostly going out and finding all these earlier studies and so forth but you get to some point where suddenly you become the expert in the field and that’s a tough transition to make. It’s scary, as you said.

Alex: It is. It’s quite the watershed moment when you realize… when you are presenting your research to a room of 30 presumably learned scholars that maybe collectively they know more about the topic than you, but you know more than any individual person in that room… and becoming that expert and then owning it, so that you can write confidently is, I agree, a very tough transition

Rebecca: I like the emphasis on the owning it part. I think that’s key.

Alex: Yeah, imposter syndrome is real.

John: Everywhere.

Rebecca: So, we usually wrap up by asking “what are you gonna do next?”

Alex: Well, that’s a good question. I really wanna keep doing what I’m doing, making my class better year on year, teaching PhD students and training them year on year and working on research, hopefully research that people will actually find interesting and useful.
I have a sabbatical coming up.

Rebecca: When’s your sabbatical? What’s the countdown?

Alex: Well, the sabbatical is a year from now and in between then, one of my colleagues with whom I co-teach a doctoral seminar, he’s on his sabbatical. So that PhD course, that normally I teach half of, I’m now teaching the entirety of next fall. So, I think the first thing I’m gonna do is prep the rest of that class.

John: Excellent.

Rebecca: Deadlines make a difference.

Alex: Yes, indeed.

John: That procrastination thing… and that time preference…

Well, thank you!

Rebecca: Thank you so much.

Alex: Thanks.


John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

29. Learning about learning

Learning is hard work. The most commonly used study techniques often provoke the illusion of knowing. David Parisian, a member of the Department of Curriculum and Instruction at SUNY-Oswego joins us in this episode to discuss how he helps students overcome their misperceptions by introducing them to the science of learning.

Show Notes


Rebecca: Learning is hard work. The most commonly used study techniques often provoke the illusion of knowing. In this episode, we discuss one faculty member’s success in helping students correct misperceptions by introducing them to the science of learning.

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today our guest is Dr. David Parisian from the Department of Curriculum and Instruction at the State University of New York at Oswego.

Rebecca: Today our teas are…

John: Barry’s Irish tea.

David: I’m not drinking tea today.

Rebecca: No?

David: Nope, I’m trying to cut back. [LAUGHTER]

John: With some people, it’s an addiction.

Rebecca: I have a problem. I’m drinking Prince of Wales tea today. Could you tell us a little bit about what you teach and how you became interested in incorporating evidence-based practices in your classes?

David: Well, I teach a few different courses. When I’m on loan to the Computer Science Department for the CSC 103, which is “Computer Tools and Informational Literacy for Educators” and then in the teacher prep program, I do ADO 394, which is “Interdisciplinary Methods” and then EDU 303 and oversee the “Block One Practicum” students, and do the online EDU 430, which is taking concurrent with their student teaching.

Rebecca: What is the 303 and the last one you just mentioned?

David: 303 are the “Block One Practicums.” So, their first semester junior year, when they entered the the block sequence for education, they have to spend time in a classroom. So, Field Placement secures their placements and then they spend a semester observing and helping where they can, but they’re getting their feet wet of being in a classroom. And the EDU 430—the online course—is “Professionalism and Social Justice” and that’s an online course that they take concurrent with their student teaching. So, a lot of the modules that they’re based on actually helps them in preparation for the edTPA that they have to submit for certification.

Rebecca: So how does the evidence-based practices fit into these courses that you’ve been teaching?

David: Well, one of the things that happened is the CSC 103 is designed for freshmen or transfer students that come to us for teaching certifications. So, one of the things we look at ishellip; approach that class from how technology is integrated in education. And one of the thingshellip; there was the book-read we had a few years back by Peter Brown, Make It Stick, kind of reaffirmed my doctoral work as we went through it… and I created a little quiz… a little matched-pair quiz… so that if they had to learn effective means… “does repetition build memory or does retrieval build memory?” and taking the material from the book and organizing this quiz… and what I found out was that all my students failed. So when it came to the 12 Principles within the book that we were testing on, less than one percent passed… which meant that a lot of the students we see coming to us, even though they can come to college, don’t have the knowledge or foundation of what strategies are most effective to learning. And that, in a teacher preparation program, my thought was, “do I continue that propagation through the pipeline or do I try to stop this stop that flow of students who are not effective or aware of the most effective strategies as they go out to be teachers or do we want them to have those effective strategies so they can implement them once they start teaching?”

Rebecca: So you staged an intervention?

David: Yes, it really becomes an intervention at that point and it’s really interesting because they have so many misconceptions coming in. And what we try to do is… they change their relationship with how they view sleeping. We set it up within the course so they take Barbara Oakley’s MOOC Learning How to Learn. So, they’re taught how sleep flushes out the chemicals overnight that are built up in the brain. They talk about procrastination and the Pomodoro techniquehellip; the benefits of flashcards and how to integrate that in spaced practice… and that’s one of the interesting thingshellip; I was just talking students today because they’re preparing presentationshellip; but a hundred percent of the class knew about spaced practice but no one knew how to implement it, or what it might look like.

Rebecca: Right.

David: So we provide opportunities for them to explore those areas, whether it’s creating flashcards where they’re using a flashcard app or have them making flashcards to build in the spaced practice to know when to practice and how to interleave and all those components that research has shown to be most effective.

John: Many of our listeners are familiar with all of these, but could you go back and talk just a little bit about the Pomodoro Technique.

David: The Pomodoro Technique is really fascinating. What it is… is for those who procrastinate, you block out 25 minutes …and actually the Pomodoro Technique is based on a tomato kitchen timer… where you set it for 25 minutes and then at the end of 25 minutes, you take a five-minute break… and then you set the Pomodoro timer and you go through it again. And what happens is you can begin to measure your workload in terms of how many Pomodoros it’d take. For the students, what I notice is they start changing their perception of assignments from a product driven “what’s the end product?” to a process of “how to get to that end product?” So many have commented on that and you can get apps on your phone that are the Pomodoro Technique, that will set up a 25-minute block. There are apps that will stop any notifications for that time so you don’t have to be keep looking at their phone because that’s the distraction that I’ve noticed, is that people are attached to their devices and have to be aware of every beep and every notification and everything that comes as opposed to blocking time. So the Pomodoro Technique has been pretty effective and most students have implemented that in terms of working on their procrastination skills or adjusting… changing behaviors.

John: So why do students have these misperceptions?

David: Good question. I come from a K-12 environment. I was a secondary science teacher for 18 years. I was a district administrator for another 12. So, coming from the K-12 environment, what my message was to them is that you were taught by loving, caring educators who were passionate about what they did and did the best they could with what they had… based on the information. That might not have been entirely best practices of what we now know about neuroscience and how the brain learns and the effective study strategies on the materials from Make It Stick. So, I think they grew up with whatever progression they went through. One thing I did notice is that students who struggled… that worked hard in school but just got into collegehellip; still struggle with the workload. What surprised me was talking to some studentshellip; is the students who were the bright ones… the ones that got it quick in school… that went through high school with no challenges… come here and all of a sudden, they’re placed in a situation where now they have to study and do all those things and they don’t have the skill sets to study. Just coming up through, I think our assumption is we feel that students know how to study because they’re in college and really what’s taking place is students aren’t really taught how to study. I think the assumption that students know how to study probably backs all the way down into fifth, sixth grade. I think we had some earlier comments where we were talking about us being in school and teachers made us write flashcards. We didn’t understand why and they might not have understood why, but we made flashcards. Now, as that is an appropriate practice, or one that works that you know, that can be integrated into a spaced practice and it’s really just teaching them a little bit about neuroscience… how the brains learns… how that all we do is encode, consolidate, and retrieve… and how do you build those principles and practices using technology… using skillsets… managing their time and trying to put that into a package where they can begin to see it. Because once they see it, then they want to improve their studies so they may begin doing it… so they go to bed earlier, they get better sleep and they begin to change their behaviors. I didn’t expect that but it was an awesome sidebar from that.

John: Part of it is, as you said, they haven’t always had much practice or training in learning how to learn and partly that may be because many of the teachers didn’t have the same…

David: Exactly.

John: …people have been just doing the same things that seemed right for an awfully long time… and one of the problems, though, with some of the evidence-based practices is that it doesn’t feel quite as good… because when you try to work on retrieval practice, after you’ve been away from a topic for a while, it doesn’t feel as good as perhaps repeatedly rereading something until it looks so familiar that you think you understand it. So there is that fluency illusion that people get that feels really comfortable and when you do some low-stakes testing or when you do some attempts at retrieval, you realize you may not understand it quite as well and it doesn’t give you the same sort of reward immediately. So it takes some training, I would think, for students to be convinced that these methods really do work. It sounds like you’ve been able to achieve that in your classes.

David: One of the advantages is it’s a setup so that we use the class and the content within the class as a training mechanism. So, they’ll practice working on a flashcard app to learn the app and then create flashcards for the content. One of the things they do have to do because the modules I set up are in a worksheet form so the initial encoding part is our instructional part. The consolidate part is them now going into another class and applying it in another class and showing the evidence in mine. So that they’re forced to, not only just use it in mine, but now go into a Psych 100 or a LIT course or whatever they’re in to create flashcards, create mind maps, begin to incorporate that… and we map out, When are your tests,?” So, how far do you already have to back up. So it’s really just coaching them on how to utilize those strategies and to get them so they can begin to internalize that to meet with their success.

Rebecca: Do you find that the students are a little resistant at the beginning or do you think because you’re providing the evidence and the science behind… the reasons behind… why you might use a method, is that what’s helping?

David: Well I think it’s a combination of both. Part of it is they have to go through the MOOC… so it’s not me presenting the science. Then we have a series of videos. There’s a gentleman by the name of Thomas Frank, who has a series of videos on studying and using it, so he has a more modern twist to it, being late twenty-something or just through college so it’s a more animated video. And then I’m in class to give suggestions, those type of things, but once they saw that on the first day school, they take a quiz and fail it and then realize that they don’t know what they think they know, then they’re pretty open to “this is what we’re gonna do for the semester.”

Rebecca: Sounds pretty motivating.

John: It is a motivating technique, yeah.

David: Well, I had one student once say like, “first day in college and I’ve already failed a quiz.” I’m like, “rather fail it now then in 15 weeks.”

Rebecca: Yeah.

David: …that’s the poise. It’s really just redefining their tool sets and introducing new ones… and not everyone takes to everything. Like, mind mapping is a classic example. whether it’s Tony Buzan’s method, but I’m very mind map oriented, but a lot of students have a negative experience towards it and I said, “if you want to be a teacher you may never do a mind map but you will know how to do a mind map, because you may have a student that needs to know how to do it and you have to be able to teach it so even if it’s not for you and you don’t like it, you darn well better know how to be able to help someone else learn it.” It’s redefining a different role for the teacher as our candidates come through as be more knowledgeable base, more pragmatic, and more understanding on what effectively works because they can then share that information with their students.

Rebecca: So what do we do for all those students who aren’t in your class in their first year, right, who aren’t becoming teachers but they’re, you know, trying to exist in this system, who might also have the same exact struggles that your students have? What’s your recommendation?

David: Well again, Thomas Francas has great videos and whether it’s how to read a college textbook, how to take flashcards, how to do any of that. That’s one aspect– knowing that you need help is another aspect. We have the success tutors that that have those skill sets, are using the same information to help other students. So, I think there’s resources on campus to support that.

John: …and our Student Success specialists or I do are very familiar with all these techniques and they work with students that are struggling to help build that up.

David: Right, and I think looking at it from a instructor side of the house, just as people become familiar with the strategies… is making those known..making the strategies known. If you’re teaching an Economics course or teaching whatever, if you say there’s a set list of vocabulary or concepts that you need to know, you know, put them in flashcards, go to these links, it’s not like you have to prepare everything because you can’t make meaning for someone else… they have to make it on their own… but you have to provide those initial concepts or the information because the student has to be able to take it in, but telling them how to take it in or just pointing them to links can facilitate that process. I don’t think anyone really comes here with a desire to flunk out, you want to be successful, but some of the times, they get paralyzed by the paralysis of the whole situation.

John: Some of it can be done through course design too, so that they are designed to include some level of retrieval practice,

David: Absolutely.

John: … some amount of interleaved practice,

David: yeah

John: ….and some spaced practice.

David: Yup, yeah, absolutely. And those type of structures in the redesign isn’t that large of a jump to do for the professional side of the house. The biggest thing I can tell for any student, the simplest thing to do if you’re feeling that stressed is breathe. It’s kind of funny… whenever you’re upset, what do people tell you to do they tell you to breathe and just from a physiological standpoint, taking deep breaths… slow deep breaths… once you start breathing about six breaths a minute, there’s a branch of the vagus nerve of the parasympathetic system of our physiology that automatically slows the body down. So, when you’re taking a test, what do you do? Breathe slowly. If you get stressed out on it, draw a circle around every fifth question so that every fifth question, you’re just taking three deep breaths just to say, “this is all right, we can do this.” And you that will calm the physiology and reduce that stress response of seeing an exam… the fight, flight, or freeze… and we’ve all experienced those… with varying degrees of success. [LAUGHTER]

John: But we want to set up a system where students generally be successful.

David: Absolutely… Absolutely.

Rebecca: So you’ve talked about the Pomodoro Technique… you’ve talked about breathing… you’ve talked about flashcards. What are some other key things that instructors could help students think about as strategies to be successful?

John: Well, you mentioned sleep, too.

Rebecca: Sleeping’s good.

John: And that’s something students often have trouble with.

David: Well it was interesting because there was a presentation done once and one of my students… I commented that she would stay up till 2-3 in the morning texting her friends and then they got that part of the MOOC and talking about sleep and she’s the one that started to go to bed at a more reasonable hour to be able to sleep. I think if one hasn’t read Make It Stick… that’s a great book and I think for a lot of us on the professional side of the house… as instructors we’re like the great white shark of our content, we’re the rogue, we’re at the top of the food chain. But one of the things, I think, the book does is lays out for you this way in which people encode information, consolidate, and retrieve it and I think having that as a foundation and then, reaching out to the center here for like, “how would you design something? I’m struggling with this” and just say “I want to reach the students” because I think a lot of professors do want to make those connections with the students. I think there’s help and support on campus to help people in designing those strategies. Personally I start with trying to give the overview… try to give the gestalt of the course… “what’s the wholeness gonna look like” and then just lay out the clear objectives and then integrate and make the assignments, being those flashcards and studying and there’s just so many ways to communicate, I guess. Even taking Barbara Oakley’s MOOC Learning How to Learn and just learning some of the basis from the neuroscience, even from the students. Have students get together informally and go through it. It doesn’t really take a lot, all the information’s out there.

John: And if I recall, it’s offered every month isn’t it? It’s a four-week MOOC and it’s offered very regularly.

David: Well it’s a four week…, like every week now, I think. For those who aren’t aware of the MOOC through Coursera, a MOOC is a massively open online course, Coursera is a vendor for that and Barbara Oakley’s MOOC. I think last year i read in the New York Times where it was the most popular MOOC on the planet.

John: It is i believe still the most popular MOOC…

David: I know when I took it, I took it with two hundred and seventy-five thousand other people at the same time. That was a few years back. Integrating that type of information and layering it into a course just gives a different feel for the students. And as I tell them even though they’re freshmen and transfer, I said “if you’re going to be a teacher, down the road, some of your professional development is going to be delivered through these.” So, we’re just getting them ready orientating them. Well there’s just you know some really simple information that people can do to design, redesign, and be more effective, more engaging and have students be more successful.

Rebecca: I’ve had students respond very positively when you explain why you might do a particular technique or a method. At first it seems like “why are we doing this? why are we doing quizzes?” but if you explain why and how it helps they’re much less resistant and actually embrace the idea. I’ve had students ask for more quizzes because they’ve learned how it’s helpful…

David: That just reminds me of, in our lesson plan development, as the students are going through it… one of the parts of the lesson plan development I have after activating prior knowledge is setting the purpose… and setting the purpose is “so, why is this lesson important? why do you need to know it? How many courses have we ever been in that no one’s ever told us why we’re there and you’re absolutely correct. Once you say the why and give the explanation, it creates an association or connection with the student and that’s one of those underpinning things… as the more associations you can connect to them, then the more apt you’re gonna have their buy-in to engage you in the content.

Rebecca: I think a lot of times students see assessments like quizzes and tests as some sort of penalty…put you in the penalty box or whatever… and that’s a faculty member’s way of torture or something, but as soon as you say that the purpose of doing this is to help you recall information and to make sure that you have that foundational knowledge, you can continue building in these more advanced classes. They stop seeing it that way and it’s pretty amazing that it doesn’t take much of a conversation… it takes having the conversation.

John: And the more frequently you do that, the easier it is for them to get past that because high-stakes testing is stressful, but if you replace it with lower stakes testing and more regular testing, it’s easier for them to see that they’re learning from this experience and it doesn’t hurt them as much if they screw up… that it’s an opportunity for them to improve and continue…

Rebecca: However your students responded after your class like moving into these upper-level classes because you’ve been doing it long enough now that probably some of them are now in those advanced courses I would imagine.

David: Part of it is coming up through the pipeline and having viewed or patterned assessments or quizzes as the “gotcha” and getting hammered. Ken and Rita Dunne, one of the things they stressed that I always kind of stress is “high content, low threat.” So, whenever you can engage and have your material be of high content but a low threat, students are more apt to engage because they don’t have that fear reaction going on and I think it’s changing that patterning by having the conversations that these are beneficial for you. So, that’s my thought on the quizzes. Some of them continue that patterning and that’s one of the things where I try to keep it going for the semester in hopes that they’re gradually continuing to do those processes, those strategies to continue their learning. The one thing about teaching the freshmen… I’m probably one of the few faculty members that have this interesting purview because I see them as freshmen or transfers coming in, I’ll see them as their block one junior EDU 303 practicum. I’ll touch base with them again for the ADO students that take 394 as their block two, so I have them as a class and than in block 4 when they’re seniors going through the the end of the pipeline. So I can see their growth along the way from that and…

John: And ADO is adolescent

David: Yeah, the adolescent, yeah… and that one they’re constructing the lesson plans, they’re trying to see how this all fits together, trying to pull on multiple layers. We revisit Make it Stick because in all my courses that’s just part of it. So, they’ll read the book as a textbook in that section… so then they can begin to refresh like “Oh, I remember when we did this” and “how’s it going?” and then we have conversations on when they’ve been using it and how effective it’s been. I just keep trying and plugging away and stepping up to the plate…. every day you get to take a new swing

John: And the more people who do that in their classes and certainly the more people who are trained to do that at lower levels in elementary and secondary school, the better off students will be.

David: Yeah… especially nowadays when you you look at the various challenges in a k-12 environment. If you can begin to seed the ground with what works and just focus on that then we’ll be okay.

John: Do students go on and use these practices in other classes after they’ve been exposed to them in your intro level course?

David: Part of it, once they leave me is trying to continue that propagation based on the courses they’re in. Some of the strategies are more effective in some content areas than in others. Math is always a struggle in terms of looking at applying the flashcards while you can do color coding or dual coding where you’re including images or multicolored in the equations as you follow different variables through an equation sequence, those type of things. The other thing is that the strategies, and this might be one of the misconceptions students have, learning isn’t easy. Some people comment like “well if I just sit here I should get an A”…… but learning is messy… learning’s organic… learnings dynamic… and learning takes a lot of work and sometimes, depending on the student’s course load and what they’re taking… if you’re taking a 4-credit science course, you got three hours of lecture and a three hour lab somewhere in the week… having been a science major and having a lot of hours on a lab. So part of it is finding time to create the materials… the mind maps… the flashcards. Those students who have a better time management… work ethic, those are the underpinnings I think that makes this a successful student and that they put forth the time and schedule that to do all those things that are necessary, whether it’s creating a mind map or whether it’s creating flashcards or creating the time intervals for the spaced practice or when to do the spaced practice. I was talking to a friend who used flashcards and whenever she was grocery shopping for her family picked the longest line because it wasn’t about getting out of the grocery store fastest, it was about being in the line the longest as she pulled out her set of flashcards and reviewed them in the line at the grocery store. You can find intervals to do those type of things.

John: I always wondered why there were always more people in the longest lines. Maybe…Maybe they’ve been in your class [LAUGHTER].

Rebecca: Yeah… everybody’s doing flashcards.

David: Then again yeah, flashcards is just one aspect of that but you can integrate that from a quiz standpoint… from a retrieval…. and one thing that could be interesting is when you look at the research on flashcards, or how to create them, there isn’t the level of “how do you create” going down Bloom’s continuum of higher processing from a flashcard aspect. A lot of the information we see is low level…. vocabulary words, or those type of things… but how do you all of a sudden take two flashcards and compare them and say “compare and contrast these two concepts of something…” and so how do you get a bigger cognitive load going from using those… and the designing of flashcards… that’d be a great study for someone to do. I’lljust put that out for anyone who’d want to. [LAUGHTER]

John: You get to work with students a bit later in their academic careers that you worked with earlier. How do they respond when they come back in upper-level classes? What do they say about their experiences?

David: Usually, semesters later they forgot about me. [LAUGHTER]

Yeah, but one of the things we try and do at the end of the course is they have a five-minute presentation they have to do… and I give them the slides and so it’s like “how has this changed you?” and they go through and reflect on that. So, I give them a template of what the presentation is and it’s their five minutes of fame where they get to begin to find their teaching voice, and it’s the first time they’re in front of the class talking about it. So, you talk about how they how they did it. In terms of seeing them later on, the people who use flashcards and grew up continue using them and then you have various levels of people who took the buy-in to create those processes. The other thing is you try to encourage them to use it, so as they’re developing their mini unit you have them do the flashcards to go with the unit… you have them do the mind map. So, you have them go through it, and I think from that aspect, they recall fairly quickly what it is they needed to do to generate it… and then it reminds them like “oh, yeah.”

John: But, if we do this in more classes and we use them or we structure our classes so that students naturally adapt some of these practices, it’s going to help reinforce these things… and the more people are reminded of how important this is and how their usual practices may be really helpful in cramming for a test the next day but aren’t going to allow them to remember the things much past that day.

David: Right. In the real world you have to remember those things past that day.

Rebecca: A couple of the themes that you were mentioning most recently is about time management and the work ethic component fitting into this and so it seems like that’s the next discussion. How do you make sure that students know what is a good work ethic? Do they even know? Do these conversations happen? I’m not sure that they do.

David: You can see those students that have work ethic pretty quick …and I’m just trying to flashback through courses and images through my head of students… from a freshman those that are asking questions… those that are getting work done and turning things on time… those who are turning it in early… those who who show up to class earlier… and sitting there… they have a certain level of comfort within themselves… where they’ll ask a question. Truth be told, I didn’t feel comfortable asking a question in college till after seven years of college… not that it took me that long to get a bachelor’s degree… it wasn’t till after I did my student teaching a long time ago… and after I did my student teaching I learned how to study. I did the outlines before the chapter. The following semester I went for a neuroscience certification taking our geology courses. So, I outlined the chapters before I walked in there. I pre-taught myself the material. I laid out all my notes and stuff before the lecture… had the conversations because then it was internalized to me that this was important… and I think until someone gets that into their intrinsic fabric of themselves… where they want to take this as being important… and I still don’t know what I want to be when I grow up. I think that’s the big thing. People who are serious know to get the job done. They’ll do the work and they’ll do it to what they need to do… and if you give them criticism and feedback, they correct it.

Rebecca: I’m hearing a growth mindset described.

David: It is.

Rebecca: …you’re open to feedback… The the real challenge is how do you get students with the fixed mindset, who maybe don’t have that work ethic. or good time management skills. to get on board.

John: …and you mention that case about some students earlier who had done really well in high school and then suddenly struggle when they get here. Those, empirically, tend to be the students who have a fixed mindset… where they’ve been successful with the techniques they’ve used, but once they have to move into a new environment or they have to engage in more transfer than they had to earlier… suddenly face some struggle and it’s a shock to them and they often give up.

David: Well, and one of the interesting things, though, and this they’ll need to be reminded of, but within all of us we have growth and fixed mindsets.

JOHN. Yes.

David: So, I could have… I’m not picking on math but I’ll pick on that…. Most students… math is not a strong aspect, because really the only time you do math is in math class. No one asks you when you’re walking down a street to factor a quadratic equation, right?

John: Well, rarely… [LAUGHTER]

David: …rarely… but you consider a musician. They know with practice they get better. In sports, you know with practice you’ll get better. In math, if you practice you’ll get better. So, part of that is understanding the context of which it is. I may struggle in math but I might be a musician in the band, and so I know. So, you have to kind of transfer where they have been successful in showing them, in this part where they’re not being successful, how they can be. Because, if you can tap into a person’s growth mindset… and it could be in a K-12 environment. You have somebody I saw students riding their skateboards… phenomenal skateboarder… can do all these tricks… will spend hours learning a trick, right? That whole idea can be shifted to their studies as well. How long did it take you to learn this trick?

John: How many times did you fall you know along the way.

David: …and how many times did you fall? The culture is changing where, not only are we imparting the information to the students, but we’re also being their coach. We’re trying to nurture them. We look at them as adults coming here… without the parents for the first time. The baton we’re handed is actually trying to nurture them into the adult working life and understanding how we learn… how we process information… how we interact… the building of rapport. how do the rapport aspect is all part and parcel… I feel… what we need to do… or what we do… I know it’s what I do.

John: How did you integrate the MOOC into your class?

David: I build the class around modules that last two weeks. So, with the beginning of class, I took some of the material from the MOOC or what the topics were and then I created worksheets based around that. So, if it was procrastination then there’s articles that they were reading in, and picking up on, and getting their takeaways… So, that part was teaching the foundational aspects of of how these strategies work, and then giving them time to practice and doing them within the two weeks. The MOOC they can view offline. They can take the quizzes. The other thing I did, is with Barbara Oakley, she had Coursera set me up as an administrator for my course. So, then I could just upload my class list to it and then it would keep track of the quizzes…

John: Oh, nice.

David: …and then I could download the the grades or whatever. So, she was giving him the content on one side; we were building in related practiced and article support on the other side. Then the consolidation part… and I broke it down into it an encoding section of the worksheet, a consolidation, and a retrieval part basically patterning our learning process. So, watching videos and reading we’re encoding… applying the material was the consolidation… and then their reflection and the reflections based on making a video, responding to an interview question, or reflecting on their experience over the two weeks… and they were able to communicate those. So, part of it is just finding out what the MOOC is doing, getting materials that kind of pattern that (that’s where I brought in you know the Thomas Frank videos and other support materials).

John: The learning scientists also have some good ones.

David: Yeah, exactly, and I use a lot of the Learning Scientist’s material and McDaniel’s site deals with retrieval practice, so there’s a variety of things. We try to overlay the MOOC with Make it Stick and strategies there to create an environment that over the first four weeks they’re experiencing… they’re learning… and they’re beginning to apply… and then as we build out the other modules, we still keep repeating… For example this is module four we’re in and one of the things I’m training them in is advanced search strategies. What are the topics they’re going to be searching? Well, there’s eight setups within the lab so… elaboration… retrieval… spaced practice… those are the deep research things. So, each group has to now prepare a presentation, but they have to do the research. What’s the research that supports this? and what are strategies to help? So, now there’ll be eight presentations of the 8 strategies that they’ve learned. Trying to deepen and make touch points through the semester and keep reminding them… constantly reminding… constant… constant…[LAUGHTER]

John: So, what are you going to do next?

David: Well, next will be a continuation with the Computer Science…. Looking for more in-depth application across the content areas to help students… and then professionally, this summer, working with Educational Administration Department and their Project Blend Symposium. We’ll be doing the third installment of Resiliency and Leadership working with the Institute of HeartMath. So, outside of that, we’ll continue to work in those areas of heart-brain synchronicity and just working and having fun.

John: Thank you for joining us. It’s been a pleasure.

David: It’s been my pleasure. Thank you very much.

Rebecca: It’s always great to hear what you’re doing in your classes and the results and thanks for sharing that for everybody else.

David: Very good, thank you .

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

6: Evidence-based teaching in large classes

Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.

Show Notes

  • Walstad, W., & Allgood, S. (1999). What do college seniors know about economics?. The American Economic Review, 89(2), 350-354.
  • Cabane, O. F. (2012). The Charisma Myth. Master the Art of Personal Magnetism. New York: Penguin Publishing.
  • Chew, Stephen L. (2010). Improving classroom performance by
    challenging student misconceptions about learning. APS Observer, 23(4).
  • Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The physics teacher, 30(3), 141-158.
  • Learning Scientists Blog
  • Resources for Economists on the Internet
  • Schwartz, Daniel L., Tsang, Jessica M., and Blair, Kristen P. (2016). The ABCs of How We Learn. New York: W.W. Norton and Co.
  • Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science, 323(5910), 122-124.
  • Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE-Life Sciences Education, 10(1), 55-63.
  • The Teaching Professor
  • Wieman, C. (2007). Why not try a scientific approach to science education?. Change: The Magazine of Higher Learning, 39(5), 9-15.


John: Our guest today is Bill Goffe, a Senior Lecturer in Economics at Penn State and a former colleague at the State University of New York at Oswego.Bill: is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists and it’s now hosted and sponsored by the American Economic Association. He is the Secretary-Treasurer for the Society of computational economics , an Associate Editor for Computational Economics and the online section of the Journal of Economic Education and he’s also an editorial board member for Netnomics.
Welcome, Bill!

Bill: Ah, thank you.

John: It’s good to talk to you again.

Rebecca: Yes.

Bill: It is.

Rebecca: Today our teas are are…

John: Are you having any tea today,


Bill: Any tea today? No, I am not. I just had some hot chocolate today, however.

John: it sounds like maybe that might be good for your cold… with some honey and so on.

Bill: I think it probably would be.

John: My tea today is ginger peach green tea.

Rebecca: … and I have Lady Grey.

Bill: Ah, enjoy.

John: You’ve been teaching large economics classes for quite some time, beginning at the University of Southern Mississippi, here at the State University of New York at Oswego and now at Penn State. How has your teaching changed and evolved over the years?

Bill: I think I’ve become much more reflective and thoughtful about what I do. It’s odd that the more you study this, the more you realize you do not know, which maybe is not surprising. I oftentimes now look at what’s going on in cognitive science research as well as in STEM education research, particularly physics education research, and apply to my classrooms.

John: …and I know you’re active on a number of listservs online in terms of following what’s going on in both economics teaching research and physics research. What are we doing differently in economics or what are we not doing?

Bill: I don’t think we’re very thoughtful if our students aren’t learning. I would guess that most economists would tend to blame the student rather than the instruction and careful thinking about things. There’s a paper by Hestenes, a physicist, where they develop a so-called concept inventory, looking at fundamental understanding of their students, and they very bluntly say that “our students didn’t learn very much.” The few that really did was by happenstance. Economics is not at that point.

John: I remember one time we were talking about a conversation you had coming back from a conference with someone from a military academy who was talking about their reaction to student learning where they found their students weren’t doing very well. Do you remember that? This was a few years ago.

Bill: Yeah. Yes, I do. It was fairly formative for me, He was flying from Colorado Springs, the Air Force Academy, back to Rhode Island for a conference and he was grading a test, his biology, genetics, or something. He noticed all his students missed a certain question, and I would tend to think (at least back then) that that’s the students’ problem, forget about it. …and it dawned on him that “oh, I mustn’t have taught this very well.” That was really an eye-opening story for me.

Rebecca: I think it’s really interesting that, you know, some of the best teachers are probably some of the most reflective teachers, and take the time to sit back and continue to try to see what they could do better, and take the time to learn new things, and read up on these things. So it’s it’s always exciting to hear other faculty members who spend the time to do this.
I’m wondering…. large classes are something that are pretty common in economics, but maybe aren’t in some other areas, but large large classes present certain kinds of issues. So, can you talk a little bit about how, some of the evidence-based practices that you’ve been studying, you’re implementing in these larger classes that might be a little intimidating for people to think about.

Bill: Several things, one is peer instruction, sometimes called think-pair-share where students will be presented with a difficult question. I tend to aim for questions that maybe half the students would get right, as students respond individually with their clickers. I then encourage them to talk to each other to: “convince them of your answer.” Typically, I see about a twenty, maybe thirty even, now and then a forty point improvement on number answering correctly, and then I actually talk out the answer as well. So they get correct answer oftentimes both from their peer as well as from me. There’s a paper by Michelle Smith. She is an educational researcher in biology. Actually you might want to look into her. She’s usually at Maine she’s on sabbatical this year at Cornell fairly close to you.

Rebecca: Wow, that is really close to us.

John: Speaking of Cornell there was a podcast issuing from Cornell, the Teach Better podcast…

Bill: Yes.

John: …with an economist actually as one of the co-hosts, and on this week’s episode they were talking about the use of concept tests and noting that we really don’t do that in economics, other than the Test of Understanding in College Economics (or the TUCE). Do you know if anyone’s been looking into, or developing, or thinking about developing concept tests for economics.

Bill: Yes, I know several. It might be worth taking a second setting back. In most of these concept tests, they have a good idea of students’ preconceptions and they test against that. We tend not to know what the students’ misconceptions or preconceptions are, but I’m not sure everything we do in economics would fit that model, if you will. Certainly students might have wrong preconceptions about how to calculate the unemployment rate, but something like aggregate supply and demand, I’m not sure they have many preconceptions there…. and I would add that Michelle Smith, who you listened to on that podcast, she’s the one I mentioned who is at Cornell. As far as people working on suc concept tests, I know of several. The projects are pretty not… not speeding along very quickly I think it’s fair to say.

John: One of the ways in which you can get some information from students on preconceptions is with clickers. Is that part of the way you use it? To test for common misunderstandings and help them get past that?

Bill: Yes, indeed it is. I sometimes test for misunderstandings first with so-called JITTs where students answer ahead of time in our LMS. We happen to use Canvas. I’ll give some essay questions on a given topic and I also ask them what they find confusing or puzzling in that set of readings, and from that I’ll base a clicker question on it. One example is students tend to think that roughly a third of workers earn the minimum wage, when in fact it’s maybe not even a tenth of that, and that’s a sort of misconception I actually do address with clickers with data from the so-called JITTs.

Rebecca: In a… in a large class setting like that where you’re using little essays in your course management system, how many how many students are you doing that technique with.

Bill: Quite a few. I have two sections of about 300 students, give or take, and in both those sections I have six graders. So each graders grading about 50 different students’ material. Works out fairly well. They typically have about one thing to grade per week.

John: So, JITTs stands for just-in-time teaching, I’m assuming.

Bill: Yes.

John: How do you go about doing that?

Bill: So, it’s assigned before class, thus the just-in-time. However, just before class and then base class upon that. To be truthful, though, now teaching this same cohort ,the same group, I don’t see that many surprises anymore… but I do review them just to remind myself of where they stand.

Rebecca: So you mentioned graders and also teaching assistants. Can you talk a little bit about, kind of, the helpers you have in the classroom and how that helps facilitate learning, and some of these techniques that you’re implementing in your classes?

Bill: Sure, so for each of these 300 student classes I have one graduate TA, he or she does administrative things. I have about four in-class learning assistants. Penn State’s a hotbed, if you will, for learning assistants undergraduates. Our College of Science used about five hundred this semester in a number of different science courses. It’s a real showcase for Penn State science. Then I also have about six undergraduate graders per class who do the grading. I joke I become a middle manager.

Rebecca: Do these student helpers help with class motivation or anything during the actual class period? or just kind of this… these outside more administrative activities?

Bill: Huh, good point The learning assistants, they go to every class. They write up a summary for me about things that went well… things that did not. You know it’s hard to get feedback here teaching and they’re a natural. They know the material, but they’re also students… and then we do a peer instruction, something like think-pair-share. We all wander around the room to get a sense of students’ understanding or exactly what they’re not understanding with these challenging clicker questions and they report back to me both during class as well as after class. It’s a very common sort of a STEM education technique.

John: How do you keep that many students’ focus? I know when I teach large classes there’s always a few students who want to chat a little bit in the back and that can be distracting.

Bill: Sure, there are several things. One is the first day of class, I do a quick assignment, I guess you could call it. I ask the students to take out a piece of paper and write on that paper what should be the classroom rules so this class runs smoothly and then they develop the for coming to class on time, not talking, and so forth. So when the student is talking, I can say: “Look the people around you don’t want to hear you talk,” because we’ve gone over that data from that first day survey. About 1/3 of the students say talking is disruptive, so I can use a student’s voice channelled through me to get quiet. I actually work on my public speaking quite a bit. I’m a member of Toastmasters. You might remember when Ken Bain came to talk at Oswego, this was maybe eight years or so ago. I’ve never seen someone command a room like that. I’ve been in rooms with two U.S. presidents and he used a lot of public speaking techniques to command the room, and I do that as well. Not as good but still, it’s helpful. Finally, I have questions every day or two that hopefully are puzzling, that will challenge students with the current material, to make it as interesting as macroeconomics can be.

John: ….and going back to Ken Bain, one of the main thing he’s argues for is for having these big questions that you organize a course, as well as individual class sessions on, and I believe you do that too in your intro macro class.

Bill: I do indeed. I ask two questions that we answer throughout the course: one is “why are some countries rich… other ones poor.” The very first day of class, I show a pair of videos: life in modern-day Australia, people living the good life (food, fashion, sports, nice cars, great life) and then I pair it with a clip from Port-au-Prince Haiti (an extremely poor place, shockingly poor) so it frames that question. The other question this semester is why are some people doing poorly an economy in spite of record low employment rates and GDP growth since 2009; in other words, why are some people being left behind? One of the things about that course is the current slow expansion, so thus we talk about the standard topics with aggregate supply and aggregate demand.

Rebecca: I know you’re a big advocate of evidence-based teaching methods, so I’m curious about how you help students make sure that they’re doing spaced practice and interleaved practice.

Bill: Sure. So rather than giving a couple of midterms and a final here’s how I break-up would be midterms into a series of seven quizzes. The first was in the third week of the semester, the seventh one is the very last week. they’re exam caliber questions, but they’re just split up, and so for any given quiz I can ask material (and indeed I do) from earlier in the semester. So that gets the spacing down. They cover a given topic: What is GDP? capital? unemployment? so forth, several times and then I always make sure to mix up all the questions so they’re actually interleaved.

Rebecca: Interesting, and that sounds like an interesting…. It’s always hard to balance, I would imagine in such a big class, like how to find that balance and communicate that to students. Do you take time in classes to explain to students why you do some of the methods that you do? To help them understand?

Bill: I do and actually what I do for that is I use a student’s voice.On the first day class ,I show student comments from my student evaluations from the previous semester, about using clickers like I do, about quizzes, and I let past students tell my new students on the first day of class about why these methods work. I think that probably helps with the buy in, in having a student’s voice rather than my own voice. I should add on the quizzes, we also go over them right then and there, so it’s example of so-called deliberate practice where someone’s trying something challenging, and then they’re getting feedback on their work, and hopefully that feedback helps them develop a richer mental model of the topic at hand.

John: …and the same thing occurs when you’re using the clicker questions, when you’re doing the think-pair-share type activity…. that you’re giving them immediate feedback, because sometimes some of them will go astray even though you see that improvement, that immediate feedback is helpful.

Bill: Yes, indeed, and that was prior to Michelle Smith’s paper where she found… they looked at peer instruction (just students talk with each other) or a clicker question (instructor explaining it} and she found much more learning when both things happened. I think what’s happening is two explanations probably come at a given topic from different directions, and help you understand things even better. There’s a fairly new book out by Schwartz and co-authors… I forget their names…. The ABCs of How We Learn. Schwartz is at Stanford, and they find when you’re doing analogies, two analogies is much more effective in teaching than one. I’m thinking that several explanations, as long as they’re brief, are more efficient, in terms of more learning, than just one.

John: Because students come with different backgrounds and different explanations can connect with some students better than others, and the more hooks they have the more elaborate the models they can develop.

Rebecca: Especially when you’re, kind of, meeting students where they’re at, and it sounds, it sounds like the opportunities that you’re providing for student… the student voice to come out is also helping the students connect and feel related to the material because if it comes from a peer it’s going to help them feel a little more related as well, I would imagine.

Bill: I think that’s the case. They feel connected, for example, I’ll put up some of their JITT responses in class when we’re just starting a section or I’ll say someone over here’s puzzled about this, let me address that… and I also get a lot of questions in class, which I think it’s fairly uncommon for large classes

John: I’ve noticed the same thing in the large class when you start giving them more interesting questions and you get them more actively engaged they’re much more likely to ask questions and participate in general.

Bill: Ah, good.

Rebecca: I’d like to shift the conversation a little bit to some faculty professional development stuff, because I think there’s a lot that you’ve already mentioned but also other things that I know that you’re doing. I’m really curious in your mention of Toastmasters and the public speaking and developing those skills. So do you have some advice for faculty about how to command the room or how to develop those skills to command the room?

Bill: Several thoughts there, one is there are Toastmasters clubs pretty much everywhere. They’re easy to join and almost every long-term member of my club who’s been there some years speaks better than almost any academic I’ve ever seen and part of it is that it’s really helpful to see really good speakers speak, and then it quickly forces you to get better on the easy things to change: cut out the UM the AHS, nervous tics and things like that. It sounds a bit odd, but I just happened to pick up a book one day. It’s called The Charisma Myth, and it talks about how one can generate charisma in different situations, and a couple of chapters deal directly with the sorts of techniques that Ken Bain used. I could literally check off the things I saw him doing. I was just quite impressed with that book, The Charisma Myth.

John: So just like anything else, more deliberate practice can improve your skills.

Bill: Yes, very much so. Most things were pretty content with being kind of average. I suspect most of us are fairly average drivers. We don’t go over curbs, but we’re not driving stunt cars in Hollywood, but when you teach I think it makes sense to ask for more effort there and to actually try to get better at it. It’s quite rewarding when you do.

Rebecca: How did you get involved in studying teaching and learning? and what advice do you offer others who might be interested in doing their own studies?

Bill: Sure, it was about 8 years or so ago. Some friends of mine, Scott Simkins and Mark Maier sent me a paper by Carl Wieman and it had the title: “Why Not Take a Scientific Approach in Science Teaching” and it just opened up a whole new world to me. I don’t know if

John: knows this story, but I was thinking about perhaps leaving the profession at that point. I was kinda bored… research didn’t interest me that much…. and teaching was okay…. and this just opened my eyes to how we could teach better using principles from cognitive science. So that was the hook for me… was these methods other disciplines are using to improve learning for their students

John: might be familiar with the paper by Walston and Allgood in 1999, that it seems economic students really aren’t learning all that much. I saw a problem that potentially could be solved.
To get others interested then, maybe look at your local STEM discipline based education research group, a lot of universities have those. They’d probably be glad to have other people involved.

Rebecca: What resources do you use to stay up on the scholarship of teaching and learning.

Bill: I used Twitter quite a bit. I follow a fair number people on Twitter. I mainly use it for professional things. I saw a real interesting paper the day on metacognition. It was in the chemistry education research, on how with repeated practice on metacognition, that is students thinking about thinking and specifically thinking about performance on homeworks and exams, they can get better at it. So, I’m a big follower on Twitter…. also use the Learning Scientists’ blog. They have a very nice blog and Magna publications has a number of different publications, like The Teaching Professor. Those are mainly my main things that I follow.

John: Along those lines, we had a podcast that came out on Wednesday involving the online metacognitive cafe discussion forum, which is designed exactly to help students improve their metacognition. It probably wouldn’t work too well in a very large class, but in a smaller setting it can help students, and in fact when I use this, I use some of the Learning Sciences videos and some of their other materials. The developer of it was Judie Little

John: at Genesee Community College, and she used some other resources and she had some different issues that she wanted to focus on. Particularly she wanted to help students understand the importance of doing reading and providing more effort into their work. Then it evolved into more broadly an examination of how students could improve their learning skills, and that’s useful.

Bill: Oh, indeed it is, that’s very nice to hear.

Rebecca: Not surprising that in metacognition, like in other areas, practice is what gets us good at things, right? So, the more you practice being aware of what, you know, the better you’re going to be at it… and I think we can all be better at helping our students and guiding our students through those processes. You know, when I’m teaching Web Design I have those same issues that students need to troubleshoot and problem-solve and a lot of the times that they don’t really even understand what the problem is or where they where they’re connecting into or where they’re getting hung up…. and so coaching them through problem-solving strategies and helping them recognize strategies to do those sorts of things can be really helpful. So, I think the more we can all do more things that help… help students focus on that is… is a good thing to do?

John: Right before we started this I was in the midst of grading last week’s metacognitive cafe discussion forum and one of the prompts I used was asking students to reflect back on what we’ve done so far this semester and connecting it to their future career and their lives, and so forth, and asking what they’ve learned that was surprising or what they’ve learned that will be most useful for them in the future… and I was really surprised to see that probably about 90% of them listed the metacognitive cafe and learning how to learn more efficiently as the most useful thing they’ve taken out of the course so far. Many of them also mentioned economics, but it was nice to see that that is having, or at least they perceive it as having, an impact.

Bill: Oh, that’s very impressive. Congratulations.

Rebecca: Do you do any sort of mid-semester feedback to kind of know what students are still getting hung up with or how they can move forward.

Bill: Sometimes I did a mid-semester revaluation. I’m sorry to say this semester I did not. I’ve taken up a new online class that I had not taught before. It’s taking up a lot of time, so that was one of the things that just fell away.

John: But you do have the student learning assistants out there who are giving you feedback from their perspective, so you’ve got a near current student perspective, right?

Bill: Yes I do, it’s very handy. I just… as you get older you don’t understand students as well, I think. It’s very handy to have a translator,if you will.

Rebecca: It’s funny how they stayed the same age and you don’t, right?

Bill: Yes. Yes, this is really not fair.

John: Well, I’m not sure it’s just due to getting older because when we talked to faculty a lot of them are talking about how…. even new faculty were saying “I was never like that as a student” and in fact they’re probably right because they… the people who choose to become faculty are not random drawings from the population of students…. and if they think back carefully, many of their fellow students were probably very much like the students that we deal with.

Bill: Yes.

John: You’ve mentioned many times that you like to try new things every year. What are some of the things you’d like to try in the future to improve student learning?

Bill: I want to work on having students use better study skills. I think that’s one of those things I suspect, and part of this project actually collecting data, is improving student study skills. I’m thinking about using some of the materials from The Learning Scientists, the blog we mentioned earlier, and also do an in-class exercise a bit like Steve Cho does on illustrating to students how deeper processing leads to more retention. You might remember Steve Cho’s project. He has students do a word list where students either study it intently, determining if words either have a positive or negative aspect, or another group studies this words more lightly, if you will, they just count letters and those that process this more deeply remember their words more fully. That’s true, but doesn’t have much resonance with our students, because we don’t do word lists in what we teach. I want to do a similar project with reading economic content and then actually see if those who use better techniques, directed by me in this case, actually learn more than students who did not. This all take place in the classroom.

Rebecca: Sounds really exciting. I’m looking forward to hearing what those results will be.

Bill: Ah, well, thank you.

John: When Michelle Miller visited a couple years ago, she talked about how they redesigned the first-year program to include a component on improving student learning and… as an online blackboard module. I’ve often argued that perhaps that would be useful if we could do it for all freshmen here in some way so that students all get exposed to some improved learning methods. Not only are faculty sometimes unaware of effective practices, but students often use practices that are not particularly useful.

Bill: Yes that’s certainly true and it actually even after students who use these more effective practices they think they’re less effective than just rereading the highlighting.

John: Part of the reason is that the practices they like to use like cramming gives them some really good immediate effects but they’re not very useful in terms of long-term learning. So they get positive feedback when they use it but they don’t see the connection between that and not remembering the stuff when they get to the next course… and how can we break that down?

Bill: That would be hard. What I’m trying to do in this project I mentioned is show to students there in the classroom that these different methods actually do lead to more learning with real actual content so that’s like my goal and hope for this project.

John: We’re looking forward to hearing more about that.

Rebecca: Yeah, thanks for joining us! As always, it’s a pleasure to hear what you’re doing in the classroom and what you’re researching.

Bill: Well, thank you so much. It was great fun and I’m probably going to go buy some tea now, to be honest with you.

John: Try some honey, it’ll probably help you feel a little bit better.

Bill: Yes, I think so.

John: It’s great talking to you,

Bill: and I look forward to talking to you again

Bill: Great. Great talking to y’all. Thank you.

Rebecca: Thanks.

John: Thank you.

4: Algorithmic questions in Blackboard Learn

Casey Raymond, the 2017 winner of the President’s Award for Teaching Excellence at the State University of New York at Oswego, joins us again in this episode to discuss how he uses Excel spreadsheets to generate algorithmic questions in Blackboard Learn.

Show Notes

John: There is an overwhelming body of evidence that supports the use of retrieval practice to encourage long-term learning. While this has been known for decades, this was once very cumbersome and time-consuming to implement. In this episode, we’ll explore how learning management systems allow students to have unlimited practice.
Thanks for joining us for Tea for Teaching, an informal discussion of innovative practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist

John: and Rebecca Mushtare, aa graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today’s guest is Casey Raymond, the acting director of the Honors Program, an associate professor of chemistry and geochemistry, and a recent recipient of the President’s Award for teaching excellence at the State University of New York at Oswego.

John: Welcome, Casey.

Casey: Thank you John and Rebecca.

John: Our teas today are:

Rebecca: Jasmine green tea

Casey: elderflower Rooibus,

John: and Twinings English afternoon.

Rebecca: So, Casey, we know that you use low-stakes testing quite a bit in your introductory level courses. Can you talk a little bit about why low-stakes testing is so important for these particular courses?

Casey: So, particularly with general chemistry, the only way to get good at it is by practice and just telling the students to do questions at the end of the textbook doesn’t really work because they don’t do it. I could collect them all but I don’t want to grade all of those. I could grade just some, but again it’s still me grading them and so being able to do something electronically that encourages… forces… students to do something… is a really good avenue to get them to practice.

John: I don’t think that’s anything unique to just chemistry. There’s a tremendous amount of research out there involving the importance of retrieval practice in effective learning in pretty much all disciplines. Here at Oswego, we’ve had a series of three reading groups where that was a common theme to each of those. We had a reading group on Michelle Miller’s Minds Online: Teaching Effectively with Technology, and then we had a second reading group on Make it Stick, and then our current one is on Small Teaching: Everyday Lessons from the Science of Learning. Each of those has a significant portion on the learning gains associated with retrieval practice.

Casey: Yeah, to a degree, these are the modern version of flashcards kind of thing. It’s just a way to really get the students doing the work… and I announce in class if you’ve had to do a question 20 times and still don’t get it right or 20 times just to get it right, you probably need to be doing more questions like that and then go to the book, go to other textbooks, go to resources the publisher has available but it’ll help you determine what you don’t need to study or what you do need to study more.

Rebecca: Yeah, I think students often have no idea what\ they know or don’t know.

Casey: Yeah, and when it gets to the test and they realize that, it’s sometimes, it’s too late.

Rebecca: Yeah, that’s not not a good time to figure that out, right?

Casey: No

John: How do you weight these? And how many chances do students have to work on these problems?

Casey: So when I do this for homework, most of the time I give them unlimited attempts but there’s enough randomness in there that it’s unlikely they’ll see the same questions again… and my homework are usually two, three, or maybe four problems is all… but I’ll offer on average probably three homework assignments every two weeks, so I’m doing a few questions very frequently and in total that’s worth fifteen percent of their course grade. I do weekly quizzes. They’re usually over the weekend and again there are two or three questions is all, and that’s worth fifteen percent of their overall grade… and then at, by comparison, their midterm exam, their one in-class midterm is fifteen percent of their grade. So, in reality, these are all weighted equally. The homeworks, they usually can do as much as they want, I’ll take their highest score. The quizzes, I give them two attempts… and in some cases I do their best score. In some cases I take an average so I tell them if you get it right the first time, don’t do it again and so the quizzes are a little more information in terms of do they really know it because they only have the two attempts… and if they’re doing well on those they should do well in the midterm….
But, there’s always the students that have really high online scores because they’re using every possible resource to get the right answer… and then they get to the midterm and they don’t have all those resources and they score low because everything there is equally weighted. They still have plenty of time to improve before they get to the final exam.

John: …and one nice advantage of allowing unlimited attempts and keeping the highest grade is there’s always an incentive for them to do it again… so if they want to go back and space their practice and review (which is also important in terms of learning. If they want to do that they can go back and redo things from earlier in the course without worrying about the chance of blowing their grade.

Casey: Right. Yep, and what I typically do, though, as well as kind of impose some due dates is I just make a copy of the homework assignment and label it as practice…. and move all of the point values to zero…. and so once the homework two closes, I open homework two practice for zero credit… and it’s categorized differently in the gradebook in Blackboard.

John: OK.

Casey: So it has no bearing whatsoever on how blackboard calculates their grade and it gives them infinite practice after that. I also allow them, on the practice version, to see all of the answers, whereas when the homework is in play, it just lets them know which question they got right and which question they got wrong. They can’t actually see the answer…. how close they were…. and part of that is, in some cases, there’s limited randomness built into the problem. So, you know, I want them to really have to be working through it, but after the fact, when they’re just practicing…. getting ready for the midterm… getting ready for the final…. they should be able to see the answers.

Rebecca: Can you talk a little bit about how you implement the kind of low-stakes testing in our learning management system?

Casey: So, I really got started doing this when we were under ANGEL as our learning management system. I used other learning management systems before coming here, but when we, as a campus, settled on using ANGEL, I realized it was viable again to do some low-stakes testing and worked with Jeff Snyder to develop ways to write algorithmic mathematically based problems where there was really infinite variability in the question…. and we actually got pretty creative and wrote some pretty complicated questions within ANGEL…. and after Blackboard purchased ANGEL and we knew that it was likely our campus would shift to Blackboard, I had a sandbox given to me to start testing and we immediately recognized that most of what we did in ANGEL was going to fail horrendously in Blackboard…. and so it was, what do we do now? And part of my time during sabbatical was working out ways to do some of the things we did in ANGEL within Blackboard…. and I got a SUNY Innovative Instructional technology Grant to put that all together as tutorials… which most of the hard copy is complete now. It’s a matter of going through and implementing the figures and probably some short video tutorials on how I do that… whether it’s multiple choice questions or numeric questions…. but ways to get around the limitations that Blackboard presents us…. which I’ve been able to do in all but one situation.

John: How have students responded?

Casey: Overall, I think they’re okay with having the homework online. There’s certainly a learning curve for the students understanding that it’s a computer doing the grading and that there are limits to how much the computer can think. So if they don’t give the response the same way Blackboard is expecting it, it’s marked wrong and so part of it is educating the students in that context, and part of it is writing the questions such that there’s less ambiguity in entering the answers. I think initially the students don’t recognize the value of the low-stakes testing model, but once they, I think, start getting towards the midterm and see that…hey, those questions I had in homework and quizzes are just like questions on the midterm, I think they start to realize the value of it.

John: I’ve had a similar issue when dealing with algorithmic type questions where I have a question pool with a mix of variants and one of the problems, as you said, is if you have free response questions, students will come up with creative ways of giving you answers that you may not have anticipated. I’ll often have a list of ten or fifteen different possible correct answers and students will find new ways of writing it. Then I have to go back and vary that a little bit or search within a string. Have you been updating the questions or just giving them more guidance in the question itself.

Casey: For the most part I’ve been giving them more guidance in the question and especially the very first few homework assignments tend to be simpler… tend to be more straightforward… and to introduce that “how do you enter your answer?”… and in the sciences it’s really important to us that students include units in numeric responses and we really want to see that on their written work in their answers on it exam, but units are a disaster for online systems because there are so many different ways that you could write the units that I can’t even begin to fathom how many variations or iterations I would have to try and come up with to avoid problems …and so we’ve taken the stance of just not asking for units on the questions we asked the students to give your answer in nanometers or give your answer in meters or kilograms. We tell them what units we want the answer in, but we don’t ask them to put the units label on it.

Rebecca: Are there other workarounds you’ve had to also kind of adapt for this system? Some tips that you might have for others?

Casey: So to deal with the mathematical problems, the limitation that we encountered with Blackboard is that it only allows one dependent variable. I mean there were some limits with the way ANGEL l worked that we dealt with, but the real hurdle is you can only have the one dependent variable when you…

John: this is in terms of algorithmic…

Casey: in terms of algorithmic or even numeric questions… any kind of mathematical problem — and so if you wanted a multiple choice question that had a number as an answer and you wanted the numbers to be different for each student, Blackboard can’t do it or as an example if you had a string of three percentages blackboard could randomly generate two of the percentages but it can’t figure out what the third percentage is if you need to calculate something else in the question because that would be two dependent variables so what I’ve determined and figured out and really have mapped is to use spreadsheets whether it’s Excel, or Numbers, or Google sheets… it really doesn’t matter, but you can code your questions in a spreadsheet and then export that as a text file that can be uploaded as a pool of questions into Blackboard.
Now the nice thing about it is once you’ve done the question once and it’s all on one row of your spreadsheet all you have to do is copy and paste it to as many versions as you want. I’ve done as many as a few hundred versions of the same question this way and so there’s all different sets of numbers that the students could get and then I upload that set of let’s say 250 versions of the question as one pool of questions in Blackboard…. and so when it’s time to write a test, an assignment, a quiz – (Blackboard calls them all tests) — I can go to that pool and say pick one of any of these 250 it’s a good.

Rebecca: So, the time for you mostly is up front getting those questions set up?

Casey: And so most of the time is up front writing them in Excel and if you can write it out on paper first, once you learn a few key functions within Excel and how to handle a few things in Excel, the time really is just copy and paste and getting the text file created and uploaded. One of the other things that it allows you to do with the Excel is use HTML coding within your text so that, if you want a table of values presented to the students, you can do that, and the students will see a table of values within their Blackboard problem. Or you want to use any other kind of formatting or symbols, you just code it all with the HTML and it will upload with the text and Blackboard understands it.

John: When you have distractors on your multiple-choice do you generate them using common misunderstandings or mistakes that students make?

Casey: …and so I rarely use a multiple choice question, but I did figure out how to do it because I know many people will want to know how to do it, and so, yes when you create a mathematical multiple choice question you can have it calculate the distractors as typical mistakes that you know students are likely going to make: a negative sign, a flipped division, an improper conversion between units, all of those can be incorporated and there are ways within the Excel to basically set it up so it does a random representation of the correct and incorrect assignments. So it’s not always version B is the correct answer.

Rebecca: There’s a lot of nuance to kind of the workarounds. Do you have these in a in a format that you can share out?

Casey: Yeah, that’s the documents that I’m kind of creating now, and the steps and I’ve decided and made the choice in doing these to kind of present it at different levels. Kind of… first level is just a set of text of this — is what I’m doing — just kind of prose. If somebody really understands Excel or understands the concepts, they can read the prose and go apply it to a few more key steps that are required for any of this that you know walks through how to do it… and then my goal is with the videos is to actually go through step by step and show people this is how you do one. For those that might not be so comfortable with spreadsheets or using HTML and then the very last part of the document is going to be some of the really common code that you might want to use like how to make a table in HTML or what are some of these special characters and how do you code those, especially ones that you might want to frequently use, and so the goal would be people can just copy and paste that into whatever text they want to write and not have to go find it or really learn how to code HTML.

John: In the show notes we’ll include links to presentations that Casey has done on this as well as a link to these documents once they are available. So we’ll be updating those as things as things are posted.

Casey: Yep, and then you know I’m happy to share the kind of the initial ones that I have so far as a draft.

Rebecca: Great! So our last question always is: what are you gonna do next?

Casey: I don’t know that there’s going to be a next right now with Blackboard. You know I want to get these documents and tutorials and such in place and I don’t foresee what the next is and this one… let me put the bed huh.

John: Very good. Well, thank you, Casey!

Casey: You’re welcome.

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts, and other materials on teaforteaching.com. Music by Michael Gary Brewer.

3: Student success specialists

In this episode, we examine the role that student success specialists may play in helping students develop more effective learning habits. Our guests are Allison Peer and Alicia King, who are both Student Success Specialists at the State University of New York at Oswego.

Show Notes


John:Today, our guests are Allison Peer and Alicia King, student academic success specialists at the State University of New York at Oswego.

Rebecca: Welcome, Allison and Alicia!

Alicia: Thank you for having us.

Allison: Thank you.

Rebecca: Today our teas are:

Alicia: I have Harney and Sons chocolate mint tea. It’s delicious.

Allison: I have Mandarin black pu’erh tea.

John: I have Twinings black currant black tea.

Rebecca: And I have Harney and Son’s Paris tea.

John: Many colleges have begun introducing student academic success specials. It’s a fairly new role. So could you tell us a little bit about what the role of academic success specialist is?

Allison: We see ourselves primarily as academic success coaches. There are a lot of differences between high school and college and we’re here to help students navigate that transition in the areas of time management, for example. So when students are in high school, they often have a lot of adults helping them manage their time for them- like teachers, parents, coaches. When they get here they have a lot more independence and free time and they have to make those decisions on their own, and sometimes they need support in doing that. Another thing we find is even though students may have been successful in high school, the strategies they relied on for success in high school may not be working for them as well in college. So we sometimes need to coach them on specific strategies for effective learning, and then we also have several students who report that they did not have to study in high school. So they, they get here and they first try to rely on, you know, the same methods that they used before, which might have included some cramming but otherwise they might not have had to study very much. And so then they find that they’re not being as successful and so we have to coach them on specific study and learn learning strategies as well.

Rebecca: How does your role complement or supplement the role of faculty members on campus?

Alicia: We have a few different things we use here. We offer to present at first choice courses and we’ll do that for any professor who’s, who would like to and-

Rebecca: What’s a first choice course?

Alicia: So it’s of course for first-year students. It’s usually a subject area of a requirement that they need for their degree program, but it’s also a class where we will teach them study strategies, teach them about the campus resources, they need to know about to be successful during their first year. So only first year students can get into those courses.

Allison: It’s actually SUNY Oswego’s version of a first-year seminar and there’s currently a group of people looking at modifying this a little bit. John you’re on that team, would you like to say a little bit about that?

John: So our Provost has talked a little bit about that. At his prior institution they had introduced a first-year program that were designed to improve student engagement and interest in the coursework. He referred to them as passion courses, where the instructor would find some topic that they were passionate about, the argument is it could help provide students with a much more engaging experience that they’d have a better tie to the community.

Allison: And we in the first choice courses, we also try to help build that tie to the college. One of the options we give to instructors that we can present for the first-choice class is “nailing your first semester,” and it focuses on campus resources to help support student success. There’s a couple other options of courses- or options of first-choice presentations that we offer, one is how to study in college, what to do and what not to do, and also balancing choices priorities and deadlines focuses more on the priority and time management aspect.

Rebecca: Are there other things that you do to support faculty other than just in the first year?

Allison: Yes, so for example we certainly encourage students to visit their faculty’s office hours. One of my students was reporting back to me on a conversation that she had with her professor in office hours he had suggested to her that, in addition to the traditional studying that she was doing for example, reviewing her notes from class he also suggested doing some additional practice problems and then if she experienced any difficulty with those problems or the ones that she struggled with she could bring those back to his office and discuss the solutions with him in office hours. And I could tell that the student really didn’t like the professor’s suggestion from my conversations with her, I know that she believes she needs to learn in certain ways and when someone was suggesting something different, she didn’t think it was going to work for her. So I took that opportunity to talk to her about why he was suggesting it. And I I showed her the book Make it Stick: The Science of Successful Learning, and explained to her that he was suggesting that she use one of the research-based strategies from that book. It’s a strategy that we know works. It’s been proven by research, and so after trying to you know convince her why she should give the the strategy a try we looked at her schedule to see you know where could she fit in some opportunities for this type of practice. So I’m looking forward to hearing back from her, to see how it went when she did some practice problems but she hadn’t gone back to the professor’s office hours yet to discuss those problems. So I’m looking forward to hearing back from her about how that part went.

Rebecca: So it sounds like the opportunities for the one-on-one interactions is really helpful. So that students can maybe start seeing the why. Certainly faculty members, I’m sure often try to indicate why they do things in classes, but maybe they’re not always good at communicating that, so having support outside can be really helpful. Because there’s often reasons why faculty might have certain requirement but you know they might not be so clear to students.

John: And students come in with some serious metacognitive issues in terms of what they find most effective in studying. As she said students generally believe that the most effective studying techniques are repeated rereading and cramming before an exam. And while that works really well and remembering things for a few minutes or a few hours, it really doesn’t do much in terms of long term learning and it’s one of the things but that you know both Make it Stick and the “Small teaching” reading group we’re doing this semester in which you’re both participating that point, out but it’s hard to convince students how successful are you, in general at convincing students a one-on-one approach I think would be much more effective than trying to have a faculty member, well in my case for example, try to convince three hundred and sixty students that they should try these techniques.

Allison: Yes, we definitely enjoy the one-on-one opportunities with students. In some ways students are already primed for new suggestions when we start working with them. We teach an academic success course in addition to seeing students on academic probation in one-on-one conversations. Students who take our academic success course, most of them are choosing to take it, because they know that the strategies that they have been relying on are not working for them. So they are ready to try something new. So we do have some buy-in already. In the past, the people in this position previously experimented with requiring students to take the course and it didn’t work as well, when the students felt like they were being made to take it, as I understand.


John:So extrinsic versus intrinsic motivation.

Allison: So another thing that we, do that kind of helps convince the students to try it, and this is suggested also in Make it Stick, take the time to explain to students how learning works. It’s about them, it’s about how their mind works and usually people are interested in things that relate to them and also that are going to help them improve and something that that they want to accomplish. So we have found a lot of engaging videos that you know help students see how learning works and then also we have videos that help introduce some strategies for effective learning such as spaced practice, retrieval practice, interleaved practice. One of the the video series that we really like is by “The College Info Geek,” Thomas Frank, I don’t know if you guys have seen him on YouTube one of our colleagues in the School of Education, I consider him to be an expert on like brain based learning strategies, he actually uses Thomas Frank with his students, so if he uses it then I feel comfortable relying on Thomas Frank as a resource also. But the videos that Thomas Frank does they are geared towards helping students improve and in all areas academically and otherwise like, for example, he has a video on how to get good sleep and how to take care of your health, so that you can then you know, in turn be more productive and successful with your academics. So, instead of students just having to listen to us the whole time we pull in some videos like that. And after we introduced the strategies to students we also ask them to share out examples of how they may already be using it, so that you know they might hear an idea from a classmate that they might think would work well for them and so then they’re willing to try it out, because they heard it from a peer who they can relate to.

Alicia: And Thomas Frank is young as well, so I think that helped quite a bit that- Allison is giggling because she knows how much I like Thomas Frank.

John: Actually a podcast we recorded just a few days ago and will be coming out probably a week or two before this we talked about a metacognitive cafe online discussion forum, where students look at similar things, but also share the thoughts with each other. So that not only are they thinking about how they’ve applied it- but applying it having hearing from peers about how it can be effective can help. And hearing from someone who’s young on a video, may also work better than hearing from an old professor, and so forth. One of the things that I’ve been really impressed with is all the materials you develop in the handouts that you’ve given out to students – we will include links to some of these in the show notes – could you tell us a little bit about the materials you’ve created and share with students?

Alicia: Sure, so using retrieval practice we understand that textbook reading isn’t always the students favored style of studying, even though to prepare for a test, you’re correct, John they they often will reread the text over and over and over again, so yes reading the text is important but also making connections to the class lectures are also important. So, we created these bookmarks using the retrieval practice strategy that helps them quiz themselves as they’re reading so that they’re pulling more information out of the text, their eyes just aren’t skimming, they’re not flipping pages and not really, they’re not really digesting the information that they’re trying to learn when they do that instead, they’re treating it kind of like a scavenger hunt. They have questions that they need to look for that they’re supposed to get out of that chapter, put it right on the back of the bookmark so they can quiz themselves with questions like what did I just learn or read? What is this mostly about so they can summarize it. Put it into their own words, which is another useful tool questions like how does this relate to what I already know which also helps them build connections so it’s just something handy that they can keep in their textbook to help them.

Allison: And we do have to coach them in how to use the bookmark, so the retrieval practice is only retrieval practice when you are forcing yourself to stop and and answer the question some students need a reminder that you know, you can’t just say “oh yeah I think I know that that’s that makes sense to me that’s easy I’m just going to move on to the next section,” they actually have to to stop and make themselves answer a question about it, put in that effort to do that.

Rebecca: I just had a student this morning that I had that same conversation with, this idea of fluency illusion where she was talking about yeah when you go over in class or we go over in class it makes perfect sense, when I go to try to do it myself like it makes- like I can’t figure it out at all and that’s you know that’s the idea of needing to practice and we were talking I was talking to her about the issue that she had been missing a lot of the review questions that we have at the beginning of class which is built-in practice, so she was missing that opportunity to kind of retrieve on her own she was only hearing like the solutions at the end, so we discussed that but it’s the same idea it’s like yeah sure does make sense to me when someone’s explaining it to me and like holding my hand through it.

Allison: Yeah, when I’m showing them the bookmark in my office, I actually find myself like turning in my chair turning away from the book and looking out of the book and you know like answering the question out of my head and then turning back to the book so they can see- yeah, I actually do need to to look out of the book and rely on my own thoughts to answer the question…

John: your own mental models

Allison: instead of you know, letting myself look back in the book to answer it. And we do tell them after you’ve answered the question for yourself, yes then you can check back in the book and see if there’s anything that you missed or it you know make sure you explain it accurately but do take the time to answer it on your own first. It’s definitely a habit that takes discipline to build.

Alicia: I like telling the students to use Cornell notes while they do it so they actually like in the left-hand column of the Cornell notes they can write a question like the ones on the back of the bookmark and then explain it and then use those notes as part with their lecture notes to study from so they get the most they can from the textbook and the lecture.

Allison: of course in our academic success course we make sure they know when you’re using those questions on the left hand side of the Cornell notes cover up the details on the right and there’s the questions to quiz yourself to do the retrieval practice.

John: Yeah, one of the things I’ve been using in my class is McGraw Hill’s Smart Book, but students have the option of either using a traditional eBook version of it which is just linear, or the smart book option which does exactly this- there’s a section they read and then they’re quizzed on it and then if they do well they can move on, if not it will tell them to go back and re-examine the material and then try it again with some different questions. What I’m finding is that the the stronger students of students who are doing better do that, but the students who are struggling find that very frustrating and they give up and it’s hard to develop a mindset to convince them that practice is really useful instead of focusing on the things that’s easiest, perhaps they need to focus more. What tactics could you use learning is hard work we have to convince some of that because when people come in with a fixed mindset, it’s hard for them to deal with from struggle.

Allison: So, you asked what tactics do we use to get them to understand that they have to practice the hard stuff, instead of going through the stuff that comes easier to them. I’m still working on figuring out the best way to do that with students but one thing that I think is helpful is a lot of students might have had the experience of playing a musical instrument or you know perhaps they took dance lessons and participated in dance recitals or sports yes, so and so if they had a good coach or teacher along the way, hopefully that person was explaining to them look when you’re playing this piece of music we don’t just start at the beginning and run through to the end every time we work on sections at a time and it’s not the section that we play really well that we need to stop and work on it’s the part that we’re struggling on. And so sometimes when we’re practicing it to try and get it better we don’t try to go home full-speed, we slow it down break it up into steps and then gradually build up the speed and work up to our performance level. And if we can relate it back to something like that that they’ve experienced, it may help them you know apply it to a new type of learning. We’re still looking for additional strategies.

Rebecca: After being sort of coached into some of these processes that we know science tells us work, you know how are students responding once they actually try? Are they seeing the effects and then although it’s a struggle they’re they’re doing it or what is your experience been?

Allison: We’ve gotten some good feedback actually we mentioned before that a lot of the students that we’re working with are primed for some change they’re ready to try some new strategies because the strategies they’ve been using have not led to success here in the college setting. So one thing that we do in our academic success course in addition to introducing the strategies with videos we give the students some opportunities to try those strategies out in the classroom on the day that we introduce it- but we also make it homework for a whole week, they have to choose a strategy they want to try out so we give them a handout actually, on the handout we list some strategies from Make it Stick: The Science of Successful Learning, we have self testing or retrieval practice, spaced practice, interleaved practice, we also list Cornell notes on there as a method of using retrieval practice so we have them choose a strategy to try out for the week, they have to write down specifically what they did to make that strategy a reality, write down the result of their actions, and then they also provide a little more detail on what their next steps are going to be. So perhaps they liked that strategy but they want to tweak it a little bit to make it even more effective for them, or perhaps they liked that strategy, found it effective for one course so they’re going to start using it in another course. So we find that when we give the students the time to do it by making an assignment, and we don’t have a lot of assignments in our class we try to make everything that we do beneficial for the students to promote their success, so we don’t feel like we’re over burdening them with work, but we do give them adequate time to try this out and as they’re doing the work for our class they’re also accomplishing work they have to do for another class anyway, so when they take the time to try it out while not being under stress about it we’ve gotten some good feedback, I have some some quotes for some students we have students write about this in their final reflection that they do for the course, they write about you know what worked well for them and kind of what their next steps are going to be as they’re working towards academic success. So for example, when students were trying out interleaving we had a couple students mention that when they were using interleaf practice or varied practice, they didn’t get tired out as fast during their study session because they weren’t studying one thing for a long stretch at a time they weren’t getting bored when they would switch between topics, students also mentioned that it helped them make connections between the things they were learning for example, I actually have an upper-division biology student who came to us because she’s considering medical school in the future and she’s doing well already but she knows medical school is going to be a challenge so she wants to start improving her learning strategies now. So she is taking several upper division biology classes and using interleaving helped her see the different connections between her different biology classes that she was taking. She cited higher grades on her midterm exam, but more importantly to her she said she didn’t feel like she had to rush to feel prepared for her exams. You know in the past she had relied on cramming and she was initially apprehensive about trying something new because she was used to studying in a certain way and she had been successful up to this point but when I shared with her that medical students use these strategies to be successful and even showed her some videos on YouTube that were geared towards medical students using the strategies I think that helped her give strategies a chance, and when she did she saw increased grades because of it, but she also saw a reduced stress because of it. Which is going to be important when she goes to medical school.

John: One thing I’ve been really impressed by talking to you since we started with the reading groups is the focus you placed on evidence-based practices giving students tools that allow them to be successful in any course rather than just focusing on specific short-term problems, and have students generally been buying in in general?

Alicia: We make sure that the students we invite into our class are students who are close to being on academic probation, or like see that they need the help so that the dynamic of the classroom is approachable for everyone.

John: So they’re more receptive.

Alicia: Yeah, and when I have a student add a class late I’m very careful to remind them this is a class full of students that are on academic probation, are close to it, and are very interested and being academically successful, and I think that helps a lot. Starting the class off with that mindset helps quite a bit, makes it a more open atmosphere. We also do some like ice breaking activities to help warm everyone up to let them know that it’s a safe zone for this kind of talk and that helps quite a bit. That seems to be my best strategy as a start.

Rebecca: It seems like you’re meeting them where they’re at, and that’s the whole structure of the course is like the assumption of being in this class is that you’re at a place where things aren’t working, we want to work better and that seems like it’s the key setup to being successful because, you know like that’s where you’re starting, you’re not starting with like well there’s some people were being really successful and some people who aren’t and you don’t have that wide range like there’s a little more focus and sometimes having that more focused group of students can make for maybe a better a better cell because, in John’s example earlier, the really large class he might talk about them maybe about the strategies but, perhaps it’s like group of students who were maybe already going to do fairly well already had that kind of growth mindset might adopt it because they’re willing to try something new, and that group of students who maybe weren’t willing who might not be successful might end up in your group of students but it’s nice that there’s kind of places that the information is getting to people in different places.

Alicia: I think you said it perfectly, we really do need to meet them where they’re at. We admitted them, they’re in the class they’re in there standing for a reason and that’s why Allison and I have jobs is to make sure we come to them.

John: It’s certainly more efficient than just discarding 20 to 40 percent of our students over first couple of years as was a common practice in colleges and universities for decades. You know people are spending a lot to be here and there’s a lot of investment in getting people here and encouraging them to be successful is a good thing.

Allison: Yeah we do find that a lot of students may not have had the opportunity to navigate through academic failure before, and certainly everyone has had struggles but some of the students that we’re seeing now may not have been having that many academic struggles while they were in high school, and you know some of the students we see they aren’t sure how to navigate a failure. So that’s another role that we and the faculty and staff as a team have to help students see those struggles and failures as a learning and growth opportunity rather than an experience that defines them negatively.

John: Going back to the mindset comment, that students with fixed mindsets are not always going to be students who are lower quality students- one of the problems is that when students have been successful but they have this fixed mindset they believe that it’s because they’re talented and the first time they experience failure can often be very disruptive and can lead them to giving up, so that growth mindset is important for all students even those who have been successful as well as those who’ve struggled. Those who’ve struggled regularly often have been forced to adopt a growth mindset because they can see it work but those students who are able to breeze through middle and high school without doing much more when suddenly they’re faced with a challenge often have those sort of troubles.

Allison: Yeah I remember having a conversation with a student like that last semester he had declared a certain major because he had been interested in it and based on his experience with the topic in high school he thought he was good at that subject that was his phrase you know “I thought I was good at it,” and then he got here and took an intro course in the same subject and he didn’t do as well as he had thought, and his comment to me was well I thought I was good at it but I guess I’m just not and you know the student definitely needs some help cultivating a growth mindset. I think it’s a long process for some students because you know their mindsets have been developing since they were young enough to start to understand language based on the types of praise they were hearing from their well-meaning parents-

John: and from some of their teachers along the way because they’ve been praised for being good at this or talented, rather than for their effort.

Allison: Right, so I think it’s something that we all need to be conscious of- the messages we’re sending to students, the way we’re praising them, what we’re praising them for, and also you know some of the feedback that we give that may not be praise but might be taken by the students as as more negative. Of course we have to give constructive criticism but some students-

John: some students just seem to have some problems in recognizing that, we’ve all seen people who say they’re just not good at math or they just can’t write or they just can give public speaking or they can’t draw, and the reason is mostly because they haven’t really tried to do those things and they haven’t put in the effort, and reminding students that they can get better at things by doing it there’s something I think we all need to work on.

Rebecca: I think it’s interesting that sometimes the the lack of a growth mindset is actually pretty prevalent and students who may be traditionally earned A’s and B’s and then those are the students who don’t want to take any risks, right? Which is really what college often is about is taking some risks, and like you know coming up with a hypothesis and finding out if you’re correct or not, you know there’s there’s a chance that you’re not, and so I certainly see this a lot in my classes too, that students you know it might be a student who maybe is a traditionally a C students and they really want to try something new you know and and try to get the most out of this situation so you know there’s there’s a benefit to maybe sharing some of the mindset strategies that maybe some of the students we might label as poor students like actually have that could that everyone else in the room could benefit from, and taking the time and energy to raise awareness about like “oh look at that risk,” you know that was a really interesting choice to make I’d like to see other people make take risks like that it could really be beneficial in helping to switch that mindset I think.

Allison: I think that’s a great suggestion Rebecca, we can’t just stop at telling students to put the time in and try and you’ll see yourself improve. I think we actually need to show them explicit ways that they can do that. So in your graphic design course you’re gonna have specific things that they can do to improve their work whereas in a writing course there’s going to be specific writing strategies students can use to improve their writing, so not only can the professor take the time to show the strategies that students can use, but also give some time to the other students in the class to share the strategies they’ve used I really do think that helps students buy in when the strategies are coming from their peers.

Rebecca: I did want to make sure that we took a little time while we were chatting today to find out like why you got into the roles that you’re in in the first place, because it’s an you know somewhat of an unusual position and I’m just like-

John: well, a new position

Rebecca: yeah, so I’m curious like how did you how did you end up where you’re at?

Alicia: So I went to school and studied finance actually, close to John over here and I started working for the school that I was going to and it helped pay tuition so that was a fantastic benefit and eventually I started academic advising, so I’ve always been an academic advisor of some sort and so advising was really important, to me not so much financial advising I found that that was a little more risky depending on the market and just tended to gravitate towards advising students so I really enjoy that aspect. My previous job here at SUNY Oswego was a Transfer Success Adviser where I helped students transfer from community college to a 4-year institution and there definitely is a gap there that’s very similar to what we experienced with our first year students, so I see that too and this just was like a very gradual happy next step for me.

Allison: I’ve had an interesting pathway to get to this role how much detail would you like?

Alicia: The whole story.

Allison: Okay well I have always been fascinated by how the brain learns and remembers, and when I was much younger I thought I wanted to be a brain surgeon, and everyone that I went to school with thought I was going to become a brain surgeon, and my parents thought I was going to become a brain surgeon, but through some different experiences that I had some before college and and some during college, I learned that physical surgery was not going to be the best way for me to help people personally, but I was still very interested in how the brain works. So I began to think about other ways that I could still learn all I could about how the brain functions and learns and remembers and help people improve their capacity for learning. So as an undergraduate, I took a lot of neuroscience courses in addition to my psychology major, got my Master’s in Education here at Oswego and taught for several years, but I was really missing the one-on-one interaction with students I found that I wasn’t really getting the time to have individual coaching sessions with students on how to improve, and how they could learn better. I had chunks of it here and there but I wanted to be able to focus more time on that, so when this opportunity became available it just seemed like a really good fit for me. So I’m glad I made the change.

Rebecca: So in your roles that you’re currently in, you’ve done so much already but what are you going to do next?

Allison: Well as we alluded to previously, we’re still you know looking for effective strategies to help students develop more of a growth mindset versus a fixed mindset and we know that the research is still taking shape in that area and I know there are some things out there that are already being discussed you know different conferences on student success also listservs it that we subscribe to first-year and transitional type listservs. Unfortunately, we don’t always have the time to digest all of that as its as it’s coming in because we do have so many things going on, but that’s one of our priorities you know I’m one of our slow times to investigate some more strategies to help develop that growth mindset with students.

Rebecca: Sounds like a fantastic plan. I think it you know taking the time to do that it’s gonna be really helpful I think we all probably wish we had a little more time to dig into the research on that topic well we really appreciate both of you joining us today and taking the time to chat with us and sharing what you’ve been up to.

Allison: Thank you.

Alicia: Thank you for having us.

John: Well thank you for being here.

2: The Metacognitive Cafe Online Discussion Forum

In this episode we discuss the metacognitive cafe online discussion forums developed by Judith Littlejohn, an instructional designer and historian from Genesee Community College in Batavia, New York. These discussion forums are designed to help students improve their metacognition and learning skills while also fostering an increased sense of community in the course.

Judith is a 2014 recipient of the State University of New York Chancellor’s Award for Excellence in Professional Service and the 2016 recipient of the State University of New York’s FACT2 Award for Excellence in Instruction.


Rebecca: Our guest today is Judith Littlejohn, an instructional designer and historian from Genesee Community College in Batavia, New York. She is the 2014 recipient of the State University of New York Chancellor’s Award for Excellence in Professional Service and the 2015 recipient of the State University of New York’s FACT2 Award for Excellence in Innovative Instruction. Welcome, Judie.

Judie: Thank you.

John: Today’s teas are:

Judie: Oh, mine is a Twining’s Forest Blend that I got in Epcot.

Rebecca: Mine is English afternoon.

John: …and mine is a Tea Forte Black Currant tea.

John: Today we’d like to talk to you about the metacognitive cafe online discussion forum that you developed. What prompted you to develop this?

Judie: At Genesee Community College, we have a Provost who really promotes critical thinking and as part of her initiatives, in January 2014, we had a critical thinking workshop with Rush Cosgrove from the Foundation for Critical Thinking. He talked a lot about the “elements of thought” and different ways to critically analyze whatever we’re reading. So I took his elements of thought and incorporated them into my online discussions then and tried to focus the students on becoming aware of bias… and you know the different…. ‘cause I’m blanking out on what all the elements of thought are….Then shortly after that, in 2016, we had Dr. John Draeger come over and spend the day with us. He’s the director of the Scholarship of Teaching and Learning at Buffalo State and he has a blog called “Improve with Metacognition” and I found his ideas really interesting. He based a lot of what he does off of an article by Kimberly Tanner. She’s a biologist; she’s at San Francisco State University and she wrote an article about helping students in biology with their metacognition, and she had a table of discussion prompts that she uses. And so John Draeger shared Kimberly Tanner’s table of prompts, and it was all about what they were getting out of the reading, and being aware of how they learn, and what’s successful for them in their struggles to try to learn all this biology content. And so Draeger works with grad students and a lot of what he did was really interesting but, where we are in the Community College, we have the fundamental problem where it’s a struggle to get the students to read. So I thought it would be great to implement questions that help students reflect on how they’re learning what they’re reading, but initially we have to get him to read. So I took the table of prompts that Draeger had and kind of broke them down a little bit in a way that would force the students to have to read at least parts of the chapter in order to answer the questions, and so that led to a series of low-stakes discussion questions separate from the content-focused, you know, the higher-stakes discussions that they do throughout the term, and so the students have to engage with specific parts of the chapter of the book and discuss those. And then from that I kind of branched out into other ways that they think about how they’re learning, think about how they’re reading, and how they can transfer their knowledge, and things like that, and just become aware of the process of learning, thinking about how they learn and thinking about thinking, which, of course, is what metacognition is.

John: How do students respond to just the term “the Metacognitive Cafe?” Many of them probably haven’t heard that term before.

Judie: Well I do put the definition at the beginning of every discussion prompt. I just copy and paste it into each discussion.

John: So to remind them.

Judie: It reinforces what you’re doing for consistency. So they they see the definition all the time. Whether or not they read it, I don’t know, but I noticed that over the course of the semester – at first they’ll call their discussion post… you know… like “meta,” “meta one,” “meta two,” and now we’re into week… we just started week 11, I think… and so they’re starting to write out the words “metacognitive cafe” and so, to me, that indicates that they’re focused a little bit more on what they’re doing. They’re paying attention a little more, I think… I survey them a lot to try to get feedback on what they think of the discussions and it’s been overwhelmingly positive. I think one of the byproducts is it builds a lot of community in the class. The students are sharing what they’re struggling with content-wise and how they’re approaching content that’s unfamiliar to them… how they reread things… or if they take notes, and things like that, and they’re really giving each other advice and strategies.

Rebecca: This is taking place in an online environment, right?

Judie: Right. It’s completely online. I have two courses right now: History 101 (which is Ancient World) and History 104 (which is Early Western Tradition)… and so currently those two courses are engaging in these discussions.

John: Earlier, when I saw you present on this I was really impressed by it, so I’ve tried it myself and it’s worked really well. One of the things, as you mentioned, is that students start talking about their struggles in the class and they start to get to know each other a lot more, at least in my experience, and you had relayed the same sort of thing when you talked about it earlier. Could you talk a little bit about that aspect of it?

Judie: Yes, I find that. Maybe, I’m not sure if it’s because it’s low stakes and the pressure is off, but the students share a lot more in these discussions. They tell surprisingly personal stories of what their home life is like. I ask them to talk about their study space, or their work space. Do you read and write in the same location? Do you try to read in a quiet place? or whatever… I encourage them to share pictures of where they do their assignments and it’s surprising how much they share. I feel like you can kind of,,, you know… I think over the course of an online term the students kind of group together… by the early posters and the late posters. But it really is solidifying this community, and they’re supportive of each other and they’re talking about what they have in common: if they have the same major… if they have the same struggles… I think a lot of it is finding that other students are struggling with the same content… that they are is really affirming for them. They feel, you know, like: “I’m not the stupid one, I can learn this. If they figured it out, maybe I can too.” And they they really are supportive. So it’s been pretty interesting to watch.

John: …and they also share study strategies. When you talked about learning spaces, they probably… at least in my class I’ve used some of the same prompts…. they’ll often talk about how important it is to have a place where they can have focused concentration without interruptions… and for some other students that’s a bit of a surprise so…

Judie: It is. I have students who work two jobs… they go to school…. they have kids… they’re primary caregivers for their aging parents… you know… all kinds of things are going on… and they’ll write about how: “Well, I’ll read for fifteen minutes in my car before I go into my second job” or whatever, and a lot of them are studying in a very haphazard way like that… and I think they watch the posts of the students who say: “oh, I plan this into my day. I schedule my time…” and things like that. There’s a lot of discussion of time management, and it really wakes some of the students up to say “Okay, if my education is a priority, my studying has to be a priority too…” and then they think… they try to reorganize their time a little bit better.

Rebecca: What role do you play in the conversations because it seems like that’s moving away from the content area stuff to more about like… how you… how you are a learner… how you exist as a learner. One of the things that we talked a little bit about in a reading group that we had on our campus recently was about the faculty member becoming a little vulnerable in certain circumstances, so that students can relate to them a little bit more… and they’re not just like some sort of authority figure who has no emotions.

Judie: Mm-hmm. So with these discussions… I try to… I don’t insert myself a whole lot, but I try to get in there a little bit now and then. I use… we use… Blackboard as our learning management system, so I do turn on the post ratings. That means you can give the students stars on their discussion: 0 through 5 stars. So, whenever somebody posts what appears to be a genuine response in their initial post, I give them the five-star rating… so they know… and they can tell that the instructor gave them the rating…. so they know that I’m right there, that I read what they what they wrote. Even if I don’t comment, they see the stars… and then where I feel like it’s appropriate, then I will comment on somebody else’s posts… and I do so. In the one where we talk about our workspace, I do share a picture of my office set up at home… I teach as an adjunct, so most of this is going on when I’m at home after my workday… and so I show them that… and there’s a few different things that they come up with. I’m trying to…. just off the top of my head… at one student. So, I have one pretty basic question, which is “why is it important to read the chapter before you start to take the quizzes?” and one of the students made a great analogy… and she said “I read the chapter before I do the quiz or before I start the writing assignment because if I was trying to change a tire on my car and I had no idea how to – where to find the jack, how to how to take the tire off, where any of the tools were, and how to go about the process, I would not be able to change the tire in my car – and it was pretty good… and so I jumped in on that one and I said well I really liked your analogy… and then that kind of led to more discussion of analogies and things like that… which I like. Because I think sometimes, when the instructor posts, it kind of ends the conversation, and sometimes people won’t post after the instructor. So I like that one but, so yeah, I’m just kind of on the sidelines I’d say… and try to jump in now and then where I feel like it’s not going to hinder the conversation and….

John: but support it….

Judie: Right, yes…support it.

John: Going back to a point you made earlier about things bleeding through and them getting to know each other, the first time I use this was last spring in a labor economics class where I, in some of the discussions I’ll ask them to talk how the material relates to their future career or how it relates to other classes they’ll be taking… and they started opening up quite a bit in my classes too: about their work environment… about their family environment… about having to work a couple of jobs in some cases… or raising small children… or having relatives who had health issues that they had to assist with… and what was really interesting is that bled over into the content discussion. So when they were talking about various labor market topics, they make references to other people’s living conditions, and how they might be able to relate to this concept, or how the things they had said in the other discussion were relevant here… and it was nice to see that bleed through… because they they were making posts that were much more meaningful in both discussions than I’d normally noticed in the past.

Judie: I noticed too that they they post more, meaning: more sentences… more…. they seem to become a little bit more articulate in their posts, instead of just barely hitting the key words and and calling it good. They really elaborate on their ideas more… and they respond to each other a lot more so instead of – I mean there’s always a handful of students who respond to another student and say “I agree” but I find much more content like “I agree, especially when you said whatever… or when you said X it reminded me of Y” and so I find that the depth of their discussions is a lot better.

Rebecca: Have students articulated that the metacognitive cafe is something that they’re finding really beneficial? Other than your observations, have they actually articulated that?

Judie: Yes, they also… I survey them in addition to the institutional end-of-course survey. I do give them informal surveys in the class through Blackboard and I typically do a half-time survey and then an end-of-course survey, and I give them five extra credit points so that does encourage them to answer – I can see who responded, but not what they said, so it’s still anonymous in that respect – and we just finished up the half-time survey and I wrote down a quote that one of the students wrote. In the surveys that I write, I can ask very pointed questions about what we’re experimenting with and, as an instructional designer, I try all different things in my classes and I want to get feedback while it’s live and while we still have the other half of the semester to make changes. But one of them in History 104, the Western Tradition, said “the metacognitive cafe is a great idea… it seems like a small break from ordinary coursework where you can actually talk about how you do the coursework, which is interesting. It seems to me that most people enjoy the metacognitive cafe.” So I like that… and a quote I had from the past, I think from last semester, was: “the discussions sometimes did not mean much to the coursework, although it did help me learn how to learn the material better.”
I was having a conversation with my mom yesterday about teaching and she was asking me a lot of questions about teaching online and I said you know my whole thing is: of course I want them to grasp the learning outcomes, but I don’t care if they memorize dates and names as long as thematically they understand the major themes throughout history, but if if I can help them learn how to learn, then they’ll be unstoppable. That’s my whole goal… to make sure the students can figure out how to tackle some new ideas… and figure them out…. and look at them with a critical thinking perspective… and make the most of it ….and take it with them.

Rebecca: That’s something that I really value in my classes too. I teach mostly web design classes, so students really have to learn… to learn… because the stuff changes all the time.

Judie: Right.

Rebecca: …and we don’t realize how important that is, and so it seems to the metacognitive cafe is a good opportunity to help students kind of move away from the idea of fluency illusion… or the idea that like… oh this is really familiar…. so therefore I know it – kind of recognizing that there is a way that we retain information and we need to practice it and retrieve it and all of those sorts of things and by having kind of those guided questions they become more aware that’s even a thing right.

Judie: Yes, I agree. I think it’s been good. I think it’s not a whole lot of effort on my part and it’s not… you know, I estimate – every week I give the students a checklist of what they have to accomplish that week and approximately how long each activity will take and I usually put about 20 minutes for this. So I don’t think it it takes a lot of time for them to go through the process of typing up their response. I don’t know how much time they spend thinking about it, but I do think it’s really been helpful, and I think it’s worth carving that time out of the week for the students to do that.

Rebecca: Can you share some examples of some of your questions?

Judie: Let’s see…. I brought an index card with a couple of questions. So the first one that I asked them is… that first week… so say in History 101, the Ancient World, we talk about the Agricultural Revolution and I just fundamentally say “What did you already know about the Agricultural Revolution and what did you learn in this chapter that was new to you?….” and they can’t answer that if they don’t open the book.

John:…and that activates prior knowledge and it helps them to make…

Judie: Oh, absolutely. But remember, my first goal is to get them to read, so at least they have to look at the book. They’ve got to at least read the five or ten pages on the Agricultural Revolution and find some new idea that they can point to.

John: … and that’s a little nudge to get them to practice…

Judie: Exactly. …and I follow it up the next thing they do… well every week, every chapter, they do mastery quizzes, and if they don’t read the book first, the mastery quizzes take close to three hours and I tell them to budget two hours every week. You know, after they spend an hour reading then two hours for the quiz.

John: Are these the InQuizitive quizzes from Norton?

Judie: Yes, I use Norton InQuizitive. InQuizitive, if you’re not familiar with it, it’s sort of a gaming method of quizzing, and so the students have to wager. They wager points according to how confident they are that they know the answer, and they have to earn fifteen hundred points in the quiz to get ten points in the grade book.

John: So, if they’re wrong they lose points. so it forces them to think about how well they know this, which is also another way of developing metacognition.

Judie: Right, how well they know it. It’s set up really well to keep targeting the questions that they’re getting wrong and it tells them where to look in the book. It gives them a lot of helpful feedback and the students overwhelmingly like those quizzes. They dislike them at first because they’re not used to their chapter quiz taking so long, but once they get into it and get used to how the system works, then I think it’s… they really do like it. They’ve got different kinds too. So you can watch a short video clip and then answer a question, or they’ve got sorting and putting things in order… which I love …. I love to do timelines and cause-and-effect type things

John: It helps them make connections across events….

Judie: Right. So, it’s not just multiple choice quizzes, or just true and false. So, I like that a lot. But, yeah… so that was one question [for a metacognitive cafe discussion prompt]… So, yeah, I asked, what interesting fact did you learn? Oh, then another week, I’ll say “Did anything new or interesting in the chapter change the way you think about X, like change the way you think about the Middle Ages?” ….or what information changed any preconceived notions that people have about certain things, like: “What did you read that….and I think students have trouble… I think at this level the students have trouble understanding the difference between what they learned in high school and what history is in college because they don’t understand that high schools teach them citizenship and not history….not major global historical themes. They think they know global history, but they really don’t… and so a lot of what they learn in college is pretty eye-opening, if they will admit that their eyes need to be opened.

John: Because your students come in with pre-existing models of how the world works…

Judie: Exactly.

John: …and much of it is wrong…. and

Judie: yes

John: …we need to tear that down but they have to recognize it… and forcing them to confront it and think about that and reflect on that…

Judie: mm-hmm.

John: …could be really helpful.

Rebecca: Especially because students… they’re trying to just get the grade sometimes… especially if they have a lot of other things that they’re balancing in their lives and they don’t take the time to reflect… so by building it into your course, right… like when I finally get students to kind of make those connections….but we have to… it becomes sort of the responsibility of a faculty member to help students develop that practice, because it’s not intuitive and it’s not something that, you know, it seems like an extra, right?

Judie: Right.

Rebecca: After a little thing but… I don’t think they always see the value in it until they’ve done it.

Judie: Right. Yeah, and it’s tough…. I think it’s really tough with a gen ed that, you know. If I have a class of 32 students, typically there’s one who will say they want to be a historian; all the rest of them are only there because they need a world civilization requirement and they don’t want to take the course. They’ll say it right out: “I’m not interested in history,” “history is boring,” and so, you’ve got that uphill struggle right there… and then for them to realize they need to take serious time on this course that is not in their program that, as they see it it, can be tough. I have a couple students right now who just don’t seem to understand why you would have to take history, say, if you’re going to be an artist. Which, [laughing] how could you not want to take history as an artist…

Rebecca: …as an artist… what the heck?

Judie: It’s like influencing artistic movements like history and art, I think reflect each other and why would you only want half the story? But so, that’s tough.

Rebecca: I’m so glad that you brought up the stuff about general education because I think a lot of faculty struggle with some of the same issues, right, like “why am I here? and then trying to get those students engaged is such an upward battle or at least it feels like it is. Are you finding that the metacognitive cafe is starting to convert some of those folks?

Judie: A little bit… a little bit… that… and I also work really hard in all my classes to help the students find something in history that they can kind of latch on to thematically… So early on it within the first couple weeks of the class they need to find a theme to track so if you if you are an artist, you could choose art, or if you’re a sports person you could choose a sport history, and kind of track how what changes and you know what new trends and things started throughout whatever era we’re studying… and I think that helps a lot too… because…

John: …it gives them a more personal connection.

Judie. Exactly.

John: It’s also activating their prior knowledge and information.

Judie: Right…and so that’s helpful and do have them kind of share that too…. you know, what topics they’re researching and why… and then, and that’s fun. Those discussions are fun because when students are working on research papers and they spend all this time looking for sources and writing their outlines and all that and finally writing the paper, to finally be able to share what they’ve been researching I think is really good. It’s easy to do that in the classroom, but online it’s a little tougher, so I try to make sure they have a forum where they can point out what the most interesting things they discovered during their research process… where they’re finding sources… and the different facts that they will discover and I think it’s helpful.

John: One of the things that I’ve asked in my classes since I’ve been doing this (at the end of the term in the last metacognitive cafe discussion) is to reflect back on the class… and I asked them also if there’s anything… based on your question, one part of it is if what they’ve learned has caused them to alter their view of the world, or what was the most surprising thing they learned in the course…. and what’s been really interesting is the most common response is that what they enjoyed the most were the metacognitive cafes. One of the things I think we both do to some extent is we try to nudge them towards better learning strategies and as part of that, I have some on spaced practice, interleaved practice, and so forth, and also having them break that myth of learning styles that they’ve come in with… and what they’re always saying is that they amazed that so much of what they had learned and what they had been told in their earlier classes about rereading and highlighting and focusing on their learning styles is not based on evidence and many of them are saying they wish they had learned much of this stuff back in elementary school and it would have made their educational career much more productive.

Rebecca: Sounds like you know in in both of your cases an interesting way of you know if it’s a general education class that maybe students aren’t buying into the subject matter, at least they can buy into the idea of like learning how to learn and finding strategies and things that could apply elsewhere through the lens for a particular discipline which you know, if anything, at least that’s something that they can like connect to if they’re finding it hard to connect to the content.

Judie: Yeah that’s that’s a good insight.

John: Now one thing we we usually ask is: “what lessons have you learned while doing this? How has your practice changed over time?

Judie: Hmmmm

John: It sounds like it’s worked pretty well from the beginning.

Judie: I think a few of the questions could be a little bit more thoughtful. I don’t know. I asked some things about you know like motivation and how will they transfer the skills they’re learning in this class to their other courses but sometimes I try to mix up, so like throughout we have a 16 week semester so the first week we don’t do one of these. So we’ve got 15 of them and I try to kind of alternate between going back into the specific chapter, so they have to look at it and then, or else, just their own personal way of learning…. and try to line things up so that, to a certain extent, it goes along with what else they’re doing in the course… if it’s research or timelines or things like that. So every now and then I just had to look back and make sure that as I change the course activities that the metacognitive cafe discussions are still aligned correctly. And I think I like, John, some of what you said about having them read about the myth of learning styles and what not. Some of that material I send out in announcements throughout the week and just give them different resources that they can look through but maybe incorporating some of that more, because some of that is hard to dispel.

John: … and some of them resist it. When I’ve done that you, some of them said they just don’t believe it. This semester, I had an interesting experience where I posted a short video as a prompt on one of them from the Learning Scientists (they have some nice videos there on how we learn) and a few students were saying “Well, they only talked about a couple of studies and I’m not convinced, I’d like to see more evidence.” So I posted a list of five or six papers and they were really grateful, and they were discussing those, and that kind of surprised me because usually in an informal discussion like that it tends not to get quite as technical and it’s been interesting. So from then on I’ve been adding some research papers on at least the things that are related to learning science and so forth, and just to provide more support.

Judie: That’s a good idea, to add that upfront in the question because sometimes I’ll follow up on something that they’re discussing and if the student isn’t… if they’re not posting till the end of the week and the late posters, sometimes the students won’t go back and read it so I may have… I may post some information that would reinforce or sort of re-explain what it is they are trying to talk about, but they’ve already moved on and they’re not going to go back and look again. So that’s one thing. That’s a tough thing in Blackboard… if there were a way that you were alerted that somebody posted, you know, responded to your post or things like that like… one of the failings of Blackboard. I think maybe some of these questions could have supporting material sort of embedded within and they could choose to engage with that or not. But that might be a good idea to try in the future.

Rebecca: I like the idea of sort of having the opportunity to engage with those extra materials but not a requirement to do so. So that might help to engage like the wide range of students that we have, some who might want to take the the deep dive, and others who maybe don’t want to, but at least you can hit them somewhere.

Judie: Right, yeah.

Rebecca: I know I allow them to engage with the subject matter. Yeah they might watch a short video, but they don’t want to read the paper.

John: And the way I’ve been phrasing is since then is I have the short video, they’re usually five to ten minutes, and then I’ll put “(For those who would like to see more evidence, here are some resources.)”

Judie: That’s a great idea.

John: So, one of the things we normally ask people is what would you like to try next? It doesn’t have to be related to this.

Judie: Actually based on what we’ve just discussed I think the next thing I would want to do is in embed some of the articles and videos that you refer to and that type of thing and I think that would be good just you know kind of update the questions… update what they’re working on and just help the students learn how to learn.

Rebecca: Great.

John: Great. it’s working really well. I was so impressed when I saw the results and you shared with us. Some of the comments from students that I had to adopt it right away, and I’m gonna keep doing it as long as it keeps working.
Rebecca: Yeah, I adopted a couple of the questions. I didn’t do the full thing, but I did use some of the questions. Some of the questions, especially the workspace one, for some of my students and it was really interesting so I’d like to figure out how, how I can do it more – yeah I want in.

Judie: I love encouraging the students to share something like their workspace. I love when they post a picture. You could see it even shows a certain amount of trust, you know, that they’re gonna show a picture of their room. One I remember really well, she studied in her bedroom. She added like a little desk next to her bed and she had a sloped ceiling and a calendar… like she used her ceiling as a calendar and wrote all her due dates on the ceiling over her bed. It was crazy but I thought… wow…. like you’re just putting it all out there when you’re sharing things like that and it’s…. it’s really really good to see that sense of community. So I just think… I think these little discussions that started out just trying to get them to read sort of compounded into all these other benefits that I didn’t anticipate at first, but it’s really great to see. So I’m glad you guys are interested in it too.

John: I think it’s come up probably at least a dozen times in workshops just a semester. I’ve suggested to many of our faculty, particularly in online classes, because it really does help build a sense of community that I’ve never seen in my online classes before.

Judie: Good. One… oh I know… okay I know I knew I had one thing…. I want to do and we, John and I, were talking about this the other day when we were arranging all of this is… I think it would be really interesting if we could go back over… this is my fourth semester of this now, so if we could kind of go back and see how the students responded to these discussions and especially the ones who said: “well I already knew this” or you know the ones who are in denial and then see how their final grades were and… kind of chart that over time. Maybe see how they do in their program and, as opposed to the students who are, who you can see by the responses, are more open and saying “oh I’ll try this” and “I’ll try that” and see how their final grades were and if there’s any patterns over time. I think that would be really interesting.

Rebecca: It would be really interesting if other people are adopting stuff too to actually have it in some different disciplines and…

Judie: Yeah, that would be great.

Rebecca: …to see how that how that might turn out.

Judie: … we can all do that.

Rebecca: ….sounds like scholarship of learning article coming your way.

John: My informal observation is that the students who tend to be the last ones to post are the ones who say “I just don’t buy this” and they generally haven’t put a lot of thought into it… and they also generally are not the strongest students overall.

Judie: Yeah.

John: …and it’s sometimes difficult to get through to them. It’s perhaps an example of that old Dunning-Kruger effect, which is that the students with the worst metacognition tend to be those who have the the highest impression of their abilities and so forth. Not always, but it’s a pattern that I think if we did do some sort of analysis of that I wouldn’t be surprised.

Rebecca: Well sounds like we have a hypothesis, perhaps amended. [laughter]

Judie: I think I would like to follow up on that I think it would be interesting. So let’s get in touch with our institutional research folks and see what we can come up with.

Rebecca: Sounds like a plan.

John: Okay, and we will post show notes that will include links which will include the resources that Judie mentioned, will include the questions that we’ve included in the notes as well as any other materials that we find interesting.

Rebecca: Thanks for joining us today.

John: Thank you for joining us again, Judie. You’ve given a lot of workshops here and we really appreciate it. You do some really good things.
Judie: Well, thank you. It’s been fun