110. Fostering a Growth Mindset

Some students with fixed mindsets enter our classes expecting to be unsuccessful while others believe that they have a natural talent in the discipline. In either case, these students often get discouraged when they experience challenging tasks. In this episode, Sarah Hanusch and John Myers join us to discuss how they have revised their classes and used metacognitive exercises to help students develop a growth mindset and to recognize the benefit of learning from mistakes. Sarah and John are both Assistant Professors in the Department of Mathematics at SUNY Oswego.

Show Notes

Transcript

John K.: Some students with fixed mindsets enter our classes expecting to be unsuccessful while others believe that they have a natural talent in the discipline. In either case, these students often get discouraged when they experience challenging tasks. In this episode, we examine how two faculty members have revised their classes and used metacognitive exercises to help students develop a growth mindset and to recognize the benefit of learning from mistakes.

[MUSIC]

John K.: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John K.: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guests today are Sarah Hanusch and John Myers. Sarah and John are both Assistant Professors in the Department of Mathematics at SUNY-Oswego. Welcome, John and welcome back, Sarah.

Sarah: Thank you.

John M.: Thank you.

John K.: Our teas today are?

Sarah: None today

John M.: Yeah, imaginary tea. No tea for me.

Rebecca: The imaginary tea…that’s what my daughter likes to drink. That kind.

John M.: Yeah, I’m in good company there&hellp;

Rebecca: I have English afternoon.

John K.: And I have a ginger tea.

Rebecca: We invited you here today to talk a little bit about how you’ve introduced a project on metacognition in some of your mathematics courses. Can you tell us a little bit about the project?

John M.: Sure, this began, I believe, in the spring of 2018 in a Calculus I course. And the idea was that, Calculus I is known across, basically the entire country…every school in the country…as being a very difficult course. So, you have a lot of students who are coming in, especially in the spring semester, who had bad experiences with calculus in the past. And in particular, I’ve been told by some colleagues that there’s going to be some students in there that more support than I suppose you would imagine. The situation was that on the very first day of class, I had students coming in who have had bad experiences with it in the past. And then at the same time, I have the students that are typically high performing. And they have difficult times also with perfection, you know, being obsessed with 4.0s and grades and that type of stuff. So the idea was that I wanted to simultaneously address failure with the students and perfection at the same time. And I was sort of led to think about this metacognition project, actually, funnily enough, on a flight back from San Diego. I was at what are called the joint meetings for mathematicians, and a lot of progressive newer teaching techniques are talked about at this conference. And I’m flying back from the conference on the airplane and I’m getting really introspective and I’m thinking like, I really need to do something to talk to my kids about failure and perfection. And then it occurred to me that there was this blog post that I had just read a couple weeks before by a mathematician by the name of Matt Boelkins at Grand Valley State University. And he had this idea for a metacognitive project that addressed all sorts of things like growth mindset, fixed mindset, productive failure, and all these different things. And I decided about a week before classes started that this is what I was going to do.

Rebecca: That’s when all the best ideas happen.

John M.: I know…right before class and on an airplane. I get really introspective when I’m on airplanes and staring out the window and thinking of all the big things in life and stuff.

Sarah: And essentially, John came to me and said, “I’m thinking about doing this project.” And I said “Well, that sounds cool. And let’s see if we can measure if it has any positive effect or not.” So, I sort of came in on the research side of it…of “let’s see if this is effective for changing attitudes towards mathematics.” And since then, I’ve stolen the project to use in my own classes. But, it really started as I came in sort of more on the research side of things

John M.: I think stolen might have been a strong word, but…

Sarah: I didn’t ask…I just took it. [LAUGHTER]

John K.: For the research project did you do pre- and post-tests on attitudes?

Sarah: We did a pre- and post-test, we use an assessment called MAPS which is the Mathematics Attitudes and Perceptions Survey. It’s a 31-item survey. It assesses, I think, it’s seven different dimensions. Some of them are growth mindset. Do they view mathematics as being answer focused or process focused? The categories were growth mindset, the applicability of mathematics to the real world, their confidence in mathematics, their interest in mathematics, their persistence in mathematics, their ability to make sense of mathematics, and do they view mathematics as being answer focused or process focused?

John K.: Sounds like a good instrument. Before we talk about the results, let’s talk a little bit more about how you implemented it. How was the project structured in terms of what activities did the students do during the class?

John M.: So the idea was that over the entire semester, they would have a selection of articles online to read, they would have a selection of YouTube videos to watch and it was essentially experts that are addressing these various topics. So, like for example, there is a clip by Carol Dweck, one of the originators of the theory of growth and fixed mindsets, and they were to watch these clips and read these articles across the semester. And then I think it was probably with two weeks or three weeks left in the semester, they’d have to write a reflective essay. It was an attempt to sort of shift the culture in the classroom towards viewing mistakes and failure as productive and as opportunities for learning. Because I think in wider culture, everybody believes that math is just about the right answer. And that if you can’t get the right answer, then there’s no worth in whatever effort it was that you put in to get to that point. And I wanted to provide sort of a counterpoint to that, so a counter narrative. Being honest about how many times per day mathematicians actually do fail, you know, that type of thing. So yeah, the main component was this essay that was reflecting on the stuff that they read and watched over the semester, and then there was sort of like daily conversations.

John K.: Were the conversations online or were they in class conversations?

John M.: In class…in office hours, just kind of whenever they popped up. I remember a couple conversations that happened after I gave back exams, for example, or rather right before I gave back exams. So for example, I would say, you know, I’m about to hand back exams. And I want you when you see the score, when you put the paper over and see your score, I want you to immediately think how are you going to frame this result in your mind. Are you going to look at that score and be happy with it and chalk it up to just your natural talents? Or are you going to say, “Oh, this is a result of hard work?” And then if you’re not happy with your score, are you going to put it away and never look at again, or are you going to engage with your mistakes and make them productive mistakes? It was sort of intervention through conversation that happened on an almost daily basis.

Rebecca: Did you notice a difference in the kinds of conversations you were having in class because they were doing these readings and watching these videos, maybe conversations you hadn’t experienced before in the classroom?

John M.: Yes. In particular, I had students come into office hours and they were relentless with trying to understand the material because they knew that they were going to have another shot to get it right. And I had never experienced that before. In fact, in one of my student’s essays, I had a student tell me that when she’s not done well on exams in the past, she would just take the exam and stuff it into her book bag and never look at it again. And she told me that just because of because of how I was structuring the course that she doesn’t do that anymore. She actually pulls it out and engages with the mistakes and the comments that I put on the exam and comes and talks to me about the exam and everything. So I did see a change in the students.

John K.: Was some of it based on the reflections or was it also partly based on a restructuring of a course to give students more opportunities to redo things or to try things again?

John M.: I believe the latter had something to do with it. Because the idea was that I could say these things out loud to them. But I wanted to actually build components into the course in addition to the essay that sort of reflect the themes that I’m trying to communicate to them.

John K.: Telling them that they can learn from mistakes, if you don’t give them the opportunity…

John M.: Right.

John K.: …to learn from mistakes might not be as productive. I think both components are really valuable. I just want to make sure we were clear on that, too.

John M.: I think that you risk sounding like a cliche motivational poster, if you don’t actually put some meat on the bones with it.

Rebecca: Can you talk about some ways that you actually built that into the course?

John M.: I did test corrections. I don’t remember exactly, I think it was get back half the credit they missed or something like that. So, the idea was that they had to engage with the mistakes on their exams and correct them. And it had to be perfect. So they had a week to turn in their test corrections, and then I would re-grade them. This was very time consuming, as you might imagine, but the students I believe, really responded to it. It really sort of hooked in with the theme that I was trying to send.

Sarah: And since then, we’ve both moved to more mastery based grading. John before I did, but a system where students keep trying things until they get it right. And that really helps sort of drive that “learn from your mistakes” message home.

John K.: Are you able to do some of that in an automated way? Or is this all involving more grading on your part?

Sarah: The way I’m doing it, unfortunately, it’s more grading on my part. Although I will say this semester I’m doing these mastery based quizzes, but I’m not collecting homework. So, it’s kind of a toss up in terms of how much…it isn’t really extra grading. I’m just grading more things in another category.

John M.: Right, I would not do test corrections again. Not only was it a lot of time to grade, but then I had issues with academic honesty. The mastery based thing I have found is, I believe, much more effective.

John K.: Another thing you may want to consider that we’ve talked about in a couple of past podcasts is having a two-stage exam, where in the first stage, they do it themselves. And then you have them break up into groups and do either all the questions or a subset of those as a group. So, you’ve got some peer instruction going on as well…and that way it’s done right in class and it can be done, if the exam is short enough or the class period is long enough you can do both of it. A common practice is to do two-thirds say individual and then one-third for the group activity, which has many of the same things. They don’t know what they’ve gotten wrong, but when they’re sharing with their peers, they’re talking it over and it means you only have to grade the group exams on the second stage, which makes it a whole lot easier than individual ones.

John M.: Right. Yeah, I have a friend I believe he has done that stuff like that. So yeah,

John K.: The Carl Wieman Science Education Institute, I believe, has a lot of information on that. I’ve been doing it the last couple of years, and it’s been working really well. Doug Mckee was a guest on an earlier podcast, we talked about that as well. Are there other things we want to talk about in terms of what you’ve done in the courses?

Sarah: One thing that we’ve both done since this initial project is we’ve taken some of the ideas of this project, but interspersed it more throughout the course. One thing I know at the time that John observed was that he felt like a lot of the students started the projects in the last week, right? And so what I’ve done instead of doing a big project of these topics is I’ve taken these articles and done the second week of class, you have to read one of them and respond on it. And then the fourth week, you have to do another one, and so on. So it’s a little bit of it throughout the whole course instead of all loaded at the end. I think it helps having some of those conversations with the students as well because they’re not just seeing the ideas in the conversations. They’re not just seeing the ideas in the paper. They’re kind of seeing both and it just helps intersperse it a little bit throughout the semester. I know I’ve done that a couple times now. I think you’ve done that since as well.

John M.: I did a pre-semester sort of essay and then I did a post-semester essay. But it was in response to the first time we did that, which is referred into the paper, and one of my students actually told me in their essay, he was like, ‘Hey, I wish I had this at the beginning of the semester.” So yeah, it’s definitely like a “duh” moment. Like, I probably should have done something earlier in the semester, instead of waiting all until the end. But, you learn as you do these things, so. But the essays that the students wrote… I provided them with prompts just to alleviate any sort of writer’s block that they may have. But, the students who basically ignored my prompts and told me their personal stories were the essays essentially that I still remember. I had students that were straight A students that were telling me exactly what I thought was going to happen: that they’ve been the smart person their entire life, and they kind of feel trapped by being a smart person. They don’t want to take any risks because if they risk something and fail, then that’s their identity as a smart person, right? They’re not smart anymore. I’ve had students from the other end of the grading spectrum who basically told me that the first day they walked into the class before I even said anything, they were already convinced that they were going to fail the class. I had students tell me about mental health problems. I had adult learners talking about balancing life and school issues. I mean, it’s just absolutely amazing what they told me, they opened up basically. That made a big impression on me.

John K.: Tying into an earlier podcast, Judie Littlejohn and I had introduced something really similar where we have weekly discussion forums. And I also noticed the same sort of thing, that I got to know the students much better because when they were talking about some of the barriers or the issues they face, they were sharing a lot of details about their life. And you get to know them better and they also seem to form a little bit more of a tighter classroom community because they also got to know each other a little bit more.

Rebecca: It is kind of interesting how when students are talking about their process or who they are as learners, is very different than talking about the subject matter. And it does get them to open up and may be engaged with faculty in a way that they wouldn’t otherwise.

John M.: And I have found being honest about my own failures in the past has been a catalyst for conversation, right? Because they view us as professors, they view us as the authority figures, the experts in that we never fail. And basically telling them how many times I fail on a daily basis in my own mathematical research. It goes a long way, I think… finding common ground with them. And acknowledging how difficult the subject material is. I mean, there’s a reason that calculus has a high failure rate because it’s a hard course, among other reasons. Yeah, just having the humility with the students and kind of stepping down off of the pedestal in front of them, I think that it helps.

Rebecca: So do you want to share some of the results that you got from your study?

Sarah: We saw some very significant quantitative results. I mentioned the MAPS instrument is what we use. It’s a 31-point scale. Its reliability and validity has been established pretty well, especially in calculus classes. One of the things that they did was they looked to see if the items were consistent with expert consensus…. So, with how mathematicians view it and all of the items were valid with the attitudes of mathematicians except some of the growth mindset scales. Research says that that’s an important scale as well. And on this 31-point scale, we saw an almost 4-point improvement from pre-test to post-test…of the students becoming more aligned with the expert opinions, which is a really significant amount…I mean, almost 10% improvement, which is even more remarkable, because when this assessment was first validated, they found that there was usually a negative result from taking a Calculus I class. So, the attitudes get worse pre-post in a calculus class and ours had statistically significant improvement. In addition, we saw statistically significant improvement among all of the sub scales. Now some of them were better than others. Some were just barely below .05 in terms of significance and others were much more significant. I mean, we really saw that over the course of this semester, they really did change their attitudes. We also had some evidence, as John’s already talked about, from their essays…where they said how they started to view mistakes as productive, and they started to feel like there was value in making mistakes and learning from them.

John K.: You mentioned alignment with an expert scale, can you explain that for our listeners?

Sarah: Essentially, what the original authors and it was Code et. al. that did this paper and develop this instrument. They gave this survey to students and they gave it to mathematicians and looked for alignment. Particularly they were looking for whether or not the mathematicians agreed on the items. And the idea was our goal is to get math students to have attitudes more like mathematicians, because that’s our goal, right? …is to develop future mathematicians. And so we would like those attitudes to get closer to how mathematicians view mathematics. They had high agreement among the mathematicians on every item, like I said, except one or two of the growth mindset questions. So, in other words, this survey reflects how mathematicians view mathematics. And that was how they determined the right answers on the survey, whether a particular item is something you should agree with or something you should disagree with. They went with the expert consensus.

John K.: So now, I may be misconstruing this, but are you suggesting that perhaps a lot of mathematicians had adopted a fixed mindset? So, there was a bit more variance there on that?

Sarah: I will say that was what the results of their validation showed.

John K.: Okay.

Sarah: And leave it at that. [LAUGHTER]

John K.: It does remind me of that study a few months ago, that found that when instructors had a growth mindset, the achievement gap narrowed and the drop-fail-withdrawal rate was much lower in courses, then for those instructors who had a fixed mindset. I think that maybe even more of an issue in the STEM fields than it is in humanities and social sciences, but I think it’s not uncommon everywhere.

Rebecca: I say it’s a common problem everywhere.

John M.: I’ll say it…mathematicians suffer from fixed mindsets. I’ll just say it, right? [LAUGHTER]

John K.: Many academics do.

Sarah: Yeah.

John M.: Yes, of course.

Sarah: I mean, the people who choose to become academics are often the people that were successful in school and they decide to continue with it. I mean, it is less likely that people who felt unsuccessful decide to keep going and to go into academia.

John K.: Selectivity bias there and that reinforces a belief in a fixed mindset, perhaps.

Sarah: Precisely.

Rebecca: What kind of response have you seen from students from…I mean, it sounds to me like this one study lead to good results, and then that changed many classes in that you’ve taught or the way that you’re teaching, how have students responded?

Sarah: Generally positively. I think doing the projects at the end of the semester wasn’t the best idea because they just feel so overwhelmed at the end of the semester with exams and projects and everything coming due. So, I did get some responses of “W hy do I have to do this now.” But generally, I think they appreciated learning about learning.

John M.: I think that given the opportunity to talk about their past experiences, I think they appreciated that. For the most part, I’ll agree with Sarah. I think that the message landed with an awful lot of students like I wanted it to. Some of my favorite essays were students who told me that they thought I was crazy on the first day. I mean, you go into a math class to learn math, you don’t go into a math class to study metacognition, or whatever it may be. I had one student the first time around, who basically told me it was all a load of crap, like why this is not working at all. And I had a student the last time that I did this, she was very skeptical towards the end even. Basically, aliken it to just some cheesy self-help stuff. I think that most students responded positively.

Rebecca: Have you seen the response impact other faculty in your area? For example, if they really liked having those techniques and things introduced in your class, have they asked other math faculty to do that in future classes or are you finding that its not many math students who were actually in that particular class?

Sarah: We haven’t done any tracking, so I don’t know where his students have gone. I mean, I’m sure some of them went on to Calc II…I’m sure some of them did not. Right. I mean, I guess most of them would have had Jess the following semester, right? Did she say anything?

John M.: No, she didn’t say anything. I’m teaching Calc III right now, and I have some of my former calculus students that were in this and they’re doing well.[LAUGHTER] Small sample size, but yeah, they’re doing well.

John K.: That could be an interesting follow up though to see how successful they were in the subsequent classes.

Sarah: Yeah.

Rebecca: Sometimes we’ve heard anecdotes, of departments and things when there’s been change that if students really respond well to whatever the techniques are, that they will demand it of other faculty members, and John’s talked about this before in economics.

John K.: Yeah, when you can show results…

Rebecca: Yeah.

John K.: …that there’s been some gain, and especially if it comes from students at the same time, it often puts pressure on other people in the department because if you’re able to show people that your technique has been successful and students are coming in and saying, “G ee, I wish you would consider doing this. I did this in my intro classes, and it was really helpful.” That sometimes helps make change much easier.

Sarah: Yeah, so one of the things that we did look at was we compared the final exam scores of John’s sections to the other sections of calculus that semester. Now, there was some other issues that clouded that data a little bit. His scores were a little bit lower than the other instructors. But what was really surprising, essentially, if you look at, I don’t remember if it were just the final exams or the semester grades. The DF rates were the same among the sections, but the withdrawal rates were significantly different. And that almost no one withdrew from John’s sections. I think there were two if I remember the data correctly, whereas there was like five or six on average from the other sections. And so the DFW rates were different, but the DF rates weren’t. So I just thought that was an unusual circumstance. So, it seems like the students were sticking with his class… and pushing through.

John K.: And if there is a larger portion of students staying with the class, then perhaps a slightly lower average grade is not necessarily a bad sign…

Sarah: Exactly.

John K.: …because student success is partly measured for persistence to completing the course.

Sarah: Exactly. I think because there were more students who stuck it through to the final exam, then his final exam scores ended up being a little bit lower. But again, if you looked at like overall course grades, they ended up being pretty consistent, other than the W rates. I wanted to make sure that there weren’t significant differences in the rates and I think it was just shy of being statistically significant. Like, if you had one more student that would’ve been significant. But just to make sure that, especially like adding the test corrections in wasn’t substantially making the class too easy, right? Because that’s often a critique that, you know, “Well you make these changes, but is that just making the class too easy and people who aren’t really prepared, are they passing?” And so I just did this analysis of the, like I said, it was really just a t-test analysis, but just to see whether or not it was significantly lower and it wasn’t significant. It was lower, right, just not significantly. And then like I said, I looked at retention rates just more as an explanation for why the average was lower.

John K.: In a lot of studies of interventions, the dependent variable is the drop-fail-withdrawal rates, because that’s a measure of success in completing the course. That by itself could be an interesting focus of a study. I’ve been running this metacognitive cafe in my online classes for a while and I did have a student in the class who wrote a few times about the metacognitive development that was introduced in one of your classes. They didn’t specify who but they said, we’re also doing some work on metacognition in the math class, and they said it was really useful and it was nice to see it in two classes.

Sarah: Yay!!

John M.: Good.

John K.: So there’s at least one positive data point there or one additional data point there. So are you going to continue this in the future? And if so, what might you do differently?

Sarah: Well, I think we’ve mentioned already that we’ve worked on including some of the ideas at the beginning of the semester and throughout the semester, rather than one project at the end. For the reason that it really benefits them most at the beginning of the semester when things are getting started. I think we’ve also both changed different things about our grading systems to incorporate more opportunities for growth.

John M.: The last time I did this, I introduced some articles that were a little bit more rigorous with the data and the science, because I sort of wanted to counter that kind of criticism that all this “Oh this is just a bunch of TED Talks…” that kind of thing. So, I really wanted the students to see some of the science behind it, the science of learning, because I really wanted to send that message that “No, this is not me just standing up here saying, ‘Oh, this is going to help you or anything, right?’ This is actually stuff that researchers have thought about before.”

John K.: I had a very similar response the first time I did this. I had a video I posted which was a TED talk by a cognitive scientist who talked about research that showed that learning styles were a myth. And some students had come to believe in the existence of learning styles because they’ve heard of them and often been tested, multiple times in multiple years, on their learning styles. Sometimes even through college and that’s rather troubling. The students said, “Well, this is just one researcher, I’m sure there’s lots of other studies. I don’t believe it because it’s not consistent with what I’ve always been told or what I’ve heard.” So I decided to modify it then and I added to that discussion, five or six research studies. In case you don’t believe this TED talk by someone who’s done a lot of research on this, here’s a number of studies, including some meta analyses of several hundred studies of this issue, and that has cut much of that discussion. They’re less likely to argue against it when it’s not just a talking head or not just a video when they can actually see a study even if they don’t understand all the aspects of it.

Sarah: Yeah. So I think that’s one thing we’ve tweaked what articles and what videos are we showing. I know the semester I gave my students a article that had just come out this September, that students perceive active learning as being less efficient, even when they’re learning more. In some physics classes at Harvard, they gave two weeks at each thing… two weeks of active and two weeks of lecture, and then they had them switch. And the students learned more with the active learning, but felt they learned less. And my students have been feeling frustrated because they feel like they’re not learning enough and that I’m not telling them what to do.

Rebecca: You’re not “teaching” them.

Sarah: I’m not teaching them. And we spend the class period, letting them vent. So all their feelings were out in the open. But, then I sort of countered with this article saying, “Look, I promise you really are learning things. You just don’t feel like you are. But you really, really are. And you’re actually learning it better than if I were using a different style.” So, that’s one way that we’re tweaking the articles because sometimes the research comes out that’s pertinent.

John K.: We refer to that Harvard study in a few past podcasts. We touched on it in a podcast that will release on October 9th. I haven’t shared it with my class yet, but I’ve been tempted to.

Rebecca: What was the discussion like talking about that particular article? Given that they were frustrated?

Sarah: I mostly was just trying to acknowledge that I understand their frustrations…and that, yes, the way I’m teaching this class can be frustrating. I agree. Sometimes I get frustrated about it. But I know that ultimately, they are learning things and that they are going to be stronger writers and stronger students of mathematics by using this structure. And so I kind of use it as evidence of I’m not changing.

Rebecca: So I hear you…

Sarah: Yeah.

Rebecca: …nut…

Sarah: I hear you, but…

John K.: I had this very conversation with my class today. They’re coming up for an exam very shortly. And I asked them, how did they review before an exam and the most common answer was they like to reread the material over and over again. And I mentioned some of the research on that. And I said, the best way to review is to work on problems with this. And I gave them several ways in which they could do that, that are built into the course structure. And I said, “But that doesn’t feel as effective. Why?” And one of the students said, “Well, I get things wrong.” And I said, “And when would you rather get things wrong, when you’re reviewing for an exam, or when you’re taking exams?” And I think some of them got that message. So I’m hoping we’ll see when they take the test next week.

John M.: Right? It seems like anytime you do anything that’s just not a standard straight lecture, there’s a certain amount of buy in that you need to get from the students. And sometimes that can be very difficult. There’s almost a salesmanship that you have to do throughout the semester to make sure that everybody’s on the same page and to kind of fight those feelings where the students give you a lot of pushback. Yeah, that’s the great fear is that when you innovate or you experiment that’s going to go horribly wrong. And sometimes it does, but, you know, we still keep going.

John K.: Because students are creatures of habit. They’ve learned certain things and they want to keep doing things the same way. And anything new can seem troubling, especially if they’re getting feedback along the way that says they need to work more on things…that’s not as pleasant as rereading things and having everything look familiar.

John M.: Right

Rebecca: Passively sitting in a lecture when things all seem like it makes perfect sense to you, because an expert is describing it who knows what they’re talking about, right? Always feels easier than trying to apply it yourself. And I think that students, even though the lecture might feel better, and learning is hard…over time…at the end, when they’ve seen how much they’ve accomplished, and you do have them reflect…many of them appreciate or come around. Sometimes, it’s not in that same semester, sometimes it’s emails, months or years later.

John K.: Yes.

John M.: Right. Right, right.

Sarah: If only if we could do course evals, you know, a whole year later,

John K.: Or five years later. That may not work too well in my tenure process, though.

Rebecca: We always wrap up asking what’s next?

Sarah: Well, the first thing is we’re hoping our article gets published. It’s been submitted. We’re waiting for reviewers. I’m going on maternity leave next semester…that’s really what’s next.

Rebecca: Sounds like a new adventure.

Sarah: It is a brand new adventure.

John M.: Wow, I don’t think that far ahead, I guess. Yeah, I guess I’m that unoriginal, huh. But, yeah, no I’m just trying to…

Sarah: We’re moving to a new building.

John M.: Yeah, moving to a new building, and getting a new department chair. Yeah, that’s right.

John K.: A new desk to go with the chair?

John M.: No. Ah… Yeah, funny, funny, funny.

Sarah: if only…

Rebecca: Well, thanks so much for joining us, this has been really interesting.

[MUSIC]

John K.: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John K.: Editing assistance provided by Brittany Jones and Kiara Montero.

108. Neuromyths

Faculty design their classes based on their perceptions of how students learn. These perceptions, though, are not always consistent with the science of learning. In this episode, Dr. Kristen Betts and Dr. Michelle Miller join us to discuss the prevalence of neuromyths and awareness of evidence-based practices in higher ed.

Kristen is a clinical professor in the online Ed.D. program in Educational Leadership and Management in the School of Education at Drexel University. Michelle is the Director of the First-Year Learning Initiative, Professor of Psychological Sciences and the President’s Distinguished Teaching Fellow at Northern Arizona University. She’s also the author of Minds Online: Teaching Effectively with Technology and a frequent guest on this podcast.

Show Notes

  • Miller, M. D. (2014). Minds online. Harvard University Press.
  • Online Learning Consortium
  • Betts, K., Miller, M., Tokuhama-Espinosa, T., Shewokis, P., Anderson, A., Borja, C., Galoyan, T., Delaney, B., Eigenauer, J., & Dekker, S. (2019). International report: Neuromyths and evidence-based practices in higher education. Online Learning Consortium: Newburyport, MA.
  • Mariale Hardiman
  • Tracey Noel Tokuhama-Espinosa
  • Dekker, S., Lee, N. C., Howard-Jones, P., & Jolles, J. (2012). Neuromyths in education: Prevalence and predictors of misconceptions among teachers. Frontiers in psychology, 3, 429.
  • Alida Anderson
  • Macdonald, K., Germine, L., Anderson, A., Christodoulou, J., & McGrath, L. M. (2017). Dispelling the myth: Training in education or neuroscience decreases but does not eliminate beliefs in neuromyths. Frontiers in psychology, 8, 1314.
  • Universal Design for Learning,” CAST website
  • Mchelle Miller, “65. Retrieval Practice” – Tea for Teaching podcast, January 23, 2019.
  • Vygotsky, L. (1987). Zone of proximal development. Mind in society: The development of higher psychological processes, 5291, 157.
  • Michelle Miller, “86. Attention Matters” – Tea for Teaching podcast, June 19, 2019.

Transcript

John: Faculty design their classes based on their perceptions of how students learn. These perceptions, though, are not always consistent with the science of learning. In this episode, we examine the prevalence of neuromyths and awareness of evidence-based practices in higher ed.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today are Dr. Kristen Betts and Dr. Michelle Miller. Kristen is a clinical professor in the online EDD program in Ed.D. Educational Leadership and Management in the School of Education at Drexel University. Michelle is the Director of the First-Year Learning Initiative, Professor of Psychological Sciences and the President’s Distinguished Teaching Fellow at Northern Arizona University. She’s also the author of Minds Online: Teaching Effectively with Technology and a frequent guest on this podcast. Welcome, Kristen and welcome back, Michelle.

Kristen: Thank you so much for having us.

Michelle: Hi, it’s great to be here again.

John: Were really pleased to talk to you. Our teas today are…

Kristen: I’m drinking Apricot Oolong, a green Tea. Nice for the afternoon.

Michelle: And, I have a wonderful hibiscus tea.

Rebecca: And, I have… big surprise… English Afternoon tea.

John: And, I have ginger peach black tea.

We invited you here to talk about the study that you both worked on together on neuromyths and evidence-based practices in higher education. Could you tell us what prompted this study?

Kristen: Sure. As a lifelong learner, I decided I would enroll in a wonderful program being offered at Johns Hopkins University several years ago in mind, brain, and teaching led by Dr. Mariale Hardiman. In one of the courses, I read several articles that looked at the high prevalence of neuromyths in K through 12 education. And, one of the things that caught me by surprise was: One, I was a K through 12 teacher early in my career. I was, at the time, a professor in the School of Education, and in looking at some of the neuromyths, they actually looked like things that I had studied as part of professional development. And, I had not assumed they would be neuromyths. And, so it really intrigued me in terms of: Why is there this high prevalence and why are we not more aware of some of the evidence-based practices that are out there? Not just in the United States, but clearly these were studies that were taking place internationally. So, I decided to start looking at this through the lens of higher education, because that’s where I work and it’s my area of expertise, and I reached out to Dr. Michelle Miller. I was at the Online Learning Consortium conference. Her focus is on cognitive psychology. So, I approached her after the session and told her about this interest in looking at neuromyths within the field of education… really, across disciplines, in trying to see was it similar to what the findings were in K through 12 education, and what was really being done to integrate evidence-based practices into pedagogy or even andragogy. So, we decided to connect and start looking at this. I had a wonderful PHD student who I was working with at the time as well, who is from Armenia, very interested in this topic, and we quickly grew our small group to include a total of ten researchers from the total of seven different institutions nationally and internationally across three countries. And, everybody brought different expertise, everyone from two-year colleges, four-year colleges, public, private. And, we also were very fortunate because we were able to find, really some of the seminal researchers in the area of mind-brain education science, such as Tracey Tokuhama-Espinosa. And, we reached out to the researchers who actually conducted the studies looking at neuromyths like Sanne Dekker, and we reached out to a Alida Anderson who worked with McDonald et. al. in their 2017 publication. So, it quickly grew from a point of interest in trying to identify what was happening in higher education, to really a much broader international study.

Michelle: Oh, and just echoing what Kristen has said here, we first did meet through the Online Learning Consortium, first at a conference and then they set up calls where we got to talk to each other and realize that even though we came from somewhat different academic backgrounds and published in some different areas, we really had this common ground of interest in how do we bring more evidence-based teaching to faculty in higher education and really throughout the world. And, to me, as a cognitive psychologist, it’s just an inherently fascinating question of, even though we live in our own minds, why do we not sometimes understand some basic principles of how the mind and how the brain works? So, that’s just an intellectually interesting question to me. But then it does take on this tremendous practical importance when we start to look at teaching practices throughout the world and bringing that really quality evidence-based design of teaching and learning experiences for our students.

Rebecca: Can you talk a little bit about how, once all of these researchers are now together, how did you put the study together and how was it conducted?

Kristen: I have to say it was not easy. Thank goodness, we reached out to some of the original authors. The survey instruments that looked at neuromyths and general knowledge about the brain. And, what was so interesting is almost all the studies were truly K through 12 focus, so the questions were very different. Even looking at lexicon, “girl and boy,” where we would want to look at male/female. So, we had a look at absolutely every question and make sure that we were able to revise that question within the framework for the lens of higher education. So, it was not an easy process, just in terms of time, because we had to go through so many iterations. And, I think that really helps with the integrity of the research. We had two pilot studies, even down to looking at the Likert scales that we used. One of the things that really stood out was the primary study that we looked at, which was a 2012 study by Sanne Dekker and several other researchers. They had a Likert scale that looked at correct, incorrect and I don’t know. There was a study by McDonald and colleagues in 2017 and they changed it to true and false. So, we decided early on, we would go with true and false. And, when we did that pilot, we ended up with half the participants stopping midway and simply putting, “I’m not sure if it’s true or false…” and they just didn’t complete the survey. And, I think, just looking at how we phrase the questions, it really affected the participation of our respondents. So, we went back, we modified some of the questions based on that, and we change the Likert scale. And, I think being able to have the ability to say whether it was correct, incorrect, or you didn’t know took away from saying it was true or false, because you can base it on knowledge or what you perhaps had been exposed to. And, we ended up having a wonderful pilot making some additional changes. And the feedback that we got, even after sending out the survey, we had a flood of emails saying “Can you please send us a copy of the study, we’re really interested?” So, we really looked at everything. And, I would say one thing that stood out most; and again I go back to the time we spent over two years on this study from point of inception to where we actually send out the survey, collected this study and then published it, was when we looked at the neuromyths, what we quickly realized was we needed to examine evidence-based practices as well. And, we looked at all of this from a metacognitive perspective. The prior studies that were done, looked at what they called “endorsing neuromyths,” and we weren’t so much looking at endorsing, we wanted to look at awareness, because all of us were involved in teaching… professional development. And, so it was a matter of trying to identify what the gaps were, what were instructors, instructional designers, and professional development administrators aware of and, if there is that gap, how could we develop a study where people would say “Wow, I also thought that was correct, but it’s incorrect… but, I would love to find out what the response is and how I can change my knowledge or understanding.” And, so we looked at absolutely everything and wanted to create a study that people would pick up and say, “This is where I am now. Gosh, after going through this in reading the report, this is where I am and my circle of knowledge needs to continue to expand, as things continue to expand through mind-brain education science.”

Michelle: As a collaborative effort, I haven’t been involved really in a study of this scale and scope. And, it’s simply the level of collaboration. You just heard about one of the iterations of the survey instrument that we put together and just how that piece of the study came about. But all the way through the analyses, the writing, it was such an opportunity, even apart from what we were able to share with the rest of the world, just from my own niche piece of the study as well. The opportunity, as a cognitive psychologist, to start infusing what I feel is more attention that needs to be paid to cognitive psychology and learning sciences. The opportunity to infuse that into this field in this area of thinking was also really exciting as well.

Kristen: So, in terms of how it was conducted, we sent the survey out for the Online Learning Consortium. When we originally started, we were just going to look at instructors, we were looking at neuromyth prevalence in instructors because all of the other studies that had been done were primarily K through 12 teachers and pre-service teachers. (although the McDonald study looked at a wider range). Once we started to bring together our team, then we started thinking, “Gosh, well, it’s not simply the instructors. It’s going to be the instructional designers, it’ll be anybody conducting some type of professional development as well because no course is truly an island.” There are so many people today involved in course design, course development and so the Online Learning Consortium was such an amazing partner for us and they touch on absolutely every part of that population. So, we reached out to them early on and said “We’d love to collaborate with you. You’ve got an extensive membership and listserv. Would we be able to develop this survey instrument, send it out through your membership, and ask them through snowball sampling to share it with others who may actually be involved in higher education, in one of these roles.” And, they could not have been a better partner. They’re just incredible to work with. So, that’s how it was conducted.

John: And, we were actually part of that snowball. I sent it out to a list of about 1200 faculty, staff, and professional development people on my campus alone. How large was your ultimate sample?

Kristen: We ended up with approximately 1300 respondents. And, then we actually looked at the full study, we ended up with 929, who met the criteria for inclusion. So, one of the things we wanted to make sure when we looked at the criteria for inclusion that they worked in higher education. You’d be surprised. So many people complete surveys, but they don’t necessarily meet the criteria. Even when you explicitly state you have to be within higher education: teaching or one of these areas. So, we had a total of 929 who met the criteria, and of those they also had a complete 95% of the questions for the neuromyths, and also for the evidence-based practices because we didn’t want to have any gaps. I would say it was an incredible response rate, especially for those completing the survey. They filled out I would say the majority of everything within the survey itself. The respondents were just incredible as well, because you talked about the cross section of participants, but we ended up with really an incredible number of instructors and that was broken down into full-time, part-time, instructional designers, the professional development administrators and it allowed us to run a lot of different tests that we’ll talk about when we look at the findings.

Rebecca: I think one of the things that’s really interesting about how you discuss the setup of the study is thinking about how many different individuals play a role in perpetuating myths, or even perpetuating good evidence-based practices too. That administrators is where funding comes from, so you have to have everybody in the institution on board with what you actually want to essentially Institute.

Kristen: Well, what’s interesting, and you bring up such a great point. One of the top neuromyths out there is learning styles. And, so when you’re looking at learning styles, this is something that almost seems to permeate. It doesn’t matter when you started teaching, whether it’s K through 12, or higher education at some point if you’ve been involved in education, you’ve come across learning styles. Now there are learning preferences and there’s lots of wonderful research on that. But this concept of teaching to learning styles, I think, unfortunately… we talk about this in section seven of our report kind of got mixed in with multiple intelligences. And, that is not at all what multiple intelligence was about, but it was almost the timing of it and so, having been a K through 12 teacher, I remember going through a professional development where we learned about learning styles and how it was something to look at in terms of teaching to learning preferences. And, even to this day when I do presentations, and I know Michelle has run into this as well, especially when we co-teach some of the OLC workshops, somebody will inevitably raise their hand or type in the chat area “Are you kidding? Learning styles is a neuromyth? We just had somebody on our campus six months ago, who taught us how to do an assessment to teach to learning styles.” So, it’s still out there, even though there’s so much in the literature saying it’s a neuromyth. It’s still prevalent within education across all areas.

John: So, you mentioned the issue of learning styles. And, that’s something we see a lot on our campus as well. We’ve even had a couple of podcast guests who we edited out there mention of learning styles and then had a chat with them later about it. I won’t mention any names because they had some really good things to say, but it is a really prevalent myth and it’s difficult to deal with. So, you mentioned learning styles. What are the most prevalent myths that you found in terms of neuromyths?

Kristen: When you look at the report, the first part of our survey had 23 statements. We had eight statements that were neuromyths. If you look at the K through 12 studies, they had many more neuromyths, but we had eight. And, I will tell you, the top five neuromyths in higher education, very closely parallel what you find in K through 12. Now our prevalence is not as high, but it still shows that instructors, instructional designers, and administrators are susceptible to them and that goes back to awareness. So, the top one: listening to classical music increases reasoning ability and that’s really that Mozart Effect. Another one: individuals learn better when they receive information in their preferred learning styles. Some of us are left brained and some of us are right brain due to hemispheric dominance and this helps explain differences in how we learn. So, that’s really that concept of “Oh, I’m right brained. I’m left brained.” And, this again, is something that goes across higher ed and K through 12. Two other really big ones: We only use 10% of our brain. And, if you look at section seven of the report, you will find all of the responses, literally evidence-based practices or research-supported responses to make sure that people aren’t simply saying, “Oh, it’s incorrect. Well, we want people to know why it’s incorrect. So, they can reflect on that and change their understanding, really the rationale and the research behind it. And, then lastly, it is best for children to learn their native language before a second language is learned. This, again, is a big neuromyth. And I think one of the things I’m hoping that will come out of this study, because we talked about this really when we go into evidence-based practices, is this concept of neuro-plasticity, the fact that the brain changes every time you learn something new. When you’re engaged in an experience, the brain is changing. And, sometimes the brain is changing at a cellular level before you might even see that change in behavior, and so we’re able to see now through technology through f-MRI through fNIR so much more than we were able to see before. So, really keeping abreast of what’s happening in the research should be informing our practice because we have more information available than ever before. But, somehow we need to get that into our professional development training, seminars, and workshops or into the classes that we’re teaching in our schools of education or into our onboarding. But yeah, these are the top five neuromyths in terms of susceptibility, and they cut across higher ed and K through 12.

John: In your paper, you also provide some crosstabs on the prevalence by the type of role of individuals, whether they’re instructors, instructional designers, or administrators. Could you tell us a bit about how the different groups due in terms of the prevalence of these neuromyths?

Kristen: Well, the one thing I will say is, everybody is susceptible to neuromyths, so it wasn’t as if there was one group, and I know that’s always in the back of someone’s mind, “Gosh, who’s the most susceptible?” Well, we didn’t find any significant differences, and one of the things that we wanted to do as well was to really be break the participants down and look at other factors. So, when we look at full-time versus part-time faculty, is one group more susceptible to neuromyths. And we found no significant difference in terms of gender, in terms of age, in terms of working at a two-year institution, a four-year institution. And I really think that talks to the amazing reality of the opportunity to integrate professional development in looking at the learning sciences and mind-brain education science in the opportunity to decrease that gap. So, it wasn’t one group over another. But it’s everybody who has this opportunity to increase this awareness across all of these areas.

John: Didn’t you also find that some of these myths were less common among instructional designers relative to faculty,

Kristen: We found with evidence-based practices, when we looked at significant difference with evidence-based practices, instructional designers actually had in terms of percent correct, higher awareness of evidence-based practices. It wasn’t a large difference, but there was a significant difference and Michelle can certainly talk to this point as well. But, this is really the importance of having an incredible team when you’re looking at course design, course development, and part of that may have to do with, when you look at instructional design, there is so much new literature and research that’s getting infused in to that area, and so that may have something to do with it. But, I think there’s lots of additional studies that we could do to follow up.

Michelle: Kind of circling back to the point of the design and delivery of instruction in a contemporary university or college is fundamentally more collaborative than it was in prior eras. And, so I think we definitely need to have everybody involved start to really break out of that old school mold of class is identified with the teacher who teaches it and that’s what a course is. And no, courses reflect, today, everything from the philosophy and the support that comes down from the top to the people that the students may never meet, but who put their stamp on instruction such as instructional designers. And, this is something that I get pretty fired up about in my just practical work as a program director and just being involved in these things in the university, that there are still faculty who you say, “Hey, do we have any instructional designers who are working with us on this project to redesign? Is anybody assigned to help us as we develop this new online degree program or something?” …and you sometimes still get blank look.? Or you get “Oh, aren’t those the people who you call when the learning management system breaks down and that’s their specialty?” I mean, this report, I think, just really hammers home that idea that instructional designers are a key part of this collaborative team that goes into really good quality higher education instruction today. And it isn’t just about the technology. I think that they’re getting exposure to and staying abreast of what’s going on in research that relates to teaching and learning. And, what a great opportunity for faculty to not just rely on them for technology, but to learn from them and to learn with them as we build better courses together.

Rebecca: Can you talk a little bit about the awareness that you found in general about evidence-based practices? So, we focused a lot on the neuromyths, but what shook out when you started looking at the evidence-based practices?

Kristen: Well, one thing that stood out was awareness was much higher. And, that’s really exciting. I think that’s a huge testament to the professional development that we are offering. But, there were still gaps in areas where there certainly could be a lot of improvement. So, a couple of examples that I’ll give because we literally spent months looking evidence-based practices, and we wanted to make sure that we could support them. So, for example, when we look at percent correct, where most individuals across all three groups were not as aware, like “differentiated instruction is individualized instruction.” So, we know that this is incorrect. But most of the respondents did not put that that was an incorrect statement. So, they either stated it was correct, or they didn’t know. So, again, this is an area that we certainly want to explore. Because differentiated instruction is something that really, I think, adds to the classroom. And, there are other ones. For example, we’ll look at Universal Design for Learning. So, one of the statements we had in there actually comes directly from the CAST website, and it says “Universal Design for Learning is a framework to improve and optimize teaching and learning for all people based on scientific insights into how humans learn.” Well, the instructional designers, they were the most aware. So, 87% of them got that correct. Of the professional development administrators 74% got that answer correct. For the instructors, 58% got that correct. So, you can see the difference in the responses and when we share this nationally or internationally…. when we talk about the study, you’ll have a lot of individuals who’ll say “No, universal design for learning, that’s about accessibility.” Well, it certainly is about accessibility. But, most importantly, it’s about learning and how humans learn. It is probably the most dynamic and the most powerful aspect that we can add into pedagogy or into andragogy. But just by looking at the data here, it may not be something that everybody’s aware of, and that’s again a great opportunity to integrate that into professional development. So, there are a number of things. I mean, it’s exciting because when you look at it, there are 28 statements. And, as I mentioned, overall, the awareness was much higher across all three groups, compared to neuromyths or general knowledge about the brain.

Michelle: Just to jump in here, again, from my kind of cognitive psychology perspective, those evidence-based practices that we’re talking about also include, specifically, some items that are related to memory, a topic that’s really close to my heart. So, I think those are just fascinating as well. So, for example, we asked a variation on a classic question that many cognitive psychologists have looked at: “whether human memory works a lot like a digital recording device or a video camera.” So, is your memory basically taking in information that’s in front of you? And, here again, we’ve got 69% of our instructors saying, “Oh, yeah, that’s right. That’s how it works.” And, that is not how it works. 79% of our instructional designers identify this as an incorrect statement and 74% of our administrators, and we have a few other related things such as we asked people whether testing detracts from learning. And, as Tea for Teaching listeners know, that goes to retrieval practice. Testing doesn’t detract from learning, testing builds up learning. So, these are some as well that I think it’s very interesting to tap into what people know and really think about while these maybe seem like inside baseball, or very metaphorical or philosophical questions, if I’m an instructor, and I believe these things, that students are basically just running video cameras in their heads… well, that is going to lead to some different practices. I might be very puzzled as to why I got up and gave this lecture and the students eyes were pointed at me and yet it didn’t end up in memory. So, those are some of the items that I was particularly interested to see when we got all the numbers in.

Kristen: You know, I would say one thing: when anybody reads the report, what we want them to do is look at how it’s presented in terms of the tables, because everything is looking at the percent of correct or accurate responses. So, as Michelle said, when we look at “human memory works like a digital recording device,” 69% of the instructors got that correct. 79% of the instructional designers got that correct. And, 74% of the administrators got that correct. So, that means we still have a fairly large percentage, basically 20 to 30% that either got the answer incorrect, or they didn’t know. And, even looking at these responses, do they actually know why they knew it? Or did they guess or did they make that assumption like, “Oh, that’s got to be right.” And so, really, the intentionality of this study was awareness, really bringing out statements from the literature to help anybody who’s involved in teaching, course design, professional development to look at these questions, and really think “Do I know this?” And, “If I know it, how do I know this? Is it based on some type of research or literature? Could I defend that? If I don’t know with certainty, where do I find that answer? And how can I learn that? And, how can I integrate those practices?”

John: On the day when your report came out, we shared that on our campus to everyone on our mailing list. One of the nice things about the report is that it has all the questions and also provides references for the answers explaining why the specific answer is true or false. And, it’s a really great resource and we’ll share a link to that in the show notes. It is long. When I shared it two people sent back email saying “Maybe we should use this as a reading group for next semester.” And it’s not a bad idea, actually. But, much of that is appendices and so forth. And, it’s a really informative document. I believe in your survey, you were asking people about their participation in professional development, and you looked at the relationship between participation in professional development and the prevalence of these myths. Is that correct?

Kristen: We did. So, one of the things that we wanted to look at was trying to find out if educators were involved in professional development, whether it be neuroscience, psychology or in mind-brain education science, did that actually increase their awareness of neuromyths, general information about the brain, and evidence-based practices? And it did. We found that that it was definitely a predictor and it was found to be a significant predictor and so, for us, again, it looked at what a wonderful opportunity to be able to say that training does have a positive impact. And, that was really the crux of the study… and it’s interesting, you talked about the length of this study, because originally we had thought about doing two different or three different studies. So, we do one on neuromyths, one on evidence-based practices, one on professional development. Then when we brought the data in, the question was: “Do we separate them out into three different long articles or three different reports?” And, we collectively, across all disciplines said, “No, we need to bring them together.” Because first and foremost, it’s about awareness. You can’t really talk about evidence-based practices, until you’re aware of what the neuromyths might be. What are some of the fallacies that you might actually believe? What are things about the brain that you may or may not know? And, once you’re there, and you have that understanding, you can then move into the evidence-based practices, because it’s all really connected. So, when Michelle talks about memory, you can’t really talk about memory without having some understanding of the mind or the brain. And, so we decided collectively, we would bring it together as hopefully a seminal piece that would really present anyone with a continuum as to: “Where am I? What am I possibly doing in my classroom?” …being able to really do that self assessment and then find the answers, as you said, in that section seven, and realize that they’re not an outlier. I mean, chances are anybody that goes through this is going to fall within that span in terms of their understanding and knowledge.

Michelle: And, what I hope is coming out here is that this study is unusual, not just in its scale, its scope and that we focused on higher education, but that it is so explicitly geared to not just identifying gaps in knowledge or awareness, but addressing those. It’s not like we came along six months later and said, “Oh, by the way, here’s a really nice resource we put together.” It is one stop, it’s right there. And, what an exercise that was, as well. Kristen, I think you’ll remember back just saying, “Okay, in a paragraph… this item, all of us look at this and go ‘oh my gosh, that’s wrong’ or ‘that’s right.’ Why is that? and what are the very best empirical sources that we will trace back to, to demonstrate that?” So, we are trying to provide that and also to really be a model to say: next time that you get that handout or that workshop that says, “Oh, here’s some great stuff about the brain.” What are they backing that up with? Can you trace it back to the solid research sources that makes some of these really powerful principles for learning, and make other things just misconceptions.

Kristen: One of the things that I would say was probably the most exciting and the most challenging. We had 10 researchers, we had 10 researchers from different fields: people from nursing, biomedical engineering, psychology; we had people who work in the area of neuroscience, education (as I mentioned), and we needed to come out with a collective voice, writing a report that would be understood across disciplines. And, so when we wrote section seven, all of us had to be reviewers and we vetted it multiple times. Not just within our group, but outside, to make sure when you read about neural pathways, it actually made sense. Because to write something where somebody would not understand or not be able to connect would be a challenge. And, we wanted people to walk away. I know one of the things that we were looking at: Why neuromyths? Well, a lot of the research out there looks at the fact that when you teach, your teaching and your pedagogy is based on your knowledge, and in your understanding of how people learn, and so we wanted to really look at this area in terms of awareness, because it may impact pedagogy. Our study did not do that. And, I want to make sure it’s really clear. Our study was not designed to say, “Oh gosh, the awareness of neuromyths wasn’t very high in this area, therefore, you must be integrating neuromyths into your teaching. That was not the intentionality of our study and that’s not something that we’ve ever said. There are certainly recommendations we put in the study to look at. If there is a high prevalence of neuromyths,how does that affect pedagogy? But ours was simply looking at awareness and could professional development address gaps? So, we could do this across all different groups that would be involved in course design and delivery.

John: That’s one of the things I really like about it, that you do address all these things well, you provide the evidence, and it’s going to be a great go to reference for those of us when faced with neuromyths, with issues about evidence-based practices. We can just go and grab some of the citations and share them back out or refer them to the whole document as I’ve done several times already. These things are really common even in professional development. I was at a session not too long ago, where there were two neuromyths presented during the session. One was the learning styles thing. But the nice thing is, unlike other times when I’ve seen that done, there were two of us who went up and waited until everyone else talked to the presenter. And, we were both ready to do it after other people had gone so we didn’t embarrass her, but it’s starting to get out there. And, I know on our campus, we’ve got a growing number of people who are aware of this partly because of the reading groups we’ve had, where we’ve had a growing number of participants… and that all started actually with Michelle’s book about five years ago now when we first did the group. You came out, you visited, people wanted to do more, so we started a reading group. We’ve done four additional reading groups since then. We’ve had many of the same participants, but it’s spreading out wider. I’m hoping we’re making a difference through these reading groups.

Michelle: And, that’s so gratifying as an author and as a researcher, and I remember well working with your group in Oswego and the great ideas I took away as well. So, I’m a big believer in virtuous cycle. So, maybe we’ve started one.

Kristen: I think what really came out of this study is the passion that everybody has for student success. Everybody from those that are offering the professional development, the instructional designers that want to make sure that the students are successful, even though they might not be teaching the course. And, then the instructors themselves… and so to be able to work with that many individuals who are not only subject matter experts across their disciplines, but so passionate about making a difference. But I think being able to integrate all of this new research relating to neuroscience, psychology and education, it’s going to transform not only how we teach, but it’s going to transform pedagogy, andragogy, and this whole concept of learning.

Rebecca: I really appreciate the bringing it together and that you decided to keep it all together and not to make three separate reports. I think it’s actually really important to understand how these are all connected and related. And, I think that’s one of the most unique things about the report. I think the community is probably very grateful that we have this resource available now.

Kristen: Oh, thank you.

John: One of the things I’ve often been concerned about is how some of these neuromyths, particularly the left brain – right brain thing, and the learning styles belief, often serves as a message to students that they can only learn in certain ways or they only have certain types of skills, and they’re not able to make progress in other ways. And, it can serve as a barrier and can lead, perhaps, to the development of a fixed mindset in students which may serve as a barrier.

Rebecca: …or not even allow those students to feel like they can enter particular disciplines.

John: If people become more aware of this, perhaps it could lead to more opportunities for our students or fewer barriers placed in the way of students.

Rebecca: …or maybe even just more inclusive pedagogy in general.

Kristen: You bring up such a great point. So, if you believe in learning styles, and you believe that you are truly a visual learner, Michelle and I’ve talked about this a lot, it almost becomes a self-fulfilling prophecy. But you probably are an incredible visual learner because you’ve been told you learn better in this learning style, so you’re going to seek materials in that learning styles. So, the challenge with that, especially when you’re looking at younger students or anybody during their education, you’re precluding really other ways to enhance your learning. So, when you look at Universal Design for Learning, it’s so important because you’re looking at multiple means of engagement, representation, action, and expression. And, when you’re looking at learning styles, if a student believes they’re a visual learner and suddenly asked to go in and take a Spanish oral exam, it could trigger, all of a sudden, stress. Well, what do we know about stress? And, Michelle can talk more about that. But, when you’re stressed, it affects working memory. And, so just that thought of, “Oh my gosh, it’s an oral exam. I’m a visual learner. How can I perform well on that?” And it’s really creating, as you talked about, a barrier or it may decrease, possibly, performance. I know that Dr. Tracey Tokuhama-Espinoza is very passionate about this. And, you’ll see in her presentations, she’ll come out and say “Neuromyths do harm.” And so, I think it’s certainly something that needs to be explored. And, Michelle, from a psychological point, I’d be curious to find out what you have to say as well.

Michelle: When you say “self-fulfilling prophecy” and things like that, it also kind of reminds me of a placebo effect, in a way… and learning styles, and continuing that as an example, yeah, I might go: “Oh, visual learning. It is absolutely me,” like “Now I feel like I can tailor all this to myself. I’ll just find teachers, opportunities, and disciplines that are right there in visual learning.” And, I might have some subjective impression that that’s helping me, or from the teacher’s perspective, I might feel like “Well, I brought in some different materials and engaged different modalities and, what do you know, because of learning styles, we’re doing better.” Well, there’s lots of different reasons why that might be happening. An individual may walk away, and maybe they weren’t individually harmed. I just feel like… just like in modern medicine, there’s sort of a promise that we can do better than mere placebos. I think that ought to be the promise of modern pedagogy as well, that we can do better than simply trying to build up expectations or giving people a false sense that they have something based on science that’s going to help them individually do better. And, I hear so many kind of missed opportunities that really kind of get me activated as well. I think about, for example, the energy that goes into faculty professional development. These things come from good impulses. I really believe that. I believe that people who really pursue something like learning styles or things like that, they want to do better and they want to be more inclusive, but that effort is directed down the wrong path simply because of this gap in knowledge and gap in information in getting the right information to the right people at the right time. And, I can’t stand the thought of faculty, especially as limited as faculty time is and as spread as thin as faculty are, to think that they might try to pick up on some better information about teaching and learning and go down the wrong path. I never want that to happen again. And, maybe our report will be a step in the right direction.

Kristen: I’ll say one thing that we’re trying to do with the report, is really to align the report with best practices and evidence-based practices. So, when you look at the concept of neuromyths the wonderful study that was written by McDonald (and this was in 2017) and her colleagues, the title is “Dispelling the Myth: training and education in neuroscience decreases but does not eliminate beliefs in neuromyths” and so professional development is not a silver bullet. Simply offering one workshop that’s going to address neuromyths is not going to necessarily get rid of neuromyths. So, we have to do what? We have to look at spacing. We have to look at interleaving. So, with professional development, how do you take information related to evidence-based practices and integrate spaced practice into our own professional development? How do we integrate interleaving? How do we integrate low-stakes assessment? So, maybe when faculty or instructional designers come in, you do a quick self assessment and find out what that baseline knowledge is, and then at the end to say, “Okay, at the end of professional development, we need to get to 95% or higher.” But, they’re able to actually test their own knowledge. So, we need to kind of turn professional development upside down and make it active learning and really engage everybody in what we’re looking at within pedagogy and andragogy.

Rebecca: Yeah, I always find it really ironic that a lot of training and things on evidence-based practices is not using evidence-based practices… or using really traditional formats: lecture or getting lectured at and not really engaging with the material. And, it’s no different when we’re working with our students. And, if they’re practicing in a way that’s not going to be effective for them, and they’re not successful. They could spend tons of time on something and just not really make progress. The same thing can happen with our faculty and staff who are designing curricula and what have you as well. They can be really invested.

Michelle: Absolutely.

John: We do have an excellent podcast on retrieval practice. In fact, it’s one of our most popular episodes. We’ll share a link to that in our show notes. We don’t yet have any podcasts on interleaved and spaced practice, but I’m sure we’ll be asking Michelle to come back and talk about these things at some point in the future, if she’s willing. So far, we’ve been focusing on the types of neuromyths that are common. What can we do to reduce the prevalence of these neuromyths?

Kristen: Professional development is certainly key. But, I would look at things such as onboarding, making sure that when people are getting hired on, that they’re really introduced to evidence-based practices from the very beginning. And, even individuals that would say, “Gosh, I’ve been in instructional design for 20 years, I’m familiar” …there may still be those gaps. And, it’s almost like adaptive learning. Everybody that comes in very much like the Vygotsky’s work of zone of proximal development, they may have all been teaching for 20 years, but it doesn’t mean that we don’t have neurodiversity in terms of experience, knowledge about different practices. So, it’s important that it’s from the very onset of when people get hired and making sure it’s understood that we’re committed to best practices, evidence-based practices and what we do builds upon the literature and the research. Not only do we introduce it here, but we move it forward and integrate it into our pedagogy and what we’re doing in our classrooms.

Rebecca: So, we always wrap up by asking: What’s next?

Michelle: Conference season is upon us. We’re recording this fall of 2019. I’m gearing up to go to the Online Learning Consortium’s Accelerate conference in November. And so, I will just personally say come find me if you’re there and you want to talk more about this. I will be presenting on a related but different topic having to do with our ongoing Attention Matters project, which is also the subject of another Tea for Teaching episode. So, I’m really working on getting ready for that, and also the upcoming POD network conference. So, for those educational developers who will be attending that, I’ll be speaking there and hopefully having lots and lots of sidebar conversations with plenty of other people who are interested and fired up about these very topics. So, I/m working on those. I’m working on what I will now call a forthcoming book. It’s under contract with West Virginia University Press, tentatively titled Remembering and Forgetting in the Age of Technology. So, maybe someday in the not too far off future, we’ll be talking about that project as well.

John: We should note that this podcast will be released during the OLC conference. In particular, it’s coming out on Wednesday of the conference.

Kristen: Oh, that’s exciting.

John: And, I should also note that we’ll be presenting there as well. I’m hoping we’ll get some people to listen to this podcast because we’re presenting the next day. So, we might get some new listeners. [LAUGHTER]

Kristen: Oh, that’s exciting. In terms of projects that I’m engaged in and working on. We’ve just launched a new lab in our School of Education at Drexel University. So, we’re bringing everything together and trying to align projects coming up for 2020. But it’s a lab called ELABS, Education, Learning, and Brain Sciences Research Collaborative. So, we’ll be looking at different studies related to the learning sciences and mind-brain education science. I am wrapping up an article with several researchers at Drexel University, some of our PhD students, that looks at immersive virtual reality and practice as well as transfer of learning. We also have a report that I’m working on. It’s an update to research that I conducted earlier on online human touch. So, I’m wrapping up that study and putting together an article there. And, then also looking at two publications for books looking at neuro plasticity and optimal learning. One would be for students to really understand neurodiversity, neuroplasticity, how you can optimize the stress response, and then looking at neuroplasticity and optimal learning from the instructor or instructional design perspective. How do you integrate this into your practice? So, those are the initiatives that I’m working on.

Rebecca: Sounds like lots of things for all of us to look forward to.

John: Thank you very much for joining us. This was a fascinating conversation. And, we’ve been looking forward to this report since I first heard a bit about it when you initially did the survey, and when I saw a preliminary presentation at all see last year.

Kristen: Well, thank you so much for having us. It’s such a pleasure to discuss this topic with you. And, I’m looking forward to listening to many of your upcoming podcasts that clearly is connected to this report.

Michelle: Thank you so much. It makes all the hard work worthwhile and we love the opportunity to get the work out to exactly the people with the power to spread it to faculty and instructional designers and leaders in universities today.

Rebecca: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

107. Project NExT

Faculty beginning their teaching careers often rely on the teaching methods that were inflicted on them when they were students. These practices are not always consistent with evidence on how we learn. In this episode, for Assistant Professors from the Math Department at SUNY-Oswego join us to discuss how our math department is transforming its instructional practices through the use of professional development opportunities provided by the Mathematical Association of America.

Show Notes

Transcript

John: Faculty beginning their teaching careers often rely on the teaching methods that were inflicted on them when they were students. These practices are not always consistent with evidence on how we learn. In this episode, we examine how one department is transforming its instructional practices through the use of professional development opportunities provided by its national professional organization.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Today we are joined by four assistant professors from the Department of Mathematics at SUNY Oswego. Our guests are:

Sarah: Sarah Hanusch.

Rasika: Rasika Churchill.

Jessalyn: Jessalyn Bolkema.

Zoe: And I’m Zoe Misiewicz.

Rebecca: Welcome everyone!

John: Our teas today are:

Rasika: I’m having Earl Grey.

Jessalyn: I just poured myself a cup of lemon ginger.

Sarah: I’m not having any tea today. I’m not much of a tea drinker.

Zoe: I’m not having any tea today either. I just haven’t unpacked to that point yet.

Rebecca: And I have… English afternoon.

John: And I have Bing Cherry Black Tea. So, we invited you here to talk about Project NExT, which is something that people in our math department have been involved with. Could you tell us what Project NExT is?

Sarah: So, Project NExT stands for New Experiences in Teaching. It’s a program that is sponsored by the Math Association of America that brings new mathematics faculty…so you have to be in your first or second year of a full-time job…but they bring these new mathematicians in from all over the country to teach them about active learning.

Rebecca: How did your involvement, or the department’s involvement, with Project NExT get started?

Sarah: I learned about it as a graduate student, and was highly encouraged by a lot of people to apply. And so I kind of brought it into the department by saying, “Dear Department Chair, will you pay for this?” And since then, in part because of my starting it, we’ve encouraged everyone we’ve hired to apply. And as a result, there’s now five members of the department that have either completed or are still in Project NExT.

Jessalyn: Yeah, I will echo that experience. It was something that I was aware of as a graduate student, in part because some of my mentors had gone through Project NExT…it’s now 25 years old…just celebrated 25 years. And so for me, it was something that I knew I was interested in. And in fact, when I visited Oswego for a campus interview, and the department said “Oh, yeah, we have Project NExT fellows on the faculty, and we would be happy to support you in that,” that was a really exciting and encouraging thing about the department.

Rasika: For me, actually, I didn’t heard about that before. But, when I got the job offer, it came with that. I said “Yeah, sure.”

Zoe: I was just hired this past year and so I’m doing Project NExT, but I think I can already see the effects that it has had. It was a program I already knew about, I really wanted to participate in. So, as I was going through the hiring process, one of the first things I would ask the chair at a place was “Would you support an application for Project NExT?” …because it does require a bit of funding. And so seeing that there were already multiple Project NExT fellows in this department was also a good sign for the department as a whole when I was thinking of what sort of department I’d want to be at. And so I think it’s just showing that it’s already been recruiting people who are interested in it already, at this point.

Sarah: I was just going to clarify a little bit about how the funding for it works. There’s actually no fee to participate in Project NExT. The way it’s organized is that you attend special sessions at three of the national conferences in mathematics. So, you attend two math fests in the summer, and then the joint math meetings, which is in January. And so these are big nationwide meetings in mathematics. And so the idea is that you’re going for some special sessions during the meetings. And then your first year, you go for a couple days pre-conference for the really heavy duty workshop. So, the financial commitment from a department is just the funding to go to those three conferences.

Rebecca: You mentioned active learning. Can you talk a little bit more about how those workshops and things are structured?

Zoe: There were a lot of workshops about active learning and just using evidence-based pedagogy, so saying not only active learning is good, but we have evidence to support it and here are some of the things that you could do in terms of active learning. And all the sessions obviously are structured with that in mind. So, we’re not just sitting there listening passively to someone tell us about active learning, but they really make sure you’re doing something, whether it’s a fun little game like building a marshmallow tower, or some other interactive activity in each session. The sessions aren’t only about active learning, there’s a lot about inclusivity and diversifying the profession. So, a lot of sessions on that, or maybe I just chose sessions on that. But, there’s also a whole professional development stream. So, there’s stuff about how to get started in your career in terms of grants and so on. It’s really a lot of everything in there.

Rasika: It’s categorized like if you interest on the tactile learning, so are you interest on the group work, are you interest on some other…you know, inquiry based and mastery grading and so forth. So, depending on your interest, actually, they give more opportunity to listen, go talk with people and have a conversation: what they had, what they tried and what failed and what succeed. Which is like a really nice thing for us, as a beginner, to see what people have gone through and what I should expect, and so forth. Actually, I was interested about the whole program.

Sarah: So, they do some three-hour breakout workshops where you get to go based on what your interests are. So, I did one that was focused on teaching future educators because that’s my background, but I doubt any of these other ladies chose that same session because that’s not their expertise and not what their job is going to be about fundamentally.

Jessalyn: I will add, I attended two workshops that stand out to me in retrospect. One on making active learning intentionally inclusive. That was all about inclusive pedagogies and ways to incorporate group work in the classroom in a way that benefits all students and allows all students to participate fully. I also did a longer breakout workshop that was building a toolkit for student-centered assessment, that was all about learning objectives and exam structures from a more experienced instructor. And then there are also facets of Project NExT that extend well beyond the physically meeting in person. So, as Rasika mentioned, there are lots of ways that you can navigate the workshop according to themes that are of particular interest to you. So, if tactile learning or kinetic activities are of interest, or you’re really focused on educating future teachers or whatever that might be, you might be encouraged to declare a goal for yourself in your first year related to one of those areas of interest. And then we’ve got little email exchanges that go on for people who’ve declared interest in one of those goals like “this email list is all about mastery based grading, check in when you’ve tried something. check in with your questions.” So, there’s a little bit of accountability built into that structure that these people know what you’re trying to do, and they’re going to check in with you on it. But, then just the larger structure of email lists is that you have this cohort of other new instructors who will fire off questions like, “Oh, I’m teaching this class next semester I’ve never taught before, what textbook might I use?” or “I had this really strange interaction in my classroom, and I’m not sure how to handle it” or “I think this part of my syllabus is just crashing and burning. Help! Has anyone been here before?” And so you have this sort of communal resource and the community experience of brainstorming and problem solving together.

Sarah: …and included in that they assign each of us a mentor. So, a more experienced instructor that’s a mentor is assigned to each person in the program currently, and it’s always someone that is outside of your department. In fact, they will not allow anyone to be a mentor who has a fellow in their department. So, as long as we keep having fellows, we won’t have any mentors here. But, what’s nice is when you do send emails out on that list of “I’m trying this and it’s not going well, help!” you do get responses from your peers. You also see responses from all of the mentors for that cohort, which I think is also valuable because sometimes they have a little more experience than your actual cohort.

Rasika: We have a group that people who are interested on the inquiry based or tactile work, they have their own little Zoom conversation whenever they have time together. You get to know all different schools, what they’re doing and, you know, share your experience.

Rebecca: Would you all like to talk a little bit about how Project NExT has influenced your own teaching?

Rasika: For me actually, I was really interest on the tactile experience from this Project NExT. So, I decided to do some activities this semester starting as a beginner and also some group work. And also something that… not exactly what I’m getting from the Project NExT, but it’s like I will say, part of the SUNY Oswego Reading Group, that I was so interested on the book that we are reading. And I decided to give a couple of pages for the students every week to read, and I assigned them 5% for the final grade that they have to read and write half a page to one-page report for me and tell me what they think. Do they think like it’s feasible for them to change and try and do the things in that nature? So far, it’s really going well, and I have good comments from students saying that “you are opening up different ways of thinking…that we were stuck and never complaining about everything. But, we are now having, you know, in a broader way of looking at the things about growth mindset and so forth.” So, I was speaking here and there like chapters from some interesting books. So, that’s what my experience so far this semester, as a beginner.

Sarah: I think for me, it just gave me a lot more lesson plans and ideas to draw from. I already had a pretty active approach to my teaching, but it just opened a broader view of what kinds of things could work well. Especially in some of the more tactile things available that can be helpful for helping students to learn.

Jessalyn: Within my own teaching, I think it’s been really easy or natural to draw on resources from Project NExT in setting up my class or setting up lessons. When I taught Calc I, on day one, we made zip lines out of ribbon and key chains and measured average velocities and it was fun and it was memorable and it got students working in groups and they reported at the end of the semester. “Hey, remember when we did zipline? That was fun!” and I 100% would not have pushed myself to do something that involved or non standard, I’ll say, without thecontext of Project NExT saying “Oh, just try one new thing each semester.” I completely overhauled the Calc II class to be entirely mastery based grading in response to some of my own frustrations with how I had been setting up my class. And Project NExT supplied a whole lot of resources, a whole lot of people, a whole lot of information and motivation to try something like that, which I think was helpful. As far as department culture goes, I think the fact that we’ve had this many Project NExT fellows and continue to have Project NExT fellows gives us a shared language to talk about teaching. Some shared frame of reference on “Oh, yeah, you know, this person who tried this technique,” or “Have you heard anything about…. “”…Oh, hey, this came through on my Project NExT list.” That I think has encouraged just our conversations about teaching and being intentional in how we’re structuring our classes, or how we’re handling things.

Sarah: I’m experimenting with mastery based grading this semester because of the information you and John got, from your experience in Project NExT. And so your experiments with it last year has led me to experiment with it this year. So, it definitely has changed just how we even hear about new things to try.

Jessalyn: That’s delightful. I appreciate that it’s trickling around.

Sarah: It is trickling for sure.

Zoe: So, I’d say it’s still obviously fairly early. We’re only one month into my first semester after going through the first part of Project NExT. But, I’d say a lot of it has been both an affirmation of things that I have been doing and also it’s sort of given me the confidence to do the things that I was doing even more fully and to advocate for these approaches, even though I am brand new in this department. So, I’m not afraid to send to the whole department email list like “we need to be more positive toward our students and not say that it’s all their fault if they’re struggling. we need to take responsibility for that.” Or just to try things that may or may not work well. For example, I’m doing mastery-based grading just of the homework in my general education math course. And I’m using an online system that,it turns out, is not that great for mastery-based grading of that course, even though I’ve used it for other courses. Students, I think, still benefit from it, but it’s not quite as effective as I might have hoped. But, I’m just willing to try these things and willing to speak up about things, so those are the main impacts in my courses.

John: Could you tell us a little bit about how you’ve implemented mastery learning technique?

Sarah: I think we’ve all done it a little different. Why don’t you start, Zoe, since you were just talking about it.

Zoe: I’ve done it only in the homework, so not in their exams. So, the homework is done online, it’s 15% of their grade. And so for each little subtopic, they have to do a little quiz. It’s five questions: three medium, one easy, one hard, and they need to get at least 90% on it. And they can try as many times as they want, but they do have to keep trying. And so, in courses like college algebra…is the one that’s most similar to where I’ve done it before…the material all builds on itself and it divides nicely into little component and there, I’d say it’s going well. The students complain about it at the beginning, but already after I asked them to reflect on their first test performance, a lot of people said, “Oh, it’s actually really helpful that I had to go back and keep learning these things until I fully understood them.” Whereas in the first couple of weeks, there’s always a bit of pushback about “Why do we need to get 90% on this. It’s too hard to get 90%…couldn’t it be lower?” And then once the results come in, they see it’s worthwhile. The other course I’m doing is similar, the gen ed math course…it’s also their online homework…15% of their grade, but that textbook just doesn’t break down the material into as nice sections and the questions are longer and the grading of the online system is pickier. So, that one has some issues, but the same basic idea.

John: Are you using publisher provided questions then, and tools?

Zoe: Yeah, publisher provided questions and tools.

John: Are you allowing unlimited attempts or is a limit on the number of attempts?

Zoe: Yeah, unlimited attempts, and flexible deadlines too. So, I do say they need to achieve a certain amount before each of the test. But, the idea is that if you haven’t yet mastered something, you can still go back and do it several weeks later. As you keep practicing the material, we keep building on it. So, it’s not that you have just one chance and you’re done. The goal is to get them all to understand it fully by the end of the semester.

Jessalyn: My approach to mastery based grading in my first implementation was to go totally off the deep end, and just structure the whole class with a mastery-based grading scheme. So, what this meant was that I did away with midterm exams, everything was broken down into learning objectives roughly correlated to the sections that we were intending to cover in the textbook. And the primary mode of assessment was quizzes. So, my students had quizzes that they could retake as many times as they needed to. And each quiz had three questions and I wrote problem banks of many many questions for each quiz. And in order to earn an A at the end of the semester, the expectation was something like, “Oh, you need 18 of your quizzes to be three out of three and the rest of you two out of three.” So, it was not a points accumulation scheme, it was just quizzes and repeated quizzes. They also had online homework through web work and that was unlimited attempts. There were deadlines, and they just needed to… there was sort of a threshold percentage associated to an A or a B, or a C. And then I had a few more other activities and elements going on. But, primarily, the structure involves these mastery quizzes. And I owe a great deal in the structure of this class to Laura Taalman from James Madison University, who shared a lot about how she structured her class that way and so I sort of borrowed and adapted from her setup for my experiment.

Sarah: So, my class is pretty similar to Jess’s. The main difference is I’m doing it in a proof-based course, so it’s fewer questions. She had three questions per objective. I have one, because they’re a little bit longer questions. The only exam in my class this semester is the final and that’s only because I’m required to have some common questions on a final exam. So, I had to have a final exam…instead I’m doing weekly quizzes. Each week, we add one to two new objectives. There’s about 20 for the entire semester. So, our first week we had two questions on the quiz. The second week, we had four questions on the quiz, but questions one and two were the same objective as one and two from the first quiz. So, the questions are just going to grow cumulatively…so our last quiz will have about 20 questions on it. Although I did tell them once everyone has mastered a question, it’s just going to say mastered, it’s going to be no new question writing and at some point, I’m going to recycle some of the early ones.

John: Your building in some interleaved practice and spaced practice as well.

Sarah: But, the idea is that once they have mastered a question, they no longer have to do it again. They’ll have the questions for practicing and for getting ready for the final. In addition to these mastery quizzes, I’m having them write a portfolio, which is going to have a little bit more of that interleaving practice and making sure that at the end of the semester, they still remember how to write some of these early proofs and it’s also to focus on the writing aspect. So, to help make sure they’re really using the language precisely. Sometimes with a quiz when it’s timed, you’re a little more flexible, but I want to make sure that they have that precision of language down by the end of the semester. So, I’m sort of balancing those two aspects of it that way. They have “unlimited attempts” in air quotes…restricted by what? …there’s 12 times I could quiz during the semester…13 for something…So, restricted to…they need to do number one all semester long. They can have all semester to do it, but we are eventually going to run out of time.

Rasika: So, for me, I haven’t tried to mastery based grading yet. Maybe in the future.

John: Are there any other new techniques any of you have used in your classes?

Sarah: I’ve done a lot of experiments with this idea of embodied cognition, where you actually have students sort of using their bodies to experience things mathematically. One way that we did this with my pre-service elementary school teachers, I give them a bunch of clothesline, and I have them make a circle. So, you may think, “Okay, no big deal.” But, what happens is, it’s not good enough until it’s a perfect circle. Part of this is to elicit the definition of a circle, because to non-mathematicians, I’m going to pick on you for just a moment, Rebecca, how would you define a circle?

Rebecca: One continuous line that’s in a loop.

Sarah: So, a lot of times they come up with something like that. Well, how does that distinguish, though, a circle from an oval. So, it’s not really a precise definition of a circle, right? With the precise definition is being it’s all of the points that are a fixed distance from the center. But, what happens is, by forcing them to make their circle better and better and better and better, they actually all know that’s the definition of the circle. Maybe they don’t remember it, but they know that there’s this radius thing involved. And so by not allowing them to sort of quit until they actually are in a perfect circle, the only way to do that is you have someone stand in the center, and you take another piece of clothesline to measure your radius, and you move everyone in and out as appropriate. So, that activity of physically making the circle and by having to have that person in the center, and that radius gets them to say the definition of the circle properly, first of all, but they get to experience it in a way that they don’t get to otherwise. And that’s an activity that I never would have thought of without going to Tensia Soto’s session at my first Project NExT meetings.

John: It is certainly safer than giving them all compasses with sharp points where they can stab each other, which was how people used to do it.

Sarah: We still do compass and straightedge constructions in geometry, but again, that doesn’t actually help you really understand what the definition is. I think doing this physically actually helps them understand why a compass works. I know that sounds silly, but it really helps make those kinds of connections. I have another activity where we take clothesline and I make a triangle on the ground, and I make them walk the interior angles of the triangle and you spin 180 degrees and it, again, helps them experience that the sum of the internal angles of a triangle is 180 degrees. And again, that’s something that, the first time I did it, it was baffling because first of all, it’s hard to turn the interior angles. Your instinct is to turn the exterior ones, but you end up backwards. From a geometry standpoint, it makes sense, but somehow that physical aspect just really changes things.

John: It makes for a much more memorable experience, where they’re seeing things from a different perspective. And I think that’s really useful.

Sarah: I agree. That’s why I do it.

Rebecca: Does anyone else want to chime in about how having so many fellows from Project NExT has influenced the larger department? Because you’re not just five people in your department, you’re how many?

Sarah: There’s 14 of us tenure/tenure-track now. I do think it’s changing the way some things are done. It’s slow going. I think everyone would concur with that. Jessalyns’s smirk is definitely confirming that. It’s slow going, some of us would like change to happen faster. But, I do think change is happening. I think there’s a lot of respect from our colleagues that we are trying new things. I think a lot of them have a “You can do what you want, but don’t make me change yet.” But, I think we’re starting to get them a little bit, too.

John: If your students are more successful, that often convinces people and sometimes when students say, “I did this in this other class and was really helpful,” that’s often really persuasive to other faculty. But, it’s convenient that you had so many people all come in at once, because that’s not typical in most departments that have such a large cohort, in a short period of time.

Sarah: We have had a lot of retirements, one right back on top of each other. So, we have had an influx of young faculty in our department, which…that alone…to have so many in this program as well. Definitely.

Rebecca: I think it really helps to have models of ways that you can do things because if you didn’t learn using these methods or you didn’t have exposure to that as a student, you have no way of knowing how those really play out unless you have examples. So, it sounds like Project NExT played that role for you, but then you are playing that role for other faculty in your department.

Jessalyn: Thinking about department culture more broadly, not just among discussions and relationship among faculty, but in terms of the student experience, and this engagement that we’re having from our majors and the sort of activities that we’re involving them in. I think there has been a Project NExT influence there as well. Sarah, you and John started the Putnam Competition before I came even and a lot of other conversations and gatherings have come out of that, like we’re getting together with our majors and talking about preparing them for graduate school if that’s something they want to do. The math club or other organizations have taken on a different role in the department and I think a lot of that comes out of some of the ideas in Project NExT, like hearing about how another department celebrates their students participating in something like the Putnam Competition. But, it also comes out of the relationships you build in an active learning classroom and the way that we connect with students when we are trying new things. And we’re being honest with them and saying, “Hey, I’m trying something new. And I’m going to want your feedback.” The community that you build in a classroom flows into the community that we support and foster as a department.

Zoe: So, it’s a bit hard for me to talk about departmental culture change in the one month that I’ve been there not having seen it before I did Project NExT. But, I can certainly talk about how the department seems different from other departments, just in the willingness to embrace new ideas. And there’s also a sense that these ideas are just supported. Even if we haven’t had an explicit conversation, I know that there will be support for trying something new that was suggested in Project NExT. And it seems, when it comes time to make policies, that we have almost a majority just of Project NExT people. Obviously, we need a couple more people, but there are other people who haven’t participated in the program who would still support these sorts of initiatives. Knowing that that base of similar views is there, makes a big difference in what sorts of ideas we would even suggest or consider.

Sarah: I think a lot of our Project NExT fellows have also been very active with doing undergraduate research with students.

Rasika: I think even like talking to colleagues. For me, like I have a personal experience, because my husband is also a mathematician and teach at SUNY Oswego. If I learn something new, I share with him of course, he’s not a Project NExT fellow, but…

Rebecca: So, it sounds like the program’s working really well. You’re all really excited about it. It sounds like it’s engaging all of you. So, glad that you’re able to share it with us.

Sarah: The MAA has definitely done a lot to support improving teaching in mathematics and I do think it is a program that other disciplines could look at and possibly model. I will say they have put a lot of money and a lot of investment into making this a success. It is well run and has been well funded, which is a testament to how important professional organization views it.

John: We always wrap up by asking, what’s next?

Sarah: Well one thing that’s next is we’re trying to get one of our other new faculty, his application was rejected last year. We’re also hiring two people, hopefully this year…So, possibly trying to send them next year as well.

Jessalyn: Another immediate thing that’s next is that our two current NExT fellows will be attending the joint math meetings in January and maybe organizing some Project NExT sessions or at least attending some sessions.

Zoe: I’ll be helping to organize a session on getting started in math education research, which I was made part of because I said it was something I wanted to do, but it’s not something I have any background in. So, I’m finding it a bit of a challenge to assist in this organizational process. But, I also, possibly for Math Fest next summer, helping organize a session on reducing math anxiety, which is something that a previous NExT fellow who I follow on Twitter help organize this session. So, having attended NExT, I think, gave me the confidence to respond on Twitter to this senior mathematician and say, “Oh, yes, I’m interested in this topic.” And so that will come later. And that’s something I actually feel like you could contribute to in a meaningful way, unlike math education.

Rebecca: Well, thanks so much for joining us. This has been really interesting.

Sarah: It’s our pleasure.

Jessalyn: Yeah, thanks for having us.

Zoe: Thank you very much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Brittany Jones and Kiara Montero.

98. Developing Metacognition

Many students arrive in our classes with relatively little understanding of how they learn. In this episode, Dr. Judith Boettcher joins us to discuss how well structured project-based or problem-based learning activities can help students develop their metacognitive skills so that they become more successful as learners.

Dr. Boettcher is the author of many books and articles on higher education and has long been a leader in the field of online education. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips, co-authored by Judith has been an important resource for faculty transitioning to online teaching. At Oswego (and many other institutions), many faculty have been using materials that Judith has developed for ACUE (the Association of College and University Educators).

Show Notes

Transcript

John: Many students arrive in our classes with relatively little understanding of how they learn. In this episode, we examine how well structured project-based or problem-based learning activities can help students develop their metacognitive skills so that they become more successful as learners.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Today our guest is Dr. Judith Boettcher. She is the author of many books and articles on higher education and has long been a leader in the field of online education. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips, co-authored by Judith has been an important resource for faculty transitioning to online teaching. At Oswego many of our faculty have been using materials that Judith has developed for ACUE (the Association of College and University Educators). Welcome Judith.

Judith: Thank you very much, Rebecca and John, it’s great to be here.

John: We’re really pleased to have you here. Our teas today are… are you drinking tea?

Judith: I actually… yes, made a special point. Here’s my cup, which you can’t see. But I chose one of my favorite teas, which is a lemon and ginger tea from England, of course,

John: the Twinings version.

Judith: No, this is a Hamptons tea from London.

John: We have the Twinings version of that in our tea collection.

Rebecca: I’m drinking Jasmine green tea today.

John: And I’m drinking oolong tea today.

Judith: Sounds good.

Rebecca: Judith, your mug looks really interesting. Is it abstract art. Is that what was on it?

Judith: It’s actually pears.

Rebecca: Okay, I only saw the bottom part of it. I can see it now.

Judith: Yeah, right. Well, I will confess that at some point, I finally decided to clear every vendor cup out of my cupboard.

Rebecca: That sounds refreshing.

Judith: Yes.

John: I have vendor cups all over… in my vehicles… in my offices, everywhere.

Judith: Well, that was part of my retirement process that I went through. I said, “Okay, that’s it.”

John: One of the things we had trouble with is picking a topic because you’ve worked on so many topics, but we settled on having you talk a little bit about how students can work to improve their metacognition using project-based or problem-based learning. But before we do that, could you talk a little bit about what metacognition means?

Judith: I would really like to John, partially because I had this book with the title of metacognition that I was reading when I was with a family event. And one of my relatives said, “What in the world is this? Meta what?” [LAUGHTER] So, an easy way I like to think about metacognition is the definition of just it’s thinking about thinking. It’s a definition, I think. that we can all just really grab on to and we can really use. But then I kind of like to expand that definition into one that’s in the How People Learn report that I go back to pretty regularly and that is that “metacognition is the process of reflecting on and directing our own learning.” And I really like that one, because it’s got the two steps, I think, what we want to really kind of focus on with metacognition, and that is reflection… really stopping thinking, pausing… and then actually directing our own thinking, because that leads to action. So then we have reflection and action, which I think is the core of metacognition skills.

Rebecca: One of the things I think we often talk about in education context is this reflection piece. And we always tell students to reflect, but we don’t always give them the time and space to do that.

Judith: Yeah. [LAUGHTER] Very much.

Rebecca: So can you talk a little bit about project-based and problem-based learning and how metacognition connects to those rather than standard ways of operating in the classroom?

Judith: Yes, I would love to do that. In fact, that was one of the first things when we’ve got into online learning, was that there was a real struggle as to how do we maintain the security and everything of people taking tests. And so it turned out that we decided that one of the best ways of gathering evidence of student learning was not by doing these tests… that we would actually have the students do projects. And that kind of evolved into the following process. And that is that in some of the work I’ve done for ACUE, and also in the book, I’ve mentioned that I really like to design a course that starts with the students selecting and doing a project, actually in week one or two of the course. And then that students actually focus on that project throughout the entire course. And that’s the mechanism by which we gather evidence of student learning. It kind of also avoids this whole process of buying papers and buying other kinds of things. Because you really have milestones along the way.

I’m going to stop and talk about the first step for just a moment. And that is choosing a project. One of the wonderful things about choosing a project is that then students actually have to stop and then think about the kind of project that they want to do. It gives them an opportunity to actually customize a course to their interests, which starts getting past that big motivation problem that sometimes teachers might say, “Well, how do I motivate them?” Well, we give them the opportunity to choose something that is of interest to them. So they choose a project, and then they actually write up what that project is that they want to do. But that’s not the end of selecting the project. Before the project is really kind of finalized that they’re going to be working on, they actually then sit down and talk with some of their other students and the other peers. They switch and swap their proposed project descriptions, so that they actually talk out loud about the project. And then hopefully, by talking to their peers about it, they get some additional ideas, and they refine it a bit. And then it goes to the faculty member. So the faculty member doesn’t get it right away, but it goes through this first the individual students thinking about it, and then the other students thinking about it, and then the faculty member can take a look at it. And that’s only milestone one that can take up to about the third week of a course. But by then, hopefully, that gives them a real focus of the course rather than having just the topics thrown at them from week one through week 16… that they’ve got a real focus of: “Oh, how is this going to affect my project?” We can come back to that I just want to mention briefly then there’s like four other milestones, five other milestones for every project throughout the course. And the first is that project description, then the second one would be planning how do I plan to do it, which is a really important metacognitive skill. And then another milestone would be some type of other “Just checking in, how are you doing?” …kind of a thing. And then there’s two final things. One is where they actually share their project with the other students, like in a mini conference, whatever. And then the final thing is the actual final thing, which might not be a paper, it might be a video or an interview, it could be one of a number of things that would go into their student portfolio then. But, that gives the student a focus throughout the entire course that way. That’s a lot, isn’t it?

Rebecca: It is a lot. [LAUGHTER]

John: When you have them give feedback to each other, do you recommend that that’s done synchronously or asynchronously If it’s an online course?

Judith: You know it’s actually best, I think, that the students do it somewhat like a brainstorming event. What you and I are doing right now here with Zoom, students can do with FaceTime, they can do it with just an online gathering chat, whatever. If they’re really, really busy online students, and they have to do it asynchronously, that can be done too with email. I mean, that works too. But it’s often really great for the students themselves to talk out loud. We don’t have them talking enough, I think. They read passively, and they kind of think and everything else, but we don’t have ways for them to use their voice to talk about what they’re thinking.

John: It sounds like the project’s really nicely scaffoldied. But how do you bring in the metacognitive development? How do you get students to improve their metacognitive skills? Is that something they’re explicitly thinking about? Or is it something that’s done as part of the structure of the project?

Judith: Well, of course, it depends on how the faculty member wants to do it. As I had the chance to go back and look more intensely at the metacognitive skills, it occurs to me that thinking metacognitively is such a basic intellectual skill in many respects, particularly now in our 21st century. It’s as fundamental as reading and writing. And, you know, I think we need to look at our entire curriculum from pre-K through whatever as to really how do we explicitly teach and model and coach metacognition skills, which includes with that initial project proposal, we can talk about the fact that what they need to do is think and be sure to build on their interest to give them the criteria and coaching as to how that’s going to really work for them. It’s really a problem when they have to select a project. So what are the constraints on that problem? What are the features or benefits that they’re going to get from this? …and to build that into the assignment and actually writing out the initial proposal and then meeting with their colleagues and peer students, that’s all part of the coaching and the modeling of the metacognitive process.

Rebecca: I think your emphasis on talking about things is really interesting. Because I find that students often will passively write something, and it makes no sense to anybody. But as soon as they try to explain it out loud, they realize it makes no sense. Because when it’s in writing, or they’re just reading in their head, they don’t often have that realization. But as soon as you try to say it out loud as a sentence, it’s not structured in a way that makes sense. They catch themselves or they realize, “Wait, there’s a big hole in what I’m talking about here… nobody knows what I’m talking about.” You can tell it by other people’s faces looking blankly back at you.

Judith: And then they realize that they don’t know what they’re talking about.

Rebecca: Yeah. [LAUGHTER]

Judith: So yes, in fact, that’s apart of the power of talking with their peers about the project, because the peers would most likely say, “Well, wait a minute, why did you choose this? Why are you interested in this?” And so that’s when they have to dig more deeply. It almost goes back to that Socratic questioning, “Why do you think that? Why is that important to you? Do you think this is going to make any difference in your life?” One of the things I really recommend in online learning is that the students, when they read and look at the objectives for a course, to stop… and in the first week, actually, to have the students process those learning objectives, and then personalize one or two of them and say, “This is what I really want to make certain I know when I finish this course. This is how I’m going to be a different person.” Thinking “What skill am I going to be able to do when I finish this course?” They kind of set that goal and set those expectations early on. Hopefully, students setting those goals and objectives become again, more personal to them, and something that they can check themselves. And then as they go through the course, “What progress am I making on this particular goal? How is this coming?” …kind of fun.

Rebecca: Are there ways that you would suggest a faculty member model metacognition early on in an online course?

Judith: Well, I think a really easy way for faculty to model meta cognitive skills is… say, for example, in biology or one of sciences, one of the things I love to look at is the biography of a scientist and looking at, “Well, how did they come to this point? What made them think X, Y, or Z as opposed to A, B, or C?” and we find it’s really rooted in their personal lives and their thinking that they’ve been doing. So faculty member can somewhat do the same thing, in fact, very early on in their introduction to the course. Now, what do you teach Rebecca?

Rebecca: I teach web design.

Judith: You teach web design, okay. So in your introduction, then, to your students, you can say the reason I love web design is as follows. And you can go back into your life and your experiences and say, This is what happened to me… and this is what I was thinking… this is how I got to this point. So you really speak out loud, and you share your processes by which you arrived at that point. And we can do that for any kind of thinking… as we’re talking about an experiment. We do want faculty to share their expertise and their dissertation, for example, they say, “Wow, you know, this is why I’m interested in this. And you know, after 25-30 years, I’m still interested in this because of this.” So they share their thinking processes. Another really easy time is when students ask them a question. And obviously, this can happen online in the synchronous activities. And the students might ask them a question about, “Well, gee, Professor, so and so what does this really mean in this instance, or in this context?” And the faculty member can say, “You know, I really am not sure about that. And I would like to, before I answer that, I would like to do a little more research and thinking, and I will get back to you on that.” So it sends the message that the process of thinking is something that we keep doing all the time.

Rebecca: I think that moment of admitting that you’re not always the expert in everything in the moment, is always a good thing for students. And they respond really positively to that.

Judith: That’s right. It also gives them the opportunity to say, “You know, Professor, I would like a little more time to think about that. Let me get back to you on that.”

John: How can we tell that students have improved their metacognitive skills? I like the idea of having the peer instruction do that. Or to work with students to help them recognize what they know and what they don’t know. But how can we measure that? Or how can students know that they’re more metacognitively aware? How can they observe improvement?

Judith: Well, one of the things I did as I was thinking about this was, you know, I was kind of preparing for you asking me the question, “What are the actual metacognitive skills?” and I came up with five thinking skills that we really want to encourage and model for students. The first one is an easy one. And that is, number one, we really want to encourage students to think. I know that sounds somewhat simplistic, but oftentimes think about in the online environment, we give them an assignment and students, they don’t even read it, really, they kind of scan it. And they kind of assume without thinking that that’s what that assignment is going to give back. And so one metacognitive skill is to pause and stop to really take some time to think and process. In fact, I kind of like to think about the assignment as a briefing. You know, we use briefings in business and politics and detective crime solving and all the rest of it we use briefings. Maybe as a faculty member, you want to encourage the students to think of an assignment as a briefing and a briefing is, “Well, this is what we know now, what do we want to know about next? How are we going to find out what that next is? And when you finish this assignment, what do you expect to know or think about or become clear about?” So step number one is really pausing and thinking. There’s a book out recently called, was it Kahneman? It’s that Thinking Fast and Slow or Slow and Fast?

John: Yes, Thinking Fast and Slow. It’s a great book.

Judith: Thinking Fast and Slow, right. Well, you know, he makes the point that we live in such a fast-paced world that our first response to anything is that, “Oh, I know the answer to that.” It’s like Jeopardy, and you really don’t stop and pause and think. And so, because it’s really easy, that’s the short-term memory, we don’t have to think about it. We know that in order for learning to occur, that we really have to stop and give time for information going into our short-term memory to get into the long-term memory. The only way anything goes from the short-term to the long-term memory is for us to connect what we know already with the new information coming in. And you know what? …that takes time. So that’s the one thing for all of us. Time is so seriously, we don’t want to take the time. But if we don’t take the time, the only way learning occurs, you know, if we grow dendrites in our brain, and so if we don’t take the time the dendrites don’t grow, and nothing lasts. It’s kind of a fast bullet shot, so to speak. And then we forget it. As soon as we use it in a sentence, it’s gone. So the idea that we have to really discipline ourselves to stop and think and process, what is it that I’m needing to do? What is it that I want to learn? What do I want to get out of this? And when I finish this assignment, how will I know that I’ve really finished the assignment. Which brings us to the next thinking skills. One of the things I really liked, I use myself a great deal. And this is kind of a hard won practice of my own. And that is that, whenever I was given an assignment whether it was at work or wherever, whenever I had a hard time getting started… you know, getting started is sometimes one of the most difficult parts of anything, right? So anyhow, I finally figured out that one of the reasons it was hard getting started was that I didn’t quite know what it was I was going to do. I finally realized that my best practice is that I visualize what it is going to be when I finish. I get requests to review articles for journals, etc, etc. Well, I know now that my visualization works is that I’ll be done when I finish answering all those questions, you know, read the article, finish answering all the questions, I compose my response back to the editor. And that’s when I’m done, okay? So with any kind of a project, we want to encourage the students to say, “Okay, what is my assignment going to look like when I’m really done? When I have finished reading this article, or reading this core seminal research project? When will I really be done? Will I be done when I can explain the research to someone else? Will I be done when I can actually implement these ideas in a web design project? Is that when I’m going to be done? Just what is that project going to look like when I’m really done?” So we just celebrated the Apollo landing on the moon, in July. I just had the recent opportunity to see the Apollo 11 program that they created out of the original footage from everything that was happening at Cape Kennedy and Houston… the pictures, it showed like, literally, a room full… it almost look like 100 guys, and they were all guys at that time… 100 guys sitting at computer terminals, and you think when they worked on that project, how did they envision success? They had to envision success as actually a man being on the moon. How did he get to the moon? What did he land on the moon in? And then how are we going to get him back from the moon safely? I mean, think of all the things that had to be planned and worked on and everything had to be coordinated to make that thing happen.

So thinking skills, we have to think, we have to visualize, and then we have to plan. Once we know what that final vision is, we need to then plan each of those steps along the way. And again, we can model some of that and coach that by building the planning into the assignments for the project. We definitely want the students to give me a plan with the date, the milestones, the resources they’re going to be needing,]… if they need to make appointments or interviews when that’s going to happen. So we help them realize that projects just don’t happen. They happen after all of these various steps. And then, of course, we build in step number four, evaluating and pausing to then debrief each step along the way: How am I doing? Do I need more time? What else might I need? What else do I really want to know, if I do have enough time? I mean, this really does happen with web design, right? We get to a certain point, we think of it’d be great if I did this. Great if I did that. Do I have enough time to do that? Do I have the knowledge to do that? Or do I need to learn a new skill to insert that into that. So those are all questions that we want to ask encourage the students to plan for and to ask along the way. And then of course, the final one is the final debriefing. When you hand something in, you get feedback both from your peers and from the faculty member. And in many online courses, you do have these little mini conferences where you invite alumni or experts or just friends in to say take a look at this and see what you think.

Rebecca: I like your framework of the briefing. And it’s actually one that I use in my classes pretty regularly.

Judith: Oh, great.

Rebecca: When I’m doing long term projects, I have students on a weekly basis do basically a little briefing of what did they do? What do they need to do? What are the planning steps? So not just like that big scope of the whole project, but on a very routine basis, checking in with their schedule and checking in with their plan and what they’re struggling with to demonstrate what they’re doing, and what their thought processes is as they’re working on it. So I asked them, “What were the big design decisions you made this week? And what did you base those on?”

Judith: Wonderful. In fact, I sometimes like to use the example of Mark Harmon, the actor in NCIS. I love those programs, actually. Like every morning, he kind of just drives into the office and says, “Okay, what do we have.” And each one of his team members have to then report “Okay, since we saw you yesterday, this is what we’ve done. And this is what we know.” And then from that briefing, decide on their next steps, “Okay, we need to do X, Y, or Z.” It’s a really kind of a nice example of planning.

John: Building it into the project just seems to make an awful lot of sense. If you want students to improve their skills, have them apply those skills and structuring it so that they’re doing it is a very reasonable way of doing this.

Judith: And I love the fact that, Rebecca, as you were saying is to ask the students to say what have you been thinking for? And what made you arrive at the decision that you’d like to do something a little different there? So yeah, again, verbalizing the thought processes, is part of the metacognitive abilities.

Rebecca: I think one of the things that surprises students with that assignment is that they don’t realize that they’ve made decisions.

Judith: Oh, interesting.

Rebecca: That happens in projects. But also, if they’re writing a paper, whatever, they’ve made decisions, but they don’t necessarily think of it like they’ve made a decision. They’ve just kind of moved forward. So it causes them to stop and pause and do that first thing that you were talking about, and think for a second before they move forward.

Judith: Yeah.

Rebecca: I do have to say, that since I instituted that, my students are far more articulate when they’re talking about their projects. [LAUGHTER]

Judith: When they’re doing their own projects, they get their passions involved. There was another thing I wrote recently on curiosity, how important curiosity is and how we want to really build that in. In fact, one of the things when I was doing a workshop, I suggested to faculty, I said, “You know, let’s get the students to stop answering questions. Let’s get them posing questions. And let’s give them problems that, in our discipline, we don’t know the answers to, because what fun is that? Ok, so we set up the situation that we the faculty know the answers, and then they have to figure it out. So switch it around a little bit and say, here are some problems and we don’t know the answers, how am I to approach this? And then we’ll give them the challenge.” And that really, particularly John, I’m sure you’ve seen this with the TIP program, the students come up with things that you never would have expected.

John: It’s one of the reasons I enjoy it so much.

Judith: Yeah.

Rebecca: So one of the things that I think can be a challenge is if you’re doing a big project throughout the semester, we don’t want it to end up being just one big high-stakes assignment. Are there methods or ways or strategies that we can manage bigger assignments to have some lower-stakes moments or ways so that there’s not so much pressure, and that they’re allowed to make mistakes and improve and learn?

Judith: Yes, and I’m glad you asked that. Actually, somewhere in a couple of tips online that I wanted to refer to. It’s, I think, tip… is it 38 and 60, that talk about project-based learning online… that’s something to think about. But I also have a chart that recommends the grading process for an online course. And so you assign points to each of the milestones in the project. So you assign points with the project proposal and selection, but you also then have smaller team-based events where it’s worth a little bit. Also, the discussion board is a really important aspect of online learning. But it’s hard. How do you evaluate and grade that? So I recommend that about 15%, even 20% on the discussion board. And it’s pretty much a given like the classroom discussion is… so long as students are reasonably consistent about following the rubric for the discussion boards. So you’ve got a little bit there, you have also shorter essays and shorter leadership opportunities in an online course where they might summarize a week’s discussion. So there’s other kinds of activities within an online course. At no point is everything dependent on that one big project, but it does require the students to invest time and energy in that.

John: Many people often think of a need to develop metacognitive skills most for our less able students, thinking of the Dunning-Kruger effect, and so forth… that those who know the least often overestimate their learning by the most. But I think those issues may apply for students at all levels.

Judith: Yeah, we assume that great kids, actually, automatically have these skills and think that way, and they don’t.

Rebecca: Which I think is also a really great thing to talk about, too, because we often talk about metacognition, and helping students who are struggling, do better. But it’s also a great way to challenge students who are already doing well, to do better.

Judith: Absolutely, a great point.

John: For the last 32 summers, I’ve been teaching in the Talent Identification Program (or TIP) at Duke University. And one of the things I’ve observed is that many of the students there have generally been able to breeze through their regular classes without ever having to really learn how to learn. So they sometimes face a little bit of a challenge when they arrive at TIP, and suddenly they’re faced with a challenge. But a really nice part about this is that it’s an ungraded program, so that they can develop their learning skills without worrying about what sort of grades they’re going to get in their classes. And it’s much better to do it there than it would be at some future point, perhaps in a physical chemistry class, or a differential equations class, or some other class later in the career, when they haven’t really had to develop those metacognitive skills that are going to be useful in their future. Developing metacognitive skills really are important for students all along the spectrum.

Judith: That’s a really good example too, John, it’s an interesting phenomenon to talk about, really.

John: How can we tell whether students have higher levels of metacognition? Or how can we tell when students have not developed their metacognitive skills?

Judith: Well, you know, this is obviously a great question. And so let me share just a little bit about my thoughts on the students. How do we know when the students are totally clueless about their thinking processes? And I think one really red flag, particularly in online courses, that these are the students who rather consistently post comments on the discussion board: “What are we doing now? What was the assignment about? When is this? Oh, there’s a rubric, I didn’t know that.” These comments often are the ones from students that they really haven’t taken the time to really read the assignment and to make plans on that. These are also the students that we know for online students that they’re trying to do too many things. They’ve got 1000 things going on with their families, and work, very likely. And they just simply don’t take the time. So I think… just simple reminders all the way along the way in an online course from the faculty member. In the assignment, to be really clear about the assignment and to say, “Be sure to think about the following.” For example, if an article has been assigned, that you as a faculty member say, “Okay, this is why I’ve selected this article. Here are some of the core concepts that are really important in this” and guide the reading to say, “Why do you think the person did this? Be sure to be able to answer these kinds of questions?” And then also to be clear about maybe what you want to do when you finish this reading assignment, to talk about it. If necessary, explain it to your 12-year old. See if you can explain it to that person. So being really clear in the assignment would be, I think, super helpful for all students in an online course, because again, it’s the kind of thing a faculty member would do in the classroom, you’d say, “Okay, I want you to read this article. And you know, this is one of my favorite articles. This is why I want you to do it” …and all the rest of it.” I think we need to do more of that in the assignments in the online course. That’s one thing.

The other students that are clueless are the ones that keep thinking that all they have to do is passively read and reread and reread. We hear that all the time from students that “Oh, well, I read it 15 times. I don’t know why I don’t understand it.” Without saying, while giving help and modeling, as you’re reading this, think about the following.

Just as an aside, I signed up to take a course on brain, dendrites, and synapses from the Hebrew University of Jerusalem, and oh, my gosh, lesson three, they said, “Well, you know, if you’re not familiar with the electrical circuits, you probably need to go into the Wikipedia and get all this information.” I’m thinking, “Oh, I’m lost right here.” But the idea was that there was some foundational content that I didn’t have. And so again, if in an online course, if some of their content is dependent on some of those other core concepts that they might not have, to remind students that, “hey, this is a difficult reading. And you may want to get some further help in X, Y, or Z.” So again, going back to that whole briefing, and using the assignment as a briefing and giving them clues how to do well in the reading. Some of those comments really have to do with whether students are struggling or not. Because we want to emphasize more clearly the reasons for why we make choices. When we design a course, we really make lots and lots of design decisions and selections as to why I want my students to read X, Y, or Z. And I think sharing that rationale and sharing those reasons, I think, really helps give insight into the metacognitive thinking of the faculty member as well.

Then John, you ask the question, where we talked about little bit about how even when students are doing well, they may not be thinking metacognitively. And I think that it’s important to recognize that these explicit thinking skills about thinking, about visualizing, and about planning, etc, that it would be a good idea to build those visualization and planning into the assignments.

Now, one of the challenges with online courses, and I’ve seen this for years, is the fact that some students don’t have good places to study, they can be living in very busy environments, they don’t have an office, they don’t have a really quiet place to go. And so one of the things that we can do is actually ask the students to post a picture of where they’re going to be doing their studying. It kind of gets them thinking, “Okay, where am I going to be doing my studying? Is it going to be a place that I can really concentrate?” There was one study hint… that I actually just pulled it out, because I gave it my granddaughters…. and it was called “How to study” …and the little hint in there was that when you are sitting down to study, what you do is you find, select or design, some kind of a flamboyant little hat… think Cat in the Hat kind of thing, you know… just flamboyant things, design some kind of a flamboyant hat so that when you put that hat on, you tell your brain and you tell the other people around you that you’re going internal now… you’re really going to think… you’re going to work, you’re going to study. We almost do that nowadays… when I go into my Starbucks, or my local coffee shop… that’s where I do my writing, by the way… I put on my Bose headphones, because again, that signals to the people around me that, “Hey, I’m working, I’m really concentrating.” So if we asked the students to really stop and think, “where am I going to study?” If I’m a family person, and I’m working, “When am I going to study?” That I have to schedule my week, and the times and places that I’m going to do my work and do my studying. And oh, by the way, I kind of have to get my family and my friends on board to say “Yes, John or Rebecca, they’re studying now I can tell they’re studying, you got to leave them alone, give them the time and the space to leave them alone.” For online students I think this is super important for them to build a schedule ahead of time. And again, it recognizes the fact that metacognitively takes time and it takes space in order to do it well.

John: That reminds me of a couple of our previous podcasts, a few episodes back Mathew Ouelett from Cornell, when we were recording, he mentioned that he has a drawing or a painting, I think he did of a tomato, I don’t remember if it was a painting or if he colored it, but he has a big poster with a tomato on his door that he puts up as a signal for faculty to leave him alone because he’s engaged in a pomodoro technique, and he wants to be just focused on this. In a much earlier podcasts, our very second one, Judie LittleJohn was here and we talked about a metacognitive cafe online discussion forum she and I had both used. And one of the things she uses in it, and Rebecca has introduced that in her classes too, is exactly what you suggested, having students describe their study space and perhaps post a picture of it. And then to address all those issues about how well it works, how they deal with distractions, and so on. This ties in nicely to some of our earlier discussions,

Judith: Well great, actually as I was in my Lucky Goat local coffee shop, actually getting ready for this podcast a couple weeks ago, it turns out this young business person came in and he had a briefcase and all the rest of it and I could tell he was kind of settling in… a little too close to me… but that’s okay, there weren’t many seats available. But anyhow, he was talking pretty loudly. And I thought, “Oh, he’s on his phone.” You know, there’s everybody who talks to themselves these days is on the phone, right? And anyhow, we ended up talking to each other partially because he was talking. He said, you know, what he was doing was he was talking to himself about what he was going to do during his time there. And obviously setting his own personal goal. And then he had learned that from his mother. [LAUGHTER] But that does remind me of something else I do want to share. And that is when I do sit down… my own metacognitive practice… when I sit down to do a task… say, I go to the coffee shop, and I got a couple of things I need to do, I write out a mini plan. And list the time I’m starting… list my first subtask, the second sub tasks with approximate time to completion, and everything else. And of course, it doesn’t happen exactly like that. But I do get done, and I do manage to check the pieces off. But then another really important metacognitive practice that I’d like to share is that when I finish that task, I say “What is my next important step?” In fact, David Allen and his book Getting Things Done on Stress Free productivity, thee whole environment, even in business. His primary question is, “What is your next step?” So before you stop where you are, particularly in a writing project, you write down what your next step is. Because it’s totally short circuits, that transition. Because when you sit down, “Yeah, what was I doing now? What do I have to do? What is my next step?” Hey, I can’t tell you the number of hours that that particular little hint has saved me in terms of really making progress and stuff. So that’s another little hint to build into all that project planning. “What is my next step? What do I have to have next?” And sometimes for online learners, particularly, it’s not sitting down and studying, they have to get a resource, or they have to get a book or they have to make an appointment or they have to do something. But then that’s something that doesn’t have to necessarily be done exactly in order. To kind of almost wanting to build a little calligraphy thing, saying “What is my next step?” is a really, really great little hint.

Rebecca: I think experience certainly teaches us that, but it’s something that students who have less experience don’t explicitly know to do. Because we all know that we’ve tried to cut corners and hurry through something. And then if we don’t do that… I know I’ve spent hours figuring out now what was I doing? Like it’s been a while since I’ve worked on this project. Now, where was that? What was my filing system?

Judith: Yes. And what was I thinking? And oh, dear, I wish I bought that along with this other thing with me. Yeah, exactly.

Rebecca: I also wanted to circle back a little bit to the workspaces too. One thing that I found with asking students to talk about their workspace is also to show them professional workspaces. And to talk about the different kinds of environments and how space can help facilitate certain kinds of activities, but also can short circuit certain kinds of activities as well.

Judith: I think that’s really, really helpful. I know, I can’t tell you the number of conversations I had with people when I was in a work environment, and management was making all kinds of decisions that were not conducive to good collaborative work or good independent work. I mean, it’s a real discipline to think about.

John: And you need both spaces, but you will often only have something that’s better for one or the other.

Judith: Exactly, exactly. For any kind of a course, as we’re talking about metacognition, we get overly focused on “what” the students are learning as opposed to the “how.” So just in a capsule comment, that as we are designing online our classroom based courses that we really include in our assignments.. in the design and the various activities… that we include both the “what” of the content and the “how…” How do I get there? And I think we started getting there when they start setting those personal goals. Because some of the courses now are starting to include a goal or a learning outcome that they want their students to think like a scientist. You’re not just learning biology, we want you to think like a scientist. Or we want you to think like an engineer, or an entrepreneur, or whatever. But getting this… it’s a different mindset. You just don’t want to learn what biology is and learn about the content of biology, but I love finding out about the biographies of like Einstein and a few other folks. It’s just fascinating to get into their heads as to the process that they use to do that. Eric Kandle, by the way, is another Nobel Prize winner. He’s got some fabulous book out about his thinking and his processes by which he actually investigated and learned about memory. So… fun.

Rebecca: I think that using biographies is a great way to introduce students to that a little bit.

Judith: Yes.

John: A number of software packages, such as Lumen Learning’s Waymaker package, Norton’s Inquiszitive, and some of McGraw-Hill’s and CENGAGE’s products include attempts at building student metacognition in their products. For example, they’ll ask students questions and they’ll also ask them about their confidence in their responses. The Norton Inquizitive package, in particular, sets it up in a somewhat game-like situation, where they get to bet points on how confident they are. And then it gives them feedback on how they did versus how they perceived they were doing. Do you think this type of approach might be useful?

Judith: I was just reading a research study about that, John, and the students… Who was that by? I would have to go back and find it, I’ll have to email it to you. But the results showed that the students who were less confident…

John: …did better.

Judith: Yeah. Who was that?

John: I don’t remember. I think it was on the POD list, but maybe not, it might have been on an economics list. Consistent with the Dunning-Kruger effect, the students who did relatively poorly, had relatively high self perceptions of how well they did that were not reflected in their test scores.

Judith: …So important to be asking the question: “What do I know?” and “What don’t I know?” Because that’s a core, isn’t it? If we can answer that question. In fact, one hint that’s in that book called Make It Stick… which is a real good one… one of the key things I took away from that was the technique or practice of having in a textbook, taking the heading and turning it into a question and then seeing if you can answer that when you finish reading something. So again, “What do I know when I start? What do I know when I finish? And am I able to answer that question? Or am I able to pose another good question based on that?” So explaining to ourselves what I know and what I don’t know, I wanted to go back to that study that we were just both talking about, because my question that occurred to me as I was reading this is that, you know, the problem, I think, is that the study really didn’t really give the students an opportunity to verbalize why they felt confident or less confident, which is, I think, a whole missing piece of that. And I don’t know how they obviously would make designing a study… actually, my dissertation, in fact, I will bore you with the title of it is “Fluent Readers’ Strategies for Assigning Meaning to Unknown Words in Context.” And the thing is that, half the time, they didn’t know that was an unknown word, they just assumed they knew the word. So again, if we don’t know we don’t know something, we can be very confident. [LAUGHTER] But anyway, in order to get the answers to that question of the study, I actually had the students talk out loud to me, they verbalized and I would go back and ask them about the word and say, “What were you thinking when you came to that?” …and all the rest of it. So they had to verbalize their thinking. For me, it was a good study, it really worked. So again, going back to the value of verbalizing,”What do I know?” and “What don’t I know?” and “What do I think I know?” And Rebecca, we got that when you said, “Well, you started talking about what to do, and then you realized you didn’t know anything.”

John: As you suggested before, a really good thing with any project is to think about where you’re going next. And we always end our podcast with the question, “What are you doing next?”

Judith: What a great question. Okay, what am I working on next? I did mention that I just finished a little video for the Distance Learning Conference in Madison in early August. On my back burner, and I’ve been saying this for a couple of years, I would really like to write a book on concepts. What I’d like to do is… I mentioned I love working with faculty, and when we design courses it so often on a topic-to-topic basis. And yet, as I started working with faculty towards the last 10 years or so, I started asking faculty to tell me what their core concepts of a course is. And, you know, faculty, that’ve been teaching for 10,15, 20, even 30 years, they would pause and they’d say, “Uhhhh, [LAUGHTER] you know, we just teach topics,…” as opposed to core concepts. And we think about core concepts are what stay with us when we finish a course, hopefully. So the question really is, what do you expect your students to know, five years, 10 years down the road from what you have been spending all this time and energy on? So I’d like to write a book about thinking about concepts and how to design a course, and focus on problems and concepts rather than topics.

Rebecca: I like that, I think that would be really helpful. Can’t wait to read it.

Judith: [LAUGHTER] I can’t either.

Rebecca: You gotta visualize, you gotta visualize. [LAUGHTER]

Judith: Thank you. You can give it right back to me.

John: We talk a lot about backwards design. But a lot of the classes that many of us teach were not designed in that way. They did not start with those major course learning objectives, and then work backwards to get to that point. And they’re just series of topics taught in the same way that they were taught to them when they were students. And they were taught in the same way as their previous generation taught them. And there’s not always a lot of thought going into that. And that sounds like a really good project.

Judith: Well, thank you, I may call you and see. I’ll need faculty to work with on that project. So I may contact you for that.

Rebecca: I’ll sign up.

John: Be happy to.

Judith: I’d be willing, God be willing that I get going on that. [LAUGHTER]

John: I think we all have a few projects like that. But eventually they often happen.

Rebecca: Eventually.

Judith: Yes, well, actually talking about it is a good thing. Because the more things we write down, and the more things we actually talk about are more likely to happen.

Rebecca: And I have just told the world so it’s gonna have to happen, right?

Judith: Oh, dear. [LAUGHTER]

John: Although if you change your mind, we can edit it out.

Judith: Scratch that…

John: I know I come up with some things after a podcast where I say I want to do this in my class next semester. And once it’s in a recording now, I pretty much have to do it. [LAUGHTER]

Judith: Well, John, before we break up here, when you do go to Duke, what do you teach?

John: I teach economics, introductory micro and macro economics.

Judith: Okay. Sounds great.

John: It’s been a lot of fun. I love doing it. The kids are just so amazing.

Judith: Well, kids are amazing at that age. They really are. It’s wonderful to see them evolving to young men and women. You know, I’ve got eight grandchildren. My oldest is now 19 and a half. In fact, she did microeconomics online, both one and two this summer.

John: Thank you for joining us.

Judith: Well, thank you very much. I really enjoyed being here.

Rebecca: This is a lot of fun. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

95. Specifications Grading

Faculty often find that grading student work is a stressful and time-consuming activity. Students sometimes see grades as a subject of negotiation rather than as an assessment of their learning. In this episode, Dr. Linda Nilson joins us to explore how specifications grading can save faculty time while motivating students to achieve the course learning outcomes.

Linda is the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. She is the author of Specifications Grading: Restoring rigor, motivating students, and saving faculty time as well as many other superb books, book chapters, and articles on teaching and learning.

Show Notes

  • Linda Nilson – Director Emeritus of the Office of Teaching Effectiveness and Innovation (OTEI) at Clemson University
  • Nilson, L. (2015). Specifications grading: Restoring rigor, motivating students, and saving faculty time. Stylus Publishing, LLC.
  • Nilson, L. “80. Self-Regulated Learning.” Tea for Teaching podcast. May 8, 2019.

Transcript

John: Faculty often find that grading student work is a stressful and time-consuming activity. Students sometimes see grades as a subject of negotiation rather than as an assessment of their learning. In this episode, we explore how the use of specifications grading can save faculty time while motivating students to achieve the course learning outcomes.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Today our guest is Dr. Linda Nilson. She is the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. She is the author of many superb books, book chapters, and articles on teaching and learning.

Rebecca: Welcome, Linda.

John: Welcome.

Linda: Thank you very much. It’s great to be here.

Rebecca: I think it’s more of a welcome back, right?

John: Yes. Welcome back.

Linda: Welcome back. Yes, yes, yes. Good to be here again.

Rebecca: Are you drinking any tea today?

Linda: As a matter of fact, I am. Yes, yes, yes. Of course I had coffee this morning as well. But I am drinking tea. I am drinking a berry tea, but it also has black tea in it. So it’s still a bit of a stimulant anyway, but it tastes really good.

Rebecca: Sounds good.

John: I have ginger peach green tea.

Rebecca: And I have English breakfast.

John: In an earlier podcast we talked with you about your work with self-regulated learning and one of the topics that came up with that was specifications grading.

Linda: Yes.

John: So we’d like you to tell us a little bit about your book, Specifications Grading: Restoring rigor, motivating students, and saving faculty time… especially that last part. [LAUGHTER]

Linda: Yes, saving faculty time. Yeah, I figured that would help sell the book. But it’s true. It does save faculty time. And one of the things that inspired this book was just my hearing so many complaints from so many faculty over the years, about grading and the aftermath of grading and returning grading material to students. And the constant steady stream of students trying to get another half a point, just arguing… just conflict… constant conflict… students being stressed… faculty being stressed… faculty getting larger and larger classes with less and less help… fewer and fewer TAs, if there ever any TAs. And I got tired of it. But there was a part of this that I did not invent. I heard it from a faculty member in the School of Management. And she was doing that pass-fail grading. It was saving her tons of time. She had huge classes, online classes. And she just invented this and she was also sick and tired… getting complaints from students and students not paying attention to her feedback, which of course, took her hours and hours and hours to write and return. So I took some ideas from her. But I also wanted to tie grades somehow to outcomes. And this is where another aspect of specs grading comes in. And that is with respect to bundling assignments, or turning them into modules or whatever. But I prefer the term bundles because it’s much more universal. So anyway, this was a solution to a problem. And that’s what a lot of my work has had to do with, making the faculty members job easier and more rewarding.

Rebecca: I think that you’ve been spying on me for the last many years. [LAUGHTER]

Linda: Yes I have. Yes I have. [LAUGHTER] I’ll admit it.

Rebecca: I knew it. I knew it. So I think John and I are both really interested in the idea of saving time, as are many faculty. But you also talked about, in your book, the history of grading… how its evolved… the 4.0 system. How is this different? And how does this relate to the history of grading?

Linda: Well, let’s look at the history of grading first. Grade started, well everywhere, in 1783. It was Yale’s idea. And what they started doing in that year was an achievement-based student classification system. They were not using As and Bs, what they were using were Latin designations of like optimi outstanding and pejores for failing, like as in pejorative, right? Anyway, then in 1800, Yale dropped the Latin designations and started using numbers 0 to 4. Sound familiar? But that was Yale doing that now. In 1850, the University of Michigan initiated grades, but for them, it was strictly pass/fail, and it only took 50% to pass. So we talk about grade inflation now… look backwards. [LAUGHTER] Mount Holyoke, though, just a few years later set passing at 75%. Harvard, also a little bit later, invented the A to F system, but passing was only 26%. So if you were wealthy enough, you had to know less, right? Okay. But anyway, that’s where ultimately grades came from before 1783. This was in Europe. And this started hundreds of years before our notion of grades. There was something… it was like a Jeopardy game, where graduates… or graduates to be… the hopeful graduates… were answering questions in a tournament style, but the stakes were really high. And so yes, if you are winning throughout, you were really showered with honors. But if you were at the bottom, you lived years in shame. It was terrible. We talk about high stakes. Oh my. So anyway, grades were invented after universities. Socrates didn’t talk about grades, right? It’s a relatively new invention for sorting students.

Rebecca: How does the specification grading relate to this letter system or this 4.0 system?

Linda: Well, it kind of takes a break away. Because first of all, all this grades that I was talking about was with respect to courses, there have been pass/fail courses. Sometimes they worked well, sometimes they didn’t work so well. They work pretty well in medical school ‘cause we’re dealing with a highly motivated students who really understand the need to know. But other than that, most students would do the absolute minimum to get their C-… whatever, it didn’t work very well in terms of like motivating students to learn, they learned the absolute minimum. With specs grading, the pass/fail is within the course, the assignments and tests in the course. But, you don’t pass with a C or C-. You set the passing level at what you would regard as a B level. And this is what restores the rigor. I think we have been sold a poor bill of goods, when we don’t set our students to high standards. We say, “Well, maybe they can’t do it, and then they’ll get…” They can do it. Come on. They’re just not doing it because of our partial credit system and our point system ‘cause they can always get another half a point, right? …just by wearing you down. [LAUGHTER]

Linda: Really, they do and they know it works, it worked in high school, so it works now. Now it may have worked the previous semester, these students aren’t stupid. And they can do what we ask them to do for “B” work… certainly at the undergraduate level, or actually at the graduate level as well. They could do it. But they choose not to, because they can survive otherwise. They could do well enough to pass or get their “B” or even get their “A.” So why should they sweat it, they can always get the partial credit. And students get partial credit. It’s almost like going back to the University of Michigan 50% passing for assignments and tests. I mean, we pretty much lowered the bar, because we want so many people to get over it. Well, people are getting over it without preparing… without doing a decent job. And they’ll spend 10 minutes… 20 minutes… the night before on an assignment knowing that no matter what they do, they will get partial credit and they will pass. So again, they’re not stupid, it’s just that, for them, college is a game that doesn’t have much to do with learning. That’s not where the focus is, and at least with specs grading, there is a lot more focus on learning, because you are tying the assignments and tests to student learning outcomes. And that’s a really nice part of the system in that those grades, the ABC, mean something in terms of outcomes achievement, and these, in turn, might be tied to program outcomes as well. So all of a sudden, you’ve gotten rid of an entire step at the departmental level of having to measure program outcomes. And why do you have to do this? Because accrediting agencies know that our grades don’t mean much of anything, as far as learning is concerned, as far as what they’re interested in, which is outcomes achievement. And outcomes achievement, it’s an up-or-out sort of thing. You can either do something or you can’t, and as long as there are standards set for that, and certainly the accrediting agencies want standards set for that. So let’s say “Okay, so your goal is a student can write a good quality, maybe not great quality, but a good quality business proposal.” Okay, fine. So if that is your outcome, what really does that entail? What exactly are you looking for in terms of a very good business plan? And that should be incorporated in your directions to students. But we don’t articulate that, do we? We really don’t. We speak very vaguely, “Well a business plan should have this and here’s your rubric.” Well, first of all, students often don’t understand the language, but is simply not detailed enough such that all students can understand the directions and actually follow them. We need to put more detail. Usually what we assigned undergraduates is some sort of a template or a formula to do this. And we’re not talking about that. We’re not sharing this template or formula with our students. And that’s what we need to do… not to say that you can’t allow for creative work. Matter of fact, with this system, you can and you don’t have to worry about tearing your hair out in different ways. “Well, I don’t know how to grade a movie.” Well, no, but you can talk about certain qualities of that movie, as in just simple things like the length, or perhaps the number of scenes that you want to see. And certainly they’ll be a learning goal and a communication goal connected to it. And that’s pretty much all you have to do. For most of our assignments, they are formulaic, but students don’t know the formula. And we need to tell them, and those are what you would call our specs for an assignment. So that’s really what we need to do. So all our work is up front and laying out what those specs are. It’s like a one-level rubric. But we don’t have to worry about different levels of the rubric, where there are four-level rubrics and five-level rubrics. We don’t have to worry about that. All we need is one. So we can afford the time to actually specify what we’re looking for… what that template involves.

John: And then the focus is helping students reach the standard, rather than negotiating with them over the grades for what they’ve received.

Linda: Exactly, exactly. Yeah, forget negotiation. There’s some things that ought to be non-negotiable. I mean, we’re not sloppy when we grade, I know, we get sick and tired of it. But we are so diligent in our grading. And then we have a line of students outside our office saying, “I deserve another two points on this because Susie said the same thing. She got 12 points, I only got 10.” …like write an essay about it… justifying it.

John: One thing I really like about it, though, is that right now we have these two levels of assessing students’ learning, we have these complex assessment plans for each department. And then we have a grading system, which often bears little resemblance to the assessment. And this is making the assessment transparent and obvious to students. And it’s forcing both faculty and students to focus on the learning objectives for the course.

Linda: Yes, as we should be, you think about it, what does it mean, when a student gets an A? Does that mean that that student has achieved all the objectives? all the outcomes at the level we want to see? Well, maybe for some of the students who get an A. What does a “B” mean? Now we’re starting to get into really ambiguous territory. Does that mean that the students sort of achieved or barely achieved all the outcomes or maybe achieved some really well and then didn’t achieve others? And a “C,” forget it. You can’t tell what’s going on there, especially given the way we give out “Cs” these days. So maybe the “C” student achieved one outcome well, but which one? Yeah, I mean, no wonder accrediting agencies pay no attention to our grades. They don’t even want to know them. And I can understand why, if they’re focused on learning outcomes. And so yeah, we’ve got this whole extra level of work that we have to do and the department chairs have to do and the Provosts have to do. I mean, who needs it?

Rebecca: Can you talk a little bit about the relationship between specs grading and contract grading, because it seems like they have some things in common?

Linda: Yes, they do have something in common. With contract grading, students individually work it out with a faculty member as to exactly what they’re going to do. And if they do all those things that they will get there “A” because I’ve never heard of a student contracting or a ”B,” it just doesn’t happen. Now, contract rating goes back to the 60s, and the first half of the 70s. And what was happening was since there was this individual relationship, first of all, faculty didn’t specify enough about what they wanted… what constituted “A” work… same problem we have had up to now… and faculty weren’t all that much different then, in fact, they probably gave less guidance. But in any case, there were the specifications. And then faculty and students would develop this relationship, this individual relationship. And it became particularly difficult for the faculty member to give anything but an “A” to the student. And the student would have thought, “Well, why didn’t you tell me to do this to get my ‘A,’ because that’s what I contracted for.” So contract rating was: 1. sloppy, but 2. highly individualized. With our class sizes, we can’t even talk about contract grading. And because it was something that was mutually established, between the faculty and the students, and specs grading is nothing like that. This is all in the faculty members’ hands. Well, at least, setting out what is required to get the “As” and “Bs” and “Cs” and “Ds” . But students choose what grade they’re going to go for. And according to what grade they believe that they need, but also according to what kind of a workload they want to shoulder for this particular course. And you know, maybe all they need is a “C.” And you know what, that’s okay. And we don’t look down on that student if the student says: “Well, all I need is a ‘C.’ We do otherwise. We say “Wow, this is a lazy student.” That’s what we’re thinking in the back of our heads. We have negative thoughts about students who settle for low grades. But with specs grading, you don’t have to feel that way. This student chose a seat for whatever reason, I don’t care. My course is not in that student’s major. So I’m not going to take it personally. So, that’s nice. But the fact that that’s making that choice, gets rid of all kinds of grading complaints, or that thing at the end of the semester: “What can I do to bring up my grade?” Well, it’s very clear what you could have been doing. If you went for a “C” now you could have done these additional assignments, or taking this additional test to get a “B.” But you didn’t do that. Now, you’ve got a week left, maybe you want to try to do that. But don’t ask me, I laid out the contract, I laid out the terms, I laid out the specs at the beginning of the semester, and you make your choices. This whole thing makes students feel a lot more responsible for their grades. They made the choice. And we respect that.

Rebecca: So I think a question that would come up for many faculty is you’ve laid out that certain assignments need to be accomplished to get even a “C…”

Linda: Yes.

Rebecca: …and what the specifications are. What happens when a student struggles to meet those specs?

Linda: Oh gee. Well, now when you say struggle, you can struggle before the assignment, or you can be disappointed and angry after the assignment. So if you’re talking about struggling before and say, “Professor, I don’t understand these two specs. I don’t even know if I can do it. Could you give me some more guidance?” No problem, and then because the stakes are higher, you get credit or no credit whatsoever… doesn’t count towards your bundle… doesn’t count towards whatever grade you’re doing. So you’d better figure this out. Now, afterwards, don’t meet the specs. Okay. And obviously, you didn’t come to see me in advance, did you or otherwise, we would have hashed this out. Anyway… so you didn’t pass? So you missed these two specs. All right, well gee, that’s most unfortunate. But there’s this system called tokens… or I call them tokens, but you can call them… I don’t know, you can call them pigs in a blanket for all I care, it doesn’t matter. They’re like get out of jail free cards. I’ve seen them called hail Mary cards. In a geography course, they were called globes, but anyway, they are opportunities to either redo an assignment that didn’t meet the specs, or to get a 24-hour extension. And there might be some other things that you make up along the way if absence counts against you in your course, you can get out of absence. But let’s say it’s for redoing this assignment, which is really important early in the semester, because students are not going to believe that you’re going to grade them pass/fail, because this is all new for them. And they go, “Well, how can this be…” and “There must be some sort of partial credit or something…” They won’t believe you. So there will be a number of students who will need this token or get out of jail free card to redo the assignment. It’s up to you, as a faculty member, as to how many get out of jail free cards or tokens that they might have. Three is a nice number. I like it, but some people give five. Some people give two. But I think it behooves you to have some reasonable number of tokens with specs grading, because that takes the pressure off of students; it allows them to screw up at least a few times. So this is something that students can get second chances out of this because the stakes are higher than typical. But the thing is, when you’re grading these assignments, all you’ve got to do is check the specs that weren’t met, and say, “This is why your assignment failed.” Now, if you want to give additional feedback, by all means, I’m not going to stop you. But I want you to have a life. That’s the only thing. And you can give positive feedback as well. And then you know what, you’ve passed something, then you get positive feedback from your instructor that you figure, “Wow, the instructor gave me this positive feedback out of the kindness of her heart, because she really cares about me and my success… told me how I could maybe do better in the future.” First of all, I’m going to read this, because this is meaningful feedback. This is not justification for having taken off points, because that’s what most of our feedback is about: why you didn’t get full credit. And now this is actual substantive learning feedback. So, it decouples feedback from evaluation in a way, but students get the chance to do it over and now that you have their attention, that let’s say they didn’t pass a certain assignment, that they can come to you and say, “Okay, how should I do this better?” Oh, okay. Isn’t this lovely? So the worst that will happen is you’ll have a conversation with a student about how to improve his or her work, that’s the worst that will happen. Isn’t that wonderful? [LAUGHTER] And sometimes students know, they just got lazy.

Rebecca: How would you suggest structuring specification grading for a class that’s more project based or even collaborative work?

Linda: Oh, sure. Whether it’s projects, or papers or whatever, again, it’s a matter of specs. Oftentimes, with our projects, that we might allow our projects different media, what we have to do is lay out specs for each media. Or say, if you can think of a different media, you can do a film, you can do a play, you can write pamphlets, you can write a paper if you want to, or you can do any of, let’s say, a half a dozen things. And let’s say if you want to do something else that’s not on the list, come talk to me, and we’ll work it out, we’ll work out some specs for it, too. So what you have to do is you have to lay out specs, but oftentimes the specs that you have to lay out, and students love creative assignments, is you have to lay out, let’s say, length, and so you might say, “Okay, if you’re going to go with a film or video, I want 20-mintues of that. Going to go with pamphlets? I want at least four pamphlets, I want each pamphlet to have at least 250-words in it. If you’re going to go with writing a play and performing a play for your peers, I mean, wow. But I do want it to be a half an hour long. And I want it to involve everybody in the group, everybody’s acting. And if you’re going to write a paper, I want it to be at least, let’s say, 2000-words long… or maybe that’s too long, whatever… and you lay it out, and I want it to follow this kind of an outline.” You can lay out specs for different kinds of media, and let students run with the project. Now a design project, what you might allow students to choose is exactly, let’s say, in architecture, what kind of a building they’re going to design. Let’s say, “Okay, you can design a residence, single family, a separate dwelling, but I want it to be at least four bedrooms, I want to be two stories, this and that. And you could do an office building too. But I want this to be at least two stories, and I want it to be so many thousands of square feet.” And then you let them run with it. And you know what, we know from our history, and from a lot of publications, that students run with this sort of thing. So I’ll give a little guidance and off they go, it’s a beautiful thing. Now in terms of projects having to do with group work, I don’t know that that specs grading will make group work smoother. [LAUGHTER] And I’ve talking about out-of-class group work. If you want to play marriage counselor, you just go right on ahead. But you don’t have to. I mean, there are some people who say, “Hey, you guys work it out on your own, I don’t want to hear about it and you can fire freeloaders. And if you are doing all the work, you can resign from your group and any other group will be happy to pick you up. But all this has to be done by the end of the fifth week,” whatever. So I don’t know that we’re going to get rid of those problems. [LAUGHTER]

REBBECCA: I think my question was more about the high-stakes nature or the long-term nature of projects.

Linda: Yes.

Rebecca: So would you recommend doing scaffolded specs or something like that to help make sure they’re on track?

Linda: Well, yes, what you need to do, as with any major project, you need to divide it up into smaller tasks that students can be at accountable for along the way, and maybe even get feedback for. Now, you can give feedback… your peers… people in other groups… can give feedback. Because after all, we’re doing criteria referenced grading, we’re not grading on the curve. So there are different ways for students to get feedback. But in any case, this divides a massive task into little pieces. So maybe starting out with, “Okay, by the end of the third week, we want your group to have a literature review put together.” And we put specs for the literature review. “And we want so many references. And we want so many of them in the last five-years,” And what have you. So we can even put the specs, I even specify specs for the format that we want: APA format, or ASA format, or Chicago, whatever it is that we want. That’s something that we can put in the specs if we want to, and that way students will actually proofread. Now I think we ought to be somewhat tolerant with respect to format anyway, we ought to allow three little errors in format. Because you know what, when we submit a paper to a journal, they’re tolerant of us, right? [LAUGHTER] I mean, you can’t be sloppy, but if we put a period instead of a comma, hey, we are forgiven. And we’ll catch it later on the proofs. So we need to allow some errors on something like that. But anyway, yeah, we divide up a big task into pieces. And then students get feedback on the pieces. And that’s also a good way to find out early about who are your freeloaders, who’s not doing the work early, so they can get rid of the freeloaders if they need to. Here’s another thing that you can do with specs grading, when you do bundles, you know, the project, the big project, you can make that big project required for an A, period. And then you will only have groups with students who are going for As. So it depends on the size of your class, of course. You might wind up with only, let’s say, three groups who are going for As. Or you know, the number of students that will fall into three groups. Well, guess what, you’re going to get some really fine projects, you’re not going to have to worry about it, and they might not be freeloaders. But anyway, in the examples in my book that a lot of faculty members and I recommended as well, that group projects be reserved for getting an A, for the A bundle. And other assignments be in the B bundle and C and the D bundle for that matter, which might be very minimal. But hey, if you want to a D, far be it for me to stop you. So, anyway, that’s another way to solve the problem, you just put those projects in just for the students are going for an A.

John: You’ve mentioned bundling a few times in terms of grades, could you talk a little bit about that and how you can go from the specs grading to the course grades that are assigned in the course.

Linda: Okay, there are two ways and the first way I’m going to just mention and not recommend. But you can keep your point system, and you might have some haggling over grades. But in any case, you can keep the point system and say, “Okay, my course is 100 points and if you get 91 points or more, you’ll get an A.” And okay, fine, but the bundling system allows students to choose their grade. So what you do is, you set up clusters of assignments and tests that students have to pass at that B level to get credit for their bundle. So you set those up. And the easiest way to do this, so you got a D bundle, is pretty minimal. And you can tell students, “Look for your D, here are the only outcomes that you will be able to achieve if you go for the D.” Some students say, “Hey, I don’t care, not in my major.” Fine. So it’s minimal work, maybe it involved just passing tests at let’s say, a 70% level, and maybe doing a couple of little written assignments along the way. But, they all have to meet whatever the specs might be. Whatever specs you set out for those assignments… And C bundle, “Okay, you got to do everything that you do in that D bundle, plus there are extra assignments as well.” And so maybe what you are having students do, and I’m just making this up as an example, is the students have to turn in 15 out of the 18 reading assignments that you have, they have to turn in a typed up outline on a particular reading. This way, at least your C students are going to do the readings, right? Because at least most of the readings, almost all the readings, and you’d give your teeth for almost all your students to do almost all the readings, right? But that’s all they’ve got to do. Maybe for a B you have to do all the requirements for a C, maybe even more of those reading notes, or something different. Perhaps keep learning journal, keep a journal on how you are learning, what you’re having trouble learning, the different strategies you are trying to learn the material, and it’s going to be collected. But you really have to specify what questions you want students to answer. So that will be collected maybe four times during the semester, and you might collect some every week or so and look at a few of them. But again, you’re just looking for the answers to the questions. That’s what you have to do for a B. You’re going to get a good handle on those readings and you’re going to find out how to best learn this material. For the A, you got to do everything for the B, but you have to do some sort of a group project where you’re going to learn, let’s say application, analysis, evaluation, what have you, whatever higher thinking levels that you have in there. At the C level, that’s pretty low. But good lord, at least they’re understanding the readings, at least they’re reading the readings and presumably understanding it. And you know, you can spot check some of them to see that they are understanding and if they’re not, you can help them with that. And as soon as they’re learning how to learn for that B, but students are learning how to analyze the material, even higher-level learning outcomes for that A. Now if your student learning outcomes are dictated by a professional accrediting agency, then you’ve got to put all of those outcomes in what’s required for a C, or maybe your institution considers D passing, some institutions don’t. Whatever you consider passing, that’s where all of those outcomes have to go. B students and the A students, they will achieve even more outcomes. If you’ve got an outside accrediting agency telling you what your students have to be able to do, then guess what? All the students that pass the course have to be able to do these things. And you’ve got to set out the specs accordingly. There are some accrediting agencies that layout good outcomes with active verbs. And other accrediting agencies, they give you crummy outcomes where you’ve got to essentially rewrite them so you can assess them. But I’m not going to get into that. [LAUGHTER] Did I answer your question, I hope?

Rebecca: Yeah, I think a lot of your examples focus on a more traditional class and not necessarily a lab or a studio kind of class where there’s a lot of class work that happens inside of class. What might a bundle look like in a situation like that? Where a lot of the learning activities are happening in person rather than for homework. So, for example, I teach studio classes that meet for six hours a week, and they presumably do about three hours outside of class. Versus the reverse, which many people have and labs are similar.

Linda: What area are you in?

Rebecca: I teach graphic design.

Linda: Okay.

Rebecca: But it would be similar to something that would happen in the sciences too and having to do lab.

Linda: Yeah, to an extent, it depends on whether it’s a decent lab or not. [LAUGHTER] Some labs are a waste of everybody’s time. But anyway, what you have to do there is, you’ve got projects, right? …throughout the semester, different things that students are doing. but how many projects do you have in a semester?

Rebecca: Usually, it’s like four big projects, or three.

Linda: Okay, well, now this sounds very, very radical. But what if for a D, you only had to do one project? …and I mean, at that B-level that you set to pass. What if for a C, as a student, I only have to do two. Now, for the first two, because I presume this is somewhat cumulative. For a B, I only have to do three projects. But an A, of course, I have to do all four, if there are four projects. Now, what does this mean? Oh, by the way, do you have tests?

Rebecca: No.

Linda: Just curious. Okay, it means that assuming your D student does passable work, that first project, you can say bye bye after that first project is done. If the person wants to sit in… you can’t, you know… it’s a free country, right? Well, not really. But they can sit in if you want them to, that’s fine. For a C student again, you can say bye bye after the second project, assuming they do it at that passable level. And by the end of the semester, you have only you the most motivated and committed students that you have to be concerned with. And you can give them some very challenging work. Now, I don’t know how you feel about that. But that’s the way you could do that. Now, you could also bundle it differently. You can expand the number of projects that you have as well and simply make them shorter if that’s what you would be required. But I don’t know how you react to that.

Rebecca:I think my accrediting agency would react to that in that they have to put a certain number of hours in. [LAUGHTER]

Linda: Okay, okay, so your accrediting agency is time focused.

Rebecca: It’s part of it. Because presumably with more practice, you get better.

Linda: Well, presumably. [LAIUGHTER] …practice with feedback, you get better. But yeah, if we are committed to this hours business, that makes things very unruly, everybody’s got to put in the same amount of time…

Rebecca: In class anyway.

Linda: Oh, in class, okay. That’s very unruly, with respects to specs grading because specs grading isn’t about time. I mean, you can recommend that such and such assignment should take you at least three hours for it to be passable. But other than that, that’s your recommendation for a certain given assignment. But yeah, that makes it very unruly when your accrediting agency says, “Okay, they’ve got to spend so many hours in class and not just looking at their phone.”

Rebecca: Yeah, I mean, we do critiques and things like that. We provide a lot of feedback. But yeah.

Linda: Yeah. Okay. You do critiques?

Rebecca: Yeah lots of critiques.

Linda: Yeah, like art classes and things like that?

Rebecca: Yeah.

Linda: Okay. Well, that’s nice. But now, you could hold your B and C students to doing critiques on all the assignments, just so they can put in their hours. But they don’t necessarily have to do the assignments, at least they can see models. But again, I don’t know if your accrediting agency would be happy with that either.

John: This may not exactly be specs grading, but could you give, say, different levels of activity in each of the assignments? If you have four-projects, you could have one bundle in that project that would give a C, another that would give a B, and another that would give an A, changing the scope of the projects and it would be specs grading within that scope.

Linda: Yes, that has been done and I’m not sure about what kind of projects you’re doing. But yes, you could do that. And you could set different specs for each level. So D students, lets say, have to do a D-level project, which is not nearly as time consuming and not nearly as high a level cognitively, it doesn’t demand that much, doesn’t demand much time, it doesn’t demand that much thinking. And so you could do that and that way students would put in their time, and as long as you can set out the specs for those levels. Yes, absolutely you could do it that way.

Rebecca: Yeah.

Linda: But yeah, accrediting agencies can really get in the way of flexibility. Rebecca, would your accrediting agency go along with that?

Rebecca: With scope, probably. Yeah.

Linda: Yeah. Okay, good. I want to tell you about a course that does this, it’s in computer science, they have students writing programs, and there are six projects that students have to do, they do these individually. If you are going for a C, you get some pretty easy projects to do, pretty easy problems to solve. If you’re going for a B, well you get sort of intermediate-level problem, you don’t do the C one. Because if can do the B ones, you can do the C ones. Now for an A you get some much more sophisticated ones. But still, for every level, you only get six. So there is that, that as long as you can designate level of difficulty, or for that matter, breadth of knowledge, if you can designate that… of course, usually people can do that… that you can make levels out that you can make different bundles out of that. This is another little interesting take, in this particular course, if you are late handing in your projects, that you’ve got to do two more projects at that level. So in other words, you are penalized for lateness by having to do more work. Now that really hits students where it hurts. And if you’re like super late, you got to do yet two more. So any way that keeps students on their toes. [LAUGHTER] Anyway, so in this particular course students are rarely late. So yes, there are ways that you can do that just in terms of level of complexity or level of cognitive operations demanded.

John: How do you explain this to students, if you’re going to introduce a specs grading system?

Linda: First of all, you’ve got to sell specs grading, because they’ll look at you like, “Huh?,” which is fine. And you should explain this, but you can tell them, “You know, I’m going to hold you to higher expectations.” And we know that holding students to higher expectations feeds into student success, student learning, but also student success in general. You could explain to them the concept of andragogy. In other words, that pedagogy really has to do with little kids. Andragogy has to do with adults. Saying, “I’m going to treat you like an adult.” And students like that kind of thing, you might be choosing the kind of project you’re doing. It’s easy to build choices into this. “This will be a safe but a challenging environment because you know exactly what you’re supposed to do. I’m going to give you the descriptions of what you have to do for the assignment to pass in advance and in detail. So you almost can’t screw this up unless you don’t read the directions or pay attention to the directions.” So you want to emphasize the choice and control over their grade that they’re going to have. You want to tell them about tokens, we’ve got some wiggle room, if you will. Now these tokens, by the way, are not physical, you make them physical students will develop a black market, [LAUGHTER] believe me, but virtual and you keep track, or you can ask for an extension, but you don’t ask me for an extension. You’re late, I just take a token away. That’s all. We don’t have to talk about it. I don’t care why you’re late, this is just the way it’s going to be. Just like your boss won’t care why you are late or why you are absent, it counts as like a holiday. Now we’re going to tie what your grades is to what you are learning. If they look at the syllabus at all, everybody’s going to know what you have learned in this course. So that’s how you sell it to students. Now students will need to be reminded about how the system works, because this will strike them. “This is so weird.” Even if they initially like the idea, they still need to be reminded a few times because it is really strange. According to my research, students way prefer this system to traditional grading. And one of the reasons why they do is because we give better direction, we tell them what we want, we give them the formula, we give them the template. or we give them tons and tons of freedom to meet the specs. Especially with students when we talk about length or whatever, length means depth to the students. It doesn’t mean that necessarily to us. Students really like this because they feel way more secure in it. And they do like the element of choices that are built in, that’s motivating for them. And they do learn more. And if they are more motivated, they’re likely to do more of the work at a higher level, they are more likely to excel. And the A students are going to do A work anyway, because they don’t know any better. All they know is A work. And there’s a sense in which they want us to love them. I mean, really, so they’ve got this strong sense of loyalty towards us and they really want us to respect them. And they’re used to feeding off of that they’ve been doing it all their lives. Don’t worry about them getting sloppy, they’ll never get sloppy on you. [LAUGHTER]

John: If someone wanted to transition to specs grading, how should they get started?

Linda: What you might want to do if you’re switching, transitioning to specs grading, you want to look at your assignments, and for that matter, look at your tests, like what you have in your tests. You want to look to see what can be transferred into pass/fail. You also want to look at your tests, if your tests are very objective, and you’re relying on a test bank, remember, it’s so much easier for you to grade with the specs that you can start assigning more written work or design work on tests, what have you. But definitely higher-order thinking types of questions and you will have the time to grade them, because you’re going to lay out the specs and without giving away the farm, you will tell your students the specs of your essay questions in advance, and they will study accordingly. But anyway, you want to look at your outcomes and then you want to identify the cognitive level of your assignments and your test questions which again, you might want to change. You want to be able to group your assignments and tests by that cognitive level so you can develop bundles. It’s kind of a radical way to do it, but it makes your life so much easier once you have those bundles. Because all you’ve got to do is say, “Okay, let’s see. So I had four assignments in this bundle, this student passed them all, hey B. It’s so much easier, you just have to be sure that you set out the deadlines for the different assignments in the bundles. You don’t want everything being submitted to you in the last week or two. So you still have to have your deadlines along the way. Just warning you about that. I want to refer people to my book for examples of courses, and not just my courses either, that have been specs graded. And they cover a range of over a dozen different disciplines. So they can see all kinds of examples.

John: That’s a great resource for those who are considering moving to specs grading.

Linda: Yes.

Rebecca: And it also sounded to me like a couple of cautionary tales that you have about transitioning are about time and making sure that you spread things out…

Linda: Yes.

Rebecca: …but also being very clear in what the specs are, and then following through with whatever you said.

Linda: Yep, yep. And if you didn’t lay out enough detail, well, there’s always next semester. [LAUGHTER] But you can’t change those specs in midstream. “Oh, I wish I thought of this and then start grading students.” No, no, no, no, no. But again, we live and learn, right? We’ll get better the second time.

John: We always end with the question. What are you doing next?

Linda: Oh, what am I doing next? Well, what I’ve been doing is I’ve been doing a lot of traveling, doing keynotes and workshops… oftentimes on self-regulated learning, sometimes on specs grading, and sometimes on any number of different topics having to do with either teaching or academic writing, or something like that. So I’m going to be taking a little bit of a break from that until the end of, or late, July anyway and then things start up again. But that’s kind of nice. I don’t mind that because this keeps me semi-retired and that’s the way I want to be. A semi-retirement is Nirvana, just letting y’all know. [LAUGHTER] So, that’s what’s next for me, that and to finish my tea.

Rebecca: Sounds like the key to semi-retirement is the semi not retired part. [LAUGHTER]

Linda: Yes, it is, it is. Your brain is still working and you still got your thumb in the pie so to speak. And it feels good because you’re not under that same 60 hour a week pressure that you otherwise have with a regular job and then doing these other extra things on the side and it doesn’t work real well.

Rebecca: Thank you so much for joining us.

Linda: Well, thank you for this opportunity. I normally say… autograph books and things like that… happy teaching, but I want to wish everybody happy grading. [LAUGHTER] An odd phrase right? Happy grading. [LAUGHTER] What an odd phrase, right? Happy grading!

John: Thank you.

Rebecca: Thanks, Linda.

Linda: Bye bye.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

93. Reflective Writing

Formative feedback, reflection, and practice are all essential to improve our skills. In this episode, JoNelle Toriseva joins us to discuss her approach of combining faculty and peer feedback with reflective practice to improve student writing skills.  JoNelle is a writer and an Assistant Professor and Director of English Communications and Media Arts at Genesee Community College. JoNelle has won the Patricia Goedicke Prize in Poetry from Cutbank. Her work has appeared in The North American Review, Salt Hill, The Literary Review, The Saranac Review, The Cincinnati Review, Descant, and JACKET, among others, and included in Days I Moved Through Ordinary Sound published by San Francisco’s City Lights, and Best Canadian Poetry in English.

Show Notes

  • JoNelle Toriseva
    • Assistant Professor and Director of English Communications and Media Arts at Genesee Community College.
    • winner of the Patricia Goedicke Prize in Poetry from Cutbank (The Literary Journal of the University of Montana). The judge for that prize was Oliver de la Paz.
    • Selected by Shane McCrae, J.R. Toriseva’s Barbed Water, was the winner of Saudade’s annual poetry contest, and is now available from Argus.
    •  Toriseva’s work has appeared in The North American Review, Salt Hill, The Literary Review, The Saranac Review, The Cincinnati Review, Descant, and JACKET, among others, and included in  Days I Moved Through Ordinary Sound published by San Francisco’s  City Lights, and Best Canadian Poetry in English.
  • Accelerated Learning Program (ALP) at GCC
  • 2. The Metacognitive Cafe Online Discussion Forum – The 11/8/2017 Tea for Teaching podcast with Judith Littlejohn that was referenced in this podcast.
  • National Novel Writing Month

Transcript

John: Formative feedback, reflection, and practice are all essential to improve our skills. In this episode, we examine one professor’s approach of combining faculty and peer feedback with reflective practice to improve student writing skills.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is JoNelle Toriseva, an assistant professor and director of English Communications and Media Arts at Genesee Community College. JoNelle has won the Patricia Goedicke Prize in Poetry from Cutbank. Welcome JoNelle.

John: Welcome.

Jonelle: Thank you. Very happy to be here.

John: Our teas today are…

Jonelle: Bombay breakfast.

Rebecca: That sounds tasty.

Jonelle: Which I get from Jasmine Pearl, my favorite tea company.

Rebecca: Yum. I think this morning I have black currant.

Jonelle: Mmm, nice.

John: And I have ginger tea.

Rebecca: Yum. So we invited you here today to talk a little bit about reflective writing. Can you talk a little bit about some of the kinds of classes that you teach and how you use reflective writing as part of that practice?

Jonelle: Yes, well, I came to academia through being a community artist. So I went to a lot of schools, and a lot of community programs… prisons, different places and taught writing in very different situations and in coming from that background, building community was really important, and also goals. So when people want to write, they want to write for a variety of reasons. And so to help people get what they wanted out of these community writing courses, I would have people tell me their goals. And I had done some reading in how to help people achieve their goals and a big part of that can be having people write down their goals. So my whole reflective writing practice has come out of goal-setting, and having people write their goals and then share their goals with each other and learn how to achieve their goals. Some people like to break it down step-by-step, other people haven’t told another person what they wanted and haven’t experienced that kind of support that comes when you share what you really want to have happen. And a lot of magical results can occur when people talk about their goals, write down their goals, learn that other people will help them achieve their goals. So this reflective writing came out of that I learned that that was working, and that was helping people. And I thought, “Oh, why don’t we do this three times a semester. So people can kind of check in with themselves, and people can adapt what they’re doing if necessary.” Sometimes people come into a class with one set of goals, and they’ll realize, “Oh, no, what I really wanted to do was this, or that.” So I have different points in the semester when they can go in and adapt, and rewrite, and edit their goals.

Rebecca: How important is that piece of community and sharing of the goals as part of this reflective process?

Jonelle: It’s highly important. And I didn’t realize how important it was at first, but a lot of people haven’t had the experience of supportive community before. And so just stating a goal out loud… First of all, the person has to figure out how they’re going to put that into words and they can help them plan a little bit more what that goal might look like if they are actually thinking about it. A lot of people come to writing and they want to write a bestseller. And they may not have thought it through. But once they have to state their goals, they break it down, “Oh, I want to write in this genre.” Or some people will want to write a memoir about their family history and so they’re not really looking to write a bestseller, they’re looking to write something that honors their ancestors. So it helps people understand exactly what they’re trying to get out of their writing practice.

Rebecca: How do you make sure that students feel safe to share those goals and actually are taking that goal sending really seriously?

Jonelle: I do it by setting standards and also doing it right away in the course. So people see what is expected. So they’re still, at the beginning of the course, they’re still meeting each other and getting to know each other and so it’s just part and parcel of what happens in the course, it’s one of the expected criteria: that they respect each other, that they listen to each other, and that they respond in a positive and supportive manner. So I set the tone right away with it. And people respond really well to that because I find that people, they’re coming to school, they’re coming to class for a reason, they do want to get a certain thing out of it, and they’re able to say what it is that they do want to get out of it. And I also set it up ahead of time that they can’t say, “Oh, I want an A in the course.” We talk about breaking down what the writing process is, and so they’re working on specific goals. Like one person may have to work a lot on transitions, or one person may be working on creating an edited draft that can go to a certain publication. But no matter what that goal is, it’ll be broken down into chunks that are manageable and also skill related.

John: Are these online or face-to-face class?

Jonelle: I do it in both. So if we’re in a face-to-face class, I’ll have them do a writing assignment and then they will share with a neighbor close to them. If it’s an online class, they will do a post about what their goals are and then they’ll respond to two other students posts. So they’re doing one-on-one sharing with each other in either mode.

Rebecca: What are some tips that you have for other faculty in helping students prepare to write those goals?

Jonelle: What I usually advise people to do is to have a time set aside for individual reflection. So they’ve been thinking about what they want out of the course, we’ve been talking about what constitutes the course, what kind of skills they’re learning. And then we’ll do five-minute writes on them in their own process on them and their goals, and so they’re used to already doing short writing sprints about these goals. So we do practice and then also I have examples of what other students have written for their goals so they can kind of see the level of detail that goes in, and it’s something that’s very practical and it can be used as a “to do” list. And I also share with them my own writing goals so that they know everybody might have a problem getting started, or ending, or writing a good conclusion. So I’m very honest with my own writing process and I think that helps them also be honest about their writing process.

John: Some of your work is in the accelerated learning program or the ALP program at Genesee Community College. Could you tell us a little bit about that program and those classes?

Jonelle: Yes, Accelerated Learning Program is really exciting. It’s a process of where you just sit side-by-side with learners and help them achieve what they want to achieve. It’s something that I first learned how to do in graduate school at Mills College in Oakland. As part of my graduate school training, I was tasked with teaching a small seminar group of students who were selected from the freshman first-year writing course. So I had a group of six students from this writing course who I would go to class with, I would go to their composition course with, and then I would meet with them as a group and teach in seminar style weekly, thesis statements, organization, researching skills. Whatever the teacher was teaching in the one-on-one course, I would supplement and also do some more extensive teaching as the students needed it. And then I also met with each of the students individually weekly and talk to them about how their work was going, how their classes were going, supporting them if they needed any kind of academic help. And this kind of one-on-one, close involvement really helped these students because they knew there’s somebody that they could talk to, and that there was somebody that could get them help. And a lot of the problems that these students were having were not cognitive problems. They were just not as well read as some of the other students and they didn’t have some of the same training. One of these students also didn’t pick appropriate material to write about. For some reason this person only would talk about and write about Bruce Lee. So no matter what we’re talking about, she would only write about Bruce Lee. So I worked with her a lot and helped her widen our writing scope. So things like this, people that just didn’t have some of the same experiences as the other students in the class.

John: So the classes are for students who are in need of getting them up to the level of the courses, but it’s accelerated because they’re working with the regular classes, but getting additional assistance within that environment so that they don’t get behind in their progress?

Jonelle: Exactly, yes. So what was being done at Mills College then started being done across the nation in different ways, different people started different programs. And the program that we accepted and did for GCC is Accelerated Learning Program. And in that program, it’s a little bit different than the one-on-one tutoring, but it has a very similar philosophy. And in that course, a student will be enrolled in English 101, and then they will have another course called English 100 that gives them supportive training and workshopping directly from that 101 course. So it’s roughly half of the amount of students, there’s 12 students in it and they get individualized workshops on whatever topic they need to have more reinforcement on. And it addresses non-cognitive issues and addresses skill-based issues as well. So a student takes a credit bearing course along with a non-credit bearing course at the very same time. And we found that this improves retention, it improves class performance, it gives students a lot of confidence when they’re in that English 101 class and gives them the start that they need to catch up work that they didn’t have, for some reason, in their high school experience.

Rebecca: Do you find that the role of reflective writing is different with these students? Or do you do more reflective writing with this group of students?

Jonelle: Depending on the group, depending on what they need, but a lot of times writing about their thinking can be incredibly good for them because they’re able to learn how to follow their thought—and that writing is thinking—it gives them bigger writing muscles so that they’re able to write for longer periods of time and write for bigger projects. They learn how to write themselves into a project so that they know how they’re going to address it, how they’re going to shape and form the narrative, what kind of sources they’re going to be talking to. A lot of writing is the writer in conversation with texts that have gone on before or texts that are currently being created. So it’s kind of part and parcel of the writing process in many ways.

John: So you also teach classes on writing for science and business, do you use similar types of reflection in those classes as well?

Jonelle: Yes, I found that a lot of writing is very similar in that you follow many of the same kinds of steps and ask yourself many of the same types of questions. So if you’re writing an academic paper about theater, or an academic paper about a biological situation, or you’re writing something in the field of chemistry, your approach is very similar although your results are going to be very different. But a writer always has to decide how they’re going to frame their writing, how it fits into the landscape of the current writing in the field, how they can work to improve their writing. So the reflection works for any sort of writing that someone will do.

John: What’s the feedback and revision process that you use in your classes?

Jonelle: Many different layers… the first feedback is, of course, given by the writer themselves, the writer talking to themselves, talking to their piece, and this can be done in whatever way is best for the writer, either verbally or on paper. Then the writer will do peer feedback. So they’ll be reading their work to a peer and a peer will be reading a work to them, and they’ll give their views and also ask questions. So it will be helpful in the developmental stage to get more questions about what is actually on the page, what could be on the page, what needs to be on the page, and then going into revision where the student will work with me, or work with tutors, or work with each other on sentence crafting. Are the sentences crafted the way they need to be? Or do we need to have more sentence variety? How is the point-of-view working? Does that need to be shifted? Are there places where the focus is dropped? Has everything been included that needs to be included? Those kinds of questions get answered. So we go from the larger, macro view to the micro view, looking at, you know, is everything in place? Is everything nailed down? And that will be through working with themselves, through working with me, through working with each other, through working with tutors, and then going back to model texts and looking at their work in comparison with other papers. Are they meeting the standards? Are they meeting the scope? Taking a look at how they measure up to other writers out there. So there’s lots of different layers of reflection and feedback that go in. And hopefully that process is continual, and they will keep doing it after the class as well so they can continue to improve as a writer.

Rebecca: Can you talk a little bit more about the relationship between an iterative revision process, and reflection and reflective writing?

Jonelle: The reflection is what will help a writer have an increased amount of success throughout their career. So when a student is writing a paper for 101, that’s one paper, but if they realize the steps and realize how this writing will connect to their future writing, then they’re able to set themselves goals, meet these goals, so they can become a better and better writer with each paper and with each writing experience. Because we’re not just training somebody to write a good research paper for a first-year class, we want them to be able to also work well at the transfer institutions, go on and work well at the masters level, at the doctoral level. Writing is a process that we continually redefine for ourselves and we continually have to get better at. So it’s this lifelong skill and the reflective writing helps people take baby steps in that larger arc of the lifelong writing, hopefully.

Rebecca: You talked a little bit about model writing. Do you use your own writing or your own process and the ways that you struggle through your own process as a model for students as well?

Jonelle: Yes, I definitely do. I tell them what’s worked for me, what hasn’t worked for me. A lot of times my talk about how I’ll do short chunk writing like write for five minutes, write for 10 minutes, write for 15 minutes using the kitchen timer when writing gets hard to do. Also taking breaks, how I motivate myself to write if I’m in a really bad funk, and this makes them laugh and also makes them realize, it’s a very human process. So a lot of being a writer and improving your writing is just learning how to work with yourself and talk with yourself so that you keep on getting better, you keep on going back to the writing process and treating yourself with kindness is important.

Rebecca: Something that seems to be coming up in the way that you’re talking about is also the need to see yourself as a writer. So it seems like the reflection is a way to help students start to identify in that way.

Jonelle: Yes, that’s a great, great insight. A lot of people have to be given permission to write or they have to give themselves permission to write in some way. If I’m teaching a creative writing class, often I’ll have people introduce themselves to each other and say, “I’m a writer,” and then they’ll hear it back. And just that ability to say that they’re a writer and be seen as a writer can be incredibly motivating and supportive for people. So yes, it’s important for people to be able to just be able to state that they are a writer. And so many of us are writers in so many different areas that it’s important for people to inhabit that role for themselves.

John: Sounds like a lot of this is building a growth mindset in students, especially students who perhaps may have struggled earlier with writing to let them know that they can, in fact, become more proficient through working on this.

Jonelle: Yes, yes. And that is something that is going to come from inside of them. We want to build up that writing muscle, that writing ability so that they do well in this class, but that they also do well in their next classes, and that they have resources of people that they can reach out to when needed if they come to a standstill. But yes, it’s creating lifetime learners and lifetime writers who will be there for each other.

Rebecca: I’m particularly interested in the student response, especially from students who are in an English 101 class, who don’t identify themselves as a writer to start off and are not in some sort of major where writing seems obvious.

Jonelle: Yes, writing has become such a big part of our lives. We text all the time, we are writing emails to each other. It’s so important that we can present ourselves well in writing and that we can understand what audience we’re writing for. So we talk about all the different writing situations so that they’re aware of how much they are writing, and I tried to use topics that are interesting to them: food, music, things that we all have in common. And so they’re talking with each other, they’re seeing each other as writers, they’re reading each other’s writing. And they start to realize, “Yes, I’m part of this community too. And I can define my role as a writer in my own unique way.” It’s fun to see them blossom, and also redefine the role of writers… a serious role… someone in an attic with crazy hair, or is it somebody writing a text… They can see themselves as a writer in their soccer playing… concert going. If they’re a math major, if they’re a science major…. Everybody writes, so they can see the role of writing in their fields and in their lives.

Rebecca: I think that sometimes faculty even don’t have that identity, despite the fact that they might do a lot of writing and maybe even a lot of publishing. They don’t always recognize themselves as writers.

Jonelle: Right.

Rebecca: And that makes it difficult to filter down to students if you don’t also see yourself as a writer.

Jonelle: Right, and it’s important for people to give themselves credit for the writing and the thinking they do. And also all that goes into writing an article, they may do things just automatically and not realize all of the steps that have actually gone into what they create and how complex it is. So they can sit back and give themselves a few kudos. And I find many departments will have weekly meetings where they will have people read papers, and that can also highlight what’s happening in their academic field and give a spotlight on the writing and the writing process. And that also helps build community.

Rebecca: How have your colleagues responded? Do other colleagues use similar approaches? Especially maybe some of this goal setting strategy that you laid out at the beginning.

Jonelle: Yes, I find that a lot of people do reflective writing in their own way. One of my colleagues, Marie Iglesias-Cardinale, has students write letters to themselves and to her and that is another way of framing the reflective writing. So you can do it in many different ways and have it work in a way that is more natural to whatever subject you’re doing, or also to your personality too. Some people like the journal writing, some people like the epistle letter writing, some people could frame it as a podcast, some people could frame it as a selfie, a writing selfie. So people can do it in a variety of ways. And a lot of my colleagues here, Judie Littlejohn, does a lot of metacognitive work and having students look at their learning, which is what this is also having students look at their writing. And I think that more and more people are coming to realize that people have to train their brain to learn a subject or to write a certain format that they’re going to be writing and that our brains are so plastic and able to respond to these tasks that we set before them in a way that is only helped by self reflection.

John: Judie was a guest on our second episode actually. We’ll include a link to that in the show notes. But it does sound like much of what you’re doing is helping students become more metacognitively aware of what they’re doing and processing their own learning much more actively.

Jonelle: Yes. And it helps them get tasks done in other places. So if they can remember “Oh, I did these steps for this paper. What if I start doing them for this article I’m writing?” This is a way to get into that so they can fall back on past successes to have future successes. But a lot of times we have to train ourselves to write in a new way for a new writing task, and that can be daunting. But if you’ve had success before, you can talk yourself through the new tasks.

Rebecca: Good advice to remind ourselves during a summer writing sprint perhaps, right? [LAUGHTER]

Jonelle: Yes. Yeah. [LAUGHTER]

Rebecca: One of the things that I’ve recognized over time doing reflective work with my students is, needing to give and dedicate class time to that activity. Because we know that it’s important to student learning, but students can see it as a throwaway kind of an assignment if you don’t actually dedicate time and effort as part of the actual in-class time. Because if it’s just a homework assignment, sometimes it’s not taking quite as seriously.

JONELL: Very true. And also having them be in conversation with each other helps them with the process as well, I think. So it’s not just that they’re spending time on the project, it’s that they’re doing social interaction, and that social aspect of learning can be so crucial. And they may not have the confidence yet to talk about their work at home or with their friends. But if they’re in a class, they all have to be doing it, there’s this is kind of group impetus that can help them reach that point.

John: That type of peer support can be really helpful in helping them work through difficulties.

Jonelle: Yes, and they may not have that kind of support anywhere else. So that’s why I think it’s important to have it be a part of class because they may not have outside support in their family or among their friend group. So for us to build that academic dialogue is so important for them to see themselves as writers, and to give credence to what they do, and to make space for what they do.

Rebecca: One of the things that I find challenging not only with writing but also, I teach design so we do the same kind of process with visual work, is to help students move from the “pat on the back” motivational feedback, to actually critical, useful feedback so that they develop those skill sets so that they do start to have that community of practice around them outside of a classroom setting. Do you have any tips on how to help move students in that direction?

Jonelle: Well, I think that continued conversation is really important. I will model what kinds of feedback that they should be giving. If it’s an online class, they have to have 350-words in their response to make sure that they have a level of detail and also to give them a rubric so they have to address the elements. I don’t know what they would be for design, but say for a piece of fiction: that you’re looking at dialogue, you’re looking at plot, you’re looking at characterization, theme. So if they’re incorporating some of that vocabulary into their conversation, that can also be a starting point for a future conversation. They’ll be looking for the line in something else, but they’ll be looking for a round character versus a flat character in something that they’re looking at outside of class. It’s kind of creating that atmosphere for the magic to happen and then hopefully, it translates outside of class too.

Rebecca: To round out some of our discussion on reflection, can you also compare the early kinds of reflection that you do in your class versus ones that happen at that midterm or later on? You mentioned those briefly at the beginning, but we didn’t really explore those opportunities.

Jonelle: Yes, the first reflective writing is mostly just goal setting, what they want to get out of the course. And I originally started them on that because students will come into a creative writing course and they may have ideas of writing an entire novel in a course when they haven’t written anything at all before, and then they may leave the class upset with themselves… they haven’t finished a novel. So I try to find out why they’re really in the course at first, so that I can help them achieve the goal that they want to achieve, knowing that I may have to give them resources outside of the class later on. Say they’re writing a novel, that can be like a five-year journey. So it’s something that we can’t address just in 16 weeks, they’re going to need to have something outside to keep them going. So I take a look at what their goals are. And this writing is pretty short and it may not be as nuanced. And then as we go throughout the semester, I devote midterm and end-of-term writing reflection, and those ask more detailed questions, and I have them give more detailed responses. And by then they’re more comfortable with each other as well, so they’re more willing to call each other out if they’re not meeting their goals, if they’re not putting in the writing time, if they’re not putting in the work with their workshop group, or if they need to be doing more editing or revising. They’re more willing to see that in themselves and sometimes they’re more willing to see that in each other but I find that if they can look at one of their peers and say, “Oh, your peer wants this and yet they’re doing this.” They can say like, “Oh, well, I can see easily how you should change this.” Whereas what they don’t see that as readily in themselves. Why I have them work with each other is so that they can see how each other have a different approach to a writing problem, because they’re often able to solve each other’s problems easier than they can solve their own. But then they’ll see, “Oh, I just need to do this. Like maybe I need to wake up and go to a coffee shop instead of writing in the kitchen. Or maybe I need to go write in my car at lunch instead of going to the cafeteria because I’m just going to talk with my friends and not get what I need to get done.” The reflection process becomes longer… more questions… more involved as the semester progresses.

Rebecca: Do you have your peer feedback groups change over time?

Jonelle: Yes, I always mix those up. Otherwise they get complacent with each other and the social interaction kind of overplays that artistic interaction. So always mixing it up.

John: We always end with a question. What are you doing next?

Jonelle: Well, I’m always creating different writing communities. One of the largest writing communities that I really love, and I love to tell people about is National Novel Writing Month where everybody writes a novel in November and it creates a huge community. It’s fun, it’s free. So if people are interested in writing, I suggest that they look at nanowrimo.org. And it’s something that I always love to do every November is to start a novel and to write with other people in community. So you’ll be seeing me in November in the nanowrimo world doing that.

Rebecca: That sounds fun.

Jonelle: Yes, it’s very fun. You can do other projects also. It’s fun to have that group impetus of working together.

Rebecca: Cool, well we’ll look forward to seeing what that novel looks like then.

Jonelle: Thank you.

John: Thank you for joining us.

Jonelle: Thank you, have a wonderful day.

Rebecca: You too.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

81. Intentional Tech

Some faculty try to use each new educational technology tool they find. Others are reluctant to try any new tools. In this episode, Dr. Derek Bruff joins us to examine how to productively choose educational technology that will support and enhance student learning.

Derek is the director of the Vanderbilt University Center for Teaching and a principal senior lecturer at Vanderbilt Department of Mathematics. He’s the author of Teaching with Classroom Response Systems: Creating Active Learning Environments. His new book Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching will be available from West Virginia University Press in November 2019. Derek is also a host of the Leading Lines podcast.

Show Notes

Transcript

John: Some faculty try to use each new educational technology tool they find. Others are reluctant to try any new tools. In this episode, we examine how to productively choose educational technology that will support and enhance student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

Rebecca: Our guest today is Dr. Derek Bruff. Derek is the Director of the Vanderbilt University Center for Teaching and a principal senior lecturer at Vanderbilt Department of Mathematics. He’s the author of Teaching with Classroom Response Systems: Creating Active Learning Environments. His new book Intentional Tech: Principles to Guide the Use of Educational Technology in College Teaching will be available from West Virginia University Press in November 2019. Derek is also a host of the Leading Lines podcast. Welcome, Derek.

Derek: Hi, I’m happy to be here.

John: We’re happy to have you here. Our teas today are…

Rebecca: Do you have anything that you’re drinking Derek?

Derek: So I do, I have some coffee here. [LAUGHTER] I’m not a tea drinker. But there’s a bit of a story. I’m drinking a coffee called Kaldi’s Dog from a local coffee vendor called Bongo Java. And a couple years ago—I work here at the teaching center—we had been serving Folgers coffee in our coffee machine in the break room for several years. And some of us claimed that it was terrible and others of us claimed that people can’t actually tell the difference between coffee brands. And so we actually had a taste test at one of our staff meetings, a blind taste test. [LAUGHTER] From Folgers and several other kind s of fancy coffees and I have to say, I was justified actually. It was very clear that that some coffees were more alike than others. And this was actually the winner, Kaldi’s Dog… the winner of our taste test.

John: So there was no p-hacking or anything going on there? [LAUGHTER]

Derek: No. No.

Rebecca: The nice thing about our tea selection is that we just make hot water and then you can have any of the many varieties that we have in our office.

Derek: That’s true. That’s true.

Rebecca: So speaking of which, what do you have John?

John: Ginger Peach Black tea.

Rebecca: And I have Gold Monkey still.

John: Okay. Mine is nearly empty. [LAUGHTER]

John: We’ve invited you here to talk about your new book. Could you tell us a little bit about the new book and what prompted you to write that?

Derek: Sure. So my work at Vanderbilt involves working with a lot of faculty around their teaching—much like your work—helping them kind of think through the choices they have as teachers, what kind of objectives they have as teachers, and what are some teaching strategies, activities, tools that they could use to try to kind of reach those objectives with their students. It’s a really great job, I get to work with faculty all across the campus, lots of different disciplines. And, in recent years, it’s taken me to other universities as well—and colleges—to kind of talk to faculty there. And so my area of expertise and kind of specialty is around educational technology. And I kind of feel like a lot of faculty come at technology in their teaching from kind of three different areas. Some faculty are told by their administrators that they need to use more technology. And they’re not always sure why. [LAUGHTER] Like what’s it good for? Why do I need this? How can it be helpful? And then kind of at the other end of the spectrum, we have all these faculty who are easily distracted by shiny objects and they see a new technology, and they’re like, “Oh, Pokemon Go, how can I use this in my teaching,” right? [LAUGHTER] And they’re great, these folks are great to work with, they’re all great to work with. But there are also a lot of faculty kind of in the middle who just want to teach well, right? They want to connect with their students, they want their students to succeed, and they want some sensible tools to help them get there. For all three faculty, sometimes they struggle with figuring out how to match technology with learning goals and teaching principles. They know kind of what they want to accomplish, but they’re not sure how to select or use the technology that helps them get there. And so the example I often give in my talks is, I’ll say that my favorite teaching technology is actually wheels on chairs. [LAUGHTER] When I walk into a classroom, right, I have stuff I want to do with my students, I have learning experiences I’ve constructed for them, and I want the furniture in the room to be flexible enough to support what we need to do. Maybe it’s small group activities today, maybe we circle up and have a whole class discussion, maybe we use a debate. I want the technology in the room to support my teaching choices. And so it’s pedagogy first, but then we find tools that help us accomplish those pedagogies. As I’ve talked with faculty at Vanderbilt and elsewhere, I see a lot of patterns in how they use technology and what I’ve done is I’ve tried to distill these patterns down into seven teaching principles, because it’s a book and you have to have seven principles, right? That’s the rule.[LAUGHTER] So seven teaching principles that kind of give you a reason for using technology. And so the intent is to guide faculty to say, “Oh, here’s why I would use technology,” and then each chapter explores one of those principles and has lots of examples of actual teaching practice from faculty in a variety of disciplines. What does it look like in English to use technology to accomplish this goal? What does it look like in biology? What does it look like in communication studies? That kind of thing. I love telling stories and one of the reasons I’m excited to be a part of the Teaching in Higher Education series of West Virginia University Press, it’s edited by Jim Lang—who is a fantastic writer—and he takes this kind of storytelling first-person personal approach to his writing and I was really excited to be a part of this series, because that’s how I like to write too.

John: And you start your book with a chapter on a time for telling, speaking of narratives. Could you tell us a little bit about what the focus of that is?

Derek: This is a little counterintuitive, sometimes for faculty, but it’s really one of the most useful principles I found when working with faculty around designing especially—I mean, to some degree, it works at all scales—but it’s really helpful in kind of a lesson-plan scale, like one day in the classroom. And so I think sometimes there’s this impulse that faculty have to explain the thing, and then have the students do something with it, right? Here’s what it is, here’s the background, here’s the context, here’s the theory, and then let’s have the students do something with that. But the example I give actually comes from my daughter’s preschool. This was 10 years ago now, her preschool had science day and they asked the parents to come in and do sciencey things. And so I was the dad who brought the Mentos and Diet Coke. [LAUGHTER] So, you’ve seen this, right? You take a Diet Coke two-liter and you put some Mentos breath mints in there and then half a second later, you get this huge geyser of Diet Coke. It’s rather dramatic. Mine only got maybe seven-feet tall, but I’ve seen them much higher on YouTube. [LAUGHTER] And so I do this kind of fun thing in front of the five-year olds. And then they ask me “Why did it explode?” And so one could lecture—maybe not to five-year-olds—but you might lecture to a bunch of chemistry students for fifteen minutes on the physics and chemistry behind this and then do the demonstration. Or you could start with the demonstration and have students conjecture. Why is this working? Why does this explode, right? Bless her heart, my daughter asked, “Why is it Diet Coke?” I was like, “That’s a good question.” Why is it diet and not regular Coke? So this is the idea behind time for telling—this is a term from the literature Schwartz and Bransford wrote about this back in the 90s—that if we can create these times for telling with our students where they’re ready to learn and they’re interested in learning, then they’re going to get a lot more out of it, they’ll learn more deeply. We can use technology to do this. One of the stories that I share in the book is a grad student in English at Purdue University, Alisha Karabinus, and she’s teaching a first-year writing composition course and she has her students play this text-based online game. It’s all text, and you’re typing in commands and telling your character where to go. You kind of wake up in this apartment and you’re not sure what’s happening and you have to navigate and walk through the apartment. And there’s this kind of sequence where you need to take a shower and so you walk into the bathroom and you try to take a shower and the game’s like, “You still have your clothes on, you have to take the clothes off.” And then they’re like, “You probably don’t want to walk into the shower with your clothes in your arms.” And so you’d have to put the clothes down and you have to take her watch off, right? Like there’s all these kind of step-by-step things. I’m of a certain age where I played games like this back in the 80s…

John: The old adventure games, yes.

Derek: Right. And so there’s a kind of, you know, interface here that you have to master and you have to learn. It needs to be very explicit about what you’re doing. So she has her students play this game outside of class and then they come in and they debrief the experience. And it’s really lovely because they get so frustrated with the interface, then she makes this nice, clever little pivot where she says, “Well, when you’re writing, when you’re trying to express yourself, you’ve got all these ideas in your head. If you’re not explicit with your reader, they’re not going to know what you’re actually saying.” And so she uses that to talk about transitions and topic sentences and things like this. And so I think it’s a really lovely example of using a little bit of technology that was not at all designed for teaching to give students an experience that then prepares them to learn this lesson about how they communicate and how they write.

Rebecca: Do you have any advice on how to find some of those key little demonstrations or technology pieces that could lead into particular ideas?

Derek: Yeah, I’ve got some more examples in my book. I mean, part of it is that I think—especially for the time for telling—there’s this kind of experiential piece that’s pretty great…

Rebecca: Yeah.

Derek: A lot of faculty will show a video clip. This is one way to kind of do it. The Office is very commonly used to introduce various topics in different courses. I have several examples in the book of games, either video games or board games, and so I think there’s some real value in this experiential piece. And so, there’s no silver bullet. I think a lot of this involves being open to taking something outside your area and bringing it in. In this case, part of it was the interface. It wasn’t the content of the game that was interesting, it was the interface of the game that really helped. And so those are things to look for. Is it the content? Is it the interface? Those are helpful. I also talk about what tools are designed for teaching that can help create this time for telling. And so my first book was all about teaching with classroom response systems—which used to be called clickers—and now in most places students bring their own devices and answer on their phones. But the idea is that you can pose a question to all of your students and they all answer and then you can show the distribution of answers up on the big screen. And if you’ve asked a question that really taps into some type of misconception that students have, and they get the question wrong, the technology is important here, because you want everyone to answer so they all have that experience of grappling with the question. So you need a way to hold all the students accountable for answering in a way to collect all their answers. So you need some tech for that. And then by displaying the distribution of answers and the wrong answers on the board, you let everyone know, “Hey, this is a hard question, right?” It’s not like everyone got this right? You’re split across these two different answers. I share an example in my book of a colleague here at Vanderbilt in the law school, Ed Cheng, and he’d ask a series of questions of his students about Carl and his rhinoceros. So this was a situation that was perhaps prone to disaster when Carl keeps a pet rhinoceros and so he plays out these different scenarios of things going wrong. And then he basically asked these multiple choice questions about who can sue whom for what. And so the first two questions are actually really straightforward, right? The rhinoceros escapes and runs into a car or something, and Carl should have known that was going to happen. And there’s clear cut answers to the first couple of these clicker questions that my colleague asks. But then the third one, it’s a little bit different. And the students, when they respond to the scenario about Carl and his escaped rhinoceros, they’re actually split across three different answer choices. Ed has this great move in class where he’s like, “Well, you’re kind of all right. There are parts of law that are really clear cut and there’s a clear answer, and we just talked about a couple of questions that fell under that category, but we’ve moved into this area where there’s actually not a clear answer and a good lawyer could argue any of these.” It’s critical for his law students to know when they’ve moved into that part of law because that’s when they have to really do the hard work and marshal the resources and make the arguments and work with evidence. And so he’s using this short technology exercise to create this moment where they’re like, “Oh, right, I need to really pay attention here.” It’s that time for telling moment that I think is really lovely and having the bar graph on the screen that has the three-way tie is really important to creating that moment.

Rebecca: I think those are really good examples that I think will help faculty get started.

John: Actually, we did talk about one in an earlier podcast where we had Doug McKee on and he was talking about using this technique in his econometrics class, where you give students a problem that’s just a little bit above what they’ve been working on and it forces them to recognize the need to develop new tools, and then they’re primed to be receptive to a solution if they don’t quite make it all the way there themselves.

Derek: Absolutely. And again, this is a little bit counterintuitive. I think some faculty are hesitant to give their students a problem they know they can’t finish, or they can’t solve, or they haven’t been fully prepared for. But by having that experience, starting class with this hard problem that they can’t quite finish and getting stuck and recognizing the limits in their mental models or their need for additional resources, then they’re ready for the second half of class when the faculty member’s like, “Oh, here’s the resource, here’s the concept, here’s the tool.” And again, very non-intuitive and one of the things I think that’s important about my book is that my focus is on using technology to accomplish these things. But all of these teaching principles are true regardless. You don’t have to use technology to create a time for telling, but it is an interesting and useful way to think about certain types of technology and how you might bring them into your classroom.

Rebecca: I think that’s an important point to hit. The technology is supposed to follow the pedagogy like what you said earlier.

Derek: Yeah.

Rebecca: So remembering that you need to make good teaching choices first, and then finding ways to support.

Derek: Yeah, and sometimes tech is not the answer, right? Or low-tech is a better choice, right? So I have a chapter on knowledge organizations and so this is the idea that when we all organize the information in our head in various ways and so you can kind of imagine in your head like this concept map of ideas and examples and facts, and there are connections between all of them and novices in a domain, right? When our students walk into our class, their knowledge organizations are not as robust, they don’t have as many nodes, they don’t have as many connections, and connections are not as meaningful, and part of our work as instructors is to help them develop more robust knowledge organizations. Well, if we just leave them to their own devices, they’ll do okay, but we can actually help them learn and see the big picture in our course, if we can give them activities that help them develop, construct, represent, and visualize their own knowledge organizations. And so I teach a first-year writing seminar at Vanderbilt—and I talk a little bit about my own teaching in the book, because I think it’s important that I’m using these tools myself and trying to figure out how they work—my first-year writing seminar is on cryptography. Codes and ciphers. And we talk about privacy and surveillance and the role of encryption in today’s society. As part of this, I teach a novel, even though I’m in the math department. [LAUGHTER] It’s not something I was trained to do in grad school, to teach a novel, but I do work a teaching center so I’ve picked up some ideas. But there’s this book called Little Brother by the author Cory Doctorow and it kind of imagines a terrorist incident that happens in San Francisco and then the kind of surveillance and security apparatus that comes after that and the lead character is this teenage hacker who’s kind of fighting against this. And so what I have my students do is they read the book, I have them blog about it in the course blog—so that is a digital technology that they use—but when they come to class, I ask them to get into small groups, I give each group a couple of large Post-it Notes—so these are the kind of five-inch by seven-inch brightly colored Post-it Notes—and some markers. And I say, “Your job in the group is to find two arguments in the book in favor of surveillance and two arguments in the book in favor of privacy.” And so they have to kind of page through the book, they’re looking for arguments made by specific characters in favor of one of those two things. And so the privacy arguments go on Post-it Notes of one color and surveillance arguments go on Post-it Notes of another color. And so then, once they’ve done that piece, the second phase is they in turn go up to the chalkboard and they put their Post-it Notes on the board one at a time, and they have to do two things here. One is they have to put practical arguments towards one end of the board. Like, “If we monitored everyone’s subway movements, are we actually going to catch bad guys?” Like that’s a practical argument. And then have to put principled arguments at the other end of the board. So one of the characters says that, “Hey, it’s about life, liberty, and the pursuit of happiness in that order. If you’re not alive, you can’t be happy, so we got to keep you safe.” That’s a principled argument. And so the students have to put their their Post-it Notes along this axis and then they also have to use the chalk to connect their argument to something already on the board. Because basically, there’s this really complex debate space around safety, and security, and privacy, and surveillance, and I want my students to know how complex that is and to start to see the relationships between some of those arguments and ideas. “This argument is a counter to that argument, or this argument is a support to that argument.” So by the end of class, they’ve constructed this debate map on the chalkboard out of Post-it Notes. And they have, I think, a much better sense of the complexity of this debate. They’ll do more with this. They’ll write about this topic throughout the semester. And so that debate map, that knowledge organization that they’ve constructed collaboratively, can then inform the arguments they make as they take positions within that debate later. And in this case—as I said—this is pretty low-tech, it’s Post-it Notes and chalkboards. I actually tried it once using some software, but having students build this map in a collaborative digital space at the same time was just too chaotic and so I needed to kind of slow the process down, and the Post-it Notes were really great for that. And so this is something I’ve used several times in my course and I think it’s a really great way to help students see how the ideas in an argument space are connected.

Rebecca: One of the things that I’ve liked about using Post-it Notes in some of the kinds of things that I do in my classrooms is that it is easy to change your mind too. You can easily pick it up and move it and it doesn’t seem as intimidating as trying to navigate software to make a decision or something. It somehow lowers a whole bunch of barriers and then it’s a little more flexible and fluid.

Derek: Yeah, absolutely.

John: And you’ve got them thinking about it among multiple dimensions and also making connections, which really, I would think would help them develop a lot of scaffolding there and a lot of connections that they wouldn’t necessarily do with their own reading. So it’s forcing them to develop better close reading skills and analytical skills, and so forth. It’s a wonderful exercise.

One of the principles of learning is that it helps for students to have lots of feedback opportunities and lots of practice, and I see you have a chapter on that. Could you tell us just a little bit about what you focus on there or some of the points that are made in that chapter?

Derek: Sure. So I actually start the chapter of the story about how I learned how to ski a couple of years ago, because learning to ski as an adult is a challenging process, as I found. [LAUGHTER] It involves much falling down. And every time you fall down when you’re learning how to ski, your body is getting a little bit of feedback about what works and what didn’t. And so in a very kind of physical motor skills way, to learn to ski, you have to practice skiing. You fail a lot, your body gets feedback, and then hopefully over time you get better at manipulating your limbs and controlling your muscles so you’re going kind of where you want to go. And so our students need this too, this is actually so key to learning is that we have to practice with the stuff that we’re learning, we have to do stuff with it and we have to get feedback on that practice. It’s a key part of learning. In the chapter, I use this as an opportunity to talk about the so-called flipped classroom because I think there’s a traditional model in some of our disciplines where students get an introduction to information during class. And then after class, they go and do something with it. They do the practice, they have a problem set, right? And the problem is that the practice and feedback part, it’s really important and also really hard. And so to have students do that when they’re left to their own devices, is a lost opportunity actually. And so the flipped classroom model says, “Let’s take some of that activity. Instead of doing it later on your own, let’s do it together collaboratively during class.” And so in the chapter I talk about some ways that some faculty have used technology creatively to help students practice the skills of their discipline during class. I mentioned the classroom response systems as certainly an option for this. I think sometimes when I talked to faculty around technology in the classroom, there’s sometimes an assumption that you’re talking about AV tech. We have a projector, we have some speakers… and that’s great, we need that, that’s helpful. But all my examples involves students using the technology because I think that’s really important. One of my favorites, actually is Kathryn Tomasek from Wheaton College. She’s a historian, and she wanted her students to practice doing the kind of close reading that historians do. When they get a primary-source document, a lot of that reading they do is looking at it line by line, word by word, figuring out who is that person? What is that term? What does it mean? Looking at the very building blocks of this primary-source document, because especially if you’re separated in time by one hundred or two hundred years, you got to do a lot of this close reading to kind of make sense of what it is. And so she had her students work with… she started with historical documents in her library’s special collections and asked her students to do what’s called a TEI. It’s a text-encoding initiative, it’s a way of marking up the text—kind of like HTML a little bit if you know web development—where you’re actually kind of tagging things in the text and labeling them as to what they are. So this is a date, this is a noun, this is a person, this is a location, or in her case, this is a theme that comes out in the sentence. And so her students, they would do this together in class. Like she’d take a piece of it and walk them through it collaboratively on the big screen and then have them take their own pieces and do this markup. And the neat part is the students were actually contributing to a larger digital history project because their markup would be kind of incorporated in this bigger database and shared online. And the work that they did with the primary source documents would then inform the writing and the argumentation that they do later in the semester. But in this case, she wanted to target a very particular skill that’s kind of close reading in history and she found a technology that digital historians use actually pretty regularly to create these opportunities for practice during class to help our students do this kind of work. I share another example of Richard Flagan from Caltech and he was doing chemical engineering. Very different course. But he used little mini projectors to kind of turn his lecture hall into an active learning classroom so his students could work in groups and do some coding—they were doing MATLAB coding in this case—and he found that when he introduced the coding in class and had them work on it after class, they would get hung up on these really small errors around grammar and syntax in the code. So he shifted that work into class to do group works in class and so then he’s able to kind of circulate among them, see what they’re doing on their little projector screens, and intervene and ask questions and help them. And so again, it’s kind of targeting these very particular skills that students need practice with that will inform often bigger projects later in the semester, but creating some time and space in class through technology to give them a chance to practice those skills and get feedback either from each other or from the instructor.

Rebecca: A lot of those examples I think are opportunities for faculty to also see where misconceptions are happening because it’s happening in class soyou can address them one-on-one, but you can also address those bigger themes that bubble up as well as a bigger group rather than having the same conversation 20 plus times.

Derek: Absolutely. Yeah,you may walk over and talk to one student or a small group or you may see a pattern across the students and then kind of take a moment to kind of gather everyone’s attention and try to kind of walk them through together as a whole class.

John: Doing some just in time teaching type of techniques, which is much more efficient use of class time.

Rebecca: That seems really tied to the knowledge organization that you were talking about as well because I think those same kinds of things happen when you’re doing those sorts of activities in class too, right? Like, “Oh, I didn’t realize that you thought this was connected to that,” right? And you can help negotiate that. [LAUGHTER]

Derek: Yeah, and these teaching principles overlap, right? So when I had my students do that debate map activity in class, we were doing practice and feedback. We were taking class time, it was just that this kind of analysis level where they were making connections across topics as opposed to Kathryn’s example of the close reading. That wasn’t necessarily a big picture kind of practice and feedback, it was a very skill focused practice and feedback.

Rebecca: I also really like that these are examples that don’t necessarily make feedback more work for faculty. It’s embedded in the practice in the classroom and it’s just when they need it. And it makes more sense because they’re getting it while they’re doing something so they’re probably more apt to listen to said feedback rather than getting it on an assignment that you hand back and they put it in the garbage or something.

Derek: There’s this book by Walvoord and Anderson called Effective Grading that you may be familiar with and I remember the first time I read about what they called light grading. L-i-g-h-t grading. I thought, “Oh, this makes so much sense. I don’t have to grade everything the students do very rigorously, I could give them a grade on the effort or I could give them a zero, one, or two if the work that they’ve invested in that little piece then shows up later.” So if I have them write a blog post before class to get ready for class, I don’t have to grade that very intensely because we’re going to talk about that material in class and that’s where they get the feedback. I may need the grade it enough to motivate them to do it, but I don’t have to give them detailed feedback at that stage, it will happen during class discussion. And I think that’s kind of freeing for instructors to know I don’t have to grade the heck out of everything. I can kind of design a sequence where students get the feedback they need apart from the grade itself.

Rebecca: I think faculty always appreciate those opportunities. [LAUGHTER]

John: And in the podcast that’ll be coming out a week before this, we talked briefly about specifications grading, which is a variation on the same theme.

Derek: Oh sure.

John: One of your chapters is entitled “Thin Slices of Learning.” Could you tell us a little bit about that?

Derek: I am so glad you asked. [LAUGHTER] Yeah, so this may be my my favorite chapter in the book just because I think the creativity that faculty brought to their use of technology in some of the examples I share, it’s just really amazing. And I also get to quote, one of my mentors a couple of times. Randy Bass is, as I like to say, the Vice Provost of Awesomeness at Georgetown University. [LAUGHTER] That’s not his actual title, but he gets to do some really amazing things there. He’s also really active in the Scholarship of Teaching and Learning world and I think he’s just a really deep thinker about how learning works and so there’s a couple of things that he talks about that I’ve heard him talk about before. He was doing some video projects with his students in an American Studies class a while back and he would look at their finished products—these short videos that they put together as a class assignment—and he realized that he wasn’t seen all of their learning. That—as he said it—there’s a lot of learning that’s left on the cutting room floor. And actually, in the book I talk about how my daughter wrote—she created a short film a couple years ago just because she wanted to—and she filmed, I think three hours of footage for a two-minute film. And so to decide what footage to use, what footage not to use, which angle, which take, even kind of which characters she wanted to include in the final product, there was a ton of decisions that went into those final two minutes. But if you just look at the two minutes, you may not know what those decisions are.Ttechnology, though, can be really good at making visible student learning, and in particular, thin slices of student learning. The kind of choices and sense making that they’re doing in the middle of learning, or creating, or designing, or producing something, and the more we can learn about how our students learn, the more we can kind of get those thin slices of learning in front of us. We can be responsive, we can be helpful, we can guide, and we can direct. And so the examples in this chapter about using technology to make visible (or sometimes audible) kinds of learning that students might not kind of share with us naturally. My all time favorite example of teaching with Twitter is from Margaret Rubega. She is a biology professor at the University of Connecticut. She’s also I think, Connecticut’s State Ornithologist, and she teaches a course on ornithology. So it’s a course on birds and so she has—wait for it—she has her students tweet about tweeters. [LAUGHTER] She’s so articulate about it, like students come to this class—and it’s a fairly large class, I don’t know, 40, 50 students—and they’ve seen really cool birds on National Geographic or YouTube and they think of birds as doing really amazing things in the Amazon and in Africa and far away. And she wants her students to know that birds in rural Connecticut also do really interesting things. Their ecology, their behavior, their biology is all very interesting. And so what she has them do as they’re learning about birds in the course is several times a semester they’re asked to tweet about their observations of birds as they go about their lives. So they’re on their way to work, they’re on the way to school, they see a bird, they see it do something or behave in a way that connects with what they’re learning in class. And she asked them to tweet about it. They have to kind of include their observation and where they are, and they have to connect to the course material, and they’ve got now 280 characters to do that on Twitter. Sometimes they get photos of birds that they see, which Twitter is really good at handling that. The thing I like about it is that it’s leveraging the field observation device that they carry around with them in their pockets—also known as their smartphone—or their regular phone, whatever it is. When they’re in the moment when they see that bird doing something, they’re able to kind of capture that and then share that back with her and with the entire class actually. One of my favorite tweets is a student who was walking by a golf course and he noticed the bird song sounded different in different parts of the golf course and in his tweet he conjectured that the golf course itself was dividing bird territory. [LAUGHTER] I was like, “That’s genius,” right? I don’t know if he’s right, but he’s really paying attention. And what Margaret’s doing is she’s helping her students practice transfer, taking what they learned in the classroom and apply it to real-world situations outside of the classroom and that’s one of the hardest things about learning, is how to transfer learning to new context. And she’s giving her students explicit practice in this, but then also making visible that practice by having this class hashtag on Twitter, #birdclass. And if you go search on Twitter for bird class you’ll see some of the tweets from her students because they’re sharing their observations with her, with each other. I just think it’s a really beautiful use of a very particular technology for a very particular reason. I love the bird class example.

Rebecca: That’s really fun.

John: Yeah, that’s a great example.

Rebecca: I couldn’t help but think that if my two-year-old could tweet, she’d be really into it right now. [LAUGHTER]

Derek: Does she have a lot of observations about life that she tries to share?

Rebecca: A lot about birds lately. [LAUGHTER]

Derek: But you think about all the learning that students do when they’re not with us that may be really important. I have another example from Mark Sample at Davidson College, where he had his students live tweet the film they were watching in his sci-fi class. They watched it on their own time in their own rooms or whatever, but it was Blade Runner and they would live tweet their observations about the film. And it’s one thing if we have our students read something or watch something and then write a response paper and bring that to class and then we discuss it, right? That’s great. But he was getting kind of a next level down. They are kind of immediate in the moment reactions to what they were seeing in the film and kind of surfacing that and making that visible. And this is a great use of technology. There’s other ways to do this but technology can be really good at this kind of thin slices of learning piece and that’s one of the connections I want folks to make in the book is that, if you think about it, “Hey, Twitter seems really useful. What can we do with Twitter and our teaching?” Well, there’s a lot of things you can do. But one of those is to surface thin slices of student learning. And that provides some focus for thinking about how you might use Twitter in your class for a very particular purpose.

Rebecca: Sounds really fun.

John: It does, and you can see this in other areas of biology or botany. I think Michelle Miller was on a while back and she talked about a class where students went out in the field to identify plants and tweet back photos and so forth.

Derek: Yeah, and I shared this with a grad student here in civil engineering and she has her students tweet about structural things they see in the built environment that connect to the material that they’re learning. I’ve ran into teacher educators who have their students—while respecting some privacy—but they’ll tweet about what they see in the field when they’re in classrooms and they’re observing teachers in action. They’ll tweet those observations and so yeah, I think there’s a lot of different uses for this kind of application.

Rebecca: This also moves a little bit into the idea of learning communities, because this is a community practice using a hashtag where you’re kind of seeing things outside, but you also have a whole chapter on just learning communities.

Derek: Yeah, and I mentioned the bird class, because there’s a lot of things going on with the bird class piece. And part of it is that yes, by making the students tweets visible to the other students, you have this other dynamic going on which is that the students are starting to learn from and with each other. And students can learn a lot from their professors certainly—and we have a lot of expertise and authority that we that we use in the classroom—but if you think about the places where you learn naturally when you’re picking up a new hobby, or you’ve got some interest of yours that you want to pursue, you’re often connecting with other people who do that too. So like, I have a friend who just went to a quilting conference in Nashville because she loves to quilt and she’s going to connect with other quilters. And that’s how she learns how to quilt, it’s this kind of peer-to-peer learning that she does. And so we can leverage that in the classroom. It’s always a little authentic. Students in a statistics class aren’t going to just get super passionate about statistics and learn from each other necessarily, they’re going to bring their own levels of intrinsic motivation to this. But if you think about all of the different perspectives and experiences that you have in the room with your students, they have a lot to bring and they can actually learn a lot from each other and you can learn a lot from your students. But you’ve got to create some mechanisms for that, it’s not necessarily going to happen naturally. And so bird class is a great example because as the students, I mean, they’re all in Connecticut, right? But other than that, they’re going to different places, they’re seeing different parts of town, then they go to different locations for spring break, and see different birds, and so they’re all bringing their kind of different perspective on this. And by making that visible, they can start to learn from each other. In my chapter I talk about, I use a social bookmarking tool called Diigo, which allows basically students to share links with each other. And so we create a group for a class and I give them these assignments, in my cryptography class especially. So, find an example of cryptography in the news or find an example of military cryptography or let’s find out something about the National Security Agency. And so what’s really cool is—especially for a course like this—students bring a lot of different interests into this topic. And so I’ll have students who have kind of like a literature interest—I had a Sherlock Holmes buff in the class once and so she was always finding really interesting examples of cryptography in literature to share—I had students who were always interested in kind of cybersecurity and computer science and so they’re bringing in kind of modern news and technology. I had one student—bless her heart—she loved Russia, she found a way to find a Russian connection to everything that she did. And so it was really great, right, because she found all these examples of cryptography, especially kind of Cold War espionage stuff that we wouldn’t have seen if she didn’t have this passion for Russia and then found resources and shared them with the class. And so by having students share these resources, in a shared space and then talk talking about them in class, I can really leverage the fact that we’ve got a number of individuals in this room that have different experiences and perspectives and if we can make advantage of that, we can actually all learn more deeply.

Rebecca: I’ve used Slack in my web design classes to do troubleshooting and technical help but I use the same Slack channel across semesters. And so what I found is that people who have graduated who are out in the field will sometimes randomly pop in and answer questions, and it’s really cool, but I remind the students that I’m not the only one that can answer questions. They can help each other out. But sometimes—you never know—some other lurker might pop in and help out. And they have some sort of vested interest, you know, because they were also in that class at one time.

Derek: Absolutely, and that’s one of the advantages. One thing that can happen when you shift away from a course management system, course management systems are good for a lot of things but they’re not good for semester to semester continuity. They kind of put courses in little boxes by semester and the students can’t get out of those boxes. And so once you move to Slack or social bookmarking—like my Diigo group, we’ve been doing it for like seven years. We’ve got hundreds of resources collected by students over time. Course blogs are really good for that too—and so that’s really exciting when you can use some technology to make some student work public and persistent in a way that invites future or past students to participate. It’s still a learning community, it’s just expanding beyond the time and space of this one particular course offering

John: One of the issues with learning management systems is—as Robin DeRosa and other people have called it—is that the assignments often take on the nature of a disposable assignment, that they’ve done the work and then at the end of the semester, they even lose access to it unless they keep it outside. And there’s a lot of advantages to having a sort of persistent work that you’re describing there.

Rebecca: It seems like you’re moving right into another chapter of Derek’s book on “Authentic Audiences.”

Derek: I mean this is the other thing—and again, the book is not a critique of course management systems—but I will say course management systems also make it hard for student work to escape, to be seen by anyone not in the course, and often that’s appropriate. When students are first learning a topic or a discipline, they need a private space to practice, and screw up, and say dumb things, and get feedback, and get better. And that’s true for the assignments, right? Sometimes we have students turn in an assignment to us and we’re the only one who looks at it because they’re still learning the skill set and they need some good practice on that. But when we have students construct work or produce work for authentic audiences outside of the course, that can be hugely motivating for students. Hugely motivating. I’ll quote Randy Bass again, he’s got this white paper where he coins this term social pedagogies. These are pedagogies where we’re asking students to construct their knowledge by representing that knowledge for an authentic audience other than the instructor, and it could just be each other. That can be really powerful as well. But when students see that the work that they’re doing is not disposable, it’s not going to be gone. Students often do write a paper and there’s only one human being on the planet who ever looks at it, but if you can build toward some assignments where students are writing or constructing or producing for an external audience or an authentic audience, there’s a lot of motivational benefits to this for students and they start to take their work very seriously and invest in it in ways that they don’t sometimes in the disposable assignments. One of my favorite examples, Jonathan Rattner teaches cinema and media arts here at Vanderbilt and he had connected with a colleague of his from grad school who was teaching a writing course, Bridget Draxler, she was at another institution. Jonathan was teaching students how to create short experimental films and they needed an audience to share that work with. Bridget was teaching her students to critique media and she wanted her students to find media to critique where they could interact with the creator, and so they just set up a course blog for the two of them, these two courses. It wasn’t public to the world but Jonathan’s students were creating media for her students and her students were critiquing it and then they would have this conversation. And this idea of connecting your course, with just one other course—somewhere else on your campus or maybe at another institution where you have colleagues working—all of a sudden, you have this really authentic audience for the work. And in this case, this was really intentional too, this wasn’t just a random pairing “We want to share stuff with someone,” but there was this kind of synthesis that worked well across the two courses. But that’s a fairly easy way to create some authentic audiences for your students. I also talked about Tim Foster who used to teach Spanish here—he’s out at one of the University of Texas schools now—and he had his students write for Wikipedia. This is actually becoming increasingly common in higher education where you have students write for Wikipedia. There’s certain standards that you have to follow and it’s kind of hard to get content to stick on Wikipedia because of that. He was actually teaching an introduction to Portuguese course—so this was first semester Portuguese language learners—and what they realized was that the Portuguese language page for Nashville on Wikipedia was very skimpy. And what his students didn’t know at first is that Portuguese Wikipedia is not just a translation of English Wikipedia. Portuguese speakers create their own Wikipedia. And so the national page was kind of skimpy. So as a class project, he had his students create content for the Portuguese language Wikipedia page for the city where we are. And so it was great as a language production task for them because they could focus on writing two or three sentences, first semester language learners, but they knew that actual people are going to look at this so they took it very seriously. Some of them went above and beyond. I think this is just a really powerful pedagogy. And again, you don’t have to use technology but this is something technology is actually good at, is connecting people across time and space. Having students use some technology to create something for an authentic audience can be really powerful.

Rebecca: I think you have one last chapter that we didn’t quite get to yet and that’s…

John: “Multimodal Assignments.”

Rebecca: Which, you know, technology is also really good at that whole multimodal thing, right? [LAUGHTER]

Derek: Sure. I was so close to calling this chapter “Beyond the Five-Page Paper.” [LAUGHTER] Because again, the five-page paper has its place where students write a thing, and it’s just text, and they give it to the professor, and they get feedback. Again, there’s a lot of practice and feedback that happens in activities like that. But there’s a lot of research that says, not so much that learning styles exist. The research actually doesn’t support this idea that I need to match my teaching modality to my students learning preference. So the matching hypothesis would say that I have some visual learners and some verbal learners and some kinesthetic learners and so I should do visual stuff with the visual learners and verbal stuff with the verbal learners and kinesthetic stuff with the kinesthetic learners. There’s no research that supports that actually. So there’s kind of this learning styles myth that I like to debunk when I can but where the research does support is multimodal learning. Now, we all learn better when we encounter stuff in multiple ways. And so I think this is the reason the learning styles feels so compelling to a lot of instructors, is that if they’re doing that, if they’re thinking about their lesson plan and saying, “Oh, I got to have some visual stuff, I got to have some verbal stuff, I got to have some activities.” It’s not that they match those with individual students, it’s that all students are benefiting from those three different modalities happening in the same classroom. This chapter is all about multimodal assignments, ways to tap into this dual coding that our brains do where we take in information in verbal ways and in visual ways and kind of put that together and it’s stronger. We’ve done a lot of work at Vanderbilt. We call it students as producers. This is kind of a course design and assignment design approach that we work a lot with here through our course design institute and elsewhere. It’s helping faculty move away from some of those traditional text-only assignments and moving to more open-ended assignments, more multimodal projects, student projects that have authentic audiences. And so actually, this chapter is kind of all Vanderbilt examples which sounds a little self-serving, but I just happen to know a lot of faculty here who are experimenting a lot in this area. I share an example for my own classroom about infographics in a stats class where I’m asking students to represent quantitative information visually. There’s an English grad student here, Kylie Korsnack, who has her students take a paper they wrote, and revise it in a different medium. So it starts off as a traditional paper but they have to revise it as a Prezi or a concept map or choose your own adventure novel, or one student did a Pinterest pin board. And so by kind of re-seeing their work, moving into this other sort of medium, the students are often able to see their work in new ways and realize, “Oh, that’s what my argument really was,” or, “Oh, my transitions are terrible. Now I know how things have to be connected.” And so there’s a lot of value in having students move into different media than straight text as a way to help them make sense of things. I’ve been experimenting a lot with podcasting. So I got this idea from Gilbert Gonzales, a colleague of mine here in health policy, who had his students create podcast episodes instead of research papers. And he really wanted them to be fluent with the language of healthcare policy. HMOs and PPOs and all this kind of stuff. And so an audio assignment made a lot of sense, actually, for the students. And he founded it with a lot more fun to listen to a few podcasts than grade a few papers [LAUGHTER]. And podcasting is a low bar, right? Not to say that what we’re doing isn’t super challenging here, but you can, you know, create a pretty decent podcast with your phone, right? It’s not going to be super high quality, but you can record and you can edit using some free software and so I now have my students do a podcast assignment in my cryptography course. And with about 25 minutes of in classroom technical training, they’re able to produce some interesting things and then we can focus on “How do you tell a story through audio?” Or in my class, how do you explain this kind of technical mathematical stuff that they’re studying through audio only without pictures? And so, again, all of these are about kind of moving to different modalities and shifting between modalities to help students see and understand the material in different ways. And if you keep doing that, they’ll start to kind of triangulate and, and make a lot more sense out of it.

John: I would think it would force them to think about it a bit more deeply to make connections that they might not otherwise. Just seeing things from a different perspective seems to have a lot of value in it.

Derek: Yeah. I would also add that when you move to a non-traditional assignment—this is something that I realized kind of late in writing the book—is that we asked you to do a podcast they walk in and they don’t know how to do podcasts. So Gilbert and I were like, “Okay, so we have to listen to some podcasts together and critique them, and then maybe come up with a rubric together, and they need to outline it and maybe even turning the script and get it approved.” We have this whole scaffolding process around preparing students to do this type of work. Well, some of our students come in and they don’t know how to write a five-page paper either. We assume they do, we assume they’re good at it, that they’ve learned that in high school or something. But for some of these traditional assignments, we have students who really struggle and so one of the things that non-traditional assignments do for faculty is help them realize, “Oh, I really have to get in the head of my students and figure out what’s the scaffolding they need.” And we really should be applying that to more traditional assignments as well, because a lot of our students struggle because we don’t have them submit a proposal, or get feedback on a rough draft, or practice how to find a credible source. These are all things that we that it’s easy to assume our students can do, but they can’t always actually do that. And so moving to a non-traditional assignment often then helps faculty move back to more traditional assignments with a new lens with greater intentionality.

Rebecca: So we have to wait until November to get this book? [LAUGHTER]

John: Yeah, I know I saw when you posted this on Twitter, I said “I’d like this now.” [LAUGHTER]

Derek: Well thank you, yes. The book production process is a long timeline as I’ve found.

John: Yeah.

Rebecca: It’s a good tease though, right? [LAUGHTER] So you were also just talking about how you’ve been experimenting with podcasts and you’ve been the host of Leading Lines since 2016. How did you get interested in doing all this podcasting stuff in the first place?

Derek: Part of it was that—at the time—I had a 45 minute commute to work so I was listening to a lot of podcasts and appreciating podcasts and wishing I had more podcasts like Tea for Teaching that talk about teaching, learning, and higher education. And so that was part of it. I think, also, I was involved in a pretty big online course project that involved a ton of video work. And I saw how powerful that was, but how much work it was to put together really high-quality video and I thought, “What if we had a podcast on educational technology?” There’s folks that I run into here on campus and elsewhere who are doing really cool things and I would just love to kind of give them a bigger audience for the innovative teaching that they’re doing, and producing a podcast seems way more tractable than producing a YouTube series. [LAUGHTER] So I mentioned this at a meeting here, we were having this meeting on campus with some other folks who deal with educational technology. One of our Associate Provosts, John Sloop, for digital learning, he’s like, “I would love to do a podcast.” We kind of both had been thinking about this idea for a while and so we combined forces. And so it’s the Center for Teaching, it’s our libraries, it’s our Institute for Digital Learning. We created Leading Lines, we’ve been doing it for a few years now, each episode is an interview with a faculty, staff, or grad student who’s kind of doing something interesting in the educational technology space. We call it Leading Lines because it has this kind of connotation of looking into the future. So leading lines in a photograph, are those kind of straight lines that draw your eye into the frame. And so we’re not really trying to predict the future—because I think that’s a fool’s errand—but I’d rather kind of shape it and influence it and so we’re looking for folks who are doing things that are kind of one or two steps down the road with technology. And it’s been really great, I mean several of the examples in my book are drawn from interviews I did for Leading Lines. It just gives me this occasion to talk to interesting people who are doing interesting things.

Rebecca: Now you know our secret. [LAUGHTER]

Derek: Right.

John: We’ve gotten this opportunity to talk to all these people doing some wonderful research that we wouldn’t be able to be in contact with so many of them otherwise.

Derek: Yeah, absolutely. I’m always referencing these people that I’ve met and the work that they’re doing and my other work here, connecting faculty, and it’s been a lot of fun. And it’s been fun to work here. So we have about six of us who do interviews for Leading Lines and so we have a little bigger team then you guys have and they go in different directions. Sometimes I’m like, “That’s really not an interesting topic,” and then they do an interview and it’s an interesting topic. And so it’s been really fun to kind of work with my colleagues here and having this collaborative project across multiple units at Vanderbilt, that’s been pretty great too.

Rebecca: Cool.

John: On one of the recent episodes, you had a discussion of the VandyVox project. And in particular you had a podcast from there. Could you tell us what this project is and how that came about?

Derek: So I had this idea actually just last summer. We were running a course design institute here at the teaching center and we had several faculty who are really interested in doing podcast projects because I think I had shared Gilbert Gonzales’s Health Policy Radio podcasts with them. And several faculty started thinking, “Oh I could really use this in my teaching,” and I thought, “This is great, but if we have Vanderbilt undergrads, especially, who are producing really interesting audio for class assignments all over campus, wouldn’t it be fun to curate that to have a podcast of podcasts?” So then I reached out to my colleagues at Vanderbilt student media and they’re like, “Yeah, that sounds great. We love to help students make media and share media with the world.” And so they were able to do all the heavy lifting on the technology piece, all I had to do was reach out to some faculty members and ask them to recommend some student produced audio for this and so this spring we launched VandyVox. It’s the best of student produced audio from all over campus. It’s a bit of a fudge, right? Like if some student has a Sports Radio Podcast, we’re not covering that. But if there’s an academic component to it, if it’s curricular or co-curricular, we’re happy to feature it on the podcast. And so it kind of serves two purposes. One is to kind of shine a spotlight on some student work, show this great stuff that our students are doing to provide some inspiration maybe for faculty and students to have students engaged in this kind of work. In the show notes for each episode, we have some background information about what the assignment was, or how the faculty members worked with students around this. So there’s a faculty development piece to it as well. But it’s been really fun to see what students are doing all over campus. You know, I highlighted some Health Policy Radio piece, we had a student from an anthropology course on health care politics. She created a 10 minute speculative fiction audio story…

Rebecca: Oh, cool.

Derek: …dramatized it as her project where she kind of imagined what would happen in the future with gene editing and baby selection. It’s just a really great sci fi kind of look at the course topic. Well researched, right? Like she turned in an annotated bibliography with all this so it’s all kind of backed up by the latest research. We had law students who were doing podcast episodes on immigration and refugee law talking to some immigrants and refugees. For that audio. We have Robbie Spivey in our Women’s and Gender Studies class teaches a course called Women Who Kill [LAUGHTER]—which is a great name for a course—and so she had her students do kind of true-crime podcasts about women who kill and how we make sense of that as a society. And then our last episode of season one, which came out recently, featured some work by Anna Butrico, who was a senior here last year, an English major. She did her senior honors thesis on podcasting and kind of connected it to ancient Greek rhetorical forms, which is really great. But her senior thesis had audio pieces to it. It’s hard to do a senior thesis on podcasting without creating a podcast and so we featured the audio introduction to her senior thesis, which I was really excited because Anna actually did a lot of work with podcasting while she was at Vanderbilt and her technical skills and her composition and storytelling skills are really strong. So it’s been really fun to kind of see something of a critical mass here at Vanderbilt around student podcasts and to be able to kind of highlight that a little bit. And I’m really excited, we’ve got some good stuff lined up for season two this fall as I’m kind of reaching out to more faculty and students about the the audio work that they’re doing. And again, part of it is getting started with a podcast is not hard. Doing it really well is still very hard, but the bar for entry is pretty low actually. And so if you want to have your students kind of move into a different modality—and again, you need to kind of be intentional about why you’re doing it and how it connects to your course goals—but podcasts offer a really great option for that. And I’ve just seening more and more faculty start to embrace this as a kind of creative output for students.

John: Going back a little bit, you mentioned that video project or the the video intensive project, I’m assuming those are the two MOOCs you have on teaching in STEM courses. I participated in the first one when it first came out.

Derek: Oh, that’s great.

John: It was a lot of fun, it was really useful. I didn’t do the second one. I think we both recommend those to a lot of faculty and encourage more people to take those. I believe they’re still running on Coursera?

Derek: Oh, yeah, we’re running one every semester. They’re not on Coursera anymore, they’re on edX. But you can always go to stemteachingcourse.org and you can find out information about those courses.

Rebecca: So you’ve already talked about the podcast that you’re working on and your book, the editing process and such that takes a long time, so you’ve got a lot of things in the cooker but we always wrap up by asking, what’s next?

Derek: Short run, we’re running a couple of course design institutes in the first of May and so that will occupy the next several weeks as kind of prep for that and those are always fun because I get to work with faculty. It’s on the theme of students as producers so we’ll be working with faculty around these creative nontraditional assignments. That’s always pretty exciting. There’s also—this is just an idea right now—but I keep running into faculty who are teaching with games or having their students design games, board games especially, as course assignments. I mentioned this text-based game that Alisha Karabinus uses and so I just keep finding examples of games and simulations that have a learning goal or learning purpose and so I’m hoping maybe this fall to put together a little one day symposium on campus on games for learning, games for social change, that kind of thing. I think that’d be a fun space to explore. And the other thing that I’m seeing—and I talked a little bit about this in the book—is this move towards active learning classrooms. I mentioned I like to walk into a classroom and see wheels on the chairs, because we can move them around and make them do what we want. The affordances that our classrooms have really matter for the choices we make as teachers. And so classrooms that are designed to facilitate small group work, student collaboration, active learning, this is a strong trend in higher education. I’m really kind of shocked how even from like three years ago, where I was having to tell people about this idea for the first time, now and here on our campus, our campus planners have decided this is a standard classroom configuration going forward. And so I see a lot of campuses moving towards active learning classrooms. Again, digital and analog technologies that support learning and so I want faculty to use them in intentional ways. And so I think we’re going to be doing a lot more with active learning classrooms here on campus, probably starting a learning community on that in the fall and I’m excited to dig into that work too.

Rebecca: Sounds like a lot of exciting things going on.

Derek: I try to stay busy. [LAUGHTER]

Rebecca: It’s going to be hard to keep up with all of them.

John: Well, we appreciate that and we’re looking forward to the book coming out.

Derek: Awesome.

Rebecca: Well thanks so much for joining us. It’s been really exciting and I know we all have a countdown now.

Derek: Thanks so much for having me on. This has been a really fun conversation. I’m happy to get the chance to share a little bit.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kim Fisher, Chris Wallace, Kelly Knight, Joseph Bandru, Jacob Alverson, Brittany Jones, and Gabriella Perez.

80. Self-Regulated Learning

Most students arrive at college with serious misconceptions about effective learning strategies. In this episode, Dr. Linda Nilson joins us to examine what we as faculty can do to help students develop their metacognitive skills and become self-regulated learners.

Dr. Nilson is the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. She is the author of many superb books, book chapters, and articles on teaching and learning. In this episode we focus on discussing one of her books: Creating Self-regulated Learners: Strategies to Strengthen Students’ Self-awareness and Learning Skills.

Show Notes

  • Linda Nilson—Director Emeritus of the Office of Teaching Effectiveness and Innovation (OTEI) at Clemson University
  • Nilson, L. (2013). Creating Self-regulated Learners: Strategies to Strengthen Students’ Self-awareness and Learning Skills. Stylus Publishing, LLC..
  • Nilson, L. (2014). Specifications Grading: Restoring Rigor, Motivating Students, and Saving Faculty Time. Stylus Publishing, LLC.
  • Professional and Organizational Development (POD)—Network in Higher Education Conference
  • Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6), 1121.

Transcript

This transcript has been edited to improve readability.

Rebecca: Most students arrive at college with serious misconceptions about effective learning strategies. In this episode, we examine what we as faculty can do to help students develop their metacognitive skills and become self-regulated learners.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Today our guest is Dr. Linda Nilson, the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. Dr. Nilson is the author of many superb books, book chapters, and articles on teaching and learning. Welcome.

Linda: I’m very honored to be here. Thank you.

Rebecca: Today’s teas are…

John: Are you drinking any tea?

Linda: Yes, I am drinking tea. I am drinking Lemon Lift.

Rebecca: Oh that sounds like a great way to start the day.

Linda: It is. It’s a very good way. Well, I also started it with coffee, but… [LAUGHTER]

John: And I’m drinking Ginger Peach Black tea.

Rebecca: And I have my Golden Monkey tea today.

John: We’ve invited you here today to talk about your book, Creating Self-regulated Learners: Strategies to Strengthen Students’ Self-Awareness and Learning Skills. Could you define what it means to be a self-regulated learner?

Linda: Yes. Self-regulated learning is the conscious planning, monitoring, and evaluation of one’s learning for the purpose of maximizing it. That’s a very fancy way of putting it. It’s that voice in your head that asks you questions about your learning as you’re involved in some sort of learning task, questions like, “Okay, I’m going to do a reading now, what strategy works best for me?” Now you just might brush over that because you’ve done readings of this type a dozen times, a hundred times, whatever, but you’ve asked yourself that question along the way. “What’s my best strategy? What kind of a task is this? And monitoring: are my strategies working for me? Am I getting it? Can I paraphrase the last couple of paragraphs that I just read?” It’s a reading thing, but it works also in lecture. And then at the end, you evaluate yourself. “Well, let’s see. I had a goal, being able to recite five main points from this chapter, let’s see if I can do it,” without looking at the chapter of course. [LAUGHTER] So you evaluate your abilities, you evaluate your strategies. That’s really what it’s all about and it involves a great deal of talking to yourself.

Rebecca: So how did you get interested in talking to yourself? [LAUGHTER]

Linda: Well, I heard voices. [LAUGHTER] Well, how did I get interested in this topic? Actually, it was an accounting professor at Clemson that got me interested in it. This is 2007, right, this is a long time ago. And so she said to me, “What about giving a workshop on self-regulated learning?” In my head I said, “Huh? What’s that?” I’d never heard of it. And so I decided to go find out about it and it took me a few years to really get a workshop together on it and I decided, “Gosh, this is wonderful. This is learning how to learn. This is familiar to me,” because I’ve been talking to myself for years. [LAUGHTER] So I thought, “Okay, I’m not crazy. This is a learning strategy, a major learning strategy, one that you can use throughout your life.” And so I gave the workshop, I started giving workshops, like at the POD Network Conference—which is made up by people like me who go to this conference every year—and then I decided—well, I didn’t decide—a book publisher came up to me and said, “Please write a book on this, I will publish it.” Since I was in love with the topic anyway, I decided to do it. And so I did and delved into it deeply.

John: As you’ve described it, it sounds like part of this deals with improving student metacognition, but you note that it goes a bit further. Could you talk about the additional aspects of it?

Linda: Metacognition is the cognitive part of self-regulated learning, which is a major part of it. However, there are a couple of other elements to it that I don’t know that you could say are really focused on cognition. There is the emotional element to it, which involves getting yourself to be motivated and interested in the topic. Remembering, reviewing what your professor told you about the relevance of this topic, and thinking about it yourself. We can motivate ourselves, we can reframe a task for ourselves, and we can certainly reframe what is going on in terms of a learning experience. That’s a major, major part of it. The emotional part of the end is: “If you didn’t reach your goal, what do you do about it?” Do you give up, walk away, and say, “Well, I wasn’t born to do engineering,” or whatever the topic is. No, what you should say is, “Let’s try another strategy. Let’s look into possible strategies.” As instructors, we need to familiarize students with various strategies because they come to us—I like the phrase—“as feral children” in terms of the life of the mind and what they know about learning. We don’t have cognitive psychologists—unfortunately—teaching first grade or fifth grade, and so we need to equip them with how their mind works. There is one other element, a physical element, and that involves planning, monitoring, and evaluating your physical setting, where do you study best. If doing a reading or writing assignment, is it in a coffee shop, or do you have to essentially be in a soundproof booth where you don’t have any stimulation? How much coffee should you have? Or tea? [LAUGHTER] What kind of an environment should you set up for yourself—perhaps putting your digital distractions in another room. How should you schedule your breaks? Other things that you might want to consider is the amount of sleep that you have had because that can be a very important element of learning and writing. Some people study better to mild background noises as long as they are familiar music they’ve never heard before. You’ve got to try out these different things and find out your best setting.

John: In your book you describe how you became a self-regulated learner. Could you relay that story?

Linda: Yes, it was based on fear and terror. [LAUGHTER] As a child, I went to a private Catholic girls school—great education, but not in the sweetest of ways. From about fifth grade on, we had what was called recitation every single day in English and history classes. The nuns would ask a question and would randomly call out students’ names—we were in small classes so it wasn’t an absurd thing—and we had to get up and we’d better have the answer to the question. Now, not all the kids did, but I needed to be Little Miss Perfect because I wanted to get into college. Somehow I thought that the answers that I gave in fifth and sixth and seventh and eighth grade would or would not get me into college. So I learned to quiz myself while I was reading chapters or essays. Constantly quizzing myself. And another thing we had to do, usually later in the day, was called “exercise period.” We had 35 minutes to write an answer to an essay question that was generally related to the readings. And we couldn’t look at our books. It was out of desperation that I was trying different strategies so I could perform very well every day, and I was really quite successful. So when I was reading about self-regulated learning, I said, “My God, I was doing this as a child.” That’s why it sounded so familiar to me.

John: But that’s not the experience I think that the students we have entering our colleges have.

Linda: You are so right. You are so right. And we don’t do this to children anymore, okay? There is a good side to it—there really is—because you do buckle under and get very serious about your homework, very serious about your studies, or you look like an idiot the next day. Anyway, the sport I’m trying to think of is that sport where you get the ice or any kind of ice particles out of the way.…

Rebecca: Curling?

Linda: Curling! Curling, yes, we are curling teachers and curling parents. We try to clear the way for our students. We don’t want to put them in stressful situations. We don’t want to ask Johnny to read if Johnny might not be able to read—and I mean, get up and read well—and the problem with this is that students are denied the opportunity for achievement. And there is no achievement without the possibility of failure, there just isn’t. So students have no idea what fear and terror in school might be. There’s bullying and all that, but I mean from the learning experience. So no, they don’t have any kind of the experiences that I had.

Rebecca: How do we start coaching students then to become self-regulated learners if they’re coming out of this really different environment that’s much more supportive and doesn’t allow for failure seemingly?

Linda: They start failing in college. We’re still sort of, you know, curling them a little bit, but they are really facing a much greater challenge. They get insecure really quickly because they’ve been told how special they are and how smart they are, and then they begin to question that, because they’re not doing as well as they were in high school, where you could get an A relatively easily. Now, “Oh my, it can be really hard,” and then they start getting C’s, and then you have their attention. That’s a way that you can tell them that, “There are ways that you can get A’s, you did not learn how to study, here is a way to learn.” Does it involves a sort of effort? Sure, but it’s really just talking to yourself and deciding what strategies would be best for you, testing out strategies, seeing how they work, and you will be more successful. And there have been studies of students—like in developmental courses—that show that the students who are struggling the most tend to know the least about self-regulated learning strategies and start to do better if they use these strategies.Of course we’ve got to get them to use the strategies, we’ve got to explain these strategies. It can be life changing for them in the most positive way.

John: I think part of the issue is that faculty generally haven’t been taught these strategies themselves. They somehow found ways to be successful, so they become self-regulated learners, but faculty are the exceptions. They’re not the typical student, and they’ve never really been trained to teach students how to become more effective learners, in part because they never learned that directly themselves quite often. What can faculty do to be more effective in this way?

Linda: Well first of all, faculty have to realize that they’re the weird ones, and everybody else is normal. [LAUGHTER] So we have to stop projecting our learning abilities, our strategies, our interests in the life of the mind onto everybody else. We have to not only sell our material, but we have to equip students to learn our material. We don’t want to do that. We say, “They should know by now.” Well, guess what? They don’t. So what are you going to do about it? You’ve got to start from where they are. Teaching students learning strategies takes a couple of sentences every class. Now, if you really want to get into self-regulated learning activities and assignments with them, that might take a few minutes per class period, but you don’t have to do it every class period, and a lot of self-regulated learning activities can be homework, in which case they take no class time at all. This is so easy to do. This is why I think faculty have really been attracted to my book and why I’m asked to speak on it so often, because there are so many little things you can do that don’t take away from the content at all—rather, they reinforce the content—that make this huge difference in the performance of most students. You can’t always bring everybody along with you. There are some people—some students—who’d just as soon shoot themselves in their foot, but most do not. They find these activities so easy to do. They don’t take a lot of time and they get to know themselves and start doing better, so students don’t complain about this.

Rebecca: Can you describe what a couple of those activities might be?

Linda: Sure, absolutely. Well, let’s consider the different parts of the course and I’ll just give you just a few, some of my personal favorites. For starting a course for instance—starting and you can also end it with these sorts of activities—but one of them is a goal setting activity. You can assign this as homework, you can have students do it in class with students write on “How I earned an A in this course.” Now, you would be surprised and students will be surprised—C students and B students will be surprised—that they know what it takes to earn an A in a course, and they will come up with, “Well, I’ve got to come to class every day, don’t I? And in class I can’t fiddle around with my mobile device, and I have to start a paper sort of early and I have to keep up with the readings.” For many people, writing this down is goal setting for them. They think, “Well, you know, maybe I could do this, maybe this isn’t so absurd.” If you make a discussion out of it afterwards, the A students will say, “Yeah, I do these things, it’s not unrealistic.” And then the C and B and F students will say, “Well, let’s give it a whirl.” Then at the end of the course, you give them another little essay assignment, “How I earned an A in this course—or not.” [LAUGHTER] True confessions time, right? And so students assess how well they met their goals. Goal setting is definitely a part of self-regulated learning, the planning and then self-evaluation at the end. Another thing that you can do is you can give your students essay questions. If you give an essay final—or have any essays on it at all—you can give them the essays on the final to take the first day. This will not take much class time at all, because students will know very little, or they’ll try to BS an answer. So they will try but they can be really quite wrong. Now at the end, for the final, they correct their answers, and then rewrite these answers given the knowledge that they have gained throughout the course. This can be really interesting for faculty—for not just faculty… well, it can be interesting for them, too—because they can see exactly what they learned. So it is a measure of learning. Faculty will never get that comment on the student evaluations saying, “Well, I didn’t learn anything in this course.” Never again, that’s gone. So anyway, those are a couple of things that you can do. Little assignments you can make on the readings. Little reflection exercises like, “What did you think was the most important point in this reading? What surprised you the most? What connections can you make between what you read and your prior knowledge, what you already know? Or to your life? Or your emotional reactions to it, if the material is amenable to that?” So those are little reflections you can give on the readings. Another exercise, a self-testing exercise, is called “read, recall, review.” This is the best way to do reading. Forget about rereading, that’s what students really do… It’s really a waste of time. What students should do is to read a portion of the chapter or the whole chapter, put their notes away, close the book, and then recall as much as they can and write it down. Then they should go back and look in the chapter for what they forgot and what they might have gummed up. And they know that, “I didn’t really get that point.” And so they go back and look at it, and then they recall again. Read, recall, review. Studies that have been done on this showing it is so much more effective than rereading. It really doesn’t take that long, and then you actually have the material in your head, even in your long-term memory. You get retrieval practice, you get deliberate practice, so there’s nothing as good as testing yourself except—well, the nicest thing we can do for students is to test them. In lectures—I should say mini-lectures—it’s a good idea to have students do this. You stop, let’s say, every 15 minutes or so and have students do the same thing. Write down everything that they can recall, and then work with their neighbor to fill in the blanks—their own blanks—and ask any questions they have. First they ask their neighbor their questions, and then they ask you. This doesn’t take very long at all, maybe five minutes, but then you know that the students got it and can remember it. Again, most effective… studies done on this, too. So this makes students aware of their learning or their lack of learning. You can give students what are called active learning checks. You give your mini-lecture, and then you stop—and by the way, you can warn students you’re going to do this so they’re listening—and ask them, “Okay, what are the three major points in my last mini-lecture that I talked about in the last 15 minutes?” Then they write those things down—and it could be two things or four things depending—and turn them in. They don’t really have to turn them in, but you know, you might want to see them yourself. Then you reveal the three most important points, and they monitor and evaluate their learning skills. Now, students are motivated to want to learn how to listen to you, so they want to improve. According to a study that was done, they improve really quickly. The first time, 45 percent of students got all three points correct. By the third time, 75 percent of the students got these correct. Remarkable progress, really remarkable. Then there are meta-assignments. In a problem solving field like chemistry or math, we are denying students learning opportunities when all we do is mark the wrong answers as wrong or incomplete and then drop the subject. Students should be able to correct their mistakes to get half the points back, let’s say. In other words, they’re going to learn how to solve that problem if it’s the last thing they do. [LAUGHTER] Again, you give them some sort of an incentive, then they learn. There have been studies on this technique as well. It’s extremely effective. And students can learn not just from you, but in peer groups. Peers can help each other very effectively. There is a wrapper—they’re called “wrappers”—for an exam, a reflection that students do after they get their exams back where they answer questions like, “How did your expected grade compare with your actual grade? How do you feel about that?” So they have to look at the exam and your feedback. “How many hours did you study? Was that enough? What did you do while studying? Might you want to change your strategies? Why did you lose points? Were there any patterns that you see here? How are you going to study more effectively for the next exam?” This has been life changing for students because they’ve never thought about this before. They’ve never really looked at their exams, their mistakes. They drop them too, right? They don’t want to see what they did wrong. Yet these are the best learning opportunities possible, and they will remember them. We remember our mistakes, we learn from our mistakes, and it’s sad that we don’t stop and use those errors. These are just a small sample of self-regulated learning activities. I can give you many more. [LAUGHTER]

John: And there are many in the book, which we strongly recommend to people.

Linda: Yes, yes, yeah…

Rebecca: A lot of what you’re talking about seems tied to growth mindset as well.

Linda: Exactly, and this creates, this generates the growth mindset because students learn that they can learn, they can do better. Otherwise they feel like their learning is like the weather. “Maybe it’ll rain on me, and maybe not. [LAUGHTER] There’s really nothing I can do about it. Because it’s all about you, professor, you are responsible for my learning, just like the fates are responsible for the weather. [LAUGHTER] And if I’m not learning, you’re not a good instructor, or you’re pitching the material over my head, or your teaching strategies are wrong.” And so everybody else gets blamed, and then they start to realize, “Oh, I can do this.” Now, this isn’t the best news for them in the world because then they have to start taking responsibility for their learning. And that can be, for some students, a hard pill to swallow. For other students it will be very empowering, and what we want to encourage in students is that sense of empowerment.

John: And that’s especially important, I think, in freshman-level classes, because students generally don’t come in with that type of mindset. They’ve often been able to blame it on the teacher and do things over and over again until they get the grade they want or get the extensions and so forth with a focus on self-esteem in many classrooms.

Linda: Oh, yeah, self-esteem without achievement.

John: But it’s an adjustment. So if they come in with a fixed mindset, and they’re confronted with failure, it’s pretty easy to give up. So we need to encourage students, I think, to see failure as a learning opportunity as you’ve mentioned. As instructors, I think we have to somehow convince them of that, because they don’t come in naturally picking it up, but the techniques you’ve mentioned are very good for that.

Linda: You know, our whole society makes them feel they’re not responsible for their learning. Look at what happens in K-12. Students have to take standardized tests and if they don’t do well, who gets blamed and who suffers? The teacher and the school, and that’s nuts. In the final analysis, we teach ourselves. We are responsible for our own learning. Good teaching can make a big difference because we can be motivated or unmotivated by teaching. We can acquire learning strategies through teaching. So it’s not that students are just left adrift on their own, we do have to help them. We do have to put them in learning experiences where learning becomes attractive for them, or you can’t help but learn, right? But they’ve got to pick up that learning and run with it themselves.

Rebecca: So you mentioned the idea of encouraging students to see learning and the self-regulation as empowering. What about those students who are a little resistant to that because it’s surprising to them that they’re not getting it and they’re failing and that it’s going to be more work? What are some things that we can do to encourage those students to see things a little differently?

Linda: Yes, first of all, if they’re failing and they subconsciously want to—it happens, it really does —[LAUGHTER] there is not a whole lot you can do about it. They might need some counseling and they might need to get some help from professionals like psychologists. But again, it can be difficult for students to realize that the ball is in their court because it’s a whole different gestalt for them. The only cure for that is success—a little bit of success—where they start doing a little better, let’s say, on the quiz on the readings or they start being able to solve more problems. That’s really the only cure. And we are assuming that they want to be successful. Again, if they prefer failure, then they are responsible for their own failure.

Rebecca: Right, they’re the ones that are normal and we are not, right? [LAUGHTER]

Linda: Exactly.

Rebecca: Maybe that should be the refrain of this interview, right?

Linda: Yes. They are the normal ones and we are strange. And we always have been strange. We were the strange kids in school, too. [LAUGHTER]

John: In your book you mentioned the Dunning-Kruger effect as being a barrier to some students, that the students who don’t understand things as well often overestimate their understanding. How can we overcome that?

Linda: Self-regulated learning helps when we give them activities and assignments where they do self-evaluation, because the only way to learn self-evaluation is through practice, practice with feedback. And that feedback doesn’t even have to come from you; it can come from peers or a computer program. We don’t give students a lot of practice in self-evaluation, and they certainly haven’t had much of it in K-12l. But the nice thing is that when we have students look back to see if they met their goals, or to evaluate their study strategies, or to assess their mistakes and the reasons for their mistakes, it makes all the difference in the world. After low-stakes practice, you can introduce higher stakes self-evaluation assignments and see more savvy self-evaluations.

John: What recommendations do you have for faculty who’d like to start building more self-regulation? Are there small steps that faculty could take to get started on this path?

Linda: Absolutely. There’s a sense in which most of the assignments, most of the activities, are little things. Here’s a little thing you can start off the course with, I was talking about essay questions, but you can just have students do a little reflection the first day and then again on the last day about the subject matter, as in, “What do you think chemistry is? Why is it a science?” You can find out a lot about students’ misconceptions just by looking at these reflections. And then of course, they’ve hopefully corrected a lot of misconceptions by the end. This could take like all of five minutes the first day. There’s so many little things. Here are some ideas for experiential learning. It’s so easy for students not to make a connection between a simulation, an interesting role play, a service-learning experience, or field work to the course. So it’s important that whenever you do an experiential assignment or activity, students reflect on what they are learning—for a simulation, to look back and explain what their goals were, to evaluate how well they met their goals, to assess their strategies, to explain how their strategies changed and their responses to other players. It’s very important that students become conscious of what’s going on in their heads. Only by becoming conscious can they remember the strategies? [LAUGHTER] And then they can write them down and articulate them. You can have students do short papers associated with papers and projects where they record, while they are doing it, the process they are following. If you’ve given them a process to follow, they even have a skeletal outline of what they should be doing. This is a place also for self-evaluation. If you have students do a revision, oftentimes you give them feedback on what they should revise, and they may or may not read your advice. So you can have them paraphrase your feedback back to them and write out their goals for the revision. What are they going to do? What are their strategies for revision? These are just little things. Students start to realize the value of this. And again, this is an assignment where you can’t screw up. It’s not a test, it’s just a reflection of what’s going on in your head. Students like to learn about themselves. And this is like the reading reflections. This is no stress. How do you mess this up? It takes less stress to just write an honest answer than it is to make one up that sounds credible. [LAUGHTER] I want to make faculty aware that the activities don’t have to be graded at all. The assignments don’t really have to be graded. You “grade” them pass-fail. Students pass just by completing the assignment. Let’s say, you had them answering three questions, three reflections. Did they answer three reflections? Is it vaguely on the chapter? Okay it’s not about football, it’s something about what’s in the chapter. [LAUGHTER] And did they meet the length requirement? It’s always a good idea to give length requirements on these reflections because for students, length means depth. So if you ask them to write a minimum of 150 words, you know, they’ll tend to do that. Those who don’t fail.You don’t count every word that students write. You eyeball the reflection. Essentially you are “grading” pass-fail at a glance. It doesn’t take much time. Plus, it gets students to do the readings in the easiest way and most productive way for them. It’s all about them, and it’s not about us. We just have to hold them accountable in some very quick way. Even the longer assignments that you might associate with a paper or project can be graded pass-fail. You have to make them worth some points if you’re still on a point system. But there are alternatives, that’s what specifications grading is all about. You don’t have to use points. In any case, you do have to at least eyeball the reflections and give some value in your course however you are grading. That communicates to students that this is important to you, that you put value on this meta-assignment or assignment wrapper, as you might call it. The same thing with the post-exam wrapper, these reflections on this exam. You make students do it because it’s worth 10 points if they simply complete it and hand it in—even though it’s for them—and they will realize right away that there is some value to this. Again, for some students, it will be life changing in the most positive way. And they will start to realize the way that they’ve been preparing for or taking exams may not be the best. They will realize what they tend to do when they’re taking an exam, such as to misread the question, or to be careless, or to not budget their time, or to not really thoroughly study all the material. Cramming is not very effective. You don’t have to spend time grading these exercises or giving any feedback at all. They can give themselves their own feedback. If they did it and they get the 10 points, okay, that’s plenty of feedback for them. They did it. You regard it as their meeting the requirements of the assignment.

John: And this is a topic you cover in another book on specifications grading, which is also another book we’d like to recommend. We’ll include a link to both of those on the show notes.

Linda: The title of the book is Specifications Grading: Restoring Rigor, Motivating Students—it’s been found to be motivating—and Saving Faculty Time, saving you time. If there’s one thing we don’t have, it’s time. Time is really more precious to us than money. Otherwise, we’d be working in some venture capital firm or something. [LAUGHTER] But time is really quite a precious thing for us. So, in terms of these sort of assignments, for self-regulated learning assignments, they’re all what we’d call “specs graded.” You set out the specs, they’re very simple, and you just grade them pass-fail.

Rebecca: I think pointing out how it doesn’t have to be complicated for faculty is important because I think we all want students to learn. We all want them to be self-regulated learners.

John: We all want to give students feedback, but we don’t want to make it impossible for us to keep up with our work.

Rebecca: Yeah. Or feedback that’s going to get ignored anyways.

John: Right.

Linda: You’re worried about students reading the feedback, and our feedback is valuable. We’ve given it, we’ve taken the time, and so you make them paraphrase it back to you. And this could be a learning experience for us because we might be “misread.” Students might not understand something that we’ve said. Awkward—that’s my favorite one—a sentence structure is awkward. What does that mean? That student didn’t set out to write an awkward sentence. That in itself will not help them because they don’t know what you’re talking about, and this is most unfortunate. But again, it’s a learning experience for us and we can learn to express ourselves somewhat differently. Too often, students get back a paper from us and look at the grade, read the paragraph at the end of the paper, and put it in their “circular file.” They dump it. They don’t read that feedback, so how are they going to get better? So paraphrasing our feedback back to us can be a very valuable exercise for them. And you can let them gain back some points for it. I just think that faculty should look at themselves as responsible for helping our students learn. They don’t come to us with those skills. We can be the finest instructor in the world, have the most interesting classes, hold their attention, and motivate them, but if they don’t know how to process that material in their own minds, it’s all for naught. Now, maybe, hopefully, seniors have learned to learn their material along the line. And by the way, there can be different learning strategies for different subject matter. There are different self-regulated learning activities and assignments for problem solving mathematically-based fields, and different ones for the social sciences and humanities. There can different kinds of assignments, different kinds of readings, actually different kinds of lectures. So we have to respect that. But we have to become conscious of study strategies, learning strategies, our strategies, and other strategies that are out there. But self-regulated learning strategies, to my mind, they’re the shortest distance between two points. Shortest distance between ignorance and learning because it’s all going on in your head, and it’s so powerful. The value of it to students becomes evident rather quickly.

Rebecca: And it’s a skill that can help through a whole lifetime, not just while they’re in college and I think helping students realize that is also really valuable.

Linda: Like no other generation before, these younger generations are going to have to learn to learn on their own. They’re going to have to keep up with their field, whatever their field is, and they might have to—will very likely have to—pivot into another field because their first field might run its course. They’re going to have to learn on their own. They aren’t going to have employers holding their hand. Not at all. They’re probably going to have to learn online, where you really are responsible for your own processing, more so than you might feel let’s say in a face-to-face class, and for your own motivating as well. There needs to be more motivation than simple fear that you will go hungry and won’t be able to get a job. [LAUGHTER] Yes, students are going to really have to learn how to learn. If they consider that a bitter pill, that’s too bad. This is reality, this is life, and most of them have not learned that life is hard. Many of them are wondering where their next meal is coming from, but a lot of students have not. Students need to learn along the line that life is not easy, that nobody does curling on their path. And they will face challenges, but if they have the strategies for facing these challenges, no problem. They needn’t be paralyzed. They needn’t freeze.

John: You foster some really good advice and I think our listeners will appreciate this and it’s really powerful.

Rebecca: Yeah, I agree. We’re all wondering, what’s next for you?

Linda: I’m actually Director Emeritus. I’m actually retired from Clemson University. But you know how academics are, [LAUGHTER] they don’t disappear, they just sort of like fade away. So I’m trying to ease into retirement because it’s not an easy thing to do? Not when you love what you’ve been doing. But I have sworn off writing books. That’s progress! [LAUGHTER] I’ve written some articles and chapters in other people’s books, so that’s fine. And I’m still traveling to give keynotes and faculty workshops. That’s hard to give up because it’s interesting to go somewhere else, somewhere new.” And I still give webinars and podcasts. But eventually I won’t be doing that anymore. Ultimately I want to work with animals. I do love animals but I’m still busy doing this and still loving doing this, but also loving just as much not having to do bureaucratic tasks for the university [LAUGHTER] and not having to stay up until two in the morning doing my email. When I’m traveling, not have to worry about, what’s going on back at the office. So I’m not complaining about retirement. I really like where I’m at right now, but I know that I will eventually fade into the sunset. That’s okay because then I’ll reinvent myself.

Rebecca: Sounds like some self-regulation was going on there, I’m pretty sure. [LAUGHTER]

Linda: Yes, I’m trying, I’m trying to retire but not too quickly.

John: Well, we’re glad you haven’t, fully yet.

Rebecca: This was really great. Thank you so much for spending some time with us today.

Linda: Well thank you for this opportunity. I hope that I have helped some faculty members out there to help them help their students to achieve more, because again, we all do want our students to learn. We’re all in love with our material, it’s worth learning, and we just have to help our students do that. So thank you ever so much, and thank all of you listeners for listening.

Rebecca: Thank you so much.

[Music]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kim Fisher, Chris Wallace, Kelly Knight, Joseph Bandru, Jacob Alverson, Brittany Jones, and Gabriella Perez.

68. Mobile Music instruction

There are apps for just about everything but choosing when to embrace them for instruction needs to be a careful decision. In this episode, Trevor Jorgensen joins us to discuss how the decision to use mobile apps in music instruction is affected by where students are developmentally, convenience, cost, and other factors. Trevor is an Assistant Professor of Music and the Student Learning Outcomes Assessment Coordinator at SUNY Oswego.

Show Notes

Additional Resources

Transcript

John: There are apps for just about everything but choosing when to embrace them for instruction needs to be a careful decision. In this episode we consider where students are developmentally in a discipline, convenience, cost, and other factors on the choice of supporting music instruction using mobile applications.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

Rebecca: Today our guest is Trevor Jorgensen, an Assistant Professor of Music and Student Learning Outcomes Assessment Coordinator at SUNY Oswego. Welcome, Trevor.

John: Welcome.

Trevor: Hey.

John: Our teas today are:

Trevor: Pure Leaf Peach Flavor tea.

John: Iced tea.

Trevor: Iced tea. Sorry. Thank you.

John: …and Rebecca.

Rebecca: I have Chai today…

John:…and I have black raspberry green tea. We’ve invited you here to talk about how you’ve been using mobile technology in applied music instruction, specifically related to performance. Could you tell us a bit about that?

Trevor: Yeah, I think one of the most important things is to decide when to use technology where it benefits as opposed to be either a substitute for something you should learn in a different way, or whether you should implement it because it’s just convenient. Sometimes when it’s convenient, though, it takes away some of the learning behind it, and I’ll get into some of that later. So I think that’s the first thing. I have to make that decision on every app that I use, or every device that I employ. So, one of the main things that I do is decide what’s best for the student and what’s best for me. And a lot of it does substitute for other things, but in the end, it can’t. For instance, if I do a collaborative piano thing with my colleague, there’s a lot of tools that we’ll talk about today that will substitute for that collaboration. Except for, I cannot phrase with the app in the the same way that I can. We can’t make decisions on a moment’s notice, like I can with another musician. We can’t have talks… at least I shouldn’t be talking to my apps… about what the best way to do something is and then make that interpretation or change colors of stuff. But what it allows me to do is, instead of having a more rehearsals with somebody that’s really busy, it allows me to rehearse by myself. And then therefore, save some time when we’re actually in the rehearsal to talk about those musical decisions as opposed to just to try to learn notes or trying to learn one anothers parts.

John: What are some of the apps that you use with instruction or performance?

Trevor: One of my favorite ones is Amazing Slow Downer, and I use this for both jazz and classical, being a saxophonist, and also a multi-reed player…I play clarinet, oboe, and bassoon. I like to use it in every circumstance I can. And it’s really a great app because even though there’s things like YouTube where you can now click over the three little dots in the corner, and you can slow down to 25%, 50% and 75% and speed it up if you so desire. This tool is just something that I always have with me and I can always use. And I’m also more familiar with it, it allows me to do multiple things. So let me give you example if I can play musical selection here. The first two are not going to be on the music slower downer, it’s going to be on Amazon Music. But I want you to listen. These are Brahms’ Clarinet Trio. I performed this with a colleague and somebody from Symphoria recently. And one of the things you’re doing as a musician is you have to interpret the history of music, and that includes listening. So if I listened to multiple recordings of something, you know, we always ask “Why are there a thousand recordings—might be overstated—but of Beethoven’s Ninth Symphony?” That’s because the composer had a certain thing in mind. And then the conductor who prepares the music, and the individuals in that group have different ways of interpreting that music and every one is different. But for a lot of younger students, and sometimes myself when it’s really advanced players, I’m not exactly sure what they’re doing, either articulation wise or what they’re doing phrasing wise. And the slower downer allows me to think about what they’re doing and compare all these great artists together. So I’ll do multiple recordings on something like this Brahms’ clarinet trio in A minor and have my students listen to it. I’m going to play one of them for you, and then I’ll play another one, and then I’ll play the third one. And then the third one I’ll slow down to show you how we can listen for different articulations. You may or may not tell the difference between the three, some of them are obvious, like there’s a tempo difference between them. But as far as the phrasing and the articulation used by the cellist, and by that clarinetist, and by the pianist, is something to think about. What do I want to sound like? It’s great to imitate people, but it’s also great to make decisions between all these different things you’re, in a way, sourcing and decide what you want to do and why you’re doing those musical decisions.

Rebecca: Before we jump into listening, for our audience who maybe is not as familiar with the music terms that you’re using? Can you clarify the difference between phrasing and articulation?

Trevor: Sure. Great. So articulation is the front of the note, and thank you for your much for that. So as a saxophonist, it’s how I tongue the note, if I put an emphasis through air or through my tongue. Another word for that is “attack,” how it attacks into it. For a string player, it might be the direction—up bow or down bow—how fast they’re moving the bow. For the pianist, how they voice the chords and are there some fingers stronger than the others. A third comes out of the chord and the fifth comes out of the chord. For a chord you have multiple, you know 1-3-5s, or the seventh of the chord. And each one of those, as a pianist, when they’re putting down multiple fingers, they can decide which ones are the most important. Whether it be the melody, whether it be the bass part, whether it be the inner harmony. So, from a pianist’s point-of-view, they can listen for that, or how the touches on the piano, how much they pressurize it, or if they use pedal in this area that’s not dictated by the score exactly. So phrasing, the best way to describe it is like minute dynamics. Or just using what we normally do. We, as human beings phrase, if you like somebody you change your voice, or if you get excited like, “I have go to the bathroom,” you don’t think, “Hey, I should elevate the dynamic of my voice. I should make it sound strained, so that I show the excitement of me having to go to the bathroom.” But in music, we have to think about that. We have to think about where, based on the harmony that’s below the music, how much am I going to emphasize that note or more importantly, maybe even growth of that note, or that musical phrase. So phrasing’s how people interpret the music beyond playing the right notes in the right place, and doing the pianoforte dynamics; it’s how they’re going to go to a note and resolve a note. And not only that, they think about overall structure of a piece for phrasing and how everything fits together, if they’re amazing musicians. When we listen to amazing musicians—like the ones who found in these recordings, then what we’re going to do is interpret what they did. And then maybe in our own minds figure out why you think they did that, if that’s a choice that you like to do based on the harmony or just a different tone color they do. And that’s obviously easier to figure out when you slow down to 70% than it is when you’re listening to it at 100%. So before I play the examples, the actual artists will be in the show notes below in the description. First one is, I’m just going to name the clarinetist, Karl Leister, and this is them playing the Brahms’ Trio. And I’ll do the first 30 seconds or so, so you can hear how it sounds.

[KARL LEISTER PLAYING BRAHMS’ TRIO]

Trevor: The next example will be by the clarinetist David Shifrin, that trio that he’s in. I’ll play about the same amount of time for it.

[DAVID SHIFRIN PLAYING BRAHMS’ TRIO]

Trevor: The last one I’ll do on Amazing Slower Downer, the app. And we’ll actually play it in tempo first, since its Martin Frost as the clarinetist, playing the same piece.

[MARTIN FROST PLAYING BRAHMS’ TRIO]

Trevor: So what we’ll do now is we’ll actually slow it down to like 70% so we can actually hear a little bit better the articulations, the dynamics—as the artist gets louder, you’ll be able to hear it more individual. And we can even slow it down after that into something ridiculous like 25%, where you can actually hear the front of each note. It’s almost painstaking how slow it is, but interpreting different things, it’s interesting to see if they use a breath attack or a tongue attack, and some of them are so subtle, I’d have to listen to it 25 times without slowing it down where I can actually hear it at 25% right away. So let’s go back and do it about 70%.

[BRAHMS’ TRIO PLAYED AT 70% SPEED]

Trevor: Here’s 25%.

[BRAHMS’ TRIO PLAYED AT 25% SPEED]

Trevor: I’m just going to jump ahead a little bit so we get into the clarinetist and stuff. I’m just going to scroll ahead.

[BRAHMS’ TRIO PLAYED AT 25% SPEED]

Trevor: First of all it’s very unfair because I would never want to be heard at 25%—it’s just impossible to keep a phrase sometimes or keep a note perfectly that way—but what it does is for us is we can actually hear them the front of the note… we can hear the shape of that note even if it’s a quarter note if he—in this case it’s a he—de-crescendos or crescendos into it and we can look at the overall structure of the dynamics too. You can see if he’s actually growing on that note, if you’re taping the note as I said before… the overall picture or phrase too. I think 25% might be a little excessive, you have to find the happy medium based on the tempo of the piece. I think initially 25% for this one might have been a little painstaking at the beginning.

John: But for students, perhaps having the ability to slow it down to that level would be much more helpful than for someone with more experience?

Trevor: Right, there’s definitely that and then we’re looking into transcription later on too. There’s this great book called School for Cool: The Academic Jazz Program and the Paradox of Institutionalized Creativity by Eitan Wilf and I hope I’m pronouncing his name right. He wrote it in 2014 and he basically went to Berklee School of Music and then The New School in Manhattan. And he noticed that students were using this application even back then—and when I say back then is only five years ago, or four years ago, or three, because in the time of technology, that’s way back then—and they were actually interpreting, transcribing not only John Coltrane and getting his notes like I would have done when I was in grad school or undergrad, but also then listening to his articulation at the front of the note. So trying to make their tongue sound exactly like that slow, and then they’d be speeding it up. Also, in a real fast passage we use a fake fingering. What fake fingering is, is there’s a normal way of fingering D on saxophone which is like 123-123 fingers down and then there’s a way of playing the high D but not using the octave key so it comes out in kind of a muffled way, I could probably demonstrate it for you if you’d like.

[TREVOR PLAYS D NOTE 3 DIFFERENT WAYS]

Trevor: So you use three different fingerings for the same note, which I’d probably not do in classical, but for jazz, it adds a different color. A lot of people know certain people will play that way so that you can actually determine those notes when it’s really fast it’s hard to tell. Just that nuanced playing that you can actually interpret when you’re listening to John Coltrane, that I didn’t when I was that age of the students that he was, you know, surveying. But, slowing it down, it’s not something new. There’s another great book too, Thinking in Jazz: The Infinite Art of Improvisation by Paul Berliner who was published in 1994. He talks about how the old school players would get a vinyl record and they’d put their thumb on it and slow it down to like a third of the speed and then obviously transpose. Now they would be in the wrong key and way too low, and then they just transpose it up. So this technology, or using this technology, has been available as far as transcription. But if you did put your thumb on a record, it would probably distort the sound and the way that you can figure out articulation and you couldn’t figure out phrasing.

John: And the advantage of this is that students can preserve the pitch because all these apps maintain the pitch and you can hear the attack… just more slowly.

Trevor: This one is specifically and I’m sure a lot of things you can probably do this on YouTube. Also, a lot of my students have to in order to avoid buying a $15 app like this one is—at least when I bought it it was—they’ll find other sources, which I think is great. It doesn’t matter to me what they use as long as it does the same thing. I think on YouTube, you can do the same thing where you can loop it, and there is these three seconds or 10 seconds or whatever, and just loop it and here you can loop it. You can also save the file, save the section of the piece, call it what you want, and the next time you open up the app, it’s there. And you can also share your ideas with people. You can download stuff from things you buy on iTunes or you can use Spotify and there are other ways of importing into this app that’s really beneficial too. The nice part about this app is it can be used for classical and jazz. Most people use it mainly for jazz and transcription but I use it for everything and additionally you can actually slow your own self down which is always painful and hear that you weren’t phrasing that well or that your end of notes aren’t as tapered as they could be. And once you listen to amazing players you realize that they have control over everything. I’ve slowed myself down. I always know the differences between me and like the best person in the world, although I knew already but but it’s so evident when you slow it down to 50% and you hear the attack of your note is not as clear and not as shaped as the people that are just geniuses.

John: And you can also use that with students, though, to show them what they’re doing, right?

Trevor: Exactly. I mean, it’s a very revealing tool. So as far as students are concerned, I think it’s great tool for them but also can be somewhat depressing to hear yourself. It has to be a good balance and the student has to be at the point where we’re finding things not initially at the beginning, or maybe they’re even in their freshman or sophomore year, but at the end when they’re preparing for grad school or something like that, and it depends on the student as it does with every situation… what you can push them on and what you can’t. But, it’s just a great tool for that. So I also use the Amazing Slower Downer when I have a piece that I don’t have a—we’ll talk about Smart Music later where it is an intuitive accompaniment tool that takes away the saxophone or takes away or whatever it is you’re working on—but when it’s in pieces not in there or I don’t know the piece, I’ll find a recording of it and then I’ll play along with it. And this I can adjust a little bit of pitch if I need to and more importantly speed if I needed to get slower or I decided to take the piece at the end faster. Maybe now’s the time to talk about what the disadvantage of some technology is. I’ve got colleagues that don’t like using slower downers, not just colleagues but I’ve read articles and everything else, because you don’t have to go through the work and the persistence of doing so, and you should be able to hear it in the moment in the spot. And I think there’s something to be said for that. But I look at myself as one of the things where I get discouraged. I always think about one of my favorite reads, which is a psychologist by the name of Csikszentmihalyi and he’s got the flow theory, which is a book he published in 1990, which a lot of people are aware of. And number three of his flow theory is that there has to be a balance between the challenge and skills. For me, there wasn’t, my skills were low and my patience were also maybe lower when I was younger. [LAUGHTER]

John: And I think that’s true for most starting students. And a lot of people get discouraged when they don’t see that immediate progress.

Trevor: Yeah, that’s exactly it. So for me, I was able to use the tools that helped me a lot more. I painstakingly did it when I was my students’ age that was also because of the slower downers were $400 and I didn’t have the money. Now most students have phones but they use them for social media or actually using them as phones, or texting more likely. But now you can just download an app instead of buying a $400 machine for about $15, or you can go to YouTube and use that function or other tools that come with any kind of PC nowadays. Microsoft has their recording software and you can slow it down too. Not in such a certain way, but any of those tools are great too. Anyways I’ll slow my own recordings down of me playing along with other things. But the problem is trying to figure out that balance where I should be taking it to the tempo there I just took it or slowing it down. You just have to progress and move it forward so that you take it faster, faster each time. The thing about anything is once we do it more often, it becomes easier.

John: And the more they practice with this at a slow pace, the better they’ll be able to do it in real time as well.

Trevor: Right, and their ears develop immediately, faster and faster, and then therefore they can get to that point. I commend my colleagues and all the articles that I’ve read where you know, it’s better to do it that way. But for me, it was a wall, as it is for some of my students.

Rebecca: Sometimes it’s good to recognize when you are a beginner and that sometimes you need those training wheels just to feel like you got a little success.

Trevor: Right. That success as we all know it from pedagogy is that wields more success and it gets better and better every time. Let’s go on to a different app. Some of the obvious apps are metronome apps, every student can get a free metronome. I was excited in the mid 2000s when they dropped down to $20 as opposed to like $200, or the tuner is the same way, and then balancing that out. For me it’s I could buy a $20 tuner and put one on every one of my instruments but I always have my cell phone on me that’s what I love about it. I always have a tuner, I always have a metronome and I have no excuse, nor do my students. You know it’s a free app; you already own the phone. But there are some tuners and some metronomes that are better than others and I am not an authority, I don’t know every app that’s out there because there’s thousands. The ones that I know are the ones that I like… people have introduced to me and I’ll switch apps all the time if I find one that’s better. As far as the tuner or metronome it doesn’t really matter what my student brings in… whether she brings in the one that I prefer, like Cleartune for everyday apps, or if she brings in like TonalEnergy Tuner, but I’m going to talk about that one because it’s one that I found really interesting and it has a lot of functions on it. And it’s not only a tuner but a metronome. I use Tempo app for metronome primarily when I’m practicing but I’m getting into the TonalEnergy app for inside of it has also a metronome that I’m trying to investigate. So the app has so many different features to it, and I’ll show you and then maybe both of you can also chime in on what you think they look like, but…

John: I use the Cleartune app myself.

Trevor: Yeah, I loved that app until I saw this app. To give you a total side story, this one does analysis by waveform as it does it by another analysis and note names, too. This weekend, I judged a competition for high school students to get into all-county bands. And one of the other judges in the brass room said one of the students—and I don’t know the name of the student nor the judge—but the performance was really not great at all, where he actually pulled out his TonalEnergy Tuner to figure out what note the student was on to see if he could find out where he or she was was in the piece. That being the extreme of what the app is used for… nor do you want to use it for transcription because there are transcription apps where you plug it in and it’ll transcribe it for you. For playing our transcriptions—there’s worth to all those things—as far as actual transcriptions… a tool that you should have so that it ingrains in your memory and you have those phrases for jazz specifically. So this one here you can choose from waveform, you can do spectral harmonic analysis and note analysis. As you can see right now as we’re talking, sometimes I’m talking in tune and he gives me a happy face—it could be a she—gives me a smiley face, or if I’m really badly out of tune, there’s this little question mark with the thumb and finger or very sadness as you get way out of tune. But more importantly, as I do the waveform thing, it will record what I do.

Rebecca: I was also noticing that the color was changing as well. So, there was a lot of different identifiers to the user so if you had a certain kind of disability it’s actually providing that information in multiple ways.

Trevor: Yeah, and that’s great. And then over on the side it tells you pluses and minuses, I’m 13 cents sharp or whatever, 13.6 or flat. So, I can show the pitch, I can show the wave, and I can record it. Let’s go ahead and record it. You’ll see as we’re talking the spectrum, and then if I do with the notes, it’ll show me the notes, which ones they are, and how badly out of tune they are and we’re just talking. I could play my saxophone. But it would give you the same thing.

Rebecca: Right.

Trevor: So as far as I’m concerned, if I turn this away, and I can record it, and then go back and look at it, I’ll know that if I do nothing, or how much I need to do on the D. The D that I played for you before, nd gave you three different examples is the worse out-of-tune note on saxophone. So, I probably was sharp but this will tell me every note that I need to. You can even slow it down and stuff. You can also record your student visually and with audio. The visual thing is interesting because a lot of movement in the face will affect tuning. When I came to college, I chewed a lot, I had movements in my throat and my face and my teacher would just tell me what they were. But now I can like hold the video against them, show what their movements are, and how it’s affecting tuning and it’s, you know, just another another tool to get you there beyond just recording their facial features and hearing it.

Rebecca: It’s an interesting way to note take because it’s a visual note-taking method that you’re not just having to remember the conversation that you had with your faculty member or mentor, but that you can go back to and see again, if you’re able to record it and save it for later.

Trevor: Right. Yeah, I know, it’s great. It’s a neat tool. So, I’m starting to love this tuner and this metronome as much as the others. It does a lot of specific things where if I’m just pulling out a tuner like Cleartune, as we talked about, or Tempo, that’s more my functional as opposed to how badly I’m out of tune when I’m playing this passage. And the same thing I notice whether I’m using this app, TonalEnergy Tuner, or I’m using Amazing Slower Downer, or I’m using what we’ll talk about later, Smart Music, is that when I’m playing along with the pitch, in the case of Amazing Slower Downer, I know I can hear I’m out of tune, and then I can check it against this or if I’m playing along with Smart Music and you check it against here and see where I’m out. And I play multiple instruments so I’d like to tell you I’m perfectly in tune on every instrument but sometimes I’ll be playing A clarinet and I haven’t played that for a while where I normally do B-flat clarinet and I’m like, “Oh, that’s right. This B-flat is flat on the A but sharp on the…” you know, you have to adjust and unlearn and it allows me to do that prior to getting in there with my collaborative pianist and realizing that I’m out of tune. It’s a great tool. It’s got so many more features that I’m going to start looking into but there are funny features like… not really funny… but the band director or the the judge using it to try to figure out where they were, which hopefully we won’t have to use it very often for.

Rebecca: I wonder for a beginning student who has very little experience… maybe someone who’s not in the music program, but is learning an instrument. Have you worked with students using these kinds of apps with that population of students… like total beginners?

Trevor: Prior to teaching here at SUNY Oswego, I was teaching beginning students… so, seven to eighth in junior high program and then had fifth and sixth grade students. I haven’t used it and mainly because of physicality. Until you develop your armature, until you learn how to use the air, it’s almost impossible to use this effectively because if you don’t have the proper air you can’t make those minute adjustments. That being said, to get the initial tuning for every instrument, whether you’re in fifth grade or fourth grade, you do want to make sure the violin is in tune so students will start using that, you do want to make sure that your concert B-flat’s in tune, which only means after that all the other 11 notes are out of tune, but it’s a good reference point. And then students start to use tuners that way, and amazingly some of them like apps and stuff, and they get into and they show me different things they’re using for that I never thought. But initially, I try not… until those physical things are employed properly. It’s almost futile to actually do it all the time.

John: They need the fundamentals before you can work on tweaking.

Trevor: Right, yeah, especially for you know, with piano you don’t have to worry about tuning. Guitar, that’s got frets so it’s good to know how to tune it. You know that more…

John: You’ve never seen my piano, but… [LAUGHTER]

Trevor: Yeah, but yeah, no… We’ll talk about the next thing, Smart Music, and how it’s used more, or at least, I think more band directors in the public schools use it at elementary all the way through high school, then maybe as far as band music and material is than college professors. I use it for the solo literature, like clarinet concertos and sonatas ….but in the public schools they’re using this technology we’ll get into to not only evaluate their students—how they’re doing, to keep track of their practicing—but as a preparation tool, it’s great. And it can also be not as great for certain specific things and we’ll talk about that when we get there. The next tool—and we’re kind of focusing on my phone right now—the only app that is not available on a phone, a mobile platform that way, but is only on an iPad—is the Smart Music. As I said, we’ll get into that later. The next app I like a lot and I just got an email right before this, there’s a physics professor on campus who’s named Shashi and he’s a saxophone player. And he does exceptionally well, he’s way better of a saxophone player than I am in physics by any means. [LAUGHTER] He loves it. But he gave me an update “I used iReal Pro yesterday at an open jam session. I played these tunes” and went “Great, I’m excited.” It’s great to see that it can be used by by anyone. And as I said, he’s come miles and he loves this because for a physics professor he probably jams with people and he’s been doing a lot more of that. But initially, he’s like, “Who do I get to play with?” and if you don’t play music with somebody—it’s a social thing too… or people hear it—then you kind of like “Why am I doing this?” Even myself as a professional musician, my colleagues and I will prepare harder and more for either recordings or performances coming up then we will during the summer when there’s nothing to do. That being said, we’re preparing always for the next semester, but it’s always great to have that motivation. If I never played with anybody I probably wouldn’t continue playing, especially being a saxophone. A pianist is different, a guitarist is different, perhaps. Drummer is probably different too. I can’t imagine one person only being a drummer and doing that for the rest of life without collaborating with someone.

John: Did you tell us what iReal Pro is?

Trevor: In jazz, there’s these things called real books or fake books. Now, they’re legal. Back about 20, 30 years ago you can only buy them through illegal sources, and they were the melodies of famous popular music standards and jazz compositions, with chord changes behind them. And then you would get together with musicians and you’d play. Now back in the 70s, a guy by the name of Jamey Aebersold used professional musicians who recorded them on CDs—or back then I guess vinyl—and then gave a play along book with it. So he’s got the Jamey Aebersold play along. And what this is done is taking non-live musicians and put it into an app where you can play along with. You can see the chord changes. There are no melodies because you’d have to pay money and royalties for the melodies. But there is no royalties needed for chord changes, and so of the title of the tune and then the chord changes. It comes with no actual pieces on it until you go to the forum and then you can download thousands of pieces that are already programmed in that you can play along with. So I’m just going to pick up right now blues “Straight, No Chaser” by Thelonious Monk. Although it could be any blues until you hear the head and I’m just going to play you a little bit of how it sounds and then I’ll give you a visual description later about it.

[“Straight, No Chaser” PLAYS]

As you notice, when the music goes by, or the chord changes goes by, there’s a highlighted measure so that you know where you are, which is handy for students that are learning to improvise. Later on it’s not necessary, you could just look off a real book lead sheet and play along with it. There’s many features, one I can make the transposing instrument like I’m playing saxophone in the key of B-flat instead of C. I can see the chords transposed in front of me even though it’s playing the key of C. I can do that for any instrument. There’s like a balancing… if I’m a pianist, I can take the piano out or if I’m a drummer, I can take the drummer out. I’m not sure why a drummer would want to do that. And I can control the reverb… I can change the tempo… I can change the style—this is a jazz medium up swing—I can change it to almost anything—Latin, Gypsy jazz, trio guitar, slow swing, traditional jazz—and there’s pop and Latin stuff so I can change the beat behind. I don’t do that as much unless you’re working in the Latin situation but it’s nice to have and as I said, you can also program pieces in. So we just did a concert with this great alto saxophone player by the name of Dick Oatts down in the Village Vanguard Band—that’s where he is primarily and teaches at Temple University—a world renowned musician. And he sent us his charts which are really hard and don’t follow normal progressions. They’re really kind of tricky so I actually programmed them, or one of them, into here just so I could play along with it a little bit… hear the changes a little bit better. I mean I’ll voice them out on my saxophone like normal but it’s a really handy tool to have. And speed it up too like I can’t take it to the tempo he wants it so I’m gonna slow it down and feel good about myself and they’re not really that odd but they’re more unusual chord changes. They made sense but just needed more time to practice and I just programmed in it by…you kind of scroll up and type in F major and put a seven and you can do all the different things. It’s really interesting and you can save them and if you want and you can share them on the forum so other people can look at tunes. So this is an amazing tool. I love it for that because I can speed it up and slow it down, but there are other options. I think Aebersold things are still available online unfortunately for Aebersold ‘cause you know, he’s losing royalties and stuff on that. But then on YouTube, there’s “Learn to Play Jazz” I think if you type that in and whatever tune you want to do there’s that for the people that actually have iReal Pro copies on YouTube. You can’t slow it down, you can’t transpose it. You can adjust the balance and blend and stuff but it’s there. So my students will find those things too and use them. And I’m totally fine with that. You know, financially it makes sense. One of my favorite features is…and John, you have this app you said?

John: Yeah, I do.

Trevor: So you use this app, and have you have you looked at the tablature part of the app?

John: I haven’t been using it recently. Since we started the podcast I haven’t been playing so much music.

[LAUGHTER]

Trevor: Got ya. Give me a second here. So I’m going to show you…and do you play an instrument at all?

Rebecca: No. That’s why I was asking all those music questions.

Trevor: And please continue to do so. [LAUGHTER] In the corner of the app there’s this little thing that looks like a tablature and you can choose from multiple things. One is for guitar so you can select guitar chords. You can do piano, and the piano you can do one hand or two hands. You can do ukulele, which is a new feature and more importantly for me is the chord scale library which I use all the time. So let’s go back and add the chord scale library. So if I press play and then I pause it right away it’ll give me chords down at the bottom. So for that C7 chord I can do either dominant mixolydian chord if I scroll over. Actually for this one, there’s only one, but for a lot of times when it’s a different chord like flat 5 or something like that it’ll give you multiple options of what scaless you can use over top of it. Once you talk to a pianist—they’re not really fond of this app, because it’s not exactly like you would teach jazz piano—but if you have no skills, and you want to learn from an app, it can give you some things. It’s a really nice tool for myself and all of my students too. And as I said, our physics professor on campus uses it a lot and as do you prior to doing the podcast.

John: I used to use it more before we started this podcast.

[LAUGHTER]

Rebecca: Trevor, we found that students were using these tools—or when you’re using these as part of your teaching—that the students abilities have improved or that they’re playing has improved. Have you noticed a difference in the integration or?

Trevor: In some regards more about how they practice and how long they practice. But yeah, I’ve noticed an improvement. By having no one to play along with it’s difficult to improvise—you improvise in different ways—it’s a very valuable skill to be able to cut the changes… and me as a saxophone player, if I play and my colleague walks in the room they say, “Oh, you’re playing this tune” based on the solo I’m doing then I know I’m really cutting the changes. But when I don’t know a tune as well, it’s nice to have the harmonic progression so I can use my ear a little bit more than I can my mind and eventually, it’s a combination of both, I’d hope. But my students will then play along with the blues a lot more than they would if they were just doing it by themselves.

John: And much of the benefit is students can work and get up to a point where they’re ready to play with other students so that everyone can use their time more effectively when they actually do collaborate and meet at once. Rather than each person struggling just to do their own parts, they get to work out playing it together.

Trevor: Exactly, it saves so much time. In addition to that, when we think about rural areas. A lot of times you’ll be able to find a great collaborative pianist for doing your solo so it might be okay to use Smart Music—I hate to say that—or you might not have a chance to prepare so you play along with records—or you play along with, in the old days—and now you can play along with this. You know, even getting out and playing along with it at open mic night it’s just totally acceptable because it’s expensive to find musicians and especially in different places. Where in New York City I think would never see anybody using iReal Pro to play along with at a club, you know, just because…

John: …There’s a few musicians there, yeah.

Trevor: But I’ve heard of people pulling it out and it does irritate people. There is the idea that once you graduate from a jazz program—specifically, if you focus on jazz. I myself, did music education. I play jazz a lot and I also did classical clarinet, classical saxophone—but if you’re only a jazz major, or if you’re good jazz player, you’ll have like 150 to 200 tunes memorized. So you’ll call a tune, you’ll have the head, and you’ll know the chord progressions. But nowadays, a lot of the younger players are… from what I hear and what I hear from complaints from older generation people is that they’ll pull out the iReal Pro if they don’t know a tune and play along with it. And there’s pluses and minuses. Now you can actually play that tune. The minus is it’s not as ingrained as if you had to learn it and memorize it and ingrain in yourself, and that you won’t be able to interact with the high level players you’re playing with in the same way as if you did that work. So there’s, you know, again, pluses and minuses. It never should be a substitute for the ideal, but it should be a tool like you just said to get you there. Or in the cases of where there’s not a bass player to play with, at least it gives you a way to proceed until you can find one.

Rebecca: How do students respond to using these tools?

Trevor: They like them. They enjoy them. They think they’re really neat. This freshman student that came in—he was using YouTube, one of the play alongs he warmed up to it—but yet again, the disadvantage of this is all electronic and they sound electronic. So at least some of the “Learn to Play Jazz” are actually real jazz musicians. But you can alter the tempo—or you can’t—I guess you can with the 75%, 50%. I’ve never tried it, now that I think of it, on YouTube…but you know, it just it takes adjustment. You know, there’s nothing to substitute live musicians. The drummer can’t react to what phrase you just played. Nor can the keyboard voice omething different if you’re going there harmonically. But it is a tool. And yet again, a tool that’s not a substitute, but a tool that’s effective in preparing. The next tool we’ll talk about is actually on my iPad, and it’s Smart Music. Smart Music is an app which originally was on a computer and way back in the 90s when it came out it looked like the old Atari cartridges you’d buy with the machine for $2,000 and each cartridge would be like $80 or something and have one piece—Mozart’s Clarinet Concerto. But nowadays it comes on an iPad and you can also get it on your computer. The computer has more features on it but obviously is not as portable as the iPad. And what it does and what it’s being used for is amazing, but also can be negative and we’ll talk about that after we talk about the positives. It has all the band music that you’ll want to play in public school and what the band director can do—including myself as a band director—is that I can say, “I want you to play Holst’s ‘First Suite,’” is which we picked here, and gives you the saxophone part but I can also push the button as you can see, and I can pick whatever instrument I’m playing. So I’m playing bassoon, it gives you the bassoon part. It has the music in front of you and then once I play along with the rest of the band without that bassoon part or that saxophone part, it has little X’s and O’s and tells me what percentage of the music I got right rhythmically and everything else. So band directors will use this to say, “I want you to record this and get it to me and then that’s your percentage on it.” Now it does have some flaws. I’ve listened to students play and myself play and it’s no way that I’m getting 60% I’m getting closer to 85 or something like that. So some advantages and it’s gotten better over the years.

Rebecca: So just to clarify, it’s like Guitar Hero video game?

Trevor: To clarify, Guitar Hero has nothing to do with guitar.

Rebecca: Right. Yeah.

Trevor: It’s more of a rhythmic pushing buttons…

John: Pattern matching, yeah.

Trevor: Pattern matching and it has something to do with the music because it’s actually, you’re hitting it rhythmically along with the song, that’s about it. But you’re right, it’s like a play along, it’s like karaoke without the part you’re on. But this karaoke, in this case, records where your notes are wrong with red and green, where you’re behind, and it tells you where you’ve missed things, and then that goes back to the band director and they can grade you accordingly. So it saves them time.

Rebecca: The visual cues are similar is really what I’m…

Trevor: Oh I see what you’re saying. Yeah, I haven’t I haven’t played a lot of Guitar Hero and I apologize. Most of my knowledge of Guitar Hero’s from South Park and we won’t talk about that. So it allows you to do all those different things and then record them. As far as soloists are concerned, that’s where I use it more. I use it more to play along with a collaborative pianist that’s not there. And it follows you. So for instance, I’m going to pick the clarinet Copland concerto that I’ve prepared for, and it’s got a piano reduction to it. It’s a very difficult piece to put together when you’re a clarinetists like me, who’s a saxophone player that also plays clarinet well. So I use it a lot of time to prepare before I meet with my collaborative pianist. And I’m going to pick an arbitrary section in the middle so you can hear the piano part. And the piano part will play and then once I play along with it—and I’m not actually going to play—it will record it, and I can play it back so I hear all my mistakes which is really nice, and also my pitch errors and stuff. And then if I am unsure of myself, I can actually set it so that it will play with the clarinet part along with me so that I can hear that part to make sure I’m rhythmically accurate with it. That comes to one of the problems with being able to play along with something that has 100% rhythmic accuracy, but you don’t. You know, our ear can pick up on things so quickly that… let me give you the example. When I was in high school, there’s this lady named Amanda and she was the first-chair alto saxophone. And Amanda would have the rhythm perfectly the first time and then me, who would after listening to her having it perfect the fifth or sixth time, so I didn’t really learn how to count properly. And by playing along with the Smart Music, where the piano is actually playing your saxophone part or clarinet part, and you’ve been listening to that. Then when it comes to actual real performances, if you play along with it all the time, then you’re relying on it so much and your rhythm goes out the window. So that’s where some technology can get in your way, specifically with rhythmic attributes, but also with other things too. But yet again, it’s a tool and if you recognize that, you know, I prepare it rhythmically with the students first, I don’t let them use Smart Music until they’re at almost the performance level. So we’re going to play a little bit of the Copland Clarinet Concerto somewhere, but 140 measures into it. I’m going to play it initially with the accompaniment but with nobody playing, and then I’m going to set it and you can hear the difference now with the clarinet part being played by a different sounding piano.

[COPLAND’S CLARINET CONCERTO PLAYS ON PIANO]

So that was just the piano alone. Now I will do it with the clarinet part being played by a secondary piano, and you’ll hear how it fits in and that’d be what I’d be playing along with if I needed the rhythmic help or I wanted to check my pitch.

[COPLAND’S CLARINET CONCERTO PLAYS ON PIANO WITH SECONDARY PIANO]

That wasn’t as obvious an example but if you knew the part you’d be able to hear there’s extra additional things added to that which should be the clarinetist. So this tool is just remarkable for that and as I said, for band directors in public school and at the university level too, it seems like an appealing thing. I just haven’t figured out a way to automate it or include it in my, you know, as part of the practicing routine for my students. And if they don’t come from it from the schools, it’s a little bit more difficult to implement. I’m not sure at the university level it’s ideal for me to implement. I’m still wrestling with that as I am with implementing any technology.

John: So how have your colleagues responded?

Trevor: One of my colleagues, Rob Auler, he uses it as much or more than I do, we always check out, “Hey man, check this app out. Do you think it works really well?” So we check out different apps and sometimes those apps apply—or in the case of Rob, a tuning app wouldn’t apply for him because, you know, somebody else tunes his piano for him—but he really loves Amazing Slower Downer. He uses the iReal Pro probably less than I do, because he could collaborate with himself in his left hand while he’s playing solo lines in his right hand. And he doesn’t use it as a tool like Smart Music because, you know, pianists are not usually playing along with other pianists, they’re the collaborators. And we’ve talked about that, too. He knows it’s not a tool that can replace a pianist. But when we work together, Rob is a consummate musician and so he learns things way faster than I do. His mother would feel the TV to see if he was practicing five hours a day when he was younger, where I was probably you know, playing out in the backyard with dirt or something. He learns things so much faster and he doesn’t want to spend time rehearsing. So for me I’ll prepare on Smart Music and I’ll prepare on these other apps and get to that point where now I don’t have to spend so much time rehearsing with him. We’ll just do it once or twice like professional musicians do and he’s always been that way. For me, it’s upped my ante and I’ve really enjoyed that. So we’ll talk about all kinds of technologies, he’s huge into it. Other colleagues, for vocal people, they prefer having a pianist collaborate with them—which makes total sense—so it’s not as prevalent and I understand that. But most of them are not against, they just don’t understand how to tie things into doing it because what they’re doing already is fine. And that’s one of my philosophies too: If something is working, and it’s not going to be any better by adding the technology, there’s no need to add it. But if it can be an efficiency, the fact that usually in history—and as I got older, I realized this—that convenience wins out. And if I look at audio recording, you know, tapes. Small tapes were horrible audio quality compared to vinyl and CD had finally came up to a very good audio quality, probably not as good as vinyl, but most people can’t hear that spectrum anyways for those people that believe it.

John: And usually it’s better the first couple times you play the vinyl. After, the needles have worn down the grooves, unless you’re reading it with a laser, the quality of vinyl degrades really quickly.

Trevor: It does. And then MP3’s everybody thought, “Oh that won’t stick,” you know, everybody kept to analog, and really, the quality doesn’t sound nearly as good. But most people are listening through cheap headsets and on phones and using it for different background thing. Most people are not audiophiles and most musicians are and they think that technology is not going to last. So unless there’s a technology that other than convenience as a musician that I think benefits my students, I don’t really buy into it too much. A lot of people have embraced it and for different reasons. I think everybody owns a tuner on their phone. Everybody has a metronome on their phone. My colleague Juan La Manna uses the digital recordings and visual recordings of Zooms and stuff like that for his conducting students to record them and give them feedback on it and he uses it to record himself on concerts and a lot of technology being used, but not the Smart Music so much outside of wind players, I think. And it’s also ingrained in the band world. Jazz musicians have always used technology as far as like play along recordings with Jamey Aebersold and later on so it’s only a natural progression to use something like iReal Pro, I think. But you know, most of my faculty enjoy technology when it’s beneficial if it doesn’t help them out, there’s no need to learn it.

Rebecca: So I always wrap up our podcast by asking, what’s next?

Trevor: One of the things I’ve held off on and my colleague Rob Auler has not is reading off of tablets for music instead of bringing paper along with them. So for him it made a lot more sense, because he can use the AirTurn Duo Bluetooth pedal, and he doesn’t have to rely on anyone to turn his page. It’ll take them a little time to practice it. You can actually write and notate things in there if you’re in the middle of rehearsal, you know, we want more piano here, or we listen to the cello or whatever. But for me as a single line instrument, it didn’t make that much more sense other than the convenience of it, which we’ve talked about, which is to not have a lot of music in your backpack. But then a few things have changed over the course of the last few weeks that made me think I need to do it for my students and myself. They should know this technology. So I went to a rock gig—I hadn’t played a rock gig for 20 years at a wedding one of my friends asked me to do—The bread, the money was pretty good so I decided to do it for the first time in a while. And most of the people were playing on that, I had this like four inch binder that the guy gave me. He wouldn’t give me his iPad, and I played there and then I did a soloist with the Syracuse Symphoria and I was there just watching the orchestra play in one of the pieces and the harpist… and harpists are one of the people that are not known for…you know, they play an antiquated instrument…. It’s a gorgeous instrument… But she was using an iPad. Ironically she was reading a hard copy book while she was waiting for her turn to play in different pieces [LAUGHTER] so I thought that was ironic but she was using an iPad and I thought, “Oh, maybe I should do this.” And also the advent of the iPad Pro, where it’s large enough. You know I tried it when Vornhagen came here when I was doing some pieces, and I wrote it and sketched it and put it on the iPad, it just wasn’t conducive to reading with low light. And then Rob mentioned something to me that I thought would be the ultimate reason why to get one. He goes, “Wouldn’t it be nice if everybody played off the score?” By that I mean piano score that has a cello and the clarinet part there and we could all take a look to see vertically what’s going on as opposed to what we normally play off, the cellist has a single line as do I. Obviously the problem behind that would…we have to like learn how to read music differently. Instead of going from one line to the other, I’d have to skip them things, but I’m a conductor so that’s not a problem. But it would actually save a lot of time in rehearsal if we got off, or more importantly, if we want to phrase something together or we’re talking about something like, “What happens there?” I’d have to go over and look over Rob’s shoulder. So that to me, outside of the convenience of it, and that I probably should get back into the 21st century and actually get playing off them so our students know how to do it. I think it’s something that my next step is going to be starting to do that. And yet again, you can edit it. You can draw on it. You can put in repeats. It just seems so much more logical to do. I put in for grant for this and at the end, they’re evaluating it and they asked me specifically, “What would you do if you didn’t get this?” and I said, “Probably use paper” other than than what Rob just mentioned to me right after that session where it’d be great to read off the score ourselves. The advantage of it is more mobility and portability. These fake books are two and a half to three inches thick and there’s three or four of them and he has them all on his iPad and I have to slug them around with me, so I have nothing against paper.

John: But it’s a whole lot easier carrying around an iPad where you can have tens of thousands of scores with you.

Trevor: It is. And a battery pack too. That’s the other thing, if you don’t have a way to charge it and stuff like that, but that nowadays that doesn’t seem like it’s a concern at all.

John: And iPads generally got 10 or 12 hours at least of battery life.

Trevor: Right.

Rebecca: Well, thank you so much for joining us, Trevor. It’s been really interesting to hear you talk about your practice with mobile technology. And I know for me as a visual artist, it was making me think about the kinds of tools that we use as a visual artist and kind of translating the kinds of activities that you’re thinking about with students to different kinds of activities that I could do with my students and also thinking about how to use those to kind of push students just a little bit further.

Trevor: Nice. That’s great.

John: Thank you.

Trevor: Thank you, appreciate it.

[Music]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Kim Fisher, Chris Wallace, Kelly Knight, Joseph Bandla, and Jacob Alverson.

65. Retrieval Practice

Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, Dr. Michelle Miller joins us to discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Her academic background is in cognitive psychology and her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications.

Show Notes

  • Miller, M. (2014). Minds Online: Teaching Effectively With Technology. Cambridge, MA: Harvard University Press.
  • Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210.
  • Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319(5865), 966-968.
  • Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 1199327.
  • Kahoot
  • Bray, Niki (2018). 43 to 0: How One University Instructor Eliminated Failure Using Gamified Learning. Blog post
  • Retrievalpractice.org
  • Lang, J. M. (2016). Small teaching: Everyday lessons from the science of learning. John Wiley & Sons.
  • Pennebaker, J. W., Gosling, S. D., & Ferrell, J. D. (2013). Daily online testing in large classes: Boosting college performance while reducing achievement gaps. PloS one, 8(11), e79774.
  • Pauk, W. (1984). The new SQ4R.
  • Thomas, E. L., & Robinson, H. A. (1972). Improving reading in every class. (a discussion of PQ4R)

Transcript

John: Retrieval practice has consistently been shown to be important in developing long-term recall. Many students, however, resist the use of this practice. In this episode, we discuss methods of overcoming this resistance and examine how retrieval practice may be productively used to increase student learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

John: Today we’re welcoming back Dr. Michelle Miller. Michelle is the director of the First-Year Learning Initiative, Professor of Psychological Sciences, and President’s Distinguished Teaching Fellow at Northern Arizona University. Dr. Miller’s academic background is in cognitive psychology. Her research interests include memory, attention, and student success in the early college career. She co-created the First-Year Learning Initiative at Northern Arizona University and is active in course redesign, serving as a redesign scholar for the National Center for Academic Transformation. She’s the author of Minds Online: Teaching Effectively with Technology and has written about evidence-based pedagogy in scholarly as well as general-interest publications. Welcome back. Michelle.

Michelle: Thank you so much. It’s so great to be here today.

Rebecca: We’re so happy to have you again. Today’s teas are:

Michelle: Well, I’m drinking Coco Loco, which is a blend from a local tea shop here in Flagstaff, Arizona. Steep Leaf Tea. And Coco Loco is a lot like what it sounds like. It’s chocolate and banana. So tea snobs may scoff at my choice, but it’s wonderful.

John: IIt sounds good.

Rebecca: And I think I saw a nice silver teapot that was poured into a green and blue tea mug.

Michelle: Yup.

Rebecca: A nice tall one.

Michelle: That’s what I need.

John: And I’m drinking ginger, peach green tea.

Rebecca: I went with a Christmas tea today. So Michelle, we invited you here today to talk a little bit about retrieval practice. Can you first start with defining that for us and letting us know what it is?

Michelle: Right. So retrieval practice is essentially the act of pulling something out of memory. So that is, in memory research, what we would term retrieval. So something is stored in memory and we want to pull it out so we can actively use that information, have it in our conscious minds and so forth. And so we go through this usually very fast process called retrieval. So retrieval practice is specifically the act of doing it, and we contextualize that with learning. So, when I’m trying to learn something or I’m the process of learning something, to say, “Oh, what was that fact I remembered, or what can I say about this?” When we do that, it produces something that we can call the testing effect. So this is kind of the clearest example… not the only… but the clearest example of retrieval practice in action during learning is when we sit down to take a quiz, take a test or something like that. So all the excitement that’s happened around retrieval practice in higher education, and really in the rest of education today, is around this finding, which has been replicated many, many times: that tests are, as one person put it, not neutral events in learning. When we take a test on something, that has a very powerful effect on our ability to remember it in the future. So, really simplified down to its core, tests help us remember in the future; when we take a test it strengthens our memory. So that’s what retrieval practice is and it can, as we’ll maybe talk about today, take many, many forms in learning settings. And I did want to clarify too, this is something that I definitely don’t want to take any kind of credit for discovering this. This has been around and has been known about for a long, long time. Some of the big names who are associated with this: Jeffrey Karpicke, Robert Bjork. Roddy Roediger, there are quite a few really heavy hitting cognitive scientists and cognitive psychologists who have established this. But there are many, many of us who are out there trying to disseminate this to other teachers around the world so that we can all tap into the power of this. And I have done a little bit of work in this area with my colleague here at Northern Arizona University, Laurie Dixon, who’s another psychologist… and we teamed up some time ago to look at a very practical implementation of retrieval practice in an Introduction to Psychology course that we conducted some years ago and this is a course, you can imagine, where just trying to get students to perform even a little bit better is a big project. So we examined how even something kind of basic… it was very high tech at the time… but just basic web quizzes that came packaged with the textbook. We said, “Well, if we actually assigned students to do these as part of the course, and if they went through and treated these as opportunities to learn, not just assessments in and of themselves, would that have any systematic impact on course performance?” And we found that in fact, there was a significant improvement associated with that. So that’s kind of the landscape of what retrieval practice is, and why we’ve been so interested in discussing this in the psychology of teaching and learning.

John: In fact, I saw you present on that about 11 or 12 years ago in Orlando at one of the NCAT conferences and it convinced me to completely revise how I was giving my classes and it’s made a big difference and resulted in some significant improvements in student learning. Given that we know so much about retrieval practice. Why are faculty resistant to doing this?

Michelle: Wow, that’s a great question, and that is one that I have been really facing a lot these days in my own practice, talking to other faculty members, and disseminating this through some different activities I do in this area. It is easy for those of us who work in this area, cognitive psychologists in particular, but a lot of us who like you heard about this a long time ago. We’ve seen the power of it. We forget that to other faculty, this can be a very off-putting concept. And so it’s really great for us to think about why that is. And I always think that’s really good for me to kind of go back to that and say “Yeah, not everybody is sold on this idea. And there are good reasons for that.” So like with a lot of things that we talk about in teaching and learning, I think that these really break down into two neat categories: there’s the philosophical issues that people have with it, and then there’s more practical and logistical issues with it. So kind of tackling those one at a time. Philosophically, when people say, “Yeah, I understand about the research but this just goes against something that I believe as a teacher or how I want my classes to be. Here are some ways that that can play out. First off is this idea, that I’ve heard in one form or another quite a few times, and that is this notion of superficial learning. So “Okay, sure, there’s a study that showed that maybe people retain something a little bit better. But surely that’s not this deep learning, whatever that is.” That’s a concept that we all want. So do tests and exams… just testing… create just a superficial form of learning? And while, of course, I understand that, and I absolutely applaud faculty for really thinking deeply about that issue, and caring about it. I Here’s the thing… we got to define that. We social scientists, that’s what we do… we have to kind of break things down and say, “Okay, what does deep learning mean?” and I don’t know that anybody has kind of definitively done that. But when I look at that, I say, “Well, this is not just one research study that showed a little improvement in a lab test. There’s quite a few studies that do use realistic types of materials. It’s not all just contrived laboratory studies. Furthermore, there’s also studies that show that when students engage in more quizzing and testing on material that they actually are able to transfer that learning better. And that is a very, very big deal in teaching and learning as a lot of us know is not just getting students to be able to solve a problem in one context or work a concept in one context, but can they do it in the next circumstance? And that very difficult process is aided by quizzing… and to me, what could be deeper learning than learning that transfers? So that’s part of it. Some of it is perceptions around multiple choice quizzes and tests. There’s an assumption too that if we’re talking about quizzing we must be talking about multiple choice questions… and first off, sometimes in larger classes, those are the final assessments… in that Introduction to Psychology course that we studied years back, that’s what the assessments were… so, having students practice in that format I don’t think that we should necessarily dismiss that. And as we can talk about in a little bit, there’s lots of ways to induce retrieval practice that actually don’t involve multiple choice questions. So, there’s a bit of that as well. And something that I’ve talked about with some faculty recently, too, is this baggage around K through 12. And maybe that’s something that’s resonant with you all.

John: Yeah, that’s given the testing effect a somewhat bad name, because high-stakes testing is being used in a lot of what’s going on with K to 12. But that I don’t think is what retrieval practice as you’re suggesting is all about.

Michelle: Right, and I have to be cautious here. I really like how you laid that issue out in K through 12, that there is a reputation problem… and that has happened because of the high-stakes standardized testing policy in the United States. And I got to be careful because I don’t want to represent myself as an expert in K through 12 or in K through 12 education policy. But I don’t think you have to be an expert in that to know that there’s been a lot of same pretty well justified public pushback against over-testing in K through 12. And yeah, I think that we absolutely do have to be aware of that. Students come to us in higher education… that’s a system that many of them have been through and our faculty are very aware and very cognizant of that too. So, nobody’s a blank slate here, not our students, not our fellow faculty. We have assumptions and ideas and experiences about testing that happen. I think those can be addressed. But, yeah, that is definitely another very big barrier. We got to differentiate between high-stakes standardized testing for the reasons it’s done in K through 12, and low-stakes testing and quizzing for learning as proponents of retrieval practice would have it.

Rebecca: Some of the pushback I’ve heard from faculty fall into two categories related to this as well. One is that they assume that retrieval practice is best implemented in 100, 200 level introductory classes instead of upper level 300, 400, graduate-level classes. And then the other area is that paper and pencil tests don’t make sense and all disciplines. And so they assume that a test has to be in a paper/pencil format, which could be online testing, or it could be multiple choice or it could be essay questions. But I think that, from being someone in the arts, like there’s other ways to test beyond that, but we don’t think of those as tests.

Michelle: Right. That’s another great lens through which to look at this issue; that we do need to broaden the definition to draw more attention to this and to make it a more appealing concept. But yes, how can we make it broadly appeal across lots of disparate disciplines? Not only does it not have to be a multiple choice type of exam, maybe it’s not a pencil and paper exam at all. And we as faculty have to think about what makes sense there. You make a good point about the levels concept. I think, these days most of us have heard of, “Well, there’s one particular Bloom’s taxonomy…” which is a wonderful framework for getting us thinking about being systematic about what we’re asking students to do with the information that we’re bringing to a course and trying to do things like align the teaching we do with the assessments that we have. That’s wonderful. However, I think it does ingrain in us that idea that “Well, just knowing things is sort of at the bottom… You sort of get that out of the way, and then we go on to the good stuff.” And from a cognitive perspective, these relationships are much more fluid and much more interdependent, so that yes, absolutely, the higher thinking that is what we want. That is what we should want. Or if we’re in highly applied disciplines (if we’re in the arts, for example), we need students to be able to do things with that information. But they have to have that. So I think it challenges us to think of new ways with that concept as well.

John: One of the barriers I think some people have is they don’t like to grade tests and so forth. But one of the things you mentioned in your book is that the testing effects been known for a long time, but it was really difficult to implement in terms of low-stakes testing, particularly when you’re teaching at a larger scale. But, as you’ve done yourself, and as you suggest in your book, computer technology makes it easy to automate some of this… certainly more easily for multiple choice and free response and similar things. But it makes it a whole lot easier for both students to have multiple attempts at learning something using some type of mastery quizzing and it makes it a whole lot easier for faculty who don’t have to spend all their time grading.

Michelle: Right and that’s absolutely where the practical stuff comes in. So, we’ve worked through some the philosophical objections, that: “No, this does not turn your classroom into some terrible assembly-line concept of learning. It’s not going to create a bad relationship with your students. It’s not going to simply carry on a legacy of bad policy as people perceive it. It’s not going to do those things.” And then, we do get to: “Alright, but what is this going to do to my life as a faculty member?” …and this is important stuff. Those who know me know I’m a big fan of James Lang’s work in his books. One of his more recent books is called Small Teaching and there’s a lot of different takes on it in that book, but it really hits home with respect to “We do have to think about not everybody’s in a position to, nor should we even try sometimes, to just take everything down to the foundation and rebuild.” That we do really need to think about “Do I want to do this, if it’s going to create 800 more questions for me to grade?” Is this a sort of a situation where, “Well, I don’t want multiple choice, so I’m going to have to give these open-ended questions. I’m gonna have to give feedback and I have 200 students, what will happen? If I am going to go with multiple choice questions… well, how am I going to do this? to have to write all of this?” And yes, it’s one of the most powerful outcomes of the educational technology revolution that makes it workable, and scalable in a sense, even with large classes to do these types of things and to bring them in. So, that is definitely a message that I hope faculty think about if they’re on the borderline of wanting to bring in more retrieval practice into their classes.

Rebecca: I’m in a discipline where the multiple-choice questions are using things digitally doesn’t always work for testing and practicing some of these basic things, and there’s not a good way to automatically grade it. But one of the strategies that I’ve used is actually some self grading, which has actually worked pretty well. I just check and I have them write notes about anything that they got wrong. So, it demonstrates that they’ve tried to understand when we go over it, and I give credit based on how thorough those notes are, rather than whether or not they got the question right or wrong. And that’s made a difference in my classes. And had I not come up with that solution, I think I would have abandoned it because it would have been too much work. But it it actually is working pretty well.

John: When we had a reading group on Small Teaching last year, one of the things that was widely adopted by faculty was a very simple form of retrieval practice where they had students at the start of each class reflect back on what they had done in the previous class. And most of them have continued to do that in subsequent classes as well.

Rebecca: One of the other barriers that faculty might raise is the idea that students don’t take low-stakes things seriously, or that they don’t put the same kind of time into it that they might for something that’s high stakes. Can you talk a little bit about how we might help students find value in retrieval practice and subsequently also with the faculty then?

Michelle: Right, so that, ”Well, we can put it out there, but will they do it?” I’ve kind of crossed this philosophical and practical barrier for myself of giving some credit for pretty much anything that I am hoping that students will do. I put the work in to set it up and I do believe as a teacher that there’s reason to believe that will help their performance, that I need to work it into the syllabus somewhere. I don’t think it detracts from learning, necessarily, to say, “Yeah, there’s some points associated with this.” And especially with our students who, for many of us, are going in a million directions at once. They’re juggling jobs, multiple classes, sometimes their own families. So having an incentive in the form of points—having some kind of a payoff—I think, helps them make that decision that this is at least worth the time to do. I think the other thing that we probably can all do more of, and that I’ve done more over the years, is framing and honestly marketing this to students… communicating with them about why. And when students disengage from an activity like this, when they say, “Ah, why do I have to sit down and do this thing? This is just another test. Oh, no.” …really conveying the excitement and the goodwill that we have in setting those things up can go a very, very long way. Of course, a student, if they just look at it inside and they have no context for why this was put into place, they’re going to have them say, “Well, maybe I won’t do that.” But when we can market to students, we can say “There is a lot of research that shows that this is a very, very good use of your time. And hey, you’ve probably taken a lot of tests in your life that were really about measuring or sorting you and figuring out what you know and what you could do. This is a very different kind of test.” So that can go along way and get the C students nodding: “Alright, I get it. I get why she put this assessment right here.” I think a lot of us have hit on the practical strategy too, that the little Easter eggs or goodies that we plant in the form of questions that get re-used on the higher stakes assessment. So most of us will have tests for measurement at some point in our courses. And yes, students really do pick up on it when you use one, two or more of those items that were in, say, the gamified quiz that you ran in class or the reading quiz that they did beforehand. They can see those and say, “Oh, wow, I got feedback on that. I got an opportunity to practice…” and if it draws in a few more students who see it as almost a legitimate form of cheating, honestly…. like a fun and sanctioned form of getting an advance sneak peek at the exam, then great! Then that maybe is an opportunity for them to come in and see that. So there is that. Actually taking it seriously ourselves, not just in the form of saying, “Well, here’s the points I’m going to give you for this,” but spending class time on it. That’s a big bridge for a lot of us to cross, right? Because we as teachers tend to be very focused on “Oh my gosh, class is for covering material. We use all these sort of distance metaphors to talk about what we want to do with our class time. But if I say, “You know what, guys, I believe in this, and I believe in it enough to where we’re going to spend the entire class period before the final exam….” I did that twice last week myself, when running exams. Or we’re going to spend five or 10 minutes of the beginning of every class period doing this, as one project recently published about doing. If you show yourself doing that, and offer them that, I think that also goes a long way towards it. And I guess to just say, well, taking it seriously… here again, what does that look like? What does that mean to different people? And we can kind of a little tongue-in-cheek say, “Well, why do we have to take it so seriously?” Sometimes games and learning can happen when it is presented in a more fun context. So not everything has to be deadly serious or spending hours and hours and hours of stressful time on. There are occasions when a light-hearted approach can be perfectly good and can still get us involved in that really critical activity of retrieval

Rebecca: I can share an example of doing that in my classes. We’ve done design challenges and sometimes we challenge other classes that are happening at the same time. That reinforces some of the basic principles that we think that students should be doing and reminds them of it… and they might work in a team…and then we have a competition. And it’s fun and what have you. And students like those, it breaks up the day, it makes it more fun. And then I’ve also done things where I give class time to do little design challenges in class that might be individual and then they can level up to working with a partner to finish solving a problem or something… and students value that. They recognize that next time they’re trying to do a project on their own,that it’s easier because they’ve had that practice or that opportunity for the retrieval practice. And my students have actually ended up asking for more of those opportunities.

Michelle: Great.

John: Could you go back just a little bit and tell us what you did in those couple of classes last week before your final. We’re recording this, by the way, in early December during finals periods in both of our campuses, but we’ll be releasing it a few weeks later… to put that in context.

Michelle: Oh, okay. Well, I’d love to. …and I’ve been talking about retrieval practice as you pointed out for years, and I still discover new ways to infuse this into courses. And the context for this is my cognitive psychology undergraduate course. It’s a 200 level. So it’s a lower-division course. And it’s about 60 or 70 students. And as you can imagine, it is a bit of a tough sell. For many of the students it’s their first encounter with this side of psychology. It’s not as intuitive as some other areas of psychology. So there’s a lot to learn and a lot of motivation to be done. One of the ways we bring this and in this course is using a technology called Kahoot, that’s spelled K-A-H-O-O-T,, and it’s really very intuitive and functions very smoothly… relatively free of bugs. That’s good stuff. It’s a program for doing gamified quizzes of various kinds. What I did, and at different points in the semester, and then really amped up in the last week of the semester is running these gamified quizzes. And this is something, by the way, that I hit on and got the idea to try based on a colleague named Niki Bray, who’s from Tennessee, and has actually done some really systematic work in reformulating some of her courses around in class quizzing in just really ingenious way. So I saw some of her presentation and I said I’ve got to try this for myself. I went with multiple choice questions. Kahoot does have some parameters… questions do have to be short…very, very short. And to some people that may be off-putting, but you can put together quite a few of these. And so we would put this up on the projector and students have the option of dialing with either their laptop or their smartphone and weighing in on each question. The neat thing about it is it has an algorithm for giving points based on your speed as well as your accuracy and it’s got a little leaderboard so you can actually have a little in-class competition. Now some people who use this do require all students to do it and they actually issue points for performance. Now, I presented it very much as a practice activity… and made it very, very clear because of my philosophy, I’m not going to assume that all students have devices or have smartphones or laptops or that they want to do that. But I said, “Look, remember we’re talking about retrieval practice, guys. So the real meat and potatoes of this is not buzzing it on your phone. That’s fun. But the real benefit of this activity is what you’re doing sitting there in your seat. And you could be doing this with a piece of paper if you want.” And that is what a few students opt to do. They try to answer the questions, they know what they need to go back and review and so on. And it’s nice because it spits out at the end, a whole report that tells me right away… Okay, which questions do we need to revisit? Which ones did students have the hardest time with and so on. That’s one of the things that I just brought in. And yeah, it was a big deal. I sacrifice a chapter of “coverage” so that we would have more time at the end of the semester. But to me, I would rather have students going into the final knowing that they’ve had this retrieval practice and they have a better chance of performing really well. Earning a good grade on this material I care about than honestly cramming in a little bit more mileage in terms of the quantity.

Rebecca: Sounds like fun.

Michelle: You know what, it does really bring a fun factor. There’s been a lot of different variations on in-class polling, and I will admit to this, I actually purchased and am the proud owner of a physical buzz-in quiz device, complete with a whole spaghetti nest of wires and an incredibly abrasive, buzzer sound, and everything. So previous to this, my educational technology did include… I think I could have up to eight intrepid volunteers who would play a quiz game and then I would have to appoint a points keeper and all this… and props are always a lot of fun. So when I say I believe in bringing in retrieval practice, I really do walk that walk. But I will say doing it by a smartphone does allow for more participation, and I don’t have to worry as much about the minutiae of scorekeeping and stuff like that.

John: I played with Kahoot a little bit at my classes at Duke and students have loved it.

Rebecca: We’ve talked a little bit about framing things so that students take the practice seriously. What do we do about the students who just push back, it’s like “This is too much work. This is a lot of extra time…” or that sort of argument.

Michelle: I think that that’s another piece of that barrier to more faculty adopting this, not just the work involved, but realistically, student opinions. Student evaluations matter a lot to faculty life. And of course, we all want to have that wonderfully rewarding semester, not the semester where we feel like we’re at odds with our students. So I do think a little piece of this is we do anticipate sometimes worse and more pushback than what actually happens in the end. So I think we have a fair amount of sort of a dread factor when we go into something new like this. But that said, when students have an issue with more quizzing or more testing, here’s how those come out. I think, first of all, we do have to sometimes separate out the technology aspects of it from the quizzing or testing, per se. So, as John mentioned, this is one of the amazing things that educational technology does. But the flip side of that is, if you’ve ever used technology for education, in any shape or form, you know that it breaks down. And when it breaks down at a big class, that’s a headache for everybody, it’s a misery. So if you’re trying it, and things are not working out, you just got to figure out “Okay, how much of this is the assignment, the activity per se, and how much of it is that wonky quizzing thing that I got from the publisher and it fell apart, or students hanging up when they’re trying to dial in with the poll and fine tuning those. I mean, when I first used kahoot, I decided it would be a lovely idea to put four chapters worth of material into a 40-question quiz. And when you get used to as you know, that they took a long time, I mean, 12 questions can keep you going for a very long time, especially if you’re discussing… and some students got kicked out part way through and then they weren’t on the leaderboard and I could have set the whole thing aside. But really, that was more I needed to get the technology working. So that’s a big piece of it. I think that sometimes it is a perception issue with the timing. Now, those of us who are just all over this as a teaching technique, we like to do reading quizzes before we talk about that material in class. So chapter three is up on the syllabus and your chapter three reading quizzes due on Sunday night before we do that…, and that can provoke a fair amount of confusion and honestly griping with students. They say, “Why did I get tested on this when we didn’t do it… we didn’t cover it in class yet?” And what do you know, that’s another framing and communications issue. Once we know that that is why we’re doing that you get so much less on that side of things. But here too we do also ourselves have to follow along with what we say. If we say you’re doing this reading quiz so that we’re establishing a foundation. we don’t have to teach everything in the class itself. We could spend the time applying. Well, guess what, then you do have to do that. So if I assign you to do the reading quiz on chapter three, and then on that Monday morning I go, “Okay, we’re going to go over chapter three starting at the top. And I’m going to show you all the PowerPoints for all these things that you already read.” Well, yes, then student morale and student support will fall apart at that point.

Rebecca: Those are very good reminders.

John: Yeah, I’ve pretty much adopted this approach all through my class beginning when I first saw you present it. So I have students do the reading in advance, I have them take reading quizzes with repeated attempts allowed on those. And then in class, I have them working on clicker questions, and I and the TAs go around and help them when they get stuck on problems. But there is an adjustment and students, especially when I first started doing it, would generally say, “But you’re not teaching me” …and you do have to sell students on this a bit. One source of resistance is that when students take quizzes, they often get negative feedback. When they read something and they read it over again, it looks more and more familiar. They’ve got that whole fluency illusion thing going and they become comfortable with it, and they feel that they’re learning it… or similarly, if they hear someone give a really clear presentation on a topic, it feels comfortable. They feel like they’re learning it until they get some type of summative assessment where they get negative feedback, and then they feel the test was somehow tricky. But it’s a bit harder to convince them that actually working through retrieval practice, watching videos at their own time and pace, reading material as needed, and then spending class time working through those problems is as effective. I’m getting better at it. But it’s been a long time trying to convince students and I did take a bit of a hit in my evaluations, especially the first few times.

Rebecca: Do you mean learning’s not easy?

John: Students would like learning to be easy.

Michelle: You know, it’s funny, I think almost in a way… see what you think about this idea. But it’s almost a mirror image of our illusions as teachers, right? That I gave a wonderful clear lecture and I assigned wonderful readings and I saw students highlighting them. Therefore, students must have assimilated this knowledge and it must be in there. So I think it challenges our students but it also challenges us as well. It can be quite an eye-opening experience to running something like a Kahoot. And that brilliant point that I gave this great example for…. what do you know…. 7% of the students actually nailed it. And so we can all use a reality check… teachers and the students. And you mentioned re-reading, that’s another one where I’ve had some pretty intensive conversations with other faculty, I’ll kind of say, “Oh, well, and there’s this great research that shows that students tend to re-read when they’re studying and we know that from a memory standpoint, that is really, really ineffective.” And faculty will say, “Whoa, whoa, whoa.I want my students to be re-reading.” Of course, now when they say re-reading, they may be picturing deeply interrogating a text… annotating it… looking at it from a different perspective. And absolutely, that’s a wonderful part of scholarship. But that isn’t what we’re talking about. We’re talking about students re-reading as a study form and mistaking highlighting for deep interrogation. So, just like with the rest of knowledge bases versus higher-order skills. This is another words “both and” it cannot be “either or.” Yes, we want students re-reading in the right ways, but students or teachers cannot mistake that for learning sometimes.

Rebecca: So, let’s say you’ve just convinced all of our listeners that we need to be doing this, how do we bake it into our course designs?

Michelle: Well, I think that, really getting creative with this, and as I talked to faculty when I visit other schools or talk to faculty of my own institution, I just see all of these new ideas all the time about how to do this. Once you do get that critical epiphany of alright, a test doesn’t have to look like a test on the surface… it does not have to be the ritual of “Okay, you’ve got a number 2 pencil that’s breaking and I’m standing over you while you have a panic attack for an hour.” That’s not what it is—that it’s about the retrieval. Once you get that, all kinds of creativity opens up. So I’ve also started sometimes on the very first day with the syllabus quiz. So, especially if you have a smaller class, we all struggle with that “I spent all week writing the syllabus and it’s incredibly important, and then I’m getting questions about stuff that’s on it all the way through the semester.” So, really on the first day, what I do is I take a very light-hearted approach and I divide the students up into teams, just physical teams. Everybody’s got a copy of the syllabus, sometimes I fake them out and I say, “Okay, we’re going to go over the syllabus point by point…” and I say, “No, you can read the syllabus. We’re actually going to do this other thing which research shows will actually help you remember it.” So the task here is that each team formulates a set of questions… you can make just a few. like three… so a little bit of teamwork. So formulate three good questions off the syllabus. And you’d be amazed, they come up with questions that even I can’t answer sometimes without having to cheat. So they talk about it. And then of course, you go around the room in some arrangement and each team gets to ask the other team their questions, and if you really want you could keep score and everybody loves bragging rights for being the team that stumped the other teams and won the most points. So, it can literally start right then. With that idea that I don’t read stuff to you in this class, this is about you. And I’m also not piling a huge amount of work on myself either apart from being the moderator and the MC and having written the thing in the first place. I don’t have to write questions. They’re writing the questions and they’re answering them too. So those are some of the ways that we can do that. I think especially in our larger classes we do want to think about things like peer grading or peer review of open-ended question responses. We do want to take advantage of things like publishers’ test banks to set those up as reading quizzes. And you’d mentioned earlier about “Well, what about this not being suitable for upper-division classes?” I had an upper-division class that just wrap this semester where we had reading quizzes as well. It may have been an upper-division course but it also serve that purpose of “Hey, you take a basic quiz over the chapter on Sunday and that really sets the stage for us to have a more substantive discussion.” All of those things. are ways that we can do this. I think open-ended reflection as well… so tests that look nothing like a test but are still retrieval practice. You’d mentioned about reflecting on what you learned last class period. So this sometimes goes by the name “brain dump.” And I did a bit of this last semester as well. So, this by the way, I do want to credit the great website retrievalpractice.org. So retrieval practice is actually that high profile it has its own website. It’s an absolute treasure trove of ideas. So, with the brain dump the way that I did it, or similar to what it sounds like you did, every now and again we start off class with you writing down everything you remember from last time on a piece of paper. I had students then turn to their neighbor and compare notes and see what they came up with. And I didn’t grade those. This was not a heap of grading for me. But in the end what they did turn into me for accountability and a few points was a very short reflection, just on an index card. So, “What surprised you?” and I review those and they’re very eye-opening but again I’m not there to police or micromanage what they put on their cards. So, that’s retrieval practice too. There’s lots of different flavors that we can bring in.

John: I do something similar with asking students to reflect on the reading in each module we work through. But because I teach a fairly large class, I didn’t want to have to deal with all the index cards. So I just have them fill up a simple Google form. And then I can skim through it and assign grades much more easily than shuffling paper. But it’s the same basic idea and it’s worked quite well.

Michelle: Absolutely.

Rebecca: Do you have any other examples of the kinds of little tests that faculty can run that don’t look like tests?

Michelle: Right? So tests that don’t look like tests. Well, here’s another that was probably a little bit more practical for a small class, and this is how I ran this. It, like the brain-dump exercise the way I ran it, was also very cyclical and very student generated. So we started out each class at the beginning of each week with students generating a set of quiz questions based out of the assigned reading. So we didn’t actually, in this particular example, have pre-quizzes or something like that. But students came in knowing that they can bring their book if they wanted. But they would need to sit down and write for me three questions, whether short-answer or multiple choice. Then I can flip through those and I would really quickly put those together for a quiz that went out the subsequent day. So, we’re alternating between generating questions and then returning to those questions. And then I would have pass out. I told them treat this like a realistic test. Actually try to retrieve everything, but, you know, when times up, you’re going to get to go back to it. And then they would grade it themselves. So I didn’t actually do the grading either. And that is really great for spurring discussion. And “Oh, my gosh, I thought I knew the difference between reliability and validity, but now that I tried to answer it, I realized that I didn’t.” And then you can throw it back out to the student who now has bragging rights for having had their question selected and say, “Hey, what did you mean? What was the right answer? And why did you put that down there?” And at the end of the day, they kept that quiz too. So it was really very much in their own hands to do. So there’s that. Other creative ideas that I’ve run across over the last couple of semesters… There’s a great project out there, run by Bruce Kirchhoff at University of North Carolina at Greensboro. So I got the wonderful pleasure of talking to that group last year. And he and some colleagues working in the area of botany actually put together a freestanding, custom built mobile app that students could take with them that presented different kinds of quizzes over the sorts of things that botany students really need to know like the back of their hands like how to identify different plants and how to discriminate among different examples and they found some empirical evidence that this actually raised performance up quite a bit. I’ve heard too of another Professor put together some surveys in Qualtrics, the surveying software took advantage of its ability to actually text message people and send them the questions. So this was an opt in sort of activity and it’s one that they didn’t just have it run 24-7 because it was a little intrusive, but what it did is it sent students questions that they could answer at different intervals throughout the day, which also takes advantage of another principle from applied memory research which is spacing. So students are getting these unpredictable questions, they have the option to answer them, and they could be happening even when they’re not in class. So those are some other ways. Some of them look like tests, some of them don’t. But those are creative ways to get students engaged in that practice.

Rebecca: As a faculty member, we often advise students and mentor students who might be struggling in other classes, ones that we don’t have control over. Are there ways that we can help those students use this methodology to do well in those classes where it might not be embedded?

Michelle: Right. And that’s so great that you bring that in as well because ultimately that is what we need as teachers and that maybe circles back to yet one other piece of objection that I’ve seen… actually this time in a published article from a few years back that said, well are we doing too much for students? Are we scaffolding them too much so they’re going to grow really dependent on these kinds of aids like reading quizzes, and reading questions. But if we also have in our mind very intentionally, that what we want students to have at the end of the day is also something they can walk away with, I think that we do have to be very mindful of like, “Okay, let’s not create the impression that just because retrieval practice is so important that you have to sit there and wait for me to put together this specific kind of reading quiz for you.” So I think here, the really powerful message is once again that one students take this to heart… once they’ve not just been told this, that “Oh well, you should quiz yourself as a study strategy,” but they’ve seen that I believe in it to the point where I’m going to put time, energy, and work into it as part of my class. And maybe they’ve even seen the results… they’ve now had their own little before and after experience of what happens when I do this… that they can be more likely to take this forward. So having more faculty across the curriculum endorse this is a powerful idea that “Hey, this is how your mind works. This is how your brain works. Your brain doesn’t just soak up stuff that’s in front of it. You soak up stuff that you have to answer questions about.” That I think is going to be a powerful message. And there’s actually another article out there that I just absolutely love it was done by a psychologist named Pennebaker and some colleagues at University of Texas at Austin some time ago. They replaced high-stakes assessments in their Introduction to Psychology course with these mobile in-class quizzes that were done every day. And one of the things that they report in this article is not just that students did better in that class, but that certain subgroups of students actually showed improvement, specifically closing of achievement gaps, really, in classes that they took after the psychology course. And these were classes that we’re not even in psychology. So you think about that for a minute. How does that happen? Now, nobody knows exactly for sure, because this was kind of an unexpected finding in the study was my impression, but a real possibility is that when students have sat in this class and every single day they have seen the power of taking a quiz… of spacing out their learning… and attacking it in a very active learning approach… Once they’ve seen that happen, they’re more likely to go home and say, “You know what? I can do this in my biology course too. I don’t have to sit there with the teacher’s quiz. But, if I attack it in the same way I might get the same results.” So I think with those things, we can have students walking away with that enduring practice that we want them to have. And it is funny too, because it brings up one of the, I’ll just say it was a really heartwarming teacher moment that I had this last semester when I did bring in these Kahoot quizzes. In the run up to the first exam, I had done my fancy little in-class quiz and was kind of patting myself on the back of what a great leader I was. So I came in to, I think it was the class right before this exam, and I come plowing into the classroom and it’s dead quiet. And there is a student at the front of the classroom. He has commandeered the podium and the computer and what has he done? He’s accessed the Kahoots, which I gave them the links. You know, they’re out there for all the students to see. And he is running them and the rest of the students in the class are taking them just as seriously as when I was administering them. So they’re running through the questions again, giving it another shot. I didn’t tell them to do this at all. It was one of these moments where I just backed out and closed the door after me. And I came back in when they were finished. And so I think those are the kind of moments that we can set ourselves up for when we really do bring this into our classes and we get behind it and a really authentic way.

Rebecca: I think one thing Michelle, in the way that I asked the question, I had also asked it from the perspective of a student hadn’t had the experience of doing a retrieval practice in a class. So, if you’re working with a student, maybe outside of class or as an advisor, are there things that we could do to help those students adopt those practices, even if they haven’t seen it modeled for them?

Michelle: So how do you help students when they haven’t had some of these experiences on their own? And I think this is a part of a bigger package of goals that I think a lot of us should have in supporting our students to really put students in the driver’s seat. To say, “Yeah, you don’t have to wait for a certain style of teaching or certain subject material in order to succeed with it.” I see that isn’t really part of a larger package of growth mindset honestly. So what can students do to make themselves the masters of this? Now, I think that there are some old standard, very traditional, approaches that are worthwhile. And when you look at those approaches, sometimes retrieval practice is at the core. So it may not be a matter of trying to get a very unfamiliar set of terminology or anything for students before, but really getting them to look at some of these approaches in a new way. So things like you’ve ever heard the term SSQ4R or PQ4R, there’s a couple of those that have acronyms and the things they have in common are that they tell students when you sit down to study or read a text or prepare for a test, here’s what you do. You don’t just start reading from the top with no goal in mind other than “Oh, I want to get a good grade later.” What you do is first of all, you set yourself up with questions. Which is, after all, a lot like retrieval practice. Start with a question, say, “What do I want to answer? Can I answer that now?” And if not, why not? To read your text or go through your material very intentionally around this questions, and then there’s always this piece of recite or review. That’s what one of those Rs stand for in some of those traditional systems. And that means closing the book. That means closing the book and saying, “Okay, I’ve sat here with this material for this amount of time. What can I actually say about it at this point?” So, sometimes directing students back into some of those and saying, you need to adopt these strategies, which are completely teacher-independent, they’re fairly discipline-independent as well. That can be good. If you’re doing them for the right reasons and with the right approach in mind, that’s very good. Encouraging students to take advantage of the publishers’ companion sites… Now this is a little bit more of an uphill climb. That is where we run into “Well, if there’s no points for it, Why should I do it?” But encouraging students to say “Look, you already paid for this textbook. To be crass about it, you paid all this money, did you know there’s a website over here and if you interact with those materials in this particular way that’s really, really likely to pay off for you?” So, besides just ensuring that our students have what I consider to be these basic foundational pieces of knowledge about the mind and brain: that we remember through testing, we remember through retrieval and an active engagement. All students should have that, but those are some specific things that we can counsel students to use across all their studies that really should pay off.

John: Do you have any other suggestions for those faculty that are thinking about expanding their use of retrieval practice?

Michelle: You know, just to really encourage and support faculty who are starting this journey, as it sounds like that you all have, to really re-examine how we can bring in this incredibly powerful principle, and to really reassure each other that “Yeah, this is not about really just piling so much more work on yourself.” We can sometimes even just re-examine the assessments and assignments that are already in the course and so that’s kind of one last piece of practice that I think that a lot of us can really stand to bring in. And that was a big thing for me as well to say, “Well, if I’m going to administer these tests… we administer tests, we administer midterms anyway… what else can we do to increase their value as learning experiences… as learning events. So here’s another where I’ve brought in various forms of test discussion activities. Instead of standing in front of the class with that deadly the day-after-the-test class period where I say, “Let’s go over it” to realize, you know what? That probably will not work that well. But one of the amazing things that we have learned from the research on retrieval practice is that it’s not just in the moment in the taking of the test, where this advantage to memory happens. It actually creates a sort of a receptive window for learning and for review when we’ve just taken a test on something. So if you’ve got a midterm in your class already, well, hey, why not carve out the time after that test is up to say give it back to the students… like I photocopy the exams before I’ve graded them. They have no feedback on them whatsoever. And I hand them back to students as a group discussion exercise. I say “Alright, here’s a blank copy of the exam, your group can fill out this blank copy together, just knowing what you know, revisiting all those questions, having those good discussions about what you understood and what you didn’t. And I offer a little bit of extra credit for really good performance on that. And there’s other ways to work that out so it actually takes off as a small group exercise in class. But regardless of the specifics of how you make something like that work, the spirit of it is the same. That when you sat down to do this, to try to drag all this information out of memory, that is not an end unto itself. It should be part of a bigger picture of learning in the class. And it’s sort of an untapped vein of potential we have as teachers and that our students have as well and that we can access it regardless of what the discipline is or how the class is setup.

John: …and students see it’s very relevant. I started doing something very similar in my econometrics class last year following a suggestion from Doug McKee who had been doing that in his class. And it worked remarkably well… and turned what was normally a pretty unproductive class period where we’d go over the test… and the people who did well with just be really happy and pretty much ignore any discussion, and the people who did badly were just sitting there unhappy and not really being very receptive. But when they sit there, and they’re explaining it to each other, it seemed like a really ripe time for them to learn the material much more deeply.

Rebecca: I remember the first time that you implemented that. You sent me a text message with a photograph of his students taking a test and it looked very active. [LAUGHTER}

John: …and they were having fun.

Rebecca: Yeah.

John: One of the nice things that came out as I was wandering around listening to their conversations, and I was hearing people say, “Oh, yeah, now I see where that came from.” Or someone would say, “Well, when did we learn this?” And then someone else would say, “Well, remember when we worked on these problems?” …and it just helped them make connections… and the power of peer instruction is so remarkable. I’m going to do it in as many of my classes I can.

Michelle: That’s perfect. That’s what we want as teachers, and that’s what our students benefit from.

Rebecca: So as you know, we always wrap up our podcast by asking what’s next?

Michelle: Oh, wow. So I am looking forward to kind of rebooting a course that I have not taught in some time. My senior capstone course in technology, mind, and brain. And that is a fun one. But it’s one that is going to take a lot of revision, since it is technology and things change so rapidly, as we all know. So that is going to be a big part of next semester. I also have a crop of research projects in various angles on teaching, learning and educational technology that I’m really excited to be moving forward in the next calendar year. One of those… really foremost among them is a project on virtual reality for learning. We have an incredibly creative and dynamic team looking at virtual reality here at Northern Arizona University. They put together an amazing series of interactive exercises that are part of the organic chemistry course here and teach some of the challenging concepts in that extraordinarily challenging gateway course. And so we now have a whole set of data from students who went through and did this at varying points during the semester. We got their feedback, we’ve got all different kinds of psychometric measures that we gathered from them at the time as well. So, I cannot wait to be tackling that and looking at all kinds of angles on how this part of technology is impacting student learning in this course.

Rebecca: Sounds like some exciting adventures.

John: That sounds like a wonderful research project. …looking forward to seeing what you find.

Rebecca: It was a real pleasure to talk to you again, Michelle. Thanks for spending some time with us.

John: It’s wonderful talking to you. Thank you.

Michelle: Oh, likewise, always a pleasure to talk to your listeners.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.