Category: assessment

290. Transparency in Learning and Teaching

Posted in assessment, backward design, equity, evidence-based practices, feedback, first-generation students, higher education, inclusion, instructional communication, learning, metacognition, pedagogy, persistence, social justice, teaching, transparency
May 24, 2023

While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, Mary-Ann Winkelmes joins us to discuss what happens when instructors make their expectations transparent to their students.  Mary-Ann has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd.

Transcript

John:While instructors know what they expect from students, these expectations are not always clear to their students. In this episode, we explore what happens when instructors make their expectations transparent to their students.

[MUSIC]

John:Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John:…and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Mary-Ann Winkelmes. She has served in leadership roles at campus teaching centers at Harvard, the University of Chicago, the University of Illinois, the University of Nevada – Las Vegas, and Brandeis University and is the Founder and Director of TILTHigherEd. TILT is an acronym for Transparency in Learning and Teaching in Higher Ed. We are very much fans of the TILT approach and have referred to it often in workshops on our campus (and on previous podcast episodes). Welcome, Mary-Ann.

Mary-Ann: Thank you. I’m really delighted to be here with you. And I appreciate the opportunity to speak with you on Tea for Teaching.

John:We’re very happy to have you here. You’ve long been on the list of people we’ve wanted to invite. So we’re very pleased that you’re here today. Today’s teas are:… Mary-Ann, are you drinking tea?

Mary-Ann: I am indeed. And I’m drinking a Sencha green tea today. That’s my new favorite kind of green tea, Sencha.

Rebecca: Nice. I have English breakfast today.

John:And I am drinking a mixed berry Twinings black tea…

Rebecca: Hmmm.

John:…which I haven’t had in a long time. I wanted to mix it up a little bit today.

Rebecca: …mixing it up with mixed berries. So, Mary-Ann, can you tell us a little bit about how the TILT project came about?

Mary-Ann: Sure. This was years back, I want to say in the early 2000s, late 1990s, where I was working at the BOK Center for Teaching and Learning at Harvard University. And I was leading a seminar group discussions about teaching and learning. And we began to think about the question: “What happens when you tell students why you’re teaching how you’re teaching, just what happens when you tell the students more about your choices as an instructor, how you’re choosing to shape the learning experiences for the students?” And that’s not often something that we think about first when we’re thinking about what’s the content of the course. But we began to think about that a lot. And we had a kind of metaphor about the Wizard of Oz, and pulling back the curtain to show what was happening behind the scenes to build the experience. And then somehow through that conversation, the word transparency emerged. And that became the word that we used pretty regularly from that time on. When I moved to the University of Chicago, that was the word we were using, and it kind of stuck. So that’s kind of where it started. And it started alongside of my career as an educational developer. And it’s kind of been, for me, in the background or on the side, as something that I’ve been kind of tracking along with as a project. It’s still there, it keeps going. And just about a year ago, I began to work on TILT as my full-time job, which I’m really happy to be doing now because it gives me an opportunity, not just to do a guest talk here or there, or a keynote address, which is usually a one time-interaction. But now I have the flexibility to connect with institutions around a longer-term project. So if there’s a faculty learning community that emerges from a first talk that I would give, I get to follow up with them later and see what’s happening and check in with them. Sometimes I get to see the assignments before and after, which I really like. And I invite those now, because we’d like to publish some of those on the TILThighered.com website. And there are some schools that I’ve been working with in the state of Washington for several years now running with their TILT projects. And that emerged from a project we did with the entire state system of Community and Technical Colleges in Washington State. So I have opportunities now like that, where I can work with larger scale TILT projects that take more time, because this is my full-time job now. And I’m really happy about how that’s working, because I feel like it’s getting larger beneficial impact for students in a way that’s more efficient than when my full-time job was at an individual institution.

John:Could you give us an overview of the TILT framework?

Mary-Ann: Absolutely. So the TILT framework is meant to be a very simple tool that is a framework for an ongoing kind of communication among teachers and students. And in all of our studies, we asked teachers to use this framework in their own way at their own discretion, because we know that it’s not really possible to expect that people would do the exact same things with it. So our research is based on the premise that people are using this framework in their own way, at their own discretion, in a way that feels consistent with their teaching style. So there are three parts to this framework: purpose, task, and criteria. And what we ask in all of our studies is for teachers to engage students in conversation about three aspects of a particular assignment or a project or even an in-class activity. Before the students do a piece of work that we want them to complete, we’re asking for teachers and students to have a conversation about three aspects of the work before the students start working on it. And those three aspects are the purpose, the task and the criteria. Now the purpose kind of consists of two pieces. The first part is talking about the skills that students will practice while they’re working on the assignment. And then how are those skills useful, not just now in this course, or maybe in college and other courses, but how are these lifelong learning skills that will be useful for the student in their careers after college or in their lives ongoing? And then the second part of the purpose is about the content knowledge. What new information or what disciplinary information will the students be researching, or gaining, or applying when they’re working on the assignment? And how will that be also similarly useful to them, not just now, or in college, but beyond in their lives? The task, that’s the second part of the TILT framework, and the task is sort of about what are the teacher’s expectations about how students will approach the work? And for the students, it’s kind of like mapping out their game plan, like, what’s the first thing they will do? Will they Google something? Will they go to office hours? Will they go seek out a research librarian? Will they go into the lab and start mixing something like, what’s the first thing they’ll do? And then a sequence of what they plan to do after that until they submit the work. In an ideal world, the teachers and the students would have similar expectations about how that would go. In some cases, though, teachers have a pretty legitimate pedagogical reason for hiding that, that they don’t want students to know how to do the task. And I found this to be the case, particularly in fields where creativity is really important: performing arts, studio arts, even engineering or some STEM courses, where teachers really want students to cast about for a while and kind of use their imagination and see if they can come up with something unique, if not into the discipline, at least unique for the student to try to figure out some new process. And there’s value in that. When teachers want to do that, we did have some pushback from teachers in our original TILT research studies, where they said, “What happens if we don’t want to tell students how to do the work, like part of the task is for them to figure out how to do the work?” So in that case, we asked for those teachers to just say something like, “Part of the purpose of this assignment, in addition to the skills and the knowledge we’ve talked about, part of the purpose is for you to struggle and feel confused, while you invent your own approach to the question.” And we think this is what helps to preserve the student’s sense of confidence and their sense of belonging. Because instead of having that moment of panic of “Oh, no, I don’t actually know how to do this, I don’t even know where to start, I don’t know where the resources are, I don’t know what I’m doing. Maybe I shouldn’t be doing this, maybe this isn’t for me, maybe I shouldn’t be in this major, or in this course.” Instead of going to blaming it on themself or to questioning whether they’re up to the task at all, students can say instead, “I am totally lost right now. And that is exactly where I’m supposed to be. I know I’m on track. I’m doing great. This is the confusion part that comes before the clarity. And I know that because we talked about that and the professor said, this is part of what we expect to happen. This is intentional, this confusion, you’re supposed to feel lost right now.” So that’s sort of what we can say about the task. And the benefit of students knowing upfront what the task is, or knowing how they plan to approach the assignment or the project, the benefit there is that students get to spend 100% of the time they’ve allocated to work on this project, doing their best quality work, and they don’t lose time trying different approaches to see if this or that is going to work or looking for resources that aren’t what the teacher intended for them to be using. Instead of losing time, on the “how,” students get to spend their time so that what teachers receive then is most of the time what we’re looking for, which is “What is the student’s highest capacity right now?” Let’s see an example of the best work that the student can do right now, so we know where they’re at and we can bring them further so that they can advance in their learning. But if we accidentally end up in a situation where a teacher didn’t intend for the students to be confused, they expected the students to take a particular approach that they may have even mentioned at some point in class. So that’s why they think the students know that that’s the expected approach. I don’t want to say the correct approach but at least what they expect students to do. So if we think that students know how to do what we expect them to do, and the students don’t know what we’re expecting them to do, then there’s this chunk of lost time, where what we’re measuring then in the end is what happens after the students spend a chunk of their time lost trying to figure out how to approach the work, and then whatever time is left after that doing their best quality work in the amount of limited time that’s left. So part of the “task” piece of the framework is about what do we want to measure? Right? Do we want to be assessing the best quality work that students can do? Or do we want to be assessing what happens when you give a really varied, diverse group of students a particular assignment to do and you don’t give them 100% clarity about how to do it, and then kind of what you’re measuring is which students have, through no fault of their own, not encountered that information in their lives before coming to this course. And then you also get to identify who are the students that maybe because they had some other kinds of privileges that not all the students had, who are the students that can figure it out faster, because they come equipped with those privileges. So you can begin to see that this is an equity issue. So if talking about the purpose of the assignment kind of speaks to the student’s motivation, and to the value that they will gain from doing the work, and maybe to their ability to assess if they’re getting that value while they’re doing the work, the task speaks to even more of an equity situation where we’re trying to get all of the students to the same starting line of understanding of how to do it, and of having all the resources they would need to do the work to complete the work. And we want to make sure that students are all at that same starting line before they start the assignment. So that’s kind of the equity piece of this. And then finally, the third part of the framework is about criteria. We want students to be able to understand while they’re doing the work, how well are they doing. We want them to be able to make corrections, if they end up with a finished version that doesn’t look like what successful work would look like in this kind of a scenario. But if the students have never seen what successful work looks like, and they probably haven’t, because why would you assign them to do something that they’ve already seen many examples of; they wouldn’t be learning anything new. So kind of by definition, students aren’t going to know what successful work looks like when it meets this or that criterion in the discipline. So what we encourage teachers and students to do there when they’re considering the criteria is to offer students more than just a checklist or a rubric, because the words on a rubric or checklist might mean something different to the student who hasn’t done this kind of work than they do to the teacher who’s really immersed in this kind of work. An example I sometimes offer is, let’s say, I asked students to write up an analysis of a 15th century wooden painted sculpture of the Madonna and child from when I was teaching Italian Renaissance art history courses. In an art history course, the word analyze, like the tasks, the actions that you take when you are analyzing something, that’s a very, very different activity than analyze in the context of an economics course, or in the context of a chemistry course. But if the student hasn’t done this kind of analysis before, you can’t know for sure that they know what you’re asking them to do. So we kind of have to talk that through and students are going to need to see some examples of real world work in the discipline so that they can, with you, in a class meeting, talk about how do we evaluate analysis in this example from the real world, or in that example from the real world. And you won’t find any one example that matches every criterion of the assignment you’re asking students to do, usually, so you need several examples. The benefit of several examples is also that you can begin to talk about the relative success with which different examples are meeting a particular criterion as well. So once we’re in a conversation with students, and we hear back from them, that they’re telling us, what we had hoped they would understand about the skills they’ll practice and the knowledge they’ll gain, that purpose, about how they’re going to approach the work, the task, and about how they’ll know that they’re doing good quality work, the criteria, once we hear students telling us that, that’s the moment that things have become transparent. It is that activity of communication, that conversation with students about purposes, tasks, and criteria, that’s where the transparency comes from. And when we are done with that conversation, we know that students are at the same starting line of readiness In terms of their understanding of what they’re going to do, and also, in terms of their confidence that everyone has the resources that they need, in order to complete that work

Rebecca: What faculty believe is important for students to learn doesn’t always align with the goals of students. Can you talk a little bit about some strategies for bringing these into better alignment?

Mary-Ann: Sure, I think that this kind of speaks to the purpose part of the transparency framework. And often teachers are expecting students to learn something that is very valuable, we wouldn’t spend our time teaching things that don’t have a lot of benefit for students or that they would only use today and it wouldn’t be useful to them later in life. We like to teach things that have value. And so, when we are communicating with students about that value, we’re talking about the skills that students will be practicing. They won’t perfect them on this assignment, but they will begin to strengthen a particular kind of skill set. And they will gain some sort of disciplinary knowledge that can be useful to them later. And we know that sometimes disciplinary knowledge changes over the years as people discover new things and publish new things in any field. Sometimes that knowledge changes. But having some knowledge now does give you important value if you’re going to continue in that discipline or if you want to understand basic principles of a discipline that you might find useful elsewhere. So if students and teachers have a transparent conversation or communication, it could be a written communication, it could be something that they record and put on a website, it could be an asynchronous kind of conversation in an online course. But whatever form that communication takes, I think students and teachers when they’re on the same page about what the knowledge is, what the skills are, that are the focus of this assignment, students will feel more motivated to do the work, because they’ll see that it has benefit for them. And it doesn’t feel like a rote exercise, or just churning out another problem set or another art history analysis paper. There’s some value here that the students know upfront what that value is. And when the teachers hear the students reflecting back to them in this communication, that this is the value that they will be gaining, then we know that students have a kind of motivation to benefit from this assignment.

John:One other issue is that students have come up with some way of learning while they’ve been in elementary and secondary school. But those methods that they picked up are not generally the ones that are most effective. How can we encourage students to adopt learning strategies that they may be resistant to because for example, students, when there have been surveys of what types of learning strategies they found most productive, students often say they prefer to be lectured at, because they learn more from the professor that way. And also, many students don’t like active learning strategies. While they learn more, they don’t perceive it that way. Partly because of those desirable difficulties you referred to before, that when they’re struggling with something, it’s a little bit less pleasant than sitting there nodding and smiling and having everything seem to make sense. How can we encourage students to accept those desirable difficulties associated with learning so that they can learn more effectively,

Mary-Ann: I want to say that this is something that the TILT framework can definitely help us with. And this is not an uncommon phenomenon at all, I even find in my TILT workshops that I do with instructors, that instructors don’t love collaborative learning either. And in fact, many of these TILT workshops that I do will begin with some kind of a research review about “How do we know TILT works? What are the studies and what do they tell us and show us the data?” So we get off on this kind of role, where we’re almost in a traditional lecture format, where like someone’s delivering some information, and people are listening, and then they have questions about it. Or maybe they have challenges to say, “Wait, this doesn’t make sense, let’s talk about this.” And then I kind of switch the method that we’re using. And I’ll ask people to break off into small groups and begin to analyze a particular assignment and talk about where do they see the purposes, the tasks, and the criteria? Before I do that, I acknowledge the fact that we are shifting gears, and that we were doing fine with this sort of Q&A format. You know, look at the research and then think about it and talk about it. Ask questions. Why would I switch that up now? Like we were on a roll, we were doing great. Everybody was sort of on board. Why would I change that now? And so I use the TILT framework to talk about why we’re shifting gears now. What is my purpose in having you use this different method? So if it’s a peer learning method, as it is in the workshops, or as it might be with students in a class, we want to tell students: “Why are we now manipulating your learning experiences this way? Why would I do that to you when I know that sometimes students resist this, when I know that it can be uncomfortable, because I don’t personally always like to do it when I’m in a learning experience?” So if we can tell students, here’s why this is going to benefit you, because you don’t just hear it, but you have to struggle to apply it, you have to fit it not to the situation that I was talking about, where it all sort of makes sense when it rolls over you and you’re hearing it. But you now have to take the principle of what we were talking about, and apply it to this new unfamiliar scenario. And the benefit of that is that you will discover you will hit a barrier at some point in that process, where you will discover the exact piece of information that’s missing for you. You will discover exactly where you hit a barrier to your understanding. And you will have an opportunity right now, right here with me, the teacher in this class, to address that confusing point. And the benefit of doing that now, as opposed to later when you’re doing a graded assignment, is pretty obvious, you get the benefit of having the difficult learning experience in a safe environment that doesn’t lose you any sort of points on your grade. It doesn’t have any negative impact on you the way that it might if you waited until the end of the term to do some massive project and you hadn’t really done a lot of the homework or done a lot of the practices and so you didn’t really know what you didn’t understand until it was kind of too late to do anything about. So I think in short, what I’m trying to say is when we’re asking students to do something uncomfortable, that has a really solid pedagogical reason, that has evidence behind that, it is an evidence-based practice, we want students to know that upfront, because that then will increase their motivation to do it, because they see how they’re going to benefit if they do this thing.

Rebecca: One of the things that students often struggle with is when they start new courses with new faculty, and new ways of doing things and determining what the instructor will expect out of them and out of that learning experience. Can you talk a little bit about how the TILT framework could allow students to shift their focus to learning if it was adopted in the design of the course rather than just an individual single assignments?

Mary-Ann: Yes. And in fact, this is a way that lots of faculty are using the TILT framework, is to think about how do I TILT not just a single assignment, but a whole course. So usually, when people are introduced to the TILT framework, the original ask for all our research studies is would you please do this two times in an academic term, just twice? Because we wanted to see how little change could you make and have a beneficial impact on students’ learning, because small change is much more likely to happen than massive change. But once you’ve made that small change as an instructor, and you see that when you do this with two assignments, there’s some real benefit for students. And on the TILThighered.com website, there are publications by faculty who talk about not just how the quality of students’ work increases, but how the teachers experience in grading, or in responding to students, or in how many students will ask for an extension at the last minute, like these difficulties that teachers often face are diminished, while the benefits for students and the quality of students work increases. So once you begin to see this in the small scale of assignments, teachers then, maybe in the subsequent term, will think about what else could I TILT? Could I TILT in-class activities? Could I TILT a unit of this course? Could I TILT the whole course? And then the effects or the applications can grow. So we can apply this to a single assignment, we could TILT a whole course, we could TILT a curriculum in a department, we could TILT a program, we could TILT an institution’s learning outcomes and thread them through not just all the courses, but through all the co-curriculars too so that students might discover in their work-study job that they’re practicing one of the critical thinking outcomes, that’s a goal for the whole university that connects with what they were doing in their accounting class. And then we can even think about this in terms of a national framework of learning outcomes as well. So there are many scales at which you can apply that to a framework. And one of the things that I’m really enjoying about doing TILT full time, is that I can work with groups of schools, groups of institutions, so not just the Washington State group that I mentioned to you, but several weeks ago I was in the state of Kentucky where working with teams of teachers from institutions across the state, for the whole state system, to think about aspects of how do you map out a path for students to succeed in fulfilling their curriculum? And then how do you pursue that path? How do you complete that path? And in that case, we were using the TILT framework as a strategic planning framework to think about once we know what the plan is, like, once we’ve mapped out our plan for how students can effectively complete their degrees, how do we then communicate the value of that degree, not just to the students who are doing the degree, not just to the students’ families who may be contributing to the costs of doing that degree, not just the costs of the student’s tuition, but the cost of the student not being an earner in that family. And we want to communicate this to all the stakeholders, so the students, their parents, faculty, and staff at the institution, to state legislators who may be voting on packages of funding to higher education in their state, to individual grantors who might be funding particular scholarships. And we want to be able to communicate the value of this degree to every stakeholder in a state system that way. And the TILT framework is very helpful for thinking across multiple audiences, because that’s a pretty difficult task to communicate clearly to all of those different kinds of audiences. But it’s pretty essential for the success of higher education in this country. And so we spent a couple of days using the TILT framework as a strategic planning framework to think about how do you communicate the value of a degree? There are lots of ways that you can apply the TILT framework. Another example is I was working with a school in Texas over the summer, and they were TILTing their entire college success course. Many institutions have that kind of course in the first year, and some of them had TILTed individual assignments. And they decided they wanted to put the team of all the teachers together, and then subdivide that so that a smaller team of teachers was working on each week of the course. And then all the assignments and the lectures or discussions that would go into that week. And then we use the TILT framework as a larger framework to connect that whole course. So that from week to week, the purposes, tasks, and criteria were pretty clear. And students understood the path for all of their learning across that course.

John:Have you tried taking on the Florida Legislature? [LAUGHTER]

Mary-Ann: I have not.

John:That’s a real challenge, I suspect.

Mary-Ann: Yeah, I have worked with schools in Wisconsin. Last week, I was working with a school in Tennessee, right after a couple of their legislators were expelled temporarily. This kind of a framework, I think, can be effective in a lot of different higher education systems and contexts. That’s one of the beauties of it. Because this is something that teachers can do, starting right now, to complement any kind of larger, institutionally driven or federally funded program that might focus on student success. A lot of the time, those programs don’t necessarily feel like they’re directly connected to what faculty members are doing in the day to day in their classes. But using this TILT framework is something that you can do that will advance students’ success that will then make you feel more like you’re connected to these larger ongoing efforts that might be focusing on something that you don’t do directly, like targeted scholarship funding, for example. But that’s part of the beauty of the TILT framework is that it can work in many, many different contexts, and across different scale sizes of projects, as well.

John:And it works nicely for faculty because you end up getting work of the quality and the type that you expect, rather than getting student work that you find disappointing. And similarly, students end up doing work that they’re much more happy with, because they were not guessing at what the instructors want. So it just seems really, really logical. But it’s not always so widely practiced. Your efforts are really helpful for all of this.

Mary-Ann: I think one of the reasons why people might be hesitant to use the TILT framework, you don’t necessarily want to try doing something different that could suck up time that could take time away from delivering important content in the course, and what teachers have discovered and written about and published in the National Teaching and Learning Forum and other places you can see on the TILThighered.com website, what teachers have discovered is that if you take some class time to talk about the purposes, tasks, and criteria for a project before students do it, by the time that practice is completed, everyone has saved time; that time gets recouped, and students have learned a larger quantity of what we had hoped they we’d learn because when we deliver content in a course, we don’t know that students are absorbing it the way that we’d hoped or that they could apply it the way that we’d hoped. So I think by the end of the course, if you’ve used the TILT framework a couple of times, you’re in a situation where you’ve worked in a way that is more time efficient, somewhat, and you arrive at a place that, as you say, is more satisfying for students and teachers, because more of the time has been spent with the students doing the highest quality work possible.

Rebecca: I think one of the things that can be challenging for faculty initially is that if you’ve never communicated in this way, it’s hard to do it the first time, because anything you do the first time is difficult. But once you have a little practice doing it, it’s easy to adopt and expand across a course or across a set of courses.

Mary-Ann: That’s so true. And I think that the way that we’ve structured the TILT framework, it looks so simple, it’s a three-part framework. Applying it then gets you into some complexities that are important to clarify. I think you’re absolutely right, the first time we try anything that’s unfamiliar, just like for students, it’s more difficult. And then we kind of get the hang of it. And then it comes much smoother, and much easier. The TILT framework for starters, is pretty simple. It’s got three parts, right? And I think you could probably share a link to the one-page version of the framework that we give to students, that sort of spells out the framework: purpose, tasks, criteria, the knowledge and the skills. And then at the bottom, there are some of the evidence behind why we know this works and some footnotes, so that students can see on one page, this is a real thing. It works, it helps you. It is, in some cases, equitable, and it is probably worth giving it a try. And if you can see all that on one page as a student, then you might be more willing, especially in a context where a teacher is describing to you why this will be good for you, why this is a benefit for all of us. And then for teachers who have not encountered the TILT framework, when students can bring in this one pager that has some studies listed at the bottom and footnotes, they can see that when the student is asking me, why should I bother? This is actually a legitimate question. This is not a troublemaker student, this is a student who actually knows that they will benefit from knowing a little bit more in advance about this assignment that they’re planning to do. So we try to make it as easy as possible to implement. And then we also try to say only a little bit of this will make a statistically significant difference for students’ learning, so that you only have to try it a couple of times in a whole term. And you’ll probably see the kind of differences that we saw in terms of increases to students’ confidence and their sense of belonging, and their metacognitive awareness of the skills that they were practicing and developing. So if you’re doing anything new or different for the first time, yes, there’s some difficulty to that, but this one is a very, very desirable difficulty. [LAUGHTER]

John:We’ll share a link to that one-sheet document as well as to your website in general. And you do have a lot of research cited on your website. And there’s also some ongoing projects. Could you talk a little bit about those?

Mary-Ann: Yes, we are sharing all the resources that we possibly can on the TILT higher ed website, because we want for everyone to have access to this. Some of the places that benefit most are places that might have the least amount of money that is allocated for faculty development or educational development. So we want to make sure that this is accessible to anyone who would want to try it. And then the studies that we’ve done in the past, there are a few studies that have indicated to us a number of the benefits of TILT. One of the first studies we did was the national study we ran with the Association of American Colleges and Universities. It was funded by TG Philanthropy and my colleagues working on that project were Tia Brown McNair and Ashley Finley. And what we did there was we worked with a group of seven minority-serving institutions from across the country that represented every possible type of minority-serving institution, as well as a range of educational contexts like urban and rural, two-year, four-year, research university, really small in scale, large, residential and non residential campuses because we wanted for teachers to look at our results and see, “Oh, well, this worked for those faculty at that institution, and there are students like my students in that mix, so maybe this would work for my students. And in that study, we started with 35 professors at seven schools and we surveyed about 1200 students and we saw that, for the students who received the more transparent instruction, their competence and their sense of belonging and their metacognitive awareness of the skills that they were developing, those increased, those were higher for the students who got more transparent instruction than for those who got less transparent instruction. And then we also saw in that study some differences that showed us that while all the students were benefiting to a statistically significant level, underserved students were benefitting slightly more. So first-generation students in their family to attend college and ethnically underrepresented students and low-income students have slightly larger benefits than the benefits for the whole group. And then in our second study, we focused on how long does this effect last. So we worked with a group of University of Nevada – Las Vegas students. At the time we were working with that study, University of Nevada – Las Vegas had the most diverse undergraduate student population in the nation, according to US News and World Report. And we know from other studies, like Walton and Cohen’s, 2011, Science Magazine article, for example, we know that when students’ confidence increases, when their sense of belonging increases, they tend to persist longer in a course. So in courses that have higher levels of confidence and belonging, fewer of the students would drop the course, for example, more of them more likely to complete the course. And we wanted to see how long does that last. Is it just that course? And some studies indicate that this could last for a year. And what we did was we kept looking at the retention rates of these students to see how many of them were still registered a semester later, a year later, two years later. And we saw that by the time students were in their third year of university as undergraduates, those students who had received transparent instruction in one of their large gateway intro courses in their first year, those students were a little bit more likely to be still registered as students in their third year. And we’re now tracking that out to six-year graduation rates. So we saw that not only does transparency have a beneficial effect, it’s statistically significant, but that effect lasts for a good long time. And then in the state of Washington, we’re now writing up that study I mentioned with the Community and Technical College System. And I think that TILT is particularly helpful in that environment, because the population of community colleges and technical colleges is a little bit more diverse. And we have more students who belong to that underserved category of students, first-generation, low income, ethnically underrepresented. And what we’re finding from that study is we’re understanding a little more about how does transparency work, and I want to thank all of the researchers who are contributing to all of these studies too, because I’m not an educational statistician, so Daniel Richard, and Carolyn Weisz and Kathryn Oleson are contributing to this study and doing a lot of the analysis, along with help from some graduate students who have been working on this project over the years. What they’re discovering is that transparent instruction has a direct impact on students’ awareness of the skills that they’re learning, and it has a direct impact, similarly beneficial, on students’ sense of belonging. And then separately, sense of belonging has a direct impact on students’ metacognitive awareness and skills that they’re developing. So TILT has this direct effect. And then there’s this other effect between belonging and skill development as well. So we’re finding out more about precisely how TILT works for the benefit of students in these studies. And I think in terms of next studies, I want to be asking questions that really matter to populations of faculty and students around the country. So we open up the TILT research team to anybody who’s curious about this, and a number of faculty have asked about, can we say something more about how this works in an online setting, in an online synchronous setting in an online asynchronous setting, and we’ve got a few publications up on the website about that, but others are looking at that a bit more. And then we have another person who’s looking into just the impact on low-income students to see if we can find out more there about the details of how this works. And I’m really curious to see if we can work with large state systems, what can we find about the most time efficient, most beneficial ways to apply transparency and learning and teaching in community college settings. And I’ve also noticed that as I begin to do more work internationally, because I now have more flexible time to be able to do that, the colleges of applied sciences, like in the European Union, for example, they have a kind of three-year degree that is similarly focused on students’ learning something from their degree like they do here in a community or technical college that will lead them on a path into sustainable long-term employment and a career. So I think that this is going to be a really beneficial place to focus TILT efforts and to do some more research about how can we long term have an impact on not just students’ education, but how that is a pathway into a career. And I’m hopeful that we can find out more about that, like the longer long-term effect of TILT. But I’m also really open to inviting anyone who wants to do more research with the mountains of data that we’re sitting on, to discover something that is of interest to them about how students are learning, and how we can help students succeed more.

Rebecca: I really love all the resources and examples and research materials, worksheets, that are on the website. They’re really handy for folks who are starting out. We always wrap up by asking: “What’s next?”

Mary-Ann: What’s next for me, and then what might be next for teachers and students too. So we’ve talked a lot in detail about how TILT works, and how we know it works, and what more we want to discover about how it works. But I want people to remember that this is really a small effort, it’s a very easy lift that has a really large benefit from the size of that lift. And so I would really encourage teachers and students, if they’re going to do anything at all, even if they have no time to adjust any assignment prompts or to adjust anything about the way that they’re teaching or learning in a classroom. If you use any one single thing, I would say use that framework that we built for the students that has the footnotes at the bottom, and it’s called the “unwritten rules” and that framework, and I think you could probably provide a link to it, that’s what I would hope people would do next, just take that framework with you to anywhere that you’re communicating with your students. And the students will tell you how to make the work more transparent for them. Ask students what they see as the purpose, the task, and the criteria. And you’ll discover very quickly, very efficiently, how you can make that work more transparent so that all students are starting to do the work with the same understanding about what’s expected and with the same set of resources that they need in order to do it. So that’s what I hope is next for teachers and students.

Rebecca: And I hear all the faculty cheering about efficiency, and quick. [LAUGHTER]

Mary-Ann: That’s good. Yeah. So that would be the most time efficient thing to do, I think is to have students teach us more about how to be more transparent. And then in terms of researchers, I’m hoping that researchers will think about what can we learn more about? Can we learn more about what motivates students? Or what forms students’ sense of belonging? Is there anything in our survey data that would shed light on any kind of work you’re doing around that? Is there anything in our survey data that would shed light on more of the research on neuroscience and how that’s impacting learning? Or is there anything in the research that we have in our survey data that might help clarify what would be most beneficial for the very most at-risk students? So if we look at federal government statistics, National Center for Education Statistics about retention rates and graduation rates of different populations of students? Can we double down and look at those students with the very lowest graduation rates? And can we find something about TILT that would be the most beneficial for that population of students? To me, that’s a really important and interesting question. And then I really do want to be finding more locations where TILT could be useful, small scale for teachers and students, large scale for state systems or national systems to be thinking about how to apply this all for the good of students success, and for the satisfaction and time efficiency for teachers work as well.

John:If you’re finding these results of long-term persistent effects from just a single intro course, imagine what would happen if all intro courses use the TILT approach. I imagine the effect would be magnified if it was adopted at a broader level and it is being adopted at many institutions at a broader level.

Mary-Ann: I absolutely agree with you that applying TILT across the largest introductory gateway required courses at any institution would be probably the most efficient way to improve retention and graduation rates. Because if you go for the largest group of students as they enter, and you reduce the number of those students who might be thinking or doubting or wondering if they should continue, and if you increase the number of students who feel confident, who are aware of the value of what they’re learning, in terms of skills and knowledge, and if you increase the number of students who persist from the first year on, then that’s where you’re going to have the best success in increasing retention and graduation rates. I agree with you. I think that’s a really strategically wise place to invest TILT effort.

Rebecca: Well thank you so much. We’re looking forward to sharing this with our listeners.

Mary-Ann: And thank you so much for the opportunity to talk with you this afternoon, I really appreciate it

John:Thank you for all the work you’re doing.

[MUSIC]

John:If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

286. Grading for Growth

Posted in assessment, authentic learning, feedback, grading, higher education, learning, math ed, metacognition, teaching
April 26, 2023

Traditional grading systems provide incentives for students to focus on maximizing their grades, rather than their learning. In this episode, David Clark and Robert Talbert join us to discuss alternative grading systems that encourage students to recognize that learning from mistakes is a normal part of the learning process.

Robert is a Professor of Mathematics at Grand Valley State University and the author of Flipped Learning: A Guide for Higher Education Faculty. David is an Associate Professor of Mathematics, also at Grand Valley State University. Robert and David are co-authors of Grading for Growth: A Guide to Alternative Grading Practices that Promote Authentic Learning and Student Engagement in Higher Education, which will be published this summer by Stylus Publishing.

Show Notes

  • Talbert, R. (2017). Flipped learning: A guide for higher education faculty. Stylus Publishing, LLC.
  • Talbert, Robert and David Clark (2023, forthcoming). Grading for Growth: A Guide to Alternative Grading Practices that Promote Authentic Learning and Student Engagement in Higher Education, Stylus Publishing.
  • Nilson, L. B. (2015). Specifications grading: Restoring rigor, motivating students, and saving faculty time. Stylus Publishing, LLC.
  • Grading for Growth blog
  • Robert Talbert’s other blog

Transcript

John: Traditional grading systems provide incentives for students to focus on maximizing their grades, rather than their learning. In this episode, we discuss alternative grading systems that encourage students to recognize that learning from mistakes is a normal part of the learning process.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Robert Talbert and David Clark. Robert is a Professor of Mathematics at Grand Valley State University and the author of Flipped Learning: A Guide for Higher Education Faculty. David is an Associate Professor of Mathematics, also at Grand Valley State University. Robert and David are co-authors of Grading for Growth: A Guide to Alternative Grading Practices that Promote Authentic Learning and Student Engagement in Higher Education, which will be published later this year by Stylus Publishing. Welcome, Robert. And David.

David: Good to be here.

Robert: Thanks for having us.

John: Today’s teas are:… are either of you drinking tea?

David: I am.

Robert: I am drinking tea. I’m more of a coffee guy normally, but I figured for the occasion I would bust the tea out.

John: What type of tea are you drinking?

David: I have, because its afternoon, a really lovely almost white jasmine right here.

Rebecca: Nice.

David: …really delicate and anybody who cares about tea is going to be horrified that I sweetened it with honey,

Rebecca: But it is in a really nice polka dotted mug. It is a lovely mug. Thanks.

Robert: And I have cheap stuff from the grocery store, because that’s my brand. This is a Bengal Spice by Celestial Seasoning, and my wife and I are addicted to this tea. We drink probably four or five cups a day of it, though to be honest when it’s cold out, but it usually is [LAUGHTER] here in Michigan.

Rebecca: And I have Jasmine dragon pearls today.

David: Oh, nice choice. Both Jasmine.

John: And I have spring cherry green tea. It’s a very cold wintry day here as we approach spring in upstate New York. We’re recording this a bit before it’ll be released, so by the time you hear this, we should be having the beginning of spring both here and Michigan, I’m hoping.

David: We hope, yeah..

Rebecca: Your tea choice is definitely a big wish for spring. [LAUGHTER] So we’ve invited you here today to discuss Grading for Growth. Can you talk a little bit about how this book came to be?

David: I don’t remember, Robert, if we were in person, or on a Zoom call or something, but I remember you saying at some point, I got a phrase for you: academic book about alternative grading. And I said, “You know, I need a project for my sabbatical.” And then I think it just happened from there [LAUGHTER], and so…

Robert: Yeah, we were in person, David. David and I, we’re not on the same hallway, we’re kind of on opposite arms of a T-shaped hallway. And so we run into each other, sometimes literally, at the intersection of these two hallways, and that happened one day. It was pre-pandemic, I guess. And it was just like, “Hey, Dave, I have this idea, Somebody should write a book about alternative grading. Because David and I have been using various forms of alternative grading in our classes for a while. And he was like, “Yeah,” and that was it. [LAUGHTER] But I think we were also tapping into some stuff that we have been hearing and around our Math Department and elsewhere, our colleagues outside of Grand Valley that started growing even three years ago, just a growing interest or dissatisfaction with the way grading is working. And we’ve been trying some stuff and thought, don’t you know, the best way to make change is to just get your ideas out there. And so this seems like the right place, right time for us.

John: So what types of alternative grading systems do you discuss in this book?

David: A pretty wide variety. And actually, something I’ll say, I think, both Robert and I tend to try to avoid labels too much for these, like there’s useful names to describe different approaches. But we do things like standards-based grading, specifications grading, ungrading, standards-based assessment. standards-based testing, but we look at a really wide variety of alternative grading approaches. And we’ve actually tried to come up with a framework that sort of describes what their common elements are, so that we don’t have to worry as much about names as what the useful features of them are. We found that people can really get locked into an idea of what, for example, upgrading means, and it’s not always a super useful thing to argue about with them.

Robert: Yeah, I would echo that too. Being mathematicians, Dave and I are both really into abstraction and so we look at these specific things that we see, but we’re more interested in the big, general overarching unification principles like what are all these models that are all good, and all applicable, in different places, to different levels of success, and what do they all have in common? And we do discuss ungrading, we do discuss specifications grading, standards-based grading, and a whole lot of approaches that are kind of in the in-between interstitial space, mostly through other people’s stories. I think the heart of this book and David’s real amazing contribution, what you really spend your sabbatical doing, Dave, was interviewing dozens and dozens and dozens of actual real life frontline professors, nobody is in one camp exclusively, everybody’s using some kind of combination, some kind of mix of all these different ideas. And so I think, to me, what the real contribution that our book makes is showing how different things can look, you can start from these basic building blocks, but real people with all kinds of different classes and life situations and professional situations are making this work by listening to their students and adapting appropriately.

David: I can’t emphasize enough how much variety there is in the people we’ve interviewed and the disciplines that they’re in, the types of classes they’re teaching, like it was an amazing thing to talk with all these people and see, okay, they’re able to use different types of alternative grading but absolutely across the board: in labs, in huge classes and tiny classes, and upper-level classes, and absolute freshman-level intro classes. And so it’s been just fantastic to hear how everyone’s doing this and try to put it together into something that could work for anyone, that everyone will find something useful.

Robert: Yeah, no two of them are alike either. And what’s even more amazing to me is that we had to cut a lot of those case studies. And we almost have enough material for a second book just for the case studies. We’re going to keep it on the blog, I think, unless Stylus wants us to do it. But, I mean, there was a lot out there that we don’t talk about in the book. So there’s more where that stuff comes from.

Rebecca: You mentioned that these alternative methods have some common themes or common threads. And maybe it would be helpful to talk about those common threads in relationship to the problems with traditional grading that lead people to these other alternative methods. Can you talk a little bit about that?

Robert: Well, I think one of the places that you begin to see some of the shortcomings of traditional grading is just looking at the history of where traditional grading came from. And it’s sort of a weird and checkered, and very short, history. Many of your listeners might be surprised to know that the current system of points based A-B-C-D-E-F, 4.0 GPA type of grading system is only about 100 years old. It’s a relatively recent innovation in higher education. Higher education, formally, is about 1000 years old. 1088 was the first university and the first readily identifiable grading system that we know of now is like 1890s. So it showed up very, very late to the party. And immediately it was co-opted by Industrial Revolution era approaches to teaching and learning which treated students more or less as subjects. The word “grading” really comes from agriculture, when it comes to like grading Grade A beef and grading grain and flour and that kind of thing. And pretty soon it became entrenched. And we have a situation now where when you look at where grading has led us, it’s highly questionable whether grades really do what we want them to do. And it seems almost certain that they do some things that we don’t want them to do. They lead to issues with ranking and sorting students, pitting students against each other. It’s not clear to me that the statistical validity of points-based grading is even well established. I mean, we put points on things, but they’re not really numerical data in any sense. Computing an average of points across a system of exams does not necessarily tell you how much a student knows. And that, to me, in my view, is the fundamental issue that I have with traditional grading, why I moved away from it. I just didn’t feel like the data was telling me anything.

David: I’d like to jump in on that one, in particular. A thought experiment that I think is really helpful for anybody to do is, let’s say you give a big exam, and a student takes it and they get 60%. So, most systems, that’s basically failing. What does that 60% tell you? What did they know? What don’t they know? Why did they do that poorly? Did they actually do poorly? Do they know 60% of the things that you’ve covered on that exam really well, and the other 40% not at all? Do they have a mediocre level of understanding of everything? Maybe they understand absolutely everything great, and they had a terrible day, and they had to get the kids to Grandma’s house or their work had to keep them late. Maybe they’re sick. All those things are wrapped up together, in that 60% and you just don’t know what it means. And then to add to that, if that student learns, and they really show great effort, and they can tell you later on, “Hey, I’ve got all this stuff, and let me show you how, maybe on like a cumulative final, that 60% is still averaged in and it’s going to permanently weigh them down and their final grade won’t represent what they actually came to know.

Robert: Right. And so grading really cuts against the very process that humans engage in when they learn anything. When I’m learning how to play a song on my bass guitar, and I play it once and I do terribly at it, if I eventually learn how to play it, people should not be looking back at the first time I tried it, [LAUGHTER] they should be looking at the last time I tried or the maybe the best time I tried it. That’s not how recording artists record. It’s not how athletes are ranked, and so forth. And it’s all because of this sort of Industrial Revolution era routes where we have everything measured and sorted out and put together and it inhibits growth. It sort of poisons our relationship with students, it leads to all these extrinsic goals that students now have. Rather than focusing on learning and growth, students get the idea that it’s wrong to fail at things the first time, whereas it’s actually normal to fail at things the first time and then grow from it. And I guess that’s where the name of our book came from, I forget where that phrase “Grading for Growth” popped up. It was way before we were asked for a title for our book. But it’s like that’s what we really want. We want to have a system, even if we must call it grading, we want it to be a system evaluating student work that focuses on and encourages growth as you are learning because that’s one of the great things about being human is that we do grow and we do learn from our mistakes. So where is it in our assessment systems? That’s kind of the fundamental question we’re asking.

David: So it might be helpful if we talk a little bit about this four pillars framework that we have in terms of where we go with that. We kind of spent a while trashing traditional grading here, and I could do that all day. So what did you do instead? What are the things you want to do instead of these things you don’t? And so from my point of view, the most important thing I want to know out of a grade is what does the student actually know? What do they learn? And so, as I said, we have this four pillars model, we talked about four pillars that any good rating system should have and one of those is really a clearly defined description of what it is you’re assessing. So we call it clearly defined standards. I don’t want to catch on to the name standards too much there, but a description of what it is you’re assessing and what it is that matters in that assessment. So how do you know that the student has achieved it? And what is it that the student has achieved? So in that way, that takes care of this issue of what does that 60% mean? 60% of what? And if you’re grading based on a specific thing that you’re testing a student about, then you can say, okay, maybe they’re achieved that or they have not achieved that specific goal. And then we try to incorporate in our other pillars, this idea of feedback loops. So humans work on feedback, they work on trying things multiple times, they work on trying and failing, and having to come back again, and that it should be the ultimate level of what they know, that really matters. So focusing on feedback, rather than focusing on numerical rates. So focusing heavily on feedback, on what’s happened, and how that relates to the standards or the specifications, or whatever the description is that they were trying to reach, and then making that feedback actionable so the student has a chance to actually act on that, and either through a new attempt show that they’ve got that idea, or through revising previous work, show that they come to understand it in a way that counts fully, so that they’re not penalized for needing multiple attempts, just like Robert was talking about with his bass, we’re not going to look at well, the first time you tried, it didn’t work out, too bad. We want to know, ultimately, where were they, so we shouldn’t penalize multiple attempts at understanding something. And finally, it feels like a technical thing, but it’s actually a really big move, moving away from points or percentages and instead, if you’re gonna put a grade on work at all, to make that grade something that’s sort of a descriptor of the feedback, something that basically says, you’ve met this standard or you’ve met these requirements or you haven’t and gives the students an idea of where to go next: you need to revise this, you should try this again, you’ve made it, something like that.

John: I know you’ve experimented with a variety of alternative grading systems. Could you talk a little bit about how you view some alternative grading systems in your own classes?

Robert: Sure, I’ll jump in on that. So around 2017, I described this in the book in my origin story where I came from with this. I had just taught a calculus class and I had just a terrible experience with grades. One student in particular, a fantastic student, she was the epitome of what we’re discussing here. She was on a 10 day delay from the rest of the class and after those 10 days were up, she’d be at the top of the class, but the exam is today, and so her exam scores just went down, down, down, down, down. It was just an awful way for a student to experience my subject. And so I started casting about and I read this book that had just come out by Linda Nilson, who is a legend in the area of professional development, and my former boss at Vanderbilt University. And it’s called specifications grading, and I said “What is specifications grading?” So I read this book, and I was converted on the spot. And I’ve been using specifications grading in almost all of my classes, except for one instance of ungrading last year, ever since. And so, I’m in various forms, and always tweaking, always experimenting, and trying to change things up. But, specifications grading is exactly along the lines of these four pillars that David is mentioning. There are clear content standards that we call specifications. The idea is not to give like points, but marks, to use a sort of a non-American term for those, marks that indicate progress, like you’ve met the standard, that’s what you get instead of a 10. Or you’re progressing, that’s what you get instead of a five, and giving lots of helpful feedback and especially letting students retry things without penalizing them for retry. To me, that’s like the thing that drives the loops. That’s been my main sort of area ever since for the last several years,

David: I’ve used a pretty wide variety of different alternatives. It’s come to the point for me where I think about all these different types of alternative grading systems that have names like standards-based grading, or specifications, or ungrading as elements to put in and they may be appropriate in some classes and not in others, depending on a whole bunch of factors. For example, I’ve taught intro level classes where I use a lot of standards-based grading where that means I’ve sort of divided up the topics into fairly fine-grained standards describing what a specific skill is, and what it would look like for a student to achieve it. And then the grade is essentially based on have they shown me that they thoroughly understand that, they can do the thing described in each of these standards and grades are based on meeting or not meeting those standards. And that can look like pretty traditional assessments, quizzes or exams or homework or such things. But instead of getting an overall grade, they get a mark, like Robert was talking about, for each standard. Yup, you’ve achieved this. Nope, you haven’t yet. Maybe they need to do it a few times. But that’s one element. It’s really useful for sort of discrete skills, intro level things. I’ve used specifications in a similar way that Robert is talking about. One of the strengths of specifications is that you sort of look at a student’s work holistically and say, “Have they overall shown me they can put these ideas together, understand the concepts, use all the different things that I think matter within this assignment,” and that works really well on written work… We’re mathematicians where students write proofs, it’s a detailed explanations of why something’s true, or in a project or in a portfolio or something longer than that. It’s really useful, maybe in upper-level classes or in places where I want students to show synthesis. And I’ve gotten pretty deeply into ungrading, or at least as I use the word ungrading, meaning removing grades entirely, even marks, in the way we’ve been talking about as another element that can be useful either in an entire class or just as individual things to do within some assignment. So I’ve covered a lot of this, I continue to push on what’s most appropriate in different classroom situations and different student situations.

John: I’ve tried some of the same things too. But one concern from a lot of faculty who have tried mastery grading systems or mastery quizzing and those types of things, is that it can put a lot more work on the instructor to do the grading on multiple attempts. In Linda Nilsen book, she suggests giving students a certain number of attempts, but limiting those. How do you keep the work manageable so that you’re not spending all of your time grading additional attempts, as students are working towards mastery.

David: Yeah, the thing you mentioned from Linda Nilson’s book, attempts without penalty doesn’t mean attempts without limits, and that is really important. And yeah, limits like that are one way to do it. There can be limits on number of attempts, or on frequency of re-attempting something or revising something once per week, once total, something like that. Those sorts of limits can just be really useful. And I know, I always encourage new instructors or people just starting to use these systems to start out limited, and then add flexibility because that way, they will be able to do it if they need to, and not do it if they don’t want to.

Robert: Yeah, you know, another thing too is, not just in the reassessments, but in your overall design of your course, including the grading system, you got to keep things as simple as humanly possible. This is something David and I just hammer home constantly, like every other paragraph ends with keep it simple, because people who want to mess around with grading systems are people who like tinkering with systems, and I am one of those people, and I know that when you start tinkering, it’s very hard to stop. And you end up with this massive Rube Goldberg like device that is your class and to you, it looks beautiful, but when you face it towards students… I mean, the first time I did specs breeding, I had 68 learning objectives that students had to meet throughout the semester and it was a nightmare. At the first of the semester, I thought, I’m a genius. I mean, look at this beautiful grading system I’ve devised. It’s going to revolutionize everything. And the only thing it did was give me a grading jail for four months. Just keep it as simple as humanly possible, and you’ve got to cut the work off at the source, I think. David and I wrote a book about grading, but we don’t like grading. I mean, nobody really likes the process of it. But it can be made at least a fulfilling undertaking if you are orienting it towards growth. But you still don’t want to be doing it 8-10 hours a day. So keep things as simple as humanly possible and put some limits on that. It’s okay to create a little bit of scarcity when it comes to the reattempt side of things. The feedback needs to be helpful that you give to students, but it doesn’t have to be incredibly lengthy. It might just be a couple of sentences, it might just be: “You did this really well, and this needs work.” …to kind of keep it right to the point, and then that helps the students who are on the other end of this too also experiencing a workflow possibly overload. And I worry more about them than I worry about myself. And so this makes it more likely they’re going to read your feedback if it’s shorter, more concise, and it makes it shorter to give.

David: Yeah, I want to emphasize that it can be a lot of work for students as well. Something that I suggest a lot is, if you want to use a system that emphasizes these pillars, especially reattempts without penalty, you’ve got to then think about those re-attempts as part of the regular workload of your class. Really what it means is you’ve got to think about the process of learning, that whole process, is part of the time that students are going to have to invest into it, and that you need to value that in your timeline for the class. So I like to think of maybe if you expect a student to work, however many hours per week on your class, account for whatever time each assessment takes, add another half for how much they might need to do revising or reassessing or reworking on things. And that can lead to really difficult decisions. You’ve got to cut stuff out if that’s going to happen. I don’t know, basically, anybody who looks at a classroom teaching and says, you know, there’s not enough in here. And so you got to make those difficult decisions, but it is worth it. Because what comes out of it is a student knowing things that are still in there way better than they would have otherwise.

Rebecca: I think one other barrier that faculty sometimes face is learning management systems, and how they guide you towards particular kinds of grading systems and evaluation systems. I wonder if you can share some tips that you have from your own experience of dealing with a learning management system to help you and your process rather than getting in the way of implementing some of these techniques.

Robert: We’re a Blackboard campus, and we’re making a transition right now and I’m honestly not sure how the new system handles those. But in at least the next to most recent version of Blackboard there was a thing called a schema, which is basically a way of going into your gradebook and defining a way to map number inputs to text outputs. And so I can set an assignment up to be graded on zero or one point. And then I can only enter in zero or one. But I could tell Blackboard if I enter in a one put “meets expectations” in the gradebook, and if I’ve put a zero put “does not meet expectations” or a happy face emoji and a frowny face emoji or whatever I feel like doing. And I have zero experience with any other learning management systems over the last 15 years, so I can’t really speak to that. But I know that many memory management systems allow you to customize the way that your numerical inputs present themselves in the students’ gradebook. And if you can find out how to do that, that’s the way to go, as far as I’m concerned.

David: And even if you can’t, the thing Robert mentioned, that essentially everything is a zero or a one, these systems all want to use points. So just refusing to play that game by making everything a zero or one. Even if students are seeing the zero or the one, it’s a lot easier to interpret the zero or one as success or not, than it is if you have points showing up somewhere else, and you’re trying to convince students No, no, no, when I put in a five, that meant something that doesn’t mean five points, that’s not going to fly. So limiting it to just a zero or one, a complete or incomplete, something like that works pretty well. That’s also true for spreadsheets. So if you just want to keep your info in a spreadsheet, they still want to work with numbers for the most part, but you can think of everything as a zero or a one. And then you start to change your mind from averaging or totaling things to counting. So I’m counting how many standards a student has completed, I’m counting how many assignments they’ve met the specifications on. And that’s really the same as saying, how many ones do I see here? One last thing is, everybody seems to have a different LMS. Even those of us who use the same one don’t quite use the same version of it [LAUGHTER]. So the best thing to do is to find someone who knows it really well and talk with them and say, “How do I make this type of thing display?” …and they probably know a way,

Rebecca: I’m gonna have this conversation with John about our LMS.

John: We’re using Brightspace from Desire to Learn, and it does have those capabilities.

Robert: And one thing I would say is that these days, as the ideas about alternative grading get more and more airplay, I think the tech companies are starting to listen to these things. I mean, you’re actually starting to build these things in as a competitive device like “Ah, you should adopt our LMS, because we can let you do ungrading and you don’t have to deal with all the hassle of hacking your own LMS. So I feel like ed tech companies should be paying attention to all this great interest that’s swelling up, it’s like this is a way to earn customers, honestly.

David: Something I’lll say most LMS’s do seem to do pretty well is to allow you to get feedback in a variety of ways. And so if you can de-emphasize that grade portion, just 0-1 or hide it or something, especially like I accept all assignments through Blackboard nowadays, giving feedback’s pretty easy on that. So that, in my mind, is a benefit.

Robert: Yeah, Blackboard does a really well, actually, that if you submit a PDF into Blackboard as an assignment, it’s easy to leave comments on it, and it keeps all the versions of it. So you can go back and easily see the students trajectory from the very first draft that they do all the way to the present day. And I think that’s really powerful, actually.

Rebecca: I’ve used rubrics to set up essentially specifications as well, like it’s met or not met.

David: Absolutely, yeah. If you know what you’re doing setting up rubrics, I don’t, that’s the problem for me. But if you know what you’re doing, that’s fantastic and it can also save you time.

John: Since the pandemic, there’s been an explosion of interest in ungrading. Does this approach work as well in all contexts, and for all students?

Robert: People are starting to use ungrading to mean anything other than traditional grading. And I think that’s leading to a lot of confusion, honestly. I was working with some teachers recently who were saying that, “Oh, I’m using ungrading but don’t have time, I’m being overwhelmed by all these quests for reattempts.” And it turns out, they weren’t using ungrading at all, they were totally grading thing. When we say ungrading, we mean like a specific approach to evaluating student work where nothing gets a mark, literally ungrading. You get a letter grade at the end of the semester, because most places require that, but it’s all based on a term that’s come up recently, that I really prefer, is collaborative grading. You’re working together with the students throughout the semester to talk about the quality of work, but nothing is getting a mark on it. I have a comment for you and you can either choose to iterate on that comment with a feedback loop or leave it alone. At the end of the semester, you’re going to sit down together and collaboratively determine what your course grade is based on the body of work that you’ve accumulated in some sort of portfolio situation with some clear standards for what an A would look like, a B would look like, and so forth. So that’s what we mean by ungrading. The explosion of interest in “ungrading” may or may not be all about ungrading the way we’re referring to it. So I just want to be clear about the terms before we answer this question.

David: And just add even another thing, it’s both that sort of umbrella term and what Robert just said and the general philosophy and a buzzword that people attempt to apply to random things they use. It’s unclear what someone means when they say ungrading.

John: When we’ve been talking about it, we’ve been talking about it exactly as you described as an extreme form of alternative grading where there are no grades other than one that is decided in consultation with the student at the end of the semester. And sometimes at midterm if that’s required in the institution. From what we’ve seen, this is used quite a bit in humanities fields, but it tends not to be used very much in the STEM fields. What are some of the barriers or some of the concerns that might be raised for ungrading in terms of in what contexts it may not work as well.

Robert: I have tried to fully ungrade a course once, and I might use it again. But the issue that I was coming up with with my students, is what I say is that ungradiing works precisely as well as students’ ability to self assess. If you have students who are struggling with the ability to self assess, or they’ve never tried it, or they’re new students, or younger students who maybe are still emerging with that idea… the whole thing about ungraded is that it’s predicated on students looking at their own work and self evaluating. And if that’s an issue, then I’m not totally sure that ungrading, as we’re defining it, best serves that population. I know I struggled with it. And this can be more me than anything else. Maybe I just don’t know how to draw students out and I need to work on that. It’s an area of growth for me, certainly. But if you have a population of students, for whatever reason, whose ability to self regulate or self evaluate is questionable, or kind of low, we’re not going to say bad students, because we’re all growing… it’s the whole point of our book… if you deploy ungrading with that population, you might be sort of targeting the weakest point in your class. And so maybe, in that case, some marks wouldn’t be such a bad thing, like use a specifications grading approach where the marks are just like: you’re progressing or you’ve met the standard. Just some kind of simple mile marker that shows students where they are. When I used this, once, it was an upper-level class, it did have a lot of writing in it, a math class. I would leave comments on their paper and I would just get questions like, so does it meet the standard or not. And I didn’t like to sort of be telling students this, but I felt like this is the way I helped them. I said like, “hasn’t met our standard yet, here is what you need to do.” So I thought, if I’m going to do that, why don’t I just put a mark on it? It seems like I’m just beating around the bush and trying to be cool by not giving marks and it’s like I’m thinking more about myself than I am about the students at that point.

David: I think that a general thing that everybody needs to think about when using any kind of alternative grading is, this is something new for students… almost guaranteed. And like any other new thing, it needs time to learn, you need to encounter it a few ways, you need to try to make sense of and probably fail initially, and then come back and you’ve got to work to understand it. Ungrading is sort of an extreme form of that. There’s more that needs to be understood about how it works, there’s more skills that need to be built. And I think, as instructors, we need to think about that and think about teaching how our evaluation systems are working and how the grading is going to work. And think about what skills we need students to build up in order to successfully engage with those. And ungrading asks a large amount of that. And I say this, as I very much enjoy using ungrading also in some upper-level math classes and I think it’s worked pretty successfully for me. But this sounds funny to say about something called ungrading, it requires scaffolding, it requires helping students build up that ability to self evaluate, advocate for themselves, to be able to understand what it is that matters in that particular system. But what I just said is also equally true for standards-based grading or for specifications grading, but with different skills that they’re building up. It’s also true for traditional grading, although it feels more familiar to students, but it still requires some scaffolding for them to understand what exactly is going on in the evaluation in this class.

Rebecca: Can you talk a little bit about some of your own experiences with ungrading and how you have done some of that scaffolding, what some of those early stages of scaffolding might look like, to understand the system.

David: So initially, I was having I think it was three check in meetings per semester, where I would meet with every single student in my class, like a 10- or 15-minute discussion, and we would talk about where they’re at. And that inevitably brings out students’ questions, and they don’t understand this part of they don’t know what that is about. And so that was one thing that I did, right? Those were the collaborative aspects of ungrading, and it was overwhelming, it was too much to meet with every single student that often. I still love being able to meet with students. And I use that as one of the tools in the toolbox. But nowadays, to sort of help build that up, I have a system where… this is actually done through homeworks for me… it’s like periodic homeworks and there’s some mathematical content. And there’s also some little additional thing where I ask a question, and early on, it might be something like, “Okay, what is a specific goal related to… and then I might have some of the math related content in the class or as a specific goal you’d like to work on related to this.” And that’s a fairly small thing. I ask students to say something along the lines of “I really want to work on improving my understanding of this one thing” and I can either respond to that as “Okay, great. Here’s the way I can suggest that you work on that,” or I can respond to it as… well, I don’t say it this way, but… “I can see that you’re having trouble elucidating the goal and so let’s talk about that” instead. And then another week or two down the line, there’s another maybe slightly higher-level task having to do with assessing themselves or thinking about their progress in the class. And so they’re scaffolded and I can, at each stage, see if a student is succeeding, in sort of the way that I’m looking for them to be thinking about their progress. And if not, then I can pull in a, “Hey, let’s actually meet and talk about this. And then I’m gonna give you a task to practice with that.” It’s really individualized, which can be a tough thing. But for me, it’s really fun to actually see students grow in that particular way, in addition to growing in the mathematical content.

Rebecca: So faculty that want to move to less traditional methods, what are some small steps a faculty could take to get started?

Robert: Well, I really appreciate that question. Because we don’t ever want to give the impression that you have to go all in on one particular grading system in order to like, be cool, or be a good professor or whatever, because everybody’s in a different position. Some people may not be in a position to ungrade, and we are never going to come out and say, “If you really care about students, you will do ungrading or you’ll do standards-based grading or whatever.” It’s like people care about students all over the place and just can’t or have mitigating circumstances. So I would say we laid out these four pillars: clear content standards, helpful feedback, marks that indicate progress, and reattempts without penalty. If a professor takes one of those pillars, and kind of drills into it on just a handful of assignments, that’s definitely progress in the right direction. So for example, you could take your next test you’re gonna give, experiment with letting students reattempt it, or reattempt different versions of it. So go one test, one class, allow reattempts without penalty, or give feedback that really gets to the point, not gives points, but gets to the point of what’s being done well and what isn’t being done well. Just pick one of these four pillars and just go with it. There are some small scale methods you might think about, let’s say, instead of grading your final exam, how about ungrading your final exam. Just say you need to lay out some standards for what constitutes successful work on your final exam and maybe that’s like 70% on the content. And then you got to write some reflective essays, and say, if you don’t meet expectations on the final exam, that’s a minus on your course grade, or something like that. So you can roll in aspects of alternative grading systems, whatever seems to resonate with you on small-scale individual assignments. And that’s really good progress. It’s really good data for you as the instructor too. You run these experiments, you ask students how it went, and then you just make notes and you iterate on that. It’s exactly the same thing that we asked students to do in these grading systems. You try something, you get feedback on it from a trusted third party, you iterate on it and try again.

David: I want to add one thing, because this is something that I always see happen when people first start using some kinds of alternative grading. Just for yourself, write a really quick list of what you think the major topics are in your class, aim for 10 to 20 things that are the important things to get out of the class, and then go pick some of your exams or quizzes, or whatever, and match them up. “Oh, on this one I was addressing these things and on this one, I was addressing those things.” And that can help you actually think about, “Oh, you know, I’m not really addressing this thing in my assessments at all, that I said was really important.” Or “Holy Moly, I have hit this one thing over and over and over all the assessments at the cost of not covering a bunch of others.” The first time you do this, even if you’re not showing students what the standards are, it can be amazing. “Wow, I’m not assessing what I say matters.” And that can adjust just how your assessments are focused.

John: Sounds like you’re advocating a backwards-design process where you start out with clearly defined objectives, and then you make sure that there’s alignment. And also, I think it might be important to make sure that students see the connection between those things, and that it’s a transparent process, which I know is something you’ve advocated in terms of your work on assessment.

David: Absolutely. And sharing your reasoning with students I think is very valuable, bringing something in from above and saying we shall now do it this way, because I said so, doesn’t work and you’re gonna get a lot of pushback, but talking with students about why you’re doing things and how it’s good for them, that makes a huge difference.

Robert: And don’t call it an experiment, okay. [LAUGHTER]

David: Yeah.

Robert: I’ve read some syllabi lately. It says, I’m going to be experimenting with this experimental new experimental grading system. And of course, you know what that is going to sound like to students… that I’m the subject of an experiment. What am I doing here? And so you’ve instantly killed your buy-in.

David: I’ll actually say, I don’t even give names to what I’m doing. So I am ungrading a class this semester, I have never used the word ungrading or any other name for it. I find that giving a name to things like that can help people sort of reduce it down to like a one dimensional idea, rather than engaging with what it actually is. I’m just “Oh, this is what we’re doing, and here’s why we’re doing it this way.”

Robert: Yeah, I have to share this story. When I started doing specs grading I did actually put we are using a thing called specifications grading on my syllabus. So, I got a call from our research office on our campus, they had gotten a call from the parent of one of my students complaining to them that I was experimenting on their child without informed consent, that I needed to give an informed consent form to my students before this happened. And I thought, well, that’s a weird flex from a parent and they call up the Office of Research Compliance [LAUGHTER] and complain about the Professor, but as you can just see even just the barest mention… Just tell students what you’re doing, and why you’re doing it. If I have to name it, nobody cares what the name is, it may not even have a name. And all this we mentioned before, that most of the people we interviewed for the book are not using any sort of canonical form of anything. It’s just like a little bit of this, a little bit of that, like cooking in your kitchen, you don’t use just one spice, you combine them, that’s where you get your unique flavor of what you’re doing.

John: Would it be fair to say that, given the title of your book and focusing on grading for growth, that one of the things you’re addressing is how to shift students from focusing on trying to maximize their grades to maximizing their learning?

Robert: Yeah, absolutely, yeah. I think a lot of frustration that I know I feel and I hear from other faculty is students tend to spend an order of magnitude more time talking about what they have to get on exam X to get grade Y than they talk about the actual content in the course. And we teach math, and I think some of the students who struggle the most with our math are doing like these amazing statistical calculations to try to figure out what the average has to work out to be. It’s like, “where was this in my actual math lesson?” I remember riding the bus home one day, from our downtown campus back to where we live, and I overheard this conversation and I said “Oh, this is this has got to come to an end.” [LAUGHTER] And absolutely, we want students to focus on their growth and I think students want to focus on their growth, too. I mean, students really, in the end, don’t want to think about all this stuff. They want to think about, “How am I growing as a human being? Is when I’m doing really meaningful? Where am I as a learner? Do I have any value in this vast educational system that I’m entering into?” And so if we can even just orient one small thing, like our grading system, towards convincing students that they have value, and are human beings in a stage of development, I think that’s worth it.

David: And that means we need to believe it, too. So I like to tell people who are thinking of using an alternative grading system, think about why you’re doing it, and what your students are going to get out of it. But definitely approach it as like, “Okay, I care about my students, and I want them to succeed.” And we’ll sometimes talk about incentives… grading systems set incentives in a class, and that can feel sort of like you’re trying to mess around with people and then incentivize you to do things a certain way. Traditional grading systems set incentives that are really kind of perverse towards learning. And if we can change those incentives, or just remove some of the ones that aren’t as good, that’s really a lot of what we’re aiming for here. And yeah, just respecting how people naturally want to learn anyway, that just allows them to do what they are as human beings going to do anyhow, or what they would naturally do.

Rebecca: So everyone wants to know, when can we have your book in our hands?

Robert: Well, it looks like on the Stylus website, it says July. So that’s what we’re going with for now. [LAUGHTER] I guess it’s sort of out of our hands at this point. We have one more round of copy edits to do. But it says July, you can preorder it now through the Stylus website and should be in your hands this summer.

John: My preorder has already been sent in. [LAUGHTER] So I’m waiting for a copy.

David: Awesome.

Robert: Awesome. Thank you.

Rebecca: And you mentioned earlier something about a blog. Do you want to tell us a little bit about that?

David: Yeah, if you’re interested in some of our ideas, go check out our blog, it’s gratingforgrowth.com …all spelled out. So, Robert and I started this sort of as a way to workshop some of our ideas for the book. So you can see proto-ideas. And we’ve also posted some examples of some of the case studies of some of these interviews that I have done with people using a lot of different alternative grading systems there. Every week, we post something new, or we have guest posters. There’s a really huge variety of ideas that show up on there.

Robert: And if any listener wants to contribute a guest post, we are accepting applications at this time… not really applications, just reach out. And we’re looking to hear. Now that the book is kind of close to coming out, we’ve had this discussion, you and I, David about where does the blog go from here once the book is out, and I feel like a great use of our time is to elevate the voices of other people. I mean, there are so many people out there doing great work, who are not education rockstars, they’re not on the lecture circuit or whatever. They’re just rank and file heads down people doing great work in the classroom. I love those people getting their ideas out there and seeing what they’re doing. So to me, that’s been a really rewarding experience.

David: So gradingforgrowth.com.

Robert: Right. It’s actually a substack. You can subscribe to it. It is free. It’s never going to cost anything and it comes every Monday. Except for taking planned breaks for holidays, we haven’t missed a Monday in two years.

David: Well, now you’ve cursed us, Robert. [LAUGHTER]

Robert: No, I put the pressure on me because then I got to step up. I think I’m next, actually. No, you’re next.

David: Oh, crap. [LAUGHTER]

Robert: Sorry, Dave.

John: We will put a link to that in our show notes.

Robert: Great.

John: Are there any other topics that you’d like to emphasize?

Robert: Well, I think I would just re emphasize simplicity. I mean, simplicity is the ultimate sophistication. I think Leonardo da Vinci said that. And so, when you’re designing a class simplify as much as humanly possible, and no more. But I think your students are going to be well served by less, honestly. I think we tried to do way too much in our classes, and we will be well served by this too. I mean, I have another blog that deals with productivity and time management in academia. And that’s the key thing. I mean, a lot of faculty struggle with overwork and burnout or just doing too much and there’s a lot of stuff you can say no to and I think keep things very, very simple and that can be a whole lot. podcast about how to simplify your life in higher education, especially applyied to course design and grading systems. You don’t want to make choices in July when you’re planning your fall courses that you end up regretting by October.

David: I will say that something that did come up earlier, briefly, but there’s not one right way to do it. You should not feel like you must go all out ungrading, you shouldn’t feel like you have to do specifications grading exactly the right way. Find something that works for you, look for models, pick and choose the things. If you look at something and say, “How could that possibly work?” then don’t do that. And yeah, we’re not judging you. Find something that works well and it’s going to be good beer students.

Robert: Yeah, and especially the last thing you just said, David is so important. All this stuff is predicated on actually communicating with your students. Don’t just sort of wait for the course evaluations. You have to really get in and talk, actually talk with your students on a regular basis and get feedback and see how things are going. And that’s one of the great strengths of all these four pillars sort of oriented approaches is that it really does get students talking with each other and with their faculty members. And I think that’s, maybe in the end, the best thing about all this.

Rebecca: Well, we always wrap up by asking what’s next?

Robert: Well, for me, I know I’ve been spending the last two years not only as a faculty member, but working in our president’s office under what’s called a presidential fellowship. And I’ve been coordinating large-scale institutional cross-institutional teaching and learning initiatives, mostly focused on active learning spaces. And so that’s been a particular focus of mine for a few years. That is wrapping up now. But we’re looking at maybe some interesting and exciting extensions of that idea where we’re going to try to surface and looking for what our faculty are already doing in terms of instructional innovation. There’s a lot of stuff out there, like we said earlier, but even in our own institution, a lot of faculty are doing really innovative and creative things and effective things but are kind of working under the radar. So the President and I, and those around us, are kind of thinking how can we look and find these faculty, get them together, elevate what they’re doing, and sort of make Instructional Innovation a normative practice at our university. So that’s that’s kind of what’s next for me is figuring out how to make that work on a practical level.

John: Sounds like a wonderful plan. I know, we’ve been talking about some ways of doing the same thing at SUNY-Oswego. And it’s a challenge.

Rebecca: It’s no small task. [LAUGHTER]

Robert: It is, it is.

David: Something that I am getting more and more interested in is what helps instructors be successful in any kind of educational innovation, not just alternative grading and changes in their pedagogy, anything like that. But especially when it comes to grading, there’s so many variations. And in so many different situations where instructors are working, what are the things that help them best succeed? And what are the things they do that help their students buy in and succeed most? And so looking at those sort of things from a more fine grain perspective, I think it’s going to be somewhere that I’m going next.

Rebecca: That sounds exciting.

John: It does. And I hope you’re both come back and talk about the success of these initiatives in future podcasts.

Robert: We’d very much like to.

David: Absolutely.

Rebecca: Well, thank you so much for joining us and sharing your stories. And I know that many people are looking forward to your book.

Robert: Thanks a lot for having us.

David: Yeah, it was great to be here. Thank you

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle, Annalyn Smith, and Joshua Vega.

[MUSIC]

276. Teaching at its Best

Posted in active learning, assessment, attention, authentic learning, backward design, classroom community, clickers, cognitive science, collaborative learning, critical thinking, edtech, educational technology, engagement, equity, evidence-based practices, feedback, first-generation, first-year classes, grading, growth mindset, higher education, inclusion, learning, motivation, online learning, pedagogy, teaching
February 15, 2023

New faculty often start their faculty roles without training in teaching. In this episode Linda Nilson and Todd Zakrajsek join us to talk about the evolving roles and expectations of faculty and explore the new edition of a classic teaching guide.

Now Director Emeritus, Linda was the Founding Director of the Office of Teaching Effectiveness and Innovation at Clemson University. Todd is an Associate Research Professor and Associate Director of the Faculty Development Fellowship in the Department of Family Medicine at the University of North Carolina at Chapel Hill. Linda and Todd are each individually the authors of many superb books on teaching and learning and now have jointly authored a new edition of a classic guide for faculty.

Shownotes

  • Zakrajsek, T. and Nilson, L. B. (2023). Teaching at its best: A research-based resource for college instructors. 5th edition. Jossey-Bass.
  • Nilson, L. B., & Goodson, L. A. (2021). Online teaching at its best: Merging instructional design with teaching and learning research. John Wiley & Sons.
  • Nilson, Linda (2021). Infusing Critical Thinking Into Your Course: A Concrete, Practical Guide. Stylus.
  • McKeachie, W. J. (1978). Teaching tips: A guidebook for the beginning college teacher. DC Heath.
  • POD
  • Betts, K., Miller, M., Tokuhama-Espinosa, T., Shewokis, P., Anderson, A., Borja, C., Galoyan, T., Delaney, B., Eigenauer, J., & Dekker, S. (2019). International report: Neuromyths and evidence-based practices in higher education. Online Learning Consortium: Newburyport, MA.’
  • Padlet
  • Jamboard
  • Eric Mazur
  • Dan Levy
  • Teaching with Zoom – Dan Levy – Tea for Teaching podcast – May 26, 2021

Transcript

John: New faculty often start their faculty roles without training in teaching. In this episode we talk about the evolving roles and expectations of faculty and explore the new edition of a classic teaching guide.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are Linda Nilson and Todd Zakrajsek. Now Director Emeritus, Linda was the Founding Director of the Office of Teaching Effectiveness and Innovation at Clemson University. Todd is an Associate Research Professor and Associate Director of the Faculty Development Fellowship in the Department of Family Medicine at the University of North Carolina at Chapel Hill. Linda and Todd are each individually the authors of many superb books on teaching and learning and now jointly have authored another superb book. Welcome back, Linda and Todd.

Linda: Thank you very much.

Todd: Really appreciate the opportunity to be here.

Rebecca: Today’s teas are: … Linda, are you drinking tea?

Linda: I’m drinking a tea called water. It’s rather dull, but I enjoy it.

Rebecca: It’s very pure.

Linda: Yes, very pure. Very pure.

Rebecca: How about you Todd?

Todd: Oh, I’ve got myself a Lemon Detox because I’ve spent most of my day getting all toxed and now I’m getting detoxed. [LAUGHTER] Wait a minute, that sounds bad. [LAUGHTER] But that will be all right. [LAUGHTER]

John: Especially at Family Medicine.

Todd: Well, we can fix it. [LAUGHTER] In general, life is good.

John: I am drinking pineapple green tea.

Rebecca: Oh, that’s a new one for you, John.

John: I’ve had it before, just not recently.

Rebecca: Okay. I’m back to the very old favorite, English afternoon. Because I stopped by the Center for Excellence in Learning and Teaching and grabbed a cup before I came.

John: And we are recording together in the same room, which has been a fairly rare occurrence for the last several years. We’ve invited you here to discuss your joint endeavor on the fifth edition of Teaching at its Best: a Research-Based Resource for College Instructors, that Linda originally developed and now you’ve collaborated on this new edition. How did the collaboration on this edition come about?

Linda: Well, let me talk about that. Because it was pretty much my idea. Jossie-Bass contacted me and said “let’s put out a fifth edition” and I said “let’s not.” [LAUGHTER] I was not in the mood to do it. I’ve been retired six and a half years now and I’m loving it. I mean, I’m really loving it. And while retired, I was still writing the second edition of Online Teaching at its Best. And then I was writing a book, Infusing Critical Thinking Into Your Course, and I guess I had had it. I mean, I wanted to really make a change and I wanted to get specifically into working at an animal shelter. So I was all occupied with that. So I thought I remember Wilbert J. McKeachie, when he was doing Teaching Tips that he came to a certain point after I don’t know how many editions that he brought other people on to really do the revision work. And so I decided I’m going to do that. So Jossey-Bass said “Okay, fine.” They wanted three names. Okay, I gave him three names, but my first choice was Todd Zakrajsek, because 1. I knew he’d finish it. [LAUGHTER] I knew he’d finish it fast. I knew he do a great job. He knows the literature like the back of his hand, I wouldn’t have a worry in the world. And guess what? Todd accepted. Hip hip hurray. I was so happy. I couldn’t tell you.

Todd: Well, this is great because I said no when they asked me. [LAUGHTER]

Rebecca: Like any smart person would, right? [LAUGHTER]

Todd: Well, I did end up doing it, of course. But the reason I said no was I knew that book very well and I know Linda very well. And I said, “There is no way. I don’t know anybody who can step in and pick this thing up. She knows so much about so much that it’s just not possible.” And they said, “But she really wants you to do this.” So I went back and forth a couple times and I finally decided to do it. And I will tell you, Linda, because I haven’t mentioned this to you. The first three chapters, I had to go back and redo those when I got done with it, because I was so scared of the first three chapters [LAUGHTER] that it was really rough. And then finally it’s like, okay, I hit my rhythm and I walked into it with impostor syndrome a little bit, and I finally caught my footing, but it’s a good book to start with.

Linda: Thank you. Thank you very much. [LAUGHTER] Yeah, I know, the plot thickens, right? It becomes more interesting as you go from chapter to chapter, right. And before you know it, there’s a happy ending after all.

Rebecca: So Linda, Teaching at its Best has been around for a long time with a first edition published in 1998. Can you talk a little bit about how that first edition came about?

Linda: Yes, that was…I can’t believe… 1998. That’s 25 years ago. It’s almost scary how time flies. But anyway, the actual seed of the book came about in about 1994… 95. But I need to give you some background because I had been writing TA training books since like, the late 1970s when I was first given the task of putting together a TA training program. So back then, I was putting out weekly mimeos,[LAUGHTER] remember mimeograph machines. Some of you don’t know, what is she talking about? But anyway, that was technology then. But anyway, smetl great, though… it really did. [LAUGHTER]

Rebecca: That’s the second time today someone has made a reference about the smell of those.

Linda: Yeah, oh yeah.

John: The dittos are what I remember having the stronger smell

Todd: The ditto did, yeah. yeah, and I’ll tell you before we move on, when I was a graduate student, we had a ditto machine. I just have to say this, Linda, because you liked the smell and all there.

Linda: Yeah, Yeah.

Todd: But they had a ditto machine. And below the ditto machine, I noticed that the floor tiles were kind of eaten away by the ditto fluid. [LAUGHTER] And then here’s the best part is that one day I was rooting around in the closet looking for something and I found the extra tiles in a box and the side of the box said “reinforced with long-lasting asbestos.” [LAUGHTER] So the ditto fluid was eating through asbestos lined tile, but that’s how strong that stuff is. So yeah, we all enjoyed the smell of that stuff back in the day..

Linda: Yeah, yeah. I guess it’s a good thing for all of us they invented something else, like copying machines. So anyway, so I started doing that at UCLA. And then that turned into like a booklet of sorts. And then I was at UC Riverside, and I was writing books there. And I sort of revised it every couple of years. And I was also writing these with my master teaching fellows. So we were doing that. And then I came to Vanderbilt, and I decided, well, I’m going to do this, pretty much on my own, I’ll get some help from my master teaching fellows. But anyway, it turned into an actual book. I mean, it turned into a happy monster. And I was very pleased with it. Well, along about 94-95, my husband recommended that I turn it into a regular book, and talk to a publisher about it. So anyway, I said, “Oh, great idea. Great idea and just sort of didn’t think about it much. Then in 1996, he died. And I thought, “Well, how am I going to pull myself through?” I bet it would be a great idea and a great tribute to him if I took Teaching at its Best, the Vanderbilt edition, and turned that into a general book. And I decided to do that and kept my mind off of bad things. And it turned into Teaching at its Best, the first edition. That’s why I dedicated the book to him, by the way, because it really was his inspiration that got me to do it. And so anyway, tribute to him. So that’s where the first edition came from. I mean, it really grew out of tragedy. But it’s been a comedy ever since, right? [LAUGHTER] So anyway, it’s been a wonderful thing.

John: And it’s been a great resource.

Rebecca: It’s interesting that it pulled you through, but then has pulled many teachers through. [LAUGHTER]

Linda: And I’ve gotten such feedback from faculty members who said, “I saved their lunch,” you know, if they were really in big trouble, and some of them said, “I was in big trouble with my teaching and you got me tenure.” Yeah, like, right. But anyway, the book helped a lot of people. And I guess maybe something in me when I first published this book said, “Gee it would really be great to be the next Wilbert McKeachie, right, which is a very pretentious thing to think. But then they wanted the second edition, I was thinking, “Hey, maybe I’m on the road to something.” And then there was a third, and then there was the fourth. And it didn’t get any easier to write the subsequent editions really, it was just a matter of keeping up with the literature. And so right now, I’m off into another corner of the world. So I just didn’t want to immerse myself in that again.

John: So that brings us to the question of what is new in the fifth edition?

Todd: Well, that’s my question. I’ve known Linda for the longest time. By the way, I do want to mention before we go on, I can’t remember, Linda, if it’s been that long ago, but it might have been the second edition. When at POD, I said, “You need to do a second edition of this book” …or second or third. But I was using the book. I mean, I learned so much from it. So for the new edition. Number one, of course, the research has been updated only because the research is always changing. And it had been a few years. So that’s number one. In terms of changing the book, though, we only have a leeway of about 10,000 words. Now, for those out there listening 10,000 words sounds like a lot of words until you’ve got a 200,000 word book, it was about 190. And they said, you can’t go over 200 Because the book just gets too big then. So it is 10,000 words longer than it was in fact, I think it’s 10,003 words longer. So it’s right in there. [LAUGHTER]

Rebecca: So you snuck an extra 3 words in.

Todd: It could have been a squeeze to put three words in there. And it’s always hilarious because when they say there’s just a few too many words I just start hyphenating things so yeah, it kind of all works. [LAUGHTER] Yeah, just any words at all. So you can do “can you” as just a hyphenated word. It works. [LAUGHTER]K So is that terminology, the terminology does change and I find this fascinating. One of the things I love to write about books is learning. I mean, Linda, the same thing what as we write, we read a ton of stuff. And as we read stuff, we learn stuff. So this one in particular, for example, is that I grew up with PBL as problem based learning. And I had done workshops on it, I had worked on everything else, but I hadn’t looked at it for quite a while. And in this particular book, as I started looking at PBL, I couldn’t find anything on problem based learning. And it was fascinating because I was doing some digging, and then I called Claire Major, who was an early person who had a grant on problem based learning and everything I ran into was about 2002, it just started to drop off a little bit, and there was some, but it started to tail off. And when I talked to Claire, she says, “Oh, yeah, I used to do quite a bit about that, it was back around 2002-2003.” And now, and the reason I’m saying this is, every time I saw the letters PBL, it was project based learning. And project based learning sounds a lot like problem based learning, but they’re different concepts. And so anyway, going through and finding some of the terminology, so it was consistent with what’s being done right now has changed. There is now a chapter on inclusive teaching, because over the last three or four years, we finally realized that there’s a whole lot of individuals who haven’t been successful in higher education, partly because of the way we teach. And so I’ve been making an argument for a few years now that teaching and learning, the classroom situation has always really been based for fast-talking, risk-taking extroverts. And we’ve suddenly realized that if you’re not a fast-talking, risk-taking extrovert, you may not get a chance to participate, classroom and other things. So I looked at some different things with inclusive teaching. There’s a whole another chapter on that. And then just the language throughout, we talk a little differently now, just even over the last three or four years than we did five, six years ago, I was pretty surprised by that. But there’s some pretty significant changes in language. So the book has a slightly different tone in language, and those are the biggest changes. Oh, I should say, before we move on, one of the biggest other changes, and I did this one, Linda put a section in there that said learning styles had changed significantly from the previous edition. And so she had pointed out that there was no longer a section on learning styles. And I put the learning styles right back in there, I told Linda and she gasped just a little bit. And then I explained that I put it back in there, and then said exactly how terrible it was to basically teach according to learning styles, because it’s the myth that will not die. So that’s back in there.

Linda: People love it. I know. [LAUGHTER]

John: We have that issue all the time, students come in believing in them and say, “Well, I can’t learn from reading because I’m a visual learner.” And I say “Well, fortunately, you use your eyes to read,” and then I’ll get them some citations.

Todd: Well, I’ll tell you, and before we move on, these are the types of things we learned. I couldn’t figure out why the thing is so hard to die. What is it that’s really doing this because other myths we’ve been able to debunk. And part of the reason is licensing exams, when you are in pre-service and you want to become a teacher, the exams you take to become a teacher, a large portion of those exams, have learning styles questions on there. So you have to answer about visual learners and auditory learners and kinesthetic. And so until we get those out of teacher education programs, we’re teaching teachers to believe this. So anyway, there you go. Public service announcement. Be careful about meshing. And if you don’t know what meshing is, look it up and then stop it. [LAUGHTER]

John: We have had guests on the podcast who mentioned learning styles, and then we edit them out and explain to them later why we edit out any reference to that. And I think most of them were in education, either as instructors, or they’ve been working as secondary teachers. It is a pretty pervasive myth. In fact, Michelle Miller and Kristen Betts, together with some other people, did a survey. And that was the most commonly believed myth about teaching and learning. It was done through OLC a few years back, about three or four years ago. Yep,

Todd: Yeah, I saw that survey. Yes, it’s pretty amazing. Michelle’s an amazing person.

Rebecca: The experience of the pandemic has had a fairly large impact on how our classes are taught. Can you talk a little bit, Todd, about how this is reflected in this new edition?

Todd: Things have changed pretty significantly because of the pandemic. There’s a couple things going on. Again, the inclusive teaching and learning, which I’ve already commented on, is really different now. And it’s interesting, because it goes back to the 1960s. We’ve known that, for instance, African Americans tend to flunk at twice the rate of Caucasians, in large machine-scored multiple choice exams. So we know it’s not the teaching, and we know it’s not the grades, it has to be something else. And it turns out that it was you put students into groups and those differences start to disappear. So I mean, even more so the last couple of years, it’s a lot of engaged learning, active learning. I’m still going to pitch my stuff that I’ve been ranting and raving about for years. And there’s no data out there that says that lecturing is bad. What the data says is that if you add active and engaged learning to lecture, then you have much better outcomes than lecture alone. But we’re learning about those types of things in terms of active and engaged learning, how to pair it with and mix it with other strategies that work, looking at distance education in terms of systems and how we can use technology. So a quick example is I used to have a review session before exams. And oftentimes, it’s hard to find a place on campus to have that. And so you might be in a room off in one hall or the library or something. And if the exam was on Monday, I’d have the review session at like six o’clock, seven o’clock on a Sunday night. And there are students who couldn’t make it. I would simply say, you can get notes from someone else. And we’ve known for the longest time, if a student misses class, getting notes from somebody else doesn’t work. Well, now I do review sessions on Zoom, we don’t have to worry about finding a place to park, we don’t have to worry about some students finding babysitters, if they’re working, it’s recorded, so they get the exact same thing. So things like Zoom have really changed teaching in a sense that you can capture the essence in the experience of teaching and use it for others, and it has helped with some equity issues. You can’t do it all the time. And teaching over Zoom is different than face to face. But there are now ways of using different technologies and using different modalities to help to teach in ways that were not really used before the pandemic.

John: Speaking of that, during the pandemic, there was a period of rapid expansion in both the variety of edtech tools available and in terms of teaching modalities themselves. In the description of your book, it indicates that you address useful educational technology and what is a waste of time? Could you give us an example of both some useful technologies that could be used and some that are not so useful? And also perhaps a reaction to the spread of bichronous and HyFlex instruction?

Linda: Yeah, I’ll take this one. And I’m drawing a lot of stuff from another book that I co-authored, with Ludwika Goodman. We were writing about Online Teaching at its Best, okay. And she was an instructional designer. And I came from teaching and learning and we put our literature’s together. And we were talking about modalities a great deal, especially in the second edition with the pandemic. Well, one thing I found out, not only from reading, but also from watching this happen was that this Hyflex or bichronous, whatever you want to call it, is a bust, if there ever was a modality, that’s a bad idea it’s that one, even though administrators love it because students can choose whether to come to class and do the things they would do in class, or to attend class remotely? Well, yeah, it sounds like “oh, yeah, that could be good.” But the technological problems, and then the social problems, especially the in-class social problems are enormous. And in-class social problems, like small group work, how do you hear what’s going on in the classroom over this low roar of small groups? Okay, so how can you help? How can the students that are learning remotely, what can they do? Now, the way this was invented, by the way, was for a small graduate class, and then okay, like, makes sense, because you’re only dealing with six students in this room and six students who are remote. But other than that, it’s so bad, the logistics, the sound logistics, the coordination that the instructor has to maintain, the attempt at being fair to both groups, at bringing in both groups, when the groups can’t even hear each other well. Now, if we had Hollywood level equipment in our classrooms, we might be able to make this work a little better, but we don’t, and we’re never going to have that. So there are just a lot of technological and social reasons why HyFlex, that’s what I called it in Online Teaching at its Best, what it was called at the time is a complete bust. Now, not to be confused with hybrid or blended learning, which we found has worked exceedingly well. So bringing in some technology, but into a face-to-face environment and that being the base of the class. Now, remote’s nice, but you might not want to do remote all the time for all things. It’s not quite the next best thing to being there. But it’s something and as long as you don’t just stand there and stare at the camera and lecture for an hour. You’ll get complaints about that quickly. And particularly with students today when they really need to be actively involved, actively engaged. So yeah, sure, fine, talk for three minutes, maybe even push it for five, but then give them something to do and you really, really must in remote because otherwise, you’re just some talking head on television.

Todd: I agree completely. In fact, it was funny because I happen to have a digital copy of the book here. And so I typed in a ctrl F and I typed in HyFlex and there’s one comment to the preface that said there’s many different formats out there and then I will tell the listeners, if you’re expecting to learn about HyFlex, the word never shows up again in the book. [LAUGHTER] So, it’s not in there. I mean, you look at the literature that’s out there. And I think it’s fair to say that maybe there are people who can do it. I haven’t really seen it done well and I think Linda’s saying she hasn’t either. And it’s so difficult, especially for a book like this. That’s not what we’re all about. I mean, again, if it even works well, which I’d love to hear that it would be a very advanced book and that’s not what this is. So we do have a lot in there about technology in terms of edtech tools, though. There are those in there, I would just say real quickly, for instance, Padlet’s one of my favorites, I’ll throw that out there. I like Padlet a lot. But there are tools out there, if you want to do a gallery walk, which for instance, if you happen to be in a face-to-face course, you’d set up maybe four stations with big sheets of paper, you put your students into groups, and then they walk from sheet of paper to sheet of paper, and they move around the room. And they can do what’s called a gallery walk. You can do the same thing online with a jamboard, you can set up jamboards so that there’s different pages, and then each group is on a page. And then you just say it’s time to shift pages, they could shift pages. So I’ve done gallery walks, and it’s worked well. I’ve used Padlet for brainstorming. And one of the things I love about Padlet, I’ll have to say is if you are doing some digital teaching in a situation, you can watch what each group is developing on the page for all groups at the same time. I can’t hear all groups at the same time when I walk around the room. So there are certainly some technologies coming out that can really do things well. There’s also things that don’t work very well, though. And I think one of the things you want to keep in mind is just learning theory. Does the technology you’re using advance students, potentially, through learning theory? Does it help with repetition? Does it help with attention? Linda was just mentioning attention, if you lecture too long, you lose their attention. If you do something ridiculously simple or not… I was gonna say stupid, but that sounds rude. But if we do something as a small group that makes no sense, you don’t get their attention either. So using clickers, I have to say, I watched a faculty member one time because they were touted as a person who was very engaging. And this is at a medical school, so I really wanted to see this. And the person used clickers, but used it in a way that asked the students a question, they responded, and the instructor looked up at the board and said, “Here’s how you responded, let’s move on.” And then moved on to the next thing. And about five minutes later gave another question said, “How do you respond?” and they clicked the clicker, and then they moved on again. That had no value at all, and in fact, there was no actual interaction there. So afterwards, I say, can’t you just ask a rhetorical question and just move on? We got to be careful not to use technology just because it’s being used, it should advance the learning process.

John: However, clickers can be effective if it’s combined with peer discussion and some feedback and some just-in-time teaching. If it’s just used to get responses that are ignored, it really doesn’t align with any evidence-based practice or anything we know about teaching and learning. But those per discussions can be useful and there’s a lot of research that show that does result in longer-term knowledge retention when it’s used correctly, but often it’s not.

Todd: Right. And I think that’s a really good point. I’m glad you said that, because Eric Mazur, and his concept tests, for a large extent, that’s where active and engaged learning really took off. And that is a clicker questions. And they can be used as great tools. But again, if you’re using it for the right reason, which is what you just said, My comment is, there’s technology out there, that is a waste of time, and not a good thing to have, because it’s just not being used in a way that’s conducive to learning. So good point, that’s fair.

Rebecca: Can you talk a little bit about who the audience of the book is?

Linda: Sure. It’s actually for anybody who teaches students older than children, I suppose, because it isn’t really designed for teaching children. But other than that, it’s really for people who teach but don’t have the time to read a book. The nice thing about Teaching at its Best is you can go to the table of contents, you go to the index, you could find exactly what you need for your next class. And it’s very oriented towards how to, so it could be for beginners or for experienced people who simply haven’t tried something specific before, or want a twist on it, or just want some inspiration. Because there there are a lot of different teaching techniques in there. And they’re all oriented towards student engagement, every single one of them. But I wanted to comment too, on just how the job of instructor or professor has changed over the past, I don’t know, 40 years, I suppose. I know when I started teaching it was a completely different job. And I started teaching in 1975, when I was 12, of course, but no and I was young to start teaching because I was 25 and there I was 180 students in front of me. So oops, my goodness, what have I done? But that’s exactly what I wanted to do. But you’d go in there, you’d lecture and you’d walk out. You were in complete control of everything. Like, you might throw out a question and you might get a discussion going. But it wasn’t considered to be essential. In fact, there were two teaching techniques back then: there was lecture, and there was discussion. And nobody knew how to do discussion. Now, I had to find out a few things about it when I was doing TA training, because TAs were supposed to be running discussions. But there wasn’t a lot out there. Thank God for Wilbert McKeachie’s book Teaching Tips, because that was about the only source out there you could go to. So anyway, but now the job, I mean, oh, it’s mind boggling what faculty are now expected to do. And they are supposed to, like, learning outcomes. Okay, I love learning outcomes. They’re wonderful. But I didn’t have to do that when I started, I just had to talk about my subject, which I dearly loved. And so, that was nothing. But you’ve got learning outcomes. So you’ve got to be like, a course designer, you have to deal with a student’s mental health problems, right? It’s part of the job, and you’re expected to respond to them. You’re supposed to give them career counseling in careers that you might not know much about, and possibly for good reason, because you’re in your own career. It’s so time consuming, not to mention fair use, oh, yes, fair use has changed, fair use has changed radically. And when you’re dealing with anything online, the rules are totally different. And you’re highly restricted as to what you can use, what you could do. When you’re in a face-to-face classroom, it’s a little bit easier. So yeah, so you got to be a copyright lawyer to stay out of trouble. And then you get involved in accreditation, you get involved in that kind of assessment. So you have to all of a sudden be totally involved in what your program is doing, what your major is doing, where it’s headed. There’s just too much to do. And there are more and more committees and oh, there’s a lot of time wasted in committees. Of course, you’re supposed to publish at the same time and make presentations at conferences. It was like that back then, too. But now, the expectations are higher, and it’s on top of more time in teaching, and more courses. I was teaching four courses a year, and you can’t find that kind of job anymore.

Rebecca: So Linda, you’re saying the animal shelter is going really well now?

Linda: Yes. Yes. Yes. Yes.

Todd: That’s hilarious. Well, I want to point out too, and I think Linda’s said it very, very well is that we are expected to do things we never had to use before. Never worried about before. And I love the fair use is great, because when I first started teaching, and I’ve been teaching for 36 years, when I first started teaching, you’d videotape something off TV and show it in class and then put it on the shelf. And I knew people who showed the same video for 10 years. Right now you better be careful about showing the same video for 10 years. But these are things we need to know. I would say also, by the way, this is a really good book for administrators, anybody who would like to give guidance to faculty members, or better understand teaching and learning so that when promotion and tenure comes along, you get a sense of this. And so if you’re saying to the faculty, they should use a variety of teaching strategies. It’s not a bad idea to know a variety of teaching strategies. And so I think it’s good for administrators as well, and graduate students. But I want to take a second and tell you, one of the reviews of the book, I guess, came in just yesterday or the day before from Dan Levy. He’s a senior lecturer at Harvard University. And what he put was Teaching at it’s Best is an absolute gem. Whether you are new to teaching in higher education, or have been doing it for a while, you will find this book’s evidence-based advice on a wide range of teaching issues to be very helpful. The style is engaging and the breadth is impressive. If you want to teach at your best you should read Teaching at its Best. And I love what he put in there because it doesn’t matter if you’re a new teacher or you’ve been doing it for a while, this book’s got a lot of stuff in it.

John: And Dan has been a guest on our podcast, and he’s also an economist, which is another thing in his favor. [LAUGHTER]

Todd: That is good.

John: I do want to comment on lenders observation about how teaching has changed because I came in at a very similar type of experience. I was told by the chair of the department not to waste a lot of time on teaching and to focus primarily on research because that’s what’s most important, and that’s the only thing that’s really ultimately valued here or elsewhere in the job market. But then what happened is a few people started reading the literature on how we learn And then they started writing these books about it. [LAUGHTER] And these books encouraged us to do things like retrieval practice and low-stakes tests, and to provide lots of feedback to students. So those people…[LAUGHTER]

Rebecca: I don’t know any of them.

John: …but as a result of that many people started changing the way they teach in response to this. So some of it is you brought this on to all of us by sharing… [LAUGHTER]

Linda: I apologize.

Todd: Sorry about that.

Linda: I apologize.

Todd: We apologize and you know, I will say too is, so yeah, sorry. Sorry about doing that. But I’m glad you said that.

Linda: We made the job harder didn’t we?

Todd: We did, but you know to just be fair for Linda and I as well as I still remember a faculty member calling me, It must have been about 20 years ago, and I just started doing a little bit of Faculty Development, she was crying, she had given her first assignment in terms of a paper. And she said, I’m sitting here with a stack of papers, and I don’t know how to grade them. And it got me thinking a little bit, how many of the aspects of the job that we’re required to do, were we trained to do? And that’s the stuff that Linda was mentioning as well, is nobody taught me. I’m an industrial psychologist. And so nobody taught me the strategies for delivering information to a group of 200 people. Nobody taught me how to grade essay tests. Nobody taught me how to grade presentations, I didn’t know about fair use and how I could use things. I mean, you go through and list all of the things that you’re required to do. And then look at all the things you were trained to do. And this is tough. And that changed. So I have one quick one I’ll mention is I was hired as an adjunct faculty member before I got my first tenure-track job. And I was teaching 4-4. So I had four classes in the fall, four classes in the spring. And about halfway through the spring, I ran into the department chair, and I was interested to see if I was going to be able to come back and I said, “Hey, Mike, how am I doing?” And this was at Central Michigan University, a pretty good sized school. He said, “You were fantastic.” And I said, “Excellent. What have you heard?” He said, “absolutely nothing.” So when it comes to teaching, what I learned was: research, you had to do well, and teaching, you had to not do terribly. And that is what you were mentioning has changed is now you’re kind of expected to do teaching as well.

Rebecca: And there’s a lot more research in the area now too. So sometimes it’s hard to keep up on it. So books like this can be really helpful in providing a lot of that research in one place.

John: And both of you have written many good books that have guided many, many faculty in their careers, and eliminated that gap between what we’re trained to do and what we actually have to do.

Rebecca: So of course, we want to know when we can have this book in our hands.

Todd: Good news for this book, which is exciting because we really cranked away on this thing and it’s listed in Amazon as being due on April 25. But it actually went to press on January 23. So it’s already out and about three months ahead of schedule.

John: Excellent. We’re looking forward to it. I’ve had my copy on preorder since I saw a tweet about this. I think it was your tweet, Todd, a while back. And I’m very much looking forward to receiving a copy of it.

Todd: Excellent. We’re looking forward to people being able to benefit from copies of it.

Rebecca: So we always wrap up by asking what’s next.

Todd: It’s hard to tell what’s next because I’m exhausted from what’s been [LAUGHTER] ever moving forward, as I’m working on and just finishing a book right now that’s to help faculty in the first year of their teaching. So it’s basically off to a good start. It’s what specifically faculty should do in the first year of getting a teaching position. And aside from that, probably working on my next jigsaw puzzle, I like to do the great big jigsaw puzzles. And so I just finished one that had 33,600 pieces. It is five feet….

Rebecca: Did you say 33,000 pieces?

Todd: No, I said 33,600 pieces.It was the 600 that…

Rebecca: Oh, ok.

Todd: …was difficult. [LAUGHTER]

Rebecca: Yeah.

Todd: When the puzzle is done, it has standard sized pieces, and it is five feet by 20 feet. So I just enjoy massively putting something together. It’s very challenging. So quite frankly, for those about and listening to this is if you imagine 33,600 puzzle pieces, that’s about as many studies as Linda and I have read to put this book together. [LAUGHTER]

Linda: Nothing to it. [LAUGHTER]

Todd: So that’s it for me. [LAUGHTER] Linda, what are you up to these days?

Linda: Oh, well, I live in la la land. So I’m still doing workshops and webinars and things like that mostly on my books of various kinds, various teaching topics. But I think what I want to do is retake up pastels and charcoals. My father was a commercial artist. And so he got me into pastels and charcoals when I was in high school. Well and then I dropped it to go off to college. Well, I want to get back into it in addition to working at the animal shelter. I know. It’s la la land and I wish la la land on everybody that I like.[LAUGHTER] I hope you all go to la la land and enjoy being a four year old all over again, because that’s the way I feel. I adapted to retirement in about 24 hours. That’s pushing it… you know, it’s more like four. But anyway, I slept on it. [LAUGHTER] That was the end of it. But I know I eased into it. I eased into it. I was still writing. I was still doing, especially before the pandemic, a lot of speaking. So then the pandemic hit and it just turned into online everything. And now I’m back on the road again, to a certain extent. I love it. So anyway, it’s a nice balance. So yeah, I wish you all la la land too.

Todd: That’s great.

Rebecca: That’s something to aspire to.

Todd: Yeah, it is. But you know, since you mentioned the speaking things, I just have to do the quick plug here. Linda, I think you and I, years and years ago, were joking around at POD about who would be the first one to get to the 50 states and have done a presentation in every state. And so I gotta tell you, I’m not even sure where you’re at in the mix, but I am at 49 states. And if any of your listeners are in North Dakota, [LAUGHTER] I could certainly use a phone call from North Dakota.

Linda: Well, I want to go to Vermont. I have not been to Vermont…

Todd: Oh, you haven’t.

Linda: …to give a presentation. So I would enjoy that. But I’ll go to Hawaii. I’ll do anything in Hawaii for you. Absolutely anything. [LAUGHTER] I’ll do gardening, [LAUGHTER] I’ll do dishes, your laundry. I don’t care.

Todd: That is good. Yeah, Linda and I had this gig. It was a long, long time ago. And I don’t know, it must have been 20 years ago we talked about it even. And there was some rules too. You had to be invited. And there had to be some kind of an honorarium or just I mean, it didn’t have to be much, but the concept was you just couldn’t show up at a state and start talking. [LAUGHTER] Otherwise, we’d have both been done a long time ago. But yeah,

Linda: Yeah.

Todd: … it was fun. This is the way nerds have fun. [LAUGHTER]

John: Well, that’s a competition that’s benefited again, a lot of people over the years.

Rebecca: Well, thanks so much for joining us. It’s great to see both of you again, and we look forward to seeing your new book.

Linda: Thank you for this opportunity. It was a pleasure.

Todd: It was so much fun. Thank you

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

261. Social Justice Assessments

Posted in access, assessment, authentic learning, diversity, equity, feedback, first-generation students, grading, higher education, inclusion, learning, mastery learning, motivation, pedagogy, social justice, teaching, universal design
November 2, 2022

Traditional methods of assessing student learning favor those students that reside in well-resourced school districts while leaving low-income students at a substantial disadvantage. These grading systems also encourage students to focus on their grades rather than on their learning. In this episode, Judith Littlejohn, Meghanne Freivald, and Katelyn Prager join us to discuss a variety of social justice assessment techniques that can help to create a more equitable environment in which all students can be successful.

Judie is the Director of Online Learning at SUNY Genesee Community College, Meghanne is an Instructional Technology Specialist at Alfred University, and Katelyn is an Assistant Professor in the English Department at the Fashion Institute of Technology.  Judie, Meghan, and Katelyn worked together on a SUNY Faculty Advisory Council on Teaching and Technology committee on social justice assessments.

Show Notes

Transcript

John: Traditional methods of assessing student learning favor those students that reside in well-resourced school districts while leaving low-income students at a substantial disadvantage. These grading systems also encourage students to focus on their grades rather than on their learning. In this episode, we explore a variety of social justice assessment techniques that can create a more equitable environment in which all students can be successful.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guests today are Judith Littlejohn, Meghanne Freivald, and Katelyn Prager. Judie is the Director of Online Learning at SUNY Genesee Community College, Meghanne is an Instructional Technology Specialist at Alfred University, and Katelyn is an Assistant Professor in the English Department at the Fashion Institute of Technology. Judie, Meghan, and Katelyn worked together on a SUNY Faculty Advisory Council on Teaching and Technology committee on social justice assessments. Welcome Meghanne and Katelyn and welcome back, Judie.

Meghanne: Thank you.

Katelyn: Thank you.

Judie: Thank you.

John: Today’s teas are:

Judie: …I have Lady Grey.

Rebecca: That’s a good one…

Judie: …In my DTL mug.

John: …a nice Desire to Learn mug.

Meghanne: I have iced green.

Rebecca: And Katelyn, how about you?

Katelyn: Mine’s water right now, if it were the evening, I would have one bag of peppermint and one bag of chamomile together, delicious.

Rebecca: Sounds nice and calming.

Rebecca: I have hot cinnamon spice tea.

John: And I have black raspberry green tea.

Rebecca: We’ve invited you here today to discuss your work on social justice assessment. Perhaps, we can start with a discussion on what you mean by social justice assessment.

Judie: So social justice assessment considers factors such as race, culture, language proficiency, socioeconomic status, and ability while working to dismantle systems of power, bias, and oppression in evaluation of student learning. So various approaches including equitable assessment, labor based grading, and ungrading, as they relate to the purpose, process, wording, and structure of student learning assessments are included. So we’re trying to focus on the learning that our diverse students achieve as it relates to specific learning outcomes just to mitigate the influence of dominant norms on our students’ grades. So we’ve all been working together for the last couple of years on a SUNY task group that was part of the Faculty Advisory Council on Teaching and Technology, which I chair. So we’re a subcommittee of an Innovations in Assessment group, and there’s a couple more of us who couldn’t make it today, but we’ve been a really close-knit group, I think, working together for over two years. And we really enjoyed the project, which resulted in a website with all these artifacts on it that people will be able to access. And we’re hoping down the road that we can continue our work, but we’ll get to that later on in this conversation.

John: And we’ll share a link to the overall website as well as your group-specific component of that in the show notes. So this was partly implied in your response defining social justice assessment, but, what are some of the shortcomings of traditional grading systems in terms of equity?

Meghanne: When we were doing our research on this topic, we encountered many drawbacks of the traditional types of assessments that we all experienced all the way up through school and into college, and I’ll share a few of them. One is that the focus is often on the grade rather than the actual learning process and what the student will actually be able to do, and be able to learn as a result of engaging in the education process. They just focus on the grade, “what’s my grade?” and that sort of misses the point. It creates a system where students are compared to each other rather than having the focus be on individual growth and achievement. It also can put students at an advantage or disadvantage based on characteristics such as race, ethnicity, socioeconomic status, disability status, language proficiency, and lots of other characteristics that students themselves don’t have any control over. We found in our research that traditional assessments tend to favor white, affluent, high-achieving students, and that really isn’t who most of our students are anymore. So we really need to remove barriers and create a way for students to accurately represent the learning that has taken place.

Rebecca: So you hinted to this in your response about traditional grading systems comparing students to one another. So thinking about that, what role should students play in determining how their learning is assessed?

Katelyn: I’ll tackle that one, and I want to answer it with a disclaimer to start because social justice assessment is an umbrella term that has all of these different strategies that are wrapped up in it, and each of those approaches, whether it’s ungrading, or labor-based grading, might have a slightly different response to that question. They all share the same goal, that students should not be systematically disadvantaged by the assessment mechanisms, and that we want to increase student agency in the classroom. We want students to be active participants in their own learning, but the actual question of how students might participate in determining their own assessments might look very different depending on who you’re talking to and what approach they use. Maybe it’s literally helping design the assessment mechanisms, the grading contract, grading rubric, maybe it’s creating flexible assignments that allow students to determine what learning is being assessed, or in the case of ungrading, maybe it’s just deprioritizing the assessment entirely in order to emphasize the individual student’s learning journey through the course. So I guess my answer, tentatively to your question, is yes, students should be participants in determining how they’re learning is assessed by the big how, and why is going to differ.

John: As you noted, there’s a wide continuum of alternative grading policies that can fit under this category of social justice assessment. Some of them are not that much different than traditional practices, and others are quite a bit different. One approach, which is much closer to the traditional grading systems that people are already using is a system of mastery learning. Could you talk a little bit about what mastery learning is and how that could be used in the classroom to provide a bit more equity.

Judie: So mastery learning is, instead of assessing a student or evaluating a student with one assessment, and giving them that grade, the students are able to go back and revisit the content and work again on any material that they didn’t understand or try things over again. So it’s an iterative process, and they should get some sort of formative feedback in between attempts so that they can understand what it is they need to work on and focus on. And this way, it’s more equitable, because the students are able to take the amount of time that they need to work on the assessment, they can access any review materials that they need to establish their foundational knowledge and continue on. And it just really helps the students learn and grow. And I think it’s a great way to establish foundational knowledge. I use it myself in all the history courses that I teach, and I just think it’s a great process. If you think about it, any athlete, that’s what they do. So if you’re learning how to play baseball, how many hours are spent in a batting cage, or like on the pitcher’s mound, how many times do you try again, and again, and then again, until you are able to do it correctly, or do things accurately? So I always liken it to use that sports analogy, because I really think that helps people understand that students’ learning… you have to practice and you can’t tell somebody something once and expect them to integrate it into all the knowledge they already have, and be able to recall it instantly. So I just think it’s a great way to level the playing field of students so that when you move on to the next part of your content, they all have the same foundation, and they’re ready to go forward.

John: And by explaining it to the students that way, in terms of a sports metaphor, it’s something that they can pretty easily connect to, and I think it also would help to promote a growth mindset, which we know is effective in increasing learning as well.

Rebecca: Another assessment strategy one might use is minimal or light grading that falls under this social justice umbrella, and is a bit different than mastery learning. Can you describe what minimal or light grading is?

Meghanne: Yeah, I’ve seen this described in a couple of different ways. This isn’t something that we really included in a lot of our research, so I kind of looked this up just a little while ago and it’s very interesting. And one approach is more on like the whole course level. And there’s another approach that can be taken on an assignment level. So for an entire course, what an instructor may do is that they would assign assignments throughout the semester, but most of them would not be graded, they would be used as like a conversation piece. And they would be discussed and gone over during class, which would then provide opportunities for the students to seek clarification and for the instructor to provide feedback in the moment. So then the assessment then becomes part of the learning process. So then when there are a small number of assessments that are given for a grade, then when the students get to those assessments, they’re not as intimidating. They’re things that they’ve done with their classmates, they’ve done them with their instructors, they’ve done them in class. So I think it’s a very interesting strategy because it removes a lot of the anxiety that students may have around assessment, because it’s just something that they’ve done in their class. Another take on this that I’ve seen is, on an assignment level, something like a paper, something that may require a lot of revision, where when the professor is grading that assessment, they would maybe not take the time to go through and mark all of the grammar and spelling and mechanical errors, but maybe they would look at a section of that, maybe point out some things the students are doing over and over again, but not mark up the entire paper, but just say, “Okay, these are the things you need to pay attention to that are recurring through your paper.” And then as they read and grade that student’s paper, they focus more on the message that the student is trying to convey and the ideas that they’re sharing, rather than the mechanics and the grammar and the spelling.

John: And one common thing I think, to both mastery learning and minimal light grading is that the goal is to provide students with feedback. In some cases that can be automated. Mastery learning systems involve some degree of automation, sometimes by textbook providers, or perhaps adaptive learning systems, or it could be questions that you put together. But if you’re going to provide feedback on writing, it can require a lot more time. And a minimal light grading approach allows faculty to provide feedback on the most important things without taking up as much time to allow faculty to provide feedback on a wider range of topics, which, again, is I think, to some extent in the same sort of spirit.

Rebecca: Light grading can help not intimidate a student with too much feedback. If you see just a paper completely marked up, it might feel like there’s no possibility for moving forward or revising. But emphasizing what’s most important to change, or most important to focus on can help a student prioritize. And this can be really important to someone new to a discipline who might not know what’s most important.

Katelyn: I’m so glad you said that.

Meghanne: There’s an element of trust there as well, because if we point out what a student needs to focus on mechanically or grammar wise in a small part of that paper, then they can be trusted to then use their judgment to go through it and read it more carefully, and then make those edits based on the feedback that they had received. So it is visually much less intimidating. Plus, it might be a motivating factor for some students too that their professor is trusting them to be in charge of that revision.

John: Another type of social justice assessment involves contract grading. Could one of you talk a little bit about how contract grading fits into this category of social justice assessment?

Katelyn: Sure, I think contract grading is one of those terms that’s gaining some broader popularity and recognition. So it’s probably a term that may be pretty familiar to a lot of instructors at this point. So maybe it doesn’t need a lot of explanation. I’ll just say there’s a couple of different models of contract grading. In some cases, the instructor might provide that contract at the start of the term. In other cases, the instructor and students would be able to negotiate that contract collaboratively together at the start of the term so that students have more of that active stake in the contract itself. Generally, the grading contract would lay out certain requirements which students would need to fulfill to receive their desired grade. And that might include requirements related to attending class or conferences, completing low-stakes assignments, completing major assignments, maybe some page- or process-based requirements. But the bottom line is that the contract gives students a clear picture from day one of the work required by the class so students can look at that contract and know exactly how much work they’re going to need to complete from day one, to get the grade that they really want to receive in the course. I think the additional benefit of contract grading for our conversation is that it decouples grades from assessment so students have more space to take risks in their work rather than aiming for correctness. And on the faculty side, faculty can respond to the content and spirit of the students work as opposed to justifying a grade. I think most important, though, because this system privileges students who are investing the time and effort into their learning, all students have the same potential to earn a high grade in the course regardless of their knowledge or ability with the subject matter prior to the start of the course. So to use another sports metaphor, it works to level the playing field on day one for students who may have very different levels of preparedness and experience with the subject matter.

Rebecca: Another strategy that folks might use, which we’ve certainly talked about quite a bit on this podcast at various times is peer assessment. Can you talk a little bit about what that looks like and how that fits into this social justice model?

Judie: So peer assessment, or I tend to call it peer review, helps to build student investment in writing, and helps the students understand the relationship between their writing and their coursework by helping them engage with the writing in a way that encourages more self reflection and works to help them build their critical thinking skills about their own work. And I think it also helps the students learn from one another, because they’re sort of trying to evaluate their peers’ work against the requirements for the course. But then you also look at your own writing in a new perspective, and you learn from what you’re seeing your peers write and from the feedback that you’re receiving from your peers.

John: Might students perhaps take feedback from their fellow classmates a bit more seriously than they do feedback from their instructors.

Judie: A lot of students self-report that they learn more from this peer review activity, because they’re trying to identify and articulate weaknesses that they’re seeing in their peers’ papers, and also in their own. And I think trying to incorporate feedback from both their peers and their instructor into their own work, I think, just helps raise that awareness and any kind of feedback that’s constructive, as they think about it and reiterate it and rewrite their work. It just helps with their critical thinking. And I think just raise awareness of how they write, and maybe they can be more thoughtful about what they’re writing going forward. I think they also, if they question their peers, say “How did you come up with this?I love this idea,” then they can apply some of this, that they’re learning from their peers to their own work, too. So perhaps that’s what you were getting at John, when you asked that question was, they may benefit more from their classmates telling them how they came up with their ideas than from their instructor just dictating what the expectations are.

Rebecca: I would expand the model to include not just writing but also other creative projects and things. It’s certainly a practice that’s pretty common in the arts, for example, to do peer review of student work.

John: And they also get to see what their peers are doing, which can serve as a positive role model. When students see that other people are doing something that they hadn’t considered doing, it could serve as a way of improving their work.

Katelyn: I think a lot of students come into the classroom thinking of their teacher as the sole reader or audience for their creations throughout the course of this semester. So anytime we can expand those audiences and have students thinking rhetorically about who else might be the consumer of their work. I think that that can benefit our students in really important ways.

Rebecca: It also seems like it’s a good opportunity to formulate community around an activity like that.

Katelyn: Absolutely.

John: One of the other areas you address with this group was the topic of labor-based grading, could you talk a little bit about that?

Meghanne: Yeah, labor-based grading removes the focus from the end product assignment and shifts it to the process of creating that piece of work. So students are provided with feedback throughout the process regarding their labor or the work that they put in. And they’re given opportunities to continue working to improve what they’re producing, and to achieve a desired grade based on a contract sometimes, so there is some overlap with contract grading, but not always. There typically aren’t penalties for students who revise and update their work, because that’s part of the learning process. And it really helps students determine what their end grade may be and how much effort they want to put in, because often, they will be given some sort of guideline for what different grades may be achieved based on certain levels of effort, or certain levels of work that are completed. And also there may be opportunities to grade based on completion rather than more of a subjective sort of qualitative grade.

John: So do you mean like using a light grading or minimal grading where you either completed satisfactorily or you haven’t, and as long as you complete a certain number of assignments or activities, you achieve that grade,

Meghanne: That or also if there’s criteria, like a rubric, and they hit all of the criteria, then they receive full credit.

John: Which becomes, actually, I think, a form of specifications grading.

Rebecca: And then one other model that you’ve talked a little bit about already today is ungrading. Can you expand upon that a little bit more?

Katelyn: Yeah, so ungrading works to deprioritize numerical grades or even attempt to eliminate them entirely. So I hope I’m not speaking out of turn when I say, I think that this is the most controversial of the approaches that we have been researching, it tends to get the most pushback from faculty because it is so different from what we have often been taught or trained to do. So instead of focusing on those numerical grades, instructors are encouraged to focus on providing learner feedback that encourages growth. Okay, I have a quote that is from an ungrading expert I’d like to share. This from Sean Michael Morris and he says, quote, “at the foundation of ungrading, lies something that could change school entirely. A suggestion that ranking and evaluation and the concomitant expertise of the ranker or evaluator is entirely an optional way of viewing things.” And I’m going to end the quote there because I think that that important kernel is that ungrading works to dismantle the hierarchy of the classroom and refocus the attention on individual student learning is an approach that requires a lot of trust between student and instructor, and a lot of student buy-in as well. Students have to be invested in the learning that’s going to happen throughout the course itself. And in a completely ungraded classroom, student grades might be based simply on a final student reflection, or even a one-on-one conversation between teacher and student about the grade that the student has earned. But because ungrading really rejects transactional grading systems, the final grade is more of an afterthought than an important outcome of the course, much less important than learning that’s occurred throughout the semester.

Rebecca: So today, we’re recording on August 9, James Lang posted on Twitter about how deep the system of creating actually is that there’s even things like discounts for insurance, for good students, or good grades. And that it’s really challenging to overcome a system that’s so ingrained beyond just our education system, but into many other systems as well. So I think that that, in part, is why there’s such a strong pushback on this particular method.

John: And we’ve always done it that way, at least for the last century or so.

Rebecca: Change is hard.

Katelyn: Yeah, I think that the traditional grading system is really embedded into not only academia but outside of academia as well. And even within a class that takes an ungrading approach, we still face that question at the end of the semester of “Well, what’s the grade going to be in the system?” because we don’t really have the option, at least at most institutions, to say, “No grade, job well done.” At least at my institution, I still have to put in a letter grade for the student. So we can work to reject that system as much as we can. But at the end of the day, we’re still operating within that same structure. And maybe that’s a question of what’s next, right? Like, are we going to see one day a future where more universities embrace this idea of learning for the sake of learning as opposed to learning for the grade? I don’t know.

John: One of the other things you address on the website is how perhaps the use of authentic assessment or UDL types of assessments might improve equity by providing a more equal playing field for students. Could you talk a little bit about how going beyond the traditional term papers and tests might provide a more equitable way of assessing students’ learning.

Judie: I think anytime you use authentic assessment that helps, or generally it allows the students the opportunity to demonstrate their learning in the way that works best for them. The students are writing a term paper, for example, they can write the paper the traditional way, or they can give a presentation or record a presentation, and still provide their citations and so forth at the end. Or they can do something visual, some sort of a PowerPoint or a nice visual display of the topic and again, cite their sources and explain their images to the group so that people understand how they’re meeting the learning outcome. And I feel like that’s just a good way if people are struggling with language, if people are just struggling with writing in general, I think that this levels the playing field, because it gives everybody an opportunity to really show their knowledge and shine and not just pigeonhole themselves into one more paper or one more multiple choice test, if they have test anxiety. Some of our traditional forms of testing or final assessment just set students up to fail. And allowing students to choose to demonstrate their learning in a way that they’re good at sets them up to succeed. And I think that’s what we really want at the end of the day. And of course UDL principles, those are Universal Design for Learning, and that does include equity in its heart. So that would definitely help to keep things equitable in the classroom. If you’re following UDL.

Rebecca: The multiple forms you were just talking about is a great example. [LAUGHTER]

Judie: Last semester, I had a student who, they’re supposed to do a blog post, and the student instead of writing a blog post, he made a video and he did it three different times. So one is on World War One, one’s on World War Two, and the third one was on revolutions, and so, this student stood in front of a whiteboard, and he had his camera set up so he could film himself. And he had his iPad in his hand. So he talked about a battle, say, for example, and he would draw it out on the board. And then he would show his citation on his iPad. And then he had other citations typed up and taped to the whiteboard. And he went on for 15 minutes, and just was making sure he explained things again, and drew little examples. And he was so animated, and so excited about his topic. And you’re not going to capture that on a written exam, or even in somebody’s written paper. It was just tremendous the way he was able to show all that he had learned and all that he was interested in, and the extra research that he had done, because he felt the freedom to pursue this topic, because he knew he was able to express it the way that suited him the past. And it was just amazing. So I think anytime we can incorporate these things, and I understand that there are times when, according to your creditor, or people have to sometimes sit for a specific certification, it doesn’t always fit, but I think if you can fit this type of assessment in, it is definitely worth it. Because just to see the joy in students when they can explore and expand their knowledge, and then feel confident in demonstrating that to you, it’s just tremendous.

Rebecca: I love the flexibility in demonstrating knowledge and understanding and skill sets because in some of our traditional methods, we are arbitrarily assessing something else. So we may be arbitrarily testing how well you can take a multiple choice test or how well you can take a test within a certain timeframe, or how well you can write, whether or not that’s actually the topic. So if I’m learning about history, there’s some learning objectives I’m trying to meet related to history that may or may not include writing. And if writing is not one of those outcomes that we’re hoping for, then we don’t need to be assessing it.

Judie: Exactly. He did this thing on medical advancements in World War One, it was just tremendous and he was so charming, because he just was so wrapped up in it that you just had to root for the guy. It was good.

Rebecca: I love that. So for those of us who may want to move towards equitable grading systems, what are some initial steps we might take? Because it could feel really daunting if you haven’t ventured down this path before.

Meghanne: Yeah, if you are not interested in overhauling your entire grading system, just to try this out, a nd to make your assessments more socially just, there are some adjustments that can be made to existing assignments. And really, the important thing is to consider the learning objectives and really think about what needs to be graded. So one of the things that we’ve talked about a lot in all of our different presentations that we’ve done is whether or not to grade for things like grammar and spelling, and mechanics, and English language proficiency. So in an example, like a discussion board, when you’re really interested in what the students have to say, and their interaction with each other, and the questions that they asked, does it really matter if their grammar and spelling is perfect in that instance, if they’re having a great conversation on a topic, and they’re learning from each other. So that’s one thing that we could suggest. Another is thinking about just the fact that sometimes students have challenges in their lives. They’re human beings, they have families, they have jobs, many of our students are athletes, and they have to travel and they have games and something like flexible due dates is very, very helpful for students because then they’re able to complete their work, certainly within a reasonable timeframe. But if those dates are a little bit more flexible, and they have access to those assignments in the learning management system beyond the actual due date, for instance, then that gives them the ability to complete that work without being penalized. So another mechanism would be in the learning management system, when students are taking quizzes, would be allowing backtracking, allowing students to go back and check their answers, that sometimes is a setting that a lot of professors really rely on, to try to avoid cheating. And as an LMS administrator, that is something that I see a lot. And I think that that can really be harmful to students, because many of our students are told to always go back and check your work. And if they’re not allowed to go back and check their work, that can be very frustrating. And also forcing completion is something that I would recommend turning off because again, that can create test anxiety. And often I think when completion is forced, there’s also a timer. So I think if any timers can be removed as well, then that does a couple of things. It can help remove testing anxiety. But then also, if there are students who require extra time due to a disability accommodation, then the professor at that point doesn’t have to go in and adjust all of the LMS settings for those students, because it’s already open ended and everyone can have as much time as they need to complete that assessment. So it really is just important to look at what the learning objectives are and what actually needs to be assessed. And the goal is always to remove barriers. So another thing that can be done is to just ask students, have a conversation about it, and find out what barriers they’ve experienced.

John: At the start of this. You mentioned the website that you were creating, could you talk a little bit more about what resources are there and how that might evolve over time?

Katelyn: Yeah, so the website, we have been slowly adding resources to over the past two years. And at this point, it’s becoming a pretty robust little outlet for people interested in social justice assessment. So, you go to the website, you can find an overview of the big picture theory of social justice assessment, as well as the various approaches that we’ve discussed today. We also have a really pretty large bibliography of resources for further reading for people who want to learn more about any one of these topics. And we’ve been working to develop a collection of sample assignments from faculty across SUNY. So we’re still working to collect additional sample assignments from faculty who might already be implementing some of these strategies within their classrooms. I think the more we can share those assignments with one another, the better off we’ll all be. I think a lot of us are doing social justice assessment in small ways in our classroom without realizing it. So the more we can share those resources and that knowledge, the more hopefully we can get people on board. So, hopefully, we’ll be able to share that link in the show notes. And people will be able to check that out.

Rebecca: So we always wrap up by asking: “What’s next?”

Judie: So for our little group, one thing that I think might be next for us is SUNY is updating the SUNY general education requirements that are mandated with the completion of any SUNY degree. And they’ve added a requirement for equity, inclusion, and diversity. So I’m hoping that our group can help contribute resources to that effort, and our website could be one more place where people go to for information on social justice assessment so that they can incorporate those into their courses that are designed to meet the DEI requirement.

Katelyn: Well, I’m gonna go take my one-year old to the pool. [LAUGHTER]

Judie: Nice.

Katelyn: I think, big picture, though, the “what’s next” I want to just give is, I hope that we’ll start to see more institutional support for some of these approaches. I think that there are still a lot of barriers, particularly for contingent faculty who want to embrace some of these practices. So I hope what’s next will be more departmental institutional support for this: more time, more resources, etc. But yeah, my personal what’s next is I’m gonna go enjoy this beautiful day.

Rebecca: Meghanne, do you want to add anything?

Meghanne: Sure yeah, at my institution, I am sharing this information, pretty much any chance I get, I’m meeting with our new incoming faculty in a couple of weeks. And this will be one of the topics that we discuss. And I’m also co-chair of our universal design for learning task force. And we have a few events and projects that we’re working on to spread the word on UDL, and also innovative assessments and social justice assessments as well.

Rebecca: Lots of great things coming and some really wonderful resources that you’ve shared today. Thank you so much for joining us.

Katelyn: Thank you.

Judie: Thank you for having us.

Meghanne: Yeah, thank you.

John: And thank you for all the great work you’ve done on this over the last couple of years and the resources you’re sharing.

Judie: I would just like to say that Shena Salvato is also in our group. She’s at Cortland, I believe. And Chris Price from SUNY is in our group, and they are missed today. They’ve been with us for all our other presentations. I know that Shana in particular wants to get the band back together and have some more meetings going forward so we can keep working together. And it was really good to see you guys again.

Katelyn: Likewise.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

257. PsycLearn

Posted in adaptive learning, assessment, backward design, cognitive psychology, cognitive science, equity, first-generation students, higher education, learning, pedagogy, retrieval practice, spaced practice, teaching
October 5, 2022

Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, Anna Yocom, Linda Goldberg, and Alan Strathman join us to discuss how the American Psychological Association has developed adaptive learning packages for core psychology courses.

Show Notes

Transcript

John: Adaptive learning platforms provide each student with a customized learning path based on the student’s individual learning needs. In this episode, we examine how one professional association has developed adaptive learning packages for core courses in their discipline.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

John: Our guests today are all involved with a psych learning project, which we’ll be talking about throughout the podcast. Let’s have each guest introduce themselves.

Anna: Hi, I am Anna Yocom. I am a senior lecturer at The Ohio State University in psychology and I am also a Content Manager for PsycLearn.

Linda: And I’m Linda Goldberg. And my role with the PsycLearn project is that of product evangelist.

Alan: Hi, this is Alan Strathman. I’m a content manager for PsycLearn and was a professor of psychology for a long time at the University of Missouri.

John: Welcome, everyone. We’re really happy to talk to you about PsycLearn. I first heard about this when I was working on a project at SUNY dealing with adaptive learning solutions. And we had seen some write ups of PsycLearn so we want to find out a little bit more.

Rebecca: Today’s teas are:… Alan, are you drinking tea?

Alan: I’m drinking coffee.

Rebecca: Oh, a coffee drinker.

Alan: Sorry.

Rebecca: I see how it is. [LAUGHTER]

Alan: I had tea this morning this morning.

Rebecca: Oh, that’s good. That’s good.

Rebecca:: How about you, Linda?

Linda: I’m having an iced tea this afternoon. It’s hot and muggy outside, so something citrusy and decaf.

Rebecca: …and perfectly refreshing. [LAUGHTER]

Linda: That’s right. And sweetened.

Rebecca: Awesome. Anna?

Anna: I had tea earlier, but I have switched to water.

John: And I’m drinking spring cherry green tea.

Rebecca: Nice. I have some Scottish breakfast. Still in my pot today.

John: We’ve invited all of you here today to discuss the PsycLearn adaptive learning platform developed by the American Psychological Association. Could you tell us a little bit about the origin of this project?

Linda: Okay, I’m going to start with that. And there are a couple of overlapping facets to the origin story, so to speak. One is really a desire to expand APA’s publishing program. APA has a very rich publishing program, which includes publishing of over 90 academic journals, a very robust books publishing program, where many of those books are aimed at clinical practice, as well as books that might support the professional researcher, books that might support self help, and so on. And we found that there was an opening for us to make a contribution to undergraduate academic instruction and contribute to that portion of the path toward the profession. We also were recognizing as we contemplated this new project that there was a tremendous amount of disruption in the higher educational publishing landscape, a lot of cost pushback, a lot of desire among publishers to leverage technology. And we thought that perhaps not simply for delivery of content, but to improve the learning experience and to improve efficacy that we saw there might be some opportunity there.

Alan: Yeah, also, let me jump in. I think the developers also saw this as an opportunity to work toward some of the goals that APA has for the field of psychology, including, for example, utilizing psychology to make an impact on social issues.

Linda: And the field of cognitive psychology is one that tells us a lot about how we learn. So we felt like we could draw from our very own field in development of our final product,

John: Which courses have been developed for this platform.

Linda: So we started with a course for research methods and psychology and added a PsycLearn statistics for the behavioral sciences to that. These two courses are typically required for all majors in the discipline. And they also represent the most challenging content for many students. So we thought that seemed like a good place to start.

Rebecca: Can you talk a little bit about how this material was developed?

Alan: Sure, we use a backwards design process where we start by identifying what we want students to learn in each module and a module is similar to what would be a chapter in a regular textbook. These outcomes that we have in mind become Learning Objectives, and they really guide the content development. Learning Objectives are developed through collaboration, a conversation, maybe a hashing out between the subject matter experts and the content managers, and are influenced by a variety of factors, including how well the learning objectives address APAs guidelines for undergraduate education. And then there is a rigorous review process where other experts in the field provide feedback and often suggest additional learning objectives

John: Was there much of a challenge getting agreement on learning objectives. I know in many disciplines, there’s a lot of discussion of what should be in the core material in the discipline? Has psychology been able to overcome that a bit by agreeing on these standards?

Alan: Well, I think it is a challenge in every case, because everybody has a sense of what they think should be taught and what is essential for students to learn. Like I said, I think it’s hashing out. It’s sometimes a little bit of trial and error, where we identify a learning objective, and then start writing about it. And then at some point, we may decide that’s not the right objective to have. But I think in the end, the subject matter experts and the content managers do a really good job of coming to some consensus on what we want to teach.

John: Was the content in the courses, the actual readings and other materials, developed by the people who created the package?

Alan: Absolutely. We are a digital first product, with original content developed by experts in various sub disciplines in psychology. And this really gives us a big advantage I think over print textbooks. So starting from scratch, in an entirely digital environment, we can develop a seamless presentation, where we start with learning objectives, we present content, we design activities and interactive exercises, and then have assessments. And we can be sure that all those things are aligned with learning objectives. And we don’t have to try and retrofit learning objectives into existing content. And then just hope that assessments that we have address the new learning objectives. So I think being able to work in a digital environment allows us to have this really seamless presentation where we start with learning objectives and we go all the way through assessment, and all those things match.

Rebecca: You’ve hinted a little bit about maintaining the content, given that it’s in a digital platform. But can you talk a little bit more about how you maintain that or what that review processes is or how that gets updated on a regular basis?

Alan: Well, that’s a good question. One issue for us is that we’re still so new, that we haven’t had semesters and semesters of feedback from students that we can use to make revisions. But what is really exciting is that whenever we decide that we want to make a change, we can do it in real time. Think about a print textbook, whenever they decide that they want something to be different, they might wait three to five years for the opportunity to make those changes. With us, since it’s digital first, we decide something needs to be changed or updated or revised and we can make those changes in a really timely fashion. And I think it really helps to keep our content current.

John: And if you find that students are not achieving the learning objectives, you have the ability, at least, to make some changes, either redefining the objectives if perhaps the objectives didn’t work too well. Or if the learning materials weren’t quite aligning with that. How often is that type of change done?

Alan: Well, so far, we’re still into the development process, that we’re making those changes just all the time. As the products get older and more mature, I think we’ll need to do that less. But again, we’ll still have the opportunity to do it as often as we’d like to.

Rebecca: John mentioned at the beginning of the episode about his interest in this project related to adaptive learning. Can you talk a little bit about the adaptive learning component of this project and the platform that you use.

Anna: The platform that we use is CogBooks, it is a platform, I think, is really unique compared to other textbooks, because it allows students to engage in their learning, and really direct their own learning pathway. And that’s based on their understanding of the content. So instead of going through a whole chapter, and then maybe quizzing yourself or testing yourself, after each content page, they get some formative questions and they get to make their own assessment. They get to say, “Okay, did I understand that? Was I not too sure? How did I do on those questions?” And then they can choose whether they want to get that support material or not, which we think of as like, “Would you follow up and ask your instructor a question.” That’s how we think of this support material. So if a student is very confident with the material, they’re essentially going to get a different path than a student who might need help in a couple of different areas or a student who might need help in a lot of areas. There’s just more examples or more practice. So it can look a little bit different for each student, which is pretty cool.

John: Linda mentioned earlier that psychologists have done a lot of research and cognitive scientists have done a lot of research on how people learn, how have some of those principles of how we learn been built into this platform?

Alan: Well, Linda is certainly right. There is a lot of research on learning science. And because we are psychologists, we’re familiar with the research, and use that knowledge to help students learn throughout the product. I think the centerpiece of our efforts to use learning science is the inclusion of metacognition, that is helping students evaluate their own learning, helping them identify what they know and what they don’t know. And when they identify what they don’t know, then they could spend a little more time learning that particular topic. Often students get to the exam, and they haven’t really done a good job of testing themselves, identifying what they know, and what they don’t know. And the first time they realize they don’t know something is when they get to the exam. And that’s obviously not the right time to get there. And so we’re able to identify ways that we can help them evaluate their own learning. We help them in this process by incorporating those learning strategies that we know are supported by research. We present concrete examples throughout. We have frequent activities and assessments. Students get practice retrieving content from memory. And these opportunities are spaced just as the research suggests they should be. They have interactive exercises. They get practice in the elaboration, or explaining content to themselves in their own words. And really the process of taking material and from the jargon in which it’s presented to explaining things in their own words, is a really useful process for students to engage in. And we could have a whole podcast just on principles of learning science that PsycLearn incorporates.

Rebecca: Can you talk a little bit about what the student experience is like for students that are using the platform? So Anna, you hinted a little bit at this with some of the adaptive learning components. But what does it look like from: I entered this space from the beginning, and now I’m interacting with this over time.

Anna: So students really, I think, approach PsycLearn differently than a printed textbook, again, not only just the fact that they would go through a printed textbook, and then maybe ask themselves some questions, probably skip over a lot of reading, because maybe it’s not very engaging or easy to get distracted with print textbooks as well. But one of the really interesting things is that they’re engaged with the content pages within them. So they’re not just engaged outside of content, like “Oh, refer to this video,” or “you can reference this activity” or “try some completely different platform to help with your reading….” but within the content pages, they get activities in which they’re asked to reflect on the course material. How does this connect to things you already know? So they get to respond to those things within the content pages. They get practice formative questions, formative meaning that it’s not just did I get this right? Yes or no. But they get feedback. So they would find out what a suggested answer was to a short answer question. They can compare that to their own. With a multiple choice question, they get follow up on why that is the correct answer, not just “Well, if I’ve missed this a couple of times, and they get multiple chances to respond, but they get some explanation about why that was the correct answer.” And so we have video questions, we have matching exercises, we have short answer, we have multiple choice questions, different drag and drop. So we’ve tried to just vary the kinds of interactions that students would have, the ways that they can practice answering questions. And this variety really helps maintain interest throughout. So they get to engage it in lots of different ways. It’s not just a rote memorization, but they’re actually applying the material as they go through each module.

John: So you mentioned formative questions. Are there are some assessment tools built in to evaluate students’ learning for grading purposes, for example?

Anna: There are. Yeah, so there are summative questions. And we try to have them associated with those learning objectives that Alan had talked about earlier. And so after maybe one, two, or a small group of those learning objectives are covered, there is a short sample, maybe 5…10 multiple choice questions where students have a few attempts, and we want them to get to about an 80% accuracy. And one of the nice things is if they don’t on their first attempt, they can actually go access those support pages then. So maybe they skipped over them earlier. And they thought, “Okay, I got this, it’s fine. I understood this page.” And then they get to these questions later on, these mastering the content and they’re like, “Well, wait a minute, I was missing some of this.” So they get another chance to go back to those support questions, and try it again. So then hopefully, they can get those right after three attempts. We don’t want them to be punitive. We want them to get engaged with the material and get to that mastery.

Rebecca: So now that you’ve talked a little bit about the student experience, can you talk a little bit about how this supports instructors? What kind of feedback is provided to instructors and maybe how instructors are using this in their teaching.

Anna: One of the things I really like, we all know, anytime that we can even maybe think back to our own experiences in classrooms, it’s a struggle getting students to read. But a lot of these classes, I mean, really all of our classes, if we’re assigning some sort of book, some reading material, we want them to read it. So how do we get them to engage? Well, that’s where the accountability comes in. So we’re giving them this tool, and then their instructor can see, “Okay, they’ve made it through this material, they engage with this.” So the instructor could choose to then reiterate that material in class, or they could choose to maybe take an active learning approach in class. And okay, now that you’ve had this background, everybody made it through, we’re going to apply it. And maybe not everybody got to 100% on every question, but maybe we got to about over 90% completion, which is what I would typically see in my classes. So you would have students who, even after a few attempts, would sometimes miss some of those mastering questions, but generally, it was pretty high. And so I could see that before I went into class. Okay, so students, as a whole, on average have got this. So I can take it a step further, where I can go back to those questions that were really challenging. So you get to track progress through each module, you can also dive a little deeper with individual students. So a student comes to you and says, “I’m really struggling,” the instructor can go in there, they can look at their pathway. Were they accessing the support material? Did they take those quizzes a few times? Where were they spending their time? Or if the instructor wanted them to cover it on let’s say, Tuesday night, were they doing it all Tuesday at 9pm? Right? Like you could actually see all of that. So you can kind of work with them like, “Well, I see you started this that night, let’s think about how we could maybe back that up. Like maybe we should pace this out at least a few days ahead of time, which is a really nice way to work with students.” And it also syncs with the LMS, the learning management system, so it really takes that burden off of instructors as well. So it’s talking to the platform that we get there. So we get this easy back and forth, which is really nice. So we can nudge students along with that, I would often encourage students to visit those support pages if they were struggling, like, “Look, I know this is really challenging,” or, “Hey, there’s more practice, if you just want a little bit more practice, you can go to some of those support pages to get that.” And the other nice thing is students can actually email right from there, which I think is pretty unique. So if a student emails me, I could see okay, they were having problems in statistics with variability and standard deviation. And it will take me as the instructor right to that page. And so then I can see that and I can say, “Oh, well, why don’t you try these examples here,” which was really nice. Instead of just a student saying, “I don’t know, I don’t get this,” they can let you know exactly where they’re struggling. So it provides a lot of options for instructors.

John: And do instructors get aggregate statistics on what areas students are struggling in addition to data on individual students?

Anna: Absolutely. So you can see that on the dashboard. You can see if there were chunks of mastering questions, for example, or a specific question that students struggled with. So you could choose to review that and maybe even go over some of those in class or go over specific topics in more detail.

John: How are instructors combining this with other face-to-face learning activities, for example, in face-to-face classes. As I understand it, most people use this as a textbook replacement, except a much more powerful replacement to a textbook. What other things will people often do to provide more of a sense of community or a social component to the learning?

Anna: Yeah, so, I think a lot of instructors are using it, just as you said, as a textbook replacement. And that’s how I have used it for statistics in particular. So it was taking the place of my other textbook, which actually was a print textbook that was online. And so I don’t feel like there were any losses. But we were gaining the ability to say, “Okay, we’re all on the same page. I know that everybody’s at least had some exposure to this. So now let’s talk about it. I can see where you were struggling. So I know we’re going to need more examples here. I know we’re going to need more follow up here.” Or “why was this question challenging? Why was this hard?” So I think drawing on those parts are how we can maintain those social ties and the social interactions we want whether it’s online, whether it’s face to face, or in any capacity.

Linda: I’ll just go ahead and point out as well that part of the content package that we’re making available to instructors who adopt, PsycLearn course material is a set of student activities that sort of exist outside of the platform itself and are available for instructors to assign or to bring to the class. They’re typically designed again to align with specific learning objectives and to give the instructor options for an in- class activity or an additional assignment activity. And that’s accessible to the instructor when they’re using the dashboard.

Rebecca: Sounds like some really great support materials. Can you talk a little bit about how this platform has affected equity gaps that we might be seeing?

Alan: Sure, I think there are two ways that PsycLearn is addressing issues of equity. First, we are designing PsycLearn to be culturally inclusive, that is we work to make sure that through examples, images, photos, figures, every student can see themselves in the product. Plus, our goal is to include the research of diverse psychologists and psychologists of color. And then I think the second way that we strive to be equitable is to design for universal access. We want the product to be accessible to all students, and so we make great efforts to do so. We use typography that’s particularly accessible. We incorporate transcripts and closed captioning, we have text-based alternatives for non text elements. And our efforts are well beyond what the minimum guidelines would be, well beyond what typical expectations would be, and I want to say that this is because we value this process so highly. We are keenly interested in making an accessible, inclusive product, and I think we’re really working hard to do that.

John: Have there been any studies of the overall efficacy of this program compared to alternatives?

Linda: Well, we have conducted some impact surveys, surveys of student users to gauge their impressions of the impact of PsycLearn on certain important aspects of their experience. Through surveying students, we looked at the impact on their knowledge gains, on their confidence in their ability to use that knowledge, and on motivation to complete assigned activities. And on the questions that we asked, to sort of tease out each of these thematic areas, we did receive very positive response. Now, these are, albeit, impressions of students. That’s sort of the first line of study available to us. And we feel pretty motivated ourselves by what we’re learning, by what we’re hearing from students. These surveys did also include some open text entry questions. And very interestingly, some of the criticisms that we heard were often this is a lot of time, this is a lot of work. But by the same token, we’ll hear in the very same response, but it’s very worth my time, and it really helps. And I feel confident when I go into the class, and I already know what my instructor is going to talk about, I feel more relaxed, I feel more able to pay attention. That kind of response has been very rewarding to receive from these student surveys.

John: With some other platforms, there have been studies that have looked at the impact on equity gaps, and they relate very much to what you just said, that basically, students come into our classes with very different backgrounds. Some come in with a very rich and strong background from prior courses, others come in with a somewhat weaker background. But to make it through an adaptive learning platform, the students who come in with less background can acquire mastery, but it takes a bit more time and effort. And the students who come in with a stronger background can race through a little bit more quickly. But much of the research does suggest that those equity gaps tend to be reduced. But it is at the cost of additional effort by those students who come in with a little bit less prior knowledge or prior experience with some of the material. And so I think that’s one of those real strengths of adaptive learning platforms compared to other formats like textbooks and lectures and so forth, where everyone is expected to move at the same pace. But as Chuck Dziuban, on a previous podcast has said, in a traditional course, everyone spends the same amount of time with the material, but they learn different amounts, when they’re working with adaptive learning platforms, they have to spend different amounts of time, but they all can achieve the same learning outcomes, they can all reach the same outcomes. But it may take more time to do so.

Linda: That’s certainly our hope, and some of what motivates the work that we’re doing. I feel like having respect for the time that our students need to spend is top of mind. We need to make sure that what we’re asking them to do is a good use of their time. And therefore this mode of delivery and the kind of content that we make available through the multiple activities and so on that Anna was describing. We’re giving our students opportunities to do that self testing that reiterative process and not simply trying to digest a narrative.

Anna: I always think back to a student I actually had in class a number of years ago, and this student went through the same class multiple times. And it was a class that was needed to get to other upper-level classes and was very frustrated, of course. And then we came to find out that this student in the multiple times of taking the class, the book was listed as required, but with either a digital or print textbook, you really don’t have any way to monitor that. You don’t know if the student’s using the book even if they have it. But this student had never got the book, just never occurred to them that, okay, this could be a really valuable tool. And so that’s really a benefit. I see. Like you were talking about equity, let’s give them this tool. But let’s make it something that’s valuable, that can really help them and if they need that added support, they’ll get it. But kind of level the playing field a little bit.

Rebecca: How do you see this platform evolving over time.

Linda: So we are able to work fairly closely with our platform partner, CogBooks. We’re able to contribute to their roadmap planning to some degree. We’ve collected feedback through a variety of channels, also, not only the student surveys that I’ve mentioned, but we also, in the past year, conducted a qualitative study that included student interviews and observations of students who were assigned to use PsycLearn. And this exercise gave us really valuable insight into how and where students focus their time and attention. And so for instance, through that process, we were able to get some real good insight into how they approach the summary of each module. And as a result, we’ll be able to give some greater attention, we’ve got some plans to enrich and make more accessible, the content that we placed there, because we were able to learn that they highly value that for later test preparation. So that’s just an example. We’re hoping to be able to continue to do those observational and interview studies.

Rebecca: So I’m really curious for each of you to answer this question. But what makes you really excited about PsycLearn?

Anna: I’m happy to go first, because I am a very energetic instructor. I will be at the front of the room, and I bring a lot of energy. And what makes me excited about PsycLearn is just the ability of students to be active consumers of their learning. I would love for them to take away something that is memorable, that feels unique. And so PsycLearn frees me up as an instructor, that I don’t have to go through every bit of background information in the way I use it in class, I can assume they’ve had things. Let’s spend time on things that are interesting. Let’s dive deeper into things. Let’s do more exercises for things that are challenging. And so the students get to think about and apply new ways of learning and engaging the material from the very first moment that they encounter the content.

Alan: Yeah, for me, it’s really about the inclusion of metacognition, I think it’s really helpful to have students be able to evaluate their own learning. I think this makes them do better on exams, and I think it makes them more successful in their entire college career. It’s also an important part of my own teaching. I have a lot of activities in class where students can answer and identify how well they know what they’re supposed to know. And what I often find is that students study, but they don’t know how much to study, they don’t know: “Do I know it as well as I need to know it.” And this product gives them a lot of feedback, really helps them understand if they understand the material as well as they need to. And PsycLearn really can do that in ways that regular textbooks cannot.

Linda: I would say that my excitement really comes from the big picture, the opportunity for us to deliver on these experiences that Anna and Alan are describing. But we’re going to be able to do that across the foundational courses in the curriculum. And so that larger picture future horizon is pretty exciting.

John: Well, you’ve already addressed some of this, at least in terms of PsycLearn, but we always end with the question: “What’s next?”

Linda: Well, we have sort of talked a little bit about future and what’s next and our opportunity to be iterative. The digital platform allows us to be continually rethinking what is the best activity to offer to reinforce or bolster certain concepts. And so that’s certainly something that is ongoing. We can take feedback from students on how well they feel various content is supporting them. We’ll be looking at what we can do to encourage a greater level and engagement with that support material, so that students aren’t feeling like, “Oh, that’s extra, I don’t really need to go there.” Indeed, it’s extra. But it’s really vital, especially for students who might sort of marginally understand. And so the opportunity for us to deliver a reframe of a concept can make all the difference at that tipping point.

Anna: Yeah, and I think what Linda had said about using the support material is really important to us. So we’re really interested in applying these learning science principles that Alan had talked about to say, “Okay, what is the best way to get students to engage with this.” We want them to see that cool stuff we’re putting on those pages, if they feel like they could benefit of it. So we want to reach the students that need that support. So I think just diving in a little deeper and figuring out the best ways to reach them, to get them to say, “Okay, yeah, I could use a little practice.” Or “I could use another example,” snd really assess that learning of their own. It’s a challenge. It’s a challenge we have constantly, especially when students are new at something, they tend to think, “Oh, yeah, I got this. I know it all.” And so getting better at that metacognition, I think, is really exciting to us.

Alan: Yeah, and I think for us, it has a lot to do with making a better and better product. I think about what it was like the first time I taught a course. And I equate that to the first iteration of a PsycLearn product. And then I think about what was it like the fifth time I taught that course or the 10th time I taught it, and how richer and deeper it was. And I think the same thing will happen with PsycLearn. We’re going to keep on creating a product that gets richer, more inclusive, and more likely to help students succeed.

John: I was really excited to hear about PsycLearn, and everything you’ve talked about makes me even more excited about the opportunity for this and I hope that other disciplines will start working on similar materials for their basic courses, because the benefits from this have been well established in terms of improving student learning and reducing some of the equity gaps. Thank you.

Linda: Thank you.

Alan: Thank you.

Anna: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

250. Hacking Assessment

Posted in assessment, classroom community, engagement, equity, feedback, grading, growth mindset, higher education, inclusion, learning, pedagogy, reflective practice, teaching
August 17, 2022

Traditional grading systems often encourage students to focus on achieving higher grades rather than on their learning. In this episode, Starr Sackstein joins us to discuss how classes can be redesigned to improve student engagement and learning. Starr has been an educator for 20 years and is currently the COO of Mastery Portfolio, an educational consultant, and instructional coach and speaker. She is the author of more than 10 books on education, including the best-selling Hacking Assessment: 10 ways to go gradeless in a traditional grades school, which has just been released in a new edition.

Show Notes

Transcript

John: Traditional grading systems often encourage students to focus on achieving higher grades rather than on their learning. In this episode, we discuss how classes can be redesigned to improve student engagement and learning.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Starr Sackstein. Starr has been an educator for 20 years and is currently the COO of Mastery Portfolio, an educational consultant, and instructional coach and speaker. She is the author of more than 10 books on education, including the best-selling Hacking Assessment: 10 ways to go gradeless in a traditional grades school, which has been released in a new edition. Welcome, Starr.

Starr: Thanks so much. I’m excited to be here.

John: Today’s teas are… Starr, are you drinking tea?

Starr: I am drinking water. No tea unfortunately, not yet.

Rebecca: Not yet. Okay. See, there we go, there’s promise there. I have Scottish Breakfast tea today.

John: And I have spring cherry green tea.

Rebecca: Well, that’s good.

Starr: Those both sound delicious, really.

Rebecca: So, you haven’t had that one in a while, John.

John: I haven’t had any in a while…

Rebecca: true that…

John: …we took a pause in recording for about a month. [LAUGHTER]

Rebecca: Fair. But even prior to that it had been a while I think.

John: I think so too.

Rebecca: We invited you here today to discuss Hacking Assessment. The first edition of your book seven years ago helped to launch the ungrading movement. Could you give us some background on what prompted you to move away from traditional grading systems?

Starr: Absolutely. In years one to five when I was in the classroom, I would say that I pretty much did grading and assessment the way it was done to me. And the one major significant thing that changed during that time was I had a child. And in his elementary school, they actually use standards-based grading. And when I got his first report card and saw just how much information I got from his teachers, and how the behaviors were separate from the actual learning and the narratives were really aligned with where he needed support and what was going on. I was like, Mmmh…for someone teaching AP English, only having the opportunity to give one grade, with pre slugged sort of comments that I was allowed to bubble into my… back then we were still using Scantrons for entering grades. I’m definitely dating myself by saying that, but it’s the truth. And I started getting really frustrated with that. And from there, I started doing a lot of reading. Alfie Kohn has really played with a lot of these ideas for a long time now. And then folks like Ken O’Connor, who had the book 15 Fixes for Broken Grades, his first edition, I think it’s been republished twice already, in the time since I’ve read from there. I read his book, and I was like, “Oh, my God, I am doing all of this wrong.” There are so many things on this list that I do, and I never thought about it that way, and it’s just not how I want to keep doing things. And I think there’s a synergy with when you decide to read a book, whether or not it resonates with you and whether or not you’re ready to start implementing the things that you learn. And I think I was very ready to first acknowledge that the practice I was doing wasn’t serving my students as well as I could. And I was looking for alternatives. So having those jumping off points, having read a bunch of different things, and then meeting Mark Barnes along the way as well, and experimenting with alright, well, these are suggestions for this kind of space. What does this look like in New York City public schools, as an 11th and 12th grade English teacher and also as a journalism teacher? How do I start making this work? And that’s sort of how it all happened and then it took years to figure out how do I make this work well, because I did it for a while before it worked well. [LAUGHTER] There were a lot of mistakes, unfortunately.

John: We’ve been dealing with a number of people starting to experiment with ungrading in college, but it’s a little bit easier in a college environment, I think, to make these changes, because there’s a little bit less structure imposed on teachers. How were you able to implement this in a K through 12 system?

Starr: So I think I was very fortunate to be in a very small community when I started doing this. We were six to 12. I was already a very established teacher in that community. I had a track record of getting students prepared for college. And most of the families when I made choices, always kind of knew that they were intentional, and there were reasons. And in my AP classes, that was probably the most struggle, because parents get nervous when they have 12th graders, what is this gonna look like on the transcript? How is this going to impact my students moving forward from school? And I just really tried to set up systems and to be super transparent about everything that we were doing so that first of all, I live streamed my class a lot, for better or for worse. And I say that because not every class was a winner. So if you were watching when it wasn’t a winner, like, well, this is reality, it wasn’t a good day. But I think they were able to see the rigor of what was going on in the space and despite the fact that it didn’t look like what normal AP classes looked like, they could appreciate my wanting to be flexible to the individual learners in my classroom… that even the creative projects I was asking them to do was often a lot more intensive than just doing a test or just writing a paper and gave that level of inquiry into that process as well so that students could be really excited about the learning they were doing. And the more comfortable I got with different technologies… I experimented with blogging to increase reading. That’s one of the biggest problems in English classes. I think most kids don’t read the books for a lot of different reasons. So how do you get them to read when you’re teaching a literature class, beyond just the five or seven or 10 books you’re reading as a whole class. So they started blogging, and we started using the blogging communities for recommendations on different books they were enjoying on their own and why they enjoyed it. And I really encouraged them to use that space too as a way to develop their writing voice. So it wasn’t like analytical writing all the time, it was more conversational… reaction sort of stuff to what they were reading and focused instead of like overviews of everything that they read… an analysis paper, which isn’t always fun for every single kid. I started tweaking that and I think parents appreciated my transparency. I did screencasts of our dashboard, because I had changed the way I was using the tool that my whole school was using. So like, if you have any questions, this is what it looks like, this is what you’re seeing. And if they emailed me, I just really tried to get back to them immediately, so that I could really put their concerns to rest before they started doing the thing that parents do, where they start making it a lot worse than it actually is. So I tried to catch that right away. To be honest, though, my colleagues were the ones with the greater pushback than parents and students… a couple of students, but just shifting the conversation away from grades, instead of what did I get? What did you learn? How can we track that progress over time? How do you know you learned it? Where do you see that evidence in your own learning. And I think very soon after getting in the routine of this is how we do things now they got it and saw that the level of metacognition as well as the rigor in the actual tasks were much greater than what they would have been seeing in a regular class anyway. So sometimes I got the: “this writing reflection is like a whole other paper that you’re asking us to do.” And I’m like, “Yeah, it is. But it also helps me give you better feedback, and it also helps me know where I need to adjust my instruction. So there’s a reason and it’s worthwhile, and it’s gonna help you, when you’re not just in school. This is a practice that you’ll probably carry with you.”

Rebecca: One of the things that you just brought up, Starr, is something that I definitely want to follow up on, which is getting our colleagues to also buy into this, and administrators. We exist within systems that require grade inputs. Grades are transcripted. So how do we get the people around us who support us professionally, to get on board? And what does that actually look like, functionally, when we’re generating grades when we’re saying we’re kind of ungrading all semester.

Starr: So those are really good questions. And in the second edition, I actually have built in leadership tips to support leaders who are unfamiliar with this kind of assessment practice and how they can support teachers who want to do it, if they’re not doing it wholesale as a school. I advocate for systemic use of this practice, because if we catch kids much younger, by the time they get to high school, their language and fluency in discussing their own learning is a lot greater. I was a 12th grade teacher, my kids had come through an entire system where this was not how it was done. So it was like, literally at the last minute, I’m like, “Yeah, I know, that’s the way you’ve been doing it this whole time. But we’re gonna do it a little differently and I promise you’ll still get into college.” It’s a different vibe. And my colleagues, I think, knew my students appreciated it, because they would start hearing from my students: “How come you don’t do this?” …which is also like a little bit of a target was put on my back, because if a school or a district is going to make the shift, it requires a lot of professional learning. And if you aren’t the kind of teacher who makes the time to do learning on your own, then there really does need to be supports put in place prior to it’s happening. And I’m a super reflective teacher, I did National Board Certification, I will go out of my way to get myself to a conference even if my school wasn’t paying for it. Because as an educator, I felt it was an essential part of my job to continually grow and model that for my students. But not every teacher is like that. And I’m not suggesting that everyone has to be or whatever their process is, but I do think it’s important to invite colleagues into your space, give them that “what you could do tomorrow kind of tips” like what are the first few steps you could take to try this out before you commit to it wholesale. And in terms of the grading aspect, the way that I got around the traditional grading was assessment conferences with my students. So really building in a vibrant and robust portfolio system where students were collecting their learning over a larger period of time, giving them the vocabulary to talk about their growth as they looked at those things, and then a conversation just like this. So based on the standards we worked on this marking period, where do you find yourself in terms of mastery? And what does that translate to for a report card grade, because I had to put a grade on the report card as well. So it was really just making them acutely aware of what exemplary work looks like, how they were meeting benchmarks to get there over time, and then also switch that transactional sort of relationship around getting grades to a more progress minded model, where they understand learning doesn’t happen in one sitting. And even though you may have successfully completed one assignment, that doesn’t mean you’ve mastered a particular skill, it’s just your first go at it. In order to get to that mastery level, you have to do it over time with less and less support, and kind of do it on your own.

John: What sort of buy in did you get from other teachers that you were working with?

Starr: It was secret at first. There were like people just dropping by out of curiosity to see what was going on in my classroom. Then a couple of other people just asking, “what would this look like in my gradebook?” I was very lucky in the one sense that our whole school was a portfolio school. So that part of it was already there. And then I also did some PD with my colleagues around reflection practices. We tried to really create something that was consistent, and also the same. So like I had created a process for doing reflection, which is that five steps sort of: first, you have to reexamine what was it I was asked to do? What were my steps for completing the assignment? Where do I think I’m meeting the goals that I set for myself? How am I doing that? What level am I doing that at? And what would I do differently in the future? And then we kind of scaffold that down to sixth grade up to 12th grade. So what is that kind of reflection look like in a sixth grade classroom, a seventh grade classroom, all the way up to 12. So that there are realistic expectations in that space around those things. And my classroom was always open. And I resented the fact that when my principal decided that she wanted us to go to a standards-based model, I implored her to not do it the way she did. I think we should have a pilot team, we should have a committee that does this, we should test it out first, try to get either a grade level team or a content area vertically to commit to doing this and then have input from more people. And then we need to train folks in the areas they aren’t already familiar with, starting with unpacking standards and getting them comfortable with that kind of language and what our expectations are. But that’s not what happened. It was like an email that went out. We’re going to do this this year. And it was a disaster. And I got attached to the disaster as a direct correlation to how all that happened. And unfortunately, you get one good shot to make a significant assessment or grading shift in a decade, because unless your folks are leaving quickly, no one forgets. So really setting up systems in the future, if folks who are listening want to do this on a bigger scale, set yourself up for a three- to five-year implementation plan, start small and grow it organically and provide tons of support along the way so everybody feels confident and not just your teachers, your community also. What does this look like for your parents? What are they going to be receiving that’s different? And just make sure that you have answers to commonly asked questions on the front end, so that when new stuff starts coming in, you’re ready to triage that, you’re not just answering the standard questions over and over and over again.

John: You mentioned in your first edition of the book that one of the motivations for this was to get students to focus on their learning rather than on grades. How successful was this? Did this work for most students?

Starr: For most, yes. And believe it or not, the ones that don’t traditionally do school well, who don’t play the game, it worked best for them. And as three educators sitting on this podcast right now, I think we can all agree that sometimes our brightest students are not the ones who do the best. The ones who do the best are the ones who are most committed to getting high grades and kind of checking the boxes and doing everything that they have to be compliant for in order to get that score. So when we shifted the focus away from that and started looking at skill acquisition and content deepening, and really getting them to be able to advocate for their own needs in that specific area, I think that it wasn’t just about them completing the tasks I asked them to do, but it required them to engage with me in a dialogue in the kinds of tasks they wanted to be doing, the way they wanted to be doing it. And it required my flexibility with taking that input and actually putting it into action. So I think that once they saw that I was listening to their feedback actively and using it right away to shift the way class looked, they understood that I wasn’t just saying, “I’m asking you to do this,” it was a real partnership, where if this is going to be successful, and you want your voice to be heard, you need to contribute or else you can’t complain when you don’t like what ended up happening, because I really did try to say “yes,” just about to everything, if they could articulate how their decisions and their choices aligned with what the objectives were, then I was totally hands off in their process to sort of help them be successful in the big picture. And it also really decreased the amount of folks who didn’t participate in the group work or didn’t participate in the learning. So when people say my students don’t finish work, or they don’t submit things, to me, that’s a red flag that either something else is going on that you need to get to the bottom of, or the kind of learning you’re asking them to do isn’t resonating. And rather than just pulling out the binder from what you’ve done for the last 20 years, you really do have to make a concerted effort to make changes so that it meets the needs of the kiddos that are sitting in front of you right now.

Rebecca: So you’ve talked a lot about reflection, and the role reflection is playing. Can you talk a bit about how you were able to get students up to the level of reflection that is really meaningful and gets to this metacognitive skill, building

Starr: Feedback, feedback, feedback. We give a lot of feedback to everything that kids do in the classroom. But the first few times we ask them to reflect, it’s so important that we’re also giving them feedback on their reflections, providing exemplars for them, really creating success criteria too, like that co-construction, like if I’m telling you, these are three examples that are wildly different, but all successful, what do you notice about all three of them? What are the things that need to be a part of every single reflection that we do. And then as they do them, rather than have them revise every single one that they do, since they’re doing them with every major assignment, it’s like, “alright, well, now take the feedback you got from the last one, apply it to this one and let’s see if we can’t grow you.” And usually by, I would say November, they’re already writing fairly good reflections and their ability to have conversation about their level of learning already starts to increase, because by November, you’ve already had a progress report conversation, you’ve already had a quarter one report card conversation. And I was doing a lot of modeling myself, like I would reflect openly on how successful projects went, in my estimation, and be really, really tied to the outcomes. And not just what I think or what I feel, but what I noticed, and how I would do it differently if we had the opportunity to do something similar again. And I think, again, that level of transparency and my comfort with saying to them, I don’t know how to make this better. What do you think? What made this experience challenging? Were my directions not as clear as they could have been? What do I need to learn from this experience? So it was very much a two-way street, which took time. And I do want to say that too. Like, I think I was seven or eight years into the classroom before I was comfortable enough to say “I didn’t know something.” That takes confidence in a way that you don’t really think. In the beginning of my career, I felt like I needed to be the expert over all of the students in my room, and I had to have an answer for everything. And I said a lot of wrong things because I was trying so hard to look like an authority. And I think the older I get, the more I work with educators, the more I realize that I’m a learner, I don’t know everything, even the stuff I’ve spent a lot of time teaching I don’t know everything about and new perspectives are incredibly useful in how I approach something because it’s the first time this group of kids is seeing something I might have tried before. Their input is extraordinarily useful for me to make changes moving forward.

John: It’s also a great way of nurturing a growth mindset in students by reminding them that we’re all part of this learning experience together. And that no matter how much experience you have, there’s always more you can learn. And so I think that’s a really great process. And it’s something that I think it generally takes a while for most people to get to.

Starr: Yeah.

John: So you mentioned having conferences with students, how often do you conference with students?

Starr: So, there’s lots of different levels of conferencing. So you have your in-class formative conversation where they’re asking questions and you’re taking the pulse of whether or not you’re going too fast or if you need to stop the class and do a mini lesson on something you notice everyone’s struggling with. Or if you pull a small group because only a small group of kids are really having an issue. So there’s that kind of on-the-fly conferencing where you’re walking around with a clipboard or an iPad and you’re taking notes on what you see. And then listening to the questions kids are asking and making a determination as to whether or not this is a small or bigger issue that needs to be addressed. And then there are formal conferences where kids are coming prepared to have that conversation where you’re giving them time in class. So part of my structuring… because remember, I said it took me a long time to find a system that worked that ended up in Hacking Assessment… so I started creating Google Forms, where there were very targeted questions that also aligned with the assessments that we did, and the different pieces of learning and the standards that we were addressing at that time. And before they could set up a conference, they needed to fill in that whole Google form, then I had all that informatio, so I could really target clarifying questions or gaps that we could spend our five minutes talking about. If they had done all the work to do certain things, they don’t have to rehash what I could read. And if I had 34 students in most of my classes, so there’s a lot of kids, there’s a little time, you really have to make that three to five minutes count, and give every student the opportunity to give you the most information that you could have to be able to determine what was going to go on the report card. So those conversations certainly got a lot better over time as well. The first one, there was a lot of prompting from me, a lot of questions to get them ready by conference number 2, 3, 4, and certainly by the end of the year, if you watch on my YouTube channel, I have examples of what those look like. By the end of the year, the student is doing 98% of the talking. And I’m just redirecting if they kind of get off a little bit, or if they miss a spot versus at the beginning, it’s more of like a 40-60 where I am interjecting and kind of bolstering confidence, helping them set goals and stuff. So there’s more of a give and take at the beginning of the year.

John: You mentioned giving students some choice in terms of the assignments and so forth. What are some of the more interesting assignments or learning activities that your students have come up with?

Starr: The one that always comes to mind was, towards the end of my time in the classroom, before I became an instructional coach, I literally gave my students my entire unit plan for Hamlet. And I said, “Alright, this is the way I always teach it. But I want to do it differently this year. So I want you to look at the overall objectives. And as a group, I want you to come up with something different, then we’re going to vote as a class, which group suggestion we want to go with, and whichever group is chosen, you’ll come meet with me at lunch, we’ll design an assignment together and work through the success criteria and benchmarks for doing it successfully.” And if I tell you some of the things these kids came up with, I would have never come up with in a million years. And what we landed on was these psychological profiles of the characters of Hamlet, where they had to first use the text, to use Shakespeare’s language, to diagnose them with some kind of psychological issue. For example, Gertrude would be a narcissist. And then they do research on the actual issue, so there’s a research component as well. And then they had to come up with a treatment plan for the character and create a movie that demonstrated the growth from whatever the treatment plan was. And what it really did was have this really in-depth character analysis of each character from Hamlet, regardless of which character you did, you were set on a course. And then we also created this Google form, so that when we had screenings of the movies at the end, students were actively taking notes about what they learned about the characters and giving feedback at the same time to the creators of those movies about what they learned and what they were still curious about. And it was really phenomenal, honestly. I think that I wish I would have started doing stuff like that sooner. Other examples would have been students creating movies in Minecraft, like for our satire movies, that’s usually so like, just technology, but I was very uncomfortable with, that they were able to use that. I was like, yeah, “If you could do it without my support, I could help you with content, but you’re on your own for the technology.”

Rebecca: So you’ve hinted at some of the changes in your second edition. Can you highlight some additional changes between the first and second edition?

Starr: Okay, so yes, there are a lot more resources. So over the last seven years, part of the reason I hadn’t made a second edition up till this point, was because I really wanted there to be a value added. I wanted there to be new voices I can highlight. I was really also looking for systems that started doing this work because I wanted there to be more case study material that kind of went in that it wasn’t just single teachers kind of playing with it, but actually systematizing it in ways that work for them. So there are brand new hacks and actions for every single chapter, all of them have read the first edition and implemented it in their own way. So what you’re getting is people’s take on how what they learned looks like. I really tried to implement K to higher ed. So Susan Blum did write a section as well on what it looks like in college for all of my reticent K-12 folks who were like, “This isn’t going to be viable in the future.” I had central office people write about stakeholder buy in and how they brought this into their space from a leader perspective, instead of just a classroom perspective. A lot of new tools that have been developed in the last seven years, lots of stuff about that, rubrics, progressions, not just in English, which was my background, obviously, really trying to span math, science, social studies, related arts. So there’s one with a music teacher writing about how they’ve done that in that area… elementary teachers. So there really are tons of resources with a lot of different fresh voices who are using this now, as well as a very intentional talk about equitable practices. I think a lot of this stuff is equitable, but I never thought of it in that lens until COVID. And then once COVID happened, really trying to talk about how these things address some of those gaps that need to be addressed, but weren’t explicitly tied to them in the past. So that’s really where the bulk of things have shifted. And then there’s an incredible appendix with lots and lots of examples of everything.

John: And your first edition was wonderful. It provides a lot of good resources. And in each section, it talks about how to deal with pushback, which is one of the things anyone introducing something new has to deal with. So I’m assuming that continues into the second edition.

Starr: Yep, sure does.

John: So your first edition was very successful, and has received a lot of traction at all levels of education, and helped spur the ungrading movement at the college level that we’ve been talking about a lot in the last couple of years with our guests, and with many of our colleagues. For those people who have read the first edition, what would be the benefits to them of picking up the second edition, and who should they share that with at their institutions?

Starr: So I’m really hopeful that this time, it’s not individual teachers picking the book up on their own, although I certainly advocate for that. I want to see teams use this as a PLN opportunity and explore the text in a way that makes sense to them. It is not narrative, necessarily. So each chapter is its own sort of entity. And so I would encourage folks to choose the chapter that they’re most ready for at this moment and pick it apart in a way that’s going to make most sense for their practice.

John: We always end with the question: “What’s next?”

Starr: Oh, I’m so glad you asked. So what’s next for me right now, we are doing a free book study with the new book when it launches August 2, and it’ll be on Amazon. And then also, once this one launches, and things are moving, I’m under contract with ASCD for my next book, which is specifically about portfolios and student-led conferences. So that is still something that’s a little thinner in Hacking Assessment, because I think that that really requires a little bit more depth than I could give it in that book in one chapter. So I am currently working on that and really trying to gather with some of the districts that I’m working with to build really great systems for building portfolios. What does that look like? And how do you parlay that piece into these student-led conferences so that you can have a robust system in your space?

John: That sounds like a great supplement. Well, thank you. It’s great talking to you. We’ve heard mention of your book from many of our past guests, and I’m glad I was finally able to get to read it. And I’m looking forward to the second edition, which should be arriving soon.

Starr: Awesome. Thank you so much.

Rebecca: Yeah, thank you so much. This is such great information and we’re looking forward to all your new work as well.

Starr: Thank you.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

246. Embedding Career Competencies

Posted in assessment, career competencies, career path, engagement, higher education, instructional communication, metacognition, motivation, pedagogy, teaching
June 29, 2022

Students generally enter college to advance their employment prospects. In this episode, Jessica Kruger joins us to discuss how explicitly embedding career competencies in the curriculum can engage and motivate students. Jessica is a clinical assistant professor in the Department of Community Health and Health Behavior and is the Director of Teaching Innovation and Excellence at the University of Buffalo.

Transcript

Rebecca: Students generally enter college to advance their employment prospects. In this episode, we explore how explicitly embedding career competencies in the curriculum can engage and motivate students.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist&hellip

John: &hellipand Rebecca Mushtare, a graphic designer&hellip

Rebecca: &hellipand features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Jessica Kruger. Jessica is a clinical assistant professor in the Department of Community Health and Health Behavior and is the Director of Teaching Innovation and Excellence at the University of Buffalo. Welcome back, Jessica.

Jessica: Happy to be back.

John: We just saw you at CIT.

Jessica: &hellipgreat conference.

John: It was nice seeing everyone back in person again. For me, and I think for Rebecca too, this was our first conference in person in at least a couple of years.

Rebecca: Yeah. Yes, it was. Refreshing.

John: Speaking of refreshing, our teas today are&hellip Jessica, are you drinking tea?

Jessica: Iced Tea.

Rebecca: &hellipthe best kind during the summer.

Jessica: Exactly.

Rebecca: I have some nice piping hot Ceylon tea again.

John: And I have spring cherry green tea.

Rebecca: Is that a new one for you, John? Or you just haven’t had it in a while?

John: I haven’t had it in a while.

Rebecca: That sounds good&hellip

John: It is very good.

Rebecca: &hellipif you like cherries, which I don’t. [LAUGHTER]

John: It’s spring cherry, it’s not just cherry, these are spring cherries.

Rebecca: &hellipthe best kind.

John: &hellipfrom the Republic of Tea&hellip actually from Harry and David. But it’s produced by the Republic of Tea for Harry and David.

Rebecca: So we’ve invited you here today to discuss your work in incorporating career readiness into the public health curriculum. Why should there be an increased focus on incorporating career readiness in our courses and degree programs?

Jessica: I like to tell people, it’s not just our job to teach students really cool things, it’s our job to help them get a job. And so ultimately, by incorporating career readiness skills, we’re equipping our students to go out into the world and get their first job. And really, in public health, there’s so much work to be done. And so if students aren’t ready to do that interview, or send out their resume, or even talk about their experiences that they’ve gained in the classroom, they’re not going to land that job.

John: Has it been hard to convince other faculty of the need to provide this career readiness for students?

Jessica: So myself and another faculty member have been incorporating the career competencies in our courses, we both teach a 200- and a 300- level public health course. And, in those, we are getting students anywhere from sophomore, juniors, or seniors. And I think it makes sense in our profession where many students go out and get jobs after their bachelor’s to have them start thinking about this early, even as early as their second year in our courses.

Rebecca: Can you talk a little bit about the kinds of careers that your students in your program pursue?

Jessica: Yes, so we have students working all over. And I’m really proud to say we have a student who’s actually working at a safe injection site in New York City, which is one of the first that has been opened. We have students working locally in health departments, working for hospital systems, and local nonprofits. And so our students are really going out there and doing what needs to be done, especially what we’ve seen over the past two years with the pandemic. We’d had so many students work in contract tracing, and also with local health departments, and it’s growing. Public health is becoming more recognized. And that’s why we need to continue to prepare our students for what’s next.

John: For those listeners who aren’t familiar with safe injection, could you talk about what that is?

Jessica: The first facility was opened in New York City. This allows for individuals who use injectable drugs to go into a place that is clean and monitored and inject safely. Someone is there to monitor them, provide them clean supplies, and even help them if there is an overdose. So this is a harm reduction technique that prevents deaths.

Rebecca: Can you talk about the ways that you have employed career readiness into your courses?

Jessica: We use the framework of the NACE Competencies and NACE stands for National Association of Colleges and Employers. And these are competencies that were set forth by actual companies, employers out there, saying what they actually want in a new graduate. So there are actually eight different areas, things from teamwork, leadership, ensuring that students are able to be critical thinkers, have career and self development, include equity and inclusion, and be proficient and technology. And so what I’ve done is incorporate first starting at the syllabus level, incorporating some verbiage saying: in this course, you’re going to learn career competencies, and we’re going to cover seven out of eight of these career competencies. Because I’m teaching a 200-level course, I’m not focusing on the skill of leadership, that’s a little bit higher level that I’m working at in this course. But in every assessment that I have in class, I have not only the objectives, why we’re doing it, but I include what competencies we’re working towards, and how they can talk about this in a job interview&hellip to a internship site. So it’s not just that you’re writing a paper, you’re working on those written communication skills, and you’re able to articulate that

John: How have students responded? I would imagine it would increase their motivation a bit when they see how directly applicable these skills will be for them.

Jessica: It all comes down to transparency, because the more transparent and applied students feel that their assessments are, the higher quality I tend to see their work. And so by telling them, you’re not just writing a paper for me to read, you’re writing a paper to practice this skill. And you’re also going to do a presentation, because in public health, you need to have excellent written and oral communication skills. And hey, you’re also going to make a poster on Canva, because you need to know how to use technology. And so by kind of stringing these competencies together, it allows students to see that what they’re doing is not just for a grade, but to help them build those skills.

Rebecca: Can you walk us through a specific example?

Jessica: Yeah, in this course, I have students do a variety of writing samples on different problems. So, in public health we’re very applied. In one of the papers, students are talking about how public health has been influenced by other areas&hellip philosophy, psychology&hellip and so they could just see it as a paper, a 1.5 page paper, or thinking about it from a career readiness standpoint, they’re learning how to write succinctly and to whatever audience&hellip so, in this case, a lay person learning about public health. And so in writing this, I include why they’re doing it. And when I’m giving the example of what I’m looking for in the assessment, I often have the students reflect on what they’re doing and why they’re doing it. I think it’s important for them to think about that, why am I writing this paper? Why are we doing this? &hellipto improve your skills, to work on that written communication. Another example that I think is probably maybe a little bit more applied here is students write a paper, they record a short pitch of this paper using Flipgrid. So they’re doing written communication, oral communication, and in that little flip grid that they’re doing, they have to dress professionally, practice that skill, and give feedback to peers. And so all of that together, which is seemingly not a too arduous of an assignment, is really hitting on a lot of those career skills.

John: Have other faculty in your department picked up on the use of expanded career readiness in their classes?

JESSICA :Yes, one other faculty member has joined me and currently we are studying the effects of this on our students. So what we have done is ask our students at the end of the semester to complete a short survey asking them about how they felt about including career readiness into their courses. And overwhelmingly, students are so appreciative that we’re thinking about this, preparing them for what’s next. And it also shows that we care, we’re not just there to be a sage on the stage. We’re there to help them get ready for that job, that grad school, or whatever else they choose.

Rebecca: Do you have other findings from your research so far? I know you’re early in this process.

Jessica: Another major takeaway from this research is students wish they had this much earlier. They wish that we started talking about it day one. And while we could have told them about the career design center on campus, they’re not forced to practice this in their courses. They’re not tying this to that NACE Competency framework. But really, when we think about it, career readiness is everyone’s job. In our University of Buffalo, we’re working to create a career ecosystem, meaning that faculty are really on the front lines of this. They’re the first people that students are going to go to for career advice. And not all faculty are equipped to have that conversation. But at least, at the very minimum, being able to direct them to the Career Design Center and other resources that are available through the Career Design Center, I think is key. One simple way that I’ve introduced students to the actual physical Career Design Center on campus is I have them go take a selfie. They have a great little selfie station, and that’s one of their first things that they do in my course. So I can start to put a face to a name. And they can also learn about why it’s important to have a professional selfie. So they can put on their digital profiles like LinkedIn.

Rebecca: I found in some of the classes that I teach, I’ve in the past done assignments where we do professional email communication related to the work that we’re doing, or reports that might be common to the discipline. And students have responded much more positively to those kinds of writing assignments, because they can see the practical application and can connect the very specific, like, I can see how I’m doing this skill, and it’s gonna result in me being able to do this other thing. We just don’t always articulate that when it maybe is a little more abstract, when it’s maybe a more traditional paper and how that might tie to the kinds of work that they might do as a professional.

Jessica: Most definitely. And I’ll say this really was not much effort, I did not change what I was doing in my courses. I really just added a sentence or two to each of my assessments, relating it back to the NACE Competencies, I also brought a little bit more awareness to what we were doing in class. When we would do group work, I’d say, “Oh, you know what? You’re working on teamwork. That’s great, because in public health, we work in teams, and this is also related to the NACE competencies.” So it doesn’t have to be like a capstone course where you’re deliberately working on resumes or other career competency. But I think weaving this in, adding to it, and really raising awareness about some of these skills can really help students go to that next level, or start thinking about what they need to do to build some more skills before they leave our universities.

John: You’ve already answered this partly by talking about teamwork and other skills that can be done in any class. But what about those instructors in a course on abstract algebra? How would you build in, say, career readiness into that, other than the types of things you’ve already mentioned?

Jessica: Well, I think teamwork and technology would be two very easy competencies to weave in to any sort of course. Also, if you’re doing advanced level math, you’re probably using critical thinking skills. And you’re probably talking about what your findings are. So you’re hitting communication, you may not have much on professionalism, or career and self-development, but still, you’re hitting five career competencies, and not having to add anything, but really just highlight what you’re already doing.

Rebecca: Were you surprised when you sat down and looked at your syllabus and your learning objectives and looked at the NACE competencies and put them side by side and how well they aligned?

Jessica: Yeah, and in fact, it’s interesting, because in public health, we have an accrediting body called CEPH. And our CEPH Competencies for preparing students at the baccalaureate level actually align perfectly with NACE. And so it made sense, once you sat it down to say, “Oh, I’m already doing a lot of this.” And yeah, it might take part of my class time the second day of class to bring in one of the career designers so they put a name to a face, but it’s not taking away any time from my instruction. And by adding the transparency in that connection. I think it’s created more of a caring environment for my students. They know I care about what happens to them after this course. And more students have come to me than ever, asking about what’s next, whether that’s grad school, or how to apply for that first job. And that’s really rewarding. Now, I know not all faculty want more meetings with students. We all have busy schedules. But it’s also great to build those connections, because those are the students who are going to continue to be connected, have mentorship and be successful.

John: And we’ve always been preparing students for their future lives and careers. But we haven’t always been that transparent about it. And it sounds like that’s a really good approach. For someone who wants to start building this into their courses, how would you suggest they get started?

Jessica: I think one of the first things is go have a conversation with someone in your career design center on campus, see what they’re offering and see how you can collaborate with them. I found that our Career Design Center at University of Buffalo has so many resources so that faculty can literally plug in modules on career development that are already created for them. It was also really enlightening to learn what it’s like to be a student, to go to the career design center. And so sitting down with someone and understanding some of the intakes that they do, some of the questions, and even some of the tools that they have, really helps give you an overview of everything that can be offered to students. So when that student comes to you and says, I’m thinking about this career in biostatistics&hellip Oh, great, I don’t actually know a lot about biostatistics, but you should go to the Career Center, because they have a great tool where you can see what your life would be like as a biostatistician. And so first learn about your career design center on campus.. Second, I think it’s important to start slow with any new thing that you’re doing in your course. It may be that you dip your toe in the water and just connect some of what you’re doing with career competencies. So when you have students work in teams, say, you know, teamwork’s important. This is actually what people care about when you get a job. And here’s how you might want to talk about your experience in teamwork, if you don’t currently do this outside of the classroom, and provide them some of those prompts. And then if you want to dive in, and really incorporate your NACE competencies, I think for most professions, in most disciplines, the alignment will be there. And it’s not a ton of effort to highlight that, especially if you start with just your assessments and maybe highlighting some of those and then moving towards other things by maybe adding some career readiness modules, or having your students go to your career design center, take a selfie, whatever they have to offer.

Rebecca: Sometimes it’s really exciting and nice to have some ideas of ways to connect with students that are just so straightforward. [LAUGHTER]

Jessica: I’ll also say, while our students have grown up in a digital age, I find that technology tends to be their weakest category when we think about the NACE competencies. And so if your field uses any technology, talking about how to be proficient in that, how to be a lifelong learner, and how sometimes it’s hard to learn something new, but you have to if you want to keep up into the field,

Rebecca: One of the things that I have been doing in my classes that students have appreciated is when there are things like free online conferences related to the discipline, assigning them as an assignment. And to do that, and to talk about what that experience is like and encourage them to connect with professionals in the field. And a lot more opportunities for that kind of an experience has been offered over the last [LAUGHTER] couple of years.

Jessica: That’s a great way to incorporate that&hellip talks, and even as you introduce a new topic, having them recognize that maybe after this class, you’re not going to be an expert, but here are ways that you can build this skill that you may need. And it might be going out and trying something new, it might be connecting with another center on campus. But recognizing there are resources there for you.

Rebecca: I think underscoring the idea that you need to continue learning in your field is something that students don’t always immediately recognize without us pointing out. They don’t recognize that one of the things they might want to ask in an interview are what the professional development opportunities are, or ways to grow as a professional in their first job.

Jessica: Most definitely, and how they want to do that and how they can identify those areas of growth. That’s something that’s not often transparent. We send them out into the world and say, “Great, you’ve got this degree.” But there may be many areas where they can become a little bit more proficient or dig a little bit deeper into a topic. I want to also highlight that I think that this is so important for our first generation and our URM students. In our undergraduate program, we have about 37% of our students being underrepresented minorities or first-generation college students. And if we don’t talk about this, no one else is going to talk about this. And by becoming someone to turn to about career readiness and about asking those questions of “What do I do for my first interview?” or “How do I prepare for this internship?” &hellipwe’re not going to be able to build that for those students because they might not have someone at home to turn to. As a first-generation college student myself, I found that no one was talking to me about this. And so I think it’s critical in higher education that we think about this as an equity and inclusion component within our curriculum.

Rebecca: That’s a really good point. I’m glad that you underscored that Jessica.

John: And one other thing that I think many people have been doing is bringing in guest speakers&hellip and you can bring in some recent graduates to talk a little bit about some of those pathways. I think we’ve all learned how easy it is to bring people in remotely to give presentations. We don’t have to physically bring them to campuses anymore.

Jessica: Yeah, I think that’s really powerful having career panels. And it’s great to connect with our graduates. I love when they say that what they’ve learned in the class actually helped prepare them for what’s next. We’re not just shooting from the hip here, we really talk to our graduates, understand where we need to focus and continue to improve our program as we continue to grow.

John: We always end with the question: “what’s next?”

Jessica: Ah&hellip summer is upon us. And so it is my time to write up all the papers that I’ve been sitting on over the semester. One of that is the paper on incorporating these competencies into our courses. And I look forward to sharing the results when we get that out there. But I’m really interested in how we work with faculty to help them think about adding career competencies to their courses. Because I don’t think this just has to happen at the undergraduate level. I think the graduate level is also key. I teach both grad and undergrad courses. And as I’m revitalizing our graduate capstone, I really think that this is perfectly aligned, along with our competencies for our accrediting body. And so really, it’s all come together for me, and I’m really excited to see what other folks think about it as the word begins to spread.

Rebecca: Sounds like a really exciting opportunity. And I think you’re right, graduate school is a great place for some of these conversations to be happening.

John: Well, thank you. These are things I think we should all keep in mind, because even if we’re not thinking about career competency as being important for our students, students certainly are. It’s always great talking to you.

Jessica: Thank you very much.

Rebecca: Thanks, Jessica.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]

242. Student Podcasts

Posted in assessment, authentic learning, collaboration, collaborative learning, higher education, hybrid, learning, pedagogy, student podcasts, teaching
June 1, 2022

Student research papers have been ubiquitous in higher education, but there are many ways in which students can demonstrate the skills that they have acquired. In this episode, Megan Remmel joins us to discuss the use of student podcasts as a more engaging alternative to traditional research papers. Megan is an Assistant Professor of Political Science at Bradley University.

Show Notes

Transcript

John: Student research papers have been ubiquitous in higher education, but there are many ways in which students can demonstrate the skills that they have acquired. In this episode, we discuss the use of student podcasts
as a more engaging alternative to traditional research papers.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners. [MUSIC]

Rebecca: I guess today is Megan Remmel, an Assistant Professor of Political Science at Bradley University. Welcome, Megan.

Megan: Hi, thanks for having me.

John: Our teas today are… Megan, are you drinking tea?

Megan: I am not drinking tea. I have rooibos this morning. But I am now currently drinking my coke zero sugar.

John: It’s not that much different than many teas.

Megan: Yes.

Rebecca: Many other rebels join us as well. [LAUGHTER] I have English afternoon today, John.

John: And I have ginger peach black tea, a return to an old favorite as we move towards the end of our semester here.

Rebecca: We’re both getting to things that are comforting.

Megan: Ginger is calming. So, [LAUGHTER] you might need that at the end of the semester.

John: Oh, very much so.

Rebecca: So we invited you here today to discuss a podcast project that you’ve been using in your state and local politics class. Can you tell us a little bit about your project.

Megan: So I was trying to think of a way to make students try to approach things in a more neutral way. Because obviously, in poli sci, there’s a lot of soapbox standing. And I’d previously been using just plain old policy analysis papers. And students don’t love them. And so I was trying to think of a way to get them to do the assignment that I wanted them to do, and having some guidance, because they’ve listened to podcasts before, so they kind of know what some of these are structured like. And so I was hoping that that would help tone down some of the opinionation that can come out of these things. And so I YouTubed, and I found John’s YouTube page, [LAUGHTER] and found his podcast project and contacted him just out of the blue asking him if he had any materials he was willing to share with me. And he did. And those came in very handy in terms of being able to guide students in the project. But it was just me trying to give them a different way to do something. Some of them still opted to do a paper this semester, but I wanted to give them the opportunity to try something different, to maybe be able to say to someone in an interview that they’ve done something in this more kind of digitalformat. So it was trying to open up the possibilities for them in class.

John: And was this a face-to-face class? Or was this an online class or a hybrid class?

Megan: So the first time I tried to do it was last spring, and that was a hybrid class and was admittedly a disaster. But that class was a disaster for numerous reasons, I think hybrid being the prime driver of that. So this class was entirely in person. I did allow them to work in groups if they wanted to, and some of them did, and some of them didn’t. And I had them do two rounds of podcasts. The others who wanted to, wrote a paper and the percentages were equal. And so a number of them who worked in groups the first time around did not work in groups the second time around.[LAUGHTER] So they got to choose their own topics. I gave them a list of I think 10 topics from that section of the course. And so there were restraints, but I let them propose if they wanted to do a topic that was of interest to them. Somehow I managed to have a Sports Communication major in the class, and when we talked about special purpose districts, I mentioned to him that there are stadium districts where cities are basically using taxpayer dollars to do massive overhauls of stadiums. And so that’s where he went. So it was still in political science. It was still state and local politics, but it was something of much greater interest to him personally than say, term limits and state legislatures.

Rebecca: Imagine that.

Megan: I know, shocking, right?

John: So, the students worked in groups, how large were the groups that they work in for these podcasts.

Megan: So, I proposed having them work in twos. This class is a 300-level political science class, but it has historically been required for the history secondary education students and criminal justice students. and poli sci students have gotten more interested in state and local politics, but I don’t think they think it’s as sexy as national level or international politics, so I think that they realize that’s where the jobs are, they’re getting more interested. So the audience is not kind of a typical political science class. Because of that, I have these history secondary education majors, who because of how tight their curriculum is, are in classes all the time and know each other really well and work together and collaborate pretty frequently. And so I did allow one group of three to work together. So I basically increased the requirements. So if they worked in a group of one, they had to have eight peer reviewed sources that they could point to in the script. If they were a group of two, they had to have 12 sources, and then this threesome had to have 16 sources. And then it went from a 10-minute requirement to 15 and a requirement to 20 minutes for that three-person group and the three-person group was actually probably the best podcast I got. And I obviously can’t attribute it to whether it was just the number of them or they’ve also been some of the best students in the class this semester, just generally, so I wasn’t surprised that they did a good job anyway.

John: So you mentioned a script. Did you have students submit a script before they recorded or was that done after the fact?

Megan: So kind of both. I had them pick a topic and then I had them submit either an outline or a script and kind of gave the pros and cons, where an outline is obviously a little more freewheeling and allows for a little more conversational style in the recording, whereas a script would be much more definitive, they wouldn’t be scrambling for words necessarily. So they’d probably have fewer filler words and they could be sure that they weren’t fading off and losing track of what they were saying. So I gave them the option of either, I think the students who wrote scripts just generally did better. So I don’t know if in the future when I do this again, if I’m going to get the option of an outline, or if I’m just going to make them write a script, because those seem to just perform better, but with the script I made them include work cited, and they had to tell me where in the script or where in the outline which source connected to that material. So I was trying to make sure that they were still using peer reviewed sources, they could obviously use stuff from outside of that. But I wanted to make sure they were still using peer reviewed sources, the way that my policy analysis paper kids were. But letting them do it in this less structured style, in comparison to like an eight to 10 page policy analysis paper.

Rebecca: How did students respond to having these options?

Megan: I was a little surprised at how few students wanted to do the paper. [LAUGHTER] I don’t know if they’re just burned out from… I assume the last two years has just been a lot of online writing assignments, for instance, and so they were just scrambling at anything that didn’t involve them having to write in such a structured way. So I have relatively few students select the paper option. So I’d say it was probably three quarters picked the podcast and a quarter picked the paper. And the ones that picked the paper… my pattern deducing… seem to be the seniors in the class. And I think they just wanted to get their paper done and be done and not necessarily have to coordinate with other people. And maybe they have prior experience with less successful group work, for instance, and they were just: “I’m going to trust myself.” But that was kind of the pattern, where the underclassmen were more likely to do the podcast and the ones that seem to be picking the paper were the seniors.

John: Was there any apprehension about recording a podcast? Because when I’ve tried doing this, I know students are often a little bit anxious about things like, “I don’t know if I have the technical skills or have theequipment to do that.”

Megan: Yeah, well, what was great was in the material you sent me, you sent me a lot of options that students could use. So for instance, regardless of what they submitted to me, in terms of the outline, I have them use, I think it was Otter’s transcription. And so they were using some of the sources that you sent to me. And so I think they felt more comfortable. And as they went, I think, obviously, from the first round to the second round, the quality of the recordings went up. And some of them realized that there’s ways on their smartphones to record and that it will partly transcribe for them. So I think they got better as they went, I didn’t try to ding them too much for production value in the rubric. So there is stuff in there just about like, “Please don’t have insane amounts of background noise [LAUGHTER] in your podcast. Maybe don’t record it in your car…” or something like that. So I tried to have a kind of minimum standard, but I wasn’t going to hold it against them if it was kind of fuzzy audio, for instance. But they actually didn’t seem all that apprehensive about the idea. They were better at it than I would have felt.

Rebecca: So the burning question is: “Did they move away from so much opinion and they’re more neutral? Or did they stay pretty opinionated? [LAUGHTER]

Megan: So actually, it went better than I thought it was going to, because spring 2021, when I tried this the first time around, I could not get them out of being on their soapboxes. And when I created the instructions for the policy analysis paper, I frame it as though you were working for a state legislator who knows nothing about the policy topic you’re writing about. And they want a policy brief from you and then recommendations at the end. So, the recommendations part is the “opinion” part. But it’s got to be based in all of the research that you’ve talked about earlier. So if it were about legislative term limits, political science agrees on very little, but this is one thing there’s kind of universal agreement on is that they are bad, and they backfire and do the exact opposite of what we want. So if that’s what the research is finally saying, then the idea is that you would recommend to the state legislator to vote against instituting term limits in the state. So I found that they were generally able to do that… it took the scripts, that initial round, to be like, “some of this language is getting a little feisty,” and “some of this, I’m not seeing any citations behind it, so, as far as I’m concerned, it’s reading like your personal opinion.” So I think that stuff was pretty necessary to get them to tone it down. I also had them submit draft recordings before the final recording. So I could ensure that the script was improved upon for the recording and so I could direct them if they were starting to go a little too far into the opinion editorial page of the newspaper. And so they were generally pretty good at it. If anything, I think they might have been overly cautious by the end of it, in that they had all this evidence about something leading to something and it was kind of repetitive so… confident that that’s actually what’s happening and still feeling like they have to do a both sides-ism. So I think I’m gonna have to try to work on that to instill in them that “No, you can take a position at the end, it’s just got to be based on the evidence you presented earlier, instead of just constantly pontificating,”

John: …and once you have your students do that, could you have them work with some journalists out there? [LAUGHTER]

Megan: Yeah, as somebody who gets interviewed pretty frequently with local media, I get a little frustrated with the both sides-ism. And yesterday, I got interviewed a lot about the Roe draft. And the reporters kept wanting to talk about the leak. And I was like, “No, the leak is not the important part, guys.” So yes, I understand some frustration there.

Rebecca: Can you talk a little bit about how students either shared or heard each other’s podcasts or whether or not the podcasts were shared more broadly.

Megan: So, inside Canvas, which is the learning management software, Bradley uses, for the ones who gave me permission to share, I posted the files inside of Canvas. And then, strangely, and I don’t know if it was because they were maybe afraid of the quality of it, some of them let me share one podcast, but not the other podcasts. And it wasn’t necessarily like, they wouldn’t let me share the first and they would the second, I just think they thought I like this one better, and so you can let people share this one. And I didn’t like this one, and you can’t share this one. So I just put them on the Canvas website. A couple of them told me, the ones who got maybe low Bs, for instance, on the first one, because everyone did pretty well… the ones who got maybe high Cs, low Bs, on the first one, they told me, they went and listened to some of the other podcasts just to kind of see what the universe looked like. And one of them went, “Yeah, I realized I need to step up my game.” [LAUGHTER] And so I think it was useful from that perspective, I don’t necessarily think they were listening to it to learn about the topic that their classmates have done. But I do think it made some of them realize the quality of their work could have been improved If this is the comparison point.

John: I think that’s a useful benefit of any type of peer review of other people’s work, that when they get to see what other people are doing, they might feel better about their own work. But more typically, they realize that there were things they could have done better. And that’s, I think, a useful experience for everyone, including faculty.

Megan: Well, I think students don’t even realize that for all intents and purposes, we have to use them as guinea pigs from semester to semester to make a class better and to improve. So I realized that maybe the pure hybrid format of spring 2021 was not a good time to maybe experiment with assignments. And so it made more sense to try something now. And it’s unfortunate that those kids maybe didn’t get the best experience, but they are our little guinea pigs, and we also need to learn from their work to see how we can make their work better by improving our assignments.

John: One of the issues I’ve had when I’ve done this, I’ve only used it in online classes so far, mostly because my face-to-face classes are relatively large and I couldn’t listen to two or three hundred of these. But one of the issues I had was that for many students in the online classes, during the depths of the pandemic, it was the only time they really got to talk to other students at the same time and I ended up with these incredibly long draft recordings, sometimes, like 30 or 40 minutes for a podcast that was supposed to be quite a bit shorter. And it did add to the amount of time it took to provide feedback. And included in the rubric was a great penalty if it was too short or too long. So I had to remind them of that. It was a tiny penalty, I think the length was only like five or 10% or so of the rubric score, but I felt bad docking them for that, because when I listened to it, it was clear that they were just enjoying getting to know each other and they were having these great conversations and getting to know their classmates. On the other hand, the focus could have been a little bit tighter. And that is one of the trade offs about having a script versus something which is a little more freeform. But it was really encouraging to hear the connections that students were forming. Although, after many hours of this, I would have appreciated them being a little more concise in some of that discussion.

Megan: Well, to your point. I’m curious, I’m not teaching the summer, but I am teaching an online Intro to American Government class this fall. And when I’ve taught it online before I just used forum postings. And it’s a lot of “I agree with this person,” even though you have directions that tell them not to do this, “I agree with what this person said.” And I’m kind of wondering, and thinking about tweaking this for the fall of kind of doing these voice responses, in hopes that it might limit some of that just repetitive nature and get maybe something a little bit more substantive. Plus, it’s more interesting for me than just reading the same post over and over and over again. And because it’s in an online environment, and it’s asynchronous, though I do have weekly benchmarks so they can access everything all at once, I think it would allow them to have a little bit more of the interaction than they get into the standard asynchronous typical shell. So it’s nice to hear that. I think I would also then have to say it was only supposed to be a 300 word post. So that’s only like maybe two paragraphs so we really don’t need to give a War and Peace sort of opinion. But maybe that would give them some of that more conversational style and make them feel like they’re at least possibly getting to know some classmates, ideally with the idea that maybe they can talk to each other and go over course material instead of being in their own little silos.

Rebecca: There’s something about hearing a voice, or seeing a face that can make all the difference. Of course, from your end, if you just make sure they have to post things in accessible format, you can either listen or read, whichever might be faster. [LAUGHTER]

Megan: What are you implying Rebecca? [LAUGHTER] It was more fun to grade these for sure, it was way more fun to grade these than a standard paper, without a doubt, because I could listen and giggle. [LAUGHTER] Because some of them would throw in a little snarky bit. And you don’t get to have that in these really structured formal papers. So, for me, grading wise, it was definitely more enjoyable.

John: For me as well, it was much more fun. And my impression was students had a lot more fun with it than they would have had doing a written assignment.

Megan: Yeah, I still feel like I’m going to continue to allow them to do a written paper. I know, if I had been in their shoes, I probably would have still pursued a written paper. And I had a couple students in there who were just quiet as church mice. And were probably never going to have the self confidence to even attempt a recording. So I think I still need to provide the option, which I did not do last spring. And I think that’s another reason why it might not have been successful is just given the… I mean, you guys know… class personalities vary wildly. And so that class was just very quiet and reserved and not super engaged. And so podcasts probably not the best approach in that class, in hindsight, but I didn’t know that before they got into the class, and I had built the syllabus.

Rebecca: Yeah, those surprises do happen.

Megan: They do, they do. And it’s a little difficult to overhaul your syllabus quite that radically in the middle of the semester.

John: There is something to be said, though, for pushing students a little out of their comfort zone. And in fact, this podcast, in part, got started because of a similar experience that I had, where I was teaching in the Duke Talent Identification Program…

Megan: I remember that.

John: …and they asked me to be on a podcast they had just started. And I said, “Well, I’m really busy, I don’t really have time for this, and I don’t think I’d really be the best person.” So I gave them a list of people’s names who they should contact to be on this. And they said, “Okay, we’ll contact them, too. But we’d like to interview you.” And after trying to get out of it for a while I agreed to do it, [LAUGHTER] and then realized it wasn’t all that bad. And then I came back from Duke that summer and Rebecca and I were talking and I said, “You know I did this podcast and maybe this is something we might want to consider.” And it’s one of the factors that led into this. I wouldn’t have probably had been doing the podcast had I not been pressured a little bit.

Megan: [LAUGHTER] I feel something similar. 18 year old me would not have done the podcast option. 35 year old me who’s done probably 80 media interviews over the last few years…much more competent doing it now.[LAUGHTER]

Rebecca: I think one of the things that I really responded to that you were saying, Megan, is that you were offering options. And that there were two that you did two podcasts or two papers and that if students chose a paper the first time but then heard podcasts, there’s a second thing. So they could do perhaps one of each, right?

Megan: Yeah, so there was more flexibility. I did not have any of them do that. But at least it was a possibility for them. I feel like I don’t know if they misread the syllabus, but it was once I picked a path that is my path, I am locked in for that path. But there was the possibility of it. So maybe some of them in the future will get maybe a little more courageous and go from a paper to a podcast.

Rebecca: Or maybe they go from a podcast to a paper

Megan: …to a paper.

Rebecca: whatever works for them. [LAUGHTER]

Megan: Also true, whatever is most appealing to their preferences.

Rebecca: I really like what you were both saying too, about personalities of students coming out. And that when they might be writing a more traditional paper, it’s just like entire personhood just disappears. And that having that kind of positionality a little bit come out and their personality come out helps us to get to know our students better and to help them get to know each other better when they’re reviewing each other’s work.

Megan: Yeah, there were some students in the class who in class itself were really quiet and then I would hear these little snarky asides in their podcasts and be like, where’s that in class, I want that in class, please give it to me in class.

John: And they would often make connections to their own lives. They were trying to connect their own experiences to what they were learning in class, at least in the podcasts they were doing for me, and those are exactly the type of connections we try to encourage students to make so that they recognize the salience of what they’re studying. I think that was really helpful.

Megan: Yeah, with my history, secondary education students. In the first half of the class, we talked about state-level interest groups. And so I gave them the option to talk about the NEA and the AFT. And most of them picked it because I think they all know that they’re probably future members of one, if not both of those organizations. And I don’t think any of them realized how different those two organizations are, I don’t think they realized not only currently how different they are, but the histories and the motivations behind them are entirely different. And so I think some of them may not join both of those organizations now, [LAUGHTER] when they become teachers, because I don’t think they like the motivations of one group necessarily versus another group. So I do like that maybe this is actually going to impact their workplace environments, and actually how they choose to behave. Same thing with the second half of the course, we talk about tax policy, which I love… shockingly, not of super interest to them… and obviously, property taxes are one of the major sources of education funding for K through 12. And so a lot of them picked that. And they knew maybe that it was bad, I don’t think they realized how bad it was. And at least in theory, some of them seem to have a little fire lit under them. How quickly that the real world maybe extinguishes that is a different story. But at least for now, I think there’s a lot of desire, at least within these particular students, to try to change school funding formulas, for instance. So I actually looked at the roster in advance of the class starting and looked at the majors of the students to try to find topics that were relevant to state and local politics as a political science class, but that students of those majors would actually gravitate toward.

Rebecca: To me that seems like one of the most meaningful choices that you made in your assignment design, because that really hooks a student and keeps them engaged.

Megan: Yeah, forcing them to talk about a topic they do not care about is hugely problematic. I teach our research methods class, and basically, if it’s a quantitative social science paper, it counts. I don’t care what topic it is. And they’re just mind boggled. So one of them, he’s a political science major, but he’s writing his paper on how video games affect stress levels in people. And so they just get to poke around in stuff that they don’t feel like they have permission to poke around in otherwise.

Rebecca: The other thing that I found interesting as a design faculty who does similar things, maybe not a podcast, but we do things that are out in the public, and we might share them, is that I often give models for students to look at that are professional, we might even analyze those together. But it’s not until they see each other’s that all the light bulbs go on. [LAUGHTER] It’s something about seeing a peer get it that all of a sudden helps bring the rest of the students along. And so they’re always clamoring for getting to see each other’s work. And it does improve the overall quality of the work, in my experience overall….

Megan: Yeah.

Rebecca: …despite the fact that they might have these professional models to look at.

Megan: Yeah, I did, because of, again, John’s instructions, I found state and local related podcasts, and linked to some of them. So they could see how they’re talking about policies, but not being super opinionated about them. And I’m looking forward to now that I have permission to share some of these, I teach this class every spring next spring, being able to give them these models of colleagues basically having done this work. So that, yeah, it doesn’t have the same production quality, and there’s no intro music and ad breaks, but they can see that their classmates have managed to do well on this. And they too, can do well on this. I mean, I always provide sample papers, I get permission from students and remove all their identifying information and post those so that students can see like “You can write a research design in my research methods class, it is possible. This was an A, this is what it takes to get an A.” So I’m glad that a few of them gave me permission to share their podcasts. And I think I’m going to share some of the better ones and some of the less better ones so that they can see for themselves, the spectrum of possibility. And if they’re cool with just putting in somewhat minimal effort, then that’s what this podcast sounds like. And if you want to put in the effort that’s gonna get you an A, that’s what this podcast sounded like.

John: One of the things that my students have commented on at the end of the class was that some of them have decided that they really enjoy podcasting, and they started their own or they plan to do one in the future…

Megan: Wow.

John: …and a few of them have also said, “I never listened to podcasts before, but now I’m listening to these podcasts.” So I was really impressed. But it did have these other side effects that I didn’t really anticipate it having.

Megan: I can’t say any of them have told me that but I’d love for at least for them to listen to some more podcasts because clearly, that’s all I listen to in the car. [LAUGHTER] So many podcasts.

Rebecca: I’ve had similar experiences. John, although I haven’t taught a podcast class. I’ve introduced students to podcasts as part of learning materials.

Megan: Yeah.

Rebecca: …and having assignments… and many of them say that they really enjoy that format more than others, but they may have never really experienced it previously.

Megan: Yeah, I can assign them a 10-page article or I can assign them a 30-minute podcast. They definitely like the 30-minute podcast better. They seem to actually listen to it in a way that they don’t with the reading. So yeah, I have been more and more frequently been trying to find either like 5-minute local NPR stories or outright organized podcasts for them to listen to,

John: I’ve been doing more of the same. And I try to find podcasts that have both the audio and a transcript, so that people can choose a modality depending on where they’re working and reading. In some cases, it may be hard to find the time to listen to audio, or they may be constrained in some way and they prefer reading the text. And in other cases, students would much prefer listening to a podcast while they’re walking or exercising, or doing something else. So they have appreciated the choice when it’s used as a basis for discussions or some other assignment.

Megan: I hadn’t even thought of that. But I’m gonna have to think about that for the syllabi for the fall.

Rebecca: The transcripts are really helpful too, because if you are listening, and then you hear the name of something, or you’re not really quite sure how to spell it, or whatever, the transcripts can be really helpful… if those transcripts have been edited, of course.

John: One thing I’ve always been amazed by are the number of people who say they only read the transcript, they never listen to the podcast. Because what I enjoy about it is the narrative and the ability to focus on a conversation while I’m driving or walking or something similar. And reading the transcript would be very much like all the other reading I do, and it just wouldn’t seem as interesting.

Megan: I agree, but to each his own.

Rebecca: There’s a few podcasts that I listen to regularly that I might go back and revisit in transcripts to pull out some notes of things that I wanted to remember. And so I really get very frustrated when podcasts don’t have transcripts for me to do that.

Megan: [LAUGHTER] Fair.

Rebecca: To support my needs.

Megan: This is about me. Thank you very much.

John: It’s important for accessibility purposes as well.

Rebecca: Yeah, definitely. So Megan, you shared a couple of things that you might want to do differently in the future related to this assignment, do you have any other thoughts about how you might frame or structure the assignment a bit differently to continue producing excellent podcasts and your classes?

Megan: I’m curious about the idea of… It’s a 300-level class, and I feel like providing them the topics was a little hand holdy. And so I’m trying to figure out if I should let them pick the topics, because then they have to work a little harder to figure out what might constitute state and local politics. Because I think you can see that in pretty broad ways. But I worry that if I do that, then they’re just going to take some really black and white literal approaches. So I’m not entirely sure what I’m going to do on the topics front of things. And it might just, again, depend on who’s in the class. I think if it were more of a straight up political science class, I’d let them maybe wander a bit more. But since that’s not the audience of this class, typically, I might try to give them a little more structure, since they’re probably of all people gonna go, “I don’t know what you want for me in a class that’s not directly my major.” So that’s thing number one. I am also trying to figure out for the groups, for when they recorded together, given that some of the groups fell apart in the second round, I’ve never had great success with peer review, because they don’t seem to want to be telling the truth about each other. And so I’m trying to figure out how to get around that because it was pretty obvious with one group in particular that someone was doing all of the heavy lifting, but that person wouldn’t fess up to it. And so I’m trying to figure out if I should allow them to work in groups, and if so, how to try to ensure that the workload is being done somewhat more equitably. Because when I’ve had them work on group projects that have involved writing, for instance, I can follow the Google Doc, because I’ve required them to give me access to something like that and I can see who’s adding what. I’m less able to do that in this scenario. So I’m trying to figure out how to… maybe in the script writing process, still have them share it as a Google doc so I can follow the trajectory of who’s adding what, when, in an effort to try to get a better handle on making sure that one person is not being carried through the assignment. And that’s just, I think, a general struggle with trying to grade group work assignments. But that’s the only thing that I can think of right now and I’d love advice. But the only thing that I can think of right now to try to address that a little bit. And I knew it was gonna be a problem, I just didn’t know quite what to do with it. [LAUGHTER]

John: While you can follow the editing history, that can be a bit of a tedious path through that. What I’ve generally asked students to do is just to use a color code where they pick a dark color, so there’s still good visual contrast to meet accessibility issues. But they each have their own color that they write their text in. So when they write a section of a document, just have them block it and choose their color. And then when you read through it, all the dark blue will be from one person, the dark green will be from another person, and the purple will be from a third and it makes it a whole lot easier to evaluate the individual contributions. And that’s worked really well.

Megan: That is beautifully simple. I appreciate that. Thank you.

Rebecca: I do a couple of things, also, because I do a lot of group work, and group work that’s not always visible. I often have shared documents where it makes sense and ways of documenting it. But I’ve also done things like having students keep timesheets, just recording what they’re doing and when and how long it takes with the frame that it might be helpful for them to better understand where they use their time. And so that sometimes is helpful. We also tend to do things in design more like a process document. So it documents the process and milestones and summaries of what they’ve contributed at various points. And so I find those kinds of documents really helpful to understand what people are doing. And I always request them to provide a little bit of information about why they made certain decisions. And as soon as you do that, then you know who did it, it becomes very clear. And the other thing that I do is a Google form as review of the other collaborators. But I do things like ask questions about how willing they were to accept feedback, what was their greatest contribution? So it’s framed in a little bit different way than maybe a traditional rating system? Like, did they do all the things?

Megan: Yeah, it sounds more, what did you do versus what did your partners do?

Rebecca: Yeah. It’s more aligned with how you might do evaluations in a workplace or something rather than maybe traditional peer-to-peer evaluation.

Megan: Those are awesome. Thank you.

Rebecca: Megan, was there anything else that you wanted to make sure we talked about?

Megan: No, I just want to thank John for being so willing to share his materials with me because I would have been floundering about how to start.

John: I was really happy that someone requested it.

Megan: It was great. Thank you so much.

Rebecca: Well, we always wrap up by asking: “What’s next?”

Megan: I think I’ve got a lot of material to work with with state and local politics. Given the recent political landscape, maybe too much material to work with and state local politics. I think one of the things in the future and I’m nowhere near this yet, I’m interested in letting them explore alternate methods of this podcasting style. So maybe actually interviewing local candidates, working with political parties, the League of Women Voters is actually very, very active in Peoria, and they still subscribe to being non-partisan, so it would be nice to try to team up with them and see if they’ve got some sort of outreach campaign they’d like to do. So I’m thinking of trying to really expand, but next year is busy. So that might be the year after.

Rebecca: Well, thank you so much for joining us.

Megan: Thank you for having me.

John: Thank you very much. And I’m really thrilled that someone actually found that material useful. [LAUGHTER]

Megan: It was, it was great. [LAUGHTER]

John: It’s an activity I’ve been doing with my class for three years now, and it’s been working really well, and I’ve really enjoyed it. And many of the students really have to.

Megan: Yeah, time two was the charm for me.

Rebecca: Yeah, you have to have a practice round.

Megan: Yeah, unfortunately, those students were guinea pigs, but I learned from them.

Megan: Thank you for having me. I appreciate it.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle.

[MUSIC]

236. ePortfolios

Posted in assessment, authentic learning, feedback, grading, higher education, learning, pedagogy, teaching, writing
April 20, 2022

As David Wiley has noted, “disposable assignments” often have small impacts on student learning. In this episode Nikki Wilson Clasby joins us to  discuss how one campus has used ePortfolios to create authentic learning experiences in their English composition courses.

Nikki is the coordinator of the English Composition Program at SUNY New Paltz.

Show Notes

Transcript

Rebecca: As David Wiley has noted, “disposable assignments” often have small impacts on student learning. In this episode we discuss how one campus has used ePortfolios to create authentic learning experiences in their English composition courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer…

Rebecca: …and features guests doing important research and advocacy work to make higher education more inclusive and supportive of all learners.

[MUSIC]

Rebecca: Our guest today is Nikki Wilson Clasby. Nikki is the coordinator of the English Composition Program at SUNY New Paltz. Welcome, Nikki.

Nikki: Thank you. It’s good to be here.

John: Thanks for joining us. Our teas today are… Nikki, are you drinking tea?

Nikki: I am certainly drinking tea, yes.

Rebecca: Woohoo!

Nikki: Would you like to know what it is?

Rebecca: Yes!

John: Yes!

Nikki: [LAUGHTER] So this is an exotic blend called Tetley, a very strong British brew, which we Brits love, unless you’re a PG Tips fan, but Tetley’s pretty up there. And I have it with 2% milk which is the best way to drink it.

John: Most of our colleagues on campus from England tend to drink Yorkshire Gold.

Nikki: Mmhmm.

John: They seem to prefer that to the other options.

Nikki: Yes, well I am a Yorkshire lass, but I have to say Tetley has that kick that I need. [LAUGHTER]

Rebecca: Good to know. I think today I have Scottish afternoon tea.

Nikki: Ooh!

John: And continuing with the theme I have an Irish breakfast tea from Twinings.

Rebecca: Oh!

Nikki: Oh very nice, that’s a good one too. I like that one.

Rebecca: This crew needs some strong stuff today.

Nikki: We need some scones now. [LAUGHTER] Then it will be complete.

John: We’re recording this at 12:30 today, and I’ve already had five meetings today including a class.

Rebecca: This is my second pot.

Nikki: I’m impressed. [LAUGHTER] You can come and have tea with me any day.

Rebecca: [LAUGHTER] Perfect. So we’ve invited you here today to discuss the use of ePortfolios in the Composition Program at SUNY New Paltz. But first can you tell us a little bit about your role at New Paltz?

Nikki: Yes, I am the Coordinator of the Composition Program. I stepped into this role two years ago, and I am also a lecturer. And so I teach mainly our upper-level writing and rhetoric courses where I specialize in visual rhetoric. And I also teach courses in what we call Practical Writing and Design which is a new course dealing with a sort of blend of graphics and writing. And I also teach a FIG, a First-year Interest Group, for the Communication Disorders. And I run practicum for our TAs.

Rebecca: So you’re not busy or anything?

Nikki: I’m not busy at all, no. Plenty of time for drinking tea. [LAUGHTER]

John: And plenty of reason to drink that tea with the caffeine.

Nikki: Which is why I drink Tetley, yes. [LAUGHTER]

John: So, we invited you here, though, because we heard about the common use of a WordPress site for the creation of student ePortfolios. And I think the first question we have to ask is… How did you possibly get agreement within a department on the use of one platform?

Nikki: [LAUGHTER] Actually it’s pretty simple, there’s no drama involved here. So in 2019 the composition committee reworked our three-credit English 180 Composition II course to a four-credit GE course, which we retitled English 170 Writing & Rhetoric. And we had been using print portfolios for a long time, and so during the process of revamping our course the composition committee reviewed how we could improve our portfolio assessment. And Matt Newcomb, who was the coordinator at the time, and I had been long advocating for ePortfolios. So during our meetings we decided that it would be a good time, seeing as we were revamping this course, to introduce ePortfolios into our curriculum. And we’d looked at options in Blackboard, but they were just too…

John: Awful? [LAUGHTER]

Nikki: Yes, too awful, but just by happenstance and pure serendipitous coincidence the university at this time decided to just opt for a CampusPress system, and they adopted Hawksites. And so it was made readily available for us to use. And so the timing was perfect, we just jumped on it right away and said, “Yes, this is what we want. This is the way we’re going to go.” So because Matt and I had been advocating for ePortfolios for so long it was pretty simple to get our program on board with the project.

John: And we should note that the Hawks are the campus mascot for SUNY New Paltz. And that Hawksites is just a campus-wide instance of WordPress, I believe.

Nikki: It is, yes. That’s exactly what it is. We just gave it the name Hawksites. Yes, it’s a campus-based university blogging website and ePortfolio tool.

Rebecca: Are students developing these in this more beginning course, and then working on the same portfolio throughout their entire curriculum?

Nikki: Well, this is what’s really interesting. So the faculty are allowed to use the ePortfolios as a tool for however they want to integrate it into their program, so they have free rein to do whatever they want with it. And we all use it in different ways and to different degrees, depending on our comfort level with technology, and how it fits into our curriculum. But as a composition program we use the ePortfolios for assessment purposes, so I can tell you a little bit about how that is organized. So whereas we can have free rein to use them however we want, we do have some very specifics that we need for our assessments. Would you like me to tell you about those?

Rebecca: Love to hear about those.

Nikki: [LAUGHTER] Okay, so for English 160 which is the basic, very first writing course, students have to go through this ePortfolio assessment at the end of the semester to determine whether or not they are fit to move on to English 170. So this is the tool that we use to make sure those students are ready for the more vigorous program. So for that 160 assessment process the students have to upload to the ePortfolio, or at least be able to visibly show on the ePortfolio, they have to have two of their strongest assignments, and they also have to have their revisions for those assignments. They can choose whichever ones they want to put on, but they have to be two major assignments. The only requirement is that they have to show that they have been able to write in different modes, different genres for different rhetorical situations. And there needs to be an element of research in their citation, you know the beginning stages of that research process. And obviously we’ll be looking for the standard of their writing as well, that’s why the revision aspect is really important. So that’s what they’re required to do for the ePortfolio. And then the 170 students, they have different requirements. But let me just backtrack just for one second. So across the board for 160 and 170 as part of the ePortfolio requirements, all the students have to create a reflective cover letter that goes up front in their ePortfolio. They write that reflective cover letter at the end of the semester and we give them questions, guidelines as to what to tackle. And what we want from them is a sort of critical overview of their progress during the course. And they have to cite examples of their writing to prove their case. So it’s a persuasive letter, and our assessors read that first, so they’ll read that reflective cover letter. And that gives us a very clear sense of what the student understands about their writing process, and that makes us feel a little bit better about whether they’re ready for 170 or not. It shows that they’re applying the techniques and skills that they’ve learned throughout the semester to that cover letter. So the 170 students, we have a very specific framework for our 170 Writing & Rhetoric program, it’s based upon a wicked question. So a wicked question might be… What should we eat? Or how do we save the world? Or what does it mean to be human in a digital landscape? And all professors can choose whichever kind of wicked question they want, that they’re excited about, and then they base all of their assignments around that wicked question. That gives us a lot of flexibility for Writing & Rhetoric, which is wonderful. So the semester is divided into two sections. We basically require two major assignments that are argument research based, and then each of those two large assignments has two smaller assignments that help students gear into those big assignments. So, for instance, you might have a proposal with an annotated bibliography that leads to a research paper. So students have to choose one of those sections. So in the ePortfolio we want to see two smaller assignments leading to a large assignment. We don’t need revisions at the stage for 170, we acknowledge that revision is part of the process, and that they will be revising anyway for those papers. So that’s the structure of the assignment sequences for those two ePortfolios. And then beyond that we add other things into the ePortfolios as we see fit. So, for instance, this semester we have our internal assessment which is for our 170 students, and that’s on basic critical reading. So that’s kind of how our ePortfolios are set up. And then at the end of the semester we have a system set up where we review each other’s ePortfolios based on a common rubric that we have put together. So that’s basically how it works.

John: It sounds like a great approach in ensuring standardization across their classes and making sure that all students meet the requirements to move on.

Nikki: It is, it’s very effective. We have lots of conversations afterwards about who’s on the cusp, borderline cases, and so it’s very democratically pieced together. And then of course we have to work on individual cases of students who are failing for various reasons. And it’s a pretty good system, and it’s been very effective over the last two years that we’ve been using it.

John: This is more of a technical question about the organization… Is each site organized by class or is it by students? In other words, does a student have their own WordPress account that they use and create an ePortfolio that is unique to them across all their classes? Or is there a class site where all the students in the class post their work? Or is it some combination of the two?

Nikki: So what happens for us is that in our composition program, we have a template on Hawksites, and our students create an account through Hawksites, and they are given the template that they have to use. So they are essentially creating their own account on Hawksites for our classes, and it’s unique to them. It’s not something that we share with other classes, this is specifically for our class. Does that make sense?

John: Mmhmm.

Rebecca: So just to clarify, if a student was in another class, in another subject area, they may have an additional site.

Nikki: Correct, students can create as many accounts as they want on Hawksites for individual programs. I have about 10. [LAUGHTER] It’s fantastic. It’s such a good resource, we love it.

John: I’m hosting a variety of WordPress sites as well for different purposes.

Rebecca: Me too. One thing I think that’s always important to ask when we’re talking about ePortfolios, is whether or not that student work is public to everyone, public just to members of the community, your classroom community? Or are they private? Or do students have a choice of the privacy settings?

Nikki: The students have a choice of privacy settings. But what we encourage students to do is to select the option that allows only people within our university that have a university ID and login to be able to access it, and only the people that the students give the link to, or the people that the faculty give the link to. And this allows us to share those ePortfolios amongst the people who are going to be assessing. So there is some choice for the students, but it also gives us the option to share easily amongst our colleagues. But I want to add something into this too, that within our template that we had created at Hawksites, we have a permissions policy embedded into the site. And that is a basic form and students can sign it, they can say yes or no. We ask the students, “Would you mind If we shared your portfolio for teaching purposes? Would you mind if we shared some of your work for research or for teaching methods?” And students can pick “yes” or “no” for all of those. And that’s nice to have that there on the ePortfolio, so whenever we’re looking for examples we can check the permissions pledge and see who’s agreed and who’s not agreed, and then of course we respect the students that haven’t signed it. So there are some levels of privacy within our cohort of teaching. There is a blog function on our Hawksite. It’s up to the professor whether they use the blog or not. But as they stand, the students can’t see each other’s ePortfolios, those are private. But there is a blog function within Hawksites, and faculty can choose whether they decide to share that blog function with other students or not. I have used that function for a different project, but I haven’t seen anybody take advantage of that because we also have Blackboard which has its own blog function too.

Rebecca: How have students responded to the idea of using ePortfolios?

Nikki: That’s a really good question. It very much depends on the instructor and how the instructor teaches the ePortfolio component. I can tell you that for TAs who are new at trying to grapple with this technology and pedagogy, some of them have in the past waited till the very end of the semester to have the students upload their work. It’s too stressful for students, they can’t handle it. It’s a lot of work to put a good portfolio together. So I make the TAs have the students sign up for an account within the first two weeks of the semester. And I encourage the TAs to find ways to get the students to engage with their ePortfolio on a low-stakes non-graded level just so that they can learn how to use all of the functions and the tools. And also get them in the habit of using their ePortfolio as a working kind of document, and not something that just gets shoved to the end of the semester. So it really depends on how it’s taught. But if you do teach it with those kinds of sensitivities in mind, and you don’t stress the students out, I find my students in particular love using their ePortfolios. They enjoy engaging with them, they enjoy seeing their work look professional on the site, they enjoy the option of using a more web-based writing process for embedding videos, hyperlinks, uploading images, embedding their beautifully designed Google slide presentations into their site. So they do enjoy that process. I give them time in class to do it so it’s very therapeutic for them. But they also appreciate learning some of the real-life skills that comes with curating an ePortfolio, and they recognize that this will help them later. So the enthusiasm for it is pretty high, and most students feel very proud of their ePortfolios by the end of the semester because they have something to show for all of their hard work, and it looks good. So they’ve adopted it really well. My worry about students is they do all this work, and they hit the submit button for grading, and then that paper disappears down the black hole never to resurface, and then they just move on. And it’s a shame because that work is good work, and we want our students to feel like they have a stake in the writing process, they have a stake in scholarship and research, and the ePortfolios provides a really nice platform for allowing them to think of themselves in that respect, and not just the humble student that struggles, if you like. [LAUGHTER] Helps them feel a little bit more professional.

John: David Wiley refers to those types of assignments that students post in Blackboard, or submit their paper at the end of the term and never see again, as disposable assignments. And having something that looks professional that they have access to, and that they can share and feel good about, is something that students really value. I’ve had students write some books in my class, and they really enjoy seeing this final product. It’s something that they can share with their friends, with their parents, with potential employers, and link to on their resumes and so forth or on LinkedIn, and they’ve appreciated that tremendously. I think you do some of the same, right, Rebecca?

Rebecca: Definitely the students love it when it’s like… It’s a real thing, a real shareable thing, with real audiences. [LAUGHTER]

Nikki: Exactly, and that’s the key thing, right? Especially in rhetoric having that real audience, it’s super, super important.

John: And in my experience it leads to a much higher quality of work when they have that non-disposable assignment. Have you seen the same?

Nikki: Yes, I totally agree. There’s a level of accountability that goes on there. So when you’re racing off an essay at the last minute and submitting it, it disappears into the black hole. With the ePortfolio it comes back to hit you in the face, and you can’t put that stuff on the web, you have to go back and revise it. And it’s really nice being in the classroom and having the students respond to your comments and make those revisions. And then you kind of hear the penny dropping, it’s like, “Oh boy, I really didn’t do this very well, I better snatch this up for the ePortfolio.” And it is very reassuring to see that in action. So yeah, it’s lovely.

John: So how have other composition faculty responded? Are they all comfortable with it? Was there any resistance?

Nikki: That’s a really good question. I know, for me, I’ve been involved in this sort of work for a long time, I came from Iowa State University here and we’ve been working with ePortfolios for a long time. And that switch, going from the paper portfolios which I hated [LAUGHTER] sorry, I hated them… Going from the paper portfolios to the ePortfolio, that’s a big mind switch to go through. So we had to work with our faculty, encourage them to set up a Hawksite of their own so they could experiment, help them feel comfortable sharing those Hawksites in the classroom so they could use that as a teaching tool. So initially there was some learning to do, and that’s great, I mean that’s great, that’s fine, perfect. So it took a while to make that switch to ePortfolios, but now that we’ve made that switch, I think we all recognize that it’s so much more accessible, it’s so much easier to organize, it’s so much easier to assess. We’ve only been doing it since the fall of 2019, but I don’t hear any complaints [LAUGHTER] about the ePortfolio. It is part of what we do now. So it’s good.

John: And that was perfectly timed to be ready for the pandemic.

Nikki: You know, it was perfectly timed for that. And what I like about it, and I think what we all agree we like about it, I encourage the faculty to have the students post the links to the faculty right from the very beginning. So that way we can just go in periodically, and we can just monitor what’s happening on there, and then we can give direct feedback to students about it. So it is, it’s a wonderful tool.

Rebecca: One of the things that you’ve mentioned is this template that you share out, and you mentioned some of the permissions that you allow students to choose. Can you talk about some of the other features of the template itself that you share with students? Like, what are some of the things built into it?

Nikki: Yes, so the template has the tabs already constructed so that students don’t have to work out how to recreate those. Obviously we teach them how to generate new tabs, but the basics are already there. So it has a homepage tab, so we encourage students to post a photograph of themselves and think about how they want to present themselves to a general audience as a student. So they have that, we work on that side of things. And then we just have the tab for the reflective cover page. Then we have the tabs for the individual assignments and their revisions,and then we have the permissions tab. And then we also include on the ePortfolio, this is a new feature, during the pandemic we had a lot of issues with attendance and accommodating students who were sick and who were in quarantine, so what we did was we posted the essentials of the course policies on the ePortfolio. And we had students acknowledge and sign that they had read them, and that they understood what those different policies were for attendance, for assignments, for what they needed to do if they were sick, all of those things we put on their ePortfolio site. So it became a quick reference guide for students that they can just pull it up, and they could see what was required of them. But also for us as faculty when students were suffering, or not keeping on track, or getting to the end of the semester and things were not looking good. We could pull that up, we could see who had signed the pledge and we could say, “Look, policy said that you needed to do this, this, and this, and you didn’t do those things.” And so that helps stem the flow of the great appeals at the end of the semester which I have to deal with. So that worked well for keeping students on track, and keeping that information transparent and clear.

John: And you have it in writing, digital writing.

Nikki: We have it in writing, and the students sign it. So it helps them take accountability for their part in this process. They can’t say, “Oh, I didn’t know about that,” when it’s on the ePortfolio and they’ve signed it. It’s like, “Mm, well apparently at some point you did read this.” So that helps. Not all students read it [LAUGHTER] of course, but at least it’s there though, that’s the important thing. Those documents are not buried somewhere else, they’re visible, they’re right up there. And I think that’s really, really important, and I’m really glad that we decided to do that, especially over the pandemic. It’s been helpful.

John: And I know I always read all the terms and conditions when I sign up for a new software package, and so forth.

Nikki: Of course we do. [LAUGHTER] The other thing that I want to add in there is that, for me, some of the professors do this too, but I have my students create a writing journal tab in their ePortfolio, and they have weekly writing journal prompts in there. And I do that so that students have a safe space just to write, and to reflect on what we’re doing in class, and to apply those ideas to material that they’re interested in. And I set that up because I wanted them to feel like they owned their ePortfolio, that it was their ePortfolio, that it was their personal sort of diary, if you like, of all of their work. So that tab is very important for my classes, and my students enjoy doing that kind of work.

Rebecca: You also mentioned earlier that the work looks professional, so I’m assuming then there’s some stylistic things that are built into the portfolio as well. There’s at least a base look for things, no?

Nikki: There is a very basic look, and I would love to be able to include more design tools in the ePortfolio because we don’t have a choice of font style, we can move our images around [LAUGHTER] to a couple of places. It’s very, very rudimentary, and it would be really lovely if we could add a few more tools in there to make it look even better.

John: So everyone in composition has agreed to use templates, but it sounds like they might use them all differently. Is that correct?

Nikki: Yes. Thankfully, even though the design elements are pretty rudimentary, there are some tools to change the actual overall look within the basic template. The students can change the background image, they can change colors, they can personalize it in a way that suits them which is really nice. So yes, that’s fun, and those are good skills to teach the students as well.

John: What about different sections of the course? Is there a standardization in terms of how the platform is used? Or does that vary from instructor to instructor to some extent?

Nikki: It varies from instructor to instructor depending on their comfortability with technology, and how they want to integrate the ePortfolio into their program. I’m not a standardization sort of person, but we do have… the basic elements for assessment are standardized, they have to have those specific elements for assessment. But apart from that they are free to use those ePortfolios as they wish, and that’s the way that I want it to be.

Rebecca: One of the things that might be helpful for listeners too, earlier you were talking about your assessment process, and that people from other sections review work, that you’re reviewing work of the students of other instructors. So I’m wondering if you’d talk a little bit about the logistics of how that actually works. Because I think for some folks it can be such a big undertaking, so hearing stories of how other people organize those sorts of things can be helpful.

Nikki: Yes, so first of all students have to be eligible for an ePortfolio review, that’s the first step. So students have to have completed all of the assignments and all of their requirements, like the library instruction, oral presentation, all of those things. The student has to have at least a D to be able to be eligible. So that sort of weeds out some of the stuff. And then what the faculty do is we take seven portfolios per class, and that’s a random selection, so you take the first student on your roster, and then every fourth student gets to go in that pile. So each faculty member has seven students randomly selected for ePortfolio assessment. Plus, we have then any student who is borderline, any student that is just clinging on there, or any student that a faculty member is really unsure about, so that goes into the pot too. And then my assistant and I, we create ePortfolio partners and we specifically place, for instance, seasoned faculty members with new TAs. And that’s the way that we do it, so we choose who assesses whose work. And that makes it a very organized system and a fair system, especially for the new TAs who are not sure about what to do, at least they’re working with someone who has experience. So that’s how we do it, and the assessments can take place whenever is convenient for that particular pair, as long as all of the results are all tabulated and submitted by a specific time period. And then after that time period we’ve got some space here to work on ePortfolios that have issues. So once that rudimentary assessment is done then anything anyone is concerned about can be given to my assistant and I, and we’ll go through case by case any of those borderline cases that we’re worried about, we can work on those. So that’s kind of how it works, and it’s a really good system. It works really, really well, it’s very efficient, it’s fair for everybody. At the end of the semester, you know we’re tired, the faculty have already been through all of the ePortfolios and given their verdict, and then we double check with those seven to make sure, it’s really a calibration thing to make sure that everybody’s on the same track. And I need to preface this by saying that all faculty members have to go through a standardized calibration training at two points during the semester, so we make sure that everybody knows how to use the rubric and can apply it effectively. So with those checks and balances it actually works out incredibly well.

John: Are there standard documents that you share with people, and then you see how they evaluated to compare against the benchmarks, for the calibration?

Nikki: Yes, we do. We have a standardized rubric, and then during our retreat sessions we will selectively pick, like, a very, very borderline portfolio for people to assess. We put people in groups, and then we make them grade the ePortfolio with the rubric, and then we discuss it and we talk about what’s working, what’s not working. And if there’s any huge discrepancies in the assessment of those ePortfolios we talk about what was going wrong with those discrepancies. So it’s pretty organized, and it’s pretty efficient.

John: It seems like a really nice way to provide equitable and fair assessments that adhere to the standards that you’re trying to meet. I’m impressed.

Nikki: Thank you.

Rebecca: You talked a little bit about students needing to meet standards to go through the portfolio assessments. Does that essentially equate to their ability to continue on in that particular program?

Nikki: Yes it does. If they’re not meeting the basic requirements for an ePortfolio review, technically it means they’ve failed. And so what we do with those students is we then decide… How did that student fail? Did they fail on their own merit? Or did they fail because they tried and tried and tried but just couldn’t get it? So we have standardized measures here that says, “Okay, so if a student has been trying really hard, and they just didn’t get it, then we will allow that student to repeat the course.” So we have checks and balances there for those students.

John: So this program is used universally in the Composition Program. Have similar practices been adopted by other departments at New Paltz?

Nikki: I’m ashamed to say I don’t know, and the reason for that is because I don’t get out much to see. [LAUGHTER]

John: That’s not uncommon especially during the pandemic.

Nikki: Yeah, I just don’t know, and I feel embarrassed to say that, but I came out of teaching a 4/4 load into this position. So that’ll be one of my next step projects is to figure out who else is using them on campus and talk to them about how they’re using those ePortfolios.

Rebecca: So another thing that is worth considering is… You mentioned that students can choose some privacy settings and things. How long do students have access to these portfolios after they’ve created them?

Nikki: As far as I know students have it for as long as they’re a student.

Rebecca: So we’ve talked about assessment being a primary motivator and maybe some professional skills as being a good motivator for putting ePortfolios in place. But are there other advantages to using student ePortfolios that we should be thinking about?

Nikki: So apart from the ePortfolios for the students being an opportunity to see themselves as professional communicators, to help boost their ethos and their confidence. I think we talked a lot about what the students get from this, but from a faculty’s perspective the ePortfolios are a fantastic tool because they are so accessible, they’re easy to coordinate for assessment, we don’t have to wade through buckets and buckets of paper. And also we don’t have to, [LAUGHTER] I know this sounds like a really minor thing, but when we used to do the paper portfolios we’d do the portfolio assessment, and then we would call the students into our offices to break the news to them whether they’ve passed or not, and give them their paper portfolio back, and a lot of students didn’t come. So we ended up with piles, and piles and piles of portfolios in our offices and it’s like, “Well what do we do with those?” I found it really distressing. You know, if you’ve got four classes and 120 students, and every semester, and then they just pile up, that was distressing. So to switch to the e-system just feels better on my soul, [LAUGHTER] for the planet doing this. But the ePortfolios, they’re just such a good tool for faculty for teaching, for training other faculty, and for sharing what we do with our students with each other, and sharing ideas and seeing what the possibilities are. The ePortfolios just offer so much more potential for pushing what writing and rhetoric is, and what we do with it in the classroom. So from a pedagogical point of view, I can’t imagine going back to paper portfolios. It’s just a fantastically amazing, creative, and soul-satisfying tool to have at your disposal.

John: That’s a really nice, positive note to end on.

Rebecca: So we always wrap up by asking, What’s next?

Nikki: Well, so last summer my colleague Rachel Rigolino and I used Hawksites to develop an online tutor training site because we need more TA tutors in our system to help with the writing program, and that was really successful. And so what we want to do now is to extend that. We would like to develop a Hawksite for our TAs so that we can put all of their innovative teaching ideas into a Hawksite, so that it’s accessible to everybody for sharing ideas. And that’s a really big project. So that’s our next big step, to do that.

Rebecca: Sounds like it’ll be really helpful, and really exciting to work on.

Nikki: I think so. I think it will be vital. [LAUGHTER]

John: This sounds like a really good program, and thank you for joining us, and thank you for sharing this with us.

Nikki: Thank you, it was a pleasure. Thank you so much.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

John: Editing assistance provided by Anna Croyle, Annalyn Smith, and Joshua Vega.

[MUSIC]

190. Academic Integrity

Posted in assessment, cheating, higher education, intellectual integrity, learning, online learning, pedagogy, teaching
June 2, 2021

The global pandemic resulted in a dramatic increase in online instruction. This was accompanied by an expansion of the use of online services that, in return for a fee, provide students with solutions to assignments and exams . In this episode, James M. Pitarresi joins us to discuss strategies that faculty can use to preserve academic integrity in their online courses.

James is a Vice Provost for Online and Innovative Education and the Executive Director of the Center for Learning and Teaching at the State University of New York at Binghamton. He is also a Distinguished Teaching Professor in the Department of Mechanical Engineering at Binghamton.

Shownotes

Transcript

John: The global pandemic resulted in a dramatic increase in online instruction. This was accompanied by an expansion of the use
of online services that, in return for a fee, provide students with solutions to assignments and exams. In this episode, we examine strategies that faculty can use to preserve academic integrity in their online courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist…

John: …and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is James M. Pitarresi. James is a Vice Provost for Online and Innovative Education and the Executive Director of the Center for Learning and Teaching at the State University of New York at Binghamton. He is also a Distinguished Teaching Professor in the Department of Mechanical Engineering at Binghamton. Welcome, James.

James: It’s a pleasure to be here. Thanks for bringing me on board.

Rebecca: Today’s teas are… James, are you drinking tea?

James: I had my cup of tea. I drink Barry’s Irish tea with a little bit of sugar and milk, and I’ve already had it today.

John: And I am drinking Spring Cherry Green tea.

Rebecca: And I’ve got a near neighbor with my Scottish afternoon breakfast.

James: Oh, nice.

Rebecca: …to James, not to John. [LAUGHTER]

John: Yeah, I’m not sure where this one comes from. It’s a Republic of Tea tea.

James: Well, I have to try the Scottish Breakfast. I’ve had the English Breakfast tea.

Rebecca: This one’s a Scottish afternoon, and there’s a morning as well, but I actually prefer the afternoon. It’s a little smoother or something.

James: Ok, Yeah, I’ll look for it.

John: And for many episodes, English afternoon was the preferred which is a little bit harder to find. Well, it’s probably comparable to the Scottish afternoon in terms of ease of locating,

Rebecca: You got to know where to look, John, you got to know where to look.

John: Well, we actually have six packs of twenty in the office where they’ve been sitting since last February when they came in.

Rebecca: We have to hurry up and drink those up.

John: So the global pandemic, which began last March, caused many faculty to shift from face-to-face instruction to online instruction, many for the first time. And we’ve seen a tremendous shift of students from face-to-face to online instruction. And that seems to have been accompanied by a fairly dramatic expansion in the use of online services that facilitate academic dishonesty. And a few years ago, at least on our campus, much of that seemed to be taking place using Course Hero. In the last couple years, much of the expansion seems to have been from Chegg. I saw a statistic recently that between April and August of 2020, the number of student uploads or questions to Chegg approximately doubled, and I think that expansion has continued since then. Why are the sites so popular?

James: Yeah, John, great summary of the challenges we face right now. I’ve done a lot of interviewing with students. And actually I stumbled into this in, I think it was June of 2019, at the American Society of Engineering Education annual conference, I think it was in Tampa. And I was doing one of the keynote breakfast things that are sponsored, McGraw Hill was sponsoring it. And we were talking about the future of educational materials. And the session was well attended by both faculty and students. We had a great panel and I was emceeing it. And the conversation came up about Chegg. Now this is pre-pandemic, and the question came up about Chegg. Many of the faculty hadn’t heard of it. All the students had heard of it. And it was an incredible conversation and eye opener. And what came out of that hour and a half meeting was, for me anyways, “Well, why were students using these type of websites?” …and there are many out there, I think Chegg is probably the leading one right now. And it was fascinating, having the conversation with the students. Of course, fast forward to the pandemic, and this all exploded. But what I’ve discovered, in having both focus groups and individual conversations with students from across the country, is that there seems to be a sense that students don’t want to leave points on the table. So if homework is worth 20%, or 25%, or 50%, they’re going to get all of those points if they can, and most students work hard to try to figure it out. But if they’re stuck, they’re not going to leave those points on the table, they’re going to go someplace to get help. And in the past, that might have been the person living in the dorm down the hall, or maybe you went to tutoring or some other source to get help. But these online sites are a couple mouse clicks away. And so the barrier to entry is so low, that if you’re struggling with a demanding curriculum and other things going on, the temptation is so great. And so what I found, in talking to students using these websites, is one of two general flavors. One was the student that was, “I’m not going to leave points on the table,” and frankly, blurring the line between academic dishonesty and trying to actually learn, and that’s sad. And we’ll probably get into that in a little bit in this conversation. The other group of students were students that were really trying to use these sites to learn, they were stuck on something, they couldn’t get past it. They’d submit, they’d take a look or search around. And to a large extent, they really want to learn. The pandemic came, everybody shifted online, and they took a look around and said, “Well, wait a minute here, this person’s cheating. That person’s cheating. I’m the one not cheating, I’m going to get the low grade because I’m really trying to do well, I’m not a cheat.” I heard this over and over again, “I’m not a cheat, but I know everyone in the class is cheating, and I’m not going to be the one getting a B, when they’re getting an A.” And so it became this crazy dynamic of a mix of “I’m not going to leave points on the table,” and “I’ll be darned everyone else is cheating.” And the barrier to entry is so incredibly low that the students who would never cheat, it’s a mouse click away. Oh, there’s the answer. Yeah, I knew that, hand it in, Sorry, long answer, John, but that sort of summarizes my exposure to it.

John: I actually had a conversation with a student just a few weeks ago that mirrors that exact discussion. Her response when we talked about this is she should have had more faith in her own ability, but she was using it as a crutch because she wasn’t that confident, so she was using this in every one of her courses. And I was just the person who happened to catch her doing it.

Rebecca: Academic dishonesty has been an issue for a very long time. But the pandemic has definitely put a spotlight on, especially this kind of digital version of academic dishonesty, even though that same mouse click was there a year or two ago, and these platforms definitely had traffic, they seem to have increased during the pandemic. Can you talk a little bit about the role of the pandemic in this particular issue?

James: Rebecca, you’re absolutely right. Unfortunately, cheating has been going on for a long time. And, usually, the way most professors adjust to that is, let’s say the homework or something might only be worth 10,15, 20%. It’s not particularly high stakes. It’s designed to be formative anyways. It’s the exams. And so in the past, they were face to face. And while young people tend to be very innovative and brilliant in solutions to circumventing academic honesty, in general, it was very hard to do. The pandemic rolls in. And now, exams are online. So we’re in a situation where the barrier to entry to cheating on an exam when it was face to face was pretty gosh darn high. Now, it’s extremely low, because you’re at home, you’re sitting in your bedroom or whatever, at the kitchen table, you’re taking an online test. Many faculty, many universities don’t use a proctoring service, we can talk a little bit about those, they’re typically a cost added, usually to the student, if they become very expensive to do institutionally. And quite frankly, I have a lot of experience that those systems don’t work, they don’t deter cheating, it’s pretty easy to cheat while these systems are being used. And I’ve unfortunately been involved with faculty that I’ve talked to, from my institution and other institutions where cheating has occurred during the exam while it was being video monitored. So in the arms race of trying to prevent cheating in the online world, we as instructors have a tendency to be a step behind. But you’re right, Rebecca, the pandemic shifted us online, including the big assessments, the big summative assessments, the exams, and we struggled with how to do it. Frankly, it was a little bit of lack of imagination on our part, and maybe an unwillingness or not recognizing that this change was afoot. The faculty that I’ve talked to that have modified their exam processes, have had some success. And interestingly enough faculty who have had upfront discussions with their students about academic honesty, and integrity, and setting standards, and a North star for yourself in terms of what your behavior is, they’ve had success in deterring cheating and academic dishonesty. But yeah, the pandemic brought it on, and it was the shift to online, plain and simple.

John: One of the things I think many teaching centers have been advocating for years is to use more online quizzing that’s automated to take some of the pressure off professors, and also to give students lots of formative assessment, as you’ve suggested, much of which is often done as a low-stakes summative assessment too, where students have multiple attempts. And so many faculty have been routinely creating these large test banks and updating them, but they pretty much all appear online pretty quickly. And the benefits of that, in many classes, have effectively disappeared. What can faculty do, without creating thousands of questions every semester, to get around this issue, to give students the benefits of that low-stakes or no-stakes formative assessment, while still making sure that students are actually learning from and ar not just looking things up on one of the services?

Rebecca: John, are you just looking for personal advice?

James: Chances are, it’s out there, John, I’m sorry to say, but hopefully…

John: I actually have been checking and I have done a few things that make it more difficult. But I’ve also been writing hundreds of new questions every week this semester.

James: Yeah, John, that’s part of the challenge is not only is it a shift in thinking for instructors, but it’s a shift in workload. So one approach is you did the work upfront, you’re writing hundreds of new questions with subtle changes, perhaps. And so it makes it very difficult to keep up with all those. The backside is to have more open-ended problems, but then your grading, so your extra work is on the backend. Part of this is having the conversation with the students, an honest conversation about their learning, connecting what you’re doing in the class, what you’re doing with the subject material to the issues that they’re going to face, perhaps in other courses, perhaps in their life and their career choice… so just an adult conversation, this is why it’s important for you to learn that, and I’ve come up with these low-stakes tests, so you can see where you’re at. And yeah, you can cheat on it. If that’s going to be your approach, then I don’t agree with that, and I think it’s eventually going to get you in trouble. So one approach, John is, yeah, you just do it, you have that conversation, and you say, “Look, it’s like taking your temperature to see if you have a fever. Just go and check. You want to know, if you have a fever, you want to know if you really have this material. I think you got to combine that when you get away from the frequent formative testing… the mantra, frequent formative assessment… when you get away from that, and you’re kind of saying, “Okay, here’s the next level test. This is a summative assessment, I need to know where you’re at with it.” That’s where the challenge is at right now, because the test banks have all been widely distributed. And unless you’re a glutton for punishment, and can write hundreds of questions, it’s going to be a real challenge. We’ve had some luck with using Gradescope. John, Rebecca, have you guys used Gradescope?

John: I have not. I know some colleagues who have, although not on our campus that I’m aware of.

James: We’ve had some, I would say modest success with it. It’s a tool that I think works best in a face-to-face type of exam, but you can do it in an online format. It helps speed up the grading process, it leverages artificial intelligence. There are some technical issues and glitches and so forth. But folks in chemistry and math have been experimenting with it with very good results. And one of the things it does is it can reduce the grading time. And that’s been the big pushback from my colleagues is “I’ll make open-ended problems, but then their grading is so hard.” One thing I did, and again, mechanical engineering, so I’ve got an advantage in that I can ask design-type questions. And what I have done in the past… this is pre-pandemic… is I would assign each student in the class slightly different parameters for a design problem. And then I was able, using some software (I think I used Mathematica or MATLAB, I don’t remember), I was able to run all the different variables and come up with approximately what their solution should look like. And so I split the difference, there was a little bit of upfront work in setting it up and a little bit of extra grading. But here’s the thing, the students loved it. And not because it was a design problem… I would say they like design problems. Here’s the insight. They loved it because they could work together, but they all had their own set of parameters that they had to do on their own. So they were like, “Oh, this is great. We got together, four or five of us got together and talked through it and explained it to each other.” But then everyone had to sit down, and kind of run the numbers for themselves. And they all diverged to slightly different solutions. That was a big insight for me. It would be interesting. I mean, I don’t know how you’d expand that to other disciplines. It would be interesting to go back and try that some more and see if that sort of assessment would get around all this. Now that said, could someone post their specific parameters on Chegg and get the answer? Yeah, unfortunately, the answer might be yes, John.

John: And in fact, I did that with my first econometrics exam when we moved online. I created seven variants of each of seven questions for a class of about 30 students, and nearly all the questions ended up online within about an hour of the test. And the first appearance was within 15 minutes of the test opening. So yeah, they can and they get a custom solution that can then be used by others. And for most of them five or six of the variants ended up appearing online very quickly.

James: Yeah, I wish we had a solution. Because if the three of us had the solution to it, we’d be going up for the initial public offering and starting up our company. But yeah, part of it is student behavior, and helping them understand what’s at stake about their learning. Part of it is changing our behavior as instructors. And while I understand sites like Chegg have introduced, I think it;s called honor shields, and so forth, the colleagues that I’ve talked to said it’s not very effective, they haven’t been happy with it. So yeah, this is a very vexing problem. And one that I don’t see a clear solution to in the future, I will say there was a math professor with a small seminar type class, and he just had oral Zoom exams with each student. He just set up a time and asked them. And the same professor, in his larger class, told students, “I might randomly contact you to explain how you solve a problem on a test.” And I think the fear of that alone probably drove students to study. [LAUGHTER] But yeah, you hate to resort to techniques like that. But there it is.

Rebecca: Especially when the platform’s themselves, like Chegg, they’re well designed for the behavior they want to occur. They’re designed in a way that rewards people uploading content, and so it’s designed in a way to be kind of effective at getting students to upload content because they want content.

James: Yep, exactly.

John: That’s actually the Course Hero model where you get free subscriptions if you upload a certain amount of graded material. And then you get access to materials that other students post. With Chegg, there’s a monthly fee, which I know, because I have to pay it just to keep track of all the cheating that’s taking place in my classes, which is really troubling. But it does have a nice interface, which is fairly similar to the interface that Netflix and similar services use. “If you like this problem, you will also like this problem.” And, in general, you can trace your way through and find many other questions from any given assessment that you posted

Rebecca: …referrals and recommendations. It’s amazing.

James: This is the Amazon model. John, one of the things that I found was interesting in talking to students who use Chegg is it recommends another problem. A number of students said that they enjoyed that, because while they saw the solution and everything, they felt that they were getting more experience with different types of problems, and they like that. But what was really interesting is when I would interview students and talk about their use of Chegg, one of the things that kept coming up, over and over again, is they liked the way, with certain types of problems in Chegg, that there could be hints. There are a whole spectrum of solutions available there. But the ones that are sort of curated, they thought were done very well, every step was explained, there was no “Oh, and it can be shown” and “then completing the algebra you get”… they showed all the steps. And some problems have hints, and you can choose to uncover the hints. And many of the students said they loved that. It was in plain language, and it showed all the steps. And maybe we should take a lesson from that when we put together course materials and study guides for our students, maybe that would be more beneficial for the students, would help them work their way through it. But I thought that was an interesting insight, that those were key aspects. Here’s another aspect, there’s not a lot of video content on Chegg. Now part of it is because “I just need to copy the answer and hand this homework in,” [LAUGHTER] but the students said, “Yeah, they didn’t really care about the video content, they were happy to read through the sort of solution walkthroughs.” And that’s interesting, because that’s sort of the opposite of YouTube, where it’s all video. And so the different learning methods and styles and approaches I thought was interesting. I just wish they didn’t cheat. [LAUGHTER]]

Rebecca: I think taking a lesson from some of those design aspects is important too. A lot of the things that students complain about is the learning management system and what that looks like and feels like or, for example, the problem sets that you’re talking about and wanting it to be in plain language rather than in language that maybe seems too difficult, or it’s not the right level of challenge. All of those things are things that could help the student maybe not want to leave your course and go somewhere else if it was built in. But it all requires a lot of time and resources and materials. And many of us don’t have that available to us with workloads expanding and especially during the pandemic having to turn around things quickly to shift gears,

James: Rebecca, you’re right, the massive shift from face to face to online, the anxiety, especially in the early days over this pandemic, and I don’t think the anxiety has gone down at all. But all that and the extra workload of learning to transition online, and figuring all this out. That’s been a consistent challenge during this whole period, that crazy shift in workload. And it’s been a challenge for all of us. It’s been a challenge for the students, certainly the types of students we’re talking about, in general, are students who went to college, they went to a residential experience. And all of a sudden we told them, [LAUGHTER] “Well, no, you’re actually attending an online institution at this point.” And Rebecca and John, I did want to point out, there was another interesting insight. I was interviewing some students on our campus, and then when I interviewed students from other universities, I was able to find similar things. And here’s what a student said to me, and I won’t use names, but they said, “Oh, in Professor X’s class, we never use Chegg.” Well, why not? “We didn’t have to. Her lectures were great. She explained everything. The homework was tied in, it made sense. And she gave us all her old exams. And she said, ‘Oh, you’re gonna know this for the exam.’” We had her old exams, and we’re like, “Oh my gosh, she’s right. We need to know this for the exam.“ And there was good support and tutorial services. There was great support. This is a quote, one student said “Chegg, Chegg who?” …obviously being a joker about it. But I thought that was fascinating, that when the course is well designed, when the material is presented in a clear way that’s student centered, when the students clearly understand how they’re going to be assessed, they know there isn’t going to be the trick problem to separate the stratospheric A from everybody else, they were like, “Yeah, we don’t need Chegg. We have everything we need from the professor and the student support services.” And when I asked students from other universities, I said, “Well, tell me about a course that you didn’t use Chegg.” And pretty much, I’d say well over half of them had “Oh, yeah, in organic chemistry, in thermodynamics, yeah, I had a great professor.” So there’s something in there that we need to learn as instructors. But that said, another quote from the same students who did use Chegg in another class, when I said “It’s academic dishonesty, you can be expelled.” And he said, “80% of the class is using it, they gonna expel all of us?” That’s an interesting perspective from a student. So yeah, but talking to students and getting their views on this has been tremendously fascinating. And it’s really helped inform a lot of the advice I give my colleagues about how to make the best of this situation. But it’s a challenge.

John: And I suspect the number of classes where it’s not being used has probably declined quite a bit, because once they’re paying that monthly fee, and now Chegg has a really nice mobile app, where you just take a picture of a problem on the screen, it uploads it, and the response comes back generally in 15 to 20 minutes, the marginal cost of engaging it in additional classes has become a lot lower for those students who might have considered it but didn’t think it was worthwhile before.

James: Yeah, the barrier to entry is now at ground level, other than the fee of $15, or whatever it is. It would be nice to engage with Chegg and other platforms. We need the students to learn this material. This is important. Let’s work together. I mean, the honor shield, okay, if it’s not working, why isn’t it working? Let’s figure out why it isn’t working on our end, working with students on academic honesty, and having a North star and having their own internal what’s right and what’s wrong, and helping young people build that set of skills and beliefs about themselves. You mentioned earlier that the young woman, she didn’t have the confidence in herself to be able to just do it. And part of it might be down the road, do departments and schools and universities and so forth, make a big fuss out of this? Is there legal action in the future? This is interesting. Where is this gonna go? I think the article in Forbes, which was very enlightening, they talked about the valuation of the company in the billions. And so if you take a very crass look at this, you got a multi billion dollar company based on cheating. That’s a hard swipe, and so forth. But let’s have a conversation.

John: And it’s basically all copyright infringement of textbook publishers’ content and faculty members’ content. So basically, they’re making millions, essentially from encouraging academic dishonesty and from infringing on everybody’s copyright. So it wouldn’t really be all that difficult, I would think, although copyright law with digital materials is a little bit tricky, because we do have the DMCA out there. And I know Chegg in particular, and I think Course Hero as well, is pretty good at responding to DMCA takedown requests, because I’ve sent dozens of them there just in the past semester, and quite a few over the last couple of years. And Chegg is actually also very good in providing faculty with information on the login ID that students use. Up until last year, in my experience at least, students were mostly using their actual college email address. Now they’ve tended to switch it a bit where they’ve created fake Gmail accounts, but they’re still logging in from the same IP address that they’re using when they submit their exams, which makes it really easy to do a look up between the exam and the person contributing the material. So there are ways of enforcing this. And if more faculty crack down on it, perhaps, it might deter a bit more of this activity. But there’s millions being made, as you said, and it might be nice if some of the publishers would work together to try to take back their ownership of the material they’ve paid to create.

James: I don’t know copyright law. But some of the problems I’ve looked at, where they were explained in more detail, whoever wrote that up, sort of wrote it up independent, they did not photocopy the instructors manual, the solution manual, they worked it out themselves. And truly I do not know what the copyright law is there.

John: The solutions, I think, would not be violations to copyright. But the photocopies of the problems and the test questions and so forth would be a violation of copyright.

James: That’s right, when we make up exams and so forth. But I mean, I’ve talked to faculty who had the exam, and had on the exam, “do not upload to Chegg,” [LAUGHTER] and it appeared, as you said, John, within 15, 20 minutes, and in those cases, it’s pretty easy to prosecute. Chegg will, as long as you go through the official academic honesty policy on campus, they’ll provide information, and that gets ugly really fast for the student. And here’s the other issue, John and Rebecca, I’ve pursued academic dishonesty cases, not involving Chegg, and it is work, it’s effort, it’s stress, and then you get the emails from the student: “You’re ruining my life” and all this kind of stuff. And It can be like, “Oh, boy, wouldn’t have been so much better if we had this conversation and you said, ‘Hey, Professor, I’m really struggling, can you give me some extra help or help me find a tutor.’” Here at Binghamton University, and I assume at many universities, students pay various comprehensive fees. tutoring is free, just go and sign up for tutoring, you’ve already paid a comprehensive fee that covers it. And I make that pitch to the first-year students all the time and say,” You’ve already paid for this, go and use it.” But yeah, I mean, think of the effort it takes for you to then go through all this work and crosswalk an IP address to this and that, I mean, that’s part of the equation we talked about, the time shift, you have to spend time on front creating assessments, on the back end grading it, and all through the process, pursuing it legally, a real challenge.

Rebecca: I think sometimes the argument, too, for students about being honest works a little better when it’s in their major, because there’s a slightly better sell of like a direct impact of “this skill set is really going to get you when you start that job and you can’t do the thing.” But we have to work on our arguments for the courses that might be in general education and things and help students recognize how those are valuable as well out in the workforce. Because I think sometimes that argument can be really compelling for students, but we have to be ready to help make it and help them want to be authentic in what they’re doing so that they can have success.

James: Absolutely. And you’re right. When I talked to students, certainly, for the courses outside of their major, they were much more willing to just survive the course, they really didn’t care. And, again, what lesson can we learn from that? Why are students feeling that way? Why are they saying that? Are those courses not connected? Or did we just not make the connection? We didn’t show them, “Oh, yes, that’s important, that gen ed is really much more important.” I’d argue, certainly in the STEM fields, as so many STEM degrees are being offered worldwide, as technology allows for so many of the things that used to require a person, now can be done by AI, that it’s our ability to work in teams, our ability to communicate, its ethics, and how do you tackle big challenging problems? I might be a mechanical engineer, I bring that background, other people bring other backgrounds and experiences. And that, what a great way to tie in general education courses to the bigger picture. Are we making that argument? Are we helping students make those connections? So, something to think about. I don’t have any answers, guys, so I’ll have to stay tuned to your podcast as you bring smarter people in to say, “Oh, well, when James said that, I have the solution.” [LAUGHTER]

John: Are there some other approaches that faculty could use in place of more traditional exams to eliminate some of the incentives and the possibility for this type of academic integrity concern?

James: Sure, there are, and they’re are more time consuming. So I’m going to be honest with myself, it’s a little easier to have 10 multiple choice questions in mechanical engineering on an exam, a little bit of a testbank, a little bit of taken from the homework, taken from my notes, super easy to grade. If you got 100 plus students, you can grade that pretty quick or you bubble source it. And I justify it, because the licensing exam is multiple choice. Well, it’s okay. It’s okay. I think we got to go back and say, “Well, maybe a judicious blend of some multiple choice questions.” Hey, you know what? The licensing exam is multiple choice. And sometimes you just got to get the right answer. With longer answer prompts that ask them to evaluate something, that’s the approach I use, and it weighs heavily on the back end. The grading is now something that you spend the weekend with a stack of papers,going through them. And you know, you guys know how that is, you can’t start grading a problem and stop halfway because you lose it in your head, right? You got the rubric in front of you. But still, you get a certain “Ok, I took a point off for that, yeah.” And so you got to sit and do that whole problem. And that’s for a lot of folks, I’ll just say, for me, that’s a shift in doing things as we shift to online. I hope that when we go back to face to face, and let’s hope it’s this fall, that we don’t forget some of these lessons, that we really should be designing better assessments that really challenge what the students know. My argument is, if you can google on it, and it’s three mouse clicks away, it’s probably not worthy of testing them on it. And the students are telling you that by saying I’m just gonna copy it. So I don’t have great solutions. I know some of the learning management systems, you can put problems in there, they’ll mix up the order, they’re timed, you can’t go back. But, the only problem I have with that is what is it we’re really testing? The time thing just puts a lot of anxiety and pressure on people and I’m not a good one under those conditions. And you know, John, I’m a “Oh shoot on problem two, wait a minute. Yeah, I did. Oh, I did that. I want to go back to problem two. I just remembered, cause in problem six, there was something similar.” You can’t do it. So yeah, there are things we do out there. But they’re not getting at the heart of what we want. And that is students to learn this material and for us to assess it in a fair and reasonable way, and help the students connect all this for whatever goals they have in their lives. What are you trying to do? Why are you in college? What’s your goal? Let’s connect what we’re doing to that goal. So Chegg’s not in front of that, that’s a deeply philosophical question for another podcast. [LAUGHTER]

Rebecca: I think combined with that, James, depending on your class size, open pedagogy or authentic assessments are also options, but it has to be the right kind of course, with the right kind of content, and the right kind of class size…

James: Yeah.

Rebecca: …for those things to all work in the mix.

James: Yeah, Rebecca, right now I’m teaching an innovation class with 10 students. [LAUGHTER] It is an absolute joy. It’s 10 motivated students, we’re using a platform called mural, which is an online collaborative platform. And so when the homework is due, I can actually go into Mural and see them doing the homework and they see each other’s homework. It’s like, yeah, we’re all going to collaborate on this. And so they can actually look and see what other people are putting in. And I mix it up, I randomize some stuff. So they have to do the reading and then they have to do parts. But, the students love it. And believe me, it’s almost like a bespoke education. This is like hand crafted somewhere between there and I used to teach some of the big sophomore engineering classes with 175 students, and I know some of your listeners who are teaching even bigger classes. That becomes almost industrial in scale. And the ability to give authentic assessments becomes very, very difficult. And until we get those AI engines up… and I’ll tell you what, when they get to the level of being able to do that. we’re all out of a job…. [LAUGHTER] or we better redefine what it is we do as educators.

Rebecca: I think what you’re pointing out, though, James, in some ways is that it tends to be those lower level classes that are bigger, those introductory classes. And so although they might get away with it, in those lower classes, they may not be thinking about the long-term game there.[LAUGHTER]

James: Yeah.

Rebecca: …because those are skills and things that they need in those upper-level classes, especially in their major, it’ll come back to get them in a way that if they don’t take an upper- level class in something else, they might not experience the same kind of consequences. But even reminding students of that, particular long-term consequences of their choices, could be useful.

James: That’s excellent. Rebecca. And as we talked earlier on, I mentioned some of the strategies that seems to work was faculty instructors talking about this, just having a very frank conversation. “Yeah, I know, all these platforms exist. Let’s talk about it. Here’s what I’m trying to do in this class.” Rebecca, I don’t think, in general, we do a great job at connecting the courses across the curriculum. And so I’m teaching my course and I do my thing. And then I hand the students off over the fence. “There you go, go take the next course,” unless we’re doing kind of self studies within our disciplines. “Hey, you know what, I teach this basic thing here at the sophomore level, and you don’t use it till last semester senior year, when does the student practice that thing?” And now we’re expecting them to pull that out of their hat and be experts at it? Why don’t we change things so that they’re constantly using it? And these are great conversations to have… a good hard look, a deep dive into how we build curriculum, helping students connect it, helping connect it to the progress they’re trying to make in their lives. Why are you studying mechanical engineering? What do you want to do with that? Let’s connect those dots through the program, so you can see where things are connected. But as long as the barrier to entry to online cheating is low, [LAUGHTER] but we got an uphill fight.

John: I have tried using some open pedagogy projects, including student-created podcasts and videos created by students, but it’s so much more work evaluating that, that it just doesn’t scale very well. I have been using them in my classes of up to 50 in my online class, but I have not yet committed to doing that in a class of 400, which I normally teach in person… last fall I taught synchronously online.

James: Wow. Yeah, you’re absolutely right. And what are the recurrent themes here? Part of it is: let’s have conversations with students, let’s rethink formative and summative assessments, are there solutions at different scales that make sense? The real challenge, Rebecca, you’re absolutely right, is the big introductory sort of classes where we’ve become used to having sometimes hundreds, if not more, students in the class, these pools of multiple choice questions that make it, and I’m using the phrase industrial and I don’t mean that in a pejorative sense at all, but it’s this ability to be able to expose students to a chemistry curriculum or mechanical engineering curriculum at scale in a very efficient way. [LAUGHTER] I mean, It’s kind of what STEM people do: how do we make this more efficient? And then a big change comes, a huge disruption, and we have to scramble. And so, I don’t see a way around the fact that this is going to be a lot of extra work. And if this were a long term shift, then absolutely, I think we’d be talking about how we restructure higher education in general and different departments and disciplines. I think because it’s short term, the concern I have is, we are going to forget all this, we’re going to wipe our hands of it and go back to face to face and go back to the old way, and not address some of the structural challenges that we’ve uncovered here: helping students understand why it’s important, and “oh, by the way, if the courses are disjointed, that’s on us, go fix that, put some effort into that, and have those conversations with the students.” That’s one of the things I always think about is what is it going to look like a year from now. And I’m concerned that we’re going to miss learning the lessons, we’re going to forget to apply these lessons, when we get back to our old ways, because we’re used to them. “I’ve got my research, my scholarship, my teaching, I’ve got it all balanced just right. I’ve been doing this for a long time, don’t make me change.” But we might have to in order to really help the students be successful. And one thing I always remember is this generation of students is going to be taking care of me when I’m old, so I want them to be good decision makers, [LAUGHTER] and have a good strong set of ethics and a moral compass. So they’re like, “oh, yeah, we have to look at society in a bigger sense.”

Rebecca: That keeps it in perspective, I think, James. [LAUGHTER]

James: Yeah, that’s right. It’s like, “Make sure I’m nice, because when I’m retired, these will be the wage earners. So I want to make sure that they keep the wheels of the economy turning and keep healthcare going and all the things we rely on, that we became painfully aware of during this pandemic, and how thin some of those threads were, how tenuous some of those systems were and are.” So really, it’s amazing how this pandemic has impacted absolutely every aspect of our lives. And we’re talking about some very specific things here. But there’s deep stuff going on here. And this is an opportunity for us to rethink how we move forward.

Rebecca: Especially because it’s easy to go back to what we had or to desire that, especially when we feel potentially burnt out with the workload and things of shifting, or just even having to have the difficult conversations with colleagues about really needing to do significant change when it’s really hard work that needs to be done.

James: Yeah.

Rebecca: So it’s easy to want to avoid it. [LAUGHTER]

James: Yeah, and I’m assuming this is a true statement. But, as a mechanical engineering professor, my training was watching my mechanical engineering professors, right? And their training was watching their professors. And so we teach the way we were taught unless we pause and take a look at the science and research and scholarship in teaching and learning, and then try to apply that. And yeah, Rebecca, you’re right, it’s work. And when you upset that balance, what’s going to happen here? And what does this mean for the future of how we do things? It would be great to not have these massive classes where you could interact with students more directly. I don’t know that that would solve the cheating problem. My students aren’t cheating in my class, there’s only 10 of them, I know them extremely well, they’re very motivated and interested, and they see where it’s connected. So how do you do that at scale and how do you do that across the curriculum?

John: On a more positive note, many faculty who had not been very involved in professional development, who had not reflected on their teaching, because they were just doing it the same way they always had, were suddenly forced to confront some new realities, and they’ve learned a lot during the past year. And I’m hoping that much of what people have learned will not be forgotten as we move past the pandemic.

James: Yeah, I agree. And I’ve been, as I’m sure you’ve seen, I’ve been just blown away by my colleagues, the huge shift, the willingness to just jump on board, to try new technology, to experiment with things and get feedback. Hey, I tried this and it really works great. And then we’ve been able to get that in the hands of other people. It was incredible to see and you’re absolutely right. So let’s pull these good pieces and bring them back. I can see myself now, even when we go face to face, just being more than happy if a student wants to Zoom with me for 15 minutes, and it’s in the evening, I could set up two evenings a week, set up an hour to do that. Like, before, I would have been: “When I get home, I’m toast. I just want to maybe catch up on my reading, catch up on my emails,” I could see that changing. And John and Rebecca, one of the things I have done, is for my entire staff, I have a weekly Zoom town hall where they can ask any questions, I give them updates. It’s fantastic. Folks really like it. And then I run a scholarship program and I just have… every two weeks, I set up a time chunk of time, vast majority of them show up, and we just chat, like, “How are things going?” And I didn’t do that. I mean, I certainly didn’t do it on Zoom before. And I might see individuals now and then, but it made me much more accessible. But in a way that was acceptable. It’s like, “Oh, I’m used to Zoom now, yeah, I don’t mind sitting in my attic studio at home and setting up a 45-minute Zoom and meeting with some students or some colleagues. So what a cool thing to be that accessible and be comfortable with it, within, obviously, within limits, and so forth. That’s a cool thing that I want to continue. A lesson we learned is that when we shifted to online, one thing became apparent right away, is there were a lot of students in socio-economic situations, they didn’t have a laptop, they didn’t have a camera, they didn’t have headphones, they didn’t have internet. And this pandemic has widened the gap to a very uncomfortable level. And so paying attention to that. And what we did at Binghamton is SUNYgave us laptops, we went out and bought a bunch of laptops, we bought mobile MiFi hotspots, 250 of them. You don’t even want to know what my monthly bill is for those… unlimited data. But we just did it. We just did it and sent it out. The other thing is we did a phone campaign just to reach out to students. And we recruited faculty and staff. And it was incredible, just to have a conversation with students. They were like, “Hey,” the overwhelming comment back was “Wow, like, thanks for contacting me.” “Yeah, I’m doing okay,” or “No, I’m not doing okay, I’ve got to take care of my younger brother and sister, my mom is working as a nurse, and it’s absolute chaos here.” So those are two things we learned in the pandemic that I hope we pay attention to, because there are students for with having all this fancy technology, that doesn’t exist. They’re in school, because there’s a computer lab and they don’t own this stuff, they don’t have it. And then the other is just reaching out to students, being a human, you know, “Hey, how’s it going?: …incredible, how powerful that is.

John: This has all become much more visible for faculty as a result of the pandemic. Those inequities were always there, but they were hidden. And now that faculty see that, it may also provide a richer appreciation of the inequities that students face as they were reaching the college level. And that’s something, as we move into the fall, that I think we’re going to see magnified because most students completed most of the last academic year remote, and some students were in well-funded school district with many resources, and all of the students and the faculty had good equipment. And in other schools, they did not have that sort of environment and much less learning occurred. So, we’re going to be faced with a student body that’s going to be experiencing greater inequities as they arrive on our campus in the fall. And I think that’s something that we all have to be prepared for.

James: John, you’re absolutely right, and well said. I think it’s always been there, and now there’s much more awareness. And I think that is something we cannot forget. And the second part of that is the students coming in… we’ve had some conversations with school districts, principals, high school principals, so forth, superintendents, and what we’re hearing is exactly what you said. It’s all over the map in terms of what that academic experience is. We know that the incoming first-year students had a pretty crazy year and a half, a lot of school districts had a “do no harm” policy. So John, if my average in your class was an 87, it didn’t matter. I couldn’t get worse than an 87. So am I motivated to work harder? Or am I going: “and thank you, I just got to take it easy.” And so I may not have learned all that material. So we have that. We have principals telling us that people have moved because of the pandemic, they had to move in with relatives. So the student isn’t showing up, they go to the house, they don’t even live there anymore. Like, where are they? So that’s a challenge. So if you think about the social aspects at home, you think about the emotional growth aspects, you think about the academic aspects, and then you add in standard test optional. The cohort of students coming in is potentially very different in a lot of ways. And we’re going to have to look at how we support them. It’s not a deficit model on their part. It’s a deficit model on our part, too, as instructors. So what are our deficits? And how do we change and modify to meet the students at a place where they can be successful? And I think that’s very important. But John, here’s the thing that was most sobering. One principal told us that it’s not just the seniors, it’s the juniors. It’s like, “Oh, yeah, so not only did the first-year’s propagate through the system, right behind them is the juniors who had a wacky…” and here’s what one principal said: “The biggest failure rate they’re seeing is in the high school sophomores.” This is a problem that isn’t going away. This is going to wash up onto the shores of higher education and if we’re not ready for it, we’re going to be in for a heck of a shock. And quite frankly, those students are going to be in for a shock. So we’ve got to figure out, what do we need to do as institutions of higher learning? And how can we best support students to be successful? No one goes to college to flunk out. They’re going because they’re trying to make progress. They’re trying to make progress in their life. Okay. How do we help them? Oh, yeah, deep and profound stuff, John and Rebecca, the effects of this are going to be with us for years to come.

Rebecca: …at least 13 years, I think, K-12. [LAUGHTER]

James: Oh yeah, right?

John: The preschoolers might have gotten past it by the time they arrive, but…

Rebecca: Yeah. A lot of preschools, I think, still maintained in person, but…

James: Yeah, so big challenges out there. I think we’re up for it. What I love about being in higher education is that we constantly question, we’re curious by nature, stubborn sometimes and have to see data in order to change our minds. But, these are all things we’re good at, and as long as we pay attention, don’t forget the lessons we’ve learned and recognize that the world has changed, and if we’re willing to figure out how we change, then I think this has a good ending. So I’m optimistic. But it’s going to be a lot of work. [LAUGHTER]

Rebecca: Definitely, and some values, of changes of things like flexibility are things that we see as value, perhaps, in students now that we didn’t see before as important skill sets and things, and adaptability.

James: Yeah, and helping students persist, and all the qualities that helped drive us to moving forward in higher education, and so forth. How do we instill some of that curiosity and work ethic? All of us, each one of us, has a unique story of why we went to school and how we want to move forward with our lives. So how do we tap into that and help students be successful? That’s what gets me up in the morning and makes me excited about my job.

Rebecca: And we always wrap up by asking what’s next?

James: Good question. So I can tell you what’s next on my horizon. I’m in the STEM field, I’m doing some research and doing some digging. I’m concerned about traditionally underrepresented groups of students in STEM fields and their success and persistence. I am very fortunate to have an NSF grant that’s sponsoring some work in this area. And when I interview and talk to students, their perspective is very interesting. For example, I was talking with a young Hispanic woman, very smart, already has a job lined up, great, great student. And she said, “Yeah, they bring guest speakers in the class, and no one looks like me, all the guest speakers look like the professor,” …which is like me, an old white guy. So where are the young people? Where are people that, I know it sounds silly, but that look different, but more importantly, have different backgrounds and different experiences and different paths to success. And that’s such an easy thing to fix at all our institutions. We have alumni of diversity who are out there. So I’m concerned about our ability to attract and retain traditionally underrepresented students in STEM fields, because that’s the pipeline for faculty in STEM. So you want to attack the faculty problem in STEM, let’s fix this problem. And you can argue let’s fix K through 12, but I can impact where I’m at right now. And down the road that’s going to impact senior-level administration. So the more people who choose an academic career, the more diverse points of view we have, the more likely that they’ll persist in the career and move into leadership. So, that’s a big problem I’m working on. That’s what’s next for me is some research and scholarship in that area. Because I want the best and brightest students no matter what their background is, because they got to take care of me when I’m old. And I’m already getting pretty old.

Rebecca: I think that’s a perfect note to end on. [LAUGHTER] Thanks so much for that conversation. Really important things to be thinking about.

James: Yeah, good stuff. And thank you so much for inviting me. I really appreciate it. Best wishes to you all. And let’s reconnect in the fall and see what lessons have we hung on to and how crazy is the fall. So, let’s circle back if you don’t mind, I’d love to catch up with you again.

John: That would be great. It’s always great talking to you. Thank you.

James: Great. Awesome. All right. Bye bye.

[MUSIC]

John: If you’ve enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer.

[MUSIC]